GFM-RAG: Graph Foundation Model for Retrieval Augmented Generation
Authors: Linhao Luo
Zicheng Zhao
Gholamreza Haffari
Dinh Phung
Chen Gong
Shirui Pan
Venue: NeurIPS 2025 poster ()
Keywords: Graph Foundation Model
Retrieval Augmented Generation
Graph-enhanced Retrieval Augmented Generation
Primary area: deep_learning
TL;DR: We introduce the first graph foundation model specifically designed for retrieval-augmented generation in large language models.

Abstract: Retrieval-augmented generation (RAG) has proven effective in integrating knowledge into large language models (LLMs). However, conventional RAGs struggle to capture complex relationships between pieces of knowledge, limiting their performance in intricate reasoning that requires integrating knowledge from multiple sources. Recently, graph-enhanced retrieval augmented generation (GraphRAG) builds a graph structure to explicitly model these relationships, enabling more effective and efficient retrievers. Nevertheless, its performance is still hindered by the noise and incompleteness within the graph structure. To address this, we introduce GFM-RAG, a novel graph foundation model (GFM) for retrieval augmented generation. GFM-RAG is powered by an innovative graph neural network that reasons over graph structure to capture complex query-knowledge relationships. The GFM with 8M parameters undergoes a two-stage training process on large-scale datasets, comprising 60 knowledge graphs with over 14M triples and 700k documents. This results in impressive performance and generalizability for GFM-RAG, making it the first graph foundation model applicable to unseen datasets for retrieval without any fine-tuning required. Extensive experiments on three multi-hop QA datasets and seven domain-specific RAG datasets demonstrate that GFM-RAG achieves state-of-the-art performance while maintaining efficiency and alignment with neural scaling laws, highlighting its potential for further improvement.

GFM-RAG: Graph Foundation Model for Retrieval Augmented Generation
Linhao Luo
Zicheng Gholammeza Haffari', Chen Gong', Dinh Phung', Shirui Pan"
Monash University, 'Nanjing University of Science and Technology,
"Shanghai Jiao Tong University, *Griffith University.
dinh. phung)hhuoottt zicheng zhao@njust. edu chen. tu. edu cn, s. wanaanngiiffe if wfw.fffiiie.eeeeee edu, au
{Linhao Luo, www.aaaree..aaaaaeee...
edu,
O Project page: https://rmast uo github. ii/iio//// ragg-m
Abstract
Netrieval-augment generation (RAG) has proven effective in integrating knowl- edge into large language models (LLMs).. However, conventional RAGs struggle to capture complex relationships between pieces of knowledge, limiting their per- formance in intricate reasoning that requires integrating knowledge from multiple sources. Recently, graph-enhanced retrieval augmented generation (GraphR AG) builds graph structure to explicitly model these relationships, enabling more effec- tive and efficient retrievers. Nevertheless, its performance is still hindered by the noise and incompleteness within the graph structure. address this, we introduce CFM-RAG, novel graph foundation model (GFM) for retrieval augmented gener- ation. GFM-RAG is powered by an innovative graph neural network that reasons over graph structure capture complex query-kma seerykknowldgge relationships. The GFM with 8M parameters undergoes two-stage training process on large-scale datasets, comprising 60 knowledge graphs with over 14M triples and 700k documents. This results in impressive performance and generalizability for GFM-RAG, making it the first graph foundation model applicable to unseen datasets for retrieval without any domain-specific fine-tuning required. Extensive experiments on three multi-hop QA datasets and seven domain-specific RAG datasets demonstrate that GFM-RAG achieves state-of-the-ar performance while maintaining efficiency and alignment with neural scaling laws, highlighting its potential for further improvement.
Introduction
Recent advancements in large language models (LLMs) 47 42 70
have greatly propelled the
evolution of natural language processing, positioning them as foundational models for artificial general intelligence (AGI). Despite the remarkable reasoning ability 488 LLMs are still limited in accessing real-time information and lack of domain-specific knowledge, which is outside the pre-training corpus. To address these limitations, strrivall-augmente generation (RAG) (12) has become a popular paradigm in adding new knowledge to the static LLMs by retrieving relevant documents into the context of LLM generation.
Existing RAG methods typically retrieve documents independently, making it difficult to capture com- plex relationships between pieces of knowledge (30 434 This limitation hampers the performance of LLMs in integrating knowledge across document boundaries, particularly multi-hop reasoning and real-world applications like legal judgment (28 and medical diagnoses 25 which
tasks 72 63
Equual Contribution.
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

require reasoning over multiple sources. Although recent methods have expanded the retrieval process into multiple steps and incorporate LLM reasoning, they still encounter high computational costs due
to iterative retrieval and reasoning with LLMs 64 59 26
Recently, graph-enhanced retrieval augmented generation (GraphRAG) 51 17 has emerged novel solution that builds a graph structure to explicitly model the intricate relationships between knowledge. This enables the development of a graph-enhanced retriever to identify relevant infor- mation using graphs. The structural nature of graphs allows GraphRAG to capture global context and dependencies among documents, significantly improving reasoning across multiple sources [9]. Methods like HippoRAG [16] enhance retrieval by employing a personalized PageRank algorithm to locate relevant knowledge with graphs. However, these algorithms rely solely on the graph structure, which is often noisy or incomplete, limiting their overall performance. Alternative methods [8]] incorporate graph neural networks (GNNs) into the retrieval process. These methods have shown impressive performance due to GNNs' powerful multi-hop reasoning capabilities on graphs 77] Nevertheless, they still face limitations in generalizability since they require training from scratch on new datasets.
Nowadays, the search for foundation GNN model that can transfer and generalize across different datasets has been an active research topic. Ideally, a foundation GNN or graph foundation model (GFM) can benefit from large-scale training and generalize across diverse graphs [40] 37.. Efforts have been made to identify transferable graph tokens (e.g., motifs, sub-trees, and relation graphs)
[1] 66 68 that can be shared among different graphs for GFM design. However, these methods primarily focus graph-related tasks (e.g., node classification and link prediction), leaving the design of a GFM to enhance L.MSs' reasoning ability unexplored.
To bridge the gap. in this paper, we propose an effective, efficient, and general graph founda- tion model for retrieval augmented generation (GFM-RAG), thereby enhancing LLMs' reason- ing ability. As shown in Figure | we create a knowledge graph index (KG-index) from doc- uments in each dataset. The KG-index con- sists of interconnected factual triples pointing to the original documents, which serves as a
Documents
KG-index
Retrieved Docs.
Doc. Ranker
Query
Answer
GFM
LLM
A
Retriever
The overview framework of GFM-RAG. structural knowledge index across multiple sources, enhancing the integration of diverse knowledge for complex reasoning tasks [16]. Then, we present the graph foundation model retriever (GFM retriever), driven by a query-dependent GNN that captures complex query-knowledge relationships in unified, transferable space of semantics and graph structure. Through multi-layer message passing, the GFM retriever enables efficient multi-hop retrieval in a single step, surpassing previous multi-step methods. The GFM retriever, with 8M parameters, undergoes a two-stage training: self-uupervise KG completion pre-training and supervised document retrieval fine-tuning on large-scale datasets, including 60 knowledge graphs with over 14M triples and 700k documents. This large-scale training ensures the generalizability of GFM retriever to be applied to unseen datasets without further training.
Figure 1:
In experiments, GFM-RAG achieves thtt--fftthe-rt performance across three multi-hop QA datasets, demonstrating its effectiveness and efficiency multi-hop reasoning. It also generalizes well across seven RAG datasets from diverse domains, such as biomedical, customer service, and general knowledge, without requiring additional training. Furthermore, GFM-RAG follows the neural scaling law [19], whose performance benefits from training data and model size scaling, emphasizing its potential as a foundational model for future improvements. The main contributions of this paper are as follows:
We introduce graph foundation model for retrieval augmented generation (GFM-RAG), powered by a novel query-dependent GNN to enable efficient multi-hop retrieval within single step.
We train large-scale model with SM parameters, marking the first graph foundation model (GFM) that can be applied directly to various unseen datasets for retrieval augmented generation.
We evaluate GFM-RAG on three multi-hop QA datasets and seven domain-specifer RAG datasets, achieving sfate-of-the-art performance across all, demonstrating its effectiveness, efficiency, generalizability, and potential as a foundational model for further enhancement.

---

2 Related Work
eetrieval-augmentte generation (RAG) 12] provides an effective way to integrate external knowl- edge into large language models (LLMs) by retrieving relevant documents to facilitate LLM genera- tion. Early works adopt the pre-trained dense embedding model to encode documents as separate vectors [3 5 43), which are then retrieved by calculating the similarity to the query. Despite efficiency and generalizability, these methods struggle to capture complex document relationships. Subsequent studies have explored multi-step retrieval, where L.LMs guide an iterative process to retrieve and reason over multiple documents 64 24 58
However, this approach is computationally
expensive.
Graph-enhanced retrieval augmented generation (GraphRAG) [5 17 is a novel approach that builds graphs to explicitly model the complex relationships between knowledge, facilitating comprehensive retrieval and reasoning. Early research focuses on retrieving information from existing knowledge graphs (KGs), such as WikiData [65 and Freebase [3, by identifying relevant facts or reasoning paths [33 38 50] Recent studies have integrated documents with KGs to improve knowledge coverage and retrieval g 35 graph structure built from these documents to aid in identifying relevant content for LLM generation [8]. Based on graphs, LightRAG [[5] incorporates graph structures into text indexing and retrieval, enabling efficient retrieval of entities and their relationships. HippoRAG [16] enhances multi-hop retrieval by using personalized PageRank algorithm to locate relevant knowledge with graphs. However, the graph structure can be noisy and incomplete, leading to suboptimal performance. Efforts to incorporate GNNs into graph-enhanced RAG [41 [8] have shown impressive results due to the multi-hop graph reasoning capabilities of GNNs in handling incomplete graphs 773. Nonetheless, these methods still limit in generalizability due to the lack of a graph foundational model.
Graph Foundation models (GFM) aims to be a large-scale model that can generalize to various datasets 40 37. The main challenge in designing GFMs is identifying graph tokens that capture invariance across diverse graph data. For instance, ULTRA [1] employs four fundamental relational interactions in knowledge graphs (KGs) to create GFM with 0.2M parameters for link prediction. OpenGraph [8]] develops graph tokenizer that converts graphs into a unified node token representa- tion, enabling transformer-like GFMs for tasks such as link prediction and node classification. GFT (66) introduces a transferable tree vocabulary to construct a GFM that demonstrates effectiveness across various tasks and domains in graph learning. Despite these successful efforts, most methods primarily focus on conventional graph-related tasks, and transformer-lis GFMs [61 60) struggle with large-scale graphs and capture logical association (52]. How to design a GNN-base GFM to
enhance the reasoning of LLM remains open question.
3 Approach
The proposed GFM-RAG essentially implements a GraphRAG paradigm by constructing graphs from documents and using a graph-enhance retriever to retrieve relevant documents.
GFM-RAG Overview. Given a set of documents D = (D1, D2,..., Dipi}: we construct a knowl- edge graph G = ((e,r, r, E & x R x E). where e, e' E E and r E R denote the set of entities and relations extracted from D, respectively. For user query q, aim to design graph-enhanced retriever to obtain relevant documents from 'D by leveraging the knowledge graph g. The whole GFM-RAG process can be formulated as:
g = KG-index(DD)
DK = GFM-Retrieverr(g (g, D, 9),
(2)
LLM(y, )
In the first step, KG-index(-) constructs a knowledge graph index & from the document corpus D. followed by our proposed graph foundation model retriever (GFM-Retrieven) on large-scale datasets. retrieves top- K documents based on any user query g and knowledge graph index G. The retrieved documents DK along with the query g, are then input into a large language model (LLM) to generate the final answer a. These three main components in GFM-RAN -RG are illustrated in Figure 2 and will be detailed next.
which is pre-trained
3

---

Graph Foundation Model for Retrieval Augmented Generation
Doc.
Dec.
Barack Osema (oern August 4, 1961, Himiikk
Decument Corpus
Honildi caphial
and most populess city.
Doc. Inwerted
paliticonn married to Ohamn
Construction
Hovell which the Pacific Ocean
Knowledge Geaph
Query Embedding
Barack
born,
Honolula
0.2 Ent. Relevance
Retrieved
Obama
palatiion_
Barack Chaami the politicion of which country?
Doc.
USA
0,9
Query Initiolizaation
Vine
Doc. Ranking
Weshingtorr
Michelle
D.C.
Dec.
Obame
Dec. 4
Q
① Query-dependent
country pomorty
The first lody the United
Message Passing
located North America
LLM Generation
Strtee from 2009 to 2017. borg
federsl union of 50
Groph Foundation
states, fecerel capitel distren Wossinghss
Obunai
GNN Model
Training Krowledge Graphs
Pairs KG- In--neesss
Supporting Dec.
Triple
Target Ent.
Question
(Bansck Obaana bortLin, ?)
GNN Model
Fourdation
Honolalu
Obanna
Barack
Graph Foundation
6NNN Model
Honoluk,
USA
Stage 1: Self seper-upeeeiied KG Completion Pre- trainnng
L
Stage 2: Supervised Document Retrieval Finetuning
The detailed framework of GFM-RAG and training processes of graph foundation model. The GFM-RAG consists of three main components: A. KG-index construction, which constructs knowledge graph index from document corpus (O); B. graph foundation model retriever (GFM retriever), which is pre-trained on large-scale datasets and could retrieve documents based on any user query and KG-index (OO); and C. documents ranking and answer generation, which ranks retrieved documents and generates final answer (00)..
Figure 2:
3.1 KG-index Construction
Conventional embedding babeeddigg-base index methods encode documents as separate vectors (0 5 45
which are limited in modeling the relationships between them. Knowledge graphs (KGs), on the other hand, explicitly capturing the relationships between millions facts, can provide a structural index of knowledge across multiple documents 9 [6]. The structural nature of the KG-index aligns well with the human hippocampal memory indexing theory (62), where the KG-index functions like an artificial hippocampus to store associations between knowledge memories, enhancing the integration of diverse knowledge for complex reasoning tasks [16].
To construct the KG-index, given set of documents D, first extract entities & and relations R to form triples T from documents. Then, the entity to document inverted index M {0, || constructed to record the entities mentioned in each document. Such a process can achieved by existing open information extraction (OpenIE) tools 77 49]. To better capture the connection between knowledge, we further conduct the entity resolution
74 to add additional edges T between entities with similar semantics, e.g., (USA, equivalent, United States of America). Therefore, the final KG-index G is constructed as G = ((e,r, leverage LLM 47
13
TUT). }. In implementation, we as the OpenlE tool (prompts shown in Table 22 and pre-trained dense embedding model 55 for entity resolution. Details can be found Appendix D.I
3.2 Graph Foundation Model (GFM) Retriever
The GFM retriever is designed to retrieve relevant documents based on any user query and the constructed KG-index. While the KG-index offers structured representation of knowledge, it still

---

suffers from incompleteness and noise, resulting in suboptimal retrieval performance when solely relying on its structure [[6]. Recently, graph neural networks (GNNs) 66 have shown impressive multi-hop reasoning ability by capturing the complex relationships between knowledge for retrieval or question answering
However, existing GNNs are limited in generalizability. as they are usually trained on specific graphs 40 371, which limits their application to unseen corpora and KGs. Therefore, there is still a need for a graph foundation model that can be directly applied to unseen datasets and KGs without additional training.
To address these issues, we propose the first graph foundation model-powered retriever (GFM retriever), which harnesses the graph reasoning ability of GNNs to capture the complex relationships between queries, documents, and knowledge graphs in a unified and transferable space. The GFM retriever employs query-dependent GNN to identify relevant entities in graphs that will aid in locating pertinent documents. After pre-training on large-scal datasets, the GFM retriever can be directly applied to new corpora and KGs without further training.
3.2.1 Query-dependent GNN
Conventional GNNs 14] follow the message passing paradigm, which iteratively aggregates informa- tion from neighbors update entity representations. Such paradigm is not suitable for the GFM retriever as it is graph-specific and neglects the relevance of queries. Recent query-dependent GNNs 78 | have shown promising results in capturing query-specific information and generalizability to unseen graphs, which is essential for the GFM retriever and can be formulated as:
H GNN, H.
where 170h E R denotes initial entity features, and Hy denotes the updated entity representations conditioned on query after L layers of query dueryddepnndent message passing.
The query-dependent GNN is theoretically proven to exhibit multi-hop logical reasoning ability 21 73 52] (detailed in Appendix [A). which is selected as the backbone of our GFM retriever. It allows the GFM retriever dynamically adjust the message passing process based on user queries and find the most relevant information on the graph with multi-hop reasoning. The path interpretation for this multi-hop reasoning process is shown in Section 4.8
Query Initialization. Given a query q, we first encode it into a query embedding with a sentence embedding model:
q SentenceEmb(()) (q), q E RRd
(5)
where d denotes the dimension of the query embedding. Then, for all the entities mentioned in the query Eg Eg S E, we initialize their entity features as while others as zero vectors:
H
q:
e C Egg
0, otherwise.
Query-depended Message Passing. The query-dependent message passing will propagate the information from the question entities to other entities in the KG to capture their relevance to the query. The message passing process can formulated as:
Triple-level:
h SentenceEmb(r). h Rdd,
(7)
m I+1 = Msg(h g ++ (KL),, (en,),)
(8)
Entity-level:
1+1 Updatee(h, Agg( (m le' e NN(e), R))),
where hi, hy denote the entity and relation embeddings at layer I, respectively. The relation em- beddings NG are also initialized using the same sentence embedding model as the query, reflecting their semantics (e.g., "born_in") and updated by layer-specific function g (-), implemented as a 2-layer MLP. The Msg(-) is operated on all triples in the KG to generate messages, which is implemented with nonpparaeetric DistMult [7] following the architecture of NBFNet (78). For each entity, we aggregate the messages from its neighbors (G( with relation r using sum and update the entity representation with a single linear layer.
5

---

After L layers message passing, a final MLP layer together with sigmoid function maps the entity embeddings to their relevance scores to the query:
Pg o (MLL((H))) Pg E
(10)
Generalizability. Since the query, entity, and relation embeddings are initialized using the same sentence embedding model with identical dimensions, the query-dependent GNN can be directly applied to different queries and KGs. This allows it to learn complex relationships between queries and entities by taking into account both the semantics and structure of the KG through training on large-scale datasets.
3.2.2 Training Process
Training Objective. The training objective of the GFM retriever maximize the likelihood of the relevant entities to the query, which can be optimized by minimizing the binary cross-entrope (BCE) loss:
LACE
Σ log log (e(e) cEAA
1 E-1
log(1 Pa(e),,
(11)
An
where A, denotes the set of target relevant entities to the query q, and E- C E A, denotes the set of negative entities sampled from the KG. However, due to the sparsity of the target entities, the BCE loss may suffer from the gradient vanishing problem [36]. To address this issue, further introduce the ranking loss to maximize the margin between the positive and negative entities:
Pg(e)
DRANK
Agg Σ EeeS Pa(e))
(12)
e,A,
The final training objective is the weighted combination of the BCE loss and ranking loss: -=aLach + (1 LRANK-
(13)
Self-supervised KG Completion Pre-training. To enhance the graph reasoning capability of the GFM retriever, we first pre-train on a large-scale knowledge graph (KG) completion task. We sample a set of triples from the KG index and mask either the head or tail entity to create synthetic queries in the form q = (e, r, ?) or (?,r,e') E'), with the masked entity serving as the target entity A, {e} or {e'}. The GFM retriever is then trained to predict the masked entity using both the query and the KG, as outlined in equation [3
Supervised Document Retrieval Fine-tuning, After self-supervised pre-training, we supervised fine-tune the GFM retriever on a labeled document retrieval task. In this task, queries q are natural language questions, and target entities Ag are extracted from labeled supporting documents D The GFM retriever is trained retrieve relevant entities from the KG index using the same training objective as in equation 13
3.3 Documents Ranking and Answer Generation
Given the entity relevance scores P. E E|x1 predicted by the GFM retriever, we first retrieve the top-T entities Ef with the highest relevance scores as:
E arg topg- =e These retrieved entities are then used by the document ranker to obtain the final documents. To diminish the influence of popular entities, weight the entities by the inverse of their frequency as entities mentioned in the document inverted index M e {0, 1} 8|x(P) and calculate the final document relevance scores by summing the weights of entity mentioned documents:
(14)
e EEE, otherwise,
Fe = EEE Mead 0,
(15)
PT=MTT F., Pa PAERRI PPI1
(16)
The top-K documents are retrieved based on the document relevance scores Pa and fed into the context of LLMs, with a retrieval augmented generation manner, to generate the final answer:
D arg top-o (Pa), D = Dk.,
(17)
LLM(g. D
(18)

---

4 Experiment
In experiments, we aim to address the following research questions: (1) How does GFM-RAG perform in multi-hop retrieval and QA tasks? (Sections
4.2 and 4.3); (2) What are the efficiency and effectiveness
of GFM-RAG in multi-hop retrieval? (Section 4.4); (3) How well does GFM-RAG generalize to unseen
datasets foundation model? (Section 4.6 (4) How does the performance of GFM-RAO RAG scale with
training as a foundation model? (Section 4.7 7 (5) How to interpret GFM-RAG multi-hop reasoning?
(Section 4.8
4.1
Experimental Setup
Datasets, We first evaluate the effectiveness of GFM-RAG on three widely-used multi-hop QA datasets, including HotpotQA (72). MuSiQue [63], and 2WikiMultHHoQAA (2Wiki) (20). We also evaluate the performance GFM RAG on seven RAG datasets from three domains, including biomedical [25],
custom support [54] 44 39 4 and general knowledge [45] 27), to demonstrate the generalizability of CFM-RAG as the foundation model. The detailed statistics of the test datasets are shown in the Appendix B
Baselines. We compare against several widely used retrieval methods under three categories: (1)
single-step nuive methods: BM25 [33, Contriever (22), GTR (46. ColBERTv2 [55]. RAPTOR [56].
Proposition 66: (2) gruph-enhanced methods: GraphRAG AG (MS) (9. LightRAG [15]. HGppoRAG [6].
SubgraphRAG (2), G-retriever [18;; (3) multi-step methods: Adaptive-RAG 23 FLARE [24] and
IRCoT (44] framework that can be integrated with arbitrary retrieval methods to conduct multi-step
retrieval and reasoning. The detailed introduction of the baselines are shown in the Appendix C
Metrics. For retrieval performance, we use recall (R @2) and recall @5 @R@5) as evaluation metrics. For final QA performance, we use the EM score and Fl score following previous works [66.
Implementation Details. The GFM retriever is implemented with 6 query-dependent message passing layers with the hidden dimension set to 512. The pre-trained all-mpnet-v2 [57] is adopted as the sentence embedding model and is frozen during training. The total parameters of the GFM retriever are 8.M, which is trained on 8 NVIDIA A A100s (80G) with batch size 4, learning rate 5e-4, and loss weight a 0.3. The training data contains 60 KGs with over 14M triples constructed from 700k documents extracted from the training set. The statistics of training data are shown in Table S and the implementations are detailed in Appendix
4.2 Retrieval Performance
We first evaluate the retrieval performance of GFM-RAG against the baselines on three multi-hop QA datasets. As shown in Table I EFM-RAG achieves the best performance on all datasets, outperforming the SOTA IRCoT + HippoRAG by 16.8% 8.3%, 19.8% in @@2 on HotpotQA, MuSiQue, and 2Wiki, respectively. The results demonstrate the efffectiveness of GFM-RAG RAG in multi-hop retrieval. From the result, we can observe that the naive single-step retrievers (c.g.. BM25, RAPTOR) are outperformed by graph-enhanced HippoRAG, which highlights the significance of graph structure in multi-hop retrieval. Although GraphRAG (MS) and LightRAG use the graph structure, it struggles with multi- hop QA tasks as retriever designed for summarization and lacks multi-hop reasoning capability. With the help of LLMs, the multi-step retrieval pipeline IRCoT improves the performance of all single-step methods through iterative reasoning and retrieval. However, GFM-RAG still outperforms the multi-step methods by large margin even with a single-step retrieval. This indicates that the GFM-RAG can effectively conduct the multi-hop reasoning in a single step (detailed in Section (48 and Appendix E.S). which is more efficient and effective than the multi-step retrieval pipeline (detailed in Section
4.3 Question Answering Performance
We then evaluate the QA performance of GFK-RAG, as it is directly influenced by retrieval quality. We adopt the GPT-4o-mini [47 as L.LM and use the top-5 retrieved documents for generating answers. From the results shown in Table 2 the single-step RAG has already achieved otate-of-the-art
7

---

Table 1: Retrieval performance comparison.
HotporQA
MuSiQue
2Wiki
Category
Method
R@2
R@5
R@2
R@5
R@2
R@5
BM25
55.4
72.2
32.3
41.2
51.8
61.9
Contriever
57.2
75.5
34.8
46.6
46.6
57.5
GTR
59.4
73.3
37.4
49.1
60.2
67.9
ColBERTv2
64.7
79.3
37.9
49.2
59.2
68.2
RAPTOR
58.1
71.2
35.7
45.3
46.3
53.8
Proposition
58.7
37.6
49.3
63.1
Single-step
GraphRAG (MS)
58.3
76.6
35.4
49.3
61.6
77.3
LightRAG
38.8
54.7
24.8
34.7
45.1
59.1
HippoRAG (Contriever)
59.0
76.2
41.0
52.1
71.5
89.5
HippoRAG (ColBERTv2)
60.5
77.7
40,9
51.9
70.7
89.1
SubgraphRAG
61.5
73.0
42.1
49.3
70.7
85.5
G-retriever
53.3
65.5
38.8
45.1
60.8
67.8
Adaptive-RAG
61.0
76.4
35.1
44.7
61.4
FLARE
73.1
81.3
44.3
55.1
67.1
73.1
IRCoT + BM25
65.6
79.0
34.2
44.7
61.2
75.6
Multi-step
IRCoT + Contriever
65.9
81.6
39.1
52.2
51.6
63.8
IRCoT + ColBERTv2
67.9
82.0
41.7
53.7
64.1
74.4
IRCoT HippoRAG (Contriever)
65.8
82.3
43.9
56.6
75.3
93.4
IRCoT + HippoRAG (ColBERTv2)
67.0
83.0
45.3
57.6
75.8
93.9
Single-step
GFM-RAG
78.3
87.1
49.1
58.2
90.8
95.6
Table 2: Question answering performance comparison.
HotpotQA
MuSiQue
2Wiki
Category
Retriever
EM
F1
EM
F1
EM
F1
None
30.4
42.8
12.5
24.1
31.0
39.0
ColBERTv2
43.4
57.7
15.5
26.4
33.4
43.3
Single-step
GraphRAG (MS)
35.3
54.6
13.4
29.5
28.3
46.9
LightRAG
36.8
48.3
18.1
27.5
45.1
49.5
HippoRAG (CoIBERTv2)
41.8
55.0
19.2
29.8
46.6
59.5
Adaptive-RAG
45.5
59.6
13.8
25.6
48.9 98.9
62.8
FLARE
48.7
60.6
16.2
28.4
46.7
65.4
Multi-step
IRCoT (CoIBERTV2)
45.5
58.4
19.1
30.5
35.4
45.1
IRCoT + HippoRAG (ColBERTv2)
45.7
59.2
21.9
33.3
47.7
62.7
Single-step Multi-step
CFM-RAG
51.6
66.9
30.2
40.4
69.8
77.7
IRCoT GFM-RAG
56.0
71.8
36.6
49.2
72.5
80.8
performance against all other baselines. Meanwhile, we also integrate GFM-RAG with IRCoT to conduct multi-step retrieval and reasoning. which further improves the performance by 8.5%, 21.2%, 3.9% in EM on three datasets, respectively. The results demonstrate the effectiveness and great compatibility of GFM-RAG with an arbitrary multi-step framework in multi-hop reasoning tasks.
4.4 Efficiency Analysis
BFM-RAG achieves great efficiency in performing multi-step reasoning single step. As shown in Table [ while the naive single-step methods get the best efficiency whose performance is not satisfying. Admittedly, the multi-step framework IRCoT could improve the performance, suffers from high computational costs due to the iterative retrieval and reasoning with LLMs. In contrast, GFM-RAG conducts multi-hop reasoning within a single-step GNN reasoning, which is more effective than single-step methods and more efficient than multi-step ones.

---

Table 3: Retrieval efficiency and performance comparison.
HotpotQA
MuSiQue
2Wiki Time (8)
Method
Time (s)
R @
Time (s)
@@5
R@5
V2
0.035
79.3
0.030
49.2
0.029
68.2
Hip
0.255
77.7
0.251
51.9
0.158
89.1
Li
0.861
54.7
1.109
34.7
0,911
59.1
GraphRAG (MS)
2.759
76.6
3.037
49.3
1.204
77.3
IRCoT ColBERTv2
1.146
82.0
1.152
53.7
2.095
74.4
IRCoT HippoRAG
3.162
83.0
3.104
57.6
3,441
93.9
GFM-RAG
0.107
87.1
0.124
58.2
0.060
95.6
Neural Scaling Law
= 0.24x 0.05 0.111y.03
R² 0.95
PubMedQA
DelucionQA
HAGRID
MNR :z
0.56
0.54
71
EManual
MS Marco
0.52
0.50
45k
24k
ExpertQA
TechQA
log10(x: #Daaa)
12k
logl0(y: #Parameters)
0.7M
GFM-RAG
HippoRAG
LightRAG
0.0RM
0.2M
Figure 3: Model generalizability comparison.
Figure 4: Neural scaling law of GFM-RAG.
4.5
Ablation Study
We conduct ablation studies to investigate the effectiveness of different components in GFM-RAG, in- cluding: different sentence embedding models (Appendix(E.I), pre-training strategies (Appendix [E.2] loss weighting strategies (Appendix F.3).. ranking methods (Appendix E.4), training datasets (Ap- pendix [. and the construction KG-index (Appendix [.9). The results show that GFM-RAG
sensitive to different sentence embedding models, and the pre-training strategy, as well as the loss weighting strategy, are both crucial for the performance of GFM-RAG.
4.6 Model Generalizability
To demonstrate the generalizability of GFM-RAG as foundation model, we test the performance (R @))) of GFM-RAG on seven RAG datasets without any domain-specific fine-tuning. Specifically, we first build the KG-index from the documents in each dataset. Then, given the query, we use the pre-trained GFM retriever to retrieve the top-K documents with the help of the corresponding KG- index. As shown in Figure 3 GFM-RAG achieves best performance on all datasets, outperforming the SOTA HippoRAG by 18.9% on average. The results demonstrate the generalizability of GFM-RAG as foundation model which can be directly applied to various unseen datasets without any domain-speciff fine-tuning, Additionally, results in Appendix E.6 demonstrat GFM-RAG's strong transferability for further performance improvement when innettund on domain fomain-specific datasets.
4.7 Model Neural Scaling Law
We further investigate the neural scaling law of GFM-RAG, which quantifies how model performance grows with the scale of training data and model parameter size. It has been validated in the recent

---

Table 4: Path interpretations of GFM for multi-hop reasoning, where r denotes the inverse of original relation.
Question
What foothall club owned by the singer of "Grow Some Funk of Your Own"?
Answer
Watford Football Club
Sup. Doc.
"Grow Some Funk of Your Own", "Elton John"l
1.095: (grow some funk your own, is a song by, elton john) - (elton john, equivalent, sir clton hercules john) + (sir clton hercules john, named stand after watford football club)
Paths
0.915: (grow some funk of your own, is a song by, elton john) - (elton john, equisalent, sir elton hercules john) - (sir elton hercules john, owned, watford football club)
Question
When was the judge born who made notable contributions to the trial of the man who tortured, raped, and murdered eight student nurses from South Chicago Commanity Hospital on the night of July 13-14, 1966?
Answer
June 1931
Sup. Doc.
"Louis B. Garippo". "Richard Speck"]
0.797: (south chicago community hospital, committed crimes at- richard speck) (richard speck, equivalent, trial of richard speck) (trial of richard speck, made contribu- tions during louis garippo)
Paths
0.412: (south chicago community hospital, were from eight student nurses) - (eight student nurses, were from, south chicago community hospital) + (south chicago community hospital, committed crimes at 1, richard speck)
foundation models 29 | As shown in Figure 4 the performance of GFM-RAG (MRR: =) scales well with the training data (x:) and the model size (y), which be fitted by the power-law scaling law ε 0.24 0.24x + 0.119.00 The results demonstrate the scalability of GFM-RAG as the foundation model and potential for further improvement. The detailed analysis of the neural scaling law is shown in Appendix E.7
4.8 Path Interpretations
GFM-RAG exhibits the multi-hop reasoning ability powered by the multi-layer GFM. We provide path interpretations of CFM-RAG for multi-hop reasoning in Table Inspired by NBFNet (78), the paths'
importance to the final prediction can be quantified by the partial derivative of the prediction score with respect to the triples at each layer (hop), defined as:
Op.(q)
$1 $2, $L = argtop- ds:
(19)
The top-k path interpretations can be obtained by the top-k longest paths with beam search. We illustrate the path interpretations in Table 4 In the first example, GFM-RAG successfully deduces that the singer of the song has a football club named after him and that he owned In the second example, CFM-RAG identifies two paths related to the murder case and the judge presiding trial. These interpretations show that CFM-RAG exhibits the ability of multi-hop reasoning within single-step retrieval. We also illustrate the distribution the multi-hop prediction in Appendix E.8
the
5 Conclusion
In this paper, we introduce the first graph foundation model for retrieval augmented generation. By leveraging the knowledge graph index, GFM-RAG explicitly models the complex relationships between knowledge and documents, facilitating more effective and efficient retrieval process. Powered by query-dependemt GNN pre-trained on large-scale datasets, GFM-RAG can effectively perform multi-hop reasoning over the graph structure to find relevant knowledge in a single step. Extensive experiments across three benchmark datasets and seven domain-specifhh datasets demonstrate that RFM-RAG significantly outperforms tfate-of-the-ar methods in effectiveness, efficiency, and generalizability. Its alignment with scaling laws also suggests the potential for scaling to even larger datasets. In the future, we plan to conduct larger-scale training and further explore GFM-RAG's capabilities in other challenging scenarios such as knowledge graph completion and question answering.
10

---

Acknowledgments
G Haffari is partly supported by the ARC Future Fellowship FT190100039 and DARPA Assured Neuro Symbolic Learning and Reasoning (ANSR) program under award number FA8750-23-2-1016
C Gong is supported by NSF of China (Nos: 62336003, 12371510). D Phung supported by the Australian Research Council (ARC) Discovery Project DP25010022 and DP230101 176. S Pan was partly funded by Australian Research Council (ARC) under grants FT210100099 and DP240101547 and the CSIRO National Science Foundation (US) AI Research Collaboration Program.
References
[1] Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D Manning. Leveraging linguistic structure for open domain information extraction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 344-354, 2015.
[2] Aijun Bai, Rolf Jagerman, Zhen Qin, Le Yan, Pratyush Kar, Bing-Rong Lin, Xuanhui Wang, Michael Bendersky, and Marc Najork. Regression compatible listwise objectives for calibrated ranking with binary relevance. Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 4502-4508, 2023.
[3] Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: collaboratively created graph database for structuring human knowledge. Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1247-1250, 2008.
[4] Vittorio Castelli, Rishav Chakravarti, Saswati Dana, Anthony Ferritto, Radu Florian, Martin Franz, Dinesh Garg, Dinesh Khandelwal, Scott McCarley, Michael McCawley, Mohamed Nasr. Lin Pan, Cezar Pendus, John Pitrelli, Saurabh Pujar, Salim Roukos, Andrzej Sakrajda, Avi Sil, Rosario Uceda-Sosa. Todd Ward, and Rong Zhang. The TechQA dataset. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1278, Online, July 2020. Association for Computational Linguistics. doi: 10. (0033//1/2220.cl-mmm org/ 2020. acl mail. 117/
URL httpps://acaanthollooot
[5] Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3- embedding: Multi-lingual, multi-functionality. multi-granularity text embeddings through self-knowledge distillation, 2023.
[6] Tong Chen, Hongwei Wang, Sihao Chen, Wenhao Yu, Kaixin Ma, Xinran Zhao, Hongming Zhang, and Dong Yu. Dense X retrieval: What retrieval granularity should we use? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 151 15159-15177, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v 1/2024. emnlp-main.845 URL
wwg.oog////222. emnlp p main.
[7] Mostafa Dehghani. Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. International Conference on Machine Learning, pages 7480-7512. PMLR, 2023.
[8] Jialin Dong, Bahare Fatemi, Bryan Perozzi, Lin F Yang, and Anton Tsitsulin. Don't 't forget to connect! improving rag with graph reranking. arXiv preprint arXiv:2405.1841. 2024.
[9] Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404. 16130, 2024.
[10] Robert Friel, Masha Belyi, and Atindriyo Sanyal. Ragbench: Explainable benchmark for retrieval-augmentedd generation systems. arXiv preprint 4xXiv:2407.11005 2024.
[11] Mikhail Galkin, Xinyu Yuan, Hesham Mostafa, Jian Tang, and Zhaocheng Zhu. Towards foundation models for knowledge graph reasoning. The Twelfih International Conference on Learning Representations, 2024.
11

---

[12] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval -uggeeendd generation for large language models: A survey, arXiv preprint arXiv:2312. 10997, 2023.
[13] Dan Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene le, and Diego Garcia-Olan Learning dense representations for entity retrieval. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 528-537, 2019.
[14] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pages 1263-1272. PMLR, 2017.
[15] Zirui Guo, Lianghao Xia, Yanhua Yu. Tu Ao, and Chao Huang. Lightrag: Simple and fast retrieval-augmmente generation. 2024.
[16] Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. Hipporag: Neurobiologically inspired long-term memory for large language models. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https
//openreview.
[17] Haoyu Han, Yu Wang, Harry Shomer, Kai Guo, Jiayuan Ding, Yongjia Lei, Mahantesh Halap- panavar, Ryan A Rossi, Subhabrata Mukherjee, Xianfeng Tang, et al. Retrieval-augmenned generation with graphs (graphrag). arXiv preprint arXiv:2501.00300 2024.
[18] Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. G-retriever: Retrieval-augmmeted generation for textual graph understanding and question answering. Advances in Neural Information Processing Systems, 37:132876-132907, 2024.
[19] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, arXiv preprint arXiv:1712.00444 2017.
[20] Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609-6625, 2020.
[21] Xingyue Huang, Miguel Romero, Ismail Ceylan, and Pablo Barceló. A theory of link predic- tion via relational weisfeiler-leman on knowledge graphs. Advances in Neural Information Processing Systems, (6:197144 197488, 2023.
[22] Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research, 2022.
[23] Soyeong Jeong, Jinheon Back, Sukmin Cho, Sung Ju Hwang, and Jong C Park. Adaptive-rag: Learning to adapt retrieval-augmentt large language models through question complexity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7029-7043, 2024.
[24] Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pages 7969-7992, 2023.
[25] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing und the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLLP), pages 2567-2577, Hong Kong, China, Novem- ber 2019. Association for Computational Linguistics. doi: 10. 1/556 URL https://aclantholoo. org D19 1259/
12

---

[26] Ashutosh Joshi, Sheikh Muhammad Sarwar, Samarth Varshney, Sreyashi Nag, Shrivats Agrawal, and Juhi Naik. Reaper: Reasoning based retrieval planning for complex rag systems. Proceed- ings of the 33rd ACM International Conference on Information and Knowledge Management, pages 4621 4628, 2024.
[27] Ehsan Kamalloo, Aref Jafari, Xinyu Zhang, Nandan Thakur, and Jimmy Lin. Hagrid: A human- IIm collaborative dataset for generative information-seeking with attribution. arXiv preprint arXiv:2307. /6883, 2023.
[28] Xiaoxi Kang. Lizhen Qu, Lay-Ki Ki Soon, Zhuang Li, and Adnan Trakic. Bridging law and data: Augmenting reasoning via a semi-structured dataset with irac methodology. arXiv preprint arXiiv:2006. 13217, 2024.
[29] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford. Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, arXiv preprint arXiv:2001.08360 2020.
[30] Vladimir Karpukhin, Barlas Oguz, Sewon Min. Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, 2020.
[31] Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv-embed: Improved techniques for training Ilms as generalist embedding models. arXiv preprint arXiv:2405. 17428, 2024.
[32] Mufei Li, Siqi Miao, and Pan Li. Simple effective: The roles of graphs and large lan- guage models in showledge-graph-basee retrieval-augmenned generation. In The Thirteenth International Conference on Learning Representations, 2025.
[33] Shiyang Li, Yifan Gao, Haoming Jiang, Qingyu Yin, Zheng Li, Xifeng Yan, Chao Zhang, and Bing Yin. Graph reasoning for question answering with triplet retrieval. In Findings of the Association for Computational Linguisties: ACL 2023, pages 3366-3375, 2023.
[34] Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.0m 032881 2023.
[35] Lei Liang, Mengshu Sun, Zhengke Gui, Zhongshu Zhu, Zhouyu Jiang, Ling Zhong, Yuan Qu, Peilong Zhao, Zhongpu Bo, Jin Yang, et al. Kag: Boosting Iims in professional domains via knowledge augmented generation. arXiv preprint 21X 2024.
[36] Zhutian Lin, Junwei Pan, Shangyu Zhang, Ximei Wang, Xi Xiao, Shudong Huang, Lei Xiao, and Jie Jiang. Understanding the ranking loss for recommendation with sparse user feedback. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5418, 2024.
[37] Jiawei Liu, Cheng Yang, Zhiyuan Lu, Junze Chen, Yibo Li, Mengmei Zhang, Ting Bai, Yuan Fang, Lichao Sun, Philip S Yu, et al. Graph foundation models: Concepts, opportunities and challenges. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[38] LINHAO LUO, Yuan-Fang Li, Reza Haf, and Shirui Pan. Reasoning graphs: Faithful and interpretable large language model reasoning. In The Twelfth International Conference on Learning Representations, 2024.
[39] Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. Expertqa: Expert-curated questions and attributed answers. arXiv preprint 23X.::330.078555 2023.
[40] Haitao Mao, Zhikai Chen, Wenzhuo Tang, Jianan Zhao, Yao Ma, Tong Zhao, Neil Shah, Mikhail Galkin, and Jiliang Tang. Position: Graph foundation models are already here. In Forty-first International Conference Machine Learning, 2024.
13

---

[41] Costas Mavromatis and George Karypis. Gnn-rag: Graph neural retrieval for large language model reasoning, arXiv preprint arXiv:2405.201.9, 2024.
[42] Meta, Build the future of ai with meta llama 2024. URL https://llama meta. com/
11 ama3/
[43] Gabriel de Souza P Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak. Benedikt Schifferer, and Even Oldridge. Ny-retriever: Improving text embedding models with effective hard-negative mining, arXiv preprint vrXiv:2407.15831 2024.
[44] Abhilash Nandy, Soumya Sharma, Shubham Maddhashiya, Kapil Sachdeva, Pawan Goyal, and NIloy Ganguly. Question answering over electronic devices: A new benchmark dataset and a multi-task learning based QA framework. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Findings of the Association for Computational Linguistics: EMNLP 2021. pages 4600-4609, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 11118865/vvl// 1/2021 win.//.iinnnnngs---aiiiaoa mandings-emnlp.399 URL https::/aclaattollleee// org. 2021. findings- emnlp .932/
[45] Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human generated machine reading comprehension dataset. November 2016.
[46] Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, et al. Large dual encoders are generalizable retrievers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9844 9855, 2022.
[47] OpenAL. Hello got-
[48] OpenAL. Learning to reason with llms, 2024. learning- to
URL https://openai..omm cot.eide///////////
[49] Liu Pai, Wenyang Gao, Wenjie Dong. Lin Ai, Ziwei Gong, Songfang Huang, Li Zongsheng, Ehsan Hoque, Julia Hirschberg, and Yue Zhang. A survey open information extraction from rule-based model to large language model. Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9586 9608, 2024.
[50] Pranoy Panda, Ankush Agarwal, Chaitanya Devaguptapu, Manohar Kaul, and Prathosh Ap. Holmes: Hyper-relational knowledge graphs for multi-hop question answering using 1lms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 12233-13222 2024.
[51] Boci Peng. Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, and Siliang Tang. Graph petrieval-augmente generation: A survey, arXiv preprint 20XV:22408.88921, 2024.
[52] Haiquan Qiu, Yongqi Zhang, Yong Li, et al. Understanding expressivity of gnn in rule learning. In The Twelfth International Conference on Learning Representations, 2024.
[53] Stephen E Robertson and Steve Walker. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In SIGIR 94: Proceedings of the Seventeenth Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval, organised by Dublin City University, pages 232-241. Springer, 1994.
[54] Mobashir Sadat, Zhengyu Zhou, Lukas Lange, Jun Araki, Arsalan Gundroo, Bingqing Wang. Rakesh Menon, Md Parvez, and Zhe Feng. DelucionQA: Detecting hallucinations in domain- specific question answering. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Find- ings of the Association for Computational Linguistics: EMNLP 2023, pages 822-835, Sin- gapore, December 2023. Association for Computational Linguistics. doi: 10. 18553/11222223.. findings-emmlp.59, URL attts://accaantholoon wwg/ findings-emmnna.. 59/
14

---

[55] Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. Colbertv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 3715-3734, 2022.
[56] Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D Manning. Raptor: Recursive abstractive processing for tree-organized retrieval. In The Twelfth International Conference on Learning Representations, 2024.
[57] SBERT. centeenc-transsormme hamp://meee...llmmmmmeee----eeee 2021. URL https: hhtg::///ggggggaggggoo co/ sentence- traassfommorsslll-mmnttttttto base---ae-t-e-ee
[58]
Weihang Su, Yichen Tang, Qingyao Ai, Zhijing Wu, and Yiqun Liu. DRAGIN: Dynamic retrieval augmented generation based on the real-time information needs of large language models. In Lun-Wa Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12991-13011 Bangkok, Thailand, August 2024, Association for Computational Linguistics. doi: 10. (0.18653311/2224.cchho acl-long.700 702/
URL htttps::/aclatahollggg...oo/ org
[59] Jiashuo Sun, Chengjin Xu, Lumingyuan Tang. Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel Ni. Heung- Yeung Shum, and Jian Guo. Think-on- Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph. In The Twelfth International Conference on Learning Representations, 2024.
[60] Jiabin Tang, Yuhao Yang. Wei Wei, Lei Shi, Lixin Su. Suqi Cheng. Dawei Yin, and Chao Huang. Graphgpt: Graph instruction tuning for large language models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 491-500, 2024.
[61] Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Long Xia. Dawei Yin, and Chao Huang. Higpt: Heterogeneous graph language model. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2842-2853, 2024.
[62] Timothy Teyler and Pascal DiScenna. The hippocampal memory indexing theory. Behavioral neuroscience, 100(2):147, 1986.
[63] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554, 2022.
[64] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with of-thugh-t reasoning for knowledge intensive multi-step questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 10014-10037, 20233
nhwwledgeintensive
[65] Denny Vrandečić and Markus Krötzsch. Wikidata: a free collaborative knowledgebase. Com- munications of the ACM, 57(10))78 85, 2014.
[66] Zehong Wang, Zheyuan Zhang, Nitesh V Chawla, Chuxu Zhang, and Yanfang Ye. GFT: Graph foundation model with transferable tree vocabulary. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL httpttt://opeeeeiiiie net/ cest/forum?in id= OMXzbAv8xy
[67] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32(1 1:44::4444 2020.
[68] Lianghao Xia, Ben Kao, and Chao Huang. Opengraph: Towards open graph foundation models. arXiv preprint arXiv:2403.01 01 121, 2024.
[69] Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.
15

---

[70]
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei. Huan Lin, Jialong Tang. Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv: 240.::2200000677 2024.
[71] Bishan Yang. Scott Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. Proceedings of the International Conference Learning Representations (ICLR) 2015, 2015.
[72] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380, 2018.
[73] Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. Qa-gnn: Reasoning with language models and knowledge graphs for question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 535-546. 2021.
[74] Alexandros Zeakis, George Papadakis, Dimitrios Skoutas, and Manolis Koubarakis. Pre- trained embeddings for entity resolution: an experimental analysis. Proceedings of the VLDB Endowment, 16(9):2255-2233) 2023.
[75] Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso- Hermmo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadch, and Jimmmy Lin. Making miracl: Multilingual information retrieval across a continuum of languages. arXiv preprint arXiv:2210.09984, 2022.
[76] Lingfeng Zhong, Jia Wu, Qian Li, Hao Peng, and Xindong Wu. A comprehensive survey on automatic knowledge graph construction. ACM Computing Surveys, 56(4):1-622 2023.
[77] Shaowen Zhou, Bowen Yu, Aixin Sun, Cheng Long, Jingyang Li, and Jian Sun. A survey neural open information extraction: Current status and future directions. In Lud De Raedt, editor, Proceedings of the Thirty-Firsi International Joint Conference on Artificial Intelligence, IJCAI- 22, pages 5694-5701. International Joint Conferences on Artificial Intelligence Organization, 7 2022. doi: 10. 01224963/jjcai.22222779 httpss///// org/ 10. 24993/ijcai..... http:/iiaaai222/77999 2022/793 Survey Track.
[78] Zhaocheng Zhu, Zuobai Zhang, Louis-Pascas Xhonneux, and Jian Tang. Neural bellman-for networks: A general graph neural network framework for link prediction. Advances Neural Information Processing Systems, 34:29476 294900, 2021.
Appendix
Table of Contents
Query-dependent GNNs for Multi-hop Reasoning and Retrieval
17
B Datascts
17
B.I Multi-hop QA Datasets
17
16

---

B.2 Domain-specilc RAG Datasets
18
Baselines
19
D Implementations Training Details
20
D.I Training Data Construction
20
D.2 Model Settings
................................
20
D.3 Training Settings
20
E
Additional Experiments
21
E,1
Elfectiveness of Different Sentence Embeddings
................................
21
Elfectiveness Different Training Strategies
................................
22
E.3
Elfectiveness Loss Weights
................................
22
E,4
Effectiveness of Ranking Methods
................................
23
E.5
Ablation Study of Training Datasets
23
E,6
Model Transferability
23
E.7
Details of Model Neural Scaling
24
Visualization of the Distribution Multi-hop Prediction
25
Cost and Impact of LLMs KG-index Construction
26
F
Prompts
27
G Limitations
27
A Query-dependent GNNs for Multi-hop Reasoning and Retrieval
We provide a detailed explanation about why query-dependent GNNs can be used for multi-hop reasoning and retrieval. Recent studies [21 52] have theoretically proven that query query-dependent GNNs are effective for capturing the multi-hop logical associations on KGs to answer queries, such as:
By: politician_of (Barack Obama, y) work_in(Barach) Obama, z Acitttyyoof(f -f v), (20)
where the right part denotes the logical associations can be executed to answer the query the left, i... poolitician_ofBBrreee Obama.y)".
This query semantic equivalent to the nature language question: "Barack Obama is the politician of which country?". By treating the input question a "soft query" (query in nature language), we apply the query-dependent GNN (GFM) to bridge the gap between nature language and logical query. The GFM tries to understand the semantic of the questions and learn to conduct complex logical reasoning (e.g., multi-hop reasoning) on KGs for retrieval [7] The learned
logical associations for reasoning shown in Section 4.8
Table 5: Statistics of the query-doc pairs and KGs used for training.
Dataset
#Q-doc Pair
#Document
#KG
#Entity
#Relation
#Triple
HotpotQA
20,000
204,822
20
1.930,362
967,218 900,338
6,393,342
MuSiQue
20,000
410,380
20
1,544,966
4,848,715
2Wiki
20,000
122,108
20
916,907
372,554
2,883,006
Total
60,000
737,310
60
4,392,235
2,240,110
14,125,063
B Datasets
B.1 Multi-hop QA Datasets
Three multi-hop QA datasets are used in our experiments: HotpotQA 72
MuSiQue 63 and
2WikiMultHopQA (2Wiki) (20). We provide a brief overview of these datasets below.
17

---

Table 6: Statistics of the datasets and constructed KG-indexes used for testing.
Dataset
Domain
#Test
#Document
#Entity
#Relation
#Triple
Ho
Multi-hop
1,000
9,221
87,768
45,112
279,112
MuSiQue
Multi-hop
1,000
11,656
100,853
55,944
319,618
2Wiki
Multi-hop
1,000
6,119
48,779
20,748
160,950
PubMedQA
Biomedical
2,450
5,932
42,389
20,952
149,782
DelucionQA
Customer Support
184
235
2,669
2,298
6,183
TechQA
Customer Support
314
769
10,221
4,606
57,613
ExpertQA
Customer Support
203
808
11,079
6,810
16,541
E.Manal
Customer Support
132
102
695
586
1,329
MS Marco
General Knowledge
423
3,481
24,740
17,042
63,995
HAGRID
General Knowledge
1,318
1,975
23,484
18,653
48,969
HotporQA [72]
multi-hop QA dataset that requires reasoning over multiple documents to answer questions. The dataset consists of 97k question-answe answer pairs, where each question is associated with up to supporting and several distracting documents. The questions are designed to be answerable using multiple pieces of information from the supporting documents.
MuSiQue 663 is a challenging multi-hop QA dataset with 25k 2-4 hop questions. requires coherent multi-step reasoning to answer questions that span multiple documents.
2WikiMultiHopQA (2Wiki) (20] is multi-hop QA dataset that requires reasoning over multiple Wikipedia articles to answer questions. The dataset consists of 192k questions, which designed to be answerable using information from
articles.
In experiments, we adhere to the official data split to obtain the training samples and follow existing methods 664 [6] use the same 1.000 samples from each validation avoid data leakage. We merge the candidate passages as the document corpus for KG-index construction. The statistics of the training and test data are presented in Table and Table respectively.
B.2 Domain-specific RAG Datasets
To test the generalizability of GFM-RAG, evaluate it domain-specific RAG datasets
[T]
including, (1) biomedical: PubMedQA (25): (2) customer support: DelucionQA
54
TechQA 4
ExpertQA 39), EManual 44; ; (3) general knowledge: MS Marco 45). HAGRID
27
We provide
brief overview of these datasets below.
PubMedQA 25 is collection of PubMed research abstracts with corresponding questions
paired with abstract chunks.
DelucionQA is domain-specific RAG dataset leveraging Jeep's 2023 Gladiator model manual as the source of knowledge, where each question is associated with 4 context documents and only relevant passage.
TechQA [4] is collection real-world user questions posted on IBMDevcloper and Devel- operWorks forums, along with 10 technical support documents relating to each question.
ExpertQA 39 is collection of curated questions from domain experts in various fields of science, arts, and law. The dataset also contains expert-curated passages relevant to each question.
EManual 44 is question-answering dataset comprising consumer electronic device manuals and realistic questions about them composed by human annotators, where each question is related with up to 3 context documents.
MS Mareo 45 is an open-domain question-anss answering dataset sourced from Bing search engine user query logs. Each question is associated with 10 context passages retrieved via Bing web search.
HAGRID 27 a multi- mingullingual information retrieval dataset with questions and passages from MIRACL (75..
18

---

In experiments, we use test sets constructed by RAGBench [0] and merge all the candidate passages as document corpus for KG-index construction. The statisties of the test dataset are detailed in Table
C Baselines
In experiments, compare with several widely used retrieval methods under three categories: (1) single-step naive methods: BM25 [53] Contriever [22], GTR [46), ColBERTv2 (55], RAPTOR (56),
Proposition [6]; (2) graph-enhanced methods: GraphRAG (MS) [9], LightRAG [15), HippoRAG
[6]; (3) multi-step methods: Adaptive-RAG [23. FLARE 24 and IRCoT 64 The detailed
introduction of the baselines is as follows.
Single-step Naive Methods are widely adopted in real-world applications due to their great efficiency and generalizability.
is classic information retrieval method based the probabilistic model that ranks a set of documents based on the query terms frequency appearing in each document.
BM25 53
Contriever [2 trains dense retriever with contrastive learning on a large-scale corpus to retrieve relevant documents for given query.
GTR [46] develops scale-up T5-based dense retriever that could generalize across different datasets and domains.
ColBERTv2 (55] is state-of-the-arr dense retriever that couples an aggressive residual compression mechanism with denoised supervision strategy to simultaneously improve the retrieval quality.
RAPTOR [56] is an LLM-augmeeted retriever that recursively embeds, clusters, and sum- marizes chunks of text, constructing a tree with differing levels of summarization to enable accurate retrieval.
Proposition (6) enhances the performance of dense retrievers by leveraging LLMs to generate natural language proposition that captures the essential information of the document.
Graph-enhanced Methods design retriever that is built upon a graph structure to conduct effective retrieval and reasoning.
GraphRAG (MS) [9] is graph-enhance retrieval method originally proposed by Microsoft. It builds a graph structure from the document corpus and use hicrarchical community detection to cluster the documents into communities and generate summary for each community. The summary together with the original documents are retrieved by the retriever for LLM generation.
LightRAG [15] is an innovative graph-enhanced RAG method that incorporates graph structures into text indexing and retrieval, enabling efficient retrieval of entities and their relationships. It employs dual-level retrieval system to gather both low-level and high-level knowledge for LLM generation.
HippoRAG [6] is a state-of-the-are training-fres frren graph-enhanced retriever that uses the Personalized PageRank algorithm assess entity relevance to query and performs multi- hop retrieval on a document-basee knowledge graph. It can be directly applied to various datasets.
Multi-step Methods are designed to conduct multi-hop reasoning by iteratively retrieving and reasoning over documents, which can be integrated with arbitrary retrieval methods.
Adaptive-RAG 23 proposes adaptive multi-step retrieval method that dynamically select the most suitable retrieval strategy based on the complexity of the query.
FLARE 24 introduces a multi-step retrieval method that actively decide when and how to retrieve documents. It also predicts the future content to the guide the retrieval in next steps.
IRCoT 64 is a powerful multi-step retrieval pipeline that integrates the retrieval with the chain-of-thought (CoT) reasoning of LLMs. It guides the retrieval with CoT and in turn using retrieved documents to improve COT. IRCoT can be compatible with arbitrary retrievers to conduct multi-step retrieval and reasoning.
19

---

Table 7: The detailed implementation and training settings of GFM-RAG.
Setting
GFM-RAG
OpenIE Entity resolution
GPT--4omini ColBERTv2
KG-index Construction
0.8
# Layer Hidden dim
6
512
Message Aggregation g'(-) Sentence embedding model
DistMult Sum
GFM Model
2-layer MLP all-mpnet-v2
Doc. ranker entities T
20
a Optimizer
AdamW
Learning rate
5e-4
KGC Pre-training
Batch size
4
Training steps
30,000
# Negative sample
128
a Optimizer Learning rate Batch size
0.3
AdamW W
5e-4
Supervised Retrieval Fine-tuning
4
Training epochs
# Negative sample
E \ Ag
D Implementations and Training Details
D.1 Training Data Construction
We extract 60,000 samples from the training set of HotpoiQA, MuSiQu, and 2Wiki to construct KG-indexes and conduct large-scale training. Specifically, we merge the candidate passages as the document corpus. In the KG-index construction, we use the GPT-4o-mini [47] with the OpenIE prompts described in HippoRAG [16] to extract the entities, relations, and triples from the document corpus. Then, we use the ColBERTv2 55 to conduct the entity resolution by computing the similarity between entities
see, e,) Emb.(e;) (eobbeeeeem((..e 'Em.,e;;)
(21)
where new triple (ei, equivalent, eạ) is generated if s(es, ez) > and e; * ej. We set the threshold 7 as 0.8 in our experiments. We divide the samples into groups of approximately lk questions and 10k documents each to control the constructed KG-index size. In the end, we obtain 60 different KG-indexes and associated question-document pairs for model training.
D.2 Model Settings
In GFM-RAG, the GFM implemented as a 6-layer query-dependent GNN with the hidden dimension of 512, DistMult message function, and sum aggregation. The relation update function g'(-) is implemented as a 2-layer MLP. We use the all-mpnet-v2 as the sentence embedding model with dimension of 768. The total training parameters of the GFM 8M. In the retrieval stage, select top T = 20 entities for the document ranker.
D.3 Training Settings
In KG completion pre-training, we randomly sample triples (e,r, () from knowledge graphs and mask out either the head or the tail entity to create a synthetic query q == (e, ?) and answer (e) in self-supervised manner. For example, given a triple (Barack Obama, born in, Honolulu),
20

---

Table 8: Comparison of different sentence embedding models used in GFM-RAG.
HotpotQA
MuSique
2Wiki
Sentence Embedding Model
R@2
R@5
R@2
R@5
R R@2 2
R@5
sentenoe-rransfomeevvlllmmmpeppe -aseev2
70.2
82.1
46.0
55.1
81.1
85.6
BAAA//bge-large-en
68.1
81.1
45.9
55.9
80.7
86.3
Aiibaba--NLPPiee--Oeeee2-- 1.t..wewnnn-.--istttreuee
69.9
81.5
46.0
55.0
79.8
86.2
68.5
81.5
45.5
55.1
80.8
85.6
nvidia/VV-Embed-v2
69.2
81.4
46.3
54.9
80.3
85.5
Table 9: Comparison of GFM-RAG with pre-trate trained and fine-tuned sentence embedding models.
HotpotQA
MuSiQue
2Wiki
Method
R@2
R@5
R@2
R@5
R@2
R@5
GFM-RAG
78.3
87.1
49.1
58.2
90.8
95.6
all all-mpnet-v2 (pre-trained)
59.4
73.3
33.2
46.3
48.5
59.4
all-mpnet-v2 (finetuned)
67.0
82.3
41.7
55.0
65.1
76.7
can create a query as (Barack Obama, born_in, in, ?), which is encoded as a sentence embedding and fed into the GFM to predict the target entity Honolulu graphs.
In supervised document retrieval fine-tuning, we obtain natural language questions and supporting documents from the multi-hop QA datasets. For each question, we identify the entities from its supporting documents as the targets. For instance, given the question "Where was Barack Obama born in?", we can extract two entities such [Honolulu, USA] from its supporting documents (e.g., Doc. in Figure 2}. The GFM is trained to maximize the likelihood of these two target entities.
In the sulf-supe KG completion pre-trate vre-training, the GFM is trained on the mixture of 60 constructed KG-indexes for 30,000 steps. Then, we conduct the supervised document retrieval fine-tuning on the labeled question-documeet pairs for 5 epochs. The weight a between losses is set to 0.3. We use AdamW optimizer, learning rate of 5e-4 with batch sizes of both training stages set to 4. Each batch contains only one KG-index and training samples associated to it, where we randomly sample from different KG-indexes during training. The model is trained on 8 NVIDIA A100s (80G) with 14 hours pre-training and 5 hours supervised fine-tuning. The detailed settings are summarized in Table 7
E
Additional Experiments
E.1
Effectiveness of Different Sentence Embeddings
In this section, we first study the effectiveness of different sentence embeddings in the GFM. We compare the all-mpnet-v2 (57), bge-large-en (69), Brre--we22-1.--insttee and gte-Qwen2-7B- instruct 34 as well as NV-Embed-v2 [31] We download the official pre-trained model from the Huggingface") The details of the models are shown in Table From the results, we can observe that the performance variance between different sentence embeddings is relatively small, where the all- mpnet-v2 achieves the best performance with respect to 3 metrics. This indicates that GFM-RAG is not sensitive to the choice of sentence embedding models. In experiments, we use the all-mpnet-v2 as the default sentence embedding model due to its efficiency. However, it has relative smaller context-size (512) which limits the length of input text. We leave the exploration larger context-size sentence embedding models (e.g., NV-Embed-y2 with 32k context) for future work.
Then, we expand our ablation study to compare GFM-RAG with variants without GNN and using solely the pre-trained all-mpnet-v2 embeddings and those fine-tuned on multi-hop QA data, respectively. The results are shown Table 9 We can observe that GNN plays crucial role in retrieval. The sentence embedding model all-mpnet-v2 is pre-trained on large-scale text data and could potentially see the QA data. However, it not specifically trained for the multi-hop QA task, which leads to
https://tugginngtoce.co/
21

---

Table 10: Effectiveneess of KGC pre-training and supervised retrieval fine-tuning in GFM-RAG.
HotpoiQA
MuSique
2Wiki
Method
R@2
R@5
R@2
R@5
@@2
R R@5
GFM-RAG
78.3
87.1
49.1
58.2
89.1
92.8
GFM-RAG w/o Retrieval Fine-tune
21.0
32.8
18.3
25.9
44.6
53.4
GFM-RAG w/o KGC Pre-train
77.8
86.5
48.3
58.3
88.3
92.5
Table 11: Knowledge graph completion result of different training strategies.
Method
MRR
Hits@
Hits @
Hits @ 10
GFM-RAG
0.193
0.138
0.221
0.293
GFM-RAG w/o Retrieval Fine-tune
0.304
0.234
0.323
0.451
RAM w/o KGC Pre-train
0.029
0.007
0.022
0.067
suboptimal performance in capturing the relationship between question and supporting documents. The fine-tuned all-mpnet-v2 achieves better performance than the pre-trained one by supervised fine-tuning the multi-hop QA data, but still inferior to GFM-RAG. This indicates that the GNN can effectively capture the relationship between knowledge and conduct multi-hop reasoning, which is not achievable by simply using the sentence embedding model.
E.2 Effectiveness of Different Training Strategies
In this section, we first study the effectiveness of the two training tasks used GFM-RAG. We compare the performance by only conducting the KG completion pre-training (GFM-RAG w/o Fine-tune) and supervised document retrieval fine-tuning (GFM-RAG w/ Pre-train). The results are shown in Table The results show that removing the supervised document retrieval fine-tuning significantly decreases the performance GFM-RAG, RAG. This highlights the importance supervised fine- tuning, as it enables the model to understand users' queries and better capture the relevance between questions and knowledge for improved retrieval.
Although the pre-training has relatively small impact on the final performance, its primary purpose is to learn the general graph reasoning ability, following previous studies like ULTRA [[]] This would enhance the generalization and robustness of the GFM, which could be beneficial to its performance on other tasks, such as knowledge graph completion. To further validate this, we conduct an ablation study to compare RFM-RAG with different training strategies the knowledge graph completion task. We report the knowledge graph completion (KGC) performance on the KG-index from the test set of the HotpotQA dataset. The results shown in Table II
From the knowledge graph completion results, we can observe that the GFM-RAG undergoes only the pre-training (GFM-RAG w/o Fine-tune) achieves the best performance, which indicates that the pre- training is effective learning the general graph reasoning ability. The performance of GFM-RAG with only supervised fine-tuning (GFM-RAG w/o Pre-train) is significantly lower than that of GFM-RAG with pre-training. This indicates that the supervised fine-tuning is only learning the specific downstream task, which would limit the generalization ability of -FM-RAG as the foundation model. The GFM trained with both pre-training and supervised fine-tuning achieves the second-best performance on knowledge graph completion task and the best performance the multi-hop QA task. This indicates that both training strategies are essential for GFM-RAC -AGG to learn the general graph reasoning ability and benefit specific downstream tasks.
E.3 Effectiveness of Loss Weights
In this section, we examine the effectiveness of the weights assigned to the BCE loss and ranking loss in training CFM-RAG. We compare performance by varying the weight between the two losses: = aLCEE + (1 ax) CRANK, with results presented Table 12 The findings indicate that using only either the BCE loss or ranking loss leads to suboptimal performance (a or 1). The best
22

---

Table 12: Effectivenees (MRR) for the weight a of two losses,
HotpoiQA
2Wiki MuSique
0
0.5189
0.3252
0,4425
0.5096
0.3214
0.4282
0.7
0.5202
0.3249
0,4348
0.3
0.5243
0.3260
0.4490
performance occurs when a is set to 0.3, which aligns with previous studies [36] suggesting that smaller weight for BCE loss is preferable when positive samples are rare in the training data.
E.4 Effectiveness of Ranking Methods
In this section, we investigate the effectiveness of different ranking methods based on inverted index used in RAG. We compare four ranking methods including (1) IDF + Top-T Pred: Our proposed method (eqs. 14) to (6)). which maps the top-T entities predicted by GFM to documents using inverse document frequency (eigh--eighted scores. (2) IDF All Pred: Uses all predicted entities
from GFM and weights them by IDF (w/ /o eq. (14). (3) Top-T' Pred: Uses only the top-T T predicted
entities without applying IDF weighting (w/o o eq. (55) (4) All Pred: Use all entity predictions and
directly map to document scores (w/o eqs. (14) and 15). The results are shown in Table 13 The results show that the proposed IDF Top-k Pred performs the best. This indicates that the inverted index is crucial component of GFM-RAG, which serves bridge between structured reasoning over KGs and the unstructured documents required by LLMs, necessitating a careful design.
We acknowledge the potential alternatives, and as promising future direction, we plan to explore end-to-een models that can jointly reason over structured and unstructured knowledge without relying on an explicit inverted index.
Table 13: Comparison of different ranking methods.
HotpotQA R@2
MuSiQue
2Wiki
Ranking Method
R@2
R@5
@
R@5
IDF + Top-T Pred (GFM-RAG)
78.3
87.1
49.1
58.2
90.8
95.6
IDF + All Pred (w/o
eq.
14]
68,1
71.4
35,8
41.2
86.0
87.5
Top-T Pred (w/o eq.
15
71.6
78.6
46.3
52.5
74.7
78.1
All Pred (w/o eqs. (14 and (15))
77.6
82.9
41.1
46.9
88.6
90.4
E.5 Ablation Study of Training Datasets
We further conducted ablation studies where GFM-RAG trained separately on each dataset, we report performance across all three benchmarks. Results are shown Table 14 These results show that GPM-RAG not only performs well on the trained datasets, but also generalizes well to other datasets. More importantly, the model trained on multi-domain datasets performs competitively across all datasets, validating its ability to generalize effectively across domains and benefit from training on diverse KGs by learning generalizable reasoning ability across domains.
E.6 Model Transferability
In this section, we evaluate GFM-RAG'S transferability by conducting domain-specific fine-tuning on the training split of dataset each domain. As shown in 15 GFM-RAG performs well in zero- shot generalization, with further improvements achieved through domain-specifie fine-tuning. This highlights its transferability when adapted to domain-speciffe datasets.
23

---

Table 14: Ablation study of GFM-RAG trained on each dataset. Best results are highlighted in bold. The second best is underlined.
Test Dataset
HotporQA
MuSiQue
2Wiki
Training Dataset
R@2
R@5
R@2
R@5
@@2
R@5
HotpotQA
79.3
87.8
46.9
57.2
86.6
92.4
MusiQue
68.8
81.8
47.6
57.5
84.4
89.6
2Wiki
72.2
77.9
46.6
55.5
89.3
93.2
All
78.3
87.1
49.1
58.2
90.8
95.6
Table 15: Model performance (R 5) and transferability comparsion.
Model
DelucionQA
EManual
ExperIQA
TechQA
MS Marco
HAGRID
HippoRAG (zero-shot)
59.0
50.0
39.5
51.1
LightRAG (zero-shot)
46.1
46.2
59.4
36.8
48.3
75.9
GFM-RAG (zero-shot)
70.8
60.6
62.7
46.6
71.0
84.7
GFM-RAG (domain-specinf specific fine-tuning)
82.7
75.9
60.8
49.5
77.5
86.6
E.7 Details of Model Neural Scaling
In this section, provide more details on the neural scaling experiments. We evaluate the changes of the model performance with respect to different parameter sizes and training data sizes. In GFM-RAG, the model parameter sizes are primarily influenced by the hidden dimension of the GFM. Thus, we vary the dimension from 32 to 512 which results in the model parameter sizes ranging from 0.08M to 8M. The detailed settings are shown in Table 16 We test models with different sizes different scales training data ranging from 3k to 45k samples. We separately report the fitted trend line of performance changing with model parameter size and training data size in Figure 5 From the trend line, we can observe that the performance of GFM-RAG increases with the model parameter size and training data size. Mcanwhile. with the larger model parameter size larger training data size is required to achieve the best performance. This indicates that the performance of EFM-RAG can be further improved by scaling up the model size and training data simultaneously.
To further investigate architectural design, we varied the number of GNN layers from keeping the hidden dimension fixed (512), and evaluated model performance across all datasets. The results are shown in Table 17 We observe that performance generally improves with deeper GNN layers, which we attribute both the increased model sizes and the ability to capture more complex multi-hop associations. This trend aligns with the neural scaling laws observed foundation models, where larger parameter counts typically yield better generalization.
while
Interestingly, we find that performance peaks around 4 layers in some cases. As discussed in Appendix and Section 4.8 GFM-RAG is designed to capture logical associations from KGs through multi-hop message passing. However, since the maximum number of reasoning hops required by our datasets is 4, additional layers beyond this offer limited benefit, likely due to the absence of higher-hop training signals. This finding supports our hypothesis that GFM-RAG effectively learns query-relevant multi-hop reasoning paths, and that deeper architectures may not improve performance without datasets requiring more complex reasoning. In summary, these results demonstrate the effectiveness and interpretability of the proposed GNN-based architecture, and confirm that both model capacity
Table The hidden dimension with corresponding model size and training batch size for scaling law analysis.
Hidden Dim.
Parameter Size
Batch size (A100, 80G)
32
78,977
40
64
215,297
20
128
659,969
20
256
2,237,441
512
8,144,897
4

---

Model Scaling
Data Scaling
3k
12k
30k
45k
3.0RM
0.7M
KM
0.59
0.59
0.58
0.58
MRR
0.55
MRR
0.55
0.54
0.54
0.53
0.53
0.50
0.50
0.49
0.49
Parameter (log scale)
(log scale)
Figure 5: The illustration of the model and data scaling law of GFM-RAG.
Table 17: The different number of layers with corresponding model size and performance for scaling law analysis.
Hidden Dim. = 512
Averge
HotpotQA
MuSiQue
2Wiki
L-Layer
R@2
R@5
R@2
R@5
R@2
R@5
R R@2
R@5
l-layer (3M)
53.9
66.7
59.3
74.2
40.7
50.2
61.8
75.7
2-layer (4M)
69.9
78.6
73.6
85.4
47.6
57.0
88.6
93.3
4-yayer (6M)
72.2
80.1
78.4
87.8
49.3
60.1
88.8
92.5
6-layer (8M)
71.9
79.6
78.0
87.0
48.4
58.7
89.3
93.1
8-layer (10M)
73.0
79.9
79.7
87.8
49.7
59.1
89.5
92.8
and logical expressibility contribute to GFM-RAG 's strong performance. We recognize the potential of other architectural designs and aim to explore them the future, inspiring the community to do the same.
E.8 Visualization of the Distribution of Multi-hop Prediction
In this section, visualize the distribution of the number of hops in the multi-hop reasoning process of GFM-RAG. We calculate the number of hops in the ground-truth reasoning path required for each question in the test set of HotpotQA, MuSiQue, and 2Wiki. Then, we compare the distribution of the number of hops in the reasoning path of the ground-truth and the predicted reasoning path by GFM-RAG well HippoRAG. The results shown in Figure (6] We can observe that the distribution of GFM-RAG is closely aligned to the ground-truth, which indicates that GFM-RAG can effectively conduct the multi-hop reasoning within single step. Meanwhile, the distribution of HippoRAC is relatively different from the bround-truth especially in 2Wiki dataset. This indicates that HG may not be able effectively capture the complex relationship to conduct multi-hop reasoning on graphs.
Table 18: The cost of the KG-index construction.
LLM
Price per 10k docs.
Total Price
GPT-4o-mini
$2.93
$216
25

---

Ground-truth (GT)
GFM-RAG
HippoRAG
HotporQA
MuSiQue
2Wiki
50
GEM-RAG is MNE: 2.03
GFM-RAG MAE
GFM-RAG GT
HippoRRA is GT 4.E:4.71
HippoRAG GT MAE 8.13
40
HippuRAG GT MAE: 16.44
Poreeningg (%)
Pareentage (5)
Perceniage
30
20
20
20
10
10
10
0
2
8
2
4
4
6
8
Hops
Hops
Hops
Figure 6: Statistics the prediction hops GFM-RAG and HippoRAG against the ground truth.
Table 19: Token cost comparison for index construction
Method
# Tokens per 10k documents
LightRAG GraphRAG GFM-RAG
55M
76M
48M
E.9 Cost and Impact of LLMs on KG-index Construction
In this section, we first analyze the cost of the KG-index construction. In experiments, we utilize GPT- 4o-minf
Fl for OpenlE extraction and construct the KG-index for 737,310 documents. The cost is shown Tables 18 and Specifically, we find that constructing the KG-index requires approximately 48M tokens per 10k documents, which costs around $2.6 using GPT-4o-mini. LightR and GraphRAG cost 57M tokens and 76M tokens, respectively. Compared to other methods, GFM-RAG more cost-effective it does not require generating community-level summaries. In addition, we compare the graph index construction time of GFM-RAC in Table Results show that GFM-RAG benefits from the quick index process during retrieval, it does not construct traditional vector database to store documents and entities.
Admittedly, using LLM for KG index construction incurs computational costs. However, KG construction has been extensively studied, and numerous alternative methods exist that do not rely on LLMs (76). Our implementation offers an easy interface for integration with any KG construction tools. We would explore the use of other KG construction methods future work.
We further analyze the impact of LLMs used for KG-index construction on the performance of GFM-RAG. We conduct experiments using different LLMs for KG-index construction, including GPT-4o-mini and GPT-3.5-turbe Then, we reevaluate the performance of GFM-RAG and HippoRAG with the constructed KG-index. The results are shown in Table 2T From the results, the performance of both methods on the KG extracted by GPT-4o-mini is higher than the ones by GPT-3.5-turbe This supports the opinion that GPT-4o-min generally outperforms GPT-3.5-turis in constructing high quality KG-index, which is crucial for the graph-enhanced retrieval. However, the performance
http:p/aaaa..cooccooeeeeeeeeeeee gpt turbo
Table 20: Graph Indexing time comparison.
Method
Indexing time (s)
LightRAG GraphRAG (MS)
1430.32
1796.43
RAG
93.55
26

---

Table 21: Comparison of the model performance under the KG-index constructed by different LLMs.
HotpoiQA
MuSiQue
2Wiki
Method
R@2
R@5
R@2
R@5
R@2
R@5
GFM-RAG (gpt-4o-mini)
78.3
87.1
49.1
58.2
90.8
95.6
HippoRAG (gpt-4o-mini)
62.2
79.3
41.7
53.6
72.1
89.5
GFM-RAG (gpt-3.5-trubo)
75.6
84.7
46.1
55.8
85.2
90.4
HippoRAG (gpt-3.5-trubo)
60.5
77.7
40.9
51.9
70.7
89.1
of GFM-RAG is significantly higher than HippoRAG under both KG-indexes. This indicates that GFM-RAG is more robust the quality of the KG-index, demonstrating the effectiveness of the GFM in graph reasoning and retrieval.
Prompts
In experiments, follow the prompts used in HippoRAG [6] extract the triples from the document corpus, which is shown in Table 22
G Limitations
The limitations of GFM-RAG are as follows: (1) The construction of KG-index can be costly and time-consuming, especially when using LLMs for OpenIE extraction. We would explore the of efficient KG construction methods in future work and optimize the construction process. (2) The model size of the GFM is relatively small (8M) compared to other foundation models like large language models with billions of parameters. Although is not faired to directly compare the GNN-based model with wansformer-basee LLMs, would explore the scaling of GFM-RAG in future work to improve its performance and generalizability. (3) CFM-RAG is only evaluated on multi-hop QA tasks and KG completion tasks. We would explore the capabilities of GFM-RAG in other tasks such as knowledge graph question answering and knowledge graph reasoning in future work to validate its effectiveness foundation model.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of work, none which we feel must be specifically highlighted here.
27

---

OpenlE Prompt
Instruction:
Your task is to construct an RDF (Resource Description Framework graph from the given passages and named entity lists. Reapond with JSON list of triples, with each triple representing a the following
relationship in the RDF graph. Pay attention requirements:
Each triple should contain at least one, but preferably two, of the named entities in the list for each passage. Clearly resolve pronouns their specific names maintain clarity.
Convert the paragraph into JSON dict, it has a named entity list and triple list.
Shot Demonstration:
Paragraph:
Radio City Radio City India's first private FM radio station and was started on July 2001. It plays Hindi, English and regional songs. Radio City recently forayed into New Media in May 2008 with the launch of music portal PlanetRadiocity, com com that offers music related news, videos, songs, and other music- related features. ccc
["Radio City", "India", July 2001", "Hindi", "English", May 2008", "FlanetRadiocity. com")
"triples", "; [
["Radio City", "located in", "India"],
["Radio City", "is", "private FX radio station"),
["Radio City", "started on", "3 July 2001"].
["Radio City", "plays songs in", "Hindi"], ["Radio City", "plays songs in" "English"] ["Radio City", "forayed into", "New Media"], ["Radio City", "launched", "PlanetRadiocity.c com"), ("PlanetRadioctt..com "launched in", "May 2008"], ["PlanetRadiocity.cort "is", "music portal"), ["PlanetRadiocity.com "offers", "news"], ["PLanetRadioctty.com com", "offers", "videos"], ("PlanetRadiocity: com", "offers", "songs"]
## Input
Convert the paragraph into JSON dict, it has named entity list and triple list. Paragraph:
rce
INPUT PASSAGE <cc
Table 22: The prompt for OpenlE extraction.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of the work Appendix G
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have detailed data construction process, model settings, and training process in Appendix D to ensure the reproducibility of our results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
30

---

Answer: [Yes]
Justification:
We have uploaded the code to an anonymous link in the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee //nips.com / cassiicyncccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We have detailed experiment settings in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification:
The experiments conducted with fixed random seed and no error bars reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
31

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The resourees used for the experiments are detailed in Appendix D.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform. in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
The research conducted in the paper complies with the NeurlPS Code of Ethics.
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The proposed method focuses on the technical aspects of the problem and do not include societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
32

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper utilizes existing datasets and pretrained models that are already released which have safeguards place.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have properly cited all code, data, and models used in our research and complied with the licensing agreements and terms of use set by the original authors.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
33

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
There are no new assets introduced in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
34

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: The usage of the LLM is described and discussed in Appendices D and D.I
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy for what should or should not be described.
35


Reviews:
[{"review_id": "wPeKRyQ0YL", "reviewer": "Reviewer_fCb4", "summary": "The paper presents GFM-RAG, a novel graph foundation model designed to enhance retrieval-augmented generation (RAG) by leveraging a graph neural network to capture complex relationships within a knowledge graph. This approach enables more effective and efficient retrieval of relevant information, addressing the limitations of traditional RAG methods that struggle with multi-hop reasoning tasks. GFM-RAG constructs a knowledge graph index from documents, representing knowledge as interconnected triples, which facilitates multi-hop reasoning and integrates diverse knowledge sources. The model demonstrates state-of-the-art performance on multi-hop question answering datasets and shows strong generalizability across various domains. It maintains efficiency by performing multi-hop reasoning in a single step and adheres to neural scaling laws, indicating potential for further improvements. Overall, GFM-RAG offers a significant advancement in RAG systems, providing improved reasoning capabilities and generalizability.", "questions": "See Summary Of Weaknesses. For each corresponding weakness, we have provided the relevant concerns and suggestions to address those concerns.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n1. Innovative Model Design\nThe introduction of the GFM-Retriever model is a significant strength. By leveraging the capabilities of Graph Neural Networks (GNNs), the model effectively helps Large Language Models (LLMs) understand and navigate graph-structured knowledge. This innovative approach addresses the limitations of traditional RAG methods, which often fail to capture complex relationships between pieces of knowledge. The use of a query-dependent GNN allows for dynamic adjustment based on user queries, enhancing the model's ability to perform multi-hop reasoning efficiently.\n\n2. Robust Experimental Setup and Results\nThe experimental setup is thorough and robust, covering a wide range of datasets and benchmarks. The authors provide extensive experimental results that validate the effectiveness and efficiency of GFM-RAG. The model demonstrates state-of-the-art performance on multi-hop question answering datasets and shows strong generalizability across various domains. This evidence strongly supports the claims made in the paper.\n\n3. Comprehensive Ablation Studies\nThe paper includes comprehensive ablation studies that explore the impact of different components of the model, such as different sentence embedding models, pre-training strategies, loss weighting strategies, and the construction of the knowledge graph index. These studies provide valuable insights into the model's performance and the importance of each component, demonstrating the authors' commitment to ensuring the robustness of their findings.\n\n\nWeakness\n1. Lack of Ablation Study on Ent-Document Inverted Index\nGFM-RAG differs from methods like LightRAG, which directly retrieve knowledge to answer queries, by using an ent-document inverted index to map retrieved entities back to source documents, which are then used as context for the LLM. This additional step is crucial for the model's operation, yet the paper does not include an ablation study to evaluate its contribution and importance. Such a study would clarify the significance of the ent-document inverted index in the overall performance of GFM-RAG, providing insights into whether this component is essential or if alternative approaches could achieve similar results more efficiently.\n\nThis can also lead to increased computational overhead and longer inference times compared to methods that do not require this step. While Table 3 compares retrieval efficiency and performance, it does not include a direct comparison with methods like GraphRAG or LightRAG regarding inference times. Providing experimental data on the inference time differences would help assess whether the additional steps in GFM-RAG significantly increase computational costs and whether these costs are offset by performance gains.\n\n2. Training Strategy and KG-Index Construction\nGFM-RAG constructs 60 KG-indices from 60,000 documents across three datasets to train a single GFM-Retriever, whereas methods like GraphRAG and LightRAG typically do not require training and instead use directly extracted knowledge triples, community reports or other info. The paper does not explain why a single GFM-Retriever is trained across multiple datasets rather than training separate retrievers for each dataset. An ablation study or an explanation comparing these strategies would provide insight into the efficiency and effectiveness of the chosen approach. Additionally, providing data on the token cost of constructing the KG-index, similar to what LightRAG offers, would help in understanding the resource requirements and facilitate a comparison of the cost-effectiveness of GFM-RAG with other methods. This information would give readers a clearer picture of the trade-offs involved in the model's design.", "was_revised": false, "final_justification": "", "rebuttal": "## W1.1: Lack of ablation study on ent-document inverted index\n\nWe sincerely thank the reviewer for highlighting this point. We have conducted a detailed ablation study to compare several alternative inverted index strategies:\n\n* **IDF+Top-T Pred:** Our proposed method (Eq. 14–16), which maps the top-T entities predicted by GFM to documents using inverse document frequency (IDF)-weighted scores.  \n* **IDF+All Pred:** Use all predicted entities from GFM and weights them by IDF (w/o Eq 14\\)  \n* **Top-T Pred:** Use only the top-T predicted entities without applying IDF weighting (w/o Eq 15).  \n* **All Pred:** Use all entity predictions and directly map to document scores (w/o Eq 14 and 15).\n\n**Ablation study of different inverted indices.**\n\n| Inverted Index | HotpotQA |  | MuSiQue |  | 2Wiki |  |\n| :---- | ----- | ----- | ----- | ----- | ----- | ----- |\n|  | R@2 | R@5 | R@2 | R@5 | R@2 | R@5 |\n| IDF+Top-T Pred (GFM-RAG) | **78.3** | **87.1** | **49.1** | **58.2** | **90.8** | **95.6** |\n| IDF \\+ All Pred | 68.1 | 71.4 | 35.8 | 41.2 | 86.0 | 87.5 |\n| Top-T Pred | 71.6 | 78.6 | 46.3 | 52.5 | 74.7 | 78.1 |\n| All Pred | 77.6 | 82.9 | 41.1 | 46.9 | 88.6 | 90.4 |\n\nThe results show that the proposed **IDF \\+ Top-k Pred** performs the best. This indicates that the inverted index is a crucial component of GFM-RAG, which serves as a bridge between structured reasoning over KGs and the unstructured documents required by LLMs, necessitating a careful design.\n\nWe acknowledge the reviewer’s concern regarding potential alternatives, and as a promising future direction, we plan to explore end-to-end models that can jointly reason over structured and unstructured knowledge without relying on an explicit inverted index.\n\n## W1.2: Inference time comparison\n\nWe conducted an empirical analysis comparing **GFM-RAG**, **GraphRAG,** and **LightRAG** in terms of both **retrieval latency (Time in seconds)** and **retrieval quality (Recall@5)**. The results are shown below:\n\n**Inference time comparison.**\n\n| Method | HotpotQA |  | MuSiQue |  | 2Wiki |  |\n| ----- | :---: | :---: | :---: | :---: | :---: | :---: |\n|  | Time (s) | R@5 | Time (s) | R@5 | Time (s) | R@5 |\n| LightRAG | 0.861 | 54.7 | 1.109 | 34.7 | 0.911 | 59.1 |\n| GraphRAG (MS) | 2.759 | 76.6 | 3.037 | 49.3 | 1.204 | 77.3 |\n| GFM-RAG | **0.107** | **87.1** | **0.124** | **58.2** | **0.060** | **95.6** |\n\nThe results clearly demonstrate that GFM-RAG outperforms both LightRAG and GraphRAG in retrieval accuracy while also being significantly more efficient in terms of inference time. Despite performing the inverted index, GFM-RAG achieves superior latency because the reasoning is executed in a single GNN forward pass over a pre-constructed knowledge graph index.\n\nIn addition, GFM-RAG also benefits from the quick index process during retrieval, as it does not construct a traditional vector database to store documents and entities:\n\n**Indexing time comparison.**\n\n| Method | Indexing time (s) |\n| :---- | :---- |\n| LightRAG | 1430.32 |\n| GraphRAG (MS) | 1796.43 |\n| GFM-RAG | 93.55 |\n\nWe will include these results in the final version to improve clarity and better highlight the advantages of GFM-RAG.\n\n## W2.1: Ablation study of training datasets\n\nWe thank the reviewer for raising this valuable point. We want to confirm that GFM-RAG is trained on multiple datasets (HotpotQA, MuSiQue, and 2Wiki), but we ensure **strict separation between training and evaluation data**, with no overlap in documents or entities, thus eliminating any risk of information leakage.\n\nThe motivation for this setup is rooted in the **foundation model paradigm**. Similar to how LLMs are trained on diverse corpora to learn generalizable linguistic patterns, GFM-RAG aims to learn **universal graph reasoning capabilities** from diverse KG-indexes. This design enables strong **zero-shot transfer to new domains and datasets**, eliminating the need for retraining or fine-tuning on every downstream domain. Results in Fig. 3 demonstrate the strong generalizability of GFM-RAG on out-of-domain datasets without retraining, compared to GraphRAG and LightRAG.\n\nWe further conducted ablation studies where **GFM-RAG is trained separately on each dataset**, and we report performance across all three benchmarks. Results are shown below:\n\n**Ablation study of GFM-RAG trained on each dataset.**\n\n| Test Dataset | HotpotQA |  | MuSiQue |  | 2Wiki |  |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| **Training Dataset** | R@2 | R@5 | R@2 | R@5 | R@2 | R@5 |\n| HotpotQA | **79.3** | **87.8** | 46.9 | 57.2 | 86.6 | 92.4 |\n| MusiQue | 68.8 | 81.8 | 47.6 | 57.5 | 84.4 | 89.6 |\n| 2Wiki | 72.2 | 77.9 | 46.6 | 55.5 | 89.3 | 93.2 |\n| All | 78.3 | 87.1 | **49.1** | **58.2** | **90.8** | **95.6** |\n\nThese results show that GFM-RAG not only performs well on the trained datasets, but also generalizes well to other datasets.  More importantly, the **model trained on multi-domain datasets** performs competitively across all datasets—validating its ability to **generalize effectively across domains** and benefit from training on diverse KGs by learning generalizable reasoning ability across domains.\n\nWe will include this ablation study and further clarify the design rationale in the final version of the paper.\n\n## W2.2: Token cost for index construction \n\nIn **Appendix E.7**, we report the **token cost** during index construction. Specifically, we find that constructing the KG-index requires approximately **4.8 M tokens per 1,000 documents**, which costs around **$2.6** using GPT-4o-mini. LightRAG and GraphRAG cost 5.7 M tokens and 7.6 M tokens, respectively. Compared to other methods, GFM-RAG is more cost-effective as it does not require generating community-level summaries.\n\n**Token cost comparison for index construction**\n\n| Method | \\# Tokens per 1000 documents |\n| :---- | :---- |\n| GFM-RAG | 4.8M |\n| LightRAG | 5.5M |\n| GraphRAG | 7.6M |"}, {"review_id": "ww4PizDq1B", "reviewer": "Reviewer_9ZFX", "summary": "The proposed GFM-RAG essentially implements a GraphRAG paradigm by constructing graphs from documents and using a graph-enhanced retriever to search relevant documents. It involves two stages of large-scale training to achieve strong generalization with 8M parameters and adhere to neural scaling laws.", "questions": "1.\tThe network architecture of graph foundation model is mainly composed of graph neural networks. The proposed version with 8M parameters is implemented with 6 graph neural networks in the appendix. Why graph foundation model should be backboned with graph neural networks?\n2.\tGraph foundation model assumes relatively stable knowledge graphs, struggling with rapidly evolving domains without frequent retraining.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.\tGFM-RAG pipeline is efficient in conducting multi-hop reasoning with single-step retrieval.\n2.\tCFM-RAG pipeline achieves the state-of-the-art performances across 3 multi-hop QA datasets and seven domain-specific datasets.\n3.\tCFM-RAG follows the neural scaling laws.\n\nWeaknesses:\n\n1.\tThe main novelty of GFM-RAG lies in the pipeline. But each module in the pipeline is fundamental and conventional, lacking novelty, such as the network architecture of graph foundation model.\n2.\tThe choice of network architectures and training methods should be considered in the ablation study.\n3.\tThere is a gap between the best performance mentioned in https://hotpotqa.github.io/ and the performance of the proposed method in Table2.", "was_revised": false, "final_justification": "", "rebuttal": "## W1: Novelty of pipeline components\n\nWe understand the reviewer’s concern that individual modules in GFM-RAG may appear conventional. However, the **core novelty lies in the integration and scaling** of these components into a **unified graph-based foundation model** designed for **retrieval-augmented generation** (RAG).\n\nTo our knowledge, GFM-RAG is the **first scalable foundation model for graph-based retrieval**, enabling **single-step multi-hop reasoning** and **adhering to neural scaling laws**, with strong **zero-shot generalization across domains**. These contributions go beyond reusing standard modules, aiming to elevate the RAG paradigm using GNN-enabled foundation models and pretraining strategies.\n\n## W2: Architecture and training method ablation\n\nWe appreciate the reviewer’s suggestion to provide a deeper analysis of the architectural and training design choices. We have conducted a comprehensive set of ablation studies, with results summarized in **Appendix E**:\n\n* In **Appendix E.1**, we compare different sentence embedding models used in GFM-RAG.\n\n* In **Appendix E.2**, we evaluate the effectiveness of the two training tasks (KG completion and document retrieval) and their impact on downstream performance.\n\n* In **Appendix E.3**, we examine the sensitivity of GFM-RAG to different task loss weightings.\n\n* In **Appendix E.5**, we analyze the impact of model parameter size on performance.\n\n**To further investigate architectural design**, we varied the number of GNN layers from 1 to 8 while keeping the hidden dimension fixed (512), and evaluated model performance across all datasets. The results are shown below:\n\n**Ablation study of different GNN layers.**\n\n| Method | Averge |  | HotpotQA |  | MuSiQue |  | 2Wiki |  |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| N\\_dim=512 | R@2 | R@5 | R@2 | R@5 | R@2 | R@5 | R@2 | R@5 |\n| 1-layer (3M) | 53.9 | 66.7 | 59.3 | 74.2 | 40.7 | 50.2 | 61.8 | 75.7 |\n| 2-layer (4M) | 69.9 | 78.6 | 73.6 | 85.4 | 47.6 | 57.0 | 88.6 | 93.3 |\n| 4-layer (6M) | 72.2 | **80.1** | 78.4 | 87.8 | 49.3 | **60.1** | 88.8 | 92.5 |\n| 6-layer (8M) | 71.9 | 79.6 | 78.0 | 87.0 | 48.4 | 58.7 | 89.3 | 93.1 |\n| 8-layer (10M) | **73.0** | 79.9 | **79.7** | **87.8** | **49.7** | 59.1 | **89.5** | **92.8** |\n\nWe observe that **performance generally improves with deeper GNN layers**, which we attribute to both the **increased model sizes** and the **ability to capture more complex multi-hop associations**. This trend aligns with the **neural scaling laws** observed in foundation models, where larger parameter counts typically yield better generalization.\n\nInterestingly, we find that **performance peaks around 4 layers** in some cases. As discussed in **Appendix A and Section 4.8**, GFM-RAG is designed to capture logical associations from KGs through multi-hop message passing. However, since the **maximum number of reasoning hops required by our datasets is 4**, additional layers beyond this offer limited benefit—likely due to the absence of higher-hop training signals. This finding supports our hypothesis that **GFM-RAG effectively learns query-relevant multi-hop reasoning paths**, and that deeper architectures may not improve performance without datasets requiring more complex reasoning.\n\nIn summary, these results demonstrate the **effectiveness and interpretability** of the proposed GNN-based architecture, and confirm that **both model capacity and logical expressibility contribute to GFM-RAG’s strong performance**. We recognize the potential of other architectural designs and aim to explore them in the future, inspiring the community to do the same.\n\n## W3: Performance gap with HotpotQA leaderboard\n\nWe thank the reviewer for pointing out the discrepancy with the leaderboard performance. As stated in Section 4.1 and Appendix B, to ensure **fair comparison** with recent methods such as **HippoRAG** and **IRCoT**, we adopt the same experimental setup that uses **the** **same** **1,000 randomly sampled HotpotQA subsets** from the validation set, rather than the full test set. This setting is widely adopted in prior LLM-based multi-hop QA works to **reduce computational costs associated with LLM-based generation** while enabling reproducible and meaningful benchmarking.\n\nTherefore, the performance reported in Table 2 is not directly comparable to the leaderboard numbers based on the full test set. We will clarify this in the final version and note this distinction more explicitly to avoid confusion.\n\n## Q1: Why use GNNs as the backbone for GFM?\n\nWe appreciate this excellent question. Currently, **graph foundation models (GFMs)** are still an emerging area without a fixed architectural definition. Our decision to use **GNNs** is grounded in both theory and practicality:\n\n* **Scalability**: Transformer-based graph models are often computationally prohibitive for large graphs. GNNs scale more gracefully with graph size.\n\n* **Logical Reasoning**: As discussed in **Appendix A**, GNNs possess strong **logical expressiveness**, making them well-suited for **multi-hop and compositional reasoning**, which is essential for retrieval tasks. Case studies can be found in Section 4.8.\n\nOur implementation (8M parameters, 6-layer GNN) is a starting point toward scaling graph-based foundation models efficiently. We view this as a foundation for future architectural exploration. In our *response to W2*, we also analyze the impact of different GNN layers.\n\n## Q2: Generalization to evolving domains and KGs\n\nWe agree that rapidly evolving KGs present challenges. However, GFM-RAG is trained on **diverse and large-scale KG corpora**, enabling it to **generalize to new or unseen domains and KGs without retraining**.\n\nAs demonstrated in **Figure 3**, without retraining, GFM-RAG performs consistently well across **seven out-of-domain datasets**, all of which involve previously unseen domains and KGs. These results demonstrate that GFM-RAG is not overfitted to a fixed KG schema or domain but instead learns generalizable reasoning capabilities. This allows GFM-RAG to **adapt to new and evolving KGs** without requiring frequent retraining."}, {"review_id": "9Rr2GPcJ3M", "reviewer": "Reviewer_b7Me", "summary": "The paper introduces GFM-RAG, a graph foundation model for retrieval-augmented generation, which leverages a query-dependent graph neural network to enhance multi-hop reasoning over knowledge graphs. The model is pre-trained on large-scale datasets and demonstrates strong performance and generalizability across multiple benchmarks.", "questions": "- Please refer to the Weaknesses section.\n- Could you clarify whether the fine-tuning is performed on the target dataset itself, or on a combined training set from multiple datasets? If it is the latter, is this approach reasonable and fair?\n- The results in Tables 10 and 13 appear to be inconsistent. Please provide further analysis and explanation.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The proposed GFM-RAG effectively integrates graph reasoning with retrieval-augmented generation, addressing limitations of existing methods.\n- The model achieves state-of-the-art performance on multi-hop QA and domain-specific RAG datasets, demonstrating its versatility.\n- The paper provides extensive experiments and ablation studies, validating the design choices and scalability of the approach.\n\nWeaknesses:\n- The proposed GFM occupies a somewhat awkward position. The core of this paper is the proposal of a GFM that can be directly applied to any dataset. However, according to Table 10, on one hand, without fine-tuning, its performance is poor; on the other hand, skipping pretraining has little impact on the results. This suggests that, beyond the framing presented in the paper, simply training and testing directly on the target dataset might be a more reasonable technical approach. The proposed GNN does not seem to have learned sufficient knowledge from pretraining to effectively support zero-shot QA retrieval.\n- Figure 5 reports results for different model sizes, but the experiment only changes the embedding dimension. Could the authors comment on how the number of layers in GFM affects the results?\n- The introduction mentions that KGs constructed by LLMs suffer from incompleteness and noise. Is there a corresponding solution proposed later in the paper? Pretraining on KGC does not address this issue. Moreover, the experiments should include further analysis to demonstrate that GFM can address or alleviate this problem.", "was_revised": true, "final_justification": "Thank you to the authors for the detailed response — it has addressed some of my concerns. I will raise my score, as this is a substantial piece of work that is likely to contribute meaningfully to the field. However, I would like to reiterate a few points that I hope the authors can address:\n\n- Although the pre-training on KG completion improves KGC performance, this paper primarily focuses on QA. The pre-training does not bring a clear benefit to the QA task, yet it accounts for the majority of the computational cost.\n\n- As also noted by another reviewer (fCb4), the main experiments are conducted using the training sets of all three major QA benchmarks. This setup is uncommon in foundation model research and does not constitute a sound experimental design.\n\nOverall, this is a solid piece of work. If the paper is accepted, I hope the authors will address these two issues to set a strong precedent for future work in this direction.", "rebuttal": "## W1: Without fine-tuning, the performance of GFM-RAG is poor and does not support zero-shot QA retrieval. This suggests training directly on the target dataset may be more reasonable. \n\n### Clarification of the term “fine-tuning”\n\nWe appreciate the reviewer’s concerns and would like to clarify that the term **“fine-tuning” in Table 10 refers to supervised document retrieval fine-tuning (Section 3.2.2)**—not fine-tuning on downstream QA datasets.\n\nAs detailed in Section 3.2, GFM-RAG follows a **two-stage training process**:\n\n1. **Self-supervised KG Completion Pretraining (stage 1):** GFM is trained to predict masked relations and entities from incomplete KGs, fostering robust multi-hop reasoning and tolerance to KG noise.  \n2. **Supervised Document Retrieval Fine-tuning (stage 2):** Analogous to instruction tuning in LLMs, this stage trains GFM-RAG on diverse retrieval instances to learn to understand users’ queries and better capture the relevance between questions and knowledge for retrieval.\n\nWe acknowledge the terminology may have caused confusion and will revise the paper to consistently use **“supervised fine-tuning”** for Stage 2 and **“domain-specific fine-tuning”** for adaptation to individual target datasets.\n\n**Table 10 Clarification:**\n\nThe experiments are conducted on three multi-hop QA datasets (HotpotQA, Musique, 2Wiki). **Removing supervised fine-tuning** significantly reduces retrieval accuracy, underscoring its necessity for effective query relevance modeling. While **skipping KG pretraining** has limited impact on retrieval performance, it enhances the model's ability for other tasks, such as KG completion (Table 11), and improves overall robustness in handling incomplete KGs. A detailed explanation is also provided in our *response to Q3*.\n\n### Zero-shot Generalizability\n\nSection 4.6 and Figure 3 demonstrate GFM-RAG’s strong **zero-shot** retrieval performance across **seven unseen domain-specific datasets**, surpassing HippoRAG and LightRAG. Table 13 and Appendix E.4 further show that **optional domain-specific fine-tuning** can further enhance performance when target data is available:\n\n**Comparison between the zero-shot performance and the performance of fine-tuning on target domains.**\n\n| Model | DelucionQA | EManual | ExpertQA | TechQA | MS Marco | HAGRID |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| HippoRAG (zero-shot) | 59.0 | 50.0 | 55.1 | 39.5 | 51.1 | 75.5 |\n| LightRAG (zero-shot) | 46.1 | 46.2 | 59.4 | 36.8 | 48.3 | 75.9 |\n| GFM-RAG (zero-shot) | 70.8 | 60.6 | **62.7** | 46.6 | 71.0 | 84.7 |\n| GFM-RAG (domain-specific fine-tuning) | **82.7** | **75.9** | 60.8 | **49.5** | **77.5** | **86.6** |\n\n### Is training directly on the target dataset more effective?\n\nTo address this, we compare GFM-RAG with methods trained directly on the target multi-hop datasets. Specifically, we compare with the following baselines:\n\n* **HippoRAG**, a strong, non-training-required baseline.   \n* **SubgraphRAG [1]** and **G-retriever [2]**, two GNN-enhanced retrieval methods that are trained from scratch on the target datasets.   \n* **GFM-RAG (target),** the proposed GFM retriever that is trained from scratch on the target dataset.  \n* **GFM-RAG (all),** the proposed GFM retriever that is trained on multiple datasets (HotpotQA, MuSiQue, and 2Wiki).\n\n**Performance of directly training on target datasets.**\n\n| Methods | HotpotQA |  | MuSiQue |  | 2Wiki |  |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| | R@2 | R@5 | R@2 | R@5 | R@2 | R@5 |\n| HippoRAG (no-train) | 60.5 | 77.7 | 40.9 | 51.9 | 70.7 | 89.1 |\n| SubgraphRAG | 61.5 | 73.0 | 42.1 | 49.3 | 70.7 | 85.5 |\n| G-retriever | 53.3 | 65.5 | 38.8 | 45.1 | 60.8 | 67.8 |\n| GFM-RAG (target) | **79.3** | **87.8** | 47.6 | 57.5 | 89.3 | 93.2 |\n| GFM-RAG (all) | 78.3 | 87.1 | **49.1** | **58.2** | **90.8** | **95.6** |\n\nThe results show that even when directly fine-tuned on target datasets, GFM-RAG (target) outperforms SubgraphRAG and G-retriever, demonstrating the superiority of the GFM architecture and the effectiveness of two-stage training. Additionally, GFM-RAG (all) performs competitively across all datasets, confirming its ability to leverage diverse KGs and develop generalizable reasoning skills across domains. \n\nWe will revise the terminology and include these expanded analyses in the final version.\n\n[1] Simple is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation. ICLR 25\n\n[2] G-retriever: Retrieval-augmented generation for textual graph understanding and question answering NeurIPS 24\n\n## W2: Impact of GNN depth\n\nWe conducted an ablation varying the number of GNN layers (1–8) while fixing the hidden dimension (512). The results are shown below:\n\n**Ablation study of different GNN layers.**\n\n| Method | Averge |  | HotpotQA |  | MuSiQue |  | 2Wiki |  |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| N_dim=512 | R@2 | R@5 | R@2 | R@5 | R@2 | R@5 | R@2 | R@5 |\n| 1-layer (3M) | 53.9 | 66.7 | 59.3 | 74.2 | 40.7 | 50.2 | 61.8 | 75.7 |\n| 2-layer (4M) | 69.9 | 78.6 | 73.6 | 85.4 | 47.6 | 57.0 | 88.6 | 93.3 |\n| 4-layer (6M) | 72.2 | **80.1** | 78.4 | 87.8 | 49.3 | **60.1** | 88.8 | 92.5 |\n| 6-layer (8M) | 71.9 | 79.6 | 78.0 | 87.0 | 48.4 | 58.7 | 89.3 | 93.1 |\n| 8-layer (10M) | **73.0** | 79.9 | **79.7** | **87.8** | **49.7** | 59.1 | **89.5** | **92.8** |\n\nPerformance improves with deeper GNNs due to larger size and better multi-hop reasoning, echoing neural scaling laws. Yet, performance sometimes peaks at 4 layers—matching the 4-hop max in training data—suggesting GFM-RAG captures relevant multi-hop paths, with deeper layers offering little gain without more complex signals. Detailed analysis can be found in our response to *Reviewer 9ZFX W2*.\n\n## W3: Addressing KG noise and incompleteness.\n\nWe appreciate the reviewer’s concern, which aligns with our use of GNNs for their robustness to KG incompleteness. Real-world KGs are inherently noisy. Unlike structure-only methods like HippoRAG, we adopt GNN and **enhance its robustness** through self-supervised KGC pretraining: **masking relations and predicting links to foster multi-hop reasoning in incomplete settings**. Table 11 confirms its effectiveness in handling incomplete graphs.\n\n**Experiment results on KG completion**\n\n| Method | MRR | Hits@1 | Hits@3 | Hits@10 |\n| :---- | :---- | :---- | :---- | :---- |\n| GFM-RAG | **0.304** | **0.234** | **0.323** | **0.451** |\n| GFM-RAG w/o Self-supervised KG Completion Pre-training | 0.029 | 0.007 | 0.022 | 0.067 |\n\n## Q2: Ablation Study of Training Datasets\n\nWe want to confirm that GFM-RAG is trained on multiple datasets (HotpotQA, MuSiQue, 2Wiki), but we ensure **strict separation between training and evaluation data**, with no overlap in documents or entities, thus eliminating any risk of information leakage.\n\nThe motivation for this setup is rooted in the **foundation model paradigm**. Similar to how LLMs are trained on diverse corpora to learn generalizable linguistic patterns, GFM-RAG aims to learn **universal graph reasoning capabilities** from diverse KG-indexes. This design enables strong **zero-shot transfer to new domains and datasets**, eliminating the need for retraining or fine-tuning on every downstream domain. Results in Fig. 3 and our *response to W1* demonstrate the strong generalizability of GFM-RAG on out-of-domain datasets without retraining.\n\nWe further conducted ablation studies where **GFM-RAG is trained separately on each dataset**, and we report performance across all three benchmarks. Results are shown below:\n\n**Ablation study of GFM-RAG trained on each dataset.**\n\n| Test Dataset | HotpotQA |  | MuSiQue |  | 2Wiki |  |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| **Training Dataset** | R@2 | R@5 | R@2 | R@5 | R@2 | R@5 |\n| HotpotQA | **79.3** | **87.8** | 46.9 | 57.2 | 86.6 | 92.4 |\n| MusiQue | 68.8 | 81.8 | 47.6 | 57.5 | 84.4 | 89.6 |\n| 2Wiki | 72.2 | 77.9 | 46.6 | 55.5 | 89.3 | 93.2 |\n| All | 78.3 | 87.1 | **49.1** | **58.2** | **90.8** | **95.6** |\n\nThese results show that GFM-RAG not only performs well on the trained datasets, but also generalizes well to other datasets.  More importantly, the **model trained on multi-domain datasets** performs competitively across all datasets—validating its ability to **generalize effectively across domains** and benefit from training on diverse KGs by learning generalizable reasoning ability across domains.\n\nWe will include this ablation study and further clarify the design rationale in the final version of the paper.\n\n## Q3: Inconsistency between Table 10 and Table 13.\n\nTable 10 examines the impact of two-stage training strategies, while Table 13 evaluates zero-shot generalizability and domain-specific transferability; **their findings are not contradictory**.\n\n**Table 10 Clarification**: \n\n“Fine-tune” and “pre-train” refer to “Supervised Document Retrieval Fine-tuning” and “Self-supervised KG Completion Pre-training,” respectively, as explained in our *response to W1*. \n\nRemoving supervised fine-tuning significantly reduces retrieval accuracy, highlighting its importance for capturing query-document relevance. In contrast, removing self-supervised pretraining has a smaller effect on retrieval but improves generalization to tasks beyond retrieval and handling incomplete KGs.\n\n**Table 13 Clarification:** \n\nThe “fine-tune” in Table 13 denotes the “**fine-tuning on domain-specific QA datasets**”.\n\nTable 13 compares the zero-shot performance of HippoRAG, LightRAG, and GFM-RAG across seven domain-specific test sets without additional training. GFM-RAG, trained on multi-hop QA datasets, shows strong retrieval performance across all unseen datasets, demonstrating robust zero-shot generalizability.\n\nWhen fine-tuned on these seven domain-specific training sets, GFM-RAG’s performance further improves, underscoring its strong transferability when target training data is available."}, {"review_id": "ZUBbPFvSDa", "reviewer": "Reviewer_U3C7", "summary": "The paper proposes a foundation model for graph retrieval for LLMs, called GFM-RAG. The method constructs a KG from a document by using an LLM as an Open IE tool to extract entity relation triples. Further, given a query sentence, the method uses a message passing mechanism over the KG after extracting the entities in the query sentence. Distmult is used to score the entities and select the top ones, followed by obtaining the relevant documents from the entity-document mapping. These documents are then used along with the query for prompting the LLM. The GFM model is trained using KG completion objective in a self-supervised manner and in the document retrieval setting in a supervised manner. Results on multi-hop benchmarks and unsupervised datasets show strong performance compared to baselines.", "questions": "1)\tIn the entity retrieval process, if the KG is large the paper doesn’t mention about how it is handled \n2)\tThe current framework uses DistMult as the KG completion scoring method. However it has been shown by previous works [1,2] that distmult has limitations in modelling antisymmetric, inverse and compositional relations. Why not use some newer approaches for KG completion modelling?\n3) In the path interpretations of sec. 4.8 (eq. 19), is the top-k taken over the product of gradients (chain rule) over the path or over the individual triple scores? This could be clarified better.\n4) See weakness section above\n\nReferences\n\n[1] Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, & Jian Tang. (2019). RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space.\n\n[2] Zhang, S., Tay, Y., Yao, L., & Liu, Q.. (2019). Quaternion knowledge graph embeddings. \n\n[3] Mufei Li, Siqi Miao, & Pan Li. (2025). Simple Is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation.\n\n[4] Xiaoxin He,Yijun Tian,Yifei Sun,Nitesh V Chawla,Thomas Laurent,Yann LeCun,Xavier  Bresson, and Bryan Hooi. G-retriever: Retrieval-augmented generation for textual graph  understanding and question answering. arXiv preprint arXiv:2402.07630,2024.", "limitations": "I don't see an explicit limitation section, but I feel the authors should specifiy the technical limitations of the method especially when the KG grows in size how would their method scale or handle such large graphs for RAG.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1)\tThe paper studies an important area of research for graph foundation models for RAG\n2)\tThe method achieves competitive results\n\n**Weaknesses**\n\n1)\tIn the entity retrieval process, if the KG is large the paper doesn’t mention about how it is handled \n2)\tThe current framework uses DistMult as the KG completion scoring method. However it has been shown by previous works [1,2] that distmult has limitations in modelling antisymmetric, inverse and compositional relations.\n3)\tThe paper doesn’t compare with contemporarymethods that perform graph learning for RAG ([3,4])\n\nMissing citation\n\n[1] Xiaoxin He,Yijun Tian,Yifei Sun,Nitesh V Chawla,Thomas Laurent,Yann LeCun,Xavier  Bresson, and Bryan Hooi. G-retriever: Retrieval-augmented generation for textual graph  understanding and question answering. arXiv preprint arXiv:2402.07630,2024.\n\nReferences\n\n[1] Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, & Jian Tang. (2019). RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space.\n\n[2] Zhang, S., Tay, Y., Yao, L., & Liu, Q.. (2019). Quaternion knowledge graph embeddings. \n\n[3] Mufei Li, Siqi Miao, & Pan Li. (2025). Simple Is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation.\n\n[4] Xiaoxin He,Yijun Tian,Yifei Sun,Nitesh V Chawla,Thomas Laurent,Yann LeCun,Xavier  Bresson, and Bryan Hooi. G-retriever: Retrieval-augmented generation for textual graph  understanding and question answering. arXiv preprint arXiv:2402.07630,2024.", "was_revised": true, "final_justification": "Overall after discussion with authors, I'm still not fully convinced from the results about the need for GFM for RAG. The provided results only show recall of entities/documents not the final results of the LLM system after using the docs. If it is only about retrieving entities and then fetching documents, I feel heuristics, standard retrieval systems or semantic closeness followed by query to the LLM would do well (if not for example due to entities not being connected in same component how is message passing going to retrieve the disconnected entity?). Also comparison with baselines may not be fair as proposed method is pretrained on a large dataset. Thus I keep my score still leaning towards accept due to the demonstrated results and importance of the problem.", "rebuttal": "## W1/Q1: Scalability to large KGs\n\nWe agree that handling large KGs is essential for real-world applications. To support large-scale KGs, inspired by LLM training, **we implement a Single Program Multiple Data (SPMD)-style message-passing mechanism that enables distributed training and inference across multiple GPUs.**\n\nSpecifically, we partition the full KG into balanced subgraphs using the METIS algorithm, with each device storing only a subset of the graph in memory. During message passing, each device first aggregates information locally and then exchanges messages with other devices to finalize the node embedding updates. This design allows GFM-RAG to scale effectively by leveraging more hardware.\n\nIn addition, instead of performing message passing over the entire KG, we can only conduct it on the localized k-hop subgraph around the query. This significantly reduces computational and memory costs, further enhancing scalability on large KGs.\n\nWe will clarify this strategy in the revised version and add a discussion in the appendix.\n\n## W2/Q2: Use of DistMult and limitations\n\nWe appreciate the reviewer highlighting the limitations of DistMult in modeling antisymmetric or inverse relations. We chose DistMult as the message function for its simplicity and computational efficiency, **as it has no training parameters and is easy to compute**. That said, we agree that more expressive models like RotatE \\[1\\] or QuatE \\[2\\] could potentially enhance performance. In preliminary experiments, we found DistMult to offer a strong tradeoff between efficiency and effectiveness in our retrieval-focused setting, but exploring alternative KGE models in our GFM is a valuable direction that we plan to pursue in future work.\n\n## Q3: Clarification of Eq. 19 (Path interpretation)\n\nWe thank the reviewer for pointing this out. In Eq. 19, the top-k paths are selected based on the **product of gradient scores over triples forming the path**, which approximates the contribution of that path to the final prediction via the chain rule. This allows us to identify influential multi-hop reasoning chains and interpret the model’s behavior. We will clarify this in the main text and provide more details in the Appendix.\n\n## W3/Q4: Comparison with G-Rretriever and SubgraphRAG\n\nWe thank the reviewer for highlighting recent related work. In response, we have conducted additional experiments comparing GFM-RAG with **SubgraphRAG** and **G-Retriever** across three benchmark datasets. The results are shown below:\n\n**Comparison with SubgraphRAG and G-retriever.**\n\n| Method | HotpotQA |  | MuSiQue |  | 2Wiki |  |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n|  | R@2 | R@5 | R@2 | R@5 | R@2 | R@5 |\n| SubgraphRAG | 61.5 | 73.0 | 42.1 | 49.3 | 70.7 | 85.5 |\n| G-retriever | 53.3 | 65.5 | 38.8 | 45.1 | 60.8 | 67.8 |\n| GFM-RAG | **78.3** | **87.1** | **49.1** | **58.2** | **90.8** | **95.6** |\n\nAs the results show, **GFM-RAG consistently outperforms** both SubgraphRAG and G-Retriever across all datasets and recall settings. While SubgraphRAG and G-Retriever both employ GNN-based reasoning to enhance retrieval, they are primarily tailored to well-constructed KGs with domain-specific fine-tuning, and often operate on shallow, fixed subgraphs. Their architectures typically involve small GNNs with limited reasoning depth and generalization ability.\n\nIn contrast, **GFM-RAG introduces a generalizable graph foundation model** trained over diverse KGs and designed to operate on noisy, incomplete, and large-scale knowledge graphs. This allows GFM-RAG to capture transferable multi-hop reasoning patterns and maintain robust performance across domains without domain-specific adaptation.\n\nWe will incorporate these comparisons and insights into the final version to enhance our empirical and methodological contributions."}]

Author rebuttal:

Meta-review: The paper introduces GFM-RAG, a graph foundation model designed to advance retrieval-augmented generation (RAG) by incorporating graph neural networks to capture complex relationships within knowledge graphs. This design improves both the effectiveness and efficiency of retrieval, particularly for multi-hop reasoning tasks where traditional RAG approaches often fall short. GFM-RAG builds a knowledge graph index from documents, representing information as interconnected triples, which supports multi-hop reasoning and integrates heterogeneous knowledge sources.

Empirical results show that GFM-RAG achieves state-of-the-art performance on multi-hop question answering benchmarks and generalizes well across diverse domains. The model further maintains efficiency by enabling multi-hop reasoning in a single step and follows neural scaling laws, suggesting potential for continued gains with larger models. Overall, GFM-RAG represents a meaningful advancement in RAG systems, offering stronger reasoning capabilities and broader applicability.

The paper proposes an interesting idea, and the experiments are comprehensive. Initially, the reviewers have concerns on the ablation studies, training strategies, and so on. However, in the rebuttal, the authors have carefully addressed them, and all the reviewers have positive scores on the paper. I recommend acceptance.
Decision: Accept (poster)