GOFA: A Generative One-For-All Model for Joint Graph Language Modeling
Authors: Lecheng Kong
Jiarui Feng
Hao Liu
Chengsong Huang
Jiaxin Huang
Yixin Chen
Muhan Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: GNN;Graph foundation model;LLM;
Primary area: learning on graphs and other geometries & topologies
TL;DR: We propose a generative graph language model, that achieve good performance on a variety of tasks, showing potential to serve as a future graph foundation model.

Abstract: Foundation models, such as Large Language Models (LLMs) or Large Vision Models (LVMs), have emerged as one of the most powerful tools in the respective fields. However, unlike text and image data, graph data do not have a definitive structure, posing great challenges to developing a Graph Foundation Model (GFM). For example, current attempts at designing general graph models either transform graph data into a language format for LLM-based prediction or still train a GNN model with LLM as an assistant. The former can handle unlimited tasks, while the latter captures graph structure much better---yet, no existing work can achieve both simultaneously. In this paper, we first identify three key desirable properties of a GFM: self-supervised pretraining, fluidity in tasks, and graph awareness. To account for these properties, we extend the conventional language modeling to the graph domain and propose a novel generative graph language model GOFA. The model interleaves randomly initialized GNN layers into a frozen pre-trained LLM so that the semantic and structural modeling abilities are organically combined. GOFA is pre-trained on newly proposed graph-level next-word prediction, question-answering, structural understanding, and information retrieval tasks to obtain the above GFM properties. The pre-trained model is further instruction fine-tuned to obtain the task-solving ability. Our GOFA model is evaluated on various downstream datasets unseen during the pre-training and fine-tuning phases, demonstrating a strong ability to solve structural and contextual problems in zero-shot scenarios. The code is available at https://github.com/JiaruiFeng/GOFA.

Published as a conference paper at ICLR 2025
GOFA: A GENERATIVE ONE-FOR-ALL MODEL FOR JOINT GRAPH LANGUAGE MODELING
Lecheng Kongl" Jiarui Feng Yixin Chen' Muhan Zhang
Hao Liu't Chengsong Huang' Jiaxin Huang'
Washington University St. Louis "Peking University
(jerry kong, feng. jiarui, liuhao, chengsong, jiaxinh @ewstt.. ychen2 www.ses.ussl.d, muhan@pku. wwt.eekkk..uuc.. edu.
ABSTRACT
Foundation models, such as Large Language Models (LLMs) or Large Vision Models (LVMs), have emerged as of the most powerful tools in the respective fields. However, unlike text and image data, graph data do not have a definitive structure, posing great challenges to developing Graph Foundation Model (GFM). For example, current attempts at designing general graph models either transform graph data into a language format for LLM-based prediction or still train a GNN model with LLM as an assistant. The former can handle unlimited tasks, while the latter captures graph structure much better-yet, no existing work can achieve both simultaneously. In this paper, we first identify three key desirable properties a GFM: self-supervised pretraining, fluidity in tasks, and graph awareness. To account for these properties, we extend the conventional language modeling to the graph domain and propose a novel generative graph language model GOFA. The model interleaves randomly initialized GNN layers into frozen pre-trained LLM so that the semantic and structural modeling abilities are organically combined. GOFA is pre-trained on newly proposed graph-level next-word prediction, question- answering, structural understanding, and information retrieval tasks to obtain the above GFM properties. The pre-trained model further instruction fine-tuned ob- tain the task-solving ability. Our GOFA model is evaluated on various downstream datasets unseen during the pre-training and fine tinettnnin phases, demonstrating a strong ability to solve structural and contextual problems in zero-shot scenarios. The code is available at pthhtps: //ti//gittt wiigtthu..... com/ Www.o///aaaiiePe/oPPPPP
INTRODUCTION
With the emergence of Large Language Models (LLMs), the field of artificial intelligence is undergo- ing a profound transformation, shifting from specialized, fragmented models to universal foundation models. A foundation model is pre-trained on large-scale datasets and can be further adapted to diverse downstream tasks using fine-tuning (Hu et al. 2022) or in-context learning (Bommasani et al.) 2021 Touvron et al. 2023). Foundation models have been developed in different domains to handle text (Brown et al. 2020 Touvron et al., 2023). ) image (Kirillov et al.] 2023 Bai et al., 2023), and mudti-modal data (Zhang et al. 2023c Li al. [023 Alayrac et al.) 2022). Because of their versatility and generalizability, foundation models have become prevalent in these domains.
However, despite preliminary efforts, foundation model in the graph domain has arguably yet be proposed. In the graph domain, data are highly flexible and dynamic. For example, social networks
receive millions of new connections daily (Hardiman & Katzir 2013), and novel molecules and
protein structures are frequently discovered (Abramson et al. 2024] Gilmer et al.) 2017). While
past rescarchers have proposed specialized models learn graph data (Ying al. 2021 Kipf &
Welling 2017), the models require retraining to accommodate new graphs (Dai et al. 2022 Mo et al. 2022). Moreover, trained models are usually tied specific applications and cannot be generalized to new domains and tasks. It becomes increasingly difficult for models to adjust to the ever-evolving
"Contributed equally. Listing order is random. 'Coresponding author

---

Published as a conference paper at ICLR 2025
Task
Sentence Completton Task
Question Answering Task
Straetural Understanding Task
Information Retrieval Task
Networks
this (hiode Aj, Creph Atbent
Whikhh type Rook comnerrryy cieditrr comtractis andhwhy?
Cpitiolle ler to: Switch OLED
Thes a Pkode CD Wilpeett
eystoon, Beverth mapoo cooee
TAG Raw Text
F
Thes (iice Atte:Attettnit you needi Abstrect: The dommnant aequarce traneduction models...
other of rocke usod for constructixi?
Secimmemrry rock, hiss aaay to sxtrect, cut, ahape
This is (Mode PJ. Pindker: Amoron arrnt
Pro 4 scrica
This is (Hoce DJ, Wippote giaphhi
this le (Plods Thts Acture merhod for stoc vestic apti mestionn Abstroct: We introuuce Acers, sigorithrs for...
Yex. igneoum rocka like are seedito thei dumbbiliyy
Switch wizh Blucard Redd Acy Cur...
(hiokk (Nliipoli Unno is familly of www.tiiiiii..................w..
Compute the shortest path betweon [bode and [Nods DJ and panacate shortest pathe fromt A]to (Node DJ
Prompl
promptt tock.
?
cortain preforcnce rocks?
output the content of (hintle DI
Absttractt We priseeti graph attendion Answer inttiorris (BATT)), novel nounal io anchitactures that oparata on gropph --.
Yes, limestone is commonly used in UK becouse se con wisttond high Lovels of raintoll and humiditt.
shortsst parth distance
Wikippddie entry: system, 7. Beventh major relaase of the classic Mac oS operating system for Hocintosh
Shartest path: [Node [Node BII [Nod6
Figure 1: Examples of our pre-training tasks.
nature of graph data. Hence, a graph foundation model (GFM) applicable to new domains/tasks with minimal or adaptation costs is urgently needed, spurring recent endeavors to study general graph models. In particular, a strong zero-shot ability is both challenging and fascinating for GFM researchers.
The success of LLMS inspired a series of preliminary attempts which use LLMs to develop general graph models. They can be roughly divided into two categories: LLM as a predictor and LLM as an enhancer (Chen et al. 2023). The LLM as a predictor approach transforms graph data into representations that L.MS can understand and use LLMs to generate predictions (Tang et al. 2023). However, as suggested by recent study (Wang et al. (023). such an approach falls short of understanding graph structures. This inspired the LLM as an enhancer approach, which adopts LLM to process and unify diverse graph data and feeds them to a GNN to train general graph models (Liu et al., 2023a Huang et al., 2023a). Nevertheless, because GNN outputs fixed-sized they can only handle specific tasks such classification, and cannor generalize to arbitrary, new tasks due to the lack of generation ability. In summary, the current two approaches cannot fully utilize structural information and be generative simultaneously. We discuss the pros and cons of existing approaches in detail in Section 2
pprrssentaiions/prccctttoo
In this paper, we first identify three desirable properties of graph foundation model (GFM), namely large-scale self-supervised pre-training, fluidity in tasks, and graph understanding. To achieve the first property, we propose a generic graph self-supervised learning problem similar the next- token prediction problem in LLMs, allowing label- aabessigoottcc and continual training on highly diverse graph data. We then propose a generative model termed Generative One-For-All (GOFA) that interleaves GNN layers into an LLM to achieve the second and third properties. Such a novel design systematically integrates GNN into an LLM, granting the LLM graph structural learning ability while keeping LLM'ss 's original free-form text generation ability. Meanwhile, this design allows the pipeline of the original LLM to remain intact, giving GOFA a close-to-LLM level of task fluidity. We pre-train the model with large-scale real-world graph data, Question Answer (QA) chain data adopted from the NLP domain, and graph structural data to empower the model with the aforementioned foundational abilities in the graph domain (Examples in Figure D. After pre-training, we further instruction fine-tune the model on a small amount of data (relative to the pre-training data) to make it understand task formats. The fine-tuned model is finally evaluated on various downstream datasets unseen during pre-training and fine-tuning. GOFA achieved impressive results on the zero-shot scenario, which demonstrates the strong potential of GOFA to serve as a graph foundation model.
2 A DESIRED FOUNDATION MODEL FOR GRAPH
In this section, we elaborate on three crucial properties a true graph foundation model should possess to motivate our GOFA model design. We note that many contemporary works (partly) propose similar ideas to ours and thus we do not claim the credit. We kindly refer readers to the latest surveys (Liu et al. 2023b Jin et al. 2023 Zhang et al. 2023db for more discussions GFMs.
2

---

Published as a conference paper at ICLR 2025
Large-Scale Self-Supervised Pre-training: One fundamental design of LLM is that it unifies all NLP tasks into a single next-token prediction paradigm, which enables self-supervised pre-training on large corpus collected from different sources. For pre-training graph models, while numerous efforts have been made from both the L.M as a predictor and LLM as an enhancer approaches, these attempts usually require the learning target to be labeled (Liu et al. 2023a Chen et al.) (023). However, a graph foundation model should have no constraint on the input graph (has labels or not) and can learn cross cross-domain knowledge from large-scale graph data in a self-supervised fashion.
next-token-reediction
Fluidity in Tasks: A graph foundation model should also possess the same level of versatility and fluidity in handling different tasks as an LLM. Specifically, such ability can be broken down into three levels: (a) The graph foundation model can naturally respond appropriately to different graph tasks based on user instructions without requiring task aseccsccc adjustment (e.g., the same model performs classification and question-answert instruction-tunie tuning, the model should have in-context learning ability on unseen tasks (e.g.. a model tuned on citation network also performs well on knowledge graphs with proper instructions). (c) Users should be able to define new, previously unseen tasks by modifying the graph structure and features in way that aligns with the universal input representation of the model. They
tasks without any modification.) (b) With appropriate
continuously train the model on new data without special adaptation. Existing approaches that use GNN models as the predictors usually either restricted in the output format (Liu et al.; 2023a Xia al., [2024] He et al. (224a) or need additional fine-tuning on the task head (Sun et al. | 2003] Wang et al. 2022).
Consequently, despite having better structural modeling ability, such models cannot accommodate task changes or deal with novel tasks, e.g., shifting from a classification task to a
task that requires outputting all shortest paths between two nodes.
Since the LLM as a predictor approach uses a generative LLM to take text input and produce text output, it naturally has the fluidity accept varied prompts to tackle different tasks. However, such an approach processes the structural information poorly (Wang et al. 2023), making the utility of these models limited on many graph tasks. More importantly, even though some recent variants can use auxiliary graph models (such as GNNs) to incorporate structural information (Tang et al. 2023 He & Hooi 2024; Zhang ct al.) 2024), the graph models are frozen and not responsive to difterent prompts, and the output from the graph models may not be the most relevant to the input prompt. On the contrary, a graph foundation model should account for the unique structural information of graphs such as node degrees, shortest paths, common neighbors, etc., and generate graph representations dependent on the input prompt. It should not only have LLM's prompt learning capability but also learn graph structure and semantic information jointly.
Graph Understanding:
3 METHOD
In this section, we first propose a generative modeling framework for graphs, serving as the graph counterpart of traditional language modeling. Next, we introduce novel GNN-LLM architecture for the proposed graph generative modeling problem. Finally, we describe the unified pre-training tasks to train GOFA towards the proposed GFM properties.
3.1 GENERATIVE MODELING FOR GRAPH
Unifed task formats. A generative model usually takes existing contexts, such as user prompts and passages, as input to generate conditional output related to the contexts, such as answers and completed sentences. Defining unified input and output formats for tasks in language applications easy, as they are purely text-based Further, because both the pre-training and downstream tasks are constructed in the same format (i.e., next-token- prediction), the downstream tasks conveniently adapt the knowledge from pre-training tasks, resulting in surprising capabilities, such as zero-shot learning. However, graph data from different domains vary significantly by input feature (e.g., nodes in a citation network have completely different vector representations as nodes in a knowledge graph) and output target, preventing direct knowledge transfer between tasks. Hence, the first challenge is to define a wnified format for graph tasks, such that the model can do large- scale self-super pre- pre-training on arbitrary graphs and transfer to downstream tasks seamlessly.
next=token-prediitton))
To unify graph task input, we follow the previous work OFA (Liu et al., 2023a) and extend the definition of Text-Attribute Graph (TAG) beyond graphs with text features such as citation and
3

---

Published as a conference paper at ICLR 2025
product networks. In fact, any node and edge features can be represented by texts. For example, in airline networks, airport and flight route details can be converted into textual descriptions for nodes and edges. Non-textural features, like numerical data, can also be transformed into text strings, in LLMs.. Even for graphs without any features, we can still attach sentences like "The degree of this node is 3" to nodes. Formally, TAG is graph G (V, E,, Xy, Xx) where V and E are the sets of nodes and edges. Each node v E V (edge E E)) corresponds to text description x(v) E Xv (ar(e) E Xx). Such format encodes almost all existing graph data and serves well general input representation.
For self-supervised language modeling, the generated output essentially completes the input sentence. Such task requires the model to have deep semantic and logical understanding of the provided contexts, which is crucial for downstream applications. Similarly, in graph modeling, we aim to achieve the same level of understanding through graph completion tasks. Given a TAG, the output should complete the graph conditioned on its semantic and structural information. We choose to use natural language the most tangible output format to complete TAG. Succinctly, all natural language tasks can be modeled as sentence completion, and similarly, we aim to model all graph tasks with graph completion.
Generative Graph Modeling. We then formally define the generative graph modeling framework for graph completion. This framework supports various graph-related tasks, including classification and free-form question answering. An L.M starts generating only from the end of the input sentence. However, in a TAG, every end of sentence users might only be interested in generating output for specific nodes. To accommodate this, we introduce Nodes of Generation (NOG), allowing users specify starting points for generation. The modeling task is to take a TAG as input and complete the TAG logically and sensibly by completing the sentences on the potentially aser-specified nodes.
node is potential generation starting point, but
We define graph generative modeling as the likelihood of the text associated with the NOG e:
plyle, G) II C,,
where y1 is the 1-th token of y, and yer is its preceding tokens. The NOG is completion target node with initial corresponding text ævv), and z(v) can be empty. G con- tains structural and textual information of neighbor nodes to help the model generate y. Under this framework, we design range of sepr-supe learning tasks. For example, the graph completion task is shown on Figure where the text on the NOG is incomplete, and the goal is to complete the sentence on it using the existing text and the neighbor information. This task is covered by Equation ( which encourages the model to have strong graph structure and feature comprehension ability. Thus, the importance of the framework is that a model properly solves such modeling problems can possess the three prop- erties of GFM discussed in Section 2 thus benefit diverse downstream tasks, even in the zero-shot fashion. Section F2 discusses how the proposed framework applies to various tasks related to the three properties.
Sentence Completion
Kim has 5 apples.
Jason has apples.
Alfred has bananas.
NOG text:
In my neighbor, bananas are
Target:
more popular than apples.
Figure 2: Task examples in TAG. Sen- tence completion/Next-wor prediction. Orange node v represents NOG.
3.2 GOFA: GENERATIVE ONE-FOR-ALL MODEL
To solve the generative graph modeling problem proposed in Equation ( we design the GOFA architecture shown in Figure 3 Overall, GOFA consists of graph language encoder and an LLM decoder. The graph language encoder interleaves GNN layers with LLM compressor layers to learn node representations containing joint structural and semantic information. The LLM decoder is then used to generate texts from the NOG representation. The LLM compressor and decoders are all pre-trained decoder-only transformers. We describe each component in detail as follows.
4

---

Published as a conference paper at ICLR 2025
Me introduce LLatt,, collection
foundation language nodels
Complettion Target: Aos mace: introdace
LLM Decoder
Teacher Forcing
foundation language 638 ranging
We
Introduce Liam
cotlection
foundation tanguage
TAG Raw Text
Text Tokeess
Graph Language Encoder
This made paper
Tiale: Hatk: Been arc Language Nodels.
LLM
CNN Layer
LIM Comproaeec Layss &
GNN Lavar
LLM
Comprosun Layert
Compressor Layer
Memory Embedding
Memory Tokens
Token level Message Passing
Edge
Manmuiie Goos
SIait Foost Morreee
Ny.
Node
Figure 3: GOFA Architecture. Text tokens of TAG's node/edges concatenated with memory tokens to be input to Graph Language Encoder. GNN layers are interleaved into LLM Compressor layers, where memory embeddings from LLM Compressor Layer used node/edge features for token-level GNN message passing. Memory embedding will be used for teacher-forcing training.
LLM compressor: Because GNNs require node and edge representations to have the same input dimension, many previous works propose to pool all tokens' output embeddings from the LLM as the node and edge vector representations and feed them GNN (Liu al., 2023a Huang et al.) 2023a He & Hooi] 2024). While this approach shows effectiveness in tasks of fixed form, such as classification and regression, is insufficient more complex tasks such as generation, as 1) the pooling process inevitably loses semantic information, 2) standard LLMs are not trained in way such that the pooled output embedding is summarization of the input sentence, and the pooled representation space is no longer compatible with the space of the downstream LLM decoder. Hence, we adopt pre-trained sentence compressor (Ge et al. 2023) that preserves as much information as possible from the original sentence in fixed-size multi-token embeddings. The core idea is compress a sentence into K embeddings instead of one embedding. Specifically, the sentence compressor has the same architecture decoder-only LLM, but the sentence to be compressed {g(2;)) is appended by sequence of K memory tokens {g(m;)) and the I-th layer of the LLM is:
[Q+11 Ontt =la (x1). (x1), 4+1 (mn1),.... + (mx))
(2)
(Lo (m)) N (m11) h (mx))) LLM H.)).
We use Q and Qt... to represent the t-th LLM layer outputs corresponding actual text tokens in sentence x and the memory tokens appended at the end of text tokens, respectively. We use H. to represent the t-th GNN layer output, which will be explained later. In Equation (2). the text tokens (Q')) and memory tokens (H processed by the previous GNN layer) are concatenated as a single sequence of embeddings, which fed to the current LLM layer. Because the last K tokens attend to all previous tokens, they can compress all information in the sentence into the output embeddings of the K tokens. This compressor architecture is inspired by ICAE (Ge et al. 2023). The compression ability is obtained through auto-encoder-sttle fine-tuning, as discussed in Appendix A.I
Token-level GNN: Conventional GNNs take one embedding vector for each node/edge. However, now each node/edge sentence is compressed into K memory token embeddings Qm. Hence, we propose a simple extension of GNNs to the token level. For node V. the t-th GNN layer is
H ('(,) [k] = GN (Om,,,(())) {(O' [k], O I'm [k)) E (.))), k= = (3)
In the GNN layer, tokens at different indices do not communicate. If we directly stack these GNN layers, they degenerate into multiple isolated GNNs for each token. Nevertheless, because
5

---

Published as a conference paper at ICLR 2025
interleave the GNN layers into the LLM layers, as shown in Figure B the isolated tokens exchange information in the subsequent self-attention layers of the LLM. This approach significantly reduces memory usage because we do not allow cross-token attention between different nodes. While edge memory tokens Q m,, are passed into GNN to assist message passing, their representations are not updated in the GNN layer but directly passed to the next LLM layer, hence H #a(e) Qmaate) In GOFA, we use a modified Transformer Convolutional GNN (Shi et al. 2021) to be consistent with the transformer architecture of LLM (see Appendix A.3 for details).
We insert one GNN layer between two transformer layers, while the first and the last layer are always transformer layers. In GOFA, we only insert GNN between the last few transformer layers, but this can be flexible depending on the computational resources. Following previous practice, we incorporate feed-forward (FF) layers into the GNN to increase expressivity and residual connections to stabilize training. Moreover, GOFA should maintain the functions of an L.M on plain texts, hence, inspired by the gating mechanism in earlier works (Hochreiter & Schmidhuber 1997 Alayrac et al., 2022), we apply a tanh gate, initialized at O, to the GNN and FF layer outputs so that the initial model ignores the information from GNN layers and is equivalent to the pre-trained LLM. We introduce weight decay in the gating module to promote gate value staying in large non-zero values only when graph information helps generate more accurate final text outputs.
After applying the model to the textual graph, the memory tokens Qm,z of every node contain information about the text on the node, the surrounding node lext, and the graph structure due to the message-passing process in the GNN layers. Then, for the NOG v and its corresponding target text y. insert Qm.z the front of the token embeddings of the target text to generate and use teacher-forcing to maximize the standard log-likelihood of using the wext-token-predictin objective. In this way, we have modeled the problem in Equation (T). The compressor, decoder,
LLM decoder:
and GNN parameters can be jointly or separately optimized. potentially with PEFT methods like LoRA (Hu et al. 2022). In this paper, we use ICAE (Ge al. 2023) as our backbone LLM, but the GOFA architecture is not tied to any specific LLM. More details are discussed in Appendix A.2
Discussion. Our proposed graph language encoder has several advantages over existing methods. Suppose a graph has V nodes, F edges, and the average number of tokens for all nodes is k. The complexity of one GOFA layer is OVV k²), as the self-attention only happens within each node. Note that we have omitted the extra computation complexity of message-passing because it only happens at individual indices with OEE) < O(VI²) in practical graphs. Instead, if we concatenate texts in all nodes and input them to regular LLM, the complexity of one layer is (((V))?, which is significantly larger than GOFA. Further, introducing GNN layers in LLMs is theoretically more powerful than pure L.Ms for modeling graph structures, which is discussed in Appendix R.2
3.3 UNIFIED TASK REPRESENTATION IN GOFA
The formulation in Equation (T) provides a natural way for users to query the graph by selecting a NOG. Following OFA (Liu al. 2023a), convert all tasks into tasks k-hhop rooted subgraphs extracted around the target nodes. For node-level tasks, the target node is a single node in the graph. For link-level tasks, the target nodes are the node pair. If the target node is not specified (e.g., the task is graph task), we set the default target nodes to all nodes in the graph. We connect a prompt node with the user query as NOG all target nodes. GOFA completes the prompted input TAG by answering the query on the NOG, which still aligns with the proposed generative modeling framework. This design has several advantages: (1) All tasks are represented by a NOG, so the distribution of all tasks can be unified into single space, helping the model generalize to unseen tasks from learned task representations; details. Connecting the prompt node to target nodes enables the prompt node to query the most important knowledge from the input graph through attention. This ensures the output embedding for NOG is conditionally learned from the GNN process subject to the different prompts. Conversely, most of the previous works (He & Hooi, 2024 Tang et al.) 2023) 2024) ZZhang et al.) 2024) only computed fixed embedding for each node before any prompt introduced.
(2) The text feature for the prompt node describes the task
3.4 LARGE-SCALE PRE-TRAINING
As discussed in Section 2 and Section 3.1 we design self-supervised pre-training tasks based on the three GFM properties to train GOFA. The training datasets include MAG240M (Hu et al.) 2021a).

---

Published as a conference paper at ICLR 2025
Pubmed and Arxiv (Hu et al. WikiGraph (proposed by us) for semantic diversity, and Ultrachat20kk (Ding et al. (023) dataset for question-answerte
b021b) for academic knowledge, Wikikg90my2 (Hu al. 202 and
ability. Details about the datasets can be found in Appendix C Each node is assigned a unique ID (e.g.. (Node A] to enable node querying in the graph. We design four
pre-training tasks shown in Figure T We describe the rationale of each task below and leave some implementation details and additional discussion in Appendix and Appendix [E.3]
Sentence Completion Task. This task aims for large-scale pre-training (GFM property one) by training GOFA to predict the remaining text in a node based on both the existing node text and the surrounding graph information. Such a task can be applied to any TAG without labeling, thus facilitating large-scale pre-training for GOFA to acquire diverse knowledge.
Structural Understanding Task. This task aims to provide structural modeling ability for GOFA (GFM property three). The structural task connects NOG randomly selected node pairs to generate the actual shortest path or common neighbors between them. Through these two tasks, the model is expected to gain the ability to identify basic graph structures fundamental for graph-related problems.
Question Answering Task. This task aims to ensure fluidity in generation for GOFA (GFM property two). Unlike language corpus, which naturally contains many question-and-answer (QA) pairs, graph data usually only contain objective descriptions of entities. Hence, we convert natural language drestion-Answer sequences into chain graphs and connect NOG with question to the chain graph for open-ended answer generation. This essential task enables GOFA to be responsive to arbitrary downstream applications expressed in free-form text questions.
Information Retrieval Task. In most downstream tasks, GOFA links a prompt node to target nodes in the graph to address related problems. To facilitate effective information extraction, we design an information retrieval task where a NOG queries a target node using its node ID. The model must retrieve and isolate information specific the queried node from the remaining target nodes, encouraging a pressage-passingg process conditioned on the input, as discussed in Section 3.2
RELATED WORK
Here we mainly discuss the two tracks of general graph models, and leave discussion about graph prompt learning and graph neural networks to Appendix [
LLMs as enhancers: One direction uses LLMs to convert the text features of graphs to unified
representations (Liu et al. 2023a| Chen et al. 2023 Li et al. 2024 He et al. 2024a Plenz Frank 2024) for downstream graph models to distinguish and transfer knowledge between different domains. For example, OFA (Liu et al., 2023a) uses LLM unify the input features in different datasets and transforms multiple types of graph classification tasks into a unified binary classification format. TAPE (He et al. 2024a) utilizes LLM to generate question answers and explanations as enhanced node features. Such approaches have good structural modeling ability, but they usually cannot generate free-form output to handle arbitrary tasks.
LLMs predictors: Another line of research proposes using LLMs predictors and aligning graph representation with LLM inputs. Preliminary attempts flatten graphs into text representations
and feed them into LLM (Chen al. 2023 Zhao et al., 2023b Guo et al. 2023 Zhao et al. 2023a
Qian et al. 2023). These approaches can benefit from LLM for task fluidity but fail to model
structural information unique graph data properly (Zhao et al.) 2023b Mao al. 2024 Ye et al. 2023). Realizing this problem, follow-up work extends methods in vision-language domain (Alayrac al. 2022 Li al. (023) to the graph domain and train adapters link graph model outputs
to LLM (Tang et al. 2023; 2024, Huang et al. 2024; Zhang et al.] 2024, He & Hooi 2024). For example, GraphGPT (Tagg et al. 2023) first implements a text-structure alignment between graph representation and text embedding to pretrain a GNN. L.aGA (Chen et al. (024) creatively uses a template to represent a subgraph with pooled node embeddings for LLM input. Inspired by Q-former (Li et al.) 2023), GraphTranslator (Zhang et al., (024) aligns node and text tokens from pre- trained GNN and LLM. UniGraph (He & 2024) pretrains GNN using masked word prediction and then tuning projector map graph embedding to language space and enable zero-shot learning. However, the GNN and LLM parts of these methods are usually detached, meaning the prompt information can not attend the message-passing process.
7

---

Published as a conference paper at ICLR 2025
Table 2: Zero-shot experiment results with instruction tuning (Accuracy).
Task
Cora-Node
WikiCS
Products
ExplaGraphs
Cora-Link
Way
2
10
5
47
10
5
2
LLama2-7B
47.92
73.45
40.10
58.77
27.65
58.71
64.33
57.76
48.15
Mistral-7B
60.54
88.39
63.63
71.90
43.99
70.16
74.94
68.77
49.43
OFA-Llama2
28.65
56.92
21.20
35.15
19.37
30.43
39.31
51.36
52.22
GraphGPT
44.65
18.84
50.74
UniGraph
69.53
89.74
43.45
60.23
38,45
66.07
75.73
ZeroG
64.21
87.83
31.26
48.25
31.24
51.24
71.29
LLaGA
51.85
62.73
23.10
34.15
39.72
88.09
GOFA-T
70.81
85.73
71.17
80.93
54.60
79.33
87.13
79.49
85.10
GOFA-F
69.41
87,52
68.84
80.62
56.13
80.03
88.34
71.34
86.31
EXPERIMENT
This section evaluates the proposed methods by answering the following four questions: Q1: Are the pre- wre-tranning tasks in GOFA effective for graph-language modeling and structure understanding? Q2: Does the pre-trained GOFA help with critical general graph model application, zero-shot learning? Q3: using GOFA more advantageous than L.MS in graph tasks? Q4: Does GOFA have the fluidity to handle open-ended graph-related tasks? Additionally, also include supervised experiments in Appendix F5
5.1 GOFA PRE-TRAINING
To answer Q1, pre-train the GOFA model using ICAE models on Mistral-7B (Jiang et al. 2023), optimizing the ob- jective in Equation (I) using the proposed tasks. The training details can be found in Appendix F33 After training, we eval- uate the perplexity of both GOFA and base L.M Cora, Product, and Wikics datasets (all three are not included in the pre-training). We report the perplexity in Table [ Note that during pre-training, we only update the weight of the GNN layers, and GOFA lower perplexity shows that the structural and semantic information in the node's neighbor can effectively help complete the sentence with more relevance than the original LLM. Further, to validate that training of GOFA will not affect original LLMs' ability, we input GOFA with single node graphs without any connections (denoted as GOFA-SN) to evaluate the perplexity, as shown in Table We can see that without connection information around the center node, generation and even better due to the pre-training process, showing that GOFA training does not destroy the desirable property of pre-trained LLM. Besides sentence completion, another important GOFA pre-training objective is the structure learning ability. We report the shortest path distance and common neighbor count prediction results Table I compared with LLM models whose inputs are textualized graphs with descriptions of edge connections. The datasets we used were Cora and Product. We see significant performance improvement of GOFA over base LLM, showing that difficult graph task for LLM can be well solved by the GNN layers with better structure modeling ability.
Table Evaluation for pre- trained GOFA. (RMSE for SPD and CN)
Perplexity SPD CN
Mistral 7B
30.12
1.254 1.035
GOFA-SN GOFA
26.20
21.34
0.634
0.326
single node graph remains comparable LLM
5.2 ZERO-SO LEARNING WITH GOFA
To answer Q2, we performed zero-shot experiments on various graph tasks. Despite using QA-chain data in the pre-training stage, the graph data does not include knowledge about task formats like classification and does not output exact matches to the answers. Hence, we first instruction-tuned the pre-trained GOFA in Section 5. on a small amount of data. We report the zero-shot results of two GOFA instruction tuning settings named GOFA-T and GOFA-F, shown in Table 2 and Table GOFA-T includes node and link classification tasks from Arxiv and Pubmed and GOFA-F addtionally adds MAG240M and Wiki90my2 datasets. The instruction-tuunie details can be found in
8

---

Published as a conference paper at ICLR 2025
Table Zero-shot experiment re- sults with instruction tuning FB15K237 and Scene Graphs.
Produdt
W8dC5
Corb Rode
MID---0 M0 Fut
Accuraa (H4)
Task
FB15K237
SceneGraphs
Format
10-Way
QA
Llama2-7B
48.32
38.62
Mistral-7B
62.48
45,95
Billons Tekers
20
FB13K237
Product
CooLink
GOFA-T 73.59
34.06
Figure 4: Performance vs Figure Pre-training Tasks Ablation Study.
GOFA-F
80.69
31,36
pre-training sample size.
Table 4: Comparison between GOFA and LLM with the same input.
Task Metric
ExplaGraphs Acc
Time sec/sample
WikiCS Acc
Time sec/sample
Cora-Link Acc ?
Time sec/sample
FB15k237
Time sec/sample
Acc f
LLM-N
74.13
1.50
OOM 71.17
OOM
50.36
3.84
51.25
3.92
GOFA-F
79.49
2.43
85.10
73.49
Improvement
7.23%
68.00%
NA
68.98%
56.51%
43.40%
14.03%
Note that the zero-shot datasets are unseen during both pre-training and instruction finetuning. The goal of instruction fine-tuning is not to let the model learn particular knowledge from these datasets but to make the model understand the task format described in Appendix [:4
Appendix F.4
While the instruction-tuning dataset only covers the relatively small spectrum of the graph datasets, we observe that GOFA achieves very non-trivial performance on all node-level (Cora-Node, WikiCS, Products), link-level (FB 15K237, Cora-Link), and graph-level (ExplaGraphs, SceneGraphs) tasks. GOFA also generalizes to different ways and even its desirable fluidity. GOFA outperforms LLM and graph foundation model baselines most datasets and exceeds best baselines by a large margin (> 10%) on WikiCS, Products, FBI 15K3377 and ExplaGraphs, showing GOFA's ability combine the advantage of both LLM and graph models. GOFA not only achieves remarkable results on the knowledge graph and academic graph, which are proximal the trained data but also excels in Products and ExplaGraphs whose distribution shifts significantly from training data, which further highlights GOFA 's substantial generalizability. Meanwhile, we observe that GOFA is only achieving comparable performance I.I. on SceneGraph dataset. We suspect that the instruct-tuning data contains information-denss texts, reducing the model's ability on common sense questions that this dataset requires. the future, we plan to diversify instruction-tuning datasets with common sense knowledge enhance such ability,
(SceneGraphs) tasks, showing
We further conducted the same experiments on intermediate pre-training checkpoints, and show results in Figure We observe that as the model witnesses more pre-training samples/tokens, the downstream task performance also increases significantly, confirming the importance of large-scale pre-training graph data. The performance continues to improve, meaning that the model can potentially scale to higher capability with more samples; we leave this to future work. In Figure 5 plot the instruction-tuning performance when remove the Wikipedia datasets and information retrieval task (w/o R+W), only remove the retrieval task, (w/o R), and full tasks, We can see that Wikipedia datasets improve the model performance of all the datasets for the diverse corpus introduced. The retrieval tasks particularly improve the knowledge graph performance due to the improved ability to retrieve key correlations between target entities. These show the necessity and effectiveness of the overall pre- pre-training task selection and design. In Appendix B. we further conduct zero-shot experiments on molecule datasets.
5.3 COMPARING GOFA WITH LLMS
Answering Q3 critical to understanding the necessity the GNN layers and the effectiveness of GOFA general graph model. We compare GOFA to LLM whose textual prompt contains the same information as the input graph to GOFA. Specifically, for a GOFA input graph, we concatenate all node texts the prompt and append the connection information to as in "Node A connects

---

Published as a conference paper at ICLR 2025
Input Graph
GOFA Generated Answer
Answer:
paceers, plasess creathvelly
wbout surmee retesarcc idless.
Based on the papers you have provided, here are some potential research ideas: 1. Adversarial attacks on generative models: The paper "Adversarial Examples in the Physical World" demonstrates that adversarial examples can crafted in real-world settings. We could extend it to GANs create adversarial examples capable of deceiving these models... 2. Generative models for drug diacovery
Please summartze pappers.
Answer:
papers is the application GANs to generste realistic images. The first paper introduces new loss function for GANs that incentivizes the generator to create images that closely resemble real ones, rather than merely aligning with those Judged as similar by the discriminetor. The second paper...
The concept
Ploase output the shortost ciscaacc betweon (NOCE.F) (NODE. Hi and all ahoriies putlis.
Answer:
The shortest path distance
The shortest paths between [NODE.F] (NODE.H] are: [NODE.F] F] - [NODE.X] - [NODE [] [NODE.H] and [NODE.F] - [NODE.Z] - [NODE.S] - [NODE.H).
Figure 6: GOFA diverse responses open-ended questions.
to Node B". The text then combined with task and question descriptions as input to an LLM for classification tasks. Approaches similar to this are widely adopted and acknowledged (Chen et al.) 2023 Fatemi et a..). We present both the classification performance and per sample inference time in Table 4 and denote the LLM method as LLM-N. We observe impressive performance improvement of GOFA on all datasets, even when the LLM is prompted with the same information, showing that GOFA, with the help of the GNN and interleaving design, utilizes the graph information much more effectively. Moreover, we also observe fundamental reduction in inference costs, confirming our analysis in Section 3.2 that, with the same input, GOFA is more efficient than LLMs. Note that when the input size is large, such as in WikiCS, LLM struggles with high memory consumption of the long sequence, whereas the GOFA avoids that by leveraging the sparsity of graph data and using edge information to compute the most important attention information.
5.4 GOFA RESPONSES ON DIVERSE TASKS
Finally, we answer Q4 by providing generation examples of GOFA in Figure where we prompt the same citation graph differently and achieved corresponding and high-quality responses. The top and middle examples have the same connection for their NOGs (both connected to the same five nodes), but when change the prompt text the NOGs, the generated texts also adjust accordingly, utilizing the neighbor node information, validating that the message-passing is conditioned on the prompt. As in the bottom example, we can also prompt the graph differently by connecting the NOG to two target nodes and querying about the shortest path distance. In this case, the model successfully generates actual paths between the two nodes, which is an ability not seen traditional graph models that can only output numerical predictions about the path length. These examples demonstrate GOFA's outstanding ability to answer open-ended questions. More examples are provided in B.4
CONCLUSION, LIMITATIONS, AND FUTURE WORKS
We introduce GOFA, a generative One-for-All graph foundation model. GOFA is pre-trained under a graph completion framework to enable large-scale self-supervieed learning. By integrating GNN layers with L.LM layers, GOFA combines the generative capabilities of LLMSs for free-form output with the structural learning strengths of GNNs for understanding complex graph connections. Our experiments demonstrate that GOFA, when fine-tuned with a small number of data, achieves impressive zero-shot performance in unseen datasets. However, our zero-shot experiment mainly focuses on evaluating models on unseen datasets with similar task formats instead of unseen task formats. Additionally, we employ frozen LLM compressor in GOFA hence, the compression capability is not naturally unified with the graph data, potentially impacting the adaptability of the model. We will explore harder zero-shot settings and better fine-tuning strategies in the future.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENT
This work is partially supported by the National Key R&D Program of China (20222D0160300) and NSF grant CBE-2225809.
REFERENCES
Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pp. 1-3, 2024.
Jean-Baptiste Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning, Advances in neural information processing systems, 15:23716-23753 2022.
Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learning for large vision models. arXiv preprint arXiv:2112.00785, 2023.
Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annic S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong. Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li. Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel J. Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong. Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R' e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang. Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. ArXiv, 0tt//1088.02555 2021. URL https::/ semant www.aanncccccoooco wrg/Corpus ID: 237091588
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL
fttp://ecppppeeeeee//eeeeeeeeee files/paper)
Runjin Chen, Tong Zhao, Ajay Jaiswal, Neil Shah, and Zhangyang Wang. Llaga: Large language and graph assistant. arXiv preprint 0rXiv:2402.08170, 2024.
Zhikai Chen, Haitao Mao, Hang Li. Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Wengi Fan, Hui Liu, and Jiliang Tang. Exploring the potential of large language models (Imm)) in learning on graphs, 2023.
11

---

Published as a conference paper at ICLR 2025
Quanyu Dai, Xiao-Ming Wu, Jiaren Xiao, Xiao Shen, and Dan Wang. Graph transfer learning via adversarial domain adaptation with graph convolution. IEEE Transactions on Knowledge and Data Engineering. 35(5):4908-493) 4922, 2022.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations, 2023.
Bahare Fatemi, Jonathan Halerow, and Bryan Perozzi. Talk like graph: Encoding graphs for large language models. The Twelfth International Conference on Learning Representations.
Jiarui Feng. Yixin Chen, Fuhai Li, Anindya Sarkar, and Muhan Zhang. How powerful are k-hop message passing graph neural networks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances Neural Information Processing Systems, 2022. URL
hitts::/eppnzeet
net/
Jiarui Feng, Lecheng Kong, Hao Liu, Dacheng Tao, Fuhai Li, Muhan Zhang, and Yixin Chen. Ex- tending the design space of graph neural networks by rethinking folklore weisfeiler-lehman. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Ad- vances in Neural Information Processing Systems, volume 36, pp. 9029-9064. Curran Asso- cc wapprrrrpppppppapppppaeaeeee..ea files/paper) www.raappprrrrreeeeeeeeeeeeeeeee pdf
ciates, Inc., 2023. URL httph: hRRLhhcpsss/pprceeiss...reetreee
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric, 2019.
Mikhail Galkin, Xinyu Yuan, Hesham Mostafa, Jian Tang, and Zhaocheng Zhu. Towards foundation models for knowledge graph reasoning. In The Twelfth International Conference on Learning Representations, 2023.
Tao Ge, Hu Jing, Lei Wang, Xun Wang. Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model. In The Twelfth International Conference on Learning Representations, 2023.
Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1263 1272. PMLR, 2017.
Jaaya Guo, Lun Du, Hengyu Liu, Mengyu Zhou, Xinyi He, and Shi Han. Gpt4graph: Can large language models understand graph structured data ? an empirical evaluation and benchmarking, 2023.
Stephen J Hardiman and Liran Katzir. Estimating clustering coefficients and size of social networks via random walk. In Proceedings of the 22nd international conference on World Wide Web, pp. 539-550, 2013.
Xiaoxin He, Xavier Bresson, Thomas Laurent, Adam Perold, Yann LeCun, and Bryan Hooi. Har- nessing explanations: LLM-to-LM interpreter for enhanced text-attributed graph representation learning. In The Twelfth International Conference on Learning Representations, 2024a. URL
https://openzeet wtpparreeeew.eeetoooruuiioooo.oe Lott://ooumldd-RPeeey..t
net
Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh V. Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. G-retriever: Retrieval-augmented generation for textual graph understanding and question answering, 2024b.
Yufei He and Bryan Hooi. Unigraph: Learning a cross-domain graph foundation model from natural language. arXiv preprint wrXiv:2402.13630 2024.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 9()): 1735-1780 nov 1997. ISSN 0899-7667. doi: 10. wttp///cco.199...... URL https
A.doi.orgre 112....11.1.cc0 1997. 9. 17351
12

---

Published as a conference paper at ICLR 2025
Zhenyu Hou, Yufei He, Yukuo Cen, Xiao Liu, Yuxiao Dong, Evgeny Kharlamov, and Jie Tang. Graphmae2: A enhanced masked self-supervised graph learner. In Proceedings of the ACM Web Conference 2023, wwW '23, pp. 737-746. New York, NY, USA, 2023. Association URL
for Computing Machinery. ISBN 9781 150339441.... doi: 10.1 00.11453543357735583333
httts:://dol.org 11t.//....1113355755..33333
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International
Conference on Learning Representations, 2022. URL httpsss/opennerllewweeee/////e// id=nzevkeeFYtt
Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. Ogb-Isc: A large-scale challenge for machine learning on graphs, 2021a.
Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. Ogb-Isc: A large-scale challenge for machine learning graphs. arXiv preprint vrXhv:2103.094330 2021b.
Qian Huang, Hongyu Ren, Peng Chen, Gregor Kržmane, Daniel Zeng, Percy Liang, and Jure Leskovec. Prodigy: Enabling in-context learning over graphs. arXiv preprint arXiv:2305. /2600, 2023a.
Xuanwen Huang, Kaiqiao Han, Yang Yang, Dezheng Bao, Quanjin Tao, Ziwei Chai, and Qi Zhu. Can gnn be good adapter for llms? arXiv preprint arXiv:2402. 0..402..2984, /2984, 2024.
Yinan Huang, Xingang Peng, Jianzhu Ma, and Muhan Zhang. Boosting the cycle counting power of graph neural networks with i$^2S-GNNs. The Eleventh International Conference on Learning Representations, 2023b. URL
net. Eorum? wtt.mmdddddSSmSmSXXXXX
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
Bowen Jin, Gang Liu, Chi Han, Meng Jiang, Heng Ji, and Jiawei Han. Large language models on graphs: A comprehensive survey, arXiv preprint arXiv:2312.02783, 2023.
Nicolas Keriven and Gabriel Peyré. Universal invariant and equivariant graph neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. netoorss In International Conference on Learning Representations, 2017. URL httttppss///eeeeeeeee
Beet/Eorum?1d=sSu0
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloc Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 4015 2023.
Lecheng Kong, Yixin Chen, and Muhan Zhang, Geodesic graph neural network for efficient graph representation learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://cpenreviie net. forum? (0=oumm11=6cc00077771
Lecheng Kong, Jiarui Feng, Hao Liu, Dacheng Tao, Yixin Chen, and Muhan Zhang Mag-gnn: Reinforcement learning boosted graph neural network. In A. Oh, T. Neu- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 12000-12021. Curran Associates, Inc., 2023. URL http::ppp//pproceddnggoo.e neur ips wtappp.cc.ppppeee wwtppccr_ppaiiiceeeeeeeeeaepeepe wwteeerfffllsepppepeeeeeee wwwlllepppppppO0/ /003/ e/ eele/18888 20/22788444om cd het.e:4422243308038828 184 -aaeedd----pprrree Conrr-Canneeecccceeemem .dd
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping Linguage-image pre- pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19740-19742. PMLR, 2023.
13

---

Published as a conference paper at ICLR 2025
Yuhan Li, Peisong Wang, Zhixun Li, Jeffrey Xu Yu, and Jia Li. Zerog: Investigating cross-datases zero-shot transferability in graphs. arXiv preprint arXiv:2402. 11335, 2024.
Hao Liu, Jiarui Feng, Lecheng Kong, Ningyue Liang, Dacheng Tao, Yixin Chen, and Muhan Zhang. One for all: Towards training one graph model for all classification tasks. In The Twelfth International Conference Learning Representations, 2023a.
Jiawei Liu, Cheng Yang. Zhiyuan Lu, Junze Chen, Yibo Li, Mengmei Zhang, Ting Bai, Yuan Fang. Lichao Sun, Philip S Yu, et al. Towards graph foundation models: A survey and beyond. arXiv preprint arXiv:2310. 1829, 2023b.
Yixin Liu, Ming Jin, Shirui Pan, Chuan Zhou, Yu Zheng, Feng Xia, and Yu Philip. Graph self- supervised learning: A survey. IEEE transactions on knowledge and data engineering, 35(6): 5879-5900, 2022.
Zemin Liu, Xingtong Yu, Yuan Fang, and Xinming Zhang. Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. In Proceedings of the ACM Web Conference 2023, 2023c.
Haitao Mao, Zhikai Chen, Wenzhuo Tang, Jianan Zhao, Yao Ma. Tong Zhao, Neil Shah, Michael Galkin, and Jiliang Tang. Graph foundation models. arXiv preprint arXiv:2402.02216. 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2022.
Péter Mernyei and Cätälina Cangea. Wiki-cs: A wikipedia-based benchmark for graph neural networks. arXiv preprint arXlv:200.00291,,20 2020.
Yujic Mo, Liang Peng, Jie Xu, Xiaoshuang Shi, and Xiaofeng Zhu. Simple unsupervised graph repre- sentation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7797-7805, 2022.
Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, pp. 4602 2019.
Moritz Plenz and Anette Frank, Graph language models, 2024. URL htttps:///aaxv.v.rrr
abs/2401. 07105
Chen Qian, Huayi Tang, Zhirui Yang, Hong Liang, and Yong Liu. Can large language models empower molecular property prediction?, 2023.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-16. IEEE, 2020.
Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. Advances Neural Information Processing Systems, 33, 2020.
Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjin Wang, and Yu Sun. Masked label prediction: Unified message passing model for semi-supervised classification, 2021.
Bing Su, Dazhao Du, Zhao Yang, Yujie Zhou, Jiangmeng Li, Anyi Rao, Hao Sun, Zhiwu Lu, and Ji-Rong Wen. A molecular multimodal foundation model associating molecule graphs with natural language. arXiv preprint Xi0::220900548// 2022.
Xiangguo Sun, Hong Cheng, Jia Li, Bo Liu, and Jihong Guan. All in one: Multi-task prompting for graph neural networks. Proceedings of the 29th ACM SIGKDD Conference on Knowledge New York, NY. USA, 2023. Association
Discovery and Data Mining, KDD '23, pp. for Computing Machinery. ISBN 9798400701030. doi: 00.1145/5803005355922 :ttpp:://6i..oo//0111155.3.
URL
14

---

Published as a conference paper at ICLR 2025
Jiabin Tang, Yuhao Yang. Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin, and Chao Huang. Graphgpt: Graph instruction tuning for large language models, 2023.
Jiabin Tang, Yuhao Yang. Wei Wei, Lei Shi, Long Xia, Dawei Yin, and Chao Huang. Higpt: Heterogeneous graph language model. arXiv preprint arXiv:2402. 16024, 2024.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:22 01022//..000 109085, 2022.
Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Rémi Munos, Petar Veličković, and Michal Valko. Bootstrapped representation learning on graphs. In ICLR 2021 Workshop on Geometrical and Topological Representation Learning, 2021.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull. David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nic, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017
Petar Veličković, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. arXiv preprint arXiv: 10341, 2018.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference Learning Representations, 2018. URL
Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov. Can language models solve graph problems in natural language?, 2023.
Song Wang, Kaize Ding, Chuxu Zhang, Chen Chen, and Jundong Li. Task-adaptive few-shot node classification. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1910-1919, 2022.
Xixi Wu. Yifei Shen, Caihua Shan, Kaitao Song, Siwei Wang, Bohang Zhang, Jiarui Feng, Hong Cheng. Wei Chen, Yun Xiong, al. Can graph learning improve task planning? arXiv preprint arXiv:2405. 191 19, 2024.
Lianghao Xia, Ben Kao, and Chao Huang. Opengraph: Towards open graph foundation models. arXiv preprint wr.iv:2403..01200 2024.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful graph neural networks? In International Conference on Learning Representations, 2018.
Junhan Yang. Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Samjay Agrawal, Amit S, Guangzhong Sun, and Xing Xie. Graphformers: GNN-nested transformers for representation learning textual graph. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL httLhttt:://// /ttps/////ppereeeeeee .ew forum?
id id=yILzFB LLLZBBBRRR ROO
Ruosong Ye, Caiqi Zhang, Runhui Wang, Shuyuan Xu, and Yongfeng Zhang. Natural language is all graph needs, 2023.
15

---

Published as a conference paper at ICLR 2025
Chengxuan Ying. Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie- Yan Liu. Do transformers really perform badly for graph representation? In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL BRL
ew. net/ fo rum?? ? d=DeRooOxF
Xingtong Yu, Zhenghao Liu, Yuan Fang, Zemin Liu, Sihong Chen, and Xinming Zhang. Generalized graph prompt: Toward a unification of pre-training and downstream tasks on graphs. 2023.
Bohang Zhang, Guhao Feng, Yiheng Du, Di He, and Liwei Wang. complete expressiveness hierarchy for subgraph gnns via subgraph weisfeiler-lehman tests, 2023a.
Bohang Zhang, Shengjie Luo, Liwei Wang, and Di He. Rethinking the expressive power of GNNs via graph biconnectivity. In The Eleventh International Conference on Learning Representations, 2023b. UR.httpp::/ppeeeeeiiiee wet.eeiwwwwweeeeeeeeeeeeeeeeeeee
net Eww.moid=rrrrrreeere ott.pp/aeeeeeeeeeeeeooooooooooaa 7
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: instruction-tuueed audio-visual language
model for video understanding. arXiv preprint arXiv:2306.02888, 2023c. URL https: //
02858
Mengmei Zhang, Mingwei Sun, Peng Wang, Shen Fan, Yanhu Mo, Xiaoxiao Xu, Hong Liu, Cheng Yang. and Chuan Shi. Graphtranslator: Aligning graph model to large language model for open-ended tasks. arXiv preprint arXiv:2402.07197, 2024.
Muhan Zhang and Pan Li. Nested graph neural networks. Advances in Neural Information Processing Systems, 34:15734-15744 157477 2021.
Muhan Zhang. Pan Li, Yinglong Xia, Kai Wang, and Long Jin. Labeling trick: theory of using graph neural networks for multi-node representation learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL /tttps://openrvviiie
net/ fort/forum?..c de
Ziwei Zhang, Haoyang Li, Zeyang Zhang. Yijian Qin, Xin Wang, and Wenwu Zhu. Graph meets 1lms: Towards large graph models. In NeurlPS 2023 Workshop: New Frontiers in Graph Learning, 2023d.
Haiteng Zhao, Shengchao Liu, Chang Ma, Hannan Xu, Jie Fu, Zhi-Hong Deng, Lingpeng Kong, and Qi Liu. Gimlet: A unified graph-text model for instruction castucction-baed molecule zero-shot learning, 2023a.
Jianan Zhao, Le Zhuo, Yikang Shen, Meng Qu, Kai Liu, Michael Bronstein, Zhaocheng Zhu, and Jian Tang. Graphtext: Graph reasoning in text space, 2023b.
16

---

Published as a conference paper at ICLR 2025
APPENDIX
A IMPLEMENTATION DETAILS
A.1 IN-CONTEXT AUTOENCODER (ICAB)
This section briefly introduces ICAE and how it helps build the GOFA model; please refer to ICAE paper {Ge et al., 2023) for the specifics of the model. ICAE contains two decoder-only LLMs. One serves as language compressor that compresses sentences into fixed-length sequence of vectors, and the other serves as a language decoder that decodes or queries into the compressed sentence representations. Specifically, during training, an input token sequence æ ["',,... #1) } is appended by a K memory tokens {m1.... mg} with trainable embeddings. The concatenated sequence is fed to the LLM compressor with a LoRA adapter (Hu et al. 2022).
(h(21),...,h (mx)), h(m)).... h(mx)] = LLMeompl complp/(pp)))).. e(ar), ( <(..,,mm)m))))))))) (4)
where e(-) and h(-) are the token embeddings and LLM outputs. Then, the decoder LLM only attends to the memory token outputs and tries to decode the original sentence from the memory tokens.
((mm))..,,Mm (mxx) ( (((=1),-,,,(1))))
(ff(mm)))) hmm).,,,,,mmm,,,, e(z1), -(m) min CrossEntropy (assEEntropp((((mm)))) (1(mx), (x))), ((.,),.., (x=-1)), (zi,,.., ).
(5)
@comp
The ICAE model is also trained on QA and Language modeling tasks to have more diverse embed- dings.
By training this auto-encoder objective on a large-scale, the compressor model learns to compress all information about a sentence to the memory token outputs like in a conventional model. In Table[5] we provide a few examples of the comparison between the original text and the text decoded from the compressed memory tokens by ICAF's decoder. Because the compressed representation contains as much information as possible, GNN can pass messages between nodes with minimal information loss.
Table 5: Comparison between original texts and decoded text from the compressed representation.
Original Text
Decoded Text
Actress Halle Berry has been sharing number of stunning photos from the time she has spent in Morocco and she just posted a new one to her Instagram page that fans will not want to miss.
Halle Berry has been sharing a number stun- ning photos from the time she has spent in Mo- rocco and just posted a new one on her Instagram page that fans won't 't want to miss.
Utah avoided the turnover bug on Saturday for the first time since its season opener. In addition, the running game was clicking and the defense was dominant as the Utes snapped a two-game win- ning streak on the road, beating Pittsburgh 26-16 14. Five keys to Utah's victory: 1. Utah running back John White IV: Running strong and with purpose from the beginning, White was a big reason why the Utes were within striking distance at halftime. White, who took couple pops that dislodged his helmet and caused a cut below his ear, seemed to get stronger the game wore on. He finished the afternoon with 171 yards on 36 carries.
Utah avoided the turnover bug on Saturday for the first time since its season opener. In addition, the running game was clicking and the defense was dominant as the Utes snapped two-game win- ning streak on the road, beating Pittsburgh 26-14 14. Five keys to Utah's 's victory: 1. 2. 3. 4. 5. Utah running back John John White IV IV ran strong and with purpose from the beginning, being a big reason why the Utes were within striking distance at halftime. He took couple of shots that dis- lodged his helmet and caused a cut below his ear, but seemed to get stronger as the game went on. He finished the afternoon with 171 yards on 36 carries.
A.2 LLM CHOICES OF GOFA
Because ICAE preserves as much information in a sentence as possible, we can use it in the GOFA model to comprehensively pass information between sentences, as shown in Section 3.2 However,
17

---

Published as a conference paper at ICLR 2025
the GOFA model is not limited to ICAE. Users can first train ICAE-like objective any existing LLM and apply the GOFA model to the trained LLM. Or, users can apply the GOFA directly to pre-trained LLM and train the GOFA without the auto-encoder training. Note that the ICAE architecture has a function similar to an encoder-decoder L.M. We do not use an off-the-shelve encoder-decoder LLM because its encoder output still subject to the sentence length, which does not fit GNN's need for fixed-sized input.
The design of GOFA can be extended beyond a compressor-decoder architecture. For example, can have a decoder-only GOFA whose LLM layer is,
(Q
H2 Q 2 LLM'((Q', H (2)
(6)
where the GNN is still applied on memory tokens inserted between the node text & and target text y. This allows the target text to attend to the node text, which may improve the performance of GOFA. However, this formulation forces every node to have target text, which is usually not what users desire and poses extra computation costs. We will explore this architecture in our future work.
A.3 TRANSFORMER CONVOLUTIONAL GNN
As mentioned in Section 3.2 we customize a Transformer Convolutional GNN(TransConv) (Shi al., (02T) as the GNN used in Equation Since GNN layers operate on token representations and tokens at different indices do not communicate, we describe the GNN at one index for simplicity. The 1-th GNN layer on node and neighbors (()) is:
h (i) W.( Σ (i)
aij (W Weenodehi (j) + W Wr,edgeh(eis)))
JEN(()
(1) * nodeed (f) + Wk.edge h(ess))
04 = Softmax
Vd
h(-) represents input node and edge features. W represents query (g), key (k), value (V), output (o) lin- ear projection for nodes and edges. The formulation closely follows the transformer design (Vaswani al. 2017) and its GNN adaptation (Shi al,, 202T]. This formulation does not aggregate the last layer embedding (i) into the next layer, because already add residual to maintain the same effect. We use pre-layer normalization following Llama (Touvron et al. 2023).
B ADDITIONAL EXPERIMENTS
B.1 ADDITIONAL. ZERO-SHOT EVALUATION RESULTS
Table 6: Additional comparison on zero-shot learning results. (The number indicates the number of ways).
Table 7: Zero-shot learning with
molecule dataset.
Cora-node (7)
WikiCS (10)
Products (10)
BBBP
HIV
DGI
20.15
13.63
21.35
OFA
35.67
GraphMAE2
37.08
19.03
25.46
MoMu
49.81
50.26
Best-LLM
60.54
63.63
70.16
Galactica
53.94
33.85
Best Graph-LLM
69.53
43.45
66.07
GIMLET
59.39
66.24
GOFA-T
70.81
71.17
79.33
GOFA
54.91
53.02
To further demonstrate the advantage of GOFA compared to previous methods, we conduct additional zero-shot experiments over different datasets and baselines. First, we additionally compare the zero- shot result with traditional GNN baselines. Since models like GCN and GAT cannot do zero-shot tasks due to their supervised training nature, we compare against graph self-supervised learning baselines. These methods learn node embeddings, which are compared to label text embeddings to make predictions based cosine similarity. Specifically, we include DGI Veličković et al. (2018) and GraphMAE2 Hou et al. (003) and follow the same evaluation procedure. The result is shown in Table GOFA significantly outperforms methods that GNNs for self-supervised learning in the zero-shot setting. This result further demonstrates the effectiveness the architecture design and the graph modeling tasks of GOFA zero-shot generalization ability over graph tasks.
18

---

Published as a conference paper at ICLR 2025
Next, we evaluate the GOFA on more numerically and structurally intensive tasks. Specifically, conduct experiments on molecular property prediction tasks using ogbg mooly and BBBP datasets. The experiments closely follow the setting of GIMLET Zhao et al. (2023a). First, we fine-tune the pre-trained GOFA on randomly sampled 1 100,000 question-answen
pairs from the Chembl pretrain dataset. The questions concentrated on asking about different molecule properties. Next, we evaluate the fine-tuned GOFA on ogbg-molhiv and BBBP in a zero-shot setting. We use AUROC as the evaluation metric. For comparison, we include OFA Liu et al. (2023a), MoMu (Su et al. (2022). GIMLET (Zhao et al. (2023a), and Galactica Taylor et al. (022). The results are shown in Table We directly report results of OFA Liu al. (2023a) from their paper, and all other baselines are referred from GIMLET Zhao et al. (023a)). We can see that with only 100,000 instruction-tuning samples, GOFA already achieves comparable results to baselines. Note that GIMLET fine-tunes its model on the whole Chembl-pretraie dataset, which contains more than 400 million question-answering pairs.
B.2 SUPERVISED EXPERIMENT RESULTS
In this section, we conduct a supervised learning experiment with the pre-trained GOFA In the supervised experiment, GOFA 's prompt does not include class optional. We show the supervised results in Table [ Specifically, we compare the result of GOFA with the following baselines: 1.
basic GNNs, which are trained individually on each dataset, including GCN (Kipf & &Welling (017)
and GAT (Velćkkvić et al.
2018)_ 2. The contrastive learning methods, including DGI (Veličković
et al. 2018) and BGRL Thakoor et al.] 202T). For these methods, we directly report the best result from (He Hooi|
2024). 3. Graph foundation model, including OFA (Liu et al., 2023a)
and UniGraph (He & Hooi| (024)). GOFA achieved competitive performance on most datasets. In
particular, GOFA achieved SOTA performance on the Pubmed dataset, demonstrating that GOFA can transfer pre-trained knowledge to downstream tasks. We also notice that GOFA is not performing as well on some datasets, possibly because in a supervised setting, we only train a small portion of the data for one epoch (specific numbers in the experimental details section in Appendix (F5e and in the supervised setting, it is important to see training samples multiple times to ensure detailed understanding of the distribution. As pre-train the model with more diverse datasets, GOFA can potentially obtain world knowledge as an LLM, which makes transfer learning in the supervised setting more accurate.
Table 8: Experiment results in supervised learning. Bold and underlined shows best and runner-up results.
Cora Link Acc t
Cora Node Acc 1
PubMed Link Acc f
PubMed Node Acc
Arxiv Node Acc f
WikiCS Node Acc f
WN Link Ace f
FB Link Acc f
Products Node Acc f
Task type Mctric
GCN GAT
12 78.9
77.5204
77.8
73.9206
77.0206 79.8
82.7=04
90.1
80.0107
60.1+a.
80.4+:04
80.5+a2
76.6
75.8
88.8+03
93.6#0.1
81.4+02
DGI BGRL
51.99
55. 55.7611066
55.21
67. 07:11
52.04-0:22
26, 2..9
64. :1.32
06.73.1023
63.77:023
62.21
70.12:0.15
06.44:0021
64.91
1:022
63.77:0223
OFA UniGraph
87.97
75.34
95.89
77.89
73.44
77.62
98.31
95.78
81443-0
74.33++223
72.91
#0.42
79.98+121
85.45
94.81
80.11+022
GOFA
89.54
76.50
93.97
83,83
74.77
79.96
92.16
88.21
79.98
B.3 EVALUATION ON ADDITIONAL GRAPH STRUCTURAL TASKS
To further evaluate the potential of GOFA to serve as a general-purpose graph foundational model, we conducted additional experiments on a variety of graph structure tasks, including node degree, shortest path distance, and node count. To provide fair comparison, we selected Mistral-7B as the baseline model and designed the prompts for Mistral following the incident prompt format proposed in (Fatemi
Table 9: Evaluation on additional structural tasks (Acc).
Node degree SPD Node Count
Mistral GOFA
98.6
99.8
99.9
98.2
99.7
86.3
et al.).. Specifically, the graph is described as "[NODE.A] is connected to [NODE.B]' We use Arxiy as the graph dataset and randomly selected 10000 samples for fine-tuning and 2000 samples for testing. For the mistral model, we remove all original text from the dataset, as the model will run out of memory with all the text kept. The evaluation result is shown in Table 9 We can that
19

---

Published as a conference paper at ICLR 2025
GOFA achieved competitive performance the LLM baseline in both node-level (node degree) and link-level (shortest path distance) structural tasks. However, LLMs perform better on graph-level tasks. We suspect that the reason lies in the fact that we keep the original text in the graph dataset, which makes the learning of structure much harder, especially when the number of nodes the graph becomes large. At the same time, our pre-training tasks foeus on understanding local graph structures. Thus, GOFA is unfamiliar with understanding global structural information like node count. In the future, we aim to include more structural tasks further improve the generalization ability of GOFA on different graph tasks.
B.4 EXAMPLE OF GOFA'S FREE-FORM ANSWER
Figure 6 in the main body illustrates GOFA's capability to respond various questions based on the same graph from ogbn-arXiv. In this section, we provide additional examples in Figure 7 to further show the ability of GOFA's free-form text answer.
Text Feature of Nodes Connected with Prompt Node
Input Graph
GOFA Generated Answer
Given these products., please find the majority cotagory amrng them.
Node.A: Tech Armor Clear Screen Protectors with Lifetime Warranty for Sarnaung Galany S3 s IlI. 600 Multicolored
Anewer:
Node. Refill Band Value
The majority category for these products is phone accessories.
Polka Dot with Clipa.
Node.C: C: DandyCase 2in1 Hybrid High impoct Hard Vintage Saa Green Floral Pottarn Pink Silicona Case Cover For Samsung Galaxy $3 19300 + DandyCase Screen Cleaner.
Products
Aode, Wikipedia entry. Entry neme: Linux, Entry
Given these vrlipedii
content: linux liuu family opan source unix-Like...
Answer:
wilpedia page
Node.Br B: Wikipedia entry. Entry name:
Wikipedia: Linux kernel. Wikipedia:
Microsoft_ Windows, Entry content: microsoft microsoftwindowwe windows
Open source software 3. Wikipedia: Free software, Wikipedia: Software development methadologies. Wikipedia: Software achietecture.
microsoft windows group severst... ...
Noda.C: Co Wikipedia entry. Entry name:
arwhhitccturre.... Entry contont: arm architocture am previously advanced risc machine...
Node.D: D. Wikipedia entry. Entry PlayStation Entry content: playstation 2 playstation 2 officially
WikiCS
abbrevisted ps2
Demonstration of GOFA's ability respond to any question to the given graph. Above is an example of the products dataset, where the model needs to output the majority category of its connected nodes. Below is another example on Wikics dataset. GOFA is asked generate Wikipedia page named based on the graph information.
Figure 7:
C DATASETS
Cora. The Cora dataset is a co-citation network, where nodes are papers related to artificial intelli- gence. Edges mean the connected two papers co-cited by other papers. The Cora dataset contains 2708 nodes and 10556 edges. We collect the Cora dataset and its raw text from OFA (Liu et al. 2023ah. We evaluate the performance of the baseline and our proposed model on Cora for both node-level and link-level tasks. For the node-level task, the aim is to classify the node into the correct paper category from 7 different categories. The split obtained from OFA. It contains 140/500/2068 samples for train/val/test set respectively. For the link-level task, the object is to predict whether two paper nodes are co-cited or not. We follow the setting OFA (Liu et al., 2023a) and randomly split all edges into train/val/test. sets with a ratio of 0.85/0.05/0.1.
PubMed. The PubMed dataset is co-citation network, where nodes are papers related to diabetes mellitus. Edges mean the connected two papers are co-cited by other papers. The PubMed dataset contains 19717 nodes and 88648 edges. We collect the PubMed dataset and its raw text from OFA (Liu et al.) 2023a). We evaluate the performance of the baseline and our proposed model on PubMed for both leve- and link-level tasks. For the node-level task, papers have different categories. The goal is to classify the node into the correct paper category. We obtain the split directly from original source. It contains 60/500/19157 samples for train/val/test set respectively. For the link-level task, the object is predict whether two paper nodes are co-cited or not. We follow the
20

---

Published as a conference paper at ICLR 2025
setting of OFA (Luu et al. 2023a) and randomly split all edges into train/val/test sets with ratio of 0.85/0.05/0.1.
Arxiv. The Arxiv dataset is citation network, where nodes are papers related to computer science and edges mean two papers have a citation relationship. The Arxiv dataset contains 169343 nodes and 1166243 edges. We collect the Arxiv dataset and its raw text from OGB (Hu et al. 2021b). We evaluate the node classification on the Arxiv dataset. The goal is to classify the paper node into the correct category from 40 possible categories. We obtain the split directly from OGB (Hu et al. 20216). It contains /9941/29799/4860 samples for train/val/test set, respectively.
WikiCS. The WikiCS dataset is graph obtained from Wikipedia. The nodes in WikiCS are Wikipedia terms and their descriptions. The edges mean there is hyperlink between two terms. We collect the WikiCS dataset and its raw text from (Mernyei& & Cangea 2020). There are 11701 nodes and 216123 edges in the graph. We evaluate the performance of WikiCS on the node classification task. There are 10 different classes. We follow the same split as OFA (Liu et al., 2023a which contains 580/1769/584 samples for the train/val/test set, respectively.
The Products dataset is a co-purchase graph. The nodes in the graph represent product items from the Amazon platform, and the edges represent that two products are co-purchased together. We obtain the Products and their raw texts from TAPE (He et al., 2024a), which a subset from the original ogbn- -rooucts (Hu al. (0211) dataset. It contains 54025 nodes and 144638 edges. We evaluate the node classification performance on Products. The data from the original source contains 47 different categories. However, we found that there two classes with missing labels. To be consistent with previous literature, we adopt the approach in LLaGA to replace the label name with special symbols. There are 14708/1572/37745 samples for the train/val/test set, respectively.
Products.
FB15K237. The FB15K237 is a knowledge graph generated from Free Base. Nodes in the dataset represent entities in the world and edges represent the relation between entities. We obtained the dataset from OFA (Liu et al. 2023a). The FB 15K33737 is used to evaluate the link classification. The dataset contains 237 unique classes. We follow the setting of OFA (Liu al. 2023a) and split the dataset with a ratio of 0.85/0.05/0.1, which results in a total of 272115 samples for train/val/test set, respectively.
The ExplaGraphs is a graph question answering dataset on commonsense concepts. Nodes the dataset represent a common sense concept and edges represent the relation between two concepts. We obtain the dataset from G-retriever (He et al. (0024b The ExplaGraphs can be used for question-answering on graphs. We obtain the split directly from G-retriever (He et al., 2024b). contains 1659/553/554 graph samples from the train/val/test set.
ExplaGraphs.
SceneGraphs. The SceneGraphs is a graph question answering dataset on scene graphs. Nodes in the dataset represent an object in an image and edges represent the relation between two objects. We obtain the dataset from G-retriever (He et al. 2024b) The SceneGraphs can be used for question- answering graphs. We obtain the split directly from G-retriever (He et al. 2024b). It contains 59978/19997/20025 graph samples from the train/val/test set.
MAG240M. The MAG240M dataset is citation network generated from Microsoft Academic Graphs. The nodes represent academic papers and the links represent a citation relation between two papers. We obtained the dataset and raw text from OGB-Isc (Hu et al.) 2021a). However, the original dataset is extremely large and contains nodes without text features (author and institution nodes), since we mainly use the dataset for pre-training, we further downsample the original dataset. Specifically, we only keep paper nodes and citation links between papers. Further, we downsample the edges in the following ways. First, we selected all nodes in the train/val/test split provided by OGB-Isc (Hu et al. 202 Next, we filter the edges through two rounds. In the first round, we only keep the edge if either the source or the target is in the selected nodes. If any node in the added edge is not in the selected nodes, we add it to the node set. Next, in the second round, we include additional edges where both the source and target are in the selected nodes (additional nodes are added in the first round). The procedure results in a total of 5875010 nodes and 26434726 edges.
Ultrachat200k. The Ultrachat200k is a question-answering dataset. each sample a multi-roune conversation obtained from the web. We obtained the Ultrachat200k from (Ding et al. 2023). However, the original dataset is not a network. To convert it to a graph dataset, we manually create graph structure for it. Specifically, if the original sample has k round of conversation, will
21

---

Published as a conference paper at ICLR 2025
generate & - graph sample. The i-th graph will contain the first round of conversation. Each node in the graph is either a question or an answer. The question and answer are linked by a directed edge indicating the order of the conversation. The conversation of + 1 round will be the question-answer pair for this graph. Since we mainly use the dataset for pre-training. We only include train-sft subset. After the conversion, there
total of 449929 graphs in total.
Wikikg90m. Wikikg90m is an encyclopedic knowledge graph dataset extracted from Wikidata knowledge base. We obtain the original Wikikg90m from OGB-LSC (Hu al. 2021a). It contains 91, 91,230,610 entities, 1,387 relations, and 601,062,2 ,002,8811 edges.
WikiGraph. The WikiGraph dataset is designed to increase the diversity of the training texts. Hence, we use WikiText (Merity et al. 2022) dataset as the seed dataset. It contains plain sentences from Wikipedia pages. We generate graphs for sentences with more than 500 characters. Specifically, we first prompt an LLM to extract meaningful entities or concepts from sentence, and these entities become the nodes in the graph. We then randomly pair concepts generate edges. Again, we use LLM to generate description of the relationship between the paired concepts and use the description as the edge text.
The BBBP is a molecule property prediction dataset. The task is to predict the effectiveness of molecule to brain blood barrier. We collect the dataset from GIMLET /Zaao et al. (2023a). The dataset contains 163 1/204/20 molecule graphs for the train/validation/tess set, respectively. We directly use the test set for zero-shot evaluation.
BBBP:
HIV: The HIV dataset is molecule property prediction dataset. The task is to predict the effectiveness of molecules to hiv. We collect the dataset from GIMLET et al. (2023a). The dataset contains 32901/41 13/4113 13 molecule graphs for the wain//alidationntesst set, respectively. We directly use the test set for zero-shot evaluation.
Chembl pretrain: The Chembl pretrain dataset is large-scale molecule property prediction dataset. for each simple molecule, the dataset contains several questions to ask about the different properties of the molecule. Overall, it contains 400 million question-answering pairs. We collected it from GIMLET Zhao et all.(22 and use it for fine-tuning GOFA for molecule zero-shot experiments.
D RELATED WORK EXTENDED
Prompt Learning in Graph: The success of foundation models inspired many works to adapt their power to the graph domain. Earlier attempts designed graph prompting mechanism such that trained model can adapt to new data by fine-tuning soft prompting vector (Liu et al.) 2023c; Yu et al. 2023; Sun et al. 2023 Xia et al. 2024). GraphPrompt (Liu et al.) 2023c Yu et al. 2023) pretrains link prediction tasks, and then finetune a prompt matrix for each downstream node or graph classification task. All in One (Sun et al. 2023) designs prompt tokens that are used to modify node features and then take meta-learning paradigm for multi-task learning. Subsequent works extend graph prompts to allow in-context learning without weight update (Huang et al. [2023a] Galkin ctt al.) 2023). However, these works only tackle limited types of tasks and do not generalize to new domains. Hence, researchers propose integrating LLM into the graph learning.
GNNs and Transformers: In recent years, GNNs have become the most popular method for dealing with graph learning problems due to their extraordinary ability in structural learning. Particularly, Previous works (Xu et al.) 2018 Morris al., (019)) show that the expressive power of message- passing GNNs can be as powerful as the I-dimensional Weisfeiler-Lchman test, powerful algorithm for graph isomorphism problems. Many recent works also try to design more powerful GNNs that beyond the 1-WL test (Zhang & Lij 2021 Kong et al. [022] Feng et al. 2022 Huang et al. 2023b Zhang et al., 2023b a Feng et al. 2023) Kong et al. 2023) for better structural ability like learning distance between nodes or counting cycles in graph. Some works try to combine the GNN with the transformer. particularly, GraphFormers (Yang et al. (021) and GROVER (Rong et al.) 2020) also insert GNN layer between consecutive transformer layers for modeling graph inductive bias. Different from us, their transformer layers are randomly initialized and directly tuned on downstream tasks without text.
22

---

Published as a conference paper at ICLR 2025
Table 10: Comparison of LLM-generateer answers with correct answers for graph structure questions.
Question
Answer
Correct Answer
In the graph node 8 connects to node 13 connects to 9, node 8 connects to 7, node connects to 13, node 3 connects to 4, node 6 connects to 6, node 11 connects 8, node 12 connects to 11, node 2 connects to 12, node 14 con- nects 7, node 6 connects 3, node 6 connects to 9, node 2 con-
To find the shortest path between node 8 and node 9, can use Dijkstra's is algorithm or any other shortest path algorithm. However, since this graph is small, we can find it manually. Here's the shortest paths between node 8 and node 9: Start node 8. Move from node 8 to node 5 (neigh- bor). Move from node to node 8 (neighbor). Move from node 8 to node (neighbor). Move from node 2 node (neighbor). So, the shortest path between node 8
8-> ->>
of
nects 8, node 2 connects 9,
node 5 connects to 8, node 3 con- nects 11, node connects to 5, node 10 connects to 11, node connects to node connects to 10, what is shortest path be- tween node 8 and node 9?
and node 9 is: 8 - 2- 9.
In the graph node 8 connects to node 8 connects to 7, node 3 connects to 13, node 3 connects 4, node 6 connects to 6, node 11 connects to 8, node 12 connects to 11, node 2 connects to 12, node 14 con- nects to 7, node 6 connects to 3, node 6 connects to 9, node 2 con-
To find the common neighbors be- tween nodes 13 and 5, we need identify all nodes that are neigh- bors of both nodes 13 and 5. From the provided connections: Node 13 is connected node 9 and node 3. Node 5 is connected to node 8. Therefore, the com- neighbor between node 13 and node node 8
No common neighbors.
node 13 connects
nects to 8, node 2 connects to 9,
node 5 connects to 8, node 3 con-
nects to 11, node 8 connects to 5, node 10 connects to 11, node 2 connects to 9, node 9 connects to 10, what are common neighbors between 13 and 5?
E ADDITIONAL DISCUSSIONS
E. GRAPH STRUCTURE QUESTION EXAMPLE OF L.M
We assessed the ability of LLMs to respond to questions related to graph structures, including shortest path distances and common neighbor counting. For this evaluation, graph edges were described using plain text, and the LLM was tasked with generating the answers. The results of this evaluation are presented in Table I These findings indicate that LLMs struggle to comprehend graph structures effectively. include examples the questions posed and the corresponding answers generated by the LLM in Table 10 to illustrate these challenges.
F.2 THEORETICAL ADVANTAGES OF GOFA'S GRAPH LANGUAGE ENCODER
In GOFA, we innovatively integrate GNN layers into LLMs to help LLMs understand graph structures. This approach theoretically more powerful and suitable for designing GFMs than using pure LLMs. Graph data have unique properties, such as node permutation invariance without fixed ordering (errvenn & Peyre 2019). making sequential models like LLMs unsuitable for modeling graphs. For a graph with n nodes, the number of possible orders is n!, which means sequential models like LLMs will need factorial sample complexity to learn that all them correspond to the same graph. Existing works therefore random or heuristic order represent graph, resulting in
23

---

Published as a conference paper at ICLR 2025
suboptimality and poor generalization structure-related tasks. One empirical example from the task planning experiments of LLM agents in (Wu et al., (204)) shows that LLMs can only perform well on task graphs with specific node ordering but cannot maintain this accuracy after nodes reordered. Methods like LLaGA (Chen et al. 2024) also fall into this category and are suboptimal the task powerful choice widely accepted by the
requires deep structure understanding. Instead, GNNs literature for encoding both features and the structure of graphs. They are permutation equivariant to graph order and can learn invariant structure information. Our GOFA, interleaving GNN layers into LLMs, naturally preserves this property. Specifically, for LLM layers, each node is processed individually, which is obvious that we can keep the permutation invariance. As GNN layers are permutation invariant, this conclude that the GOFA is permutation invariant to input graph.
E.3 ADVANTAGES OF GOFA'S SELF-SUPERVISED LEARNING TASKS
Our proposed self-superviee learning tasks are enlightened by existing graph and NLP SSL tasks. However, our tasks are novel compared to existing methods from several perspectives.
In graph SSL, most prior work aims to recover original features or graph structure contrastively or generatively (Liu et al., 2022), using learned embeddings for downstream classifiers. In contrast, our tasks aim to learn embeddings that enable downstream natural language generation. Concretely, our SSL. shortest path prediction task requires the model to output multiple actual paths (e.g., node a-nood b-onode c) hetween two nodes in text format; this requires a more fine- fine-grained and in-depth model understanding of the graph structure. Regular graph SSL tasks such as link prediction and shortest path distance prediction only cares about simple objective (binary classification for link and single number for distance). Conversely, our generation-off -eneration-oriented design allows a unified task format query into the graph from different aspects with different granularity (e.g. shortest path and common neighbor can be incorporated under the same natural language generation framework), whereas traditional SSL tasks inevitability lose much detailed information and may need artificial complicated design to accommodate multiple SSL targets.
SSL in NLP (e.g. woxe--ood--rediction only takes texts input. One may think of directly converting graphs to text and doing a similar SSL. However, as many previous works show (Wang al. 2023), converting graphs directly texts for LLM generation is ineffective. Hence, we design the sentence completion task directly on the graph to use connection information to help the model attend correct nodes for subsequent generation.
In summary, our SSL design cares more about training the model to generate any answers in natural language format so that it can accommodate arbitrary tasks, which differs from traditional graph SSL that normally focuses on classffication/eeresssoo (actually any graph SSL tasks can be incorporated into our natural language generation framework). Compared NLP SSL, our novel SSL design focuses sentence completion using neighboring sentence information rather than pure auto-regression, strengthening the model's power to leverage joint graph-text information.
F EXPERIMENTAL SETTINGS
F.1 GENERAL SETTINGS
Subgraph sampling: In the GOFA, for node/link/raph--eevel tasks, the input format is unified as subgraph task. Namely, for node/link-level tasks, we will select a k-hop subgraph surrounding the target nodes the input graph for the model. We follow a similar subgraph sampling strategy as OFA (Liu et al. 2023a). Specifically, for node-level tasks, we directly sample the k-hop subgraph rooted the target node. Meanwhile, a certain hop exceed the threshold, we will randomly sample maximum nodes from all nodes. For link-level tasks, doing the sampling on both two nodes.
set threshold for maximum nodes per hop. If the nodes in
Implementations. Both the GOFA and all baselines are implemented using Python with Pytorch, transformers, and PyG (Fey & Lenssen 2019) packages.
24

---

Published as a conference paper at ICLR 2025
F.2 DESIGN OF TRAIIIING TASKS
In this section, we describe the self- self-supervised pre-training of GOFA. The goal of the pre-training is to let the GOFA model obtain the ability to query graph structure and context but retain the ability to reason about plain text. Specifically, we perform the pre-training task using multiple existing graph datasets, including MAG240M (Hu et al.) a0) Arxiv (Hu et al. 2021b), Pubmed, Wikikg90mv2 (Hu et al. 2021a). and Ultrachat200k (Ding et al. 2023) datasets. Further, we create another graph dataset called WikiGraph, whose graphs are generated from sentences in the pure textual WikiData dataset (Merity et al. (022) using LLM. Details about the datasets can be found in Appendix C We randomly generate a unique node ID (such as [Node A]) for each node in the training sample and append it to the original node text. This ID will serve as a basis for querying nodes in the graph. We design four pre-training tasks: sentence completion, structural understanding, question-anse -awternnn, and information retrieval tasks. Figure shows an example of each task. We describe the rationale of each task below and leave some implementation details to Appendix [ We also include an additional discussion of the advantages of our designed tasks in Appendix E.3
Sentence completion task. The objective of the sentence completion task is to train GOFA to reason about the rest of the text in node given both the existing text and the information in the rest of the graph. Given an input training sample, we randomly select n nodes in the graph as the target nodes. All selected nodes' texts are split into halves. The first half forms node text xr and the second half becomes the target y to generate. The length of the first half will also be randomly determined. Finally, the output representation of these nodes will be directly input to the decoder (no prompt node will be connected) and we minimize the loss between model decoded text and target y. This sentence-completion pre- training task adapts LLMs' standard ability to the graph context.
Structural understanding tasks. The objective of the structural tasks is to pre-train GOFA to understand basic graph structural properties. In light of this, we design the shortest paths and common neighbors reasoning tasks between nodes. Specifically, For each training subgraph sample, we randomly sample node pairs the selected targets. For each selected node pair, we ask the model to compute the shortest path distance between two nodes and output all possible shortest paths between them using the assigned node IDs. Meanwhile, we also ask the model to output the number of common neighbors the two nodes have and the node IDs of their common neighbors. For the structural understanding task, a prompt node up will connect to both two nodes since our structural tasks need the model to reason about two nodes simultaneously. The text in the prompt node will be the corresponding question. Through these two tasks, the model is expected to gain the ability to identify basic graph structures, which are critical to downstream tasks.
Question answering task. Unlike language corpus, which naturally contains many question-and answer (QA) pairs, graph data usually only contain objective descriptions of entities. Nevertheless, for the model to be fluid in tasks, we need the model to understand user prompts and be sensitive to different tasks. Hence, we synthesize QA-chain dataset from Ultrachat200k, as shown in Figure [] A language QA sequence is converted to a chain graph where nodes with question texts alternate with nodes with answer texts, which connected by directed edges to represent the conversation order. The last question becomes the text on prompt node Up. which is connected to every node in the chain, and the last answer is the target text y (see Figure QA-Chain Task for an example). This QA task provides QA pseudo-graphs missing from the common graph corpus, and we found it critical for enabling the model to be responsive to arbitrary tasks expressed in free-form text questions.
Information retrieval task. For most of the downstream tasks, GOFA requires a prompt node to link to all target nodes in the graph to solve related problems. To enable the prompt node to effectively maintain related information for solving the task in the decoding stage, we design an information retrieval task to realize these goals. Specifically, for each input graph, we randomly select n nodes and we connect a prompt node to these nodes. Next, the information retrieval task is further divided into two parts: key-to-content and content-to-key. For key-to-content, we provide a node ID (randomly chosen from the selected nodes) in the prompt node and ask the model to retrieve the text of that node. For the content-tor montent-to-key -o-kett task, we provide the content of one node (selected the same as above) in the prompt node and ask the model to return the correct node ID of that node. This task enhances the ability of GOFA to utilize our provided node IDs to retrieve and maintain correct information in the prompt node, which proves useful for many downstream tasks requiring information retrieval.
25

---

Published as a conference paper at ICLR 2025
Table 11: Detailed question and answer example in pertaining task.
Task
Question example
Answer example
Sentence completion
Complete the sentence of the tar-
The rest of the sentence in the tar- get node.
get node.
Complete the sentence of the node(NODE.A].
The rest of the sentence in node [NODEA]]
Shortest paths
Compute the shortest path dis- tance between the target node [NODELL] and node (NODE.B) |] and generate all shortest paths from the target node to the node (NODE.B) Please separate nodes the path with >>. If mul- tiple paths exist, generate all of them with an ascending order of node sequences and separate dif- ferent paths with:
The shortest path distance is Shortest paths: [NODEID.L] [NODEID.G] (NODEID.
Common Neighbors
Is there any common neighbor be- tween the target node [NODE.L] and node [NODE.B]? If it exist, please give the total number and list all common neighbors in cending order of node, separate nodes with
There is / common neighbor between two nodes, including [NODEID.G].
QA-Chain
What are the rules and restric- tions in place for COVID-19 in the city?
don't 't huve any live data regard- ing the covid-19 19 rules and restric- tions. Please check with the local authorities health department for the latest guidelines and re- strictions in your city.
Information Retrieval
Please output the content of [NODEA]]
Content on [NODEA].
Given this node content: (node content], please output the node id.
[NODEID).
F.3 PRE-TRAIN IMPLEMENTATION DETAILS OF GOFA
Dataset and task construction. As we discussed, we designed four different pre-training tasks for GOFA. Here we describe some implementation details about each task and then discuss how construct each task on each dataset.
For the sentence completion task, the node text is split by the following rule: for each node, if the node text is less than 256 words, we set the maximum left-halve length be the half of node sentence length. Otherwise, we set it to 128. Next, we randomly choose length from 0 to maximum left length the final cut point to cut the sentence into two pieces. For the shortest path task, we ask the model to output both the shortest path distance and all possible shortest paths. Since there may be multiple paths, to ensure the uniqueness of the answer, we first order all paths based on the node ID (the ascending order of alphabets) for nodes in each path and ask the model to learn this order. The construction of common neighbor task is similar. Finally, for information retrieval, given an input graph sample, we randomly select 2 to the number of nodes in the graph be the target nodes.
For pre-training datasets, we use multiple datasets including MAG240M. Arxiv, Pubmed, Wikikg90mv2, Ultrachat200k, and WikiGraph. For MAG240M, Arxiy, and Pubmed datasets, each
26

---

Published as a conference paper at ICLR 2025
training sample is subgraph sampled around node. Next, sentence completion, shortest path, and common neighbor tasks are constructed. For each sample, there are 4 complete sentences, 3 shortest path, and 3 common neighbor tasks. We will also construct information retrieval tasks on these datasets. However, to ensure a moderated graph size, the information retrieval task will be constructed separately from the above tasks and also for both key-to-content and content-to-key tasks. For each information retrieval task sample, there will be only one task. For Wikikg90m, each training sample subgraph sampled around an edge. In Wikikg90m, we additionally include link prediction task. That is, for each input graph, we randomly mask e edges and ask the model to recover the content in the edge. For each sample, there are 4 complete sentences, 2 shortest paths, and 2 common neighbor tasks, and 2 link prediction tasks. At the same time, the information retrieval task will also be generated separately. For WikiGraphh, each sample itself a graph. Similar to Wikikg90mv2, each sample consists of 4 complete sentences, 2 shortest paths, 2 common neighbor tasks, and 2 link prediction tasks and information retrieval task will also be generated separately. Finally, for Ultracha200k, we only include question answer task and each sample only contains one task. The detailed task prompts and answer examples are shown in Table
Training details. The initial weight of the LLM compressor and decoder is obtained from ICAE (Ge et al. 2023). The initial weight of all GNN layers is randomly initialized. The value of all gates in the residual connection is set 0 to ensure the initialized model performs the same as the original language model. During the training, we only tune the GNN layers. For each training epoch, the training corpus includes 500,000 MAG240M samples, 50,000 Arxiv samples, 5,000 PubMed samples, 100,000 Ultrachat200k samples, 80,000 WikiGraph samples, 100,000 Wikikg90mv2 samples. At the meantime, for MAG240M, Arxiv, Pubmed, WikiGraph. and Wikikg90mv2, we will include 10,000 key-to-content and 10,000 content-to-key information retrieval tasks. This resulted in 935,000 samples for each training epoch and we trained the model for 3 epochs. The training conducted on 8 NVIDIAA 100 100AA00000AMAA0000
GPUs with DeepSpeed stage 2 (Rajbhandari et al., 2020) parallelism. The detailed training parameters are set the same for both two models and are listed in Table [2] We use Adam W optimizer with 3 (0.9, 0.95). We use cosine annealing learning rate scheduler, and the minimum learning rate is 10% of the initial learning rate. We restarted the learning rate 2 times on one-third and two-thirds of the training.
Table 12: Hlyper-paameee for pretraining.
Ir weight weight_decay batch_size dropout grad_elip gradient_ _accum aai_max_leengee optimizer
0.0001
0.1
8
0.0
0.5
128
Adam
F.4 ZERO-SHOT LEARNING
Setting. For the zero-shot learning, we select Cora-link, Cora-node, WikiCS, Products, ExplaGraphs, and SceneGraphs as evaluation datasets. For all datasets, we directly evaluate baselines and GOFA on the test set.
Baseline Details: We compare the performance of GOFA with two categories of baseline methods. The first category includes models that directly utilize large language models (LLMs). For this, we select Llama2-7B and Mistral-7B (Jiang et al. (023) as baselines. We input the content of all target nodes into these pre-trained models and concatenate the same prompt used in GOFA for evaluation. The second category consists of Graph LLM models that have zero-shot ability. We include OFA (Liu
al. 2023a). GraphGPT (Tang et al. 2023), UniGraph (He Hooi 2024). ZeroG (Li et al. 2024). and LLaGA (Chen et al. 2024) as baselines. For OFA, we extend the datasets by adding Products and ExplaGraphs and follow the original source code to train the model on Arxiv and FB15K237 for 30 epochs, using Llama2-7B default OFA configuration, and we report the test performance accordingly. For GraphGPT, we use the results reported in the LLaGA paper. For UniGraph, we use the results from the original paper. For ZeroG, we use the results in UniGraph paper. For LLaGA, we rerun the source code, adapting the settings of ways to align with our experimental setup.
the embedding model. All other settings remain consistent with the
Detail of GOFA. For the GOFA, we fine-tune the model from the pre-training checkpoint. In fine-tuning, will train the parameters of GNN and LoRA layers in the LLM decoder. To com-
27

---

Published as a conference paper at ICLR 2025
Table 13: Prompt examples of GOFA for each training dataset in Zero-shot learning.
Dataset
Prompt
MAG240M
This a citation network from microsoft academic graph platform. Nodes repre- sent academic papers and edges represent citation relationship. You are expert in computer science. You need to choose the correct paper category based on the paper content and its citation network. For example, the puper [NODEID] (<la- bel_ dedddescription... choose <label>:]. What is the most likely paper category for the target paper? Choose from the following: |<label>].
Pubmed- link
This is co-citation network from the Pubmed platform focusing on diabetes mellitus. Nodes represent academic papers and edges represent two papers that are co-cited by other papers. You are a diabetes mellitus expert tasked with determining whether two given papers [NODEIDI] ] and [NODEID2] are co-cited by another paper based on their content and network charucteristics. Evaluate the following criteria: assess whether the topics of the two pupers are similar, check if the shortest path distance between the two papers is small, and verify whether the papers have large number of common neighbors in the citation network. If the answer to most of these questions is Yes, choose Yes; if the answer to most of these questions is No, choose No.
PubMed- node
This is co-citation network from the Pubmed platform focusing on diabetes mellitus. Nodes represent academic papers and edges represent two papers that are co-cited by other papers. You are an expert on diabetes mellitus. You need to choose the correct paper category bused on the paper content and its co-citation network. For example, if the paper [NODEID] likely paper category for the target paper? Choose from the following: | |<label>).
deserppiissrr, choose <label>;]. Whut is the most
Wikikg90m
This is graph extracted from the entire Wikidata knowledge base. You are an expert in knowledge graph reasoning. You need to choose the correct relation type between two target entities based on their existing relations. For example, if two relations involve {<label description>, choose <label>:). What the relationship between two target entities? Choose from the following list: (<label>]."
rxiv_node
This is citation network from Arxiv platform focusing on the computer science area. Nodes represent academic papers and edges represent citation relationships. You are an expert in computer science. You need to choose the correct paper calegory based on the paper content and its citation network. For example, if the paper [NODEID] [ (<label description>, choose <label>;). What is the most likely paper category for the target paper? Choose from the following: (<label>).
Arxiv_lin _in_kink
This is citation network from Arxiv platform focussng the computer science area. Nodes represent academic papers and edges represent citation relationships. You are computer science expert tasked with determining whether two given papers [NODEIDI] and [NODEID2] are co-cited by another paper based on their content and network characteristics. Evaluate the following criteria: assess whether the topics of the two papers are similar, check if the shortest path distance between the two papers is small, and verify whether the papers have a large number of common neighbors in the citation network. If the answer to most of these questions is Yes, choose Yes; if the answer to most of these questions is No, choose No.
28

---

Published as a conference paper at ICLR 2025
Table 14: Prompt examples of GOFA for each evaluation dataset in Zero-shot learning.
Dataset
Prompt
Cora-node
This is co-citation network focusing on artificial intelligence, nodes represent aca- demic papers and edges represent two papers that are co-cited by other papers. You are an expert computer science. You need to choose the correct paper category based on the paper content and its co-citation network. For example, if the paper [NODEID] <<label_descriptione
choose <label>:). What is the most likely paper cutegory for the larget puper? Choose from the following: (<label>).
Cora-link
This is co-citation network focusing on artificial intelligence, nodes represent aca- demic papers, and edges represent two papers that are co-cited by other papers. You are a computer science expert tasked with determining whether two given papers are co-cited by another paper based on their content and network characteristics. Evaluate the following criteria: assess whether the topics of the two papers are similar, check if the shortest path distance between the two papers is small, and verify whether the pupers have a large number of common neighbors in the citation network. If the answer to most of these questions is Yes, choose Yes; if the answer to most of these questions is No, choose No.
WikiCS
This is Wikipedia graph focusing on computer science. Nodes represent Wikipedia terms and edges represent two terms that have hyperlinks. You are an expert in computer science. You need to choose the correct category of Wikipedia term based on the term content. For example, the term [NODEID] (<label description>, choose <label>:). What is the most like category for this Wikipedia entry? Choose from the following: [<label>).
Products
This is a co-purchase network from the Amazon platform. Nodes represent the products sold Amazon and edges represent two products that co-purchased logether. For example, if the product [NODEID] |<label description>, choose <label>:). What is the most like category for this product? Choose from the the following: /<label>).
FB15K237
This is a knowledge graph from the FreeBase. Nodes represent knowledge entities and edges represent relations between two entities. You an expert in knowledge graph reasoning. You need to choose the correct relation type between two target entities based their existing relations. For example, if two relations (<label description>, choose <label>:). What is the relationship between two target entities? Choose from the following list: (<label>)."
ExplaGraphs
This is graph constructed from commonsense logic. Nodes represent commonsense objects and edges represent the relation between two objects. You are logic expert tasked with analyzing the logical relationship between two arguments related to connected entities. Determine the arguments support counter each other based on their logical coherence. If there is no logical conftict between the two arguments and they in agreement, choose Support; if the arguments exhibit logical conflict or contrudiction, choose Counter.
SceneGraphs
This is scene graph generated from an image. Nodes represent object in the image and edges represent the relationship between two objects. <Question>
29

---

Published as a conference paper at ICLR 2025
Table 15: Hyper-parameter for zero-shot instruction tine-tuning.
Ir weight decay gradient accum Ilm max length
0.0001
0.1
64
256
prehensively evaluate the performance of GOFA, We separately fine-tune the GOFA different datasets. Specifically, we design two different settings. In the first setting, we fine-tune the model using the Arxiv and Pubmed datasets with both the node classification and link prediction tasks. In the second setting, we add mag240m and Wikikg90m additionally. We denote GOFA-T T and GOFA-F, respectively. For GOFA-T, we sample 40000, 80000, 10000, 10000 for Arxiv Jink, Pubmed_ling Jink, and Pubmed_ _oniddde_dede___e respectively. For GOFA-F, we sample 10000, 10000, 40000, 50000, 10000, 10000 for MAG240M, Wikikg90m. Arxiv link, Arxiv _ndde, Pubmed node, and Pubmed link, respectively. For all evaluation and pre- pre-training datasets, we design multiple prompt templates with instructions to let the model select the correct label from the provided label list. For each label in each dataset, we use the GPT-4 to generate short description for the label. The detailed prompt examples for all datasets are shown Table and and Table For all MAG240M, Wikikg90m, and Arxiv, since it is hard to include all ways in the prompt, we randomly sampled 10 ways during the training for each sample. For each pre-training dataset, we randomly sample a fixed number of training samples in random way. The detailed parameters for parameters not listed in the table are the same as the pre-training setting. For all training versions, we directly evaluate the model on the test set of all evaluation datasets. We evaluate the model on the test set. For datasets with less than 15000 test samples, we evaluate on the whole set. Otherwise, we only randomly select 15000 samples for evaluation, due the time constraint. For evaluation, we will match the text output generated by the GOFA with the ground true label to compute the accuracy of the classification task. For the regression task, we will extract the number from the output text and compute the metric with the correct value.
are listed in Table [5] All
F.5 SUPERVISED-LEARNING
Table 16: Hyper-parameters for supervised fine-tuning.
Ir
weight_decap grad_clip gradient_accum jin_max_lengll
0.0001
0.1
0.5
32
256
Setting. For the supervised-leanine setting, we select Cora (node/link), PubMed (node/link), Arxiv, WikiCS, WNISRR, FB 15K377, and Products datasets for the evaluation. For all datasets, we utilize the default split described in Appendix To ensure a fair comparison, we employ subgraph sampling for GOFA and all baseline methods. For all datasets, the sampling hop is 3 and the maximum nodes per hop are 5.
Detail of baselines. For the traditional GNN methods, we include GCN (Kipf & Welling 2017) and GAT (Veličković et al., 2018). To ensure a fair comparison, we use Llama2 7B to convert raw texts in all datasets to sentence embedding and use this as the model's 's input node/edge features. We re-implement both methods in order to adapt the original method with subgraph input. Specifically, for but node/link-level tasks, we will add labeling trick (Zhang al., (02T) to the target nodes at the beginning. After message passing, we will use the summation pooling on all target nodes and use the result embedding for the prediction. For traditional GNN methods, we train and evaluate each dataset independently. For all datasets, we search the number of layers and dropout parameters. For each parameter set, we repeat the experiment times select the parameter set with the best validation performance, and report the performance on the test set. For constrastive learning methods, we include DGI (Veličković et al. 2018 and BGRL (Thakoor et al., 2021). We directly report results from UniGraph (He& & Hooi 2024 For the graph foundation model, we include OFA (Liu) et al. 2023a) and UniGraph (He & Hooi 2024) as the baseline. The OFA is simultaneously trained and evaluated source and train the model on Cora (node/link), PubMed (node/link), Arxiv, WikiCS, WN18RR, and FB 1.1552377 dataset using the Llama2-7b as base LLM model. Similarly, for OFA, use the
all datasets. To ensure fair comparison, we get their code from the original
30

---

Published as a conference paper at ICLR 2025
Table 17: Detailed prompt of GOFA for each dataset in supervised learning.
Dataset
Prompt
Cora-node
This a co-citation network focusing on artificial intelligence, nodes represent academic papers and edges represent two papers are co-cited by other papers. What is the most likely paper category for the target paper? Please directly answer the category.
Cora-link
This a co-citation network focusing on artificial intelligence, nodes represent academic papers and edges represent two papers are co-cited by other papers. the two target papers co-cited or not? Please only answer yes no.
PubMed-node
This is co-citation network from Pubmed platform focusing dia- betes mellitus. Nodes represent academic papers and edges represent two papers are co-cited by other papers. What the most likely paper category for the target paper? Please directly answer the category.
PubMed-link
This is co-citation network from Pubmed platform focusing dia- betes mellitus. Nodes represent academic papers and edges represent two papers co-cited by other papers. Is the two target papers co-cited or not? Please only answer yes or no.
Arxiv
This is a citation network from arxiv platform focusing on the computer science area. Nodes represent academic papers and edges represent citation relationships. What is the most likely paper category for the target Arxiv paper? please directly answer the category.
WikiCS
This is Wikipedia resent Wikipedia terms and edges represent two terms have hyperlink. What is the most likely category for this Wikipedia term? Please directly answer the category.
focusing on computer science. Nodes rep-
WNISRR
This a knowledge graph from WordNet. Nodes represent an English word and edges represent the relationship between two words. What is the relationship between two target words? Please directly answer the relationship.
FB15K237
This knowledge graph from freebase. Nodes represent knowledge entities and edges represent relations between two entities. What is the relationship between two target entities? Please directly answer the relationship.
Products
This a co-purchase network from the Amazon platform. Nodes repre- sent the products sold on Amazon and edges represent two products are co-purchased together. What the most like category for this product? Please directly answer the category.
same subgraph sampling parameters as all other methods. For other parameters, we use the default parameter provided their code. We only run the model one time and report the final performance. For UniGraph, we directly report results from their original paper.
Detail of GOFA. For the GOFA, we fine-tune the model from the pre-training checkpoint. In fine- tuning, we will train the parameters of GNN and LoRA layers in the LLM decoder. We simultancously fine-tune the model on the train set of Cora-node, Cora-link, PubMed-node WikiCS, WN18RR, FB FB15K237, and Products. For each dataset, we will randomly sample fixed number of training samples for each epoch with random sampling. The sample numbers for each dataset is 3000, 40000, 3000, 80000, 105000, 12000, 60000. 120000, and 38000, respectively, We fine-tune the model for epochs. The detailed parameters for fine-tuning are listed in Table [6] For each dataset, create prompt for the LLM decoder to generate the desired answer. In supervised setting, we ask the LLM model directly to generate the correct answer, instead of doing the selection from the given list. The detailed prompt for each dataset is listed in Table 17 For evaluation, will
PubMed-link, Arxiv,
31

---

Published as a conference paper at ICLR 2025
match the text output generated by the GOFA with the ground true label to compute the accuracy of the classification task. We evaluate the model on the test set. For datasets with less than 15000 test samples, evaluate the whole set. Otherwise, we only randomly select 15000 samples for evaluation, due to the time constraint.
32


Reviews:
[{"review_id": "SQ2Iv0z6gk", "reviewer": "Reviewer_5txS", "summary": "The paper proposes a new joint graph and language model that can perform different graph and language tasks. The model is a combination of an LLM compressor, GNN layers and an LLM decoder. The model can be instruction-tuned to solve many graph-structured tasks, and at the same time, it remains effective on language tasks. The model is also more efficient than plain LLMs solving the same tasks, since it separates the input into different nodes and edges.", "questions": "1. Do the authors use a pretrained llm decoder or only a pretrained compressor?\n\n2. From eq. 3, since GNN is not indexed by k, it appears to be shared across all the K tokens. Is it correct? \n\n3. In the graph task, the authors \"set the default target nodes to all nodes in the graph\". How the authors combine generations from each node intto a single answer/prediction?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses an important and challenging problem of building generic and effective models for different modalities.\n2. The proposed model, while being a combination of existing methods, is logical and original. Especially, applying the GNN layers on top of memory tokens seems a very good idea to exchange information in an efficient and flexible way.\n3. The experiments are sufficiently large, diverse and challenging.\n4. The results are overall strong and promising.", "weaknesses": "1. The model is designed more for the tasks where nodes/edges are rich in text. It would be interesting and more impactful to include tasks (in the pretraining and/or eval) that are more numeric, e.g. molecule prediction/generation such as ZINC or from open graph benchmark. While the SOTA specialized GNNs are probably going to work better in those cases, it would be interesting to see the gap.\n\n2. Related to above, one potential issue of this approach is that representing node/edge features can be extremely inefficient when the features are high-dimensional (e.g. a node is an image or some other array data).\n\n3. What's the SOTA GNN results in Table 2? It's not a direct comparison, but useful to know for reference.\n\n4. The examples in Fig 2 do not seem to require much knowledge of the graph, i.e. a plain LLM should be able to solve these tasks well. The example does not show what are the nodes and what is the meaning of edges and why those are important for the tasks. It seems like all the information exists within the node.\n\n5. In Table 4, it's unclear if LLM-N and GOFA-F trained on the same data? Also does LLM-N have the complexity of O((V k)^2) as discussed in L298? \n\nMinor:\n\n- In the \"LLMs as predictors\" some representative papers like GraphLLM are not discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2XL0mCcnZx", "reviewer": "Reviewer_sGjN", "summary": "This paper extends language modeling to the graph domain and proposes a new generative graph language model, GOFA. The GOFA architecture interleaves frozen LLM layers with trained GNN layers, adding structure-aware processing layers to the LLM while keeping the original LLM intact and providing processing paths for regular language modeling through gating mechanisms. Input graphs are transformed into Text-Attributed Graphs (TAGs), and a next-token-prediction framework is adopted over various graph datasets in natural language, serving as a general self-supervised pretraining objective over graph-structured data. GOFA shows strong performance on benchmark datasets versus previous works building foundation models for graph-structured data using LLMs, and provides a flexible framework for integrating LLMs with GNNs.", "questions": "1. More clarity on the degree to which the GOFA saw examples of node classification and link prediction during instruction finetuning would improve clarity on whether the results can be called a zero-shot setting, or whether GOFA is generalizing seen tasks to new datasets unseen during pretraining and instruction finetuning.\n2. More benchmarking results across graph datasets as well as including traditional GNN baselines would greatly strengthen the empirical results of GOFA, and support its claim as a graph foundation model.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The authors offer detailed discussion about the desired attributes of a graph foundation model, including the ability to generalize to new tasks and data domains without additional finetuning of the foundation model, i.e. zero-shot settings.\n- The authors also give detailed discussion on previous methods for incorporating GNNs into LLM-based architectures (LLMs as predictors and LLMs as an enhancer), and discuss the drawbacks of previous approaches in building general graph foundation models. The graph foundation model should account for the unique structural information of input graphs and generate graph representations depending on the input prompt. It should preserve the LLM’s prompt following capabilities while also learning graph structure and semantic information jointly.\n- The authors propose an architecture which designs a graph language encoder that encodes input TAGs into a fixed number of tokens while accounting for both the semantic information of node and edge text as well as the structure of the graph. Their model controls computational complexity by only doing message-passing on each index of the memory tokens, and feeding a fixed number of vectors to the LLM decoder for the downstream graph task.", "weaknesses": "1. It is unclear whether the results in Table 2 of the paper represent a true zero-shot setting, since the authors mention in Section 5.2 and Appendix F.4 that the model is instruction-finetuned on node classification and link prediction examples. The datasets in Table 2 are unseen, however samples of the same task on other datasets has been seen by the model during instruction finetuning, making it seem like this is a generalization task to unseen datasets rather than a zero-shot setting where the model has never seen node classification samples before.\n2. Additionally, given that the authors emphasize GOFA as a graph foundation model, more comprehensive benchmarking across more graph datasets would better support this claim. Examples include Amazon Products, molecular graph datasets such as OGBN-molhiv, or protein-protein interaction datasets such as OGBN-proteins. Further benchmarks would better support the claim that GOFA supports diverse downstream tasks on graph-structured data across a variety of graph datasets.\n3. The authors do not benchmark against any traditional GNN baselines. The main comparison remains other LLM and LLM-based graph models, however a graph foundation model should compare against traditional GNN baseline models to indicate performance relative to much lower capacity GNN models on the same tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YJCdASvNUe", "reviewer": "Reviewer_qxFo", "summary": "The authors introduce GOFA, a general-purpose Graph-Language Model, which proposes an efficient but somewhat intricate approach to encoding a graph into a collection of vectors suitable for input into a large language model (LLM). The model is trained on a large dataset collection using substantial computational resources (8 A100 80GB GPUs).", "questions": "1. Could the authors clarify how the NOG is represented in practice? Specifically, is a special token used for this purpose?\n\n2. Could the authors provide more details about the graph samples used? How many nodes on average are processed by GOFA on each forward pass and how capable is it to provide accurate answers as the graph size increases?\n\n3. How many tokens are reserved for the graph input in the final question-answering LLM (purple tokens in Fig. 3)? Is it a fixed number or does it depend on the graph's size?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "GOFA is trained on multiple datasets and it appears to be indeed a general-purpose Large Graph Language Model.\n\nThe paper is well-structured, with a comprehensive presentation of both quantitative and qualitative results across the main text and appendix.", "weaknesses": "1. **Evaluation of Graph Structure Utilization.** In Figure 1 and Section 3.4, the authors outline four pre-training tasks: (a) sentence completion, (b) question answering, (c) structural understanding, and (d) information retrieval. However, apart from the structural understanding task, it is somewhat unclear how these tasks benefit from graph information. For instance, tasks like sentence completion and question answering might be achievable by directly providing the raw text input to a generic LLM, possibly yielding similar outcomes. Similarly, in Figure 2, the sentence completion and question-answering tasks are primarily language-oriented and may not require graph-based augmentation. Even in Figure 3, the abstract completion task appears somewhat independent of the graph structure. \n\\\nIt would be helpful if the authors could include a comparison where an LLM alone performs these tasks, highlighting any areas where it falls short. Alternatively, if GOFA matches or surpasses the LLM’s performance while adding distinct capabilities, this would serve as a compelling justification for GOFA’s design. Currently, the experimental section primarily focuses on comparing GOFA to LLMs in the structural understanding task (Q3), where graph augmentation evidently improves performance. Additionally, introducing more graph-relevant questions, as seen in recent works (e.g., Fatemi et al.), could further substantiate the advantages of incorporating graph structure in the evaluations.\n\n2. **Comparison with Related Work.** The proposed approach bears similarities to recent models such as GraphToken (https://arxiv.org/abs/2402.05862) and GraphLLM (https://arxiv.org/abs/2310.05845). A detailed comparison, particularly of the advantages and limitations of GOFA relative to these methods, would strengthen the discussion. Including an experimental comparison could also help clarify GOFA’s unique contributions and situate it within the broader context of graph-enhanced LLM research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9dM4Fnm0qS", "reviewer": "Reviewer_5YVL", "summary": "This paper introduces three key properties that an effective Graph Foundation Model (**GFM**) should possess: self-supervised pretraining, fluidity, and graph understanding. To meet these goals, the authors propose Generative One-for-All (**GOFA**) , which integrates a Language Model Decoder with a Graph Language Encoder, which strategically interleaves trainable Graph Neural Network (GNN) layers into frozen Large Language Model (LLM) layers.\n\nThe GOFA model is pre-trained on a diverse set of datasets, including MAG240M, Pubmed and Arxiv, Wikikg90mv2, WikiGraph, and Ultrachat200k. Notably, the WikiGraph dataset is created specifically for this work and it is derived from the existing WikiText dataset.\n\nThe pretraining results validate the effectiveness of incorporating GNN layers into LLMs, while instruction-tuning experiments further demonstrate GOFA's strong zero-shot generalization capabilities.", "questions": "I have two questions for the authors:\n\n1. Dataset quality: I would like to inquire whether the authors thoroughly validated the content generated in their constructed WikiGraph dataset. Given that the dataset is created by prompting large language models (LLMs), there is a known issue with the potential for LLMs to produce hallucinated or inaccurate outputs. Could the authors elaborate on the measures they took to ensure the accuracy and quality of the dataset? How did they mitigate the risks of including erroneous or noisy data in the final dataset?\n\n2. Could the authors provide details about the pre-training time when using 8 NVIDIA A100-SXM4-80GB GPUs?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well-structured and easy to follow. Its key strengths can be summarized as follows:\n\n1. The authors identify three crucial properties of an effective graph foundation model: self-supervised pretraining, fluidity, and graph understanding. I strongly agree with these as important considerations.\n\n2. The paper introduces GOFA, which operates under a graph completion framework to enable large-scale self-supervised learning. Furthermore, GOFA demonstrates impressive performance in both zero-shot and fine-tuning scenarios when compared to LLM baselines.\n\n3. One potential concern in developing a foundation model is whether it might negatively impact the capabilities of existing LLMs. However, the results in Table 1 suggest that GOFA's training does not degrade the performance of the original pre-trained LLMs. This is a critical design criterion when building a graph foundation model.", "weaknesses": "1. The authors suggest that integrating GNN layers into LLMs offers a theoretical advantage over standard LLMs when it comes to modeling graph structures. However, I did not find any accompanying mathematical formulations or equations to substantiate this claim in the appendix.\n\n2. GOFA performs significantly worse than the LLM baselines on the SceneGraphs dataset, despite the authors attributing this to the instruction-tuning data containing information-dense texts. It's concerning that a domain-specific foundation model, despite careful design, does not demonstrate competitive performance.\n\n3. As the authors note, using a frozen LLM compressor while tuning only the GNN layers appears questionable. This approach might impede optimal alignment between information from various modalities, potentially limiting the model's overall effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)