Title: GFT: Graph Foundation Model with Transferable Tree Vocabulary
Venue: NeurIPS 2024
Accepted: yes
Keywords: Graph Foundation Model

Transferability

Computation Tree

Graph Neural Network
Abstract: Inspired by the success of foundation models in applications such as ChatGPT, as graph data has been ubiquitous, one can envision the far-reaching impacts that can be brought by Graph Foundation Models (GFMs) with broader applications in the areas such as scientific research, social network analysis, drug discovery, and e-commerce. Despite the significant progress of pre-trained graph neural networks, there haven’t been GFMs that can achieve desired performance on various graph-learning-related tasks. Building GFMs may rely on a vocabulary that encodes transferable patterns shared among different tasks and domains. Unlike image and text, defining such transferable patterns for graphs remains an open question. In this paper, we aim to bridge this gap by rethinking the transferable patterns on graphs as computation trees -- i.e., tree structures derived from the message-passing process. Based on this insight, we propose a cross-task, cross-domain graph foundation model named GFT, short for Graph Foundation model with transferable Tree vocabulary. By treating computation trees as tokens within the transferable vocabulary, GFT improves model generalization and reduces the risk of negative transfer. The theoretical analyses and extensive experimental studies have demonstrated the transferability of computation trees and shown the effectiveness of GFT across diverse tasks and domains in graph learning. The open source code and data are available at https://github.com/Zehong-Wang/GFT.

GFT: Graph Foundation Model with Transferable Tree Vocabulary

Abstract

Inspired by the success of foundation models in applications such as ChatGPT, as graph data has been ubiquitous, one can envision the far-reaching impacts that can be brought by Graph Foundation Models (GFMs) with broader applications in the areas such as scientific research, social network analysis, drug discovery, and e-commerce. Despite the significant progress of pre-trained graph neural networks, there haven't been GFMs that can achieve desired performance on various graph-learning-related tasks. Building GFMs may rely on a vocabulary that encodes transferable patterns shared among different tasks and domains. Unlike image and text, defining such transferable patterns for graphs remains an open question. In this paper, we aim to bridge this gap by rethinking the transferable patterns on graphs as computation trees - i.e., tree structures derived from the message-passing process. Based on this insight, we propose a cross-task, crossdomain graph foundation model named GFT, short for Graph Foundation model with transferable Tree vocabulary. By treating computation trees as tokens within the transferable vocabulary, GFT improves model generalization and reduces the risk of negative transfer. The theoretical analyses and extensive experimental studies have demonstrated the transferability of computation trees and shown the effectiveness of GFT across diverse tasks and domains in graph learning. The open source code and data are available at https://github.com/Zehong-Wang/GFT .

1 Introduction

Foundation models such as Large Language Models (LLMs) and Large Vision Models (LVMs) keep reshaping our view of the world [7, 100, 51, 112, 50]. These models are designed to be generalpurpose, adaptable across various tasks and domains through fine-tuning or prompting, such as GPT-4 [1] in Natural Language Processing (NLP) and Sora [46] in Computer Vision (CV). Research attributes the success of foundation models to the uniformity of tasks and a general vocabulary that defines basic transferable patterns across tasks [98, 76, 112, 3, 50]. For example, LLMs [1, 112] treat language tasks as question-answering or next-word prediction and deconstruct sentences using a word vocabulary. Similarly, LVMs [100, 98, 3] reformulate image tasks as image question-answering, converting images into discrete tokens using a vision vocabulary. Inspired by the success of LLMs

Yanfang Ye ∗

and LVMs, as graph-structured data (e.g., citation networks, social networks, computer networks, molecular structures, and recommender systems) have become ubiquitous, one can envision the farreaching real-world impacts that can be brought by pre-trained Graph Foundation Models (GFMs).

Although there has been significant progress of pre-trained Graph Neural Networks (GNNs), there haven't been GFMs that can achieve desired performance on a wide range of graph-learning-related tasks. Unlike CV and NLP, as graphs represent complex, non-Euclidean relationships among entities [92, 48, 104, 58, 107], a grand challenge of building GFMs lies in identifying transferable patterns across graphs [50, 93, 25]. There have been extensive studies aiming to tackle this challenges, which can mainly be categorized into two groups: (1) Utilizing graphon theory: Ruiz et al. [62] provide theoretical evidence of transferability between two graphs generated from the same graphon. Cao et al. [8] further extend these findings by both empirically and theoretically analyzing graph transferability in pre-training and fine-tuning scenarios. Despite these theoretical guarantees, the stringent assumptions of graphon theory often prove difficult to satisfy in real-world, cross-domain datasets [42], thus limiting its applicability in defining transferable graph vocabularies. (2) Exploring graph transferability using subgraph structures [114, 59, 50]: Zhu et al. [114] demonstrate that the transferability between graphs is linked to the ego-graph patterns of individual nodes and establish a stability bound using the graph Laplacian. They suggest that localized subgraphs could serve as transferable patterns within graph vocabularies. Building on this finding, Sun et al. [68] develop a GFMby reformulating graph tasks as subgraph classification, enabling a single model to be applied to diverse tasks. Huang et al. [30], Liu et al. [45] expand GFMs to cross-domain scenarios by unifying the node feature space across different graphs through LLMs [60, 76]. Despite these successes, the process of explicit subgraph extraction remains time and memory intensive [30]. More importantly, numerous studies such as [20, 10, 53, 103] show that message-passing GNNs [40, 24, 21] fail to detect critical substructures or motifs within subgraphs, reducing the feasibility of using subgraphs to define graph vocabularies.

How to identify a vocabulary that can encode transferable patterns shared among different tasks and domains for the construction of GFMs? In this paper, we aim to address the limitations of existing works by answering this question. Specifically, based on message-passing mechanism of GNNs, we have observed that the learned embeddings of each node can be essentially captured in the form of its computation tree. Based on this insight, we bridge the research gap by rethinking the transferable patterns on graphs as computation trees - i.e., subtree structures derived from the message-passing process. Using computation tree as a transferable pattern across graphs will bring three primary advantages: (1) Efficiency : As the extraction and encoding of computation trees are integrated within a single message-passing GNN process [20], it eliminates the need for the explicit subgraph extraction for GFMs [30, 45]. (2) Expressiveness : Since computation trees are capable of capturing localized patterns [52], it's able to represent a graph as a multiset of computation trees [23]. (3) Learnability : As the information of computation trees is completely captured by message-passing GNNs, it can tackle the issue that certain motifs within subgraphs remain elusive. We theoretically investigate the transferability of computation trees and empirically demonstrate a strong correlation between computation tree similarity and transfer learning performance across various graphs.

Based on the key idea above, by treating computation trees as graph vocabulary tokens, we develop a cross-task, cross-domain graph foundation model - namely GFT - short for Graph Foundation model with transferable Tree vocabulary. GFT consists of pre-training and fine-tuning phases, enabling it to handle datasets across different tasks and domains effectively. During pre-training, we introduce a computation tree reconstruction task to acquire generalized knowledge from cross-domain graphs. We obtain a discrete tree vocabulary of prototypical tree tokens by quantizing the embedding space of computation trees, which theoretically improves model generalization. In the fine-tuning phase, we utilize this learned tree vocabulary to unify various graph-related tasks into computation tree classification, thereby preventing negative transfer [89, 87]. Extensive experimental results demonstrate the effectiveness of GFT in graph learning on cross-task and cross-domain datasets.

2 Rethinking Transferable Patterns on Graphs

2.1 Transferability on GNNs

Transferability refers to a model's capability to extract patterns from source tasks and apply this knowledge to enhance performance on related target tasks [5, 33, 90]. Understanding transferable

patterns is essential for developing graph foundation models. Early research focuses on analyzing transferability through the perspectives of graph spectrum [41, 42] and subgraphs/substructures [114], defining transferability as model invariance to minor permutations on the graph. A more recent study [50] investigates the transferable vocabulary in graphs by identifying key substructures relevant to various tasks. For instance, they find that triadic closure, homophily, and heterophily are vital for node classification; local and global structural proximities are crucial for link prediction; and certain motifs [103], such as triangles, k -cliques, and stars, serve as fundamental components for graph classification. Another line of research [62, 8, 69] incorporates graphon theory to provide a theoretical basis for transferability. Specifically, Ruiz et al. [62] establish a bound on the embeddings of two graphs sampled from the same graphon. Cao et al. [8] expand this to include pre-training and fine-tuning scenarios, assessing the distance between graphs based on their alignment within the graphon space. However, the stringent assumptions of graphon theory limit its practical application in the design of graph foundation models.

We identify two primary limitations in analyzing transferable patterns on graphs: (1) While certain domains [110, 87, 43, 108] or tasks [114, 111, 19] exhibit transferable patterns, the challenge of identifying universally transferable substructures is difficult. (2) More critically, basic messagepassing GNNs, constrained by the 1-WL test [94, 52], fail to recognize certain subgraphs (or motifs) [20, 10, 103], such as stars, conjoint cycles, and k -cliques, as well as heterophily patterns [113]. This limitation in recognizing substructures impedes using subgraphs as transferable tokens in graph vocabulary [68, 30, 45]. More related works are elaborated in Appendix A.

2.2 Computation Tree as Transferable Pattern

In this paper, we rethink the transferable pattern in graphs as the computation tree - a specialized subtree pattern that emerges from unfolding the message-passing process [12]. This pattern is demonstrably effective at capturing critical localized information within the graph [20, 52, 94, 12]. Treating computation trees as tokens within a graph vocabulary offers two distinct advantages: (1) computation trees preserve the essential structural information of the graph, which is learnable through message-passing GNNs, and (2) the computation tree structure occurs across various graph-based tasks. These tasks can be unified as computation tree classification by integrating a virtual node, as shown in Figure 1.

Before diving into transferability analysis, we first establish the necessary notations. Consider a graph G = ( V , E ) composed of node set V and edge set E . Each node v ∈ V is associated with a feature vector x v ∈ R d and a computation tree T v with L layers. A GNN encoder ϕ processes these computation trees as inputs, producing embeddings for root nodes z = ϕ ( T v ) ∈ R d ′ .

Node

Link

Graph

Task-relevant Nodes

Definition 2.1 (Computation Trees [12]) . Given a graph G = ( V , E ) , define T 1 v = v and T L v as the L -layer computation tree. This tree is constructed by recursively integrating the subtrees of neighborhoods. The multiset of L -layer computation trees on graph G is denoted by T L G := {T L v } v ∈V .

Figure 1 demonstrates the construction of computation trees across various graph tasks, including node-, link-, and graph-level tasks. These trees capture essential localized subtree patterns within the graphs [55, 64, 12]. If the L -layer computation trees for two nodes are similar, it indicates that these nodes share similar neighborhoods, suggesting they represent analogous phenomena [42]. Thus, it is rational to assess transferability of computation trees by examining the stability of GNNs in producing analogous embeddings for similar trees [62, 42].

Theorem 2.2 (Transferability of Computation Tree ) . Given two L -layer computation trees T v 1 , T v 2 derived from the graph G and a GNN encoder ϕ , the Euclidean distance between the tree embeddings ∆ ≜ ∥ ϕ ( T v 1 ) -ϕ ( T v 2 ) ∥ 2 is bounded as follows:

where ∆ L -1 v 1 ,v 2 ,j represents the distance between the ( L -1) -layer subtrees of the j -th children of nodes v 1 and v 2 , C 1 , C 2 are constants, and B x denote bounded norm of x . The variable d l indicates the number of children in the l -layer subtrees, and D l = d l d l -1 ...d 1 .

Remark 2.3 . Theorem 2.2 derives a recursive bound for computation tree similarity. In particular, the distance between two computation trees is closely correlated to the similarity of their subtrees, where higher subtree similarity results in a closer distance. This suggests that computation trees with similar structures are likely to have similar embeddings, which enhances their transferability [114, 33, 62]. This aligns with our empirical observations that higher computation tree similarity between two graphs leads to improved transferability.

Supportive Observations - Synthetic Graphs. Figure 3 shows that high computation tree similarity between graphs correlates with improved transfer learning performance on synthetic graphs (Figure 2). Specifically, we construct three distinct graphs: G 1 and G 2 share similar motifs but differ in computation tree distributions, while G 1 and G 3 exhibit dissimilar motifs but similar computation tree distributions. We employ the WL subtree kernel [64] and the graphlet sampling kernel [57] to assess tree and motif similarity, respectively, and utilize the inverse of the Central Moment Discrepancy [102] to measure transferability. Further details on experimental settings and additional results are available in Appendix E.1. Our findings indicate that transferability is strongly associated with computation tree similarity rather than motif similarity, regardless of the scale of graphs (# blocks).

𝒢1

𝒢2

𝒢3

20

40

60

80

100

# Blocks

6

8

10

Transferability

Source

Graph 3

Graph 2

20

40

60

80

100

# Blocks

0.7

0.8

0.9

Source

Graph 3

Graph 2

20

40

60

80

100

# Blocks

0.6

0.8

1.0

Source

Graph 3

Graph 2

Figure 2: Synthetic graphs composed of two basic blocks. More blocks can scale up the graph sizes.

Supportive Observations - Real-world Graphs. Table 1 validates the correlation between computation tree similarity and transferability in real-world graphs, including homophily Airport networks [61] and heterophily WebKB networks [56]. We evaluate transferability based on transfer learning performance in node classification tasks. Detailed experimental settings and additional results are available in Appendix E.2. Our findings in real-world graphs corroborate those in synthetic graphs: higher computation tree similarity enhances transferability, while the impact of motifs is marginal, no matter using original node features (Table 1) or randomized node features (Table 9).

3 GFT: Graph Foundation Model with Transferable Tree Vocabulary

We develop GFT, a cross-domain and cross-task graph foundation model that leverages computation trees as transferable patterns within graph vocabulary. As illustrated in Figure 4, GFT undergoes

Computation

Tree

Graph

Database

(a) Pre-training with Tree Reconstruction

(b) Fine-tuning with Tree Classification

Unseen

Graph

Encoder

𝜙

Unify Graph Tasks as

Tree Classification

Prototype Classifier

Linear Classifier

Memory Bank

Query

Tree Reconstruction Eq. (3) &

Orthogonal Regularizer Eq. (5)

Tree Vocabulary

𝜙

ℒ

!"#

𝛿

ℒ

$"%&

𝛿

ℒ

&'('

𝛿

ℒ

')&*'

Assign

Query

Single Token

Fixed

Tree Vocabulary

Freeze

Assign

pre-training through a computation tree reconstruction task to acquire general knowledge from a crossdomain graph database. Subsequently, GFT quantizes the embedding space of computation trees to form a discrete tree vocabulary, encapsulating fundamental, transferable computation tree patterns for diverse tasks. In the fine-tuning phase, GFT utilizes this tree vocabulary to unify graph-related tasks (including node-, link-, and graph-level tasks) as computation tree classification, adapting the general knowledge to specific target tasks.

3.1 Pre-training with Computation Tree Reconstruction

The pre-training stage focuses on learning general computation tree patterns on graphs, facing two primary challenges: (i) obtaining transferable patterns, and (ii) comprehensively capturing computation tree knowledge. For the first challenge, we learn a discrete tree vocabulary by quantizing the embedding space of computation trees [77]. For the second challenge, we introduce a computation tree reconstruction task that considers multiple aspects of computation trees.

Learning Tree Vocabulary. The idea of learning a discrete computation tree vocabulary originates from the principles of sparse distributed memory in cognitive science [37, 38], which stores and retrieves memory in a distributed manner. By adopting these principles, the tree vocabulary maintains a set of tokens that are reusable and adaptable across various tasks, improving model transferability.

We adopt the Vector Quantization (VQ) [77] to develop the tree vocabulary. Given a graph database 2 D = {G i } n i =1 , we randomly extract a set of computation trees T = {T i } i m =1 and employ a GNN encoder ϕ to generate the tree embeddings Z = { z i } i m =1 . We define the computation tree vocabulary as a set of learnable tokens C = { c 1 , ..., c K } . The tree embedding space is quantized by assigning each embedding to the nearest token, resulting in quantized tree embeddings q i = c j , where j = arg min j ∥ z i -c j ∥ 2 . We optimize this projection by back-propagating the reconstruction error to the tree vocabulary C and applying a straight-through gradient estimator [6] to the encoder ϕ . In particular, we jointly optimize vocabulary loss and commitment loss [77], along with tree reconstruction loss (discussed later), where the former updates the token vectors c using the fixed quantization q , and the latter ensures alignment between the tokens in the vocabulary and the quantized tree embeddings, serving as a regularizer. The pre-training objective is thus defined as:

where sg [ · ] denotes the stop-gradient operator and β 1 is the weight.

Computation Tree Reconstruction. We introduce a computation tree reconstruction task designed to enable a deep understanding of the structural and semantical attributes of computation trees [36]. We use the tree tokens to reconstruct the original computation tree, retaining general knowledge while discarding irrelevant details. Specifically, we develop three reconstruction tasks: (i) reconstructing the features of the root node L feat , (ii) reconstructing the connectivity among nodes in the computation trees L topo , and (iii) reconstructing the overall semantics of the computation trees L sem :

where ˆ z i = ˆ ϕ ( T i ) represents the semantics of the original computation trees, and ˆ ϕ is updated through a moving average of the tree encoder ϕ . The quantized tree embedding q is projected via different decoders defined by MLP, ˆ q j = δ j ( q ) , γ is the scaling factor, and E and ˆ E represent sets of existing and non-existing connections in computation trees, respectively. e ij denotes the edge embedding between nodes i and j . By jointly optimizing these tasks, we establish a comprehensive reconstruction objective:

where β i indicates the weights of respective losses. The philosophies under these loss functions separately correspond to existing works [39, 26, 74, 86]. For example, Kipf and Welling [39] reconstruct the graph structure, aligning to the philosophy of L topo , Hou et al. [26] reconstruct node feature that is similar to L feat , and Thakoor et al. [74], Wang et al. [86] employ contrastive learning to maximize the alignment between two views, aligning to L sem . Unlike existing methods that typically focus on reconstructing a single aspect of computation trees, GFT integrates multiple facets [85] to learn a general and transferable tree vocabulary.

Enhancing the Quality of Tree Vocabulary. The effectiveness of GFT is correlated to the quality of the tree vocabulary, which should be both comprehensive and expressive. A comprehensive vocabulary is inclusive enough to accommodate new patterns, while an expressive vocabulary ensures that different tree tokens do not overlap in representation [50]. To enhance comprehensiveness, we augment the computation trees during pre-training, increasing the variety of observed computation trees through node feature augmentation and structural augmentation, as described by [115]. To improve expressiveness, we regularize the tree vocabulary space by intentionally increasing the distance between distinct tokens [65]. Specifically, we introduce an orthogonal regularizer designed to maintain tree tokens orthogonal to each other, effectively expanding the tree token space:

where c i is tree token, I K is the identity matrix for K dimensions, and ∥ · ∥ F denotes the Frobenius norm. The orthogonal loss L ortho is integrated with Equation 2. More analysis is in Appendix C.2.

3.2 Fine-tuning with Computation Tree Classification

The pre-training stage encodes general knowledge into the tree vocabulary, while the fine-tuning phase adapts this knowledge to specific tasks. This adaptation is challenging because identical patterns can have different meanings across domains and tasks [8]. For example, a triangular structure indicates stable relationships in social networks (node classification) but denotes unstable chemical properties in molecular networks (graph classification). To this end, we propose computation tree classification that utilizes the tree vocabulary to unify graph tasks as the tree-level task, ensuring the adaptation is applicable across diverse tasks and domains.

Reformulate Graph Tasks as Computation Tree Classification. Graph-related tasks can be represented by task-specific computation trees, as illustrated in Figure 1. Specifically, for node classification, the task-specific computation tree, denoted as T node = T i , is derived directly from the node itself, resulting in the embedding z = ϕ ( T i ) . For link prediction, the computation tree, T link = Combine ( T s , T t ) , merges the computation trees of two nodes of the edge, with the embedding

z = mean ( ϕ ( T s ) , ϕ ( T t )) . For graph classification, the task-specific computation tree T graph = Combine ( {T v } v ∈V ) integrates the computation trees of all nodes within the graph, and computes the embedding as z = mean ( { ϕ ( T v ) } v ∈V ) . Subsequently, the embeddings of these task-specific trees are used to query the tree vocabulary and then make predictions, adapting the general knowledge encoded in the vocabulary to various tasks and domains.

Prototype Classifier. The prototype classifier f proto constructs class prototypes using tokens from the tree vocabulary. Given a set of task-specific computation trees { ( T i , y i ) } n i =1 with | C | classes, we employ the pre-trained GNN encoder ϕ to generate tree embeddings Z = { z i } n i =1 . These embeddings are then used to query the tree vocabulary and produce quantized embeddings Q = { q i } n i =1 . Subsequently, we construct a class-wise memory bank S = { S 1 , ..., S | C | } , where S k = { q i ∈ Q| y i = k } , to store tree tokens of the same class. The memory bank typically includes all instances from the training set. From this, we derive a set of prototypes for each class { p k } | C | k =1 , calculated as p k = (1 / | S k | ) ∑ q i ∈ S k q i . These prototypes are then used for predictions:

where sim ( · ) denotes the cosine distance and τ is a temperature scaling factor. We optimize the cross-entropy loss between the classifier's output and the ground truth to update the encoder ϕ .

Linear Classifier. Different from the prototype classifier, which utilizes class prototypes to adapt to target tasks, the linear classifier f lin directly applies the knowledge encoded in each tree token. Specifically, given a task-specific computation tree T i , we use the encoder to generate tree embeddings z i and then query the tree vocabulary to retrieve q i . These embeddings are used for predictions as:

We optimize the cross-entropy loss between the prediction f lin ( z ) and the ground truth to update the parameters of the encoder and the linear classifier. During inference, predictions from both the prototype and linear classifiers are combined to form the final output. It is important to note that the tree vocabulary remains fixed during fine-tuning to preserve the integrity of the encoded knowledge.

3.3 Additional Analysis

Tree Vocabulary Learns Generalizable Tokens. Learning tree vocabulary via VQ involves clustering within the embedding space of computation trees, utilizing a margin-aware classifier [14] that assigns each computation tree to a specific cluster. Assuming that each computation tree T is associated with an underlying clustering label y , and that each pair ( T i , y i ) is sampled from the distribution P T , we derive the following theorem:

Theorem 3.1. Given a set of computation trees { ( T i , y i ) } n i =1 sampled from the distribution P T , the VQ process functions as a margin-aware prototype classifier f that predicts the class of computation trees via a distance measure. The risk R ( f ) of classifier f can be bounded with probability 1 -δ :

where ˆ R ( f ) is the empirical risk, p denotes the number of tokens, C is a constant, and ρ acts as the margin, serving as a penalty factor in evaluating the distance between computation trees and tokens.

Remark 3.2 . The generalizability of tokens within the vocabulary highly correlates to the margin ρ , the number of observed computation trees n , and the number of tokens p . (i) A larger margin ρ results in a tighter bound by ensuring higher inter-cluster distances and lower intra-cluster distances. This supports the use of an orthogonal regularizer (Equation 5) that explicitly pushes tokens apart, enhancing cluster distinction. (ii) An increased number of observed computation trees n leads to a tighter generalization bound, which supports the use of augmentations to increase the diversity of computation trees. (iii) More tokens p may loose the upper bound of the generalization error, potentially due to a higher risk of overfitting. This aligns with our experimental findings that more tokens do not necessarily lead to improved performance (Section 4.4).

Tree Vocabulary Mitigates Negative Transfer. Negative Transfer (NT) occurs when the pre-training process degrades model performance on a target task. This issue often results from misalignment between the pre-training and fine-tuning tasks [89, 8]. Following the approach in [89], we characterize the NT gap, R ( S, T ) -R ( ∅ , T ) , as the risk gap on task T with ( R ( S, T ) ) and without ( R ( ∅ , T ) ) pre-training on task S , where a smaller NT gap indicates improved transferability. As illustrated in Figure 5, employing the learned tree vocabulary to align the tree reconstruction task in pre-training and tree classification task in fine-tuning can significantly mitigate negative transfer.

0

500

1000

Epochs

0.6

0.4

0.2

0.0

NT Gap

Our Method

Complexity Analysis. A comprehensive complexity analysis of GFT is provided in Appendix B. Notably, GFT employs a single GNN to decompose and encode computation trees, taking O ( L · ( |E| · d + |V| · d 2 )) . In contrast, subgraph-based GFMs [30, 45] require the explicit extraction of subgraphs for each node, taking additional O ( |V| 3 ) using adjacency matrix-based BFS. This contrast highlights the efficiency of using computation trees as transferable patterns in terms of time complexity. More discussions are in Appendix C.

4 Experiments

4.1 Experimental Setting

We employ cross-domain and cross-task graph datasets to evaluate the effectiveness of GFT. For node-level tasks, we utilize citation networks such as Cora , PubMed , Arxiv , and the web link network WikiCS . For edge-level tasks, we include two Knowledge Graphs (KGs), WN18RR and FB15K237 . For graph-level tasks, we use molecule networks, including HIV , PCBA , and ChEMBL . All preprocessing steps follow [45]. We take various baselines, encompassing MLP, supervised GNNs such as GCN [40], GAT [78], GIN [94], and self-supervised methods like BGRL [74], GraphMAE [26], GIANT [11], and GFMs including Prodigy [30] and OFA [45]. We replicate each experiment ten times and report the average performance to minimize the influence of randomness. Further details on experimental settings are available in Appendix F.

4.2 Effectiveness on Cross-Domain and Cross-Task Datasets

Pre-training and Fine-tuning. Table 2 demonstrates the model performance across cross-domain and cross-task datasets in pre-training and fine-tuning setting. We evaluate the effectiveness of graph foundation models [30, 45] in the following few-shot setting due their distinctive training mechanisms, such as in-context pre-training [30] and fully supervised training [45]. For supervised baselines, models are trained directly on the target graph; for self-supervised methods, we pre-train across all datasets before adapting to the specific target graph. Our approach demonstrates a substantial performance improvement, exceeding the best baseline by an average of over 6%. Specifically, our method outperforms the best baseline by 2% across three datasets and by 5% across another three datasets. This underscores the effectiveness of using computation trees as transferable patterns.

Few-shot Learning Table 3 presents the few-shot learning performance of GFT compared to selfsupervised methods [74, 26, 11] and graph foundation models [30, 45, 25]. We randomly select k samples per way from the training set for fine-tuning 3 . This method is similar to Prodigy [30], and is much more label-efficient than OFA [45] with supervised pre-training. Despite the extremely limited labels for fine-tuning, GFT significantly surpasses existing methods, showing the fast adaptation capability. Appendix H shows more fine-tuning instances can significantly improve performance.

4.3 Transferability

Table 5 shows the impact of different pre-training datasets under the pre-training and fine-tuning setting, where comprehensive results (including the following ablation studies) are available in Appendix I. The performance for specific tasks (node-, link-, graph-level) represent the average across all involved datasets. We examine three scenarios with distinct pre-training datasets: (i) all

datasets, (ii) only the target dataset, and (iii) datasets excluding the target dataset. These variants are compared against GAT and GIANT, which represent the best supervised and self-supervised baselines, respectively. Notably, GFT consistently outperforms all baselines, regardless of the pre-training dataset utilized. Interestingly, performance improves when using datasets excluding the target dataset compared to pre-training solely on the target dataset. We hypothesize that the computation trees from the non-target datasets provide sufficient information to facilitate the learning of a transferable tree vocabulary, thereby promoting positive transfer.

We further evaluate the impact of various combinations of pre-training datasets on the target tasks, as depicted in Figure 7. For pre-training, we select FB15K237, Arxiv, and ChEMBL, while Cora, WikiCS, WN18RR, and HIV serve as the target datasets. Our findings indicate that an increased number of pre-training datasets consistently enhances performance across all target datasets. However, for existing GFMs, transferability closely correlates with the selection of pre-training datasets, with more datasets sometimes leading to negative transfer [25, 43]. This observation underscores the adaptability of using computation trees as transferable patterns in graph vocabulary.

4.4 Ablation Study

Tree Reconstruction and Classification. Table 4 shows the impact of various reconstruction tasks in pre-training and tree classifiers in fine-tuning. All reconstruction tasks enhance model performance compared to models without pre-training. Notably, semantic reconstruction is most effective for node-level and graph-level tasks due to its comprehensive consideration of node features and graph structures. Feature reconstruction is particularly beneficial for link-level tasks, as it preserves the original node semantics, which are crucial for KGs. The optimal performance is achieved when three tasks are jointly optimized, aligning with findings in Ju et al. [36]. Similarly, the combination of prototype and linear classifiers in tree classification leads to superior performance. Furthermore, removing strategies designed to enhance the quality of the tree vocabulary results in model degradation across all settings (Appendix I.3).

+FB15K237

+ARXIV

+CHEMBL

Pretrain Dataset

76

78

y

=0.648

x

+77.310

(a)

(Node)

+FB15K237

+ARXIV

+CHEMBL

Pretrain Dataset

78

80

y

=0.340

x

+78.660

(b)

WikiCS

(Node)

+FB15K237

+ARXIV

+CHEMBL

Pretrain Dataset

91

92

y

=0.121

x

+91.355

(c)

WN18RR

(Edge)

+FB15K237

+ARXIV

+CHEMBL

Pretrain Dataset

68

70

72

74

y

=1.263

x

+70.638

(d)

HIV

(Graph)

Tree Vocabulary. Table 6 shows the importance of the vocabulary, where the use of vocabulary significantly enhances model performance, particularly in link- and graph-level tasks, which aligns to the theoretical analysis that the tree vocabulary improves generalization. However, we observe that increasing the number of tokens in the vocabulary does not necessarily enhance model performance; indeed, the improvements are often marginal.

5 Conclusion

Conclusion. In this paper, we rethink the transferable pattern in graphs as computation trees and validate their transferability both empirically and theoretically. Building on this insight, we propose a cross-domain and cross-task GFM named GFT. This model leverages computation tree reconstruction to acquire general graph knowledge from cross-domain datasets and uses computation tree classification to facilitate adaptation to various target tasks. In future work, we aim to explore its capabilities for in-context learning and zero-shot learning.

Limitations. In this paper, we focus primarily on message-passing GNNs, as message-passing can be naturally unrolled as a tree-like structure. However, our analysis excludes graph transformers and expressive GNNs with specialized computational architectures. We plan to extend our analysis to understand the transferable patterns of these advanced learning algorithms in future work. Additionally, message-passing GNNs may lack the expressiveness needed to address isomorphism problems in graphs. One can apply advanced techniques [105] to handle link isomorphism and use advanced expressive GNNs [103] to tackle graph isomorphism. Moreover, the deployment of GFT in real-world applications may encounter efficiency issues, which can be mitigated by techniques like [106, 88].

Boarder Impact. The proposed GFT is a cross-domain and cross-task graph foundation model, designed for rapid adaptation to target tasks with extremely limited labels. We wish our research can support applications where label acquisition is challenging and model training is time-consuming, such as in molecular discovery and financial fraud detection.

Acknowledgments

This work was partially supported by the NSF under grants IIS-2321504, IIS-2334193, IIS-2340346, IIS-2203262, IIS-2217239, CNS-2426514, CNS-2203261, and CMMI-2146076. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv , 2023.

Yushi Bai, Xin Lv, Juanzi Li, and Lei Hou. Answering complex logical queries on knowledge graphs via query computation tree optimization. In ICML , 2023.

Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learning for large vision models. arXiv , 2023.

Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. JMLR , 2002.

Yoshua Bengio. Deep learning of representations for unsupervised and transfer learning. In ICML Workshop , 2012.

Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv , 2013.

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv , 2021.

Yuxuan Cao, Jiarong Xu, Carl Yang, Jiaan Wang, Yunchao Zhang, Chunping Wang, Lei Chen, and Yang Yang. When to pre-train graph neural networks? from data generation perspective! In KDD , 2023.

Ziwei Chai, Tianjie Zhang, Liang Wu, Kaiqiao Han, Xiaohai Hu, Xuanwen Huang, and Yang Yang. Graphllm: Boosting graph reasoning ability of large language model. arXiv , 2023.

Zhengdao Chen, Lei Chen, Soledad Villar, and Joan Bruna. Can graph neural networks count substructures? In NeurIPS , 2020.

Eli Chien, Wei-Cheng Chang, Cho-Jui Hsieh, Hsiang-Fu Yu, Jiong Zhang, Olgica Milenkovic, and Inderjit S Dhillon. Node feature extraction by self-supervised multi-scale neighborhood prediction. In ICLR , 2022.

Ching-Yao Chuang and Stefanie Jegelka. Tree mover's distance: Bridging graph metrics and stability of graph neural networks. In NeurIPS , 2022.

Weilin Cong, Morteza Ramezani, and Mehrdad Mahdavi. On provable benefits of depth in training graph convolutional networks. In NeurIPS , 2021.

Koby Crammer, Ran Gilad-Bachrach, Amir Navot, and Naftali Tishby. Margin analysis of the lvq algorithm. In NeurIPS , 2002.

Kaize Ding, Jianling Wang, Jundong Li, Kai Shu, Chenghao Liu, and Huan Liu. Graph prototypical networks for few-shot learning on attributed networks. In CIKM , 2020.

Pascal Esser, Leena Chennuru Vankadara, and Debarghya Ghoshdastidar. Learning theory can (sometimes) explain generalisation in graph neural networks. In NeurIPS , 2021.

Bahare Fatemi, Jonathan Halcrow, and Bryan Perozzi. Talk like a graph: Encoding graphs for large language models. In ICLR , 2024.

Matthias Fey, Jan-Gin Yuen, and Frank Weichert. Hierarchical inter-message passing for learning on molecular graphs. In ICML Workshop , 2020.

Mikhail Galkin, Xinyu Yuan, Hesham Mostafa, Jian Tang, and Zhaocheng Zhu. Towards foundation models for knowledge graph reasoning. In ICLR , 2024.

Vikas Garg, Stefanie Jegelka, and Tommi Jaakkola. Generalization and representational limits of graph neural networks. In ICML , 2020.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In ICML , 2017.

Jiayan Guo, Lun Du, and Hengyu Liu. Gpt4graph: Can large language models understand graph structured data? an empirical evaluation and benchmarking. arXiv , 2023.

Mridul Gupta, Sahil Manchanda, Sayan Ranu, and Hariprasad Kodamana. Mirage: Modelagnostic graph distillation for graph classification. In ICLR , 2024.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NeurIPS , 2017.

Yufei He and Bryan Hooi. Unigraph: Learning a cross-domain graph foundation model from natural language. arXiv , 2024.

Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang, and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. In KDD , 2022.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In NeurIPS , 2020.

Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. Ogb-lsc: A large-scale challenge for machine learning on graphs. In NeurIPS , 2021.

Jin Huang, Xingjian Zhang, Qiaozhu Mei, and Jiaqi Ma. Can llms effectively leverage graph structural information: when and why. arXiv , 2023.

Qian Huang, Hongyu Ren, Peng Chen, Gregor Kržmanc, Daniel Zeng, Percy S Liang, and Jure Leskovec. Prodigy: Enabling in-context learning over graphs. In NeurIPS , 2023.

Zhongyu Huang, Yingheng Wang, Chaozhuo Li, and Huiguang He. Growing like a tree: Finding trunks from graph skeleton trees. TPAMI , 2023.

Minyoung Huh, Brian Cheung, Pulkit Agrawal, and Phillip Isola. Straightening out the straight-through estimator: Overcoming optimization challenges in vector quantized networks. In ICML , 2023.

Junguang Jiang, Yang Shu, Jianmin Wang, and Mingsheng Long. Transferability in deep learning: A survey. arXiv , 2022.

Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In ICML , 2018.

Haotian Ju, Dongyue Li, Aneesh Sharma, and Hongyang R Zhang. Generalization in graph neural networks: Improved pac-bayesian bounds on graph diffusion. In AISTATS , 2023.

Mingxuan Ju, Tong Zhao, Qianlong Wen, Wenhao Yu, Neil Shah, Yanfang Ye, and Chuxu Zhang. Multi-task self-supervised graph neural networks enable stronger task generalization. In ICLR , 2023.

Pentti Kanerva. Sparse distributed memory . MIT press, 1988.

Pentti Kanerva. Sparse distributed memory and related models. Technical report, 1992.

Thomas N Kipf and Max Welling. Variational graph auto-encoders. In NeurIPS Workshop , 2016.

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR , 2017.

Ron Levie, Elvin Isufi, and Gitta Kutyniok. On the transferability of spectral graph filters. In SampTA , 2019.

Ron Levie, Wei Huang, Lorenzo Bucci, Michael Bronstein, and Gitta Kutyniok. Transferability of spectral graph convolutional neural networks. JMLR , 2021.

Yuhan Li, Peisong Wang, Zhixun Li, Jeffrey Xu Yu, and Jia Li. Zerog: Investigating crossdataset zero-shot transferability in graphs. arXiv , 2024.

Renjie Liao, Raquel Urtasun, and Richard Zemel. A pac-bayesian approach to generalization bounds for graph neural networks. In ICLR , 2021.

Hao Liu, Jiarui Feng, Lecheng Kong, Ningyue Liang, Dacheng Tao, Yixin Chen, and Muhan Zhang. One for all: Towards training one graph model for all classification tasks. In ICLR , 2024.

Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, et al. Sora: A review on background, technology, limitations, and opportunities of large vision models. arXiv , 2024.

Zemin Liu, Xingtong Yu, Yuan Fang, and Xinming Zhang. Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. In WWW , 2023.

Tianyi Ma, Yiyue Qian, Chuxu Zhang, and Yanfang Ye. Hypergraph contrastive learning for drug trafficking community detection. In ICDM , 2023.

Yao Ma, Xiaorui Liu, Neil Shah, and Jiliang Tang. Is homophily a necessity for graph neural networks? In ICLR , 2022.

Haitao Mao, Zhikai Chen, Wenzhuo Tang, Jianan Zhao, Yao Ma, Tong Zhao, Neil Shah, Michael Galkin, and Jiliang Tang. Graph foundation models. In ICML , 2024.

Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelligence. Nature , 2023.

Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In AAAI , 2019.

Christopher Morris, Floris Geerts, Jan Tönshoff, and Martin Grohe. Wl meet vc. In ICML , 2023.

Giannis Nikolentzos, Michail Chatzianastasis, and Michalis Vazirgiannis. Weisfeiler and leman go hyperbolic: learning distance preserving node representations. In AISTATS , 2023.

Karl Pearson. The problem of the random walk. Nature , 1905.

Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gcn: Geometric graph convolutional networks. In ICLR , 2020.

Nataša Pržulj. Biological network comparison using graphlet degree distribution. Bioinformatics , 2007.

Yiyue Qian, Tianyi Ma, Chuxu Zhang, and Yanfang Ye. Dual-level hypergraph contrastive learning with adaptive temperature enhancement. In WWW , 2024.

Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. Gcc: Graph contrastive coding for graph neural network pre-training. In KDD , 2020.

Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bertnetworks. In EMNLP , 2019.

Leonardo FR Ribeiro, Pedro HP Saverese, and Daniel R Figueiredo. struc2vec: Learning node representations from structural identity. In KDD , 2017.

Luana Ruiz, Luiz Chamon, and Alejandro Ribeiro. Graphon neural networks and the transferability of graph neural networks. In NeurIPS , 2020.

Franco Scarselli, Ah Chung Tsoi, and Markus Hagenbuchner. The vapnik-chervonenkis dimension of graph and recursive neural networks. Neural Networks , 2018.

Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, and Karsten M Borgwardt. Weisfeiler-lehman graph kernels. JMLR , 2011.

Woncheol Shin, Gyubok Lee, Jiyoung Lee, Eunyi Lyou, Joonseok Lee, and Edward Choi. Exploration into translation-equivariant image quantization. In ICASSP , 2023.

Jiawei Sun, Kailai Li, Ruoxin Chen, Jie LI, Chentao Wu, Yue Ding, and Junchi Yan. InterpGNN: Understand and improve generalization ability of transdutive GNNs through the lens of interplay between train and test nodes. In ICLR , 2024.

Mingchen Sun, Kaixiong Zhou, Xin He, Ying Wang, and Xin Wang. Gppt: Graph pre-training and prompt tuning to generalize graph neural networks. In KDD , 2022.

Xiangguo Sun, Hong Cheng, Jia Li, Bo Liu, and Jihong Guan. All in one: Multi-task prompting for graph neural networks. In KDD , 2023.

Yifei Sun, Qi Zhu, Yang Yang, Chunping Wang, Tianyu Fan, Jiajun Zhu, and Lei Chen. Fine-tuning graph neural networks by preserving graph generative patterns. In AAAI , 2024.

Rajat Talak, Siyi Hu, Lisa Peng, and Luca Carlone. Neural trees for learning on graphs. In NeurIPS , 2021.

Zhen Tan, Song Wang, Kaize Ding, Jundong Li, and Huan Liu. Transductive linear probing: a novel framework for few-shot node classification. In LoG , 2022.

Huayi Tang and Yong Liu. Towards understanding generalization of graph neural networks. In ICML , 2023.

Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin, and Chao Huang. Graphgpt: Graph instruction tuning for large language models. In SIGIR , 2024.

Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Mehdi Azabou, Eva L Dyer, Remi Munos, Petar Veliˇ ckovi´ c, and Michal Valko. Large-scale representation learning on graphs via bootstrapping. In ICLR , 2022.

Yijun Tian, Huan Song, Zichen Wang, Haozhu Wang, Ziqing Hu, Fang Wang, Nitesh V Chawla, and Panpan Xu. Graph neural prompting with large language models. In AAAI , 2024.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv , 2023.

Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NeurIPS , 2017.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR , 2018.

Petar Veliˇ ckovi´ c, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In ICLR , 2019.

Saurabh Verma and Zhi-Li Zhang. Stability and generalization of graph convolutional neural networks. In KDD , 2019.

Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov. Can language models solve graph problems in natural language? In NeurIPS , 2024.

Song Wang, Chen Chen, and Jundong Li. Graph few-shot learning with task-specific structures. NeurIPS , 2022.

Song Wang, Kaize Ding, Chuxu Zhang, Chen Chen, and Jundong Li. Task-adaptive few-shot node classification. In KDD , 2022.

Yu Wang and Tyler Derr. Tree decomposed graph neural network. In CIKM , 2021.

Zehong Wang, Qi Li, Donghua Yu, Xiaolong Han, Xiao-Zhi Gao, and Shigen Shen. Heterogeneous graph contrastive multi-view learning. In SDM , 2023.

Zehong Wang, Donghua Yu, Shigen Shen, Shichao Zhang, Huawen Liu, Shuang Yao, and Maozu Guo. Select your own counterparts: Self-supervised graph contrastive learning with positive sampling. TNNLS , 2024.

Zehong Wang, Zheyuan Zhang, Chuxu Zhang, and Yanfang Ye. Subgraph pooling: Tackling negative transfer on graphs. In IJCAI , 2024.

Zehong Wang, Zheyuan Zhang, Chuxu Zhang, and Yanfang Ye. Training mlps on graphs without supervision. In WSDM , 2025.

Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime Carbonell. Characterizing and avoiding negative transfer. In CVPR , 2019.

Qianlong Wen, Mingxuan Ju, Zhongyu Ouyang, Chuxu Zhang, and Yanfang Ye. From coarse to fine: Enable comprehensive graph self-supervised learning with multi-granular semantic ensemble. In ICML , 2024.

Jun Wu, Jingrui He, and Elizabeth Ainsworth. Non-iid transfer learning on graphs. In AAAI , 2023.

Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. TNNLS , 2020.

Lianghao Xia, Ben Kao, and Chao Huang. Opengraph: Towards open graph foundation models. arXiv , 2024.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR , 2019.

Yuchen Yan, Peiyan Zhang, Zheng Fang, and Qingqing Long. Inductive graph alignment prompt: Bridging the gap between graph pre-training and inductive fine-tuning from spectral perspective. In WWW , 2024.

Ling Yang, Ye Tian, Minkai Xu, Zhongyi Liu, Shenda Hong, Wei Qu, Wentao Zhang, CUI Bin, Muhan Zhang, and Jure Leskovec. Vqgraph: Rethinking graph representation space for bridging gnns and mlps. In ICLR , 2024.

Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQGAN. In ICLR , 2022.

Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion-tokenizer is key to visual generation. In ICLR , 2024.

Xingtong Yu, Chang Zhou, Yuan Fang, and Xinming Zhang. Multigprompt for multi-task pre-training and prompting on graphs. In WWW , 2024.

Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv , 2021.

Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. TASLP , 2021.

Werner Zellinger, Thomas Grubinger, Edwin Lughofer, Thomas Natschläger, and Susanne Saminger-Platz. Central moment discrepancy (CMD) for domain-invariant representation learning. In ICLR , 2017.

Bohang Zhang, Jingchu Gai, Yiheng Du, Qiwei Ye, Di He, and Liwei Wang. Beyond weisfeilerlehman: A quantitative framework for gnn expressiveness. In ICLR , 2024.

Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V Chawla. Heterogeneous graph neural network. In KDD , 2019.

Muhan Zhang, Pan Li, Yinglong Xia, Kai Wang, and Long Jin. Labeling trick: A theory of using graph neural networks for multi-node representation learning. In NeurIPS , 2021.

Shichang Zhang, Yozen Liu, Yizhou Sun, and Neil Shah. Graph-less neural networks: Teaching

old MLPs new tricks via distillation. In ICLR , 2022.

Zheyuan Zhang, Zehong Wang, Shifu Hou, Evan Hall, Landon Bachman, Jasmine White, Vincent Galassi, Nitesh V Chawla, Chuxu Zhang, and Yanfang Ye. Diet-odin: A novel framework for opioid misuse detection with interpretable dietary patterns. In KDD , 2024.

Haihong Zhao, Aochuan Chen, Xiangguo Sun, Hong Cheng, and Jia Li. All in one and one for all: A simple yet effective method towards cross-domain graph pretraining. In KDD , 2024.

Jianan Zhao, Le Zhuo, Yikang Shen, Meng Qu, Kai Liu, Michael Bronstein, Zhaocheng Zhu, and Jian Tang. Graphtext: Graph reasoning in text space. arXiv , 2023.

Shuxin Zheng, Jiyan He, Chang Liu, Yu Shi, Ziheng Lu, Weitao Feng, Fusong Ju, Jiaxi Wang, Jianwei Zhu, Yaosen Min, et al. Towards predicting equilibrium distributions for molecular systems with deep learning. arXiv , 2023.

Wenqing Zheng, Edward W Huang, Nikhil Rao, Zhangyang Wang, and Karthik Subbian. You only transfer what you share: Intersection-induced graph transfer learning for link prediction. TMLR , 2023.

Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt. arXiv , 2023.

Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. In NeurIPS , 2020.

Qi Zhu, Carl Yang, Yidan Xu, Haonan Wang, Chao Zhang, and Jiawei Han. Transfer learning of graph neural networks with ego-graph information maximization. In NeurIPS , 2021.

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. In ICML Workshop , 2020.

Appendix: Table of Content

A More Related Work

Transferability of GNNs. Early studies on the transferability of GNNs are based on two main approaches. The first approach utilizes graphon theory. Ruiz et al. [62] derive a bound between the embeddings of two graphs sampled from the same graphon. However, identifying a common graphon in many real-world graphs is often unfeasible, limiting the direct application of this theorem in the design of graph foundation models. Cao et al. [8] employ the graphon theory to analyze transferability in pre-training and fine-tuning setting. They fit pre-trained graphs into a graphon space, ensuring transferability if the target graph can be generated within this space. Like Ruiz et al. [62], the challenge lies in acquiring sufficient data to adequately represent the graphon space for graph foundation model. Following this, Sun et al. [69] designed a fine-tuning method based on graphon theory. Although these studies consider graphon as a transferable pattern in graphs, the assumption is challenging to satisfy in cross-domain real-world graphs. Furthermore, graphon theory is generally limited to single-task applications, making it difficult to identify a shared graphon across node-, link-, and graph-level tasks. The send approach examines transferability through subgraphs [114] or graph spectrum [41, 42]. Specifically, Levie et al. [41] analyze transferability from the perspective of stability, positing that effective transferability minimizes the impact of small perturbations. Similarly, Levie et al. [42] explore transferability through stability, demonstrating that transfer between graphs is feasible when they discretize the same underlying space in a generic sense. Zhu et al. [114] focus on transferability through ego-graphs, showing a higher similar ego-graph distribution leads to better transferability. In contrast to these two approaches, we treat computation trees as transferable patterns on graphs, and conduct both empirically and theoretically analysis to show their transferability. Additionally, we develop a graph foundation model that utilizes these computation tree patterns.

Generalization of GNNs. Generalization is a closely related topic to transferability. For instance, Scarselli et al. [63] pioneer the analysis of GNNs' VC-dimension, focusing solely on the number of nodes. Garg et al. [20] leverage Rademacher complexity to evaluate GNN generalization through computation tree perspectives. Furthermore, the Rademacher complexity has been extended to transductive settings by Esser et al. [16] and Cong et al. [13]. Under the PAC-Bayesian framework, Liao et al. [44] offer a tighter generalization bound for GNNs compared to Garg et al. [20], and Ju et al. [35] further improve the bound. Additionally, Sun et al. [66] investigate these bounds through the lens of graph topology. Stability is another lens through which generalization is examined, with Verma and Zhang [80] focusing on 1-layer GNNs and linking generalization to the largest absolute eigenvalue of the graph convolution filter. Tang and Liu [72] further establish bounds for transductive node classification, highlighting the significance of graph structural information for different GNN architectures.

GNN-based Graph Foundation Models. Developing graph foundation models involves two primary steps: (i) unifying the task space and (ii) unifying the domain space. Several studies focus on aligning the task space. Qiu et al. [59] introduce a self-supervised model that empirically demonstrates the transferability of subgraphs across tasks. Sun et al. [67], Liu et al. [47] pinpoint the task gap between pre-training and fine-tuning as the primary performance bottleneck, addressing it through link prediction to unify these tasks. Yan et al. [95] further adapt this model to an inductive setting, where the pre-training and fine-tuning graphs differ, proposing methods to bridge both the graph signal and structure gaps. Yu et al. [99] implement multi-task pre-training to support various downstream tasks. Sun et al. [68] employ subgraphs as fundamental transferable patterns, integrating node-, link-, and graph-level tasks into a unified subgraph-level task. Instead of extensive model fine-tuning, they incorporate a learnable subgraph into the original graph. Other research focuses on aligning the domain space. Li et al. [43] introduce a zero-shot graph learning framework for cross-domain node classification, leveraging LLMs to unify node features across different graphs. In a similar vein, Zhao et al. [108] propose a graph prompting method for cross-domain classification, utilizing singular value decomposition to align the feature space across various graphs. However, all of these methods are generally limited to single tasks or domains, and do not effectively address the complexities of datasets that span multiple domains and tasks.

To this end, Huang et al. [30] introduce a graph foundation model that utilizes LLMs to align the feature space of graphs and utilizes in-context learning to facilitate applications in node-level and link-level tasks. In particular, they extract the subgraphs for different tasks and conduct subgraph classification. However, this approach necessitates that the pre-training and fine-tuning tasks be

identical due to the specialized in-context pre-training strategy. To address this constraint, Liu et al. [45] use LLMs to align the feature spaces of cross-domain graph datasets and introduce a prompt graph to align various tasks on graphs. Xia et al. [93] leverages a graph tokenizer to convert the graph into sequence and propose to use transformer to handle such information. Following, He and Hooi [25] concurrently train GNNs and LLMs to enhance performance further. Despite their empirical success, subgraph-based graph foundation models face challenges due to the GNNs' limitations in encoding substructures within subgraphs. Differently, we rethink computation trees as transferable patterns and propose a new graph foundation model based on that.

LLM-based Graph Foundation Models. LLMs present a promising avenue for the development of graph foundation models due to their ability to unify the output of various graph tasks [17, 22, 81]. While GNNs require task-specific adjustments for model training, LLMs can accommodate a wide range of questions and generate appropriate answers. The primary challenge, though, lies in effectively translating graph structures into a natural language format that LLMs can comprehend. Current efforts in this domain focus on two main approaches. The first is to use natural language to describe the graph structure, such as what the nodes are and which pairs of nodes are connected [17]. Such methods can be further enhanced with extra embedding [9] or prompt engineer techniques to enhance the understanding of LLM. For example, Guo et al. [22] employ a self-prompting methods to utilize the context generated by LLM as input context; Chai et al. [9] used Build-a-Graph and Algorithmic prompting to faciliate LLM understanding. Zhao et al. [109] map a graph into tree-like tokens for designing LLM prompt, further enhancing learning capability. Additionally, another line of works [75, 73] follow Visual Language Models (VLMs) to process the graph into embeddings by GNNs first and then employ LLM as a translator to decode the graph embedding.

It is noted that prior research such as Wang et al. [81], Huang et al. [29] indicate that LLMs can indeed capture structural information from graphs, which enhances their performance on downstream tasks. However, while LLMs show promise in basic graph reasoning tasks like connectivity checks and cycle detection, they struggle with complex graph patterns in graph learning tasks such as node and graph classification. Moreover, there is limited research on cross-domain graph foundation models, largely due to the diverse patterns and distributions of graphs across different domains. This underscores the importance of our work in identifying transferable patterns within graphs to pave the way for future advancements in graph foundation models.

Computation Tree. The computation tree, or more broadly, the subtree, is a fundamental structure on graphs [20]. It serves to (i) enhance the performance of existing GNNs and to (ii) measure graph similarity. Several studies [34, 18, 70, 84] employ tree decomposition to develop advanced GNNs. Specifically, Jin et al. [34] treat the joint tree as complementary to the original graphs, while Fey et al. [18] introduce inter-modality message passing between joint trees and the original graphs. Talak et al. [70] construct an H-tree by organizing nodes and subgraphs hierarchically, developing a neural tree model capable of approximating any probability distribution on a probabilistic graphical model. Wang and Derr [84] propose a more efficient tree decomposition algorithm by separating the model layer and tree construction. Furthermore, Huang et al. [31] investigate the significance of trees in learning node representations and design a framework to identify the most crucial trees in a graph. Additionally, Nikolentzos et al. [54] design a hyperbolic learning framework to utilize the computation tree structure in creating expressive GNNs. Bai et al. [2] adapt the computation tree concept to knowledge graphs by optimizing the solution in query computation trees. Another avenue of research utilizes computation tree distributions to measure graph similarity. Notably, Shervashidze et al. [64] introduce the WL subtree kernel to measure discrepancies between graphs based on subtree structures. Chuang and Jegelka [12] employs optimal transport to propose the tree mover's distance, estimating distribution shifts between graphs. Wu et al. [91] utilizes a hierarchical WL subtree kernel to assess graph discrepancies and derive a generalization bound for cross-domain classification.

Different from these approaches, our work rethink the role of computation trees. We consider computation tree as a transferable pattern on graphs and both empirically and theoretically validate its transferability, thereby expanding the analysis for computation trees in graph learning.

B Complexity Analysis

B.1 Computation Tree Decomposition and Encoding

The decomposition and encoding of computation trees can be jointly finished by message-passing GNNs. Specifically, the learning process in message passing GNNs involves (i) extracting computation trees for each node and (ii) updating node embeddings within these trees from bottom to top. Utilizing a GraphSAGE-like architecture, as detailed in Appendix F, each layer's learning comprises both aggregation and updating operations. We will analyze these two operations in the following.

Aggregation is an edge-wise operation that propagates messages from neighboring nodes to the target node. Consequently, this operation's computational complexity is linear to the number of edges in the graph G = ( V , E ) , expressed as O ( |E| · d ) , where d is the embedding dimension. The update process, on the other hand, is a node-wise operation that updates the state of each node based on aggregated messages through a neural network. Therefore, its time complexity is O ( |V| · d 2 ) , as the complexity of the neural network operations per node is O ( d 2 ) . By integrating both aggregation and update processes in each layer, the overall complexity of our model is O ( L · ( |E| · d + |V| · d 2 )) .

For node-level tasks, we directly use the embeddings of computation trees of the target node, which incurs a constant time complexity of O (1) . For link-level and graph-level tasks, we apply a nonparametric pooling function to aggregate subtree embeddings into a computation tree embedding, also with a time complexity of O (1) .

B.2 Subgraph Extraction

Current graph foundation models [30, 45] treat subgraphs as transferable patterns and explicitly extract subgraphs for each node. For a given graph G = ( V , E ) , the extraction of ego-graph for all nodes results in a computational cost of O ( |V| 3 ) when using an adjacency matrix for the BFS algorithm.

B.3 Vector Quantization

Vector quantization assigns each instance to the nearest token in the vocabulary. This process involves measuring the distance between the instance and every token, then selecting the token with the minimum distance as the quantized embedding. Assuming there are K tokens, the complexity of distance measurement (no matter Euclidean or Cosine) is O ( K × d ) per instance. Then, determining the shortest distance from K measured distances can be achieved in O ( K ) , and replacing the original instance embedding with the selected token requires O (1) . Therefore, the predominant computational cost is distance measurement, leading to an overall complexity of O ( |T | · K · d ) , where |T | represents the number of computation trees.

B.4 Tree Reconstruction

The computation tree reconstruction comprises three main tasks: feature reconstruction, semantic reconstruction, and topology reconstruction. Feature reconstruction utilizes a neural network-based decoder with a complexity of O ( d 2 ) and a mean squared error (MSE) loss of O ( |T | ) , resulting in a total complexity of O ( d 2 + |T | ) . Topology reconstruction focuses on edge information and takes a computational cost of O ( |E| · d ) . Semantic reconstruction involves an additional GNN and a distance measurement, leading to a complexity of O ( L · ( |E| · d + |V| · d 2 ) + |T | · d ) . Consequently, the overall computational complexity is approximated as O ( L · ( |E| · d + |V| · d 2 ) + |T | · d ) .

B.5 Tree Classification

The computation tree classification process employs both a prototype-based classifier and a linear classifier. The prototype-based classifier constructs prototypes from a memory bank, which incurs a complexity of O ( |T | ) . It then classifies instances by measuring their distances to these prototypes, resulting in a complexity of O ( |T | · | C | · d ) , where | C | represents the number of classes. On the other hand, the linear classifier incurs a complexity of O ( |T | · d ) . Consequently, the total computational complexity can be approximated as O ( |T | · | C | · d ) .

C More Analysis

C.1 The Difference Between Computation Tree and Subgraph

Our concept of computation trees is closely aligned with [12], representing tree-like patterns derived from unfolding the message passing process. Encoding the computation trees of a node is equivalent to encoding the node itself via message passing GNNs, implying that the information in computation trees can be fully learned by basic GNNs, demonstrating both learnability and efficiency in encoding computation trees. Notably, computation tree can be reinterpreted as a special pattern preserved on the ego-graph of the target node, differing from junction trees [34] or H-trees [70], which construct additional tree-like graphs to complement the original graph.

Subgraphs, on the other hand, are graph-like substructures within the original graph, such as motifs in molecule graphs. Sun et al. [68] identifies subgraphs as basic patterns across graph-related tasks and reformulates these tasks into subgraph-level tasks. For example, in node classification, they extract the ego-graph around each node and assign the label of the induced graph as the label of the center node, converting node classification into subgraph classification. This process involves (1) extracting ego-graphs around task-relevant nodes and (2) using GNNs to learn graph-level embeddings for classification. However, this extraction process introduces additional time consumption and increased memory usage for storing induced subgraphs. More importantly, the information in subgraphs is not always learnable by basic GNNs, as they cannot detect some critical substructures necessary for learning graph-level embeddings [10, 103], reducing the feasibility of using subgraphs to define graph vocabularies.

We provide empirical analysis for better understanding. Efficiency analysis is presented in Figure 8. Subgraphs generally incur an extra 1/3 time consumption compared to computation trees and encounter out-of-memory errors when batch size exceeds 2048, compared to 8192 for computation trees. The performance comparison is shown in Table 7, where the subgraph version (GFT - Subgraph) performs worse than the computation tree version (GFT). We use GAT and GraphMAE as additional baselines and apply linear classifiers on all models for a fair comparison.

2 8

2 9

2 10

2 11

2 12

2 13

25

50

75

100

Memory Usage (%)

GFT

GFT-Subgraph

GraphMAE

2 8

2 9

2 10

2 11

2 12

2 13

400

600

800

Time (s) / Epoch

GFT

GFT-Subgraph

GraphMAE

(a) The memory allocation in an A40 (48GB) GPU.

(b) The time consumption per epoch.

Figure 8: The efficiency analysis between computation trees and subgraphs. Our GFT is based on the computation trees and we further replace the computation trees with subgraphs called GFT-Subgraph. We compare their memory usage (a) and time consumption (b) during pretraining. With the increase of batch sizes, Subgraph-based GFT encounters out-of-memory, yet computation tree-based GFT can still fit in the GPU.

C.2 Preventing Vocabulary Collapse

Another challenge in developing a robust discrete vocabulary is known as vocabulary collapse (or codebook collapse in VQ). The commitment loss (Equation 2) effectively prevents this issue by aligning the quantized tree embeddings with the token space [77]. Furthermore, we have empirically discovered that using Euclidean distance to query tree tokens leads to vocabulary collapse. Consequently, we have switched to Cosine distance to enforce querying within a hyper-sphere space, thereby enhancing training stability [97]. Alternatively, other techniques such as expiring stale codes [101] or affine re-parameterization [32] (not evaluated in this paper) can also be employed to mitigate this problem.

C.3 Scaling to Large-scale Graphs

Due to the emergence of large-scale graphs [27, 28], efficient training often requires the use of mini-batches. We facilitate mini-batch training through subgraph sampling. In pre-training, we employ basic subgraph sampling techniques [24] to extract a smaller graph from the original graph and then extract computation trees for each node within this subgraph. This method serves as an additional topology augmentation, further enhancing the diversity of computation trees through re-sampling. In the fine-tuning phase, subgraph sampling remains effective for the linear classifier, as it directly processes the computation tree. However, the prototype-based classifier, which requires the aggregation of instances with identical labels to form class prototypes, faces efficiency challenges in this mini-batch training setting. To address this, we randomly sample a small subset of the training set for each class to construct the memory bank S . Based on our empirical observations, a limited number of samples per class suffices to achieve desirable performance.

C.4 Discussion on Homophily and Heterophily

Homophily and heterophily [49] are both critical properties for node-level tasks. The primary distinction between these types of graphs is that identical connectivity patterns can indicate different meanings. We consider our model is also effective for heterophily graphs. Although we only evaluate the performance of GFT on homophily graphs (Cora, PubMed, WikiCS, Arxiv), two considerations support its applicability to heterophily graphs: (i) The analysis of computation tree transferability shows that, similar to homophily, higher computation tree similarity in heterophily graphs correlates with enhanced transferability, matching the principle of our GFT. (ii) Our proposed computation tree classification in fine-tuning can adaptively reinterpret the patterns encoded in the tree vocabulary across various tasks. We will leave the experiments on heterophily graphs in the future work.

C.5 Comparison to VQGraph

The major connection between GFT and VQGraph [96] is the usage of vector quantization in learning a discrete vocabulary for downstream tasks. However, there are four major differences between GFT and VQGraph. (1) Model Objective: GFT focuses on building a general task reasoner, but VQGraph aims to train a structure-aware MLP for efficient inference. (2) Pretrain Dataset: GFT is pre-trained on cross-domain and cross-task datasets to acquire transferable patterns among graphs, but VQGraph is pre-trained on a single dataset to better capture the structural information. (3) Usage of Tokens: GFT treats tokens as specific transferable patterns, using them directly to build classifiers. VQGraph, on the other hand, treats tokens as external structural knowledge to complement the training of MLP classifiers. (4) Downstream Tasks: GFT can be applied to various graph-related tasks with different settings like few-shot and zero-shot learning. VQGraph is designed for node classification with a basic pre-training and fine-tuning setting.

C.6 Comparison to LLM-based Methods

Recent researches [17, 22, 81] utilize LLMs to reformulate graph-related tasks as question answering, transforming graph datasets into sentence structures and leveraging the inference capabilities of LLMs to implicitly infer structural knowledge from the original graphs. This approach exploits the transferable patterns in the word vocabulary of LLMs to reinterpret the transferable patterns on graphs. The main challenges include (i) aligning the transformed graphs (sentences) with the word vocabulary of LLMs, and (ii) employing LLMs to infer essential structural knowledge for

graph-structured data. Due to these challenges, existing methods often fall short in handling graph datasets with LLMs, resulting in inconsistent performance. Unlike these approaches, which entirely abandon GNNs, we utilize a GNN as an encoder to analyze transferable patterns on graphs. We consider these LLM-based approaches as complementary to our work.

C.7 Comparison to Subgraph-based GFMs

Several studies [30, 45] identify subgraphs as transferable patterns across graphs, unifying graph-related tasks into subgraph classification tasks and explicitly extracting subgraphs for classification. However, this extraction process incurs additional time and memory costs due to overlapping nodes in the extracted subgraphs. Moreover, other research [20, 103] suggests that certain substructure or motif patterns within subgraphs are not learnable by basic message-passing GNNs. Unlike these methods, our GFT treats computation trees as transferable patterns, offering advantages over these GFMs in both respects. Firstly, GFT does not require the explicit extraction and encoding of computation trees, instead employing message passing to inherently processes computation trees rooted at all nodes, ensuring efficiency in both time and memory. Furthermore, the

computation tree can be seen a unique subgraph structure, which is fully learnable by GNNs without information loss.

In addition, we also compare the number of parameters of these GFMs in Table 8. Considering the number of parameters, Prodigy [30] has 2 million parameters, while OFA [45] has 29 million since the use of more GNN layers. UniGraph [25] has 180 million parameters, primarily due to its explicit integration with LLMs in encoding node features in an end-to-end way. Our GFT consistently maintains 7 million parameters during both pre-training and fine-tuning phases, making it comparable to Prodigy but significantly fewer than OFA and UniGraph.

C.8 Detailed Illustration of Computation Tree Reconstruction

See Figure 9.

(1) Feat. Recon.

Dec

ℒ

Dec

ℒ

(2) Sem. Recon.

Dec

ℒ

?

?

?

(3) Topo. Recon.

D Proofs

D.1 Proof for Theorem 2.2

We restate Theorem 2.2 from the main paper as below.

Theorem D.1 (Transferability of Computation Tree) . Given two L -layer computation trees T v 1 , T v 2 derived from the graph G and a GNN encoder ϕ with parameters W = ( W 1 , W 2 ) , the Euclidean distance between the tree embeddings ∆ ≜ ∥ ϕ ( T v 1 ) -ϕ ( T v 2 ) ∥ 2 is bounded as follows:

where ∆ L -1 v 1 ,v 2 ,j represents the distance between the L -1 -layer subtrees of the j -th children of nodes v 1 and v 2 , and constants C 1 = C σ B W 1 and C 2 = C σ C ρ C g B W 2 . Here C σ , C ρ , C g are Lipschitz terms for GNN components, and B W 1 , B W 2 , B x denote bounded norms of W 1 , W 2 , x , respectively. The variable d l indicates the number of children in the l -layer subtrees, with each d l ≤ d , and D l = d l d l -1 ...d 1 .

Proof. We calculate the embedding distance between two L -layer computation trees generated from a single GNN encoder ϕ with parameters W = ( W 1 , W 2 ) . Here we use a GraphSAGE-like encoder, as described in the Appendix F.1, that W 1 transforms the target node, while W 2 transforms the neighboring nodes. For simplicity, we assume that all GNN layers share the same parameters. Without loss of generality, this assumption does not affect the validity of our proofs. The term x v represents the features of node v , and N ( v ) denotes the set of direct neighborhood in the graph, which correspond to the children of node v in the computation tree T v .

With a bit of notation abuse, we define the GNN as:

where σ as the non-linear activation function, ρ as the permutation-invariant aggregator function, and g as the update function ( ρ and g are all based on neural networks). To simplify notation, we denote the computation tree embeddings by T ( W ) = ϕ ( T ) . Since these functions and neural networks exhibit Lipschitz continuity, we represent their Lipschitz constants as C σ , C ρ , and C g , respectively. Additionally, we assume that the norm of node features is bounded by ∥ W 1 ∥ ≤ B W 1 , and the norms of model weights by ∥ W 1 ∥ ≤ B W 1 and ∥ W 2 ∥ ≤ B W 2 .

Given the absence of constraints on the tree structures, we manually align the structures of the two trees by incorporating non-sense nodes and edges, as depicted in Figure 10. Initially, the structures of tree 1 and tree 2 are entirely distinct, as illustrated by solid lines. By integrating non-sense branches, we ensure both trees have the same structure, with three branches per node in the first layer and two in the second. These non-sense branches, considered as virtual branches, are purely for theoretical analysis convenience and hold no inherent meaning, similar to the approach in [12]. Consequently, we assume the node features of each non-sense node to be a zero vector. This alignment of tree structures enhances the coherence of subsequent analyses.

Tree 1

Tree 2

True Branch in Computation Tree

Non-sense Virtual Branch

Following, we expand the stability term ∆ :

where R ( W , T L v ) = ρ ( ∑ j ∈N ( v ) g ( T L -1 j ( W ))) . We can further bound the term as:

As we already align the structures of two computation trees by adding non-sense branches to ensure |N ( v ) | = |N ( v 1 ) | = |N ( v 2 ) | , we can merge the two terms in the RHS:

where ∆ L -1 v 1 ,v 2 ,j = ∥T L -1 v 1 ,j ( W ) -T L -1 v 2 ,j ( W ) ∥ 2 .

We now establish a bound on the distance between two computation trees of identical structure by analyzing the node-wise differences from bottom to top. Denote the number of branches (i.e., children) at each l -layer as d l , we simplify this bound as follows:

This bound prioritizes the most influential children of a node to dominate all other branches. By combining Equation 11 with Equation 14, we recursively establish the bound of the distance between two computation trees:

where C 1 = C σ B W 1 and C 2 = C σ B W 2 C ρ C g .

Without loss of generality, we consider the distance between the original computation trees as the distance between the L -layer computation trees rooted at nodes v 1 and v 2 , denoted as ∆ = ∆ L v 1 ,v 2 . This allows us to recursively bound the distance. Given that all x are bounded by ∥ x ∥ 2 ≤ B x , the distance between the node features x v 1 and x v 2 satisfies ∥ x v 1 -x v 2 ∥ 2 ≤ 2 B x by the triangle inequality. Consequently, we can further develop the recursion as follows:

where D l = d l d l -1 ...d 1 .

Assuming that the number of branches (i.e., children) at each l -layer does not exceed the maximum number of branches in the tree, such that d 1 , ..., d L ≤ d . We can further simplify the recursion by:

D.2 Proof for Theorem 3.1

Before proving Theorem 3.1, it is necessary to first establish a more general version of the theorem, as detailed below:

Theorem D.2. Given a set of instances { ( x i , y i ) } n i =1 sampled from the distribution P , and a marginaware prototype classifier f that predicts the class of instances via a distance measure. The risk R ( f ) can be bounded with probability 1 -δ :

where ˆ R ( f ) is the empirical risk, p denotes the number of tokens, C is a constant, B is the bounded norm of x and p , and ρ acts as the margin, serving as a penalty factor in evaluating the distance between computation trees and tokens.

Proof. Given a set of tokens (prototypes), margin-based classification involves using instances to identify the nearest tokens and assigning the labels of these nearest tokens to the target instances. We denote the set of p tokens by { p i } p i =1 , each with a norm bounded by B , and the labels associated with each token by { c i } p i =1 . Given an instance ( x i , y i ) sampled from a distribution P , the classifier can be defined as follows:

where x ∈ R d is a random variable with a norm also bounded by B , consistent with the norms of the tokens.

In this proof, we focus on a binary classification problem with only two classes, {-1 , 1 } . Consequently, the function can be represented as f : R d → {-1 , 1 } . Without loss of generality, this binary classification setting can be readily extended to multi-class scenarios using one-versus-all or one-versus-one strategies [4]. We then define a class of functions as follows:

We denote d c + as the distance to the nearest tokens with the same label y i = c + and d c -as the distance to the closest tokens with the different label y i = c -. If d c + is less than d c -, the instance is correctly classified. Thus, the classification margin is defined as:

where a positive value indicates correct classification. Moreover, we introduce a penalty term to estimate the classification margin, defined as:

where ρ > 0 is a pre-defined margin threshold.

For this classifier, the risk R ( f ) and the corresponding empirical risk ˆ R ( f ) is defined as:

̸

We can establish a Gaussian complexity bound by applying Theorem 7 from [4], which holds with a probability of at least 1 -δ . This is expressed as:

where C represents the Lipschitz constant, and ρ represents the margin. This formulation allows us to explicitly incorporate the prediction margin into the complexity analysis. The term G n ( F ) denotes the Gaussian complexity defined over the function class F , and an empirical Gaussian complexity can be estimated as:

where σ = ( σ 1 , σ 2 , ..., σ n ) are independent standard Gaussian random variables, σ i ∼ N (0 , 1) .

̸

It is important to note that the Gaussian complexity of the function class F can be bounded by aggregating the complexities of all its sub-classes (Theorem 16 from [4]). In our model, the token classifier leverages p tokens simultaneously; hence, it is logical to define sub-classes of F that utilize only two tokens for predictions. We define each sub-class as F ij , which specifically uses tokens p i and p j with differing labels c i = c j . Consequently, the total number of sub-classes is bounded by p · ( p -1) / 2 . This allows us to simplify the complexity bound as follows:

To further simplify the bound, we must derive it for each G n ( F ij ) . It is important to note that F ij can be regarded as a binary classification function class for d i -d j , where the weights are bounded:

Based on Lemma 22 in [4], which establishes that the empirical Gaussian complexity is bounded by a kernel function defined by F ij , we can simplify the empirical Gaussian complexity of each sub-class as follows:

The difference between the Gaussian complexity and empirical Gaussian complexity is estimated to be ϵ with a probability of 2 · exp( -ϵ 2 n 8 ) (Theorem 11 from [4]). We can simplify the risk as follows:

We can readily extend the Theorem D.2 to Theorem 3.1 in the main paper.

Proof. Theorem D.2 establishes bounds on the generalization error of margin-based classifiers using Gaussian complexity. Analogously, vector quantization functions as a margin-based classifier by assigning instances to the nearest tokens in the vocabulary. Specifically, vector quantization utilizes this classifier for clustering, where each cluster center corresponds to a token. We assume each computation tree has a corresponding ground-truth cluster index based on the latent distribution, denoted as P T , where ( T , y ) ∼ P T . Thus, the vector quantization process employed in the main paper converts to a margin-based classification problem, consistent with Theorem D.2. Moreover, we can cancel the term B since the Cosine distance, used to measure the similarity between tree embeddings and tokens, ensures the bounded norm B = 1 .

E Detailed Analysis on Computation Tree Transferability

E.1 Synthetic Dataset

Experimental Setting. We randomly sample node features from a uniform distribution with a dimension of 4 and conduct experiments 100 times using different seeds to report average performance.

20

40

60

80

100

# Blocks

6

8

10

Transferability

Source

Graph 3

Graph 1

20

40

60

80

100

# Blocks

0.7

0.8

0.9

Source

Graph 3

Graph 1

20

40

60

80

100

# Blocks

0.6

0.8

1.0

Source

Graph 3

Graph 1

20

40

60

80

100

# Blocks

6

8

10

Transferability

Source

Graph 1

Graph 2

20

40

60

80

100

# Blocks

0.7

0.8

0.9

Source

Graph 1

Graph 2

20

40

60

80

100

# Blocks

0.6

0.8

1.0

Source

Graph 1

Graph 2

Since the synthetic datasets do not have labels for each node, we employ a graph auto-encoder [39] for self-supervised training. The encoder is a basic 2-layer GCN model with a dimension of 4, and the decoder uses a standard inner product approach, computing the inner product between the embeddings of two nodes to determine their linkage. We set the number of training epochs at 200 and use Adam as the optimizer with a learning rate of 1e-3 and a weight decay of 0. To evaluate transferability, we use the inverse of the Central Moment Discrepancy (CMD) [102], a measure that serves as an indicator of transferability, defined as transferability = 1 CMD . We ensure that the number of blocks in the source and target graphs is the same.

To compute the computation tree similarity (Tree Sim.), we employ the Weisfeiler-Lehman subtree kernel [64], which evaluate similarity between two graphs by considering the subtrees patterns in terms of both structure and features. To match the capabilities of the 2-layer GNN encoder used, we limit the maximum iterations of the WL subtree kernel to two. Additionally, we mitigate the impact of randomness by randomly sampling node features from a uniform distribution and repeating the process 100 times. For evaluating the motif similarity (Motif Sim.), we utilize the graphlet sampling kernel [57] with sampled graphlet size as 5.

Additional Results. We present additional results analyzing transferability on synthetic graphs in Figure 11 and 12. We observe that higher computation tree similarity correlates with better transferability when using G 2 and G 3 as target graphs. However, the impact of motif similarity is marginal. We plan to analyze link-level and graph-level tasks in future work.

E.2 Real-world Dataset

Experimental Setting. We conduct transfer learning to evaluate the correlation between transferability and specific graph patterns on real-world graphs. Similar to synthetic datasets, we utilize the Weisfeiler-Lehman subtree kernel and the graphlet sampling kernel to compute tree similarity and motif similarity, respectively. We include homophily Airport graphs, consisting of USA , Europe , and Brazil [61], where each node represents an airport and edges denote flight connections. Nodes are labeled based on airport connectivity levels. Additionally, we also use heterophily graphs [56] that represent web links from universities such as Cornell , Texas , and Wisconsin , where nodes are web pages and edges are hyperlinks. The objective is to classify nodes into five categories: categories, student, project, course, staff, and faculty. In our analysis of real graphs, we consider two settings: (1) use randomly sampled node features and (2) use raw node features. This approach will offer more comprehensive insights, as node features are also related to homophily and heterophily.

The experimental settings are detailed as follows. We evaluate the transfer learning performance using a basic 2-layer GCN model with ReLU activation, running the experiments 20 times to report the average results. We pre-train the model on the source graph with 60% of nodes randomly selected and subsequently fine-tune it on the target graph with 10% of nodes randomly selected. The hidden

dimension is set to 64, and we use the AdamW optimizer with a weight decay of 1e-6. The pretraining learning rate is set at 1e-3 for all settings with 500 pre-train epochs, while the fine-tuning learning rate is set at 1e-2 for heterophily graphs and 5e-4 for homophily graphs. In the random feature setting, we sample node features from a uniform distribution across 64 dimensions. For computing graph similarity with randomly sampled node features, we conduct the experiments 100 times using different seeds. The graphlet kernel samples motifs 10,000 times, and the maximum motif size is set to 5. The maximum iteration of the subtree WL kernel is limited to 2, aligning with the number of GNN layers.

Additional Results. See Table 9 for transfer learning performance on graphs with random features. Even though the node features are randomly initialized, we still observe that a high tree similarity correlates with improved transferability.

F Experimental Setup

F.1 GNN Encoder

We employ a GraphSAGE-like architecture to encode node and edge features in a graph G = ( V , E ) , where node features are represented as X ∈ R |V|× d f and edge features as E ∈ R |E|× d e . Considering a GNN with L layers, the ( l +1) -th layer node embedding for node v is given by:

where H ( l ) v represents the node embedding at the l -th layer, E u,v denotes the edge features between nodes u and v , and W 1 and W 2 are the learnable matrices. The function φ , used to align feature dimensions, is chosen as the identity function, Id( · ) , in this study. While we utilize the basic GraphSAGE-like framework as the encoder, alternative, more advanced encoders such as graph attention networks [78] or other expressive GNNs [52] could potentially enhance model performance.

F.2 Dataset

Dataset Statistics. We utilize nine datasets from various domains and tasks, as detailed in Table 10. We follow the preprocessing method described in [45], using the Sentence Transformer [60] to convert raw textual descriptions of nodes and edges into 768-dimensional features. It should be noted that for knowledge graphs (KGs), we do not transform edge textual information into edge features, as the textual information already provides sufficient knowledge for KG completion.

Dataset Splitting. For Cora and PubMed, we follow the common split setting with 20 labeled nodes per class for training, utilizing a predefined 10 splits with different seeds to report average performance. For WikiCS, we also employ the standard split, reporting average accuracy across 20 different training splits, each with 20 random seeds, and using 5% of nodes in each class for training. For Arxiv, HIV, and PCBA, we follow the official splits, conducting experiments 10 times

with random seeds to determine average accuracy. For WN18RR and FB15K237, we follow the splits outlined in Liu et al. [45]. Specifically, for FB15K237, the training set comprises 272,115 edges, the validation set 17,535 edges, and the test set 20,466 edges; for WN18RR, the numbers are 86,835, 3,034, and 3,134, respectively. We repeat each experiment 10 times with random seeds and report the average accuracy.

F.3 Baseline

We compare GFT against a broad spectrum of baselines, encompassing supervised GNNs, selfsupervised GNNs, graph few-shot learning, and graph foundation models.

Supervised GNNs/MLP. The supervised approaches include a basic MLP (Linear), GCN [40], GAT [78], and GIN [94].

Self-supervised GNNs. Our analysis also covers self-supervised methods for graph learning. DGI [79] utilizes contrastive learning between graph summaries and node patches. BGRL [74] employs bootstrapping to predict the same node in different views. GraphMAE [26] reconstructs node features using structural information. GIANT [11] combines language models with graph neural networks in a self-supervised fashion, achieving state-of-the-art performance.

Graph Few-shot Learning. To assess performance in few-shot learning scenarios, we evaluate GFT alongside methods such as GPN [15], TENT [83], GLITTER [82], and TLP [71]. Experimental results are detailed in Appendix H.

Graph Foundation Models. We include two primary baselines: Prodigy [30], which specializes in pre-training for in-context learning, although it is not applicable in standard pre-training and fine-tuning scenarios. For this model, we pre-train on MAG240M and evaluate performance on Arxiv, and on Wiki for FB15K237. OFA [45], in contrast, utilizes language models to align the feature spaces of different graphs and introduces a prompt graph to align task spaces, trained in a supervised manner.

F.4 Hyper-parameter Setting

Baselines. For the baseline methods, we follow the hyper-parameters reported in [30, 45]. If specific hyper-parameters for a task are not reported, we set the learning rate to 5e-3 for Cora, PubMed, WikiCS, WN18RR, and HIV, and to 1e-4 for Arxiv, FB15K237, and PCBA. We configure all GNN encoders with two layers, a hidden dimension of 768, and incorporate batch normalization and ReLU activation. AdamW is used as the optimizer with a weight decay of 1e-5. For methods that utilize attention mechanisms, we specify four attention heads.

Pre-training of GFT. We configure our model with two layers, each having a dimension of 768, and use ReLU activation complemented by batch normalization. In vector quantization, we set the number of tokens to 128 with each token dimension at 768. We empirically determine the weights for different losses as β 1 = 10 , β 2 = 100 , β 3 = 1 , and β 4 = 0 . 01 . Additionally, we set the weight for

the orthogonal regularizer, λ , to 1. AdamW is utilized as the optimizer with a learning rate of 1e-4 and a weight decay of 1e-5. The pre-training phase lasts for 25 epochs with a batch size of 1024. For data augmentation, we implement an edge drop rate and a node feature drop rate, both set at 0.2. For topology reconstruction, we selectively reconstruct 10% of links and choose an equivalent number of negative samples. The sampling factor γ for semantic reconstruction is fixed at 1.

Fine-tuning of GFT. We detail the hyper-parameters for different datasets in Table 11. λ proto and λ lin represent the weights of the losses for the prototype classifier and the linear classifier, respectively.

F.5 Running environment

We utilize an NVIDIA A40 with 48GB GPU memory for all experiments. Both the pre-training and fine-tuning phases can be conducted on a single Nvidia GeForce RTX 3090 with 24GB memory.

G Pre-training and Fine-tuning Results with std .

We report the average results of the pre-training and fine-tuning settings in the main paper. The model results along with the standard deviations are presented in Table 12.

H Additional Few-shot Learning Results

We present the extended few-shot learning performance across multiple tables: Table 13, 14, 15, 16, 17, 18, and 19. In each run, we sample 20 few-shot tasks to mitigate the impact of randomness. The baselines consist of graph foundation models such as Prodigy [30] and OFA [45], alongside few-shot learning methods including GPN [15], TENT [83], GLITTER [82], and TLP [71]. In terms of graph foundation models, we compare GFT to OFA across all datasets and to Prodigy on the Arxiv and FB15K237 datasets only, as Prodigy's application is limited to these datasets by its in-context training strategy. GFT not only significantly enhances performance over Prodigy and OFA but also surpasses a broad range of specialized few-shot learning methods. Furthermore, as the number of fine-tuning instances per class increases, there is a marked improvement in model performance, demonstrating

significant adaptability to target tasks. Notably, even with extremely limited training instances, the model substantially outperforms the baselines, showcasing rapid adaptation capabilities.

I Complete Ablation Study

I.1 Pre-training Datasets

The complete results detailing the impact of different pre-training datasets are presented in Table 20, which serves as a complement to Table 5. We explore three variants: (1) pre-training on all

datasets, (2) pre-training solely on the target dataset, and (3) pre-training on the remaining datasets. Our observations suggest that using only the target graph can still achieve desirable performance, as it provides graph-specific information without spurious noise. More importantly, performance improves significantly when the target graph is excluded and the remaining datasets are utilized. We hypothesize that observing more computation trees generally enhances model performance. Even without the target graph, the presence of numerous computation trees shared across various domains provides sufficient information. Moreover, using all datasets typically yields the best performance, as it offers a more comprehensive approximation of the computation tree distribution.

I.2 Pre-training Tasks

Table 21 presents complete results of model performance across different pre-training tasks, serving as a complement to Table 4. The observations align fully with those in Table 4, demonstrating that all reconstruction tasks enhance model performance compared to models without pre-training. Optimal performance is achieved when three tasks are jointly optimized.

I.3 Strategies for Enhancing Tree Vocabulary

Table 22 presents the ablation study on strategies for enhancing the quality of tree vocabulary, as described in Section 3.1. As previously stated, both the comprehension and expressiveness of the tree vocabulary are critical properties for its effectiveness, achieved through augmentation (aug.) and an

orthogonal regularizer (ortho. reg.), respectively. We note that removing any component results in a degradation in model performance.

I.4 Fine-tuning Tasks

We analyze the impact of different computation tree classification tasks for fine-tuning. The complete results are presented in Table 23, which complements Table 4. Specifically, GFT employs both a linear classifier and a prototype classifier to utilize information from various levels of the tree. The prototype classifier excels in node-level tasks, while the linear classifier performs better in the other two tasks. However, combining these two methods yields the best performance.

I.5 Tree Vocabulary

Table 24 presents a detailed analysis of model performance with different numbers of tokens and without utilizing the tree vocabulary, complementing Table 6. Although increasing the number of codes can enhance performance to a certain extent, it is not necessarily effective in all scenarios. Specifically, in only four out of eight scenarios, the maximum number of codes (512 tokens) yields the best results. This observation is consistent with our theoretical analysis, suggesting that more codes may increase the upper bound of generalization error, potentially due to overfitting risks. Furthermore, this phenomenon might also be attributed to the limited diversity of datasets; the eight utilized datasets originate from only four domains (citation, web link, knowledge graphs, and molecules). For these

domains, a smaller number of codes may suffice. We hypothesize expanding the number of domains might necessitate more codes; we intend to explore in future work. Regarding the variant that does not use vocabulary, we bypass vocabulary training during pre-training, and directly append a linear classifier behind the GNN for classification during fine-tuning. The results indicate that using the vocabulary significantly enhances model performance, particularly in link- and graph-level tasks, aligning with our theoretical considerations regarding the generalizability of tree tokens.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly introduce the contribution of the paper in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Section 5

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide proofs for each theorem in the paper and clearly present the assumptions. We also discuss how such assumptions can be satisfied. See Appendix D.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the code in the supplementary and also comprehensively discuss the experimental setup in Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the anonymized code in the supplementary.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We present the error bar in the Appendix G, H, and I.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the clear experimental setting in Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We carefully read and follow the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See Section 5.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We clearly cite the papers if we follow their settings.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: It does not available for our work.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: It does not available for our work.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: It does not available for our work.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper proposes GFT, a graph foundation model based on computation tree. Extensive experiments and theoretical analyses are conducted to show the effectiveness of GFT across diverse tasks.

STRENGTHS:
1. Addresses a significant challenge (identifying transferrable patterns) in graph foundation models.
2. Includes both theoretical and empirical evaluations on synthetic and real-world graphs.
3. Conducts extensive experiments including multiple graph tasks and domains.

WEAKNESSES:
1. The experiments are unconvincing (and most likely unfair): Only a few outdated supervised baselines are used, and their results are weaker than expected. For instance, the GCN's performance on Cora should surpass 80, and the results for GIANT on Arxiv are notably low. Are all baselines using SentenceBERT embedding? If not and the raw features are used, the improvements might be contributed to better text encoders.
2. Assumes a common feature space across different graphs, limiting its applicability to non-textual graphs.
3. The method is complex, involving a broad hyperparameter space, including computation tree parameters and the betas in Equation 3.
4. Certain sections of the paper are unclear, such as the sampling/construction of the computation trees and the interpretation of the y-axis in Figure 5.
5. The distinction between the computation tree and subgraph is not clearly defined.
6. The analysis of time complexity should consider the size of subgraphs, similar to GraphSAGE. Furthermore, the discussion should focus on actual wall time to better demonstrate the efficiency of the GFT.

QUESTIONS:
How are baseline results obtained? What kind of features are leveraged?

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the concept of the transferable token in graph foundation models. Specifically, the paper proposes to use the computation tree as the transferable token for graph learning and prove its efficiency from both theoretical and empirical perspectives. Then, the paper proposes GFT model, GFT first pretrain a graph tokenizer using vector quantization to tokenize text-attributed graphs across different domains through a computation tree. Next, it is fine-tuned on downstream tasks. The model shows great results on generalization ability, especially on few-shot and zero-shot experiments.\n\nSTRENGTHS:\n1. Investigating transferable tokens in graph learning is important and fundamental to the community. This paper is the first one that demonstrates the potential existence of the transferable token in text-attributed graphs. \n2. The paper is well-structured with solid theoretical and experimental results to demonstrate the advantages of the proposed model.\n\nWEAKNESSES:\n1. In Theorem 2.2, the distance bound is associated with the distance between the $j$-th neighbor of node $v_1$ and $v_2$. How to define the $j$-th neighbor related to two nodes? What if the two nodes have different numbers of neighbors?\n2. In Theorem 2.2, the distance bound is related to the distance of the feature between two nodes (as well as the distance between node neighbors, as it is defined recursively). However, in the real world, the distance of the feature itself can be unbounded, which makes Theorem 2.2 only applicable when features are close and less useful in real scenarios (Even in text-attributed graphs, sentence embedding from different domains can still be too diverse). \n3. For Section 3.1, I am not sure if I understand it wrongly or if I missed something. There are a few parts I think it is not true. For Equation 2, should loss compute between $z_i$ and $c_i$, instead of $q_i$? Otherwise, it does not make sense to me. Meanwhile, what is the $\\delta_j$, I cannot find its definition in the paper. Please correct me if I am wrong.\n\n4. Although the authors did extensive ablation studies, there is one I am particularly interested in about cannot find in the paper. How well can be model be if the pretraining datasets are totally from different domains of the downstream tasks? For example, what is the performance if the model is pre-trained on molecular datasets but tested on citation networks or vice versa?\n\nQUESTIONS:\n1. I am curious about Table 23. Since the graph tokenizer needs to tokenize both structural and textual information. The token size should be much larger than 512 even within a single domain from my sense given the abundance of both the graph structure and textural information. Could authors explain more about it? Have authors explore what are these tokens learned exactly?\n\n2. The approach in this paper is somewhat similar to the VQGraph [1]. It is worth discussing the relation and difference between the two methods.\n\n3. The author claims the proposed method does not need to extract subgraphs. I am wondering how is the method deal with the case like the graph is too large to fit into one GPU? \n\n\n[1] Yang et al., VQGRAPH: RETHINKING GRAPH REPRESENTATION SPACE FOR BRIDGING GNNS AND MLPS, ICLR 2024.\n\nLIMITATIONS:\nThere are no major limitations from my perspective. I encourage the authors to further explore the proposed methods, like using more expressive GNNs or pre-train on large-scale datasets, as discussed in the limitation section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a novel computation tree method to improve the transferability between the pre-train model and downstream tasks. This paper rethinks the transferable pattern in graphs as computation trees and validate their transferability both empirically and theoretically. The proposed GFT leverages computation tree reconstruction to acquire general graph knowledge from cross-domain datasets and uses computation tree classification to facilitate adaptation to various target tasks Comparing with other previous methods, the proposed GFT can improve model generalization and reduce the risk of negative transfer, which is suitable for the cross-domain and cross-task situation.\n\nSTRENGTHS:\na. Originality: The computation tree has been proposed in several previous works that have been adequately cited in this paper. The author reconstructs the computation tree in a new way and combines it with Vector Quantization method, effective and preventing over-fitting. b. Quality: The paper is technically sound, and the loss function proposed is comprehensive, enabling a deep understanding of the structural and semantical attribute of computation trees. The experiments in the manuscript are very comprehensive and effectively demonstrate the superiority of GFT, and important claims are well supported by theoretical analysis. c. Clarity: The manuscript is well organized and clearly written, although some descriptions could be clearer. d. Significance: The experimental results advance the SOTA methods, and the computation tree method proposed is easy for other researchers to use.\n\nWEAKNESSES:\nIt is not very clear how this work differs from previous contributions. The explanation of the superiority of computation trees over subgraphs is unconvincing because the computation tree in this paper is very different from some proposed computation trees (e.g. junction tree, H-tree, etc.) but more like the subgraph. It would be better if the author could provide detailed comparison and analysis between the computation tree and the subgraph; The quality is a bit limited. The Lfeat is to minimize the discrepancy between the local structure of a node and its feature, but is there any inherent relationship between the two? Besides, the method might not be able to deal with link prediction in a good manner. In a graph, if node va and vb are isomorphic while the links (va, vc) and (vb, vc) are not isomorphic, the vanilla GNN with the same node representations va and vb gives the same prediction to links (va, vc) and (vb, vc), but GFT does not address this issue; The clarity should be improved. The explanation for Ltree should be detailed. For example, what are decoders used for q projection? In line 269, the definition of R(f) is not given; The significance of the paper is slightly limited. It might be difficult to extend the GFT to zero-shot scenarios because the fine-tune process is necessary for GFT but it will be excluded in zero-shot scenarios.\n\nQUESTIONS:\n1. In Appendix C.5, the paper argues that the subgraph method incurs additional time and memory costs. Why can the proposed computation tree avoid these problems? \n\n2. The Lfeat is to minimize the discrepancy between the local structure of a node and its feature. Why do you use Lfeat and is there any intrinsic connections between the local structure of a node and its feature? \n\n3. For Ltree, what are decoders used for q projection? \n\n4. In a graph, if node va and vb are isomorphic while the links (va, vc) and (vb, vc) are not isomorphic, the vanilla GNN with the same node representations va and vb gives the same prediction to links (va, vc) and (vb, vc). How does GFT address this issue in link prediction? \n\n5. What is the definition of R(f)? \n\n6. From the experimental results, GFT outperforms other methods. Is this due to the computation tree or the more comprehensive loss function? Would the overall performance be worse if the computation trees were replaced with subgraphs? 7. Is it possible for GFT to be applicable in zero-shot scenarios?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new graph foundation model based on the tree structure, which is called GFT. GFT leverages computation trees to define tokens within the transferable vocabulary, which improves model generalization and reduce the risk of negative transfers.  Comprehensive experiments and theoretical analyses are provided to demonstrate the effectiveness of GFT across diverse tasks and domains in graph learning. Overall, this is a good work with good motivation, a novel method with theoretical support, comprehensive experiments, and good writing.\n\nSTRENGTHS:\n+ The paper is well motivated by the fact that current graph learning models lack identification of a vocabulary that can encode transferable patterns shared among different tasks and domains. Filling the gap is challenging and meaningful. \n \n+ The proposed GFT model is novel. Unlike existing models, it introduces computation tree as transferable patterns and encodes general knowledge of graph into a tree vocabulary, which is new. It aims to improve the model generalization and reduce the risk of negative transfers.  The theoretical analyses are also provided to support the design of GFT model, which is solid. \n\n+ Extensive experiments over different graph learning tasks (e.g., node classification, link prediction, graph classification) are conducted across many datasets. The model outperforms SOTA baseline methods. As a graph foundation model, it is good to see experiments over cross-domain and cross-task datasets. Many more analytical experiments (including Appendix) are also provided, which is impressive. \n\n+ The paper presentation is good. The organization is clear and easy to follow.\n\nWEAKNESSES:\n-  The proposed model assumes tree structure as transferable patterns. Besides the examples (i.e., basic blocks in Fig. 2) shown in the paper, is there any other structures transferable? It is not clear whether these patterns are transferred during the model training or not. How to demonstrate this?\n-  As a graph foundation model, besides general and few-shot tasks, it would be better to see experiments on zero-shot task.  Can the proposed model be applied to this scenario and what is performance compared to baseline methods?\n\nQUESTIONS:\nPlease see the weaknesses.\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}