AnyGraph: Graph Foundation Model in the Wild

## Abstract

Abstract. The growing ubiquity of relational data structured as graphs has underscored the need for graph learning models with exceptional generalization capabilities. However, current approaches often struggle to effectively extract generalizable insights, frequently requiring extensive fine-tuning and limiting their versatility. Graph foundation models offer a transformative solution, with the potential to learn robust, generalizable representations from graph data. This enables more effective and adaptable applications across a wide spectrum of tasks and domains. In this work, we investigate a unified graph model, AnyGraph, designed to handle key challenges: i) Structure Heterogenity . Addressing distribution shift in graph structural information; ii) Feature Heterogenity . Handling diverse feature representation spaces across graph datasets; iii) Fast Adaptation . Efficiently adapting the model to new graph domains; iv) Scaling Law Emergence . Enabling the model to exhibit scaling law behavior, where its performance scales favorably with the amount of data and parameter sizes. To tackle these critical challenges, we build the AnyGraph upon a Graph Mixture-of-Experts (MoE) architecture. This approach empowers the model to effectively manage both the in-domain and cross-domain distribution shift concerning structure-level and feature-level heterogeneity. Furthermore, a lightweight graph expert routing mechanism is proposed to facilitate AnyGraph’s fast adaptability to new data and domains. Our extensive experiments on diverse 38 graph datasets have demonstrated the strong zero-shot learning performance of AnyGraph across diverse graph domains with significant distribution shift. Furthermore, we have validated the model’s fast adaptation ability and scaling law emergence, showcasing its versatility.

## 1. Introduction

The growing ubiquity of relational data in the form of graphs has underscored the pressing need for advanced graph learning models that excel at generalization (; ). As real-world applications of graph-structured data continue to proliferate across diverse domains, including social networks, academic networks, transportation systems, and biological networks, the ability of graph learning models to effectively handle distribution shifts and adapt to new graph domains has become increasingly crucial (; ; ; ). Developing models with robust zero-shot learning performance and fast adaptation capabilities can unlock transformative opportunities for leveraging the rich insights encoded within graph data.

The field of graph learning has seen significant advancements in recent years, largely driven by the power of Graph Neural Networks (GNNs) (; ; ). However, the current state-of-the-art models often fall short when it comes to truly generalizable performance. Existing approaches tend to be heavily reliant on arduous fine-tuning processes, making them ill-equipped to handle the diverse array of graph structures and distributions encountered in real-world applications. This inability to adapt swiftly and seamlessly to novel graph domains poses a critical barrier to the widespread adoption of graph learning technologies. Therefore, addressing this challenge is of paramount importance if we are to fully harness the transformative potential of graph-based insights.

Inspired by the principles that have driven the development of successful foundation models in understanding vision and language data (; ), the concept of a versatile graph foundation model holds immense potential to unlock new frontiers in graph learning. By learning rich, transferable representations from diverse graph-structured data, such a model can be efficiently adapted to a wide array of graph domains and tasks. However, building an effective and adaptive graph foundation model is not a trivial endeavor. Several key challenges must be overcome, including:

(i) Structure Heterogeneity. The development of versatile graph models faces the challenge of accommodating diverse structural properties and data distributions in various graph datasets. For instance, graphs can exhibit substantial heterogeneity in node degree distributions, ranging from homogeneous to highly skewed patterns. Similarly, graph structures can vary greatly in complexity, from simple topologies to intricate, hierarchical arrangements. These structural variations can significantly impact the performance and generalization of graph learning algorithms. Effectively addressing this diversity is critical for developing unified models that can thrive across a wide range of graph-structured data.

(ii) Feature Heterogeneity. Graphs exhibit substantial heterogeneity in their node and edge features, which can span categorical attributes, continuous numerical data, and multi-modal content. Furthermore, the dimensionality and semantics of these features often vary dramatically across different graph domains. For instance, a social interaction graph may include textual content and demographic information associated with its nodes, while a molecular graph may feature atomic compositions and bond types. Effectively handling this feature heterogeneity is crucial for building a versatile graph model capable of generalizing across diverse graph domains.

(iii) Fast Adaptation for Broad Applicability. A key capability for effective graph foundation models is the ability to efficiently adapt to new graph dataset and domains. Rather than requiring extensive retraining or fine-tuning, the ideal model should be able to quickly adjust its parameters and learning strategies to handle the structural and distributional characteristics of previously unseen graph datasets. By seamlessly generalizing and performing well across a diverse range of real-world scenarios – from user behavior graphs to transportation networks and biological systems – these adaptable models can unlock transformative insights across an ever-expanding universe of graph-structured data.

(iv) Scaling Laws for Transformative Graph Capabilities. A key characteristic of successful foundation models in domains like CV () and NLP () is their ability to exhibit scaling laws - where performance systematically improves as the model size or training dataset increases. By harnessing this emergent scaling phenomenon, graph foundation models can unlock unprecedented levels of capability and generalization, far surpassing the limitations of fixed-capacity architectures. As the size of graph datasets and model complexity grow, these scaling-aware designs can continue delivering transformative performance gains.

Figure: (a) Generalizability of AnyGraph.
Refer to caption: x1.png

The Presented Work. To tackle the above challenges, our AnyGraph model is built upon a Mixture-of-Experts (MoE) architecture, which allows for effective handling of both the in-domain and cross-domain distribution shift in structure-level and feature-level heterogeneity. The proposed graph MoE paradigm empowers AnyGraph to learn a diverse ensemble of graph experts, each tailored to specific structural characteristics. This enables the model to effectively manage the distribution shift in graph topologies, ranging from homogeneous to highly skewed degree distributions, as well as handle graphs with varying levels of complexity. Furthermore, the MoE architecture of AnyGraph facilitates fast adaptation of the graph model. Rather than relying on a single, fixed-capacity model, the Graph MoE learns an ensemble of specialized expert networks, each tailored to capture distinct structural and feature-level characteristics of graph data. The lightweight graph expert routing mechanism allows AnyGraph to quickly identify and activate the most relevant experts for a given input graph, without requiring extensive retraining or fine-tuning across the entire model. The key findings of this work can be summarized as:

- •
Methodology Design Motivations of AnyGraph. Current large graph models (; ; ) often struggle when faced with the substantial heterogeneity found in real-world graph data. This is especially challenging when it comes to feature-level heterogeneity. These fixed-capacity models may encounter interference between different types of graph datasets, and can sometimes overfit to new data, leading to catastrophic forgetting. To address these challenges, the proposed Mixture-of-Experts (MoE) architecture for graph models was designed with a focus on adaptability. This new paradigm empowers the model to flexibly adjust to the nuances of diverse graph datasets, dynamically selecting the most appropriate experts to learn distinct patterns.
- •
Stronger Gernealiation Capacities of AnyGraph. Through our extensive experiments, the proposed AnyGraph model with the graph MoE framework has demonstrated strong generalization capacities across a wide range of graph tasks and domains. The experimental results showcase the AnyGraph’s ability to outperform existing graph models in terms of both predictive performance and robustness to distribution shift.
- •
Fast Adapability of AnyGraph. Our innovative dynamic expert selection mechanism enhances AnyGraph’s ability to swiftly adapt to new graph domains. By dynamically routing inputs through relevant experts, AnyGraph can quickly activate the specialized networks best suited for the task. This strong adaptation sets AnyGraph apart from baselines. Evaluation shows its superiority through rapid convergence and exceptional performance, further justifying its cross-domain versatility.
- •
The Scaling Law of AnyGraph. Our experiments reveal that AnyGraph’s performance follows the scaling law, where the model continues to improve as model size and training data increase. Additionally, AnyGraph exhibits emergent abilities, where its generalization capabilities see sudden significant improvements with further scaling. This critical scaling law property has been largely overlooked in prior investigations, but it underscores the immense value that AnyGraph derives from its scaling-driven enhancements to generalization performance.

## 2. Preliminaries

Graph-Structured Data. A graph $\mathcal{G}$ consists of a set of nodes $\mathcal{V}=\{v_{i}\}$ and a set of edges $\mathcal{E}=\{(v_{i},v_{j})\}$. In many cases, each node $v_{i}$ is associated with a feature vector $\textbf{f}_{i}\in\mathbb{R}^{d_{0}}$. To efficiently utilize such graph-structured data, the link information is typically recorded using an adjacency matrix $\textbf{A}\in\mathbb{R}^{|\mathcal{V}|\times|\mathcal{V}|}$. Each element $a_{i,j}$ of A is either 1 or 0, inddicating whether there is an edge from node $v_{i}$ to $v_{j}$. Additionally, the feature vectors of the nodes are usually represented by a feature matrix $\textbf{F}\in\mathbb{R}^{|\mathcal{V}|\times d_{0}}$, where each row corresponds to a node’s feature vector. The primary goal of learning from such graph-structured data is to generate embeddings for the graph elements, typically nodes, that effectively capture both the structural and feature-based information of the graph.

Graph Foundation Models (GFMs).
The essence of GFMs lies in their strong generalization capabilities. Specifically, a graph foundation model should be able to handle unseen graph data that exhibits significant discrepancies from its training graph datasets. These discrepancies may include differences in feature spaces, as well as variations in node and edge semantics across datasets. Formally, let’s denote the training graphs as $\mathbb{S}={\mathcal{G}_{s}}$, where each graph $\mathcal{G}_{s}$ is associated with a label set $\mathcal{Y}_{s}$. Similarly, the set of test graphs is denoted as $\mathbb{T}={\mathcal{G}_{t}}$, with labels $\mathcal{Y}_{t}$. With a differentiable training objective $\mathcal{L}$ and an evaluation criterion $\mathcal{C}$ to measure the prediction accuracy of downstream tasks, building a graph foundation model $f_{\mathbf{\Theta}}$ with trainable parameters $\mathbf{\Theta}$ can be formalized as follows:

$$ (1) $\displaystyle\mathop{\arg\max}_{f,\mathcal{L}}\sum_{\mathcal{G}_{t}}\mathcal{C }\left(f_{\mathbf{\Theta}}(\mathcal{G}_{t}),\mathcal{Y}_{t}\right),\leavevmode \nobreak\ \mathbf{\Theta}=\mathop{\arg\min}_{\mathbf{\Theta}}\sum_{\mathcal{G} _{s}}\mathcal{L}\left(f_{\mathbf{\Theta}}(\mathcal{G}_{s}),\mathcal{Y}_{s}\right)$ $$

The above formulation reveals that the key to building graph foundation models are: i) the model architecture design ($f$), which must have the capacity to encode diverse feature spaces and structural patterns, and ii) the model training process ($\mathcal{L}$), which must effectively traverse such diverse data to find an optimal solution $\mathbf{\Theta}$ for the model $f$.
In light of this, our AnyGraph employs a mixture-of-experts architecture with an automated expert routing method, to seamlessly integrate powerful prediction models for highly diverse graph data. AnyGraph is extensively trained on graphs from various applications using multiple featuring methods, with a graph augmentation technique to further enhance data diversity.

## 3. Methodology

The proposed AnyGraph framework aims to address both cross-domain and in-domain heterogeneity in graph structures and node features, while enabling fast adaptation to new data. The overall framework of AnyGraph is depicted in Fig. 2.

Figure: Figure 2. The proposed graph Mixture-of-Experts (MoE) paradigm enables AnyGraph to learn a diverse ensemble of graph experts, each tailored to specific structural characteristics. The lightweight expert routing mechanism allows AnyGraph to quickly identify and activate the most relevant experts for a given input graph, without extensive retraining or fine-tuning.
Refer to caption: x3.png

### 3.1. MoE Architecture of AnyGraph

#### 3.1.1. Addressing Cross-domain Graph Heterogeneity

To model heterogeneous graph patterns across different domains, AnyGraph employs a MoE architecture that consists of multiple graph expert models, each responsible for handling graphs with specific characteristics. An automated routing algorithm is designed to assign input graph data to the most competent expert model for training and prediction.
Specifically, the AnyGraph framework can be denoted as $\mathcal{M}=(f_{\mathbf{\Theta}_{1}},f_{\mathbf{\Theta}_{2}},\cdots,f_{\mathbf
{\Theta}_{K}},\psi)$, where $K$ denotes the number of experts. For an input graph $\mathcal{G}$, the routing algorithm $\psi$ firstly identifies the most competent expert model, and the corresponding model is then used for predicting the graph data, as:

$$ (2) $\displaystyle\hat{y}_{i,j}=\hat{\textbf{e}}^{\top}_{i}\hat{\textbf{e}}_{j}, \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \hat{\textbf{E} }=f_{\mathbf{\Theta}_{k}}(\mathcal{G}),\leavevmode\nobreak\ \leavevmode \nobreak\ \leavevmode\nobreak\ k=\psi(\mathcal{G})$ $$

where each expert model $f_{\mathbf{\Theta}_{k}}$ can be viewed as a projection from the graph space to a node embedding space with uniquely trained parameters $\mathbf{\Theta}_{k}$. And $\hat{y}_{i,j}$ represents the dot-product-based prediction of whether the entity $v_{i}$ should be related to the entity $v_{j}$. Here, $v_{i}$ and $v_{j}$ could be vanilla graph nodes, class labels, or graph labels.

#### 3.1.2. Graph Expert Routing Mechanism

Inspired by the effectiveness of graph self-supervised learning tasks (), we propose measuring the competence of expert models on specific graph datasets using the models’ self-supervised learning loss values. Specifically, for an input graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$, the routing mechanism $\psi$ calculates the dot-product-based relatedness scores for some positive edges $(v_{c_{1}},v_{p_{1}}),\cdots,(v_{c_{S}},v_{p_{S}})\in\mathcal{E}$ and analogously calculates the relatedness scores for some sampled negative node pairs $(v_{c_{1}},v_{n_{1}}),\cdots,(v_{c_{S}},v_{n_{S}})\notin\mathcal{E}$. The following score difference is then calculated as the competence indicator $\varphi_{k}$ for the $k$-th expert model regarding the input graph $\mathcal{G}$:

$$ (3) $\displaystyle\varphi_{k}=\frac{1}{S}\cdot\sum_{s=1}^{S}\sigma(\hat{\textbf{e}} _{c_{s}}^{\top}\hat{\textbf{e}}_{p_{s}}-\hat{\textbf{e}}_{c_{s}}^{\top}\hat{ \textbf{e}}_{n_{s}})$ $$

where $\sigma(\cdot)$ represents the sigmoid activation function, which constrains the competence score to the range of (0, 1). This prevents the few outlier cases where the non-activated score difference is excessively large or small, which could otherwise distort the results.

Training Frequency Regularization.
Although being empirically accurate in measuring models’ competence using the aforementioned self-supervised task loss, this routing mechanism tends to result in a winner-takes-all sub-optimal situation. In this scenario, a single model, or very few models, is predominantly selected as the most competent expert and is used to handle almost all input graphs. These expert models generally receive more or better training samples in the early training stages, giving them an advantage over other experts. Consequently, subsequent training samples are also mostly assigned to them due to their performance advantages, ultimately causing other experts to remain largely untrained.

This situation contradicts our motivation of using different expert models to learn different subsets of graph modeling knowledge. To address this, we propose a training frequency regularization approach that recalibrates the competence score as follows:

$$ (4) $\displaystyle\varphi^{\prime}_{k}=\varphi_{k}\cdot\left((1-\frac{m_{k}}{\sum_{ k^{\prime}}m_{k^{\prime}}})\cdot\rho+1.0-\frac{\rho}{2}\right)$ $$

where $\varphi^{\prime}k$ represents the recalibrated routing score for the $k$-th expert model $f{\mathbf{\Theta}_{k}}$, based on the number of previously assigned training steps $m_{k}$ for $k=1,\cdots,K$. The notation $\rho$ refers to a hyperparameter for the recalibration scale. A larger $\rho$ results in a greater adjustment to the competence score $\varphi_{k}$. With this additional step, the expert routing mechanism will assign more training instances to the less trained expert models, thereby preventing the aforementioned winner-takes-all situation.

#### 3.1.3. Fast Adaptation Capabilities of AnyGraph

With the aforementioned MoE architecture and routing mechanism, the training and inference process of AnyGraph is conducted by only one expert model. This approach consumes only $1/K$ of the computational and memory resources required for predictions and optimization, compared to other non-MoE graph foundation models based on complex networks like transformers. This endows AnyGraph with the advantage of fast adaptation when dealing with new datasets.

### 3.2. Adaptive and Efficient Graph Experts

#### 3.2.1. Addressing In-domain Graph Heterogeneity

To handle graph data with different adjacency and feature dimensionalities, the expert models of our AnyGraph employ a structure and feature unification process. Adjacency matrices and node features of varying sizes are both mapped into initial node embeddings of fixed dimensionality using a unified mapping function. Inspired by the effectiveness of singular value decomposition (SVD) in extracting important latent features, we utilize SVD for this unified mapping process as follows:

$$ $\displaystyle\textbf{U}_{\textbf{A}},\Lambda_{\textbf{A}},\textbf{V}_{\textbf{ A}}=\text{SVD}(\tilde{\textbf{A}})\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode \nobreak\ \textbf{U}_{\textbf{F}},\Lambda_{\textbf{F}},\textbf{V}_{\textbf{F}} =\text{SVD}(\textbf{F})$ (5) $\displaystyle\textbf{E}_{0}=$ $\displaystyle\text{LayerNorm}\left(\textbf{U}_{\textbf{A}}\sqrt{\Lambda_{ \textbf{A}}}+\textbf{V}_{\textbf{A}}\sqrt{\Lambda_{\textbf{A}}}+\text{Flip}( \textbf{U}_{\textbf{F}}\sqrt{\Lambda_{\textbf{F}}})\right)$ $$

Here, $\textbf{U}_{\textbf{A}},\textbf{U}_{\textbf{A}}\in\mathbb{R}^{|\mathcal{V}|
\times d}$ and $\textbf{U}_{\textbf{F}}\in\mathbb{R}^{|\mathcal{V}|\times d},\textbf{V}_{
\textbf{F}}\in\mathbb{R}^{d_{0}\times d}$ refer to the $d$-dimensional features obtained through SVD of the Laplacian-normalized adjacency matrix $\tilde{\textbf{A}}$ () and the node feature matrix F, respectively. If the dimensionality of $\tilde{\textbf{A}}$ or F is less than $d$, SVD uses a smaller rank $d^{\prime}$ equal to the smallest dimensionality of $\tilde{\textbf{A}}/\textbf{F}$, and the remaining dimensions are padded with zeros up to $d$.

Due to the nature of SVD, the dimensions of these features ($\textbf{U}_{*},\textbf{V}_{*}$) are ranked from the most important to the least important, corresponding to the descending eigenvalues in the diagonal matrices $\Lambda_{\textbf{A}}$ and $\Lambda_{\textbf{F}}$.
In light of this characteristic, we propose to better preserve the most important feature dimensions for both $\tilde{\textbf{A}}$ and F. In particular, the function $\text{Flip}(\cdot)$ reverses the $d$ dimensions of each row for the SVD features of F, such that the important features of $\tilde{\textbf{A}}$ are aligned with the less important features of F, and vice versa.

High-order Connectivity Injection.
A non-trainable layer normalization $\text{LayerNorm}(\cdot)$ is applied for numerical stability. The initialized embeddings, denoted as $\textbf{E}_{0}\in\mathbb{R}^{|\mathcal{V}|\times d}$, have consistent representation dimensionality and relatively stable semantics across datasets.
To better preserve the multi-hop connection information into the initial embeddings, AnyGraph adopts a simplified GCN without parameters () for $\textbf{E}_{0}$ as follows:

$$ (6) $\displaystyle\textbf{E}_{1}=\sum_{l=1}^{L}\textbf{E}_{0}^{(l)},\leavevmode \nobreak\ \leavevmode\nobreak\ \textbf{E}_{0}^{(l)}=\tilde{\textbf{A}}\cdot \textbf{E}_{0}^{(l-1)},\leavevmode\nobreak\ \leavevmode\nobreak\ \textbf{E}_{0 }^{(0)}=\textbf{E}_{0}$ $$

#### 3.2.2. Efficient and Strong Feature Encoder

To achieve efficiency while retaining the capacity to encode graph features, our graph experts are configured by deep multi-layer perceptron (MLP) networks. Specifically, the final node embeddings given by an expert model is calculated iteratively as follows:

$$ (7) $\displaystyle\bar{\textbf{E}}^{(l+1)}=\text{LayerNorm}\left(\text{Dropout} \left(\text{ReLU}(\bar{\textbf{E}}^{(l)}\textbf{W}+\textbf{b})\right)+\bar{ \textbf{E}}^{(l)}\right)$ $$

The final embeddings are denoted as $\hat{\textbf{E}}=\bar{\textbf{E}}^{(L^{\prime})}\in\mathbb{R}^{|\mathcal{V}|
\times d}$, where $L^{\prime}$ represents the number of fully-connected layers. And $\bar{\textbf{E}}^{(0)}$ is initialized by the aforementioned embeddings $\textbf{E}_{1}$. Each layer of our MLP module comprises a linear transformation $\textbf{W}\in\mathbb{R}^{d\times d}$ and bias $\textbf{b}\in\mathbb{R}^{d}$, followed by a ReLU non-linear activation, a dropout layer, a residual connection, and layer normalization.

Multiple Simple Experts as Strong Encoder.
It is worth noting that each graph expert in AnyGraph adopts a very simple learnable network, foregoing the capacity to mine complex hidden relations like those in heavy graph neural networks such as GATs () and GraphTransformers (). This is because AnyGraph employs a MoE architecture, where each expert is expected to handle only a sub-domain of all graph data through simple feature transformations. Therefore, no complex models are needed to accommodate different types of graphs within a single network. Compared to other graph foundation models that rely on a single heavy network, this approach further accelerates the training and inference processes.

### 3.3. Efficient Cross-domain Model Training

To maximize the cross-graph generalizability of AnyGraph, the training samples from different datasets are mixed together and randomly shuffled for model training. Each batch of training samples contains the following information:

$$ (8) $\displaystyle\mathcal{S}=\left(\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \begin{aligned} &\{(v_{c_{b}},v_{p_{b}})|b\in B\}\subset \mathcal{E}_{\mathcal{G}_{s}},\\ &\textbf{E}_{1}=\text{InitialEmbed}(\mathcal{G}_{s}),\\ &f_{\mathbf{\Theta}_{k}}\leavevmode\nobreak\ \leavevmode\nobreak\ \text{where} \leavevmode\nobreak\ \leavevmode\nobreak\ k=\psi(\mathcal{G}_{s})\end{aligned} \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \right)$ $$

Inspired by the effectiveness of link-wise graph pre-training tasks (), we utilize link prediction as the training task. Here, $(v_{c_{b}},v_{p_{b}})$ denotes the positive edges for link prediction, and $B$ denotes the batch size. To facilitate batch training, each training batch involves only one training graph $\mathcal{G}_{s}$. The initial node embeddings $\textbf{E}_{1}$ and the most competent expert model $f_{\mathbf{\Theta}_{k}}$ are preprocessed in advance to accelerate the training. Specifically, the loss function used by our AnyGraph training is as follows:

$$ (9) $\displaystyle\mathcal{L}=\sum_{\mathcal{S}}\sum_{b\in B}-\frac{1}{B}\log\frac{ \exp(\hat{y}_{c_{b},p_{b}}-\hat{y}_{\text{max}})}{\sum_{v_{n}\in\mathcal{V}_{ \mathcal{G}_{s}}}\exp(\hat{y}_{c_{b},n}-\hat{y}_{\text{max}})}$ $$

This training objective maximizes the prediction scores for positive samples $(v_{c_{b}},v_{p_{b}})$ and minimizes the predictions for all possible node pairs between $v_{c_{b}}$ and all nodes $v_{n}$. To avoid numerical instability, we employ a technique where the maximum prediction score of the batch, $\hat{y}_{\text{max}}$, is subtracted from all prediction scores.

#### 3.3.1. Feature and Structure Augmentation

To further enrich the training data, the training of AnyGraph undergoes periodic reprocessing of, firstly, the initial graph embeddings $\textbf{E}_{1}$, and secondly, the graph routing results. We demonstrate that such reprocessing augments the features and structures of the original graph data, thereby training AnyGraph using more diversified input data.

For the initial graph embeddings, we periodically reconduct the SVD and simplified GCN processes after a certain number of training steps. This helps generate different embedding spaces for the same data, thereby greatly improving the generalizability of AnyGraph regarding representation heterogeneity (). To prevent this process from consuming excessive computational time, we propose adopting different augmentation frequencies adaptive to the size of different datasets. Specifically, each dataset undergoes this representation augmentation after $|\mathcal{E}|/(10B)$ training steps.

For the graph routing results, we also periodically recalculate the recalibrated competence scores. Specifically, the positive sample pairs $(v_{c_{s}},v_{p_{s}})$ for $s=1,\cdots,S$, as well as the negative samples $v_{n_{s}}$, are randomly sampled. This essentially performs structure augmentation by using a random subset to evaluate the performance of graph experts on the input graph, thereby enhancing the model’s robustness against structural noise.

#### 3.3.2. Complexity Analysis

The training and inference process of AnyGraph is conducted by only one expert model, which has a complexity of $\mathcal{O}(B\times d^{2}\times L^{\prime})$ for each batch. Since we preprocess the initial embeddings and the expert routing, these two processes do not increase the batch-wise computational complexity. As a result, the complexity of the forward and backward steps for AnyGraph is much lower than that of other graph foundation models that involve complex GNNs and graph transformers.
Additionally, the expert routing performs $\mathcal{O}\left(\sum_{\mathcal{G}_{s}}|\mathcal{E}_{s}|\times d\times K+\sum_
{\mathcal{G}_{s}}|\mathcal{V}_{s}|\times d^{2}\times L^{\prime}\times K\right)$ computations, where the latter term empirically has a larger scale compared to the former term. This dominant term is similar to a simple GCN network of a comparable model size.
Overall, AnyGraph is more efficient than existing methods in both training and inference, and the additional computations for routing have a complexity comparable to simple GNNs.

## 4. Evaluation

Our experiments aim to answer the following Research Questions:

- •
RQ1: How does the zero-shot prediction performance of AnyGraph compare to different baseline methods?
- •
RQ2: How do AnyGraph’s various modules impact its overall performance with the contribution of each component?
- •
RQ3: How does the model size and the amount of training data influence the performance of AnyGraph?
- •
RQ4: How interpretable is the expert routing mechanism within AnyGraph’s graph Mixture-of-Experts (MoE) architecture?
- •
RQ5: How is the scalability and efficiency of AnyGraph compare to fine-tuning methods when adapting to new datasets?

### 4.1. Experimental Settings

#### 4.1.1. Experimental Datasets

To conduct a comprehensive evaluation of the cross-domain generalizability of graph models, we employ a total of 38 graph datasets. These datasets span a wide range of domains, including e-commerce (e.g. user interactions and product-wise relations), academic graphs (e.g. citation and collaboration networks), biological information networks (e.g. relations among drugs and proteins), and other domains like email networks, website networks, trust networks, and road networks.

We set up different dataset groups and conduct cross-dataset evaluations on these groups. Specifically, all datasets are divided into two cross-domain groups, Link1 and Link2, which have a similar number of total edges and a similar number of domain-specific edges.
Specifically, the Link1 and Link2 groups contain 15 and 18 datasets, respectively.
For the node classification task, we use 5 datasets gathered from e-commerce and academic information scenarios.
Additionally, we have three domain-specific groups: Ecommerce, Academic, and Others. The Others group is primarily composed of biological networks, combined with other small domains that have fewer datasets.
See Appendix A.1 for more information of our experimental datasets.

#### 4.1.2. Experimental Settings

We follow previous works (; ) for dataset splitting and evaluation metrics. Our AnyGraph model and the graph foundation models are evaluated on a cross-graph zero-shot prediction task. For baselines that cannot handle cross-dataset transfer, we evaluate their few-shot performance. Details of the evaluation protocols are provided in Appendix A.2. The Hyperparameter Settings of AnyGraph are provided in Appendix A.3. The compared Baseline Methods are introduced in Appendix A.4.

**Table 1. We evaluate the AnyGraph model (in zero-shot settings) and baseline models (with 5% and 10% training data) on link prediction (Recall@20, NDCG@20), node classification (Accuracy, Macro F1), and graph classification (Accuracy, Macro F1).**
| Data | GIN | GAT | GPF | GraphPrompt | GraphCL | AnyGraph |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Train 5% | Train 10% | Train 5% | Train 10% | Tune 5% | Tune 10% | Tune 5% | Tune 10% | Tune 5% | Tune 10% | 0-shot |  |  |  |  |  |  |  |  |  |  |  |  |
| Metric | Rec | NDCG | Rec | NDCG | Rec | NDCG | Rec | NDCG | Rec | NDCG | Rec | NDCG | Rec | NDCG | Rec | NDCG | Rec | NDCG | Rec | NDCG | Rec | NDCG |
| Link1 | 6.46 | 3.06 | 11.80 | 5.45 | 13.52 | 6.65 | 13.45 | 6.78 | 6.04 | 2.92 | 6.80 | 3.27 | 4.33 | 2.24 | 5.42 | 3.11 | 17.23 | 9.00 | 20.55 | 10.76 | 23.94 | 12.68 |
| Link2 | 6.72 | 4.50 | 21.62 | 13.41 | 9.83 | 5.91 | 15.30 | 8.84 | 7.44 | 4.25 | 16.58 | 9.84 | 6.06 | 3.36 | 6.10 | 3.62 | 29.18 | 17.62 | 31.42 | 19.91 | 46.42 | 27.21 |
| Ecom. | 3.36 | 2.58 | 13.41 | 8.06 | 3.79 | 2.94 | 9.64 | 5.78 | 7.25 | 3.84 | 18.72 | 10.94 | 4.90 | 2.59 | 6.06 | 3.36 | 22.13 | 13.19 | 26.05 | 14.59 | 26.92 | 15.05 |
| Acad. | 10.82 | 4.70 | 20.61 | 9.04 | 14.95 | 6.29 | 11.17 | 4.67 | 13.22 | 5.80 | 14.83 | 6.41 | 6.73 | 3.05 | 7.72 | 3.40 | 24.86 | 12.50 | 28.69 | 14.31 | 32.74 | 15.31 |
| Othrs. | 6.92 | 4.46 | 18.43 | 11.85 | 16.34 | 9.22 | 16.17 | 20.88 | 2.40 | 2.12 | 4.51 | 3.44 | 2.93 | 2.36 | 3.42 | 2.72 | 24.54 | 14.93 | 24.62 | 15.90 | 46.83 | 28.97 |
| Metric | Acc | MacF1 | Acc | MacF1 | Acc | MacF1 | Acc | MacF1 | Acc | MacF1 | Acc | MacF1 | Acc | MacF1 | Acc | MacF1 | Acc | MacF1 | Acc | MacF1 | Acc | MacF1 |
| Node | 20.79 | 19.46 | 36.04 | 30.60 | 53.76 | 40.14 | 54.83 | 41.61 | 12.77 | 11.45 | 16.29 | 16.00 | 18.01 | 20.59 | 23.15 | 22.89 | 43.70 | 33.72 | 48.75 | 36.15 | 64.31 | 43.24 |

**Table 2. Comparing AnyGraph framework to existing graph foundation models in zero-shot prediction capabilities.**
| Method | GraphGPT | OpenGraph |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Data | Pubmed | Cora | Ecom. w/o GR |  |  |  |
| Metric | Acc | MacF1 | Acc | MacF1 | Recall | NDCG |
| Baseline | 0.1813 | 0.1272 | 0.7011 | 0.6491 | 0.1444 | 0.1099 |
| AnyGraph-F | 0.5852 | 0.5325 | 0.7134 | 0.6003 | 0.2281 | 0.1600 |
| AnyGraph | 0.6088 | 0.5492 | 0.7809 | 0.7591 | 0.2382 | 0.1552 |

### 4.2. AnyGraph’s Zero-Shot Prediction (RQ1)

To assess the zero-shot prediction capabilities of the AnyGraph model, we conducted an extensive evaluation across 38 graph datasets from various domains. We independently trained two versions of the AnyGraph model - one on the Link1 dataset and the other on the Link2 dataset. Each trained model was then used to make zero-shot predictions on datasets it was not originally trained with. It is important to note that the Link1 and Link2 datasets do not share the same feature spaces or sources of data collection, which adds to the complexity and challenges of the zero-shot evaluation. We also compare our AnyGraph with existing graph foundation models. And in this comparison we add another AnyGraph-F version, which removes the utilization of node features. The outcomes of this evaluation are detailed in Table 1 and Table 2, and our key observations are listed as follows:

i) Superior Generalizability across Diverse Datasets. $\bullet$ Superior Prediction Accuracy. Compared to the few-shot capabilities of existing GNN models, pre-training techniques, and foundation models, AnyGraph demonstrates exceptional zero-shot prediction accuracy across various domains. This superior performance spans both link prediction and node classification tasks. $\bullet$ Effectively Handling Heterogeneity. The enhanced generalizability can be attributed to the effective handling of structure-level and feature-level data heterogeneity through unified structure and feature representations in the expert models. This approach enables AnyGraph to develop comprehensive modeling functions that are universally applicable across different graph data scenarios. $\bullet$ Comprehensive Training. Additionally, the extensive training regimen, which incorporates a variety of large-scale datasets, equips AnyGraph with a deep and broad expertise in graph modeling and prediction.

ii) Limitation of existing pre-training GNNs.
$\bullet$ Challenges of Cross-Domain Transfer. Existing pre-training and tuning methods, like GPF, GraphPrompt, and GraphCL, employ self-supervised learning and are pre-trained on half the datasets, then fine-tuned on the remaining datasets using few-shot data. However, this pre-training often fails to yield significant improvements due to substantial distribution disparities across data domains. For instance, datasets may exhibit vastly different link densities or utilize distinct node features, which significantly challenges the transfer of useful knowledge from divergent pre-training datasets during fine-tuning and prediction. $\bullet$ AnyGraph’s Robust Adaptability To address this challenge, the AnyGraph model incorporates multiple graph expert models tailored to various sub-domains of graph data. This MoE architecture effectively manages datasets from distinctly different domains, such as e-commerce user behaviors, academic networks, and road networks, demonstrating its robust adaptability.

### 4.3. Scaling Law of AnyGraph Framework (RQ2)

In this section, we explore the applicability of the scaling law to AnyGraph. We conduct experiments using 18 different versions of AnyGraph, each differing in model size and quantity of training data. Specific configurations of these variants are discussed in Appendix A.5. The evaluation results are depicted in Figure 3, which includes overall and domain-specific performance, as well as zero-shot and full-shot outcomes. Our key findings are as follows:

i) Generalizability of AnyGraph Follows the Scaling Law. As the model size and the volume of training data increase, we notice a saturation point in AnyGraph’s full-shot performance. In contrast, the zero-shot prediction accuracy continues to improve. This pattern supports the scaling law of graph foundation models, illustrating that scaling up can significantly enhance the capabilities of graph models. Two key factors contribute to this phenomenon:

- •
Task Difficulty. The saturation in full-shot performance is partly because the evaluation tasks might not be challenging enough. In-domain generalization can be more straightforward, leading to a plateau in performance improvements. This insight into the scaling law for graph data encourages further exploration of larger models on more complex graph learning tasks.
- •
MoE Architecture. The integration of the Mixture of Experts (MoE) architecture allows AnyGraph to effectively manage and utilize a broader spectrum of knowledge, particularly in this zero-shot scenario characterized by significant distribution disparities.

ii) Emergent Abilities of AnyGraph. The overall zero-shot performance curve illustrates that as the model size increases, the performance sometimes experiences periodic stagnation. With further increments in parameters, AnyGraph’s performance undergoes a sudden significant improvement. This phenomenon indicates the emergent abilities of AnyGraph, demonstrating the effectiveness of scaling up in enhancing its generalization capabilities.

iii) Insufficient training data may bring bias. In the initial stages of increasing the training data, the introduction of new datasets might negatively impact performance due to their differences from the test graphs. However, this issue can be mitigated by further expanding the training data. By providing the model with a more comprehensive set of training samples, it helps prevent overfitting and reduces bias stemming from dataset disparities.

Figure: (a) Cross-domain performance on Link1 (zero-shot) and Link2 (full-shot).
Refer to caption: x4.png

Figure: (a) Multi-domain zero-shot performance.
Refer to caption: x16.png

### 4.4. Ablation Study (RQ3)

This section evaluates the effectiveness of AnyGraph’s sub-modules by comparing ablated variants in terms of their zero-shot and full-shot performance across both cross-domain datasets and domain-specific datasets (specifically Academic data). The results are in Figure 4. We make the following observations:

- •
MoE Significantly Enhances Zero-Shot Performance. The -MoE variant, which employs a single expert model without the MoE architecture, demonstrates decent performance on datasets on which it was trained, as shown in parts (b) and (c). However, this variant exhibits a substantial decline in zero-shot prediction capabilities. This underscores the critical role of the MoE architecture in enhancing AnyGraph’s generalization abilities. The use of multiple expert models significantly expands AnyGraph’s modeling capacity, effectively managing the large disparities between various domains using multiple seperated models.
- •
Feature Modeling is Crucial in AnyGraph. In the -Feat variant, node features are omitted, leading to the most significant degradation in both zero-shot and full-shot performance. This underscores the effectiveness of AnyGraph’s unified structure and feature representation method in successfully learning features. This component is crucial for tackling in-domain graph data heterogeneity. Additionally, this outcome highlights the feasibility of unifying different feature spaces created by various methods into a single model for general use.
- •
Effectiveness of Frequency Regularization and Graph Augmentation. In the -FreqReg and -Aug variants of AnyGraph, the routing adjustment based on the training frequency of experts and the feature and structure augmentation are individually removed.
Frequency regularization specifically prevents suboptimal training outcomes by ensuring that all experts are utilized, avoiding scenarios where the majority are overlooked. Meanwhile, the graph augmentation module plays a crucial role in enriching the training graph data, thus enhancing the model’s robustness.
The outcomes from these modifications affirm the beneficial impact of these two components within AnyGraph. Omitting them can lead to biased model training, which undermines the robustness of AnyGraph in handling diverse datasets.

### 4.5. Investigation on Expert Routing (RQ4)

This section delves into the expert routing mechanism of AnyGraph. Figure 5 displays the competence scores of various expert models for the input datasets, as determined by AnyGraph’s routing algorithm based on the self-supervised loss. The figure illustrates that datasets sharing common characteristics—such as source of collection or feature construction method—are often routed to the same expert models by AnyGraph. For instance, datasets like arxiv-ta, Photo, GoodReads, and Fitness, which utilize a common text-embedding-based feature space, are assigned to highly similar experts
(expert 0, 2, 4, 5).
Additionally, ML1M and ML-10M, both sourced from the movie-rating platform Movielens, are predominantly associated with expert 1. It is also notable that this routing pattern extends to zero-shot datasets, as shown on the right part of Figure 5. Here, YelpT, SteamT, and AmazonT, which share the same feature space, are assigned to very similar expert models.
This outcome underscores the efficacy of AnyGraph’s routing mechanism in identifying the appropriate expert models for various datasets, and also showcases its explainability in revealing graph-wise relatedness.

Figure: Figure 5. Competence score between datasets and expert models, given by the routing mechanism of AnyGraph.
Refer to caption: x25.png

### 4.6. Efficiency Study (RQ5)

Tuning Curve Comparison. To evaluate the efficiency of AnyGraph, we compare its fine-tuning process with that of GraphCL and the training from scratch process of a GCN model. As depicted in Figure 6, when fine-tuned on a new dataset, the pre-trained AnyGraph rapidly achieves a high performance saturation point. In some instances, such as with the PPA dataset, GraphCL and the end-to-end trained GCN struggle to attain comparable performance levels. This advantage is based on i) the strong cross-domain generalization capabilities of AnyGraph, which bring a high starting point for the new dataset, and ii) the efficiency of AnyGraph’s MoE architecture, which requires only one MLP network for efficient but effective modeling and parameter tuning.

In addition, it is observed that pre-training GraphCL does not consistently benefit its fine-tuning on new datasets, as evidenced by GraphCL’s underperformance relative to GCN in Figure 6(b). This should be ascribed to the large distribution gap between the pre-training data Link2 and the test data PPA.

Figure: (a) Performance on Citation-2019.
Refer to caption: x27.png

Training Time Comparison. To evaluate the efficiency of the models under consideration, we compared the training times of the three models. As indicated in Table 3, AnyGraph, despite having significantly more parameters, has training times that are comparable to, or even less than, the other two models. This underscores the efficiency of our model design, and demonstrates the efficiency of AnyGraph to adapt to new data through model tuning.

Specifically, AnyGraph avoids the cumbersome process of full-graph propagation at each training step. Instead, it utilizes structure-aware embeddings derived through a non-trainable pre-processing method. This approach significantly reduces both the time and memory requirements for AnyGraph. Furthermore, the MoE architecture equips AnyGraph with the capability to use only $1/K$ of the computational resources for most prediction and optimization processes, thereby greatly reducing overall computational costs.

**Table 3. Training time for each 100 steps on different data.**
| Dataset | CS | ML1M | Yelp | Email | Cite19 | roadNet | PPA |
| --- | --- | --- | --- | --- | --- | --- | --- |
| GCN | 1.5s | 4.2s | 6.0s | 2.5s | 19.2s | 27.8s | 101.1s |
| GraphCL | 1.1s | 4.9s | 9.4s | 2.8s | 43.1s | 57.1s | 130.8s |
| Ours | 1.5s | 3.5s | 6.1s | 3.0s | 31.6s | 37.3s | 41.1s |

## 5. Related Works

Graph Neural Models.
Graph learning has garnered significant interest for its broad applicability across various fields such as user behavior modeling, social analysis, and studies in biology and chemistry (; ). Graph neural networks (GNNs) learn node representation vectors for downstream tasks like node classification and link prediction. The core mechanism involves iterative message passing, refining node embeddings to capture both node-specific information and higher-order topological structures. This process ensures that the final node embeddings effectively encapsulate both node-specific information and higher-order topological structures. Notable techniques include Graph Convolutional Networks (GCNs) (), Graph Attention Networks (GATs) (), Graph Isomorphism Network (GIN) (), and Graph Transformer (), which improves the encoding function for better graph modeling. Despite these advancements, these methods still require high-quality training data and often struggle with generalization capabilities.

Self-Supervised Graph Learning. Given the challenges with the generalizability of GNNs, considerable research efforts () have focused on enhancing GNNs through self-supervised learning objectives, aiming to capture invariant graph features. Specifically, GraphCL () introduced a contrastive pre-training approach for graph data, designed to learn authentic graph characteristics that are robust to structural and feature perturbations. Building on this, JOAO () and GCA () have developed adaptive augmentation strategies for self-supervised tasks, effectively mitigating the adverse effects of random augmentations.
Subsequent works have sought to quickly adapt these pre-trained models to downstream tasks and evolving graph data, as demonstrated by GPF () and GraphPrompt (). Despite these advancements, the generalizability of these methods remains confined to graph data with similar structural patterns and feature spaces, thus not addressing the cross-domain generalization challenges highlighted in this paper.

Large-scale Graph Pre-training.
Recent advances in graph modeling have seen efforts to pre-train large-scale graph models across multiple datasets to improve their generalization abilities, drawing inspiration from the strong generalization capabilities of large language models (LLMs) (). For instance, OFA () and ZeroG () utilize text embeddings to standardize the feature spaces across various graph datasets and tasks, facilitating cross-dataset training of graph models. Models like InstructGLM () GraphGPT () and LLaGA () synchronize graph representation spaces with the hidden spaces of LLMs,
thus enabling the application of general LLMs for graph prediction tasks.
Furthermore, HiGPT () expands the capabilities of LLMs to accommodate heterogeneous graph data.

Despite these advancements, most generalized graph models require substantial access to and integration of text features, which confines their use primarily to text-abundant environments such as academic networks. Additionally, these methods are typically trained within specific application realms, failing to address the significant variances between datasets from diverse domains.

## 6. Conclusion

In this work, the presented AnyGraph framework, an effective and efficient graph foundation model designed to address the multifaceted challenges of structure and feature heterogeneity across diverse graph datasets. AnyGraph’s innovative Mixture-of-Experts (MoE) architecture, coupled with its dynamic expert routing mechanism, positions it at the state-of-the-art of cross-domain generalization capabilities. Extensive experiments on 38 varied graph datasets have not only underscored AnyGraph’s superior zero-shot learning performance but also its robustness to distribution shifts and its adherence to scaling laws, thereby enhancing its predictive accuracy with increased model size and data volume. The model’s efficiency in training and inference, validated through comparison with existing methods, further cements its practical applicability.

## Appendix A Appendix

### A.1. Experimental Datasets

We utilize a total of 38 graph datasets across various domains. The entire dataset contains 14,437,372 nodes, and 199,265,688 edges. The dataset specifics are detailed below:

**Table 4. Statistics of the experimental datasets.**
| Dataset | DDI | Collab | ML1m | ML10m | Amazon-book | PPA | Yelp2018 | Gowalla | Cora | Pubmed | Citeseer |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| # Nodes | 4,267 | 235,868 | 9,746 | 80,555 | 144,242 | 576,289 | 69,716 | 70,839 | 2,708 | 19,717 | 3,327 |
| # Edges | 1,334,889 | 1,285,465 | 920,193 | 9,200,050 | 2,984,108 | 45,495,642 | 1,561,406 | 1,027,370 | 10,556 | 88,648 | 9,104 |
| $d$ Feats | 0 | 128 | 0 | 0 | 0 | 58 | 0 | 0 | 1433 | 500 | 3703 |
| Datasets | Proteins-0 | Proteins-1 | Proteins-2 | Proteins-3 | Products-home | Products-tech | Yelp-t | Amazon-t | Steam-t | Goodreads | Fitness |
| # Nodes | 25,449 | 6,568 | 18,108 | 13,015 | 9,790 | 47,428 | 22,101 | 20,332 | 28,547 | 676,084 | 173,055 |
| # Edges | 11,660,646 | 1,845,960 | 7,418,688 | 3,962,930 | 131,843 | 2,077,241 | 277,535 | 200,860 | 525,922 | 8,582,306 | 1,773,500 |
| $d$ Feats | 0 | 0 | 0 | 0 | 100 | 100 | 1536 | 1536 | 1536 | 768 | 768 |
| Datasets | Soc-Epinions1 | Email-Enron | Web-Stanford | RoadNet-PA | P2P-Gnutella06 | Citation-2019 | Citation-20Century | Arxiv | Arxiv-t | Photo | CS |
| # Nodes | 75,879 | 36,692 | 281,903 | 1,088,092 | 8,717 | 765,658 | 1,016,241 | 169,343 | 169343 | 48,362 | 18,333 |
| # Edges | 508,837 | 183,831 | 2,312,497 | 1,541,898 | 31,525 | 1,917,381 | 5,565,798 | 1,166,243 | 1,166,243 | 500,939 | 163,788 |
| $d$ Fets | 0 | 0 | 0 | 0 | 128 | 128 | 128 | 128 | 768 | 768 | 6805 |

E-commerce Datasets. This category includes 15 datasets from various e-commerce contexts such as user rating platforms and online retail services. These datasets vary in terms of the presence and type of node features. For instance, datasets such as Amazon-book, Yelp2018, Gowalla, Yelp-text, Amazon-text, Steam-text, Goodreads, Amazon-Fitness, Amazon-Photo, Movielens-1M, Movielens-10M, Products-home, Products-tech, Home-node, Tech-node are included. Notably, Amazon-text, Steam-text, and Yelp-text utilize the same method for feature generation, while Fitness, Photo, and Goodreads employ a different consistent method.

Academic Network Datasets. We use 13 datasets focused on academic networks, which include citation and collaboration relations among scholars and papers. These datasets represent various research fields and employ diverse feature generation methods, such as NLP embeddings, bag-of-words, and different versions of large language models. The specific datasets are Cora, Pubmed, Arxiv, Cora-link, Pubmed-link, Citeseer, CS, Arxiv-link, Arxiv-t (with features derived using an alternative method), Citation-2019, Citation-20Century, OGB-Collab.

Biological Information Networks. Our collection includes 6 datasets related to biological entities like proteins, drugs, and diseases. This category features networks such as OGB-DDI, OGB-PPA and four protein relation networks for different species, denoted as Proteins-0, Proteins-1, Proteins-2, Proteins-3.

Other Datasets. In addition to the categories mentioned above, we include 5 datasets from various other fields: an email network Email-Enron, a website network Web-Stanford, a road network dataset RoadNet-PA, a P2P web network dataset P2P-Gnutella06, and a trust network dataset Soc-Epinions1.

Dataset Groups. For conveinience of performance evaluation, we split the many datasets using different grouping methods. Firstly, two big data groups Link1 and Link2 are made using all the link prediction datasets. Notably, datasets from the same source of collection, such as Movielens-1M and Movielens-10M, or uses the same method to generate features, such as Fitness, and Photo, are put into the same group, to avoid information leakage when evaluating zero-shot performance on the other group. Apart from these two datasets, we also conduct evaluations on domain-specific groups, including E-commerce, Acadmic, and Others. Specifically, these data groups contain the following datasets, respectively:

- •
Link1: Products-tech, Yelp2018, Yelp-textfeat, Products-home, Steam-text, Amazon-text, Amazon-book, Citation-2019, Citation-20Century, Pubmed-link, Citeseer, OGB-PPA, P2P-Gnutella06, Soc-Epinions1, Email-Enron.
- •
Link2: Photo, Goodreads, Fitness, Movielens-1M, Movielens10M, Gowalla, Arxiv, Arxiv-t, Cora, CS, OGB-Collab, Proteins-0, Proteins-1, Proteins-2, Proteins-3, OGB-DDI, Web-Stanford, RoadNet-PA.
- •
Ecommerce and Academic: These two datasets contain all the domain-specific datasets as mentioned above.
- •
Others: This group contains all the biological datasets, and other datasets including Email-Enron, Web-Stanford, RroadNet-PA, P2P-Gnutella06, Soc-Epinions1.

### A.2. Evaluation Protocols

All datasets used in this study are sourced from previous research as referenced [, ]. We adhere to the original data splits from these sources to delineate our training and testing sets. Given that many baseline methods are not equipped to manage zero-shot prediction across datasets, we instead assess their few-shot capabilities. This allows for a comparative analysis against the zero-shot performance of AnyGraph. We employ specific evaluation settings tailored to each method, detailed as follows:

- •
Zero-shot Setting for AnyGraph, GraphGPT, and OpenGraph. In our study, AnyGraph and two comparative graph foundation models, GraphGPT and OpenGraph, undergo evaluations for zero-shot prediction capabilities. We pre-train two instances of AnyGraph using Link1 and Link2 datasets. The model pre-trained on Link1 is then tested for zero-shot performance on the Link2 group datasets, and vice versa. Results labeled as "zero-shot" for AnyGraph are derived using this cross-evaluation method. Conversely, results marked as "full-shot" pertain to supervised learning outcomes, where, for example, the model trained on Link1 is tested on the test sets of Link1 group datasets. For GraphGPT and OpenGraph, we utilize the models as released in their respective original studies, which were pre-trained on specified datasets.
- •
Zero-shot Node Classification for AnyGraph. Drawing from insights in prior research [], our approach to zero-shot node classification involves a novel method where label classes are represented as distinct nodes. We then connect existing nodes that have training labels directly to these new class nodes. This technique eliminates the need for learning specific parameters for each class within the zero-shot learning framework, streamlining the process. We have integrated this innovative approach into baseline methods as well, enhancing their capability to handle unseen node labels effectively.
- •
Few-shot Training for GIN and GAT. The GIN and GAT models, employed as end-to-end training baselines, undergo training from scratch on few-shot subsets of the evaluation datasets. This approach is necessary because these models are not well-suited for cross-dataset transfer, particularly when dealing with datasets that have varying feature dimensionalities.
- •
Pre-training and Few-shot Tuning for GraphCL, GPF and GraphPrompt. These category of baselien methods follow the pre-training-and-fine-tuning mode. In our evaluations, they are firstly pre-trained using the same pre-training datasets as our AnyGraph. Then, they experience an additional fine-tuning process using the few-shot subsets of the evaluation datasets.

Evaluation Metrics. For link prediction, we follow previous works [] and utilize Recall@20 and NDCG@20 as the evaluation metrics. Note that we typically use the summary results of the evaluation results across multiple datasets. Results for fifferent datasets are averaged according to their number of test samples. For the node classification task, we employ the widely-used Accuracy and Macro-F1 score as our metrics [, ].

### A.3. Hyperparameter Settings

Optimization. Our model, AnyGraph, is implemented using PyTorch. The optimization process employs the Adam optimizer with a learning rate of $1\times 10^{-4}$ and a training batch size of 4096. We use cross-entropy loss with a sampled negative set []. The learnable parameters of AnyGraph are initialized using the Xavier uniform initializer.
Network Configurations. The standard configuration of our AnyGraph includes 512 hidden units and 8 graph expert models. Each expert model comprises 8 fully-connected layers. These layers utilize a ReLU activation function and incorporate a dropout layer with a dropout probability of 0.1.
Algorithm Hyperparameters. The frequency regularization of our routing mechanism is set with an adjustment range of $\rho=0.2$. The SVD decomposition is performed using 2 iterations. For structural and feature augmentation, each dataset is reprojected after using 1/10 of its samples for optimization. A minimum of 100 training steps should be executed for each dataset before its initial representations are reprojected. The reassignment of experts occurs after all training datasets have undergone one cycle of re-projection.

The baseline methods are evaluated using theeir original code or released model. We closely follow the original code to adapt to our experiments. Grid search is conducted to search for the best hyperparameter settings for each baseline method.

### A.4. Baseline Methods

This section provides detailed descriptions of the baseline models used in our analysis. We employ seven different baseline models across four distinct categories.

Training-from-scratch Graph Neural Networks.

- •
GAT []. Graph Attention Networks (GAT) leverage an attention mechanism to dynamically weight node-to-node connections, enhancing the model’s ability to adaptively propagate and aggregate information across the graph.
- •
GIN []. The Graph Isomorphism Network (GIN) significantly boosts the expressive power of Graph Neural Networks by introducing a unique graph encoding technique aimed at effectively distinguishing between non-isomorphic graphs.

Graph Pre-training Models.

- •
GraphCL []. It enhances the pre-training of graph models via self-discriminative contrastive learning, which is applied to learned node embeddings. The method employs various graph augmentation techniques such as node dropping, edge permutation, random walks, and feature masking to improve robustness.

Graph Prompt Tuning Methods.

- •
GraphPrompt []. GraphPrompt proposes a unified approach that integrates pre-training and prompt tuning for graph models. It features a learnable prompt layer designed to automatically extract crucial information from the pre-trained model to enhance performance on downstream tasks.
- •
GPF []. The Graph Prompt Framework (GPF) is a versatile graph prompt tuning framework compatible with various graph pre-training methods. It offers two variants of a learnable graph prompt layer, tailored to different application needs.

### A.5. Details of the Scaling Law Experiment

For the scaling law experiment (RQ2), we elaborate the configurations of the developed instances of AnyGraph. For AnyGraph with different model sizes, we begin with the smallest model which has 64 hidden units, 1 fully-connected layer, and 1 expert model. The subsequent 3 model instances increases in their hidden dimensionality, from 64 to 128, 256, and 512. Then 3 larger models with more fully-connected layers are utilized, respectively containing 2, 4, and 8 MLP layers. Then we have MoE versions of AnyGraph, with 2, 4, and 8 experts, respectively. The final largest instance of AnyGraph has a larger latent dimensionality of 1024.

For the increase of training data, we begin with a subset of Link2 data including Cora and CS. The next version additionally includes Photo. The thir one includes ML1M. The fourth one includes Gowalla. The fifth one additionally include Arxiv and Arxiv-t. The sixth one adds the following datasets: collab, ddi, Yelp2018, Fitness, proteins-spec1, web-Stanford, proteins-spec3. The seventh one is trained with proteins-2, roadNet-PA, and Fitness additionally. And the final one is trained with all datasets from Link2.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Rephrasing the 'weak' points above:

Q1 (based on W1): how can a practitioner be sure that AnyGraph will work for a graph outside the 3-4 groups (academic, social, etc) that were used in the paper

Q2 (based on W2): Does AnyGraph work well on heterophily graphs (see the datasets in Table 4, bottom, of the 'slimG' paper mentioned earlier)

Q3 (based on M3): what is the total wall-clock time (a) for training and (b) for inferencing - please
soundness: 3
strengths: S1 nice idea, to use a mixture of experts

S2 also nice idea, to use of SVD for feature unification (Eq. 5), line 224, 225

S3 experiments show improvement on baselines

S4 nice ablation study
summary: The paper proposes a foundation graph model to handle graphs with (numerical) node attributes.

It used 38 diverse graph datasets for training.

It proposes a 'mixture of experts' model, thus reducing the number of parameters.

Experiments on these 38 datasets show better accuracy on link prediction and node classification, against popular baselines.
weaknesses: W1. too few datasets - hard to believe that the method may work on, say,
    patient-doctor-diagnoses graphs, that it has never seen before.

W2. Eq. 6, line243: adding the embeddings may lead to failures if the graph
    exhibits heterophily. Concatenation would be better - see, eg.,
    Table 1 of the SlimG paper
    [Yoo et al, KDD 2023] or https://arxiv.org/pdf/2210.04081

Minor, presentation suggestions:

M1. it would be nice to give the problem definition:
  given <??> find <??> to optimize <??>

M2. also, a table of symbols and definitions, like 'Flip()' on Eq5,
  $B$, $L'$, $K$,  on lines 291-294.

M3. what is the total wall-clock time for training, and for inference;
  and on what hardware (# CPUs, #GPUs, etc)
  Table 3 only gives training time per 100 steps.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: * What is the inference overhead of AnyGraph?
* Why the approach consumes only 1/K memory resources during training or inference? You still need to host all the expert in memory.
* In Figure 3. With the increasing size of parameters, when AnyGraph’s performance improves first and then suddenly experiences a drop first and improves significantly. Why? Is there any explanation?
* How do you compute the training time of AnyGraph in Table3? Which training set are you using? Link1 or Link2?
soundness: 2
strengths: * The paper highlights the key challenges of building a graph foundation model: 1/ structure heterogeneity of different graphs, 2/ feature heterogeneity of different graphs, 3/ foundation model adaptation to new domains. It is a good summary of existing challenges.

* AnyGraph employs a MoE architecture to tackle the cross-domain graph heterogeneity. It proposes a AnyGraph specific method to route the experts. Specifically, the expert indicator score is computed with dot-product-based relatedness scores for positive and negative edges of an input graph. The method is soundness and effective.
summary: Graph foundation models offer a transformative solution, with the potential to learn robust, generalizable representations from various graph data. This paper studies the key challenges of building a graph foundation model: 1/ structure heterogeneity of different graphs, 2/ feature heterogeneity of different graphs, 3/ foundation model adaptation to new domains. The paper proposes a unified graph model AnyGraph to handle the challenges. AnyGraph uses singular value decomposition (SVD) to extract graph structure information and feature information to address the in-domain graph heterogeneity. AnyGraph employs a MoE architecture consisting of multiple graph expert models to address the cross-domain graph heterogeneity. The evaluation results show that the proposed graph foundation model AnyGraph outperforms the baseline GNN models like GIN, GAT, etc, on the zero-shot/few-shot setting.
weaknesses: * The proposed graph foundation model AnyGraph cannot work with heterogeneous graphs (graphs with multiple node types and edge types).
* The paper is very hard to read. 
    * Figures do not have clear captions. Here are some examples:
        1. Figure 2 is not understandable. 
        2. Where is Figure 4(c)? What does -Feat mean in Figure 4?
    * Notations are not well presented and explained. Here are some examples:
        1. In Section 3.1, how to get the entity representation like v_{c1}, v_{p1} is unclear. (I guess it is from equation 5). 
        2. How do you define \hat{y}_{max} in equation 9?
    * Some tables do not have clear caption and explanation. Here are some examples
        1. Which model is the baseline in Table 2? Why only show the performance of Pubmed and Cora?
* Some of the design choices are not well justified and presented:
    * Why Adjacency matrices are used to init the node embeddings? The Adjacency matrices are used twice, one in node feature initialization and one in message passing.
    * What kind of reprocessing is applied to initial graph embeddings in feature and structure augmentation?
* The details of the evaluation result of each dataset corresponding to Table 1 is missing in the Appendix. 
* I also want to see the comparison between AnyGraph and the baseline models trained with full training set (not the few-shot setting). This can help us to understand the gap between AnyGraph and full-training set training.
* How to choose the number of experts is unclear? Why using seven in the evaluation? What is the impact to use less experts and more experts?
* All the graph datasets used in evaluation are small graphs with maximum of 1M nodes. Can we also evaluate the performance of AnyGraph on large graphs like ogbn-products, ogbn-paper100M?
* The way how AnyGraph compute graph embeddings is similar to NetInfoF (https://arxiv.org/pdf/2402.07999). Can you compare the difference?

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Q1. In the Feature and Structure Augmentation paragraph, how can feature reprocessing create varied embedding spaces? In my understanding, the initial embeddings should remain the same since no learnable parameters are involved with this process.

Q2. In Table 1, what datasets were used for training and evaluation for node classification tasks?

Q3. Table 2 is confusing to read. There are columns for GraphGPT and OpenGraph, and rows for Baseline, AnyGraph-F, and AnyGraph. What does each entry mean? What does Baseline refer to? What is AnyGraph-F?

Q4. Dataset sources should be specified with the corresponding citations. No citations are currently given in Appendix A.1.
soundness: 3
strengths: S1. This work tackles an important problem for graph learning methods, which is to equip graph learning models with the capability to be able to efficiently adapt to a wide range of graph domains and tasks. Due to the diverse and heterogeneous nature of graph datasets (e.g., graph-structural heterogeneity and feature heterogeneity), existing approaches yet struggle to satisfy several desiderata for dealing with such real-world data, such as fast adaptation to new datasets, broad applicability, and adherence to the scaling law. This paper fills in this gap by developing a new approach that tackles these challenges.

S2. This paper proposes AnyGraph, a new graph foundation model that tackles the aforementioned challenges using several interesting ideas, including graph mixture-of-experts, graph expert routing, and the use of a unified mapping function to process heterogeneous graph connectivity patterns and node features. The proposed model architecture and expert routing mechanism enable an efficient training and inference, while achieving high generalization performance.

S3. The paper presents extensive experimental results using 38 graph datasets, which shows the effectiveness of AnyGraph, such as its superior zero-shot prediction performance and efficiency, and demonstrates that the performance of AnyGraph follows the scaling law. In comparison to the baselines, which were trained using 10% of the training data, the proposed AnyGraph’s zero-shot performance is much better, which is an impressive result. Also, the ablation studies show the positive contributions the individual components of AnyGraph make to the overall zero-shot performance.
summary: This work tackles an important problem for graph learning methods, which is to equip graph learning models with the capability to be able to efficiently adapt to a wide range of graph domains and tasks. To that end, this paper proposes AnyGraph, which aims to address several challenges for graph foundation models, such as dealing with structure and feature heterogeneity of real-world graphs, and the ability to quickly adapt to new domains, and follow the scaling law. AnyGraph achieves this goal by using several interesting ideas, including graph mixture-of-experts and the use of a unified mapping function for graphs and features. Extensive experiments on 38 graph datasets demonstrate the effectiveness of AnyGraph, such as its superior zero-shot prediction performance compared to the few-shot performance of several baselines.
weaknesses: W1. While the most important capability of AnyGraph would be the ability to effectively generalize to new domains, it is not clear how the design of AnyGraph, in principle, can achieve strong generalization capability when given graphs from new domains. Despite strong experimental results involving cross-domain groups, it is unclear how AnyGraph can generalize to new domains that are significantly different from the domains observed during training. Given a graph from a new domain, AnyGraph will select an expert that has the best competence score. However, that doesn’t mean that the selected expert could perform well, if the domains the selected expert has been trained on significantly differ from the new domain. Thus, when given a graph that greatly deviates from observed graphs, it doesn’t seem like any trained expert could perform well in principle.

W2. The Related Works section lacks a discussion of mixture-of-expert models and graph foundation models. While graph mixture-of-experts lies at the center of the proposed AnyGraph’s architecture, it is not discussed how the proposed graph mixture-of-experts framework relates to, and/or improves upon, previous (graph) mixture-of-expert models, which makes it hard to judge the technical contributions of the proposed framework. Also, a discussion of graph foundation models in the Related Works section would be helpful for a clearer contextualization of AnyGraph.

W3. Some of the design choices of AnyGraph seem a bit arbitrary, and need better justifications, despite its strong empirical performance. For instance, in Eqn (5), the Flip() function aligns important features of A with less important features of F. This alignment seems counter intuitive, and it is not clear how this helps. What is the intuition behind this? Also, in Eqn(4) for training frequency regularization, what is the intuition of adding +1.0-\rho/2?

W4. It is not clear how AnyGraph compares with baselines in terms of number of parameters. While the paper says that AnyGraph has significantly more parameters than baselines (Sec 4.6), it’s not clear how this can be the case. AnyGraph does not seem to have many parameters due to its use of simplified GCN. Given that the model size (# params) is one of the major axes that constitutes the scaling law, it would be useful to provide a comparison among representative methods in terms of the number of parameters.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 1
questions: Please see the questions written in the previous section.
soundness: 2
strengths: The paper proposes a complex framework, that seems to work, but details are not enough to evaluate properly (even if I consider the appendix).

The experiment section has several analyses making the paper stronger. However, important details are omitted.
summary: The paper proposes the use of k expert for classification purposes. Each expert model is trained on edge samples (observed and unobserved edges) from multiple graphs (graphs are assigned with a problem routing algorithm) using a link prediction approach. Each expert model generates a |V|x d-dimensional embedded matrix by combining the SVD of the Laplacian-normalized adjacency matrix and the SVD of the node feature matrix (equation 5). This embedded matrix is modified and a multilayer perceptron (MLP) is applied for further modifications generating the final embedding (the training process of the MLP is based on the link prediction problem). When a new graph arrives, a competence score is calculated per expert model (equation 4). The score combines the embedding of each expert and the probability of predicting a sample (observed and unobserved edges) of the target network. The expert with the highest score is used for the final classification problem.
weaknesses: Please, rewrite the abstract. According to the current abstract, the main contribution is a unified graph model, but the problem is not described. Is this unified model to generate a graph, to learn the distribution of the graph, or any other problem? Something similar happens with the introduction, where the characteristics of the model are mentioned, but not the problem. Based on page 3 (line 111), it can be deduced that the model is for prediction. Finally, in section 2, preliminaries, the problem is mentioned/defined in more detail.

The readability of the paper is very low. Besides the issue of the problem definition explained before, the methodology also lacks several details, forcing the reader to derive these details. For example, in equation 3, the value S is not defined, and it must be deduced that S is the number of samples used for the graph. Also, given the importance of \hat{\mathbf{e}}_i, be more specific, and mention that \hat{\mathbf{e}}_i is the node embedding of v_i. How do you sample the nodes? What is the effect of the sampling process on the competence indicator? What is the effect of the encoder on the competence indicator function (no embedding process is mentioned until the next subsections)? 

The notation could be improved. Equation 3 has a subindex k which corresponds to the value of the k-expert, but as can be appreciated, the value k is not in the equation. Again, the reader must deduce that the embedding is specific for each expert and that the embedding used in that equation corresponds to the embedding of the expert k. A simple solution is to include another subscript/superscript making a refer to k.

According to the Training Frequency Regularization, it seems that the graph expert routing mechanism is used to assign a graph to an expert in the training process ("These models generally receive more or better training samples in the early training stages, giving them an advantage over other experts"). However, it is mentioned that each expert will be focused on graphs with specific characteristics. How do you ensure that each expert can model the different characteristics, even considering that the sample of the graph can not be representative?

Be careful with the statement "This approach consumes only 1/K of the computational and memory resources required for predictions and optimization, compared to other non-MoE". You should prove this with a memory complexity analysis. Consider the same model training one time with all the data, against K equal models each of them trained with different data. In both cases, the estimation will be the same; however, the proposed method will have to analyze K competence indicator increasing the time.

The paper is not self-contained. Most details for reproducibility are in the appendix instead of the main paper. Recall, we are not forced to read the Appendix. What is your first experiment? Did you randomly choose some edges or did you try all possible combinations of edges and use the top 20 edges for prediction? Why the Table 1 changes the Data for the different models?

What do you mean by a zero-shot prediction task? Zero-shot usually means without any further training process. In the case that you are using the same data used for a previous training approach, then this is not a zero-shot prediction task. In the case that you are using another dataset, then you are analyzing the generalization capability of the model, which could be interesting. However, given the lack of details of the experiment section and the appendix, this can not be concluded.

Why do you think that the model is able to classify correctly the node labels even though they are not even considered in the process? 

How did you setup the other baseline models? Did you use the same datasets as the data used in the proposed framework?  Why you did not use other models for Table 2? What is AnyGraph-F?

In summary, the paper seems to be an ensemble method, which is described as a complex framework, and important details for the evaluation process are not mentioned.

Minor comments:
Line 94 "topologies.Furthermore"
Figure 1 is not mentioned in the text.
Line 127 d_0 is not defined
Line 128 "indicating"
Line 242 Acronyms must be defined before their are used

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: see W1-W6
soundness: 2
strengths: S1: Nice presentation.

S2: their target problem is very important.

S3: The ambition of exploring a graph foundation model, especially getting rid of LLMs, should be encouraged, which is rare and commendable.
summary: This paper propose graph MoE with some training strategies for a graph foundatoin model.
weaknesses: W1: My 1st concern is about the semantic unifying. Different graphs usually have totally different features w.r.t semantics and dimensionality. The solution of this paper is to use SVD for dimension alignment and link prediction as a training target. However, it should be noted that even if you use SVD to achieve the same dimensionality, they are still located in different latent semantic spaces. The natural gap in the semantic space will not be narrowed with SVD tricks, and the link prediction is not helpful for further semantic alignment. 

I personally hold the following opinion (I am open to seeing further discussion if my opinion is not accepted by the authors or other reviewers): currently, the general feature alignment issues across different graph datasets is still an irresolvable problem, unless we have some detailed assumption (e.g. two graph datasets share some overlapped instances, anchor nodes, etc.) 

With the above discussion, I think the reason why this paper works w.r.t feature gap (Feature Heterogeneity as they called) may be from some good luck. For example, the experimental datasets might in nature share a large body of their feature semantics (making SVD spaces also overlap) but this is not related to their technique contributions.


W2: The 2nd concern is about structure heterogeneity. I didn't find the corresponding solution to this problem from their whole framework. In particular, the gold standard to convince the readers that you solve the structure heterogeneity issue is the combination of both homophilic and heterophilic graph datasets. The authors should see whether it will cause negative transfer issues, or positive performance than a single model. Currently, the evaluated datasets seem to be all homophilic datasets.

W3: The 3rd concern is about the truth of the Scaling Law of AnyGraph. It is interesting to see some possible trends of scaling law from Figure 1 and Figure 3. However, I personally hold a very cautious opinion of this trend, and I think we need more convincing evidence.  My concern comes from a very natural comparison: language texts only share several thousands of words (a.k.a tokens) but they collect an enormous corpus to train a very large language mode and then they see the scaling law phenomenon. Compared with text as a linear structure with limited tokens, graph datasets are a non-linear structure with nearly unlimited tokens (nodes), but the training graph dataset scale and the parameters of the existing framework are far from comparable to their counterparts in NLP. This is far-fetched to convince the readers that the observation is indeed Scaling Law. For example, how about extending the x-axis further? Currently, the best performance is still not significantly higher than the lowest performance (they seem to be all around 0.1-0.2).

W3: I feel confused about the claim like cross-domain and cross-dataset. According to their experimental settings, link 1 and link 2 have some overlap domains so the results of such a training strategy might not be appropriate to be claimed as cross-domain effectiveness. 

W4: I did not find something new in technical contributions. I neither find something huge in model size and data size like their counterpart in the NLP area. I think either of these two aspects is a notable contribution if it indeed works.  


W5: Currently, the graph foundation model might be more promising to achieve within a specific domain rather than general multiple domains. I think their claim is a little exaggerated and over-claimed. Compared with the foundation model in NLP, I think the graph foundation model is far from feasible at the current stage. We should first solve some very fundamental problems. for example, how to solve unlimited tokens in graph datasets (while in the NLP area, tokens are limited). How to increase the complexity of the graph model and make sure they will not undergo performance bottleneck (e.g. oversmooth issues for traditional GNNs). How to build a very large dataset and make sure the graph foundation model can learn so-called "knowledge", and furthermore, whether the concept "knowledge" truly exists in the graph domain compared with their counterpart in the NLP area.  Currently, the academic can even not come to an agreement on graph knowledge learned by existing pre-training approaches. I am open to seeing some revolutionary breakthrough, some entirely new infrastructure to replace existing graph models and then I think we might have some brittle confidence to modestly say that we might come closer to the "graph foundation model".

W6: There is a very weird thing: The authors compare their so-called "graph foundation model" with graph-promoting methods. I believe the authors might be lost in these concepts. Graph prompting is used to prompt a pre-trained graph model for more general task performance. Currently, studies on graph prompts face a very awkward situation: compared with their counterpart in LLMs, we do not have a very large graph foundation model. So the researchers on graph promoting have no choice but to deploy their graph prompting technique on GNNs. However, since the authors claim that they successfully invented the "graph foundation model", there is no doubt that we should immediately apply graph prompting techniques to your graph foundation model to see what will happen. But the weird thing is that you compare your graph foundation model with graph prompts. It is a logical mess. It is just like, inventing GPT 4.0, and then comparing your GPT 4.0 with GPT 1.0 with a prompt and then claiming that prompt is not better than GPT 4.0. I do not know what the points of this comparison are. One is the model, one is the usage process. I feel confused about the comparisons and the results in Table 1 (w.r.t graph prompt) make no sense.