AutoGFM: Automated Graph Foundation Model with Adaptive Architecture Customization
Authors: Haibo Chen
Xin Wang
Zeyang Zhang
Haoyang Li
Ling Feng
Wenwu Zhu
Venue: ICML 2025 oral (2025)
Keywords: Graph Foundation Model
Text Attributed Graph
Graph Neural Architecture Search
Graph Neural Networks
Primary area: deep_learning->graph_neural_networks
TL;DR: We explore the problem of graph neural architecture search for GNN-based GFMs, aiming to automatically search customized GNN architectures tailored to data with different domains and tasks.

Abstract: Graph foundation models (GFMs) aim to share graph knowledge across diverse domains and tasks to boost graph machine learning. 
However, existing GFMs rely on hand-designed and fixed graph neural network (GNN) architectures, failing to utilize optimal architectures *w.r.t.* specific domains and tasks, inevitably leading to suboptimal performance in diverse graph domains and tasks. 
In this paper, we explore graph neural architecture search (GNAS) for GFMs for the first time, which suffers from the problem of *architecture inconsistency*, i.e., the optimal architectures for different tasks and domains vary. We tackle this problem by discovering an invariant graph-architecture relationship across domains and tasks, which imposes three challenges: i) how to capture invariant and variant patterns; ii) how to customize architectures to adapt to diverse domains and tasks; iii) how to mitigate the data domination phenomenon during the architecture search process.
To address these challenges, we propose **Auto**mated **G**raph **F**oundation **M**odel with Adaptive Architecture Customization (**AutoGFM**), providing a theoretical analysis to demonstrate the limitations of existing GNAS. Specifically, we first propose a disentangled contrastive graph encoder to learn invariant and variant patterns. Then, we design an invariant-guided architecture customization strategy to customize architectures for data from diverse domains and tasks. Finally, we propose a curriculum architecture customization mechanism to mitigate the phenomenon of particular data dominating the search process. 
Extensive experiments demonstrate that **AutoGFM** outperforms baselines, achieving state-of-the-art performance.

AutoGFM:
Automated Graph Foundation Model with Adaptive Architecture Customization
Haibo Chen
Xin Wang's
Zeyang Zhang'
Haoyang Li
Ling Feng'
Wenwu Zhu'
Abstract
1. Introduction
Graph foundation models (GFMs) aim share graph knowledge across diverse domains and tasks boost graph machine learning. However, existing GFMs rely on hand-designed and fixed graph neural network (GNN) architectures, failing to utilize optimal architectures w.r.t. specific do- mains and tasks, inevitably leading to suboptimal performance in diverse graph domains and tasks. In this paper, we explore graph neural architecture search (GNAS) for GFMs for the first time, which suffers from the problem of architecture inconsis- tency, i.e., the optimal architectures for different tasks and domains vary. We tackle this problem by discovering an invariant graph-archhe architecctree rela- tionship across domains and tasks, which imposes three challenges: i) how to capture invariant and variant patterns; ii) how customize architec- tures adapt to diverse domains and tasks; iii) how to mitigate the data domination phenomenon during the architecture search process. To ad- dress these challenges, we propose Automated Graph Foundation Model with Adaptive Archi- tecture Customization (AutoGFM), providing a theoretical analysis to demonstrate the limitations of existing GNAS. Specifically, first propose disentangled contrastive graph encoder to learn invariant and variant patterns. Then, we design an invariant-guides -uidddd architecture customization strat- egy to customize architectures for data from di- verse domains and tasks. Finally, we propose a curriculum architecture customization mechanism mitigate the phenomenon of particular data dominating the search process. Extensive experi- ments demonstrate that AutoGFM outperforms haselines, achieving state-of-the-art performance.
Graph foundation models (GFMs) (Liu et al., 2023b; Xu et al., 2024; Kong et al., 2024) aim share graph knowl- edge across diverse graph domains and tasks. GNN-based GFMs (Liu et al., 2023a; Wang et al., 2024b) represent promising direction, as they enable the transfer of shared knowledge across various domains and tasks, allowing a single graph neural network (GNN) to handle node-level, edge-level, and graph-level tasks across various domains. Specifically, GNN-based GFMs leverage large language models (LLMs) enhancers, transform the textual fea- tures of graphs into unified representations, and unify graph- related tasks through subgraph classification for GNNs.
However, data from different tasks and domains may require different graph neural architectures. For instance, the vanilla GCN (Kipf & Welling, 2017) outperforms GraphSAGE (Hamilton et al., 2017) in the citation network OGBN-arxiv while failing to demonstrate satisfactory performance in OGBN- proteins (Hu et al., 2020). Since existing GNN- based GFMs rely on hand-designed and fixed GNN architec- tures, they inevitably fail to adapt to the specific architecture requirements for diverse domains and tasks.
In this paper, explore the problem of graph neural ar- chitecture search (GNAS) for GNN-based graph foundation models, which suffers from the problem of architecture inconsistency, i.e.. the optimal architecture for different tasks and domains varies. We further leverage a representa- tive group of differentiable graph neural architecture search methods (Liu et al., 2018) as an example and provide theo- retical analysis demonstrating their inability effectively search for graph neural architectures for GFMs under archi- tecture inconsistency, resulting in suboptimal architectures. We tackle this problem by discovering an invariant graph- architecture relationship across domains and tasks, which imposes three challenges: i) how to capture invariant and variant patterns, which are entangled in graph data; ii) how to customize graph neural architectures based on the dis- covered patterns to adapt data with diverse domains and tasks; iii) how to mitigate the phenomenon of data domina- tion during the architecture search process.
'Department of Computer Science and Technology. BN- RIST, Tsinghua University, Beijing, China. Correspondence to: Xin Wang <xin wang wwwgi@@ttssciuu.com. Wenwu Zhu <wwzhu@tsing trrchu@@singguu..du.n.>>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
To address these challenges, we propose a novel Automated

---

AutoGFM: Automated Graph Foundation Model with Adaptive Architecture Customization
Graph Foundation Model with Adaptive Architecture Cus- tomization (AutoGFM), which customizes graph neural ar- chitectures for graph data across diverse tasks and domains. The core idea is to train an architecture mapping function π, which maps 9 (graph data) > A (architecture), enabling the customization of architectures for each dataset to address architecture inconsistency, while simultaneously facilitating mutual knowledge sharing across diverse domains and tasks within weight-sharing super-network. Specifically, we first propose disentangled contrastive graph encoder to learn invariant and variant patterns from graph data. To achieve this, we design a subgraph-level discriminative contrastive learning that captures the invariant and variant patterns from diverse graph data. Second, we propose an invariant-guided architecture customization tailor graph neural architec- tures for diverse data. We encourage invariant patterns to retain their ability to customize architectures despite the interference from variant patterns, aiming to eliminate the spurious effects brought by variant patterns. Finally, we pro- pose a curriculum architecture customization mechanism to mitigate the phenomenon of some particular data dominat- ing the search process. We design curriculum constraint to promote the diversity of customized architectures across dif- ferent datasets. Extensive experiments demonstrate that our AutoGFM model outperforms existing baselines, achieving state-of-ther the-art art performance. The contributions of this paper are summarized as follows:
2.1. Graph Data Definition
Text-attributed Graphs (TAGs) A text-attributed graph (TAG) is a graph where each node and edge is associated with text sentence (Liu et al., 2023a). We denote TAG is denoted as g (V. &, R), where V = vi..... l repre- sents the set of nodes, & = , [|s] represents the set of edges, and R = ....... [/|]] represents the set of relations.
Node of Interest (NOI) Subgraph Given a graph g = (V, E, R). Following the previous work (Liu et al., 2023a; Wang et al., 2024b), we define the subgraph to unify graph tasks as node of interest subgraph (NOI-graph), An NOI- graph G₁, is defined the subgraph around the NOL De- note (v)v) {V Et R.j as the ht-hop ego-subgraph around consisting of h-hop neighbor nodes of and all interconnecting edges. For node-level tasks on a node v, the NOI is the node itself, such that T = {v} and (T) = (v). For link-level tasks involving a node pair (0,, 2y), we define T =(,, By and the NOI-graph is Ga((,, m5}) = Sn (1:) U S.,) For graph-level tasks, the NOI includes all nodes in the graph, making the NOI-graph Gv = EV,E, R.). We define an NOI-graph GA. (T) as:
(T = Sp = (UUET V,, U U.)T
(1)
2.2. Graph Neural Architecture Search
We propose to explore the problem of graph neural architecture search for GNN-based graph foundation model, to the best of our knowledge, for the first time.
Given a data D = (G.3) for a graph neural architecture search (GNAS), we aim to search for a function Fa..: g + Y, with architecture parameters a E A and learnable weights w E W, where A is the architecture space and W is the weight space:
We propose Automated Graph Foundation Model with Adaptive Architecture Customization (AutoGFM), an- alyzing the problem of architecture inconsistency for GFM and providing a theoretical analysis to demon- strate the limitations of existing mainstream differen- tiable GNAS methods under such conditions.
a² arg min L(F aEa
a, (a)(),)))
(2)
s.t. w' va(a)= = arg min C(Fa, (()).(()))))
(3)
wEW(a)
We propose three novel modules to tackle the prob- lem of architecture inconsistency. i) disentangled con- trastive graph encoder, ii) invariant-guided architecture customization, and iii) curriculum architecture cus- tomization mechanism.
where C represents the loss of predictions made by the architecture (.) on the graph, and a and w" denote the optimal architecture and weights for the given data D = (G. ))). Specificallyy, a typically represents the selection of GNN operations (e.g., GCN (Kipf & Welling, 2017). GraphSAGE (Hamilton et al., 2017), GAT (Velickovic et al., 2017), GIN (Xu et al., 201 etc.), which are referred to as operation choices for brevity. GNAS addresses this as a bi-level optimization problem (Elsken et al., 2019).
We conduct extensive experiments on eight datasets to demonstrate the superiority of our method over state- of-the-art baselines.
2. Problem Formulation
2.3. GNAS for GNN-based GFMs
In this section, we introduce the fundamental concepts and notations used in this paper, including graph data defini- tion, node of interest (NOD) graph, graph neural architecture search, and GNAS for GNN-based GFMs.
We define diverse data as D = {DI, De...., DN where DD = (9. V.} represents the i-th dataset with graph G; and label )d. Following previous work for
GFMs (Liiu et al., 2023a; Wang et al., 2024b), which leverage
2

---

AutoGFM: Automated Graph Foundation Model with Adaptive Architecture Customization
LLMs unify the node feature space across different graphs and leverage subgraphs unify graph tasks, enabling single GNN to be applied to diverse data D across domains and tasks. Graph neural architecture search for GFM aims to search graph neural architecture Fa,. that achieves performance in diverse data D.
ing proposition with proof in Appendix A.I.
Proposition 3.2. If there exist two datasets Di, Dj D, the optimal operation for D; is different from Dj. the operations will render optimization conflicts.
Assumption 3.1 serves prerequisite condition for Propo- sition 3.2. Proposition 3.2 demonstrates that when two datasets require different optimal architectures, current mainstream GNAS methods encounter optimization con- flicts for GFM. For instance, as illustrated in Figure the optimal architectures for PubMed and Wikics differ. When existing GNAS methods search simultancously for ar- chitecture optimal for both datasets, they fail to identify single architecture that performs best for both and are forced to compromise.
3. Preliminaries
In this section, first introduce the problem of architec- ture inconsistency in GFM. Then we provide d an invariant view of architecture customization and formulate the overall objective for our proposed method.
3.1. Architecture Inconsistency in GFM
tackle architecture inconsistency, our key idea to train mapping function π. 9 - A, which customizes archi- tectures for each data prevent architecture inconsistency, while simultaneously facilitating knowledge sharing across domains and tasks via the weight-sharing supernetwork.
GCN
GAT
Archieeture
GmphSAG2
Rank
3.2. Invariant View of Architecture Customization
Rank
We customize graph neural architectures for graph data across diverse tasks and domains from an invariant perspec- tive. Unlike conventional invariant inference approaches that aim to discover invariant relationships between data and labels (Wu et al., 2022b; et al., 2022a; Wu al., 2022a), our goal is to identify invariant relationships between the graph data and the corresponding architecture, addressing the issue of architecture inconsistency by tailoring graph neural architectures for each data individually.
GrupCoovv
Bank 5
Pubmed
HIV guaph
mode
gruph
Dataset and, Task
Figure 1. A heatmap visualization illustrating that the optimal architecture can vary across datasets with different domains and tasks. The darker the color of a block, the better the performance on the correspouding dataset. Additionally, red indicates the best- performing architectures for each dataset.
We formalize the four key variables: input graph data G. architecture A, invariant pattern Z,, and variant pattern Zv We divide the architecture mapping function a into two components: encoder G + Zz, and predictor + Z2 + A. We make the following assumptions:
According to our observation, the optimal architecture for graph data across different tasks and domains may exhibit architecture inconsistency, meaning that the optimal archi- tectures vary for data with diverse tasks and domains. To validate this, test various GNN architectures built upon a GNN-basee GFM, GFT (Wang et al., 2024b), datasets with different domains and tasks. We present the perfor- mance of each architecture on each dataset using a heatmap in Figure I. As shown in Figure I, datasets from different do- mains and tasks require distinct optimal architectures, high- lighting the presence of architecture inconsistency. Based on this observation, we introduce the following assumption:
Assumption 3.3. (1) Z, = G Zv. There are two disjoint parts in the graph data G: invariant part Z, and variant part Zy. (2) Zy Y A. The variant part Zy is correlated with the architecture A. (3) Zy Z and (Zz)). ) Z₁ shields A from the influence of Zv.
Assumption 3.3 defines what constitutes an invariant pattern for architecture prediction: i) Condition 1 indicates that the data contains two types of patterns: an invariant pattern Z,, which reliably predicts the architecture, and a variant pattern Zv. which cannot stably predict the architecture; ii) Condition 2 highlights that the variant pattern Zy is not independent of the architecture A; iii) Condition states that, given the invariant pattern Z,, the architecture A is independent of the variant pattern Zy. and Z, is sufficient for predicting A.
Assumption 3.1. There exist two datasets D:, Dj E DD, the optimal operation required D, is different from Dj-
Assumption 3.1 assumes that the optimal architectures re- quired by two difffrent datasets may differ. Then we further provide theoretical analyses showing that architecture in- consistency leads to operations optimization conflicts in existing differentiated GNAS methods. We have the follow-
3

---

AutoGFM: Automated Graph Foundation Model with Adaptive Architecture Customization
Different NOI-graph
Disentangled Contrastive Graph Encoder
Invariant-guided Architecture Customization
Customized
Pa Pq. Gt)
Weight-share SuperNet
Architecture
Invariant Predictor
Disen GNN Encoder
Invarianti
2
OCN
GCN
11
Wi
GAT
GAT
Auxiliary Predictor
GIN
GIN
2
Variant
W
SAGE
SAGE
Po (2(x (
Same data
Different data
2
Encourages diversity early
Progressive search process
Curriculum Architecture Customization Mechanism
O O different
minimize the distance
maxmize the distance
The framework of Automated Graph Foundation Model with Adaptive Architecture Customizationí (ttiomii The model consists of three modules: i) Disentangled contrastive graph encoder that discovers invariant and variant patterns from graph data, ii) invariant-guide architecture customization enabling customization of graph neural architectures based the discovered invariant and variant patterns to adapt to data with diverse domains and tasks, and iii) Curriculum architecture customization mechanism to mitigate the influence of any single data dominating the search process.
Figure 2.
3.3. Overall Objective
4.1. Disentangled Contrastive Graph Encoder
To satisfy the constraints outlined in Assumption 3.3, we formulate a learning objective that adheres to the specified conditions. Specifically, we minimize the mutual informa- tion between Z, and Zy ensure that the two parts remain disjoint. Simultancously, we maximize the mutual informa- tion between Z and A, ensuring that Z, is sufficient for predicting the architecture A. Furthermore, we minimize the mutual information between A and Zv. conditioned on Z,. to guarantee that Z, shields from the influence of Zy. The resulting overall learning objective is defined as follows:
In this section, we focus on learning disentangled repre- sentations to capture two distinet aspects of graph data. Specifically, we aim to learn two architecture-aware disen- tangled representations, Z, and Zy, we temporarily treat them jointly and denote the two-channel representations as Z (k 1,2) in this section. The main insight of our proposed method is intuitively based on the following ob- servations: (1) Data from the same sources (i.c., the same domain and task) require similar architectures, so they share a similar Zy that reflects the architectural requirements. Con- versely, graphs from different data sources will have distinct Zr (2) To satisfy the Assumption 3.3, the mutual informa- tion between Z₁ and Zy should be minimized.
max
Zv Zr),
(4)
0,6
where denotes the mutual information function, rep- resents the encoder that extracts Z, and Zy from G, vb represents the predictor that maps 7, to A, and À and are hyperparameters controlling the trade-off.
Disentangled NOI-graph Encoder Initially, we adopt GNNs with individual parameters to learn two-channel graph representations of NOI-graphs.
H = GNN, (n A,,,, k 1
(5)
4. The Proposed Method: AutoGFM
where H1 the k-th channel the node representation
the I-th layer, A is the adjacency matrix of the graph. We employ two distinct Readout functions (i.e., pooling func- tions) and MLPs to derive NOI-graph-level representation for each channel:
In this section, we introduce an Automated Graph Foun- dation Model with Adaptive Architecture Customization (AutoGFM) search for graph neural architectures for each graph data with diverse tasks and domains individually. We first introduce two modules: disentangled contrastive graph encoder and invariant-guide architecture customiza- tion. Besides, we introduce our optimization objective with a curriculum architecture customization mechanism. The overall framework of AutoGFM illustrated in Figure 2.
Z MLPx (hæ), h; Readout (H
(6)
k 1,2.
(7)
NOI-graph Disentangled Contrastive Learning In- spired by self-supervvsed contrastive learning that captures

---

AutoGFM: Automated Graph Foundation Model with Adaptive Architecture Customization
discriminative features by pulling similar samples together and pushing dissimilar samples apart in latent space (Jaiswal et al., 2020; Le-Khae et al., 2020; You et al., 2020), we propose an NOI-gran 101-graph-lever contrastive learning method to encourage disentangled representations to reflect the ar- chitectural requirements of the graph data. Initially, we encourage the representations Z, and Zy to be disentan- gled:
where A is the adjaceney matrix of the graph, H repre- sents the node representations at the 1-th layer, GNN("-1) denotes the mixed GNN operations, |O] is the number of GNN operation choices, and or,, indicates the probability of selecting the i-th operation for the I-th layer. Different from previous super-networks (Liu et al., 2018) that use learnable parameters a, we employ set of prototypes to guide the routing between data and the operations.
exp o (Z P&)
Architecture Predictor Given a graph representation 2 E (8)
Pe (3i G.)
7, we design an invariant mapping predictor, 1/. a + {ous). The probability a1,; of selecting the i-th operation for the I-th layer is calculated as follows:
1== exp @ Ps)
where & is a similarity function, G2 is a NOI-graph from i-th graph, 2g,k is the k-th chunk of the representation of a NOI-graph from i-th graph, and pa is the propotype of the k-th chunk of the representation. Then, we propose NOI-graph-level instance discriminative task to encour- age the representations Z, to capture different architectural requirements of different data. The task is defined as:
exp(ár,s)
Pt,
01,1 E exp(ár.))
On = 2
(12)
|p,:
where Pt,, is learnable prototype of the i-th operation for the I-th layer, and z is the graph representation. Following (Qin et al., 2022a), we adopt the 12 Iz-nommalization on p to ensure numerical stability and fair competition among different operations. We utilize the learnable prototypes p as the parameters of the predictor b to map architecture, i.e., if the graph representation is similar to the prototype Pt, the operation i will be selected for the 1-th layer.
exp Φ 2i 24.
po (s(zix) Gū, 2i,x)
Σ Σソ= exp Φ (z.,ks 24,88
where k(z,,) represents unique surrogate label assigned to 2,, and y is sampled from the same graph data G, as Vi,a- Then we learn the model parameters 0 by calculating the loss function as:
Invariant-guided Customization The objective of min- imizing the conditional mutual information [(A, Zv Z1) in Equation (4) is not tractable, as the mutual information of high sehhdimennional vectors is difficult to estimate. Therefore, we utilize an equivalent transformation in Proposition 4.1 to achieve it, with a detailed proof provided in Appendix A.2.
Lais Σ
-log Ee IGD (s(Zi,)) G., Zi,). (10)
-
In this way, we encourage the representations 7, to capture the architectural requirements of the graph data, while en- suring that the representations! Z, and Zy are disentangled.
Proposition 4.1. if P(A Z,,Zy) = P(A Zz). the conditional mutual information T(A, Zv 7rr) achieves its minimum value 0.
4.2. Invariant-guided Architecture Customization
To customize graph neural architectures based on the dis- covered patterns and enable adaptation to data from diverse domains and tasks, we propose an invariant- guided architec- ture customization approach. Specifically, we first establish a weight-sharing super-network with a set of prototypes, then utilize an invariant predictor, vh,, and an auxiliary pre- dictor, gEg. to guide the customization process, ensuring the minimization of T IAA, Z,) in Assumption 3.3.
Proposition 4.1 indicates that can minimize the con- ditional mutual information I(A, Zv Z,) by enforcing P(A| Zr, Zv) P(A | Zr). To this end, we utilize an auxiliary predictor th with an invariant predictor e to guide the customization process. Specifically, we first predict the architecture A based on the invariant patterns Z, using the invariant predictor dhy Then, we predict the architecture Ag based on the patterns fused from Z, and Zy using the auxiliary predictor DE:
Weight-sharing Super-network To facilitate differen- tiable optimization, employ continuous parameterization and neight-sharing mechanism (Liu et al., 2018) to imple- ment the mixed operations. The super-network layer with [O) mixed operations is defined as:
A, = vI(V),
Ag = (E(ZI,
(13)
We guide P(A Zv, = P(A 2n) by minimizing the difference between the two predictions A,, Ag. Besides, to boost the architecture predictor to fit more data with different variant patterns, we fuse Z, with Zy from other
10
H
ag.
A),
(11)
5

---

AutoGFM: Automated Graph Foundation Model with Adaptive Architecture Customization
data to predict Ag. We define the loss as:
Algorithm Training pipeline for AutoGFM
Input: data D = - .2.......y hyperparameters
I IPII
Lnn ΣΣ |A - AEGsi)
(14)
3.
fort=1,.... do
Sample NOI-graphs G; from G..
s.t. Ar, (Z1,), (E(Z),,
(15)
Extract Z, and Zy from NOI-graphs G,. Calculate Cais using Equation (10).
where Ar. is the predicted architecture for the i-th graph based on the invariant patterns ž7, A are the predicted architectures based on the patterns fused from 21,; and zV---
Obtain architectures A, and Ag predicted by th and DE in Equation (15).
Calculate Line using Equation (14).
Calculate Lour and Lrosk using Equation (18) and Equation (16), respectively.
4.3. Optimization with Curriculum Customization Mechanism
Update 0, v th by minimizing Equation (19).
We calculate the task loss of GFM using only the architec- ture predicted by the invariant predictor why:
end for
Frask ((F(())(G))))
(16)
y is controlled by a pacing function: 7 y= 1 f." where t is the current training step and Le is the step to stop the curricu- lum customization mechanism. This constraint encourages diversity in early architecture customization, thereby miti- gating the influence of any single data dominating the search process. The final training objective is:
frask aim to maximize T(Z,,, A) in Equation (4). Notably, the computation method of Ltask depends on the GFM for which we aim to search for architectures. e.g., GFT (Wang et al., 2024b) utilizes Computation Tree Reconstruction to calculate loss during the pretraining stage.
max (Zz, A)
min I(A, Zv (2a)
GFMs need to be simultaneously optimized using multiple datasets with diverse domains and tasks. However, different data have different influences on architectures (Zhou al., 2022d), mainly on the learnable weights of operations in our study, e-g., some datasets are easier to fit with certain operations but more challenging with others. As a result, operations that fit well in the early stages of training are more likely to be selected, causing other operations to be overlooked.
min 0.91.98
Ltask
A Ldis
+ B Line
+ Leurs
(19)
min (Zz, Zv)
where Liesk aims to exploit invariant patterns customize architectures, Ldis encourages the disentanglement of the invariant and variant patterns, Line discovers the invariant
patterns and variant patterns, and Lour mitigates the influ- ence of any single data dominating the search process. The overall algorithm is summarized in Algorithm 1.
To mitigate the dominance of data in the search process, we design a curriculum architecture customization constraint that encourages diversity in the customized architectures during the early stages of training. We first calculate the average a for each operation in the I-th layer as:
During the inference stage, given an input graph, we first utilize the disentangled contrastive graph encoder obtain its invariant pattern representation, denoted as Z,. Then, Zr is fed into the invariant predictor 1/ to generate a cus- tomized architecture. This architecture is then used as the GNN component within the GFM to perform prediction.
00,1 (2),, EPPl (12(z)) i
[D]
EPI (2))
D
5. Experiments
(17)
In this section, we conduct experiments on real-world datasets with diverse domains and tasks show the ef- fectiveness of the proposed AutoGFM for GFM.
where oy is the average for each operation in the (-th layer, (2,) is the probability of selecting the j-th operation for the (-th layer predicted by the invariant predictor 1 for the i-th graph, and J is the number of operations. We define the curriculum architecture customization loss as follows:
5.1. Experimental Setup
Datasets We employ datasets with diverse domains and tasks. For node-level tasks, we utilize citation networks (Cora, Pubmed, and Arxiv) and the web link network (Wi- kiCS). For edge-level tasks, we utilize Knowledge Graphs (WN18RR, FB15K237). For graph-level tasks, we utilize molecular datasets (HIV, PCBA, and ChEMBL.). Following
Lour
CV
(18)
where (az) is the coefficient of variation of the average a for each operation in the 1-th layer across different data and
6

---

AutoGFM: Automated Graph Foundation Model with Adaptive Architecture Customization
Table Accuracy (%) with std of different methods in pre-training and fine-tuning setting. The highest result is bold. The subscript v represents Vanilla GNNs. We do not explicitly report the performance of GFT separately, as GFT (Wang et al., 2024b) employs GraphSAGE as GNN architecture, which overlaps with our baselines.
Node Classification
Link Classification
Graph Classification
Method
Cora
PubMed
Wiki-CS
Arxiv
WN18RR
FB FB15K237
HIV
PCBA
Avg.
Linear
58.03+233
68.6612.24
70.36e0.8
66.50+a.14
78. 78,50eu.9
87.39
66.37a1.11
32.30
71.01
GCNv
75.65a137
75.61a2.10
75.28a1.34
71.40aa.08
73.79
82.22s0
64. 8424.8
71.3220.49
73.76
GAT
76.24-162
74.86+187
76.78+0.7
70.87+0.24
80. 16H.27
88. 18.93+0.15
65. 46.94
12
75.44
GIN.
73.59a2.10
69.5 16687
49.77
65.05
74.02.0.55
8
66,8643,48
22.69
69.34
DGI
72. 34
73.13
75.32s095
69. 15
75.75
81.34+015
59.62s1.21
63.3 +0
71.22
BGRL
71.20
75.2921.33
76.5320.69
71.1920.18
75, 44.330
80.6620.29
63,9521.06
67.09at.00
72.67
GraphMAE
73. 40
74.32+0.3
77.61
70.90ea.1
78. H4.44
85.30eo
61. 51.005
63.30
73.07
GIANT
75.13:049
72.3 10.,3
76.56e0.88
70. 10a0.2
84,36H0.0
87.45
65.44.1.39
61,49
74.11
GCN
77.97=146
77.68
76.35eo.50
67.2 67.22an80
92,20eu.40
77.35
70.89
74.94 69
76.83
GAT
78.962091
77.24.2.44
78.00s0.67
72.82+0.33
15.91
86.15s2
69.07=1.22
76.2320.61
76.80
GraphSAGE
78.24=146
76,28s2.19
79.29g0.53
72.28
91.57=0.42
89.92s0.7
72.85s2.47
78.322024
79.84
GIN
79.85
77.57x2.04
78.61 60.55
67.80tn.2
17.87
71.37
69.32
74.27
74.58
GraphConv
80.12
76.52a1.39
78.85e0.66
65.75a0.60
91.52a0.33
80.72a2.05
11.81
76.0620.46
77.67
Darts
76.97z134
77.7721.59
73.60s2:00
72.10
78.02a1.52
81.6423.00
68.85
75.16
75.51
Graphnas
76.34x2.25
77.49x2.02
70.98
68, (8.6442.04
82.63a0.44
80.7223.65
67.70s4.50
73.49
74.75
GASSO
78.24
77.82t1.8
71.90 11.9022.00
70.85
81. 1.9660.58
80.66x2.21
68.37
76.76+1.44
75.82
Graces
78.30=192
76.98a2.57
70.05
70.23
84.05
83.91
70.93
75.89s1.4
76.29
Ours
80.32a1.12
78.28
79.45=0.09
73.39
93.17
90.27s1.44
73.17
78.83+1.54
80.86
(Liu et al., 2023a), we use the textual encoder to unify the node features from different domains.
GNN performs well across all datasets, indicating that fixed architectures struggle to generalize across diverse domains and tasks. (2) Existing GNAS methods fail to discover better architectures for each dataset, highlighting their limitations in adapting to diverse domains and tasks. (3) AutoGFM outperforms all baselines across datasets, demonstrating its effectiveness in customizing architectures for different domains and tasks.
Baselines We compare our proposed AutoGFM with the five categories of baselines: (1) Vanilla GNNs: GCN (Kipf & Welling, 2017). GAT (Velickovic et al., 2017), GIN (Xu et al., 2018); (2) Self-supervised methods: BGRL (Thakoor
et al., 2021), GraphMAE (Hou et al., 2022), GIANT (Chien al., 2022). (3) GFMs: OFA (Liu et al., 2023a) and GFT (Wang et al., 2024b). (4) Manually designed GNNs: GCN (Kipf & Welling, 2017), GAT (Velickovie et al., 2017), GIN (Xu et al., 2018), GraphSAGE (Hamilton et al., 2017),
Few-shot Learning Few-shot learning challenging task that requires models to generalize well with limited labeled samples. We randomly sample a few labeled sam- ples per way from the training set for fine-tuning. From the results in Table 2, despite the limited labeled samples, AutoGFM achieves the best performance across all datasets, demonstrating the fast adaptability of its architectures. We provide more experimental results in Appendix B.3.
and GraphConv (Morris et al., 2019). (5) GNAS methods:
DARTS (Liu et al., 2018), GraphNAS (Gao et al., 2021), GASSO (Qin et al., 2021b), Graces (Qin et al., 2022a).
For manually designed GNNs, GNAS baselines, and Au- toGFM, we utilize GFT (Wang et al., 2024b) as the base model to ensure fair comparison. Additionally, we adopt the same search space (operations in manually designed GNNs baselines and super-network layers is 2) for both GNAS baselines and AutoGFM. We replicate each experi- ment ten times and report the average results. Further details about experimental setups are provided in Appendix D.
5.3. Ablation Study
To verify the effectiveness of the key modules in our method, we compare different ablated versions on five datasets: w/o D removes and replaces the disentangled contrastive graph encoder with standard GNNs; ii) w/o removes the invariant evariant-guided architecture customization module by re- moving the Linei iii) w/o C removes the curriculum archi- tecture customization mechanism. The results are shown in Figure 3. We observe that the full model achieves the best performance across all datasets, demonstrating the ef-
5.2. Main Results
Pre-training and Fine-tuning From the results Table I, we observe the following: (1) None of manually designed
7

---

AutoGFM: Automated Graph Foundation Model with Adaptive Architecture Customization
Table 2.
Accuracy (%) with std of different methods in Few-shot learning. The highest result is bold.
Cora-7 way
WN18RR-10 way
CHEMHIV-2
way
Method
1-shot
3-shot
5-shot
I-shot
3-shot
5-shot
1-shot
3-shot
5-shot
10-shot
OFA
30.38
36.03a211
25.82
30,56
32.64
77.17
59.30
57.562306
54.36
GFT
41.40
43.3
43.55
35.33
35.50
35.50.4.0
59.94.7.0
58,44.228
58.78
58.67
GCN
43.07
42.38
42.57
29.85
29.78.3
40.40=302
59,58a615
59.53
59 19.288
59.6427.2
GAT
42.12
47.31
47.71
34.50x298
34.37a34
34.70as15
56.39
59.17
59,33,n,5
59.22xx.m
GraphSAGE
40.50
42.07
42.40
38,03
38. 17.2344
38.30
58.33
58.64 28.64
59.28 $9.28
17.111
GIN
45.29
47.02a7.3
47.24
36.62es 17
36.9 36.92ssus
37. =3.10
58.72a145
57.97
59,06s4.7
57. 37.22s4.8
GraphConv
38.6748.50
40.9315.00
41.60:930
38.93
39.28s2227
39. 39.62.334
53,00
55.50
$4.67
53.06x7.47
DARTS
43,29 43.2997706
10
42.81
37 67.22a208
38.57=3
55
58.31
58.68a.72
58,07=a.7
59,05
GraphNAS
38.6 18.64
40.60
41.62:930
36.79t317
37.03
36.98.s2
57.62as70
58.52.n0s
58.91
59.32.7.7
GASSO
4
42.07
42.95
37.13
37.42at9
37.37
59,46
59,38
59,26ss445
59. 59.7245.42
GRACES
45.43
46.31
47.57
38.76:263
38.24
39.13
59.39amm
58.6229.0
59,45.5.7
59.71
Ours
46.29.7.24
47.33s7.0
47.76sxos
39.34
39.55s244
40.02
59,73
59.68.774
60.08
59.92s1.0
fectiveness of each module. We further observe the follow- ing: The disentangled contrastive graph encoder module is designed extract discriminative invariant and variant patterns from the data by pulling similar samples closer and pushing dissimilar samples apart in the latent space. Removing this module impairs the extraction of invariant patterns and reduces the distinguishability between patterns extracted from different datasets, ultimately harming the effectiveness of architecture prediction; ii) The invariant- guided architecture customization module serves to shield architecture A from the influence of variant patterns Zy given the invariant pattern Z, The substantial performance decrease observed upon removing this module highlights the importance of effectively isolating architecture predic- tions from Zy influences, reinforcing the critical role this module ensuring the invariance conditions of captured patterns; iii) This curriculum architecture customization mechanism aims to reduce data dominance in the architec- ture search process. Removing this module causes certain operations, which perform well on specific datasets during early training stages, to dominate the search process. Con- sequently, other datasets may neglect potentially beneficial operations.
90
Accuracy(%%) (%)
80-
70
Cora
Pubmed
WikiCS FB15K237
PCBA
Full
w/o
w/o I
w/o
Figure 3. Comparisons of different ablated versions of AutoGFM on real-world datasets. "Full" denotes the full version of the method.
all computational complexity of our method is given by: O(E)) + [V|2 + -O + [O](E + +Vld)))
5.4. Time Complexity Analysis
6. Related Work
Let [ and E denote the number of nodes and edges, and d as the dimensionality. We use de and d, to denote the dimensionality of the disentangled graph encoder and the customized super-network. The time complexity of the GNN layers in both the graph encoder and the super-network is O(FFld + [V]2). Therefore, the time complexity of our disentangled graph encoder is O[E) + [V))). The time complexity of the architecture customization with proto- types is (..) The time complexity of the customized super-network is (([O((E) + [V(22). Thus, the over-
In this section, review the related work GNN- based Graph Foundation Models, graph neural architecture search, graph invariant presentation learning, and graph self-supervised learning. We provide more related work in Appendix C.
6.1. GNN-based Graph Foundation Models
GNN- GFMs, which leverage LLMs as enhancers, are a promising direction for GFM (He & Hooi, 2024; Mao et al., 2024a; Zhao et al., 2024b; Xia Huang, 2024; Huang
8

---

AutoGFM: Automated Graph Foundation Model with Adaptive Architecture Customization
al., 2024c). Specifically, it involves using LLMs to trans- form the textual features of graphs into unified representa- tions and unify graph-related tasks through subgraph clas- sification for GNNs (Sun et al., 2023; Zhao et al., 2024c: Fan et al., 2024; Ren et al., 2024; Pan et al., 2024b; Mao et al., 2024b). This enables the transfer of shared knowledge across various domains. Two key challenges in developing GNN-based Graph Foundation Models (GFMs) are the uni- fication of diverse tasks and domain spaces (Yu al., 2024; Li et al., 2024b; Zhao et al., 2024a; Xia et al., 2024; Zhu et al., 2024b; Guo et al., 2023). For instance, OFA (Liu et al., 2023a) employs an LLM to unify input features from diverse datasets, converting multiple graph classification tasks into a unified binary classification format. GFT (Wang et al., 2024b) extends OFA by incorporating computation trees to discover transferable patterns across graphs. Nev- ertheless, these models face challenges in their reliance manually designed architectures, which can constrain their performance on diverse domains and tasks. More related work about GFMs is included in Appendix C.
et al., 2022c) leverages invariant structures and features with stable predictive performance across distribution shifts. However, these methods focus capturing stable relation- ships for accurate label prediction. We apply this concept to architecture search, aiming to define invariant patterns that support stable architecture prediction, and design our method based on this concept.
6.4. Graph Self-supervised Learning
Graph self-supervised learning (SSL) has attracted signifi- cant attention in recent years, with numerous methods pro- posed to learn effective representations from graph data without relying on labeled information. These methods can be broadly classified into two categories: contrastive learning and generative learning. Contrastive learning ap- proaches (You et al., 2020; Hassani & Khasahmadi, 2020; Li et al., 2021; Zhang et al., 2024a; Li et al., 2022d) aim to maximize the agreement between positive pairs of graph samples while minimizing it between negative pairs. In contrast, generative learning approaches (Tan et al., 2023; Xia et al., 2023; Hou et al., 2023) learn representations by reconstructing graph structures or attributes from partially observed data. These SSL techniques have demonstrated strong performance across a variety of graph-related tasks, including node classification, link prediction, and graph classification. In our work, we adopt contrastive learning strategy extract distinct patterns from diverse datasets, enabling the model to capture richer information and better identify architecture-specif requirements.
6.2. Graph Neural Architecture Search
Neural architecture search (NAS) has gained growing at- tention for its capability to automate the design of neural architectures tailored to specific tasks (Pham et al., 2018: 8; Qin et al., 2021a; Wang et al., 2025). In particular, graph neural architecture search (GNAS) methods address the distinet challenge of modeling the intricate relationships between architectures and complex graph structures (Gao et al., 2021; Qin et al., 2022b; Guan et al., 2022; Zhang et al., 2023e; Xie et al., 2023). These methods can be broadly clas- sified into three categories: reinforcement-leannng -basee approaches (Zhou et al., 2022b; Gao et al., 2022: 2023); evolutionary-based strategies (Nunes & Pappa, 2020; Li & King, 2020; Shi et al., 2022: Zhang al., 2022a:b); and differentiable methods (Ding et al., 2021: Zheng et al., 2023; Huan et al., 2021; Zhang et al., 2023b;d; Qin et al., 2023; Yao et al., 2024; Ge et al., 2025), which enable continuous optimization of architectures within a differentiable search space. However, existing GNAS methods are limited in their ability to search for architectures for baseb GFMs.
7. Conclusion
Existing graph neural architecture search methods fail to search for architectures for GNN-based GFMs. In this pa- per, analyze the problem of architecture inconsistency, demonstrate that existing GNAS methods cannot effectively search for architectures for GNN-based based GFMs, and tackle it by discovering an invariant graph-architecture relationship. We propose a novel Automated Graph Foundation Model with Adaptive Architecture Customization (AutoGFM) to search for graph neural architectures for each graph data with diverse tasks and domains individually. We introduce a disentangled contrastive graph encoder to discover invariant and variant patterns from graph data and an invariant-guides architecture customization module to customize graph neu- ral architectures based on the discovered patterns. We also propose a curriculum architecture customization mechanism to mitigate the phenomenon of some particular data domi- nating the search process. Experimental results demonstrate that AutoGFM outperforms existing methods. One limita- tion of our work is that we mainly focus on graph neural architecture search on the GNN-based GFMs, and we leave the exploration of other types of GFMs for future work.
6.3. Graph Invariant Representation Learning
Graph invariant presentation learning has emerged as a pow- erful approach for graph representation learning, focusing on capturing the stable relationships between graph data and tasks. Recent works have explored various applications of generaliza- tion (Ma et al., 2019; Wu et al., 2022b; Li et al., 2022b;c; Zhang et al., 2022c; 2023c; 2024b; Li et al., 2024a). For instance, DIR (Wu et al., 2022b) discovers causal rationales that remain invariant across different distributions while fil- tering out spurious patterns that are unstable. DIDA (Zhang
graph invariant learning in
9

---

AutoGFM: Automated Graph Foundation Model with Adaptive Architecture Customization
Acknowledgements
Fang, Y., Fan, D., Zha, D., and Tan, Q. Gaugllm: Improving graph contrastive learning for text-attributed graphs with large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 747-758, 2024.
This work supported by National Natural Science Foun- dation of China No. 66.62222,0, Beijing National Research Center for Information Science and Technology under Grant
No.BNR2023TD03006.
Fatemi, B., Halcrow, J., and Perozzi, B. Talk like a graph: Encoding graphs for large language models, arXiv preprint arXiv:2310.05560 2023.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Gao, Y. Yang. H., Zhang, P., Zhou, and Hu, Y. Graph and neural architecture search. In International joint confer- ence on artificial intelligence. International Joint Confer- ence on Artificial Intelligence, 2021.
References
Gao, Y., Zhang, P., Yang, H., Zhou, C., Hu, Y., Tian, Z., Li, Z., and Zhou, J. Graphnas++: Distributed architecture search for graph neural networks. IEEE Transactions on Knowledge and Data Engineering, 35(7):6973-6987 2022.
Bengio, Y., Louradour, J., Collobert, R., and Weston, Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009.
Chen, H., Wang, X., Lan, X., Chen, H., Duan, X., Jia, J., and Zhu, W. Curriculum-listener Consistency-and complementarity-awaan
Gao, Y. Zhang, P., Zhou, C., Yang. H., Li, Z., Hu, Y., and Philip, S. Y. Hgnas++: efficient architecture search for heterogeneous graph neural networks. IEEE Transactions on Knowledge and Data Engineering, 35(9):9448-9461 2023.
audio-enhanced temporal sen- tence grounding. Proceedings of the 31st ACM In- ternational Conference on Multimedia, pp. 3117-3128, 2023.
Ge, C.,, Wang, X., Zhang, Z.,, Qin, Y., Chen, H., Wu, H., Zhang, Y., Yang. Y., and Zhu, W. Behavior importance- aware graph neural architecture search for cross-domain recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 1 11708-11716 2025.
Chen, R., Zhao, T. Jaiswal, A., Shah, N., and Wang, Z. Llaga: Large language and graph assistant. arXiv preprint arXiv:2402.081700 2024.
Chen, Y., Wang, X., Fan, M., Huang, J., Yang, S., and Zhu, W. Curriculum meta-learning for next poi recom- mendation. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 2692-2702, 2021.
Gong, C., Yang. J., and Tao, D. Multi-modal curriculum learning over graphs. ACM Transactions on Intelligent Systems and Technology (TIST), 10(4):1-25, 2019.
Chien, E., Chang, W.-C., Hsich, C.-J., Yu, H.-F., Zhang, J., Milenkovic, O., and Dhillon, S. Node feature extraction by self-supervised multi-scale neighborhood prediction. In International Conference on Learning Representations, 2022.
Gong, T., Zhao, Q., Meng, D., and Xu, Z. Why curricu- lum learning & self-paced learning work in big/noisy data: A theoretical perspective. Big Data & Information Analytics, I(I):111-127, 2015.
Guan, C., Wang, X., Chen, H., Zhang, Z., and Zhu, W. Large-scale graph neural architecture search. In Interna- tional Conference on Machine Learning, pp. 7968-7981. PMLR, 2022.
Ding, Y., Yao, Q. Zhao, H., and Zhang, T. Diffin: Differen- tiable meta graph search for heterogeneous graph neural networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp- 2021.
Guo, Y., Yang, C., Chen, Y., Liu, J., Shi, C., and Du, J. data-centric framework to endow graph neural networks with out-of-distribution detection ability. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 638-648, 2023.
Elsken, T., Metzen, J. H., and Hutter, F. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1-21, 2019.
Fan, W., Wang, S., Huang, J., Chen, Z., Song. Y., Tang. W., Mao, H., Liu, H., Liu, X., Yin, D., et al. Graph machine learning in the era of large language models (Ilms). arXiv preprint Xiv:2404. 14928, 2024.
Hamilton, W., Ying, Z., and Leskovec, J. Inductive repre- sentation learning on large graphs. Advances in neural information processing systems, 30, 2017.
10

---

AutoGFM: Automated Graph Foundation Model with Adaptive Architecture Customization
Hassani, K. and Khasahmadi, A. H. Contrastive multi- view representation learning on graphs. In International conference on machine learning, pp. 4116-4126. PMLR, 2020.
Kipf, T. N. and Welling, M. Semi-supervised classifica- tion with graph convolutional networks. In International Conference on Learning Representations, 2017.
Kong, L., Feng, J., Liu, H., Huang, C., Huang. J., Chen, Y., and Zhang, M. Gofa: A generative one-for-all model for joint graph language modeling. arXiv preprint arXiv:2407.099 2024.
He, Y. and Hooi, B. Unigraph: Learning cross-domain graph foundation model from natural language. arXiv preprint arXiv:2402. 13630, 2024.
Hou, Z. Liu, X., Cen, Y., Dong, Y., Yang, H., Wang, C., and Tang, J. Graphmae: Self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 594-604 2022.
Le-Khac, P. H., Healy, G., and Smeaton, A. F. Contrastive representation learning: framework and review. leee Access, 8: 19997771933934 2020.
Li, B., Shen, Y., Wang. Y., Zhu, W., Li, D., Keutzer, K., and Zhao, H. Invariant information bottleneck for domain generalization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7399-7407, 2022a.
Hou, Z., He, Y., Cen. Y., Liu, X., Dong, Y., Kharlamov, E., and Tang, J. Graphmae2: A decoddig-enhanced masked self-supervised graph learner. In Proceedings of the ACM web conference 2023, pp. 737-746, 2023.
Li, H., Wang, X., Zhang, Z., Yuan, Z., Li, H., and Zhu, W. Disentangled contrastive learning on graphs. Advances Neural Information Processing Systems, 34:21872- 21884, 2021.
Hu, W., Fey, M., Zitnik, M., Dong, Y., Ren, H., Liu, B., Catasta, M., and Leskovec, J. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118-22133, 2020.
Li, H., Wang, X., Zhang, Z.,, and Zhu, W. Ood-gnn: Out- of-distribution generalized graph neural network. IEEE Transactions on Knowledge and Data Engineering, 35 (T):332-7730, 2022b.
Huan, Z., Quanming, Y., and Weiwei, T. Search to aggre- gate neighborhood for graph neural network. In 2021 IEEE 37th International Conference on Data Engineer- ing (ICDE), pp. 552-563. IEEE, 2021.
Li, H., Zhang, Z., Wang, X., and Zhu, W. Learning invariant graph representations for out-of nut-of-distribution generaliza- tion. Advances in Neural Information Processing Systems, 35:11828- 1828-11841, 2022c.
Huang, B., He, F., Wang, Q.. Chen, H., Li, G., Feng, Z Wang, X., and Zhu, W. Neighbor does matter: Curriculum global positive-negative sampling for vision-language pre- training. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 8005-8014, 2024a.
Li, H., Wang, X., and Zhu, W. Curriculum graph machine learning: survey. arXiv preprint arXiv:2302.02924. 2023.
Huang, C., Ren, X., Tang, J., Yin, D., and Chawla, N. Large language models for graphs: Progresses and directions. In Companion Proceedings of the ACM on Web Conference 2024b.
Li, H., Wang, X., Zhang, Z., Chen, H., Zhang, Z., and Zhu, W. Disentangled graph self- superviee learning for out- of-distribution generalization. In Forty-first International Conference on Machine Learning, 2024a.
2024, pp.
Huang, J., Zhang, X., Mei, Q., and Ma, J. Can Ilms effec- tively leverage graph structural information: when and why. arXiv preprini 230.:2309.16599 2023.
Li, S.. Wang, X., Zhang, A., Wu. Y., He, X., and Chua, T.-S. Let invariant rationale discovery inspire graph con- trastive learning. In International conference on machine learning, pp. 13052-13065 PMLR, 2022d.
Huang, Q., Ren, H., Chen, P., Kržmanc, G., Zeng, D., Liang, P. S., and Leskovee, J. Prodigy: Enabling in-context learning over graphs. Advances in Neural Information Processing Systems, 36, 2024c.
Li, Y. and King, I. Autograph: Automated graph neural network. In Neural Information Processing: 27th Inter- national Conference, ICONIP 2020, Bangkok, Thailand, November 23-27, 2020, Proceedings, Part 27, pp. 189 201. Springer, 2020
Jaiswal, A., Babu, A. R., Zadeh, M. Z., Banerjee, D., and Makedon, F. A survey on contrastive self-supervised learning. Technologies, 9(1):2, 2020.
Li, Y., Wang, P., Li, Z.., J. and Li, Zerog: Inves- tigating cross-dataset zero-shot transfcrability in graphs. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1725-1735, 2024b.
Jin, B., Zhang, W., Zhang, Y., Meng, Y., Zhang, X., Zhu, Q. and Han, J. Patton: Language model pretraining on text-rich networks. arXiv preprint arXiv:2305.12266. 2023.

---

AutoGFM: Automated Graph Foundation Model with Adaptive Architecture Customization
Liu, H., Simonyan, K., and Yang. Y. Darts: Differentiable ar- chitecture search. In International Conference Learn- ing Representations, 2018.
Pham, H., Guan, M., Zoph, B., Le, Q., and Dean, J. Efficient neural architecture search via parameters sharing. In International conference on machine learning, pp. 4095- 4104. PMLR, 2018.
Liu, H., Feng, J., Kong. L.., Liang, N., Tao, D., Chen, Y., and Zhang, M. One for all: Towards training one graph model for all classification tasks. In The Twelfth International Conference Learning Representations, 2023a.
Qin, Y., Wang, X., Cui, P., and Zhu, W. Gqnas: Graph q network for neural architecture search. In 2021 IEEE International Conference on Data Mining (ICDM), pp. 1288-1293, IEEE, 2021a
Liu, J., Yang, C., Lu, Z.,, Chen, J., Li, Y., Zhang, M., Bai, T., Fang, Y., Sun, L., Yu, S., et al. Towards graph foundation models: A survey and beyond. arXiv preprint arXiv:2310.11829 2023b..
Qin, Y., Wang. X., Zhang, Z., and Zhu, W. Graph dif- ferentiable architecture search with structure learning. Advances in neural information processing systems, 34: 16860-16872, 2021b.
Liu, Z., Yu, X., Fang, Y., and Zhang, X. Graphprompt: Uni- fying pre-training and downstream tasks for graph neural networks, In Proceedings of the ACM Web Conference 2023, pp. 417-428, 2023e.
Qin, Y., Wang. X., Zhang, 7., Xie, P., and Zhu, W. Graph neural architecture search under distribution shifts. In In- ternational Conference on Machine Learning, pp. 18083- 18095. PMLR, 2022a.
Ma, J., Cui, P., Kuang, K., Wang, X., and Zhu, W. Disen- tangled graph convolutional networks. International conference on machine learning, pp. 4212-4221. PMLR, 2019.
Qin, Y., Zhang, Z., Wang, X., Zhang, Z., and Zhu, W. Nas- bench- graph: Benchmarking graph neural architecture search. Advances in neural information processing sys- lems, 35:54-69, 2022b.
Mao, H., Chen, Z., Tang, W., Zhao, J., Ma, Y., Zhao, T., Shah, N., Galkin, M., and Tang, J. Graph foundation models. arXiv preprint arXiv:2402.02211 2024a.
Qin, Y., Wang, X., Zhang, Z., Chen, H., and Zhu, W. Multi- task graph neural architecture search with task-aware collaboration and curriculum. Advances in neural infor- mation processing systems, (124879-24899 2023.
Mao, Q., Liu, Z., Liu, C., Li, Z., and Sun, J. Advancing graph representation learning with large language models: A comprehensive survey techniques. arXiv preprint arXiv:2402.05952. 2024b.
Reimers, N. and Gurevych, I. Sentence-bert: Sentence em- beddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods Natu- ral Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP. IJCNLP), pp. 3982-3992, 2019.
Morris, C., Ritzert, M., Fey, M., Hamilton, W. L., Lenssen, J. F.,, Rattan, G., and Grohe, M. Weisfeiler and leman go neural: Higher-order graph neural networks. Pro- ceedings of the AAAI conference on artificial intelligence, volume 33, pp. 4602-4609, 2019.
Ren, X., Tang, J., Yin, D., Chawla, N., and Huang, C. survey of large language models for graphs. In Proceed- ings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. pp. 6616-6626, 2024.
Nunes, M. and Pappa, G. L. Neural architecture search in graph neural networks. In Intelligent Systems: 9th Brazilian Conference, BRACIS 2020, Rio Grande, Brazil, October 20-23, 2020, Proceedings, Part I 9. pp. 302-317. Springer, 2020.
Shi, M., Tang. Y., Zhu, X., Huang, Y., Wilson, D., Zhuang, Y., and Liu, J. Genetic- Genetic-gnn: gnn: Evolutionary architecture search for graph neural networks. Knowledgeebased sys- tems, 247:108752, 2022.
Pan, B., Zhang, Z., Zhang, Y., Hu, Y., and Zhao, L. Distilling large language models for attt-bttributed graph learning. arXiv preprint arXiv:2402. 22.4022.2222 /0022, 2024a.
Sun, X., Cheng, H., Li, J., Liu, B., and Guan, J. All in one: Multi-task prompting for graph neural networks. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2120-2131, 2023.
Pan, S.. Zheng, Y., and Liu, Y. Integrating graphs with large language models: Methods and prospects. IEEE Intelligent Systems, 39(1):64 (111))6466.) 68, 2024b.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Tan, Q., Liu, N., Huang, X., Choi, S.-H., Li, L., Chen, R., and Hu, X. S2gae: Self-supervieed graph autoencoders are generalizable learners with graph masking. In Pro- ceedings of the sixteenth ACM international conference on web search and data mining, pp. 787-795, 2023.
12

---

AutoGFM: Automated Graph Foundation Model with Adaptive Architecture Customization
Tang, J., Yang, Y., Wei, W., Shi, L., Su, L.., Cheng. S., Yin, D., and Huang, C. Graphgpt: Graph instruction tuning for large language models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 491-500, 2024a.
Wu, Y.-X., Wang, X., Zhang, A., He, X., and Chua, T.- S. Discovering invariant rationales for graph neural net- works. arXiv preprint arXiv:2201.12872 2022b.
Xia, L. and Huang, C. Anygraph: Graph foundation model in the wild. 2024.
Xia, L., Huang, C., Huang, C.,, Lin, K., Yu, T., and Kao, B. Automated self-supervised learning for recommendation. In Proceedings of the ACM web conference 2023, pp. 992-10002 2023.
Tang, J., Yang, Y. Wei, W., Shi, L.,, Su, I.., Cheng, S., Yin, D., and Huang. C. Graphgpt: Graph instruction tuning for large language models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 491-500, 2024b.
Xia, L.,, Kao, B., and Huang, C. Opengraph: To- wards open graph foundation models. arXiv preprint arXiv:2403.01121 2024.
Thakoor, S., Tallec, C., Azar, M. G., Azabou, M., Dyer, E. L., Munos, R., Veličković, P., and Valko, M. Large- scale representation learning on graphs via bootstrapping. arXiv preprint arXiv:2102.066 04, 2021.
Xie, B., Chang, H., Zhang, Z., Wang, X., Wang, D., Zhang, Z., Ying, R., and Zhu, W. Adversarially robust neural architecture search for graph neural networks. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8143-8152, 2023.
Tian, Y., Song, H., Wang, Z., Wang, H., Hu, Z., Wang, F,, Chawla, N. V., and Xu, P. Graph neural prompt- ing with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19080-19088, 2024.
Xu, K., Hu, W., Leskovec, J., and Jegelka, S. How powerful are graph neural networks? arXiv preprint arXiv: MeXiv:18/00 1ate:81000000 1810.00826, 2018
Velickovic, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y., al. Graph attention networks. stat, 1050 (20):10-48550) 2017.
Xu, Y., Liu, X., Duan, K., Fang, Y. Chuang, Y.-N. Zha, D., and Tan, Q. Graphfm: A comprehensive benchmark for graph foundation model. arXiv preprint arXiv:2406.08. 08310, 2024.
Wang, H., Zhou, K., Zhao, X., Wang, J., and Wen, J.-R. Curriculum pre-training heterogencous subgraph trans- former for top-n recommendation. ACM Transactions on Information Systems, 41(1):1-28, 2023.
Yao, Y., Wang. X., Qin, Y., Zhang. Z., Zhu, W., and Mei, H. auta-engeented curriculum graph neural architecture search under distribution shifts. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 16433-16441, 2024.
Wang, H., Feng, S., He, T., Tan, Z., Han, X., and Tsvetkov, Y. Can language models solve graph problems in natural language? Advances in Neural Information Processing Systems, 36, 2024a.
Ye, R., Zhang, C., Wang, R., Xu, S., Zhang, Y., al. Natural language is all graph needs. arXiv preprint 07144, 2023.
Wang, X., Li. H., Zhang, Z., Chen, H., and Zhu, W. Mod- ular machine learning: An indispensable path towards new-generation large language models. arXiv preprint arXiv:2504.20020, 2025.
You, Y., Chen, T., Sui, Y., Chen, T. Wang, Z., and Shen, Y. Graph contrastive learning with augmentations. Advances in neural information processing systems, 33:5812-5822 2020.
Wang, Z., Zhang, Z. Chawla, N. V., Zhang, C., and Ye, Y. Gft: Graph foundation model with transferable tree vocabulary. arXiv preprint TtV::4/11.00770 2024b.
Yu, X., Qu, M., Feng, X., and Qin, B. Graphagent: Exploit- ing large language models for interpretable learning on text-attributed graphs.
Wei, X., Gong, X., Zhan, Y., Du B., Luo, Y., and Hu, W. Clnode: Curriculum learning for node classification. In WSDM, pp. 670-678, 2023.
Yu, X., Zhou, C., Fang, Y., and Zhang, X. Multigprompt for multi-tash pre-training and prompting on graphs. In Proceedings of the ACM on Web Conference 2024, pp. 515-526, 2024.
Wu, Q., Zhang, H., Yan, J., and Wipf, D. Handling distribu- tion shifts on graphs: An invariance perspective. arXiv preprint ar Xrxx:2202.022466 2022a.
Zhang, S., Hu, Z., Subramonian, A., and Sun, Y. Motif- driven contrastive learning of graph representations. IEFE Transactions on Knowledge and Data Engineering, 36 (8)5406340075, 2024a.
Wu, Y., Yao, J., Xia, X., Yu, J., Wang, R., Han, B., and Liu, T. Mitigating label noise on graph via topological sample selection. arXiv preprint arXiv:2403.0/99 2024.
13

---

AutoGFM: Automated Graph Foundation Model with Adaptive Architecture Customization
Zhang, W., Lin, Z., Shen, Y., Li, Y., Yang, Z., and Cui, B. Deep and flexible graph neural architecture search. In International Conference Machine Learning, pp. 26362 22374, PMLR, 2022a.
Zhao, J., Zhuo, L., Shen, Y., Qu, M., Liu, K., Bronstein, M., Zhu, Z.., and Tang, Graphtext: Graph reasoning in text space, arXiv preprint arXiv:2310.01089 2023.
Zhao, J., Mostafa, H., Galkin, M., Bronstein, M., Zhu, Z., and Tang, J. Graphany: A foundation model for node clas- sification on any graph. arXiv preprint arXiv:2405 20445, 2024b.
Zhang, W., Shen, Y., Lin, Z., Yi, Tao, Y., Yang, Z., and Cui, B. Pasca: A graph neural architecture search system under the scalable paradigm. In Proceedings of the ACM Web Conference 2022, pp. 1817-1828, 2022b.
Li, X., Ouyang, W.,
Zhao, Z., Li, Y., Zou, Y., Li, R., and Zhang, R. sur- vey on self-supervised pre-training of graph foundation models: A knowledge-based perspective. arXiv preprint arXiv:2433.66137 2024c.
Zhang, Z., Wang, X., Zhang, Z., Li, H., Qin, Z., and Zhu, W. Dynamic graph neural networks under tentoral distribution shift. Advances in neural information pro- cessing systems, 35:6074-6089, 2022c.
Zheng, X., Zhang, M., Chen, C.,, Zhang, Q., Zhou, C., and Pan, S. Auto-heg: Automated graph neural network on heterophilic graphs. In Proceedings of the ACM Web Conference 2023, pp. 611-620, 2023.
Zhang, Z., Zhang, Z., Wang, X., and Zhu, W. Learning to solve travelling salesman problem with hardness-adaptive curriculum. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 9136-9144, 2022d.
Zhou, D., Zheng, L., Fu, D., Han, J., and He, J. Mentorgnn: Deriving curriculum for pre-training gnns. In CIKM, pp. 2721-2731, 2022a.
Zhang, Z.,
Wang, J., and Zhao, L. Relational curriculum learning for graph neural networks, 2023a. URL https: / /ooenneeview net wwweroooomddddddirooriereeoeeeee
Zhou, K., Huang, X., Song, Q. Chen, R., and Hu, X. Auto- gnn: Neural architecture search of graph neural networks. Frontiers in big Data, 5:1029307, 2022b.
Zhang, Z., Wang, X., Guan, C., Zhang, Z., Li, H., and Zhu, W. Autogt: Automated graph transformer architecture search. In The Eleventh International Conference on Learning Representations, 2023b.
Zhou, Y., Chen, H., Pan, Z., Yan, C., Lin, F., Wang, X., and Zhu, W. Curml: A curriculum machine learning library. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 7359 77633, 2022c.
Zhang. 7., Wang, X., Zhang, Z., Qin, Z., Wen, W., Xue,
H., Li, H., and Zhu, W. Spectral invariant learning for dynamic graphs under distribution shifts. Advances in Neural Information Processing Systems, 36:6619-6633. 2023c.
Zhou, Y., Wang, X., Chen, H., Duan, X., Guan, C., and Zhu, W. Curriculum-nas: Curriculum weight-sharing neural architecture search. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 2022d.
Zhang, Z., Wang, X., Zhang, Z., Shen, G., Shen, S., and Zhu, W. Unsupervised graph neural architecture search with disentangled self-supervision. Advances in Neu- ral Information Processing Systems, 36:73175-73190, 2023d.
Zhou, Y., Wang, X., Chen, H., Duan, X., and Zhu, W. Intra- and inter modal curriculum for multimodal learning. In Proceedings of the 31 ACM International Conference on Multimedia, pp. 3724-3735, 2023.
Zhang, Z., Zhang, Z., Wang, X., Qin, Y., Qin, Z., and Zhu, W. Dynamic heterogeneous graph attention neural archi- tecture search. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 11307-11315, 2023e.
Zhou, Y., Pan, Z., Wang, X., Chen, H., Li, H., Huang, Y., Xiong, Z., Xiong, F., Xu, P., Zhu, W., et Curbench: cur- riculum learning benchmark. In Forty-first International Conference on Machine Learning, 2024.
Zhang, Z., Wang, X., Chen, H., Li, H., and Zhu, W. Dis- entangled dynamic graph attention network for out-of- distribution sequential recommendation. ACM Transac- tions on Information Systems, 43(1):1 122, 2024b.
Zhu, Y., Wang, Y., Shi, H., and Tang. S. Efficient tuning and inference for large language models on textual graphs. arXiv preprint 2rXiv:2401.15560 2024a.
Zhu, Y., Wang, Y., Shi, H., Zhang, Z., Jiao, D., and Tang, S. Graphcontrol: Adding conditional control to univer- sal graph pre-trained models for graph domain transfer learning. In Proceedings of the ACM on Web Conference 2024, pp. 339-550, 2024b.
Zhao, H., Chen, A., Sun, X., Cheng, H., and Li, J. All in one and one for all: A simple yet effective method towards cross-domain graph pretraining. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4443 4454, 2024a.
14

---

AutoGFM: Automated Graph Foundation Model with Adaptive Architecture Customization
A. Proof
A.1. Proof of Proposition 3.2
Proposition 3.2: there exist two datasets D,, Dj E D, the optimal operation for D; is different from Dj, the operations will render optimization conflicts.
Proof. Assume the optimal operation for D; is on and the optimal operation for Dj is oz. The overall architecture function is defined as: F(G) = oifi(G)) + o2) The MSE loss function (F) is given by:
Cau(F) = + (F)),, k i
(20)
where =x M.A =x (F(G,N) u5,))
We examine the effects of changes 01 and 02 on the two terms (FFF) and (F) by calculating the partial derivatives of the two terms with respect to 01 and 02: 8C, ah,, aL,, and ac,
doy dog Do1 dog
first simplify LF) as follows:
(((F) Σ (F(GL)) yt.k)
(21)
Σ ((Isi(GG))) + (2f((Ga))))) vr,k)
(22)
[loff(GGia++o (((,,)))
(23)
E ((1 o2)f +2 (G
(24)
of Σ - f(G.x))))
(25)
(1 01)3 Σ (fa(Ga.)) -
(26)
Equation (22) expands &(G as F(G.,)) (Cfi(G()) o2f2(Grx)) In Equation (23), Vi,k is replaced with fi (G
because the optimal operation for D; is 01, i.e., Vi, (i In Equation (24), or is replaced with 1 02 due the constraint 01 0 1. Equation (25) simplifies the equation. Similarly, for (F):
(,(P)= = of
of Σ -
= -1-oz)) Σ (fa(Gr,)) ()
(27)
Then we examine the effects of changes in 0₁ on the two terms LFF and (F) by calculating the partial derivatives of ac, ac,
the two terms with respect to o₁: do, do:
C. (1 (1-) Σ -
(28)
Cj = of Σ (Je(G,,)) -i((Gn)))
(29)
We calculate ac, ac, follows: 80, Bo,
15

---

AutoGFM: Automated Graph Foundation Model with Adaptive Architecture Customization
OC: (201 22 Σ (Fe(Grt) <0
(30)
do
JLj 201 [(2(G)z)) - [1(G,,,))
(31)
d01
01, 02 E (0, 1). and fi # f2 so that Ex - and En (fe(G(xx) ((,)))))) are both positive. There- fore, UC 0 and 0L1 do1 > 0. To minimize the loss function Lis must be increased, whereas minimizing the loss function
Lj requires decreasing on Similar results about can be obtained for UC. ac, Therefore, the operation optimization objective for D; is different from Dy..e., the operations will render optimization conflicts.
Ø0z Doz
A.2. Proof of Proposition 4.1
Proposition 4.1: if P(A Zr, Zv) P(A Zr), the conditional mutual information [AA, Zv (rr achieves its minimum value of 0: I(A,Z) Zv Zr) 0
Proof. The conditional mutual information [A,, Zv
Zz) is defined as:
P(A. Zv Z1) T(A,Zy 2,) = Ez, E EA.ZviZ, log P(A Z, P(Zy Zr)
11
(32)
Based P(A Z,, Zv) P(A Zz), obtain P(A,Zv [1) P(A Z,)P(Zv 1y) as follows:
P(A L1, Lv) = P(A ZT),
(33)
P(A 7r, 7v) (Zy (7,)) P(A 7.) 77);
(34)
P(A, Zv [7.) P(A P7Z 77))-
(35)
Substituting the conditional independence into the definition conditional mutual information, we obtain:
P(A P/) Zr)]
T(A, Zv 2,) = Ezz Ez,
EA,Zv 1Z,
log P(A P(Z 77))
[A 12, [log 1J]
=0.
(36)
Therefore, when P(A Zv, P(A 7.), the conditional mutual information T(A, Zv 77) achieves its minimum value of 0. This implies that, given Z,, there is no additional dependence between A and Zy
B. More Experiments
B.1. Architecture Visualizations
To clearly visualize the customized architectures tailored to different datasets, we presented heatmap in Figure illustrating the choice weights of each operation at each layer. Firstly, observe that different graph datasets prefer distinet architectures; for example, Cora mainly prefers GraphConv and GraphSAGE, whereas these two operations rarely selected for PubMed. This observation further supports our earlier assumption that different datasets require different architectures, and some datasets exhibit inconsistent architectural preferences. Moreover, we find that many datasets prefer varying operations across different layers. For instance, the Arxiv dataset prefers GCN in the first layer and GAT in the second layer. Such fine-grained architectural preferences are challenging to meet through manual design, highlighting the advantage of automated, customized architectures.
B.2. Hyperparameters Analysis
We analyze the sensitivity of the important hyperparameters À and 3 in our method on the WikiCS dataset. We adjust X 3 E (1e le 2, 3, le 4), while maintaining the default value of the other hyperparameters unchanged.
16

---

AutoGFM: Automated Graph Foundation Model with Adaptive Architecture Customization
1.0
GralhSACE
GCN
Operatoo
0.5
0.4
0.2
GraphCanv
0.0
Cona-1
Cora-2 Pubmed-IPubmed-22 Wikiis-1 Wikies-2 Anxi-l
Arxis-2
WN-1
WX-2
FB-1
FB-2
PCBA-I PCBA-2
HIVI
HIV-2
Dataset and Layer
Figure 4. A showcase heatmap displays customized architectures for different datasets, where the number following the dataset name represents the architecture's layer.
81
Accom
79
78
77
0.1
0.01
0.001
0.0001
78
0.1
0.01
0.001
0.0001
Figure 5. Hyperparameter sensiiitiiyy analysis on WikiCS dataset. The blue lines denote the results our method and the red dashed lines are the results of the best baseline.
The results shown in Figure 5. The hyperparameter À in Equation (19) controls the trade-off between Ltask and Lais- Specificallyy, Least aims maximize the mutual information between the invariant pattern Z₁ and the architecture A, ensuring that 7, is sufficient to predict A. In contrast, Lais aims to minimize the mutual information between the invariant pattern Z, and the variant pattern Zy. thereby enabling the extraction of two disjoint patterns from the data. We adjust its value within the set (1c - 1, 1e 2, le 3, 1c - 4). As shown in Figure 5, when À is set too low, the model's performance deteriorates, confirming that proper disentanglement of Z, and Zy essential for effective architecture prediction. Conversely, when À is set too high, performance also declines, indicating that while ensuring the separation between the two patterns, is equally important that the Z, retains sufficient information to predict the architecture. Overall, is important hyperparameter for balancing the sufficiency and disentanglement. The hyperparameter 3 in Equation (19) controls the trade-off between Trtakk and Trine- Specifically, Time aims to shield architecture A from the influence of Zvv given the invariant pattern Z,. As demonstrated in Figure 5, setting 3 too low results in degraded model performance, underscoring the importance of effectively shielding A from the influence of Zv given Z,. Thus, B is also critical hyperparameter for balancing the sufficiency and invariance conditions of the patterns captured by the model.
B.3. More Few-shot Learning results
We conduct more N-way K-shot experiments on the few-shot learning task. The results are shown Table and Table 4. Our method outperforms the baselines mostly across various N-way K-shot settings, further verifying the effectiveness of the customized architectures.
17

---

AutoGFM: Automated Graph Foundation Model with Adaptive Architecture Customization
Table Accuracy (%) with std of different methods on Cora under N-way K-shot settings. The highest result is bold.
7-way
5-way
2-way
Method
5-shot
3-shot
1-shot
5-shot
3-shot
1-shot
5-shot
3-shot
1-shot
OFA
32.10
36.03a21
30.38
42.28
31.28
23.68
72.20
62.22.1.77
51.85
GFT
43.55
4 +8.11
41.40
+2.30+6.57
51.47++6.3
49.80+679
75.00+4.8
76.33
74.92+464
GCN
42.38 42.57
43.07
45.00+7.4
47.5
44.80+894
71.58++5.00
71.42++4.03
70.58=4m
GAT
2802
47.31 2777
46. 12.7.10
52.302660
51.732732
50.172741
75.922389
75.17
42.83
GraphSAGE
42.40 12
42.07
40.5022611
51.17
50.80.15.4
49.50
74.20
74.33
72.08=589
GIN
47.34
47.02.1732
45.29 16.26
49,83
49.17
48.97
75.25 18.00
76.83
71.50
GraphCony
41.60.19.0
40.93 18:00
38.67 18.50
46, 19.775
11..0
42.57 18.65
67.75.1984
67.00....45
63.42
Darts
4..91
42.10
43,29 1::5
49.23 17769
49.50.1..1
46.97
71.67.143
71.42
69.58
GraphNAS
41.62=9900
40.60=923
38. 38.64-4831
50.07+666
50.13
47.30
71.83+421
73.17
68.00=s.0
GASSO
42.95=575
42.07
40.31 16.10
49.53++6.0
50.87+6610
47.53+7.40
71.08+6622
70.92#526
68.00=441
GRACES
47.57:272
46.31 1:742
45.43
50.17
49.3011612
49.40.66.0
14.81
74.422547
72.582490
Ours
47.76_8066
47.33
46.29 1::34
53.93
52.5016s4
50.87.158
76.431s4s
76.55.4..
73.922664
Table Accuracy (%) with std of different methods on WN18RR under N-way K-shot settings. The highest result bold.
10-way
5-way
3-way
Method
5-shot
3-shot
1-shot
5-shot
3-shot
1-shot
5-shot
3-shot
1-shot
OFA
32.64+1.56
30.56=10z
25.82=107
48.32#3.19
45.04+239
34,40+147
60.72+3иz
61.29=256
51.77+2.6
GFT
35.50
35.502502
35.33=4220 20
48.80
48.53
48.1324.1
62.562271
60.67
58.44
GCN
30.40
29.78+344
29.85
44.70
44.97 +:.95
44,77 +:345
54.06#536
53.33
53.28
GAT
34.70
34.37=348
34.50=2.9
46.2332444
46.33
46.30
59.56=335
59.39=3445
58.062434
GraphSAGE
38.30+2.6
38. 17:334
38.03=203
48. 10 10=3.7
47.83
47.90
62.39
61.442114
59.39
GIN
37.47
36.92.1405
36.62.4.17
47.57 15:56
47.80
47.6013.81
61.33 15.98
61.831x33
58.22.4490
GraphConv
39.62+3.4
39.28
38.93=33
48.40=333
47.63
43.07
61.39
60.06=3.60
59.33
Darts
38.651355
38.57
37.22
47.40
46.0319.81
46,43
60.17
18.78
57.8913.82
GraphNAS
36.98+3.23
37.03
36.79=317
46.07 #3.64
47.07
45,87
58.56#5540
60.50
57.11
GASSO
37.37
37.422191
37.13
47.90 14
47.77
46.20
59.61 2:44
59.83as.0
57.83
GRACES
39.13
38.24
38.76-244
48.37
47.67
47.00
61.50
60.00
59.17
Ours
40.02 2..26
39.55
39.34 3,03
49.93
49.10
48.47 14.38
63.11 +
61.94:261
59.72
C. More Related Works
C.1. LLM-based Graph Foundation Models.
LLM-based Graph Foundation Models employ LLMS as predictors within a unified generative framework for graph tasks (Liu et al., 2023c; Pan et al., 2024a; Fang et al., 2024; Jin et al., 2023; Zhu et al., 2024a; Yu et al.; Huang et al., 2023; 2024b). For instance, InstructGLM (Ye et al., 2023) employs generative framework in which LLMs predict node labels by generating them based on the nodes' textual attributes. GraphGPT (Tang et al.. 2024b) adapts LLMs for downstream graph tasks through instruction tuning, integrating natural language with a graph-text aligner to capture and convey structural graph information. These approaches present a promising direction for the development of GFMs, as LLMs can seamlessly unify the output of various graph tasks. Unlike GNNs, which require task-speeific adjustments for model training, LLMs can accept wide range of queries and generate appropriate responses. However, a key challenge lies in effectively translating graph structures into format that LLMs can interpret. Current research tackles this problem with two primary approaches. The first involves describing the graph structure using natural language (graph to text) (Zhao et al., 2023; Fatemi et al., 2023; Zhao et al., 2023; Wang et al., 2024a). The second approach draws inspiration from Visual Language Models (VLMs), where the graph is first processed into embeddings using GNNs or projectors (graph to token), and an LLM then decodes the graph embeddings (Chen et al., 2024: Tian et al., 2024; Tang et al., 2024a). These methods demonstrate competence in fundamental reasoning tasks such as connectivity checks and cycle detection, but struggle with more complex graph tasks
18

---

AutoGFM: Automated Graph Foundation Model with Adaptive Architecture Customization
that require capturing intricate graph patterns, such graph classification.
C.2. Graph Curriculum Learning
Curriculum learning is a training strategy that involves presenting training data meaningful order, typically starting with simpler examples and gradually progressing more complex ones. This approach has been shown improve the performance of various machine learning models (Bengio al., 2009; Gong et al., 2015; Li et al., 2023; Chen al., 2021:?; Zhang et al., 2022d; Zhou et al.. 2022c;d; Chen et al., 2023; Zhou et al., 2023; Huang al., 2024a; Zhou al., 2024). Graph curriculum learning (GCL) is different from traditional curriculum learning due to the inherent dependencies of graph data(Gong et al., 2019; Zhou et al., 2022a; Wang et al., 2023; Wu et al., 2024). Researchers leverage graph structures to measure difficulty through predefined or automated strategies. For instance, CLNode(Wei et al., 2023) is a Curriculum Graph Learning method that measures local difficulty by considering the class diversity among a node's neighbors and uses global features to identify mislabeled nodes. RCh(nh al., 2023a) gradually integrates node relationships into the training process, based the complexity of those relationships. We utilize the concept of curriculum learning to enhance the architecture search process, mitigating the dominance of specific data on the search process.
D. Experimental Setup
D.I. Dataset
Table 5. Dataset statisties (Liu et al., 2023a).
Dataset
Domain
Task
# Graphs
Avg. #Nodes
Avg. #Edges
# Classes
Cora PubMed
Citation
Node
2,708
10,556
Citation
Node
19,717
44,338
Arxiv
Citation
Node
169,343
1,166,243
40
WikiCS
Web link
Node
11,701
216,123
10
FB15K237
Knowledge
Link
14,541
310,116
237
WN18RR
Knowledge
Link
40,943
93,003
11
PCBA
Molecule
Graph
437,929
26.0
28.1
128
HIV
Molecule
Graph
41,127
25.5
27.5
2
CHEMBL
Molecule
Graph
365,065
25.9
55.9
1,048
Dataset Statistics. We follow the preprocessing method described in (Liu et al., 2023a; Wang et al., 2024b), employing the Sentence Transformer (Reimers & Gurevych, 2019) convert raw textual descriptions of nodes and edges into 768- dimensional features. For knowledge graphs (KGs), we do not transform edge textual information into edge features, the existing textual information already provides sufficient knowledge for KG completion (Wang et al., 2024b). The statistics of the datasets are detailed in Table 5.
Dataset Splitting, We adopt the same splitting strategy as (Liu et al., 2023a; Wang et al., 2024b). For Cora and PubMed select 20 labeled nodes per class for training. We utilize predefined set of 10 splits with different random seeds to compute the average performance. For WikiCS, we report the average accuracy over 20 distinct training splits, each generated with 20 different random seeds. In each split, 5% of the nodes from each class are used for training. For Arxiv, HIV, and PCBA, we employ the official dataset splits and conduct experiments 10 times using different random seeds to determine the average accuracy. The FB15K237 dataset consists of 272,1 15 edges in the training set, 17,535 edges in the validation set, and 20,466 edges in the test set. Meanwhile, for WN18RR, the corresponding numbers are 86,835, 3,034, and 3,134, respectively. Each experiment is repeated 10 times with different random seeds, and the final results are reported as the average accuracy.
D.2. Baseline
We compare our proposed AutoGFM with the following baselines and provide brief description of each method:
Vanilla GNNs
GCN (Kipf & Welling, 2017): Graph Convolutional Networks (GCN) utilizes graph convolutional layers to learn node representations by aggregating information from neighboring nodes.
-
19

---

AutoGFM: Automated Graph Foundation Model with Adaptive Architecture Customization
GAT (Velickovic et al., 2017): Graph Attention Networks (GAT) attention mechanisms weigh the importance of neighboring nodes when updating node representations.
- GIN (Xu et al., 2018): Graph Isomorphism Network (GIN) employs a sum aggregation function and a learnable
MLP to update node representations, aiming to achieve maximum discriminative power among graph structures.
Self-super GNNs
BGRL (Thakoor et al., 2021): BGRL leverages bootstrap-styl contrastive learning approach without negative samples, maximizing agreement between online and target networks over augmented graph views.
- GraphMAE (Hou et al., 2022): GraphMAE employs masked autoencoders to learn node representations by reconstructing masked node features from the input graph.
-GIANT (Chien al., 2022): GIANT is self-supervised GNN framework that performs multi-granularity contrastive learning across multiple graphs to learn generalizable node representations.
GFMs
OFA (Liu et al., 2023a): OFA a GNN- foundation model that integrates LLMs process textual features across different domains and unifies graph-related tasks through subgraph classification.
-FFT (Wang et al., 2024b): GFT is GNN-based foundation model that incorporates computation trees to identify transferable patterns across graph structures, enabling the learning of generalizable representations for various graph domains and tasks.
Manually designed GNNs
GraphSAGE (Hamilton et al., 2017): GraphSAGE learns node representations in inductive manner by sampling and aggregating features from node's local neighborhood using various aggregation functions such as mean LSTM.
GraphConv (Morris et al., 2019): GraphConv is generalized graph convolutional operator that combines node features with their neighbors' features using trainable transformation, capturing local structure in the graph.
GNAS methods
DARTS (Liu et al., 2018): DARTS is differentiable architecture search framework that relaxes the search space into a continuous domain, enabling efficient gradient-based optimization of neural architectures.
-GraphNAS (Gao al., 2021): GraphNAS applies reinforcement learning to search for optimal GNN architectures by modeling the architecture design process as sequential decision miisiinnmmking problem.
- -GASSO (Qin et al., 202 GASSO enables differentiable architecture search via gradient descent and discovers more effective graph neural architectures by incorporating graph structure learning as a denoising process during the search procedure.
Graces (Qin et al., 2022a)) Graces achieves generalization under distribution shifts by designing instance-specific GNN architectures tailored to the unknown distribution of each graph.
For Vanilla GNNs, self-supervised methods, and GFMs, we reproduce the results based on their original papers and publicly available code. To ensure fair comparison between manually designed GNNS and GNAS basclines, we employ GFT (Wang al., 2024b) as the base model. Specifically, for manually designed GNNs, we replace the GNN in GFT with various manually designed GNNs and follow identical pretraining and finetuning procedures GFT. For GNAS methods, we substitute the GNN component in GFT with different GNAS methods. Architecture search is performed during the pretraining stage, whereas in the finetuning stage, we further optimize only the parameters of the searched architectures without additional architecture searches.
D.3. Hyperparameters
We evaluate different GNN architectures and GNAS methods based on GFT (Wang et al., 2024b), following the default hyperparameters of GFT to maintain consistency. To ensure a fair comparison, we set the dimensionality of all methods to 768, use the same search space and operations (GCN, GIN, GAT, GraphSAGE, GraphConv), and fix the number of layers to For our method, we explore hyperparameter X, 3 (le 1, le 2, le 3, le 4} and empirically select À and J The
20

---

AutoGFM: Automated Graph Foundation Model with Adaptive Architecture Customization
learning rate of the disentangled contrastive graph encoder is set 5e 3, and the learning rate of the architecture predictor set to 3e 2. The dimensionality of both the graph encoder and the supernet is 768. Each experiment is conducted 10 times, and report the average performance along with standard deviations.
D.4. Configurations
We conduct all experiments the following configurations:
Operating System: Ubuntu 20.04.5 LTS
CPU: Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz
GPU: NVIDIA A100-SXM4-4000 and NVIDIA A100-SXM4-80BB
Software: Python 3.9, CUDA 12.2, PyTorch (Paszke et al., 2019) 1.13.1.
21


Reviews:
[{"review_id": "5wscrFBZhE", "reviewer": "Reviewer_QHCx", "summary": "This paper explores automated graph neural architecture search (GNAS) for Graph Foundation Models (GFMs) to overcome the limitations of fixed, hand-designed GNN architectures, which result in suboptimal performance across diverse graph domains and tasks. The authors identify the architecture inconsistency problem, where the optimal GNN architectures vary across different domains and tasks. To tackle this, they propose an Automated Graph Foundation Model with Adaptive Graph Neural Architecture Customization (GFA), which incorporates: a disentangled contrastive graph encoder to learn both invariant and variant patterns from graph data, an invariant-guided architecture customization strategy to adapt GNN architectures to different domains and tasks, and a curriculum architecture customization mechanism to mitigate the dominance of particular data during the search process. Additionally, the paper provides theoretical insights into the limitations of existing GNAS methods under the architecture inconsistency problem. Extensive experiments demonstrate that GFA outperforms baseline models, achieving state-of-the-art performance. This work is the first to address the problem of GNAS for GFMs.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere gratitude to the reviewer for providing us with detailed comments and insightful questions. We have carefully considered the reviewer's feedback and would like to address each point as follows.\n\n> 1. \"Could you clarify whether the assumptions in the theories can be valid in the real world?\" \n\nThank you for raising this important point. **Assumption 3.1** serves as a prerequisite condition for **Proposition 3.2**. Specifically, **Assumption 3.1** assumes that optimal architectures required by two different datasets may differ. To validate this assumption, we evaluated various GNN architectures based on a GNN-based GFM (GFT [1]) across multiple real-world datasets. Figure 1 presents a heatmap visualization of each architecture’s performance on different datasets, showing that optimal architectures indeed vary according to the dataset. For instance, GCN achieves optimal performance on the PubMed, while GraphSAGE performs best on the Wikics.\n\n**Assumption 3.3** defines what constitutes an invariant pattern for architecture prediction. The concept of the invariant pattern is well-defined, and methods based on this concept have been validated as effective in various real-world applications, such as academic citation networks [2] and molecular structures [3]. Unlike previous work, which has primarily focused on capturing stable relationships for accurate label prediction, we applied this concept to architecture search, aiming to define invariant patterns that support stable architecture prediction, and designed our method based on this concept.\n\n>  2. \"Could you add more details on the baselines? How do you implement the baselines? The current descriptions are too simple.\"\n\nThank you for bringing up this point. For **Vanilla GNNs**, **self-supervised methods**, and **GFMs**, we reproduce the results based on their original papers and publicly available code. To ensure a fair comparison between **manually designed GNNs** and **GNAS** baselines, we employ GFT[1] as the base model. Specifically, for **manually designed GNNs**, we replace the GNN in GFT with various manually designed GNNs and follow identical pretraining and finetuning procedures as GFT. For **GNAS** methods, we substitute the GNN component in GFT with different GNAS methods. Architecture search is performed during the pretraining stage, whereas in the finetuning stage, we further optimize only the parameters of the searched architectures without additional architecture searches. Furthermore, we use the same search space for both GNAS baselines and GFA, including operations such as GCN, GAT, GraphSAGE, GIN, and GraphConv within a super-network depth of 2 layers. We set the dimensionality of all methods to 768.\n\n>  3. \"More recent advances in GFMs [1] could be relevant references. [1] Graph Foundation Models: Concepts, Opportunities and Challenges. ArXiv:2310.11829\".\n\nThank you for your suggestion. We will add this citation to the related work section of the revised manuscript.\n\n> 4. \"The writing can be improved. For example, more details in Sec. D.2 should be added for reproducing the results. The descriptions on the baselines are too simple, which brings difficulty to the readers that do not very familiar with this topic.\"\n\nThank you for your suggestion. We have expanded the description in Sec.D.2 as follows.\n\nWe evaluate different GNN architectures and GNAS methods based on GFT[1], following the default hyperparameters of GFT to maintain consistency. To ensure a fair comparison, we set the dimensionality of all methods to 768, use the same search space and operations (GCN, GIN, GAT, GraphSAGE, GraphConv), and fix the number of layers to 2.  For our method, we explore hyperparameter $\\lambda,\\beta \\in \\\\{1e-1,1e-2,1e-3,1e-4\\\\}$ and empirically set $\\lambda$ to $1e-3$ and $\\beta$ to $1e-1$. The learning rate of the disentangled contrastive graph encoder is set to $5e-3$, and the learning rate of the architecture predictor is set to $3e-2$. The dimensionality of both the graph encoder and the supernet is 768. Each experiment is conducted 10 times, and we report the average performance along with standard deviations.\n\nWe will also include a new section in the appendix with detailed descriptions of all the baselines.\n\n> 5. \"Minors: There are some typos in the manuscript. The corresponding symbols (periods or commas) after equations are missing in Eq. (4) (5) (14) (15) or wrong in Eq. (16). Initial letters should be capitalized in line 434.\"\n\nWe sincerely appreciate your thoughtful observation. We have corrected these typos and will carefully revise the manuscript to address any remaining errors.\n\n[1] GFT: Graph Foundation Model with Transferable Tree Vocabulary\n\n[2] Learning Invariant Representations of Graph Neural Networks via Cluster Generalization\n\n[3] Learning Invariant Molecular Representation in Latent Discrete Space", "claims_and_evidence": "The main point that fixed GNN architectures lead to suboptimal performance in diverse settings is justified through both theoretical analysis and empirical validation.", "methods_and_evaluation_criteria": "The proposed method is evaluated on multiple datasets and makes sense.", "theoretical_claims": "The theoretical analysis is rigorous. The authors present a formulated argument demonstrating the optimization conflicts caused by architecture inconsistency in existing methods. The proof of Proposition 3.2 convincingly shows that a one-size-fits-all architecture search approach is insufficient for diverse graph tasks. Proposition 4.1 provides a justification for the proposed invariant-guided architecture customization strategy. But I would say that the theoretical analysis is based on some assumptions. It should be discussed whether these assumptions can be true in the real world.", "experimental_designs_or_analyses": "The experimental setup is comprehensive, covering multiple datasets and a set of baseline models, including manually designed GNNs, existing GNAS methods, and state-of-the-art Graph Foundation Models.", "relation_to_broader_scientific_literature": "The work fits within GNAS and GNN literature.", "essential_references_not_discussed": "The discussion of related work is comprehensive. More recent advances in GFMs [1] could be relevant references.\n[1] Graph Foundation Models: Concepts, Opportunities and Challenges. \tArXiv:2310.11829.", "other_strengths_and_weaknesses": "Strengths:\n-\tThe paper presents an important and novel problem—GNAS for Graph Foundation Models (GFMs)—which has not been previously explored. The identified architecture inconsistency problem is a significant contribution to the field.\n-\tThe proposed GFA framework systematically addresses key challenges in GNAS for GFMs. The design of disentangled contrastive learning, invariant-guided customization, and curriculum-based customization is innovative.\n-\tThe paper provides a theoretical analysis of the limitations of existing GNAS methods under the architecture inconsistency problem.\n-\tThe empirical results demonstrate the effectiveness of GFA, showing state-of-the-art performance across multiple benchmarks.\nWeaknesses:\n-\tThe author did not explain whether the assumptions can be valid in the real world.\n-\tThe writing can be improved. For example, more details in Sec. D.2 should be added for reproducing the results. The descriptions on the baselines are too simple, which brings difficulty to the readers that do not very familiar with this topic.", "other_comments_or_suggestions": "As shown above, the authors should talk about the validity of the assumptions used in the method. Why do the authors need the Assumption 3.1 and Assumption 3.3? Are these assumptions common in the literature?\nMinors: There are some typos in the manuscript. The corresponding symbols (periods or commas) after equations are missing in Eq. (4) (5) (14) (15) or wrong in Eq. (16).  Initial letters should be capitalized in line 434.", "questions_for_authors": "1) Could you clarify whether the assumptions in the theories can be valid in the real world?\n2) Could you add more details on the baselines? How do you implement the baselines? The current descriptions are too simple.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D7TQOOG6Ip", "reviewer": "Reviewer_Dusn", "summary": "The authors introduce GFA, a framework for graph neural network architecture customization in graph foundation models. The paper addresses the architecture inconsistency problem, which arises when different graph domains and tasks require varying GNN architectures. To tackle this, the authors propose a disentangled contrastive graph encoder, an invariant-guided architecture customization strategy, and a curriculum-based optimization mechanism to improve architecture search for diverse datasets. They conduct extensive experiments across real-world datasets, showing that GFA outperforms state-of-the-art baselines. Furthermore, the paper provides theoretical analysis demonstrating the limitations of existing graph neural architecture search methods in handling architecture inconsistency. In summary, the paper presents a novel approach to automated GNN-based GFMs with architecture search.\n\n## update after rebuttal\n\nI will keep my positive opinion towards the paper after rebuttal.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the insightful comments provided by the reviewer. We have carefully considered each point raised and would like to respond as follows.\n\n> 1. \"The experiment should consider additional way settings beyond the current specific configurations.\"\n\nThank you for raising this important point. We have conducted additional experiments to validate our model’s performance under more N-way settings on the Cora and WN18RR datasets. Due to character limits, we only include a subset of the most competitive baselines below. The complete results will be provided in our second-round response and included in the revised manuscript.\n\n|Cora||5 way|||2-way||\n|-|-|-|-|-|-|-|\n||5-shot|3-shot|1-shot|5-shot|3-shot|1-shot|\n|GAT|52.30±6.05|51.73±7.32|50.17±7.41|75.92±3.89|75.17±5.36|72.83±5.48|\n|GIN|49.83±7.79|49.17±8.10|48.97±6.73|75.25±8.60|**76.83±8.36**|71.50±7.44|\n|GRACES|50.17±7.74|49.30±6.12|49.40±6.20|74.81±5.82|74.42±5.47|72.58±4.90|\n|Ours|**53.93±6.95**|**52.50±6.84**|**50.87±5.55**|**76.43±5.45**|76.55±4.48|**73.92±6.64**|\n\n\n|WN18RR||5 way|||3-way||\n|-|-|-|-|-|-|-|\n||5-shot|3-shot|1-shot|5-shot|3-shot|1-shot|\n|GAT|46.23±4.44|46.33±4.50|46.30±4.43|59.56±3.85|59.39±3.45|58.06±4.34|\n|GIN|47.57±5.56|47.80±5.29|47.60±3.81|61.33±5.98|61.83±6.35|58.22±4.93|\n|GRACES|48.37±3.76|47.67±4.04|47.00±3.15|61.50±3.31|60.00±4.01|59.17±6.35|\n|Ours|**49.93±3.63**|**49.10±3.31**|**48.47±4.38**|**63.11±5.80**|**61.94±2.61**|**59.72±4.26**|\n\nOur method outperforms the baselines mostly across various N-way K-shot settings, further verifying the effectiveness of the customized architectures.\n\n> 2. \"additional intuitive explanations could make the theoretical arguments more readable for a broader audience.\"\n\nThank you for your suggestion. We provide some additional intuitive explanations below.\n\n**Assumption 3.1** assumes that the optimal architectures required by two different datasets may differ. As shown in Figure 1(the performance of each architecture on different datasets), GCN achieves optimal performance on the PubMed dataset, while GraphSAGE performs best on the Wikics dataset.\n\n**Assumption 3.1** serves as a prerequisite condition for **Proposition 3.2**.  \n**Proposition 3.2** demonstrates that when two datasets require different optimal architectures, current mainstream GNAS methods encounter optimization conflicts for GFM. As previously illustrated, the optimal architectures for PubMed and Wikics differ. Consequently, when existing GNAS methods search simultaneously for an architecture optimal for both datasets, they fail to identify a single architecture that performs best for both and are forced to compromise.\n\n**Assumption 3.3** defines what constitutes an invariant pattern for architecture prediction. \n- **Condition 1** indicates that the data contains two types of patterns: an invariant pattern $Z_I$, which reliably predicts the architecture, and a variant pattern $Z_V$, which cannot stably predict the architecture.\n- **Condition 2** highlights that the variant pattern $Z_V$ are not independent of the architecture $A$.\n- **Condition 3** states that, given the invariant pattern $Z_I$, the architecture $A$ is independent of the variant pattern $Z_V$, and $Z_I$ is sufficient for predicting $A$.\n\n**Proposition 4.1** aims to demonstrate that our method satisfies condition 3 of an invariant pattern for architecture prediction via enforcing  $P(A \\\\mid Z_{I}, Z_{V}\\) = P(A \\\\mid Z_{I})$. \n\n> 3. \"the paper does not discuss how hyperparameter sensitivity affects performance thoroughly.\"\n\nThank you for bringing this to our attention. Due to character limitations, we will provide a more detailed discussion on hyperparameter sensitivity in our revised manuscript. Alternatively, we kindly refer the reviewer to our response to **Reviewer YeCs’s Question 3**, where we address a similar question. We apologize for any inconvenience this may cause.\n\n> 4. \"The authors should provide more detailed descriptions of the baseline methods.\"\n\nWe appreciate your suggestion. We will include a new section in the appendix providing detailed descriptions of all the baselines.\n\n> 5. \"The framework diagram (Figure 2) should be revised to clarify the technical details of the proposed GFA.\"\n\nThank you for highlighting this point. We have refined our framework by incorporating additional technical details to better align with the content and equations presented in the Method section, specifically in the following three aspects.\n\n(1) Enhancing the clarity of the pipeline by adding more numerical indexing and clear directional arrows to improve readability.\n\n(2) Including additional equations directly in the figures, enabling readers to easily identify corresponding equations from the text. \n\n(3) Reflecting the unique advantages of our method, such as the customization of different architectures tailored to specific datasets, and the progressive architecture search process driven by curriculum learning.", "claims_and_evidence": "The main claim of the paper is that customizing GNN architectures according to different datasets improves GFM performance. This claim is supported by empirical evidence, including evaluations on eight diverse datasets. The paper effectively shows that fixed architectures used in prior GFMs lead to suboptimal performance, while GFA dynamically adapts architectures, achieving state-of-the-art results. A key contribution is the theoretical analysis that demonstrates why standard differentiable GNAS methods struggle under architecture inconsistency. This analysis, backed by proofs in Appendix, provides strong theoretical motivation for the proposed approach. \nHowever, while the theoretical insights are technically solid, they could be presented more intuitively to help reader understanding.", "methods_and_evaluation_criteria": "The methodology is convincing. The three core modules, disentangled contrastive graph encoder, invariant-guided customization, and curriculum-based optimization, are integrated to address architecture inconsistency issue. The experimental design is comprehensive, covering datasets across node, edge, and graph classification tasks. The authors compare against multiple baselines, including vanilla GNNs, self-supervised learning methods, existing GFMs, and various GNAS techniques.", "theoretical_claims": "The theoretical aspect of this paper addresses why standard differentiable GNAS methods struggle to manage architectural inconsistencies across heterogeneous graph datasets. The authors provide a series of theorems (with proofs in Appendix) showing how prior methods might overlook essential dataset-specific structures when optimizing a universal, shared architecture parameter space. The derivations appear mathematically solid. Nonetheless, additional intuitive explanations could make the theoretical arguments more readable for a broader audience.", "experimental_designs_or_analyses": "The experimental results are strong, showing improvements over baselines. However, the few-shot experiments in table 2 seem to be confusing (I will explain it in weaknesses). Also, the paper does not discuss how hyperparameter sensitivity affects performance thoroughly.", "relation_to_broader_scientific_literature": "The work aligns with ongoing research in graph LLMs and graph foundation models.", "essential_references_not_discussed": "No more works should be referenced.", "other_strengths_and_weaknesses": "I think the paper did a good job at these points:\n\n-\tIt proposes a novel, end-to-end approach for GNN architecture customization in GFMs.   \n-\tIt shows theoretical analysis exposing limitations of existing GNAS methods.  \n-\tIt integrates theoretical, empirical, and algorithmic innovations into a unified framework.\n\nBut I still have some concerns on the paper:\n\n-\tOne concern relates to the “N-way K-shot” few-shot learning experiments. The authors explore different way settings (Cora-7 way, WN18RR-10 way, CHEMHIV-2 way), but it is unclear why these specific values of N were chosen for each dataset. A clearer justification would be beneficial.\n-\tAnother concern is on the hyperparameter sensitivity. The paper does not discuss how hyperparameter sensitivity affects performance thoroughly.\n-\tAnd there are limited descriptions of the baseline methods.", "other_comments_or_suggestions": "-\tThe experiment should consider additional way settings beyond the current specific configurations.\n-\tThe authors should provide more detailed descriptions of the baseline methods.\n-\tThe framework diagram (Figure 2) should be revised to clarify the technical details of the proposed GFA.", "questions_for_authors": "How about the performance in additional way settings as in table 2? Does the method still perform best in these new few-shot settings?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S9EUnjG290", "reviewer": "Reviewer_Qn8E", "summary": "The paper introduces a framework for adapting GNN architectures dynamically to improve generalization in GFMs. Existing graph neural architecture search methods struggle to design architectures for GNN-based GFMs. This paper addresses the issue of architecture inconsistency by identifying an invariant relationship between graphs and architectures. The authors propose GFA, an automated approach that tailors GNN architectures to different graph datasets, tasks, and domains.\n\nThe key contributions of the paper include: \n\n- Proposed a disentangled contrastive graph encoder to extract invariant and variant patterns from graph data.  \n- Proposed an invariant-guided architecture customization strategy to tailor GNN architectures in a dynamic way.  \n- Proposed a curriculum-based architecture customization mechanism to mitigate the effects of data domination during the search process.  \n- Provided theoretical insights demonstrating the limitations of existing GNAS methods in handling architecture inconsistency.  \n- Conducted extensive experiments on multiple datasets showing that GFA outperforms baseline methods.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere appreciation to the reviewer for providing us with detailed suggestions. We have carefully reviewed each comment and offer the following responses.\n\n> 1. \"Can you explain what claims you want to support with the showcase Figure 4?\"\n\nThank you for highlighting this point. To clearly visualize the customized architectures tailored to different datasets, we presented a heatmap in Figure 4, illustrating the choice weights of each operation at each layer. Firstly, we observe that different graph datasets prefer distinct architectures; for example, Cora mainly prefers GraphConv and GraphSAGE, whereas these two operations are rarely selected for PubMed. This observation further supports our earlier assumption that different datasets require different architectures, and some datasets exhibit inconsistent architectural preferences. Moreover, we find that many datasets prefer varying operations across different layers. For instance, the Arxiv dataset prefers GCN in the first layer and GAT in the second layer. Such fine-grained architectural preferences are challenging to meet through manual design, highlighting the advantage of automated, customized architectures.\n\n> 2. \"How does the computational complexity of GFA compare to classical GNNs and standard GNAS methods?\"\n\nWe are grateful for your feedback. In the original manuscript, we analyzed the complexity of GFA as ${O}(|E|d_e +|V|d_e^2 +|\\\\mathcal{O}|^2d_e +|\\\\mathcal{O}|(|E|d_a +|V|d_a^2))$. Considering the term with the largest complexity, this is approximately ${O}(|\\\\mathcal{O}|(|E|d +|V|d^2))$. The complexity for most existing GNN methods is typically ${O}(|E|d +|V|d^2)$, and GNAS methods also exhibit a approximate complexity of ${O}(|\\\\mathcal{O}|(|E|d +|V|d^2))$. Thus, our method's complexity is comparable to existing GNAS approaches.\n\n> 3. \"Could you show the pipeline of GFA duration the inference stage?\"\n\nThank you for highlighting this point. During the inference stage, given an input graph, we first utilize the Disentangled Contrastive Graph Encoder to obtain its invariant pattern representation, denoted as $Z_I$. Then, $Z_I$ is fed into the Invariant Predictor within the Invariant Guided Architecture Customization module to generate a customized architecture. This customized architecture is subsequently employed as the GNN component within the GFM to perform prediction.\n\n> 4. \"Ablation studies seem to validate the effectiveness of the proposed method. But the analyses on the results are a little limited.\"\n\nThank you for bringing this to our attention. We provide a further analysis of the ablation studies below.\n\n(i) The disentangled contrastive graph encoder module is designed to extract discriminative invariant and variant patterns from the data by pulling similar samples closer and pushing dissimilar samples apart in the latent space. Removing this module impairs the extraction of invariant patterns and reduces the distinguishability between patterns extracted from different datasets, ultimately harming the effectiveness of architecture prediction.\n\n(ii) The invariant-guided architecture customization module serves to shield architecture $A$ from the influence of variant patterns $Z_V$ given the invariant pattern $Z_I$. The substantial performance decrease observed upon removing this module highlights the importance of effectively isolating architecture predictions from $Z_V$ influences, reinforcing the critical role of this module in ensuring the invariance conditions of captured patterns.\n\n(iii) This curriculum architecture customization mechanism aims to reduce data dominance in the architecture search process. Removing this module causes certain operations, which perform well on specific datasets during early training stages, to dominate the search process. Consequently, other datasets may neglect potentially beneficial operations.\n\n> 5. \"The discussions on the results shown in the figures or tables could be enhanced.\"\n\nThank you for your suggestion. We will add more discussions on the results shown in the figures and tables in our revised manuscript to make our paper more readable.\n\n> 6. \"The framework figure 2 is a little simple and did not clearly show the technical details of the proposed method.\"\n\nThank you for highlighting this point. We have refined our framework by incorporating additional technical details to better align with the content and equations presented in the Method section, specifically in the following three aspects.\n\n(1) Enhancing the clarity of the pipeline by adding more numerical indexing and clear directional arrows to improve readability.\n\n(2) Including additional equations directly in the figures, enabling readers to easily identify corresponding equations from the text.\n\n(3) Reflecting the unique advantages of our method, such as the customization of different architectures tailored to specific datasets, and the progressive architecture search process driven by curriculum learning.", "claims_and_evidence": "The primary claim that adaptive architectures enhance performance across various graph settings is supported by theoretical analysis and empirical results. The paper provides mathematical proofs demonstrating why existing GNAS methods struggle with architecture inconsistency, emphasizing the need for dynamic customization. Additionally, experimental evaluations on diverse datasets show significant improvements over both manually designed GNNs and existing GNAS approaches. Ablation studies seem to validate the effectiveness of the proposed method. But the analyses on the results are a little limited.", "methods_and_evaluation_criteria": "The methodology is clearly presented and builds upon solid foundations in NAS and GNN customization. The experimental setup/evaluation is comprehensive. Table 1 covered all common graph tasks, like node classification, link prediction and graph classification.", "theoretical_claims": "The theoretical contributions are valuable. The proofs are correct after my careful check. The authors provide mathematical formulations and proofs that highlight the architecture inconsistency problem in existing GNAS methods. Their theoretical analysis shows that under the assumption that different datasets require distinct architectures, differentiable GNAS methods (e.g., DARTS) fail due to optimization conflicts. My verification confirms the correctness of these proofs, and the proposed invariant-guided architecture customization is a theoretically sound solution to this problem.", "experimental_designs_or_analyses": "The evaluation is thorough, but the study would benefit from detailed ablation experiments to analyze the impact of various components of the proposed method.", "relation_to_broader_scientific_literature": "The work follows the recent trends in adaptive neural architecture design and graph foundation model.", "essential_references_not_discussed": "I did not locate essential references missing.", "other_strengths_and_weaknesses": "Strengths:   \n\n- Strong empirical validation with extensive experiments across multiple datasets. \n- Novel use of curriculum learning to mitigate data domination effects in GNAS.  \n- Theoretical insights into architecture inconsistency and its impact on GNAS.\n- Comprehensive experimental design, including ablation studies and few-shot learning evaluation.\n\nWeaknesses: \n\n- The discussions on the results shown in the figures or tables could be enhanced.\n- The framework figure 2 is a little simple and did not clearly show the technical details of the proposed method.\n- The descriptions on the inference stage are missing.", "other_comments_or_suggestions": "It would be useful to expand the discussion on Figure 4 and explain what key insights can be drawn regarding how different architectures adapt to various datasets.", "questions_for_authors": "1. Can you explain what claims you want to support with the showcase Figure 4? What important points can be derived from the showcase? I am not very clear to that.\n\n2. How does the computational complexity of GFA compare to classical GNNs and standard GNAS methods?\n\n3. Could you show the pipeline of GFA duration the inference stage? The current algorithm and method description only focus on the training stage.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fb0Dx9Ep9O", "reviewer": "Reviewer_YeCs", "summary": "This paper introduces an automated graph foundation model with adaptive graph neural architecture customization. The authors address the architecture inconsistency problem in graph foundation models. The proposed method consists of graph encoder, architecture customization, and curriculum training. The theoretical analysis and empirical results on multiple datasets seem to show the effectiveness of the proposed approach.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere gratitude to the reviewer for the detailed comments and insightful questions. We respond to each of the reviewer’s comments point by point as follows. \n\n> 1. \"Clarify the hyperparameter tuning process and discuss more on ablation studies.\"\n\nThank you for bringing this to our attention. Regarding the hyperparameter tuning process, we pretrain the GFA model with $\\lambda,\\beta \\in \\\\{1e-1,1e-2,1e-3,1e-4\\\\}$. Subsequently, we fine-tune these pretrained models and evaluated their performance on the validation set to empirically determine the hyperparameters.\n\nAdditionally, we provide a further analysis of the ablation studies below.\n\n(i) The disentangled contrastive graph encoder module is designed to extract discriminative invariant and variant patterns from the data by pulling similar samples closer and pushing dissimilar samples apart in the latent space. Removing this module impairs the extraction of invariant patterns and reduces the distinguishability between patterns extracted from different datasets, ultimately harming the effectiveness of architecture prediction.\n\n(ii) The invariant-guided architecture customization module serves to shield architecture $A$ from the influence of variant patterns $Z_V$ given the invariant pattern $Z_I$. The substantial performance decrease observed upon removing this module highlights the importance of effectively isolating architecture predictions from $Z_V$ influences, reinforcing the critical role of this module in ensuring the invariance conditions of captured patterns.\n\n(iii) This curriculum architecture customization mechanism aims to reduce data dominance in the architecture search process. Removing this module causes certain operations, which perform well on specific datasets during early training stages, to dominate the search process. Consequently, other datasets may neglect potentially beneficial operations.\n\n\n> 2. \"How does the proposed method compare in terms of computational efficiency with other NAS-based approaches?\"\n\nWe are grateful for your feedback. In the original manuscript, we analyze the time complexity of GFA as ${O}(|E|d_e +|V|d_e^2 +|\\\\mathcal{O}|^2d_e +|\\\\mathcal{O}|(|E|d_a +|V|d_a^2))$. Considering the term with the largest complexity, this is approximately ${O}(|\\\\mathcal{O}|(|E|d +|V|d^2))$. The time complexity for most existing GNN methods is typically ${O}(|E|d +|V|d^2)$, and GNAS methods also exhibit a approximate complexity of ${O}(|\\\\mathcal{O}|(|E|d +|V|d^2))$. Thus, our method's complexity is comparable to existing GNAS approaches.\n\n\n> 3. \"However, the choice of hyperparameters for different datasets is not well discussed, and sensitivity analysis should be enhanced.\"\n\nThank you for your suggestion. Since GFA is trained jointly across all datasets, the hyperparameters $\\lambda$ and $\\beta$ are consistent for all datasets. Specifically, we set $\\lambda$ to $1e-3$ and $\\beta$ to $1e-1$. \n\nWe provide a more detailed discussion on the hyperparameter sensitivity. The hyperparameter $\\lambda$ in Eq.17 controls the trade-off between $L_{task}$ and $L_{dis}$. Specifically, $L_{task}$ aims to maximize the mutual information between the invariant pattern $Z_I$ and the architecture $A$, ensuring that $Z_I$ is sufficient to predict $A$. In contrast, $L_{dis}$ aims to minimize the mutual information between the invariant pattern $Z_I$ and the variant pattern $Z_V$, thereby enabling the extraction of two disjoint patterns from the data. We adjust its value within the set $\\\\{1e-1,1e-2,1e-3,1e-4\\\\}$. As shown in Figure 5, when $\\lambda$ is set too low, the model's performance deteriorates, confirming that proper disentanglement of $Z_I$ and $Z_V$ is essential for effective architecture prediction. Conversely, when $\\lambda$ is set too high, performance also declines, indicating that while ensuring the separation between the two patterns, it is equally important that the $Z_I$ retains sufficient information to predict the architecture. Overall, $\\lambda$ is an important hyperparameter for balancing the sufficiency and disentanglement. The hyperparameter $\\beta$ in Eq.17 controls the trade-off between $L_{task}$ and $L_{inv}$. Specifically, $L_{inv}$ aims to shield architecture $A$ from the influence of $Z_V$ given the invariant pattern $Z_I$. As demonstrated in Figure 5, setting $\\beta$ too low results in degraded model performance, underscoring the importance of effectively shielding $A$ from the influence of $Z_V$ given $Z_I$. Thus, $\\beta$ is also a critical hyperparameter for balancing the sufficiency and invariance conditions of the patterns captured by the model.\n\n> 4. \"more recent advances in self-supervised learning techniques for GNN adaptation.\"\n\nWe sincerely appreciate your valuable suggestion. In response, we will include a paragraph in the Related Works section of our revised manuscript, discussing recent advances in self-supervised learning techniques for GNN adaptation.", "claims_and_evidence": "The claims regarding architecture inconsistency and the benefits of the proposed method are grounded with theoretical justifications and experimental comparisons. The authors demonstrate that fixed architectures underperform on diverse datasets and propose a method to adapt architectures dynamically.", "methods_and_evaluation_criteria": "The proposed methodology is sound, leveraging contrastive learning and mutual information constraints. The evaluation benchmarks are relevant, with a reasonable selection of baselines. However, the choice of hyperparameters for different datasets is not well discussed, and sensitivity analysis should be enhanced.", "theoretical_claims": "The theoretical claims, particularly the propositions regarding architecture inconsistency, are mathematically sound.", "experimental_designs_or_analyses": "The experimental setup is comprehensive, covering multiple datasets and comparisons. The results indicate superior performance.", "relation_to_broader_scientific_literature": "The paper builds on prior work in graph foundation models and GraphNAS, providing a novel adaptation mechanism. The references are mostly relevant.", "essential_references_not_discussed": "The paper could benefit from citing more recent advances in self-supervised learning techniques for GNN adaptation.", "other_strengths_and_weaknesses": "A key strength is the novel combination of graph foundation models and GraphNAS, which opens a new direction on this topic.", "other_comments_or_suggestions": "Clarify the hyperparameter tuning process and discuss more on ablation studies.", "questions_for_authors": "How does the proposed method compare in terms of computational efficiency with other NAS-based approaches?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes to use automated graph neural architecture search (GNAS) for Graph Foundation Models (GFMs) on diverse graph domains and tasks. To find an invariant graph-architecture relationship across domains and tasks, it presents an Automated Graph Foundation Model with Adaptive Graph Neural Architecture Customization (GFA). GFA consists of an encoder to distinguish invariant and variant patterns, an invariant-guided architecture customization strategy, and a curriculum architecture customization mechanism. Theoretical analysis and experiments illustrate the superiority of the proposed model. The claims are supported by clear and convincing evidence. The proposed method is novel and makes sense. This seems to be the first work to explore the problem of GNAS for GFMs. Thus, it is inspiring and possesses a high impact on the field. After the rebuttal, most concerns raised by reviewers were alleviated, and most reviewers raised their ratings.
Decision: Accept (oral)