GraphFM: A generalist graph transformer that learns transferable representations across diverse domains
Authors: Divyansha Lachi
Mehdi Azabou
Vinam Arora
Eva L Dyer
Venue: Accepted by TMLR ()

Abstract: Graph neural networks (GNNs) are often trained on individual datasets, requiring specialized models and significant hyperparameter tuning due to the unique structures and features of each dataset. This approach limits the scalability and generalizability of GNNs, as models must be tailored for each specific graph type. To address these challenges, we introduce GraphFM, a scalable multi-graph pretraining approach designed for learning across diverse graph datasets. GraphFM uses a Perceiver-based encoder with learned latent tokens to compress domain-specific features into a shared latent space, enabling generalization across graph domains. We propose new techniques for scaling up graph training on datasets of different sizes, allowing us to train GraphFM on 152 distinct graph datasets, containing a total of 7.4 million nodes and 189 million edges. This allows us to study the effect of scale on pretraining across domains such as molecules, citation networks, and product graphs, and show that training on diverse datasets improves performance over single-source pretraining. Additionally, pretraining with a mixture of synthetic and real graphs enhances adaptability and stability, leading to competitive performance with state-of-the-art models across various node classification tasks. This approach reduces the burden of dataset-specific training and provides a single generalist model capable of performing across multiple diverse graph structures and tasks. Code is available at https://github.com/nerdslab/GraphFM.

Published in Transactions on Machine Learning Research (12/2025)
GraphFM: A generalist graph transformer that learns trans- ferable representations across diverse domains
Divyansha Lachi
dinl @upennn.cee edu
University of Pennsylvania Philadelphia, Pennsyloania
Mchdi Azabou
ma4 ma.76660cohumbiic edu
Columbia University
New York, New York
Vinam Arora
cinam@upennte edu
University of Pennsyleania Philadelphia, Pennsyloania
Eva Dyer
dyer @upeenn. edu
University of Pennsyloania Philadelphia, Pennsyloania
Reviewed on OpenReview: https: // openrevie.. net/ forum? id= S2TpRfRUER
Abstract
Graph neural networks (GNNs) are often trained on individual datasets, requiring specialized models and significant hyperparameter tuning due to the unique structures and features of each dataset. This approach limits the scalability and generalizability of GNNs, as models must be tailored for each specific graph type. To address these challenges, introduce GRAPHFM, a scalable multi-graph pretraining approach designed for learning across diverse graph datasets. GRAPHFM uses a Perceiver-based encoder with learned latent tokens to compress domain-specific features into a shared latent space, enabling generalization across graph domains. We propose new techniques for scaling up graph training on datasets of different sizes, allowing us train GRAPHFM on 152 distinct graph datasets, containing a total of 7.4 million nodes and 189 million edges. This allows on pretraining across domains such molecules, citation networks, and product graphs, and show that training diverse datasets improves performance over single-source pretraining, Additionally, pretraining with a mixture of synthetic and real graphs enhances adaptability and stability, leading to competitive performance with state-of-the-art models across various node classification tasks. This approach reduces the burden of dataset-specific training and provides a single generalist model capable performing across multiple diverse graph structures and tasks. Code available
study the effect of scale
1 Introduction
Graphs are fundamental data structure in biology, social systems, and recommendation platforms Hamilton ct al.] 2017b). However, many graph neural network (GNN) architectures are specialized for particular
regimes and struggle to generalize across others (Topping et al. Yan et al. 2022 Zhu el al. 2020). Methods
that perform well on homophilic graphs, such as citation networks, often degrade on heterophilic graphs that
are common in social or biological domains (Abu-El-Haija et al., 2019 Yan al. 2022 This specialization
fragments model development and limits scalability, since each dataset class often requires architecture
redesign and extensive hyperparameter tuning (Galkin et. al. Wang et. al. 2025 Zhao et. al. 2024b). A

---

Published in Transactions on Machine Learning Research (12/2025)
generalist approach that performs robustly across diverse graph structures without per-dataset customization is therefore highly desirable.
A central challenge is to integrate graphs that differ in topology, features, and size while still enabling transfer of useful inductive biases across domains. Without a shared representation space or vocabulary for graph structure, learned patterns do not transfer reliably (Galkin et al.). At the same time, experience from language and vision suggests that scaling model capacity and data diversity can unlock transferable capabilities (Wei et al. Kaplan et al. 2020). Realizing similar benefits for graphs requires architectures that can process large and heterogeneous imputs efficiently while preserving
structure (Xia & Huang 2024
We introduce GRAPUFM,, a multi-graph pretraining framework that addresses these needs. GRAPUFM uses a Perceiver-slyle encoder (Jaegle et al.) 2021 Azabou et al. 2023a) in which a fixed set of learnable latent tokens attends to the input node sequence through cross- attentinn The latent tokens act as virtual nodes that compress cach graph into a compact representation, which decouples compute from graph size and creates a shared latent space across datasets. This design provides common interface for heterogeneous graphs and supports efficient scaling.
To train our generalist model, we curate a corpus of 152 datasets for pretraining, including 80 real-world graphs from multiple domains and 72 synthetic graphs that enrich underrepresented regimes such as low homophily. The full corpus contains more than 7.4 million nodes and 163.9 million edges. We exclude popular benchmark datasets from pretraining in order to evaluate generalization on unseen graphs.
Our experiments reveal that scaling both model capacity and data diversity produces consistent and measurable gains in generalization on graphs unseen during training. As increase model size from 389K to 75M parameters and the number of pretraining tokens from 200K to 7.3M, performance on unseen datasets improves monotonically, with the largest model achieving a 2.1% increase in accuracy over smaller configurations. Training on diverse domains further enhances this effect: adding biological graphs improves accuracy even on citation datascts, and including synthetic graphs leads to the strongest overall results. Beyond accuracy, GRAPHFM exhibits strong generalist. behavior with minimal adaplation. Using a fixed learning rate and weight decay, our lightweight MLP fine-tuning (MFT) approach achieves rapid convergence within 10 to 20 steps and strong out-of-the-box performance, while node-dccoder fine-tuning (NFT) achieves competitive performance with other otate-of-the-art graph transformers. Finally, sensitivity analysis shows that GRAPHFM maintains stable performance across wide range of hyperparameteers, unlike GCN and NAGphormer, which are highly sensitive to parameter settings. Together, these results demonstrate that a single pretrained model can provide robust, transfcrable representations across heterogencous graphs without per-dataset. luning.
The main contributions of this work are follows:
We introduce scalable framework for pretraining diverse graphs using a Perceiver-based encoder with latent tokens, which efficiently handles graphs with varying sizes and topologies. Our approach includes advanced multi-graph sampling techniques that optimize GPU utilization, enabling large-scale pretraining across a wide range of graph datascts.
Scalable Pre-training Approach:
Demonstration of Benefits from Across-Graph Pretraining: We show that pretraining on diverse graphs significantly improves the model's ability to generalize and transfer knowledge to unseen graphs. To further enrich diversity, we incorporate synthetic graphs that capture underrepresented structures such as low-homophily patterns and show that this can help improve performance. This demonstrates that a generalist model can leverage common structural features across different datasets to outperform specialized models.
Scaling Analysis and Impact of Multi-Graph Pretraining: We provide the first scaling analysis for multi-graph pretraining on different domains, showing that larger models pretrained on more diverse graph datasets result in better generalization.
2

---

Published in Transactions on Machine Learning Research (12/2025)
2 Background
In this scction, provide background graph transformers, focusing tokenization of graphs and posilional encodings.
2.1 Transformers and Leff-Attention
Transformers model sequential data by operating on a set of tokens = [X,,...,XN] where dependencies are captured through self-attention (Vaswani 2017
Attn(Q,Ke K, V) softmax
QKT
V,
vdl
with Q, K, V denoting learned projections of the input tokens. Extending this framework to graphs requires a tokenization scheme that maps graph-structured inputs into a sequence of tokens, while incorporating structural information derived from the adjacency matrix.
2.2 Tokenization of Graphs
Given a graph g = (V,E) with node features {u.)) each node o E V is represented by a token embodding that concalenates projection of its raw features with a positional encoding:
ú = MLP(u:) E RR,
= (ū; E Rdr+d,
where MLP denotes multi-layer perceptron, (:-) is concatenation, and p. E R encodes structural information from the graph topology. The complete graph is then expressed sequence of tokens,
x= X1, , x
Since graphs lack a canonical node ordering, the tokenization scheme must be permutationninnaaiintt, ensuring that reindexing nodes does not alter the resulting sequence.
2.3 Positional Encodings and Sign-Invariance
To incorporate structural information, each token is assigned positional embedding derived from the graph
topology. A standard approach is to use the eigenvectors of the normalized Laplacian C = / - D-1/2 AD-1/2 where A is the adjacency matrix and D the degree matrix. Let Vi...., Vk denote the first eigenvectors of C, corresponding to the smallest eigenvalues. For node i, we construct a vector
S; [Vii, V25...., Vud E R*,
where Vji is the i-th entry of eigenvector Vj. These eigenvector values provide continuous notion of position that captures global graph structure and are invariant to permutations of node indices.
A key limitation of spectral encodings is the non-uiqqenesss Laplacian eigenvectors: each eigenvector can be multiplied by - without affecting validity, and for repeated eigenvalues, any orthonormal basis of the eigenspace may be chosen. As shown in Lim al.) these ambiguitics make raw eigenvoctors unsuitable features, since they can lead to unstable and inconsistent encodings across graphs.
To address this, SignNet introduces functions that are provably invariant to both global sign flips and basis changes (Lim et al.), while retaining the ability to approximate common positional encodings such as heat kernels and random walks. This property is particularly important in our multi-graph pretraining setting, where consistent positional encodings are required to align structural information across different graphs. We pass s, through SignNet obtain the final positional embodding p, E Rd. This transformation ensures that the resulting positional encoding are both sign- and Davii-inv while preserving global structural information across graphs.
3

---

Published in Transactions on Machine Learning Research (12/2025)
Graph encoder
Node decoder
Repeat all nodes
O
Attention
Attention
Linear Classifier W
Cross
Infomation peoting!
Kv
Learned Latent Tolens
compression
XL
XM
features of node
context for decoding
features randorn walk nodes
Figure 1: Overview GraphFM architecture and multi-graph training approach: 'The imput node-level tokens are passed through a cross-attention layer, followed by multúple self-attention layers generate a compressed graph-level representation (latents). We decode node-level properties by creating a spatial sequence with features from a query node, a subset its neighbors (sampled via random walk; see Appendix C.2b and the latents, which is then processed by node docoder that uses self attention across the sequence.
3 Methods
this section, we describe our method, including the model architecture and tokenization (Section 3.1.1 our proposed mulli-task node decoder for jointly solving node classification and regression tasks by querying from the latent space (Section 3.1.2 ), and efficient tools for scaling (Section 3.2 that allowed to build large pretrained model that could integrate the extreme diversity in our pretraining set.
3.1 Model
3.1.1
Tokenizing Diverse Graphs with Perceiver Encoder
Each graph represented as a sequence of tokens, as described in Section Formally, lot (9y1g N, denote a dataset of graphs, where each graph Gy (Vy, Eg) has node features (u:) with Ng = 12ll For
each graph, we construct token sequence Xg X1 ........ . xv,l, where each token x; combines projection of the node features with positional encoding (see Section 2
To enable training across diverse graphs with heterogencons node features and sizes, we map all graphs into a shared latent space using Perceiver encoder (Jaegle et al.p The encoder learns fixed set of latent query tokens that attend to the input graph via pross-attention producing compact latent representation. the context of graphs, this can be viewed as routing communication between distant nodes through a small number of learnable virtual nodes that summarize the input graph (Figure D.
Specifically, we maintain shared sequence of K learned latent tokens Zo 1... k0,k,, where Zo RD and K < N₂ (we use K = 512 in this work). Each graph's node embeddings are compressed into this latent space through cross-attention operation:
QK g
Z Cross-Attn Vg) Zo softmax
V
(1)
where the queries W,Zo are projections of the learnable latent tokens, and the keys and values, K, W,X, and V, W.X,, projections of the graph's token embeddings. The projection matrices (W. g. We, W.) are shared across all graphs. This is followed by L self-attention blocks operating in the latent space, yielding the final sequence of K latent tokens, Zout Each block uses standard transformer components with pre-normalization and feed-forward layers (Vaswani 2017]. The overall complexity is O + LK²), which is significantly lower than the O(N3) cost of full self-attention when K < N,
Compressing each graph into a fixed of latent virtual nodes enables the model learn shared vocabulary across graphs and domains, capturing recurring semantic and topological motifs. Moreover, because the bulk
4

---

Published in Transactions on Machine Learning Research (12/2025)
of computation occurs in the latent space, this design naturally accommodates graphs of variable sizes while maintaining constant computational cost per graph.
3.1.2 Node decoder
Our encoder model designed do the bulk of the computation when processing the graph. To be able to readout node-level leatures, developed a multi-task node decoder that combines the virtual node embeddings learned by encoder g with local information from node and its neighbors to create sequence S that can be processed by transformer to produce final node-level estimate of its class information.
The sequence Sy for the 7th node can be represented as:
Si = (x::)) (XN35 Theighhor) (XNTi Theighbor),
(Zi Natent) (ZX Natent)
(2)
node
neighbors
virtual latent nodes
where x and Toype denote the features and their token type (latent, self, or neighbor), respectively, and N/ denotes the jth neighbor selected in the neighborhood of node i. We use small encoder-only transformer with depth of M to obtain a final set of embeddings S for node Note that the complexity is MK + T 1)² < N2.
3.1.3 Multi-tast pretraining on variety of node classification and regression tasks
In the end, a per-dataset linear classifier (or regressor) WT is tasked with producing the final predictions j for node i, mapping the final embedding node i, the first token in the S sequence Eq, to the output space as:
The lincar projection effectively translates the nodo-level emboddings into task-spocific outputs, such as class labels for classification continuous values for regression. The model handles a wide variely of tasks across different datasets, e.g., citation graphs are trained to predict academic fields, and co-purchasing graphs are used to predict product categorics. Each dataset has an arbitrary label space, varying not only in the number of labels but also in the nature and semantics of the output classes.
Since the model trained end-to-end, it learns how to optimally route and query information on graphs to maximize the performance on the various pre-training tasks. The virtual nodes allow for longer-range and global interactions be encoded in the virtual node embeddings, and uses this information along with the local information provided by the node's neighbors.
3.2 Important ingredients for training on diverse graphs
3.2.1
Multi-graph packing
Typically when creating batches for training graph transformers, padding is used to extend the smaller graphs to have the same size as the largest graph in the batch (Rampášck et al. 2022 Ying et al. 2021). This approach is likely inherited from the transformer architectures found in other domains where the context. window (or sequence length) is usually fixed. But for graphs, the problem with padding is particularly pronouncod when there is significant size disparity among different graphs in the same batch. Alternative solutions exist, and in particular, the graph community have been pioneers batching variable-sized graphs. Message-past passing frameworks combine multiple graphs into single large graph over which message passing is conducted (Fey & Lenssen 2019 Krell et al. 2022). However, these out-of-the-bon implementations are not suited for transformers which use fully-connected attention.
To address this, we simply batch graphs by concatenating their node tokens into a single sequence, and use Flash Attention (Dao) 2023) to efficiently handle these pariable-lengtl sequences. This eliminates superfluous padding and leads to improved computational efficiency during training.
5

---

Published in Transactions on Machine Learning Research (12/2025)
3.2.2 Balanced GPU utilization with the DistriutedSSSampler
During multi-GPU distributed training, global batch is formed by randomly sampling graphs from different datasets, which is then equally split among the GPUs. Naively splitting the batch can lead to unbalanced GPU utilization. On one hand, we can have a large batch of relatively small graphs, and another where we can only have batch with one two very large graphs. This means that we would be forced to lower the batch size, to avoid going out of memory when multiple large graphs are batched together. Our Distributed Snake Strategy Sampler (DistributedSSamplee) employs a bidirectional filling strategy, where graphs, sorted by their size, are distributed in snake-like pattern, initially assigned to GPUs from right to left, then left to right. and on. This method effectively pairs large graphs with small ones in subsequent passes, preventing the concentration of multi- ple large graphs on the same GPU, thus achieving efficient load balancing and uniform GPU utilization. A detailed algorithm and more details are provided in Appendix C.1
baseline, b 640
0.20
- 64, b 1376
Density
0.05
0.00
50
60
70
80
90
100
GPU Memory Usage (%)
Figure 2: The computational benefits of using our multi-graph sampling approach: GPU memory uti- lization during distributed training using the default batch sampler with GPUs (left), compared to Distribut- edSSSampler with N 4 (middle, 4 GPUs and I gradient accumulation step) and N 64 (right,, 8 GPUs and gra- dient accumulation steps). The total batch size
b.
We show the effectiveness of this approach in Figure where we demonstrate significantly lower variance in GPU load compared to the default Py Torch batch sampler and near 100% utilization. The effectiveness is more pronouneed the more GPUs are used This subsequently allows us to use substantially larger batch sizes, resulting in further improvement in stability and a significant 2-4x speed-up in training time.
3.2.3 Overall time and memory savings
In total, our largest model, trained on all the pretraining data, takes -6 days to train on 8 A40 GPUs for 300 epochs. With our distributed sampler, each epoch takes approximately 56 minutes (0.93 hours), compared to 299.04 minutes (-5 hours) with the standard distributed sampler. By using the distributed sampler, we observe a speedup of approximately 5.53x, reducing the total training time from 33 days to days. Please refer to Appendix [ for an ablation study on the proposed sampler and multi-graph packing methods.
4 Datasets
In standard practice, one would train on individual datascts, one at a time. However, to build our large multigraph model, we needed to curate a large dataset of graphs that have varied structures, features, and tasks.
Datascts used for pretraining. For pre-training, we curated a large set of 80 real-world graph datasets from the PyTorch Geometric library (Fey Lenssen 2019) and Network Repository (Rossi & Ahmed) (Figure 3 These datasets span wide range of domains, including: citation networks, product recommendation graphs, webpage traffic graphs, biological protein-protein interactions, and molecular graphs, and vary in their degree of heterophily (extent to which neighbors share the same class or node-level labels). Each dataset contributes unique structural patterns and tasks, providing a rich source for our model to learn diverse graph represontations. In addition to these real-world datasets, we generated 72 synthetic graphs (Tsissulin) et al. 2022) that vary in their hetero- and homophily ratios, and overall size and density (see Appendix B.1). We note that most datasets used in popular benchmarks were left out of pretraining to enable meaningful evaluation of generalization performance new unseen test datasets
The same effect can be obtained using gradient accumulation when resource bound. See Appendix C.1
6

---

Published in Transactions on Machine Learning Research (12/2025)
Synthetic Real
30
15
frequency
15
20
10
10
10
10
5
0.0
0.5
1.0
102
10
10
homophily ratio
average degree
total number of nodes
total number of edges
Figure Characteristics graph datasets used train GraphFM: From left to right, we compute the histograms of the homophily ratio, average degree, number nodes and number edges all 152 graphs used during training. The homophily ratio provides a measure of how frequently a node is directly connected to other nodes from the same class.
In Figure 3 we show summary of various graph statistics, including the number of nodes and edges, the average degree of each node, and the homophily ratio of the graph. The homophily ratio ranges from 0 to and encodes the average amount of nodes with nearest neighbors from the same class. When comparing our real-world datasets with the synthetic graphs added the mix (Figure 3), we see a good amount of overlap between most fcatures except for the average degree. The average degrec realworld graphs spans a larger range, and the synthetic graphs have a more limited range. We also find an enrichment of heterophilic graphs with low homophily ratio in the added synthetic data. In total, counted more than 7.4M nodes and 163.9M edges across all 152 datascts used for pretraining. We point the reader to Appendix B.1 for a detailed description of all datasets.
Datasets for evaluating generalization to unseen graphs To evaluate how general and transferable the learned representations are, we evaluate the model on a set of unseen graph datasets that were excluded during training (see Appendix B.4p. These 10 datasets include academic collaboration networks such as "Coauthor-CS" and "Coauthor- Phynthhhrr-Pysicss hysics" (Sinha et al. 2015) as well as webpage link datasets like "Chameleon" and "Squirrel" [Rozemberezki et al. 2021). The latter are particularly challenging due to their low homophily ratios, where nodes are less likely to connect to others of the same class.
The unseen datasets were not included during training but may share structural similarities with the training data. The label and feature space of these graphs are entirely new to the model, making them suitable for testing generalization. Evaluating on such unseen datasets allows us to examine whether the learned representations can elfectively generalize new graphs with similar structural properties.
5 Results
5.1
Experimental Setup
Training: To train all of our models, we employed the LAMB optimizer You et al.) with a learning rate of 10-4. The learning rate is scheduled basod on a linear warmup of epochs, followed by cosine decay until the end of training. We use bfloat16 mixed-precision and Flash Attention (Dao 2023) for higher compute efficiency while training. We trained our largest model (75M parameters) for 300 epochs with batch size of 320 (~6.4 days) on 8 NVIDIA A40 GPUs. We point the reader to further details on the architecture and model training in Appendix A.1
Baselines: We compared GRAPHEM against six baseline models that were consistently reported in both heterophilic and homophilic benchmarks. This included two GNN-basod models: GCN (Kipf & Welling 2016) and GAT (Velickovic et al.) 2017); two transformer-based models: SAN (Kreuzer et al. 2021) and
NAGphormer (Chen et al.);; and two heterophily- based models: MLP and H2GCN (Zhu et al. 2020). For all of the baseline models, we include the best reported accuracy, and when there are no reported results for dataset we extensively tuned each model as in standard practice (see Appendix B.4h. We also provide additional baselines in Appendix D..5 reported for subsets of the datasets tested.
7

---

Published in Transactions on Machine Learning Research (12/2025)
Evaluation: To evaluate the quality of the learned representations, fine-tuned the model datasets that were excluded from pre-training- We employed two fine-tuning strategies for this purpose. The first strategy, low-resource MLP fine-tuning (MFT), involves freezing both the encoder and node decoder weights, allowing updates only to the feature MLP. This approach evaluates near out-of-the-box performance by leveraging the pretrained model's representations with minimal additional training. The second stralegy, combined MLP and node decoder finc-tuning (NFT), provides more flexibility by adapting both the feature MLP and the pretrained node decoder weights, enabling the model better align with the unseen graph data that it's finetuned on.
For all fine-tuning experiments, we fixed the learning rate to 10-3 and the weight decay to 10-6 across all datasets, optimizing with the AdamW optimizer (Loshchilov & Hutterd. In our NFT experiments, we additionally applied a gradual unfreezing strategy to update the node decoder weights. Further details are provided Appendix A.3
5.2 Experiments
Q1: Is it possible to build large model spanning many domains?
Recent efforts in graph neural networks (GNNs) have shown success in training models on many graphs (Baaini
0.750
et al. 2024 Mao et al. 2024). However, these approaches primarily focus on graphs with homogeneous structures, limiting their ability to generalize across different types of graphs. In this experiment, we aim to address a more ambitious question: we effectively train large model diverse, multi-graph datasets that vary significantly in their topologies, features, and downstream classification tasks? Our goal is to determine whether a generalist. model can span multiple graph domains and improve perfor performance on new unseen datasets through diverse pretraining.
0.75
0.74
0.734
mean accuracy accross
0.732
0.73
0,729
test datasets
0.730
0.72
0.724
0.725
0.717
0.71
0.70
389K
18M
0.69
75M
0.68
We trained three different model sizes: small model with 389K parameters, medium model with 18M parameters, and large model with 75M parameters. Each model was pretrained on progressively larger datasets containing different amounts of graph data, ranging from 200K tokens (small), to 2M tokens (modium), and finally to 7.3M tokens (large), created by laking random subsets of the largest dataset (refer to Appendix B.2 for more details). The datascts span a variety of real-world graph types and structures, as described in Section 4 For the largest scale of data, we also introduced synthetic graphs into the mix to further test the model's ability to generalize across highly diverse graph structures. The synthetic graphs provided addit ional variability both topologies and node features, allowing us to assess how well the model can handle graph data that extends beyond typical real-world scenarios.
10F
105
107
B
0.95
accuracy
0.94
0.93
0.92
Coauthor-CS Amazon-Phoor Photo
0.91
10a
Soc
Soc+Bio
107
number of tokens seen during pretraining
Figure 4: Scaling analysis showing how increasing model and data scale impacts downstream perfor-
mance: (A) Average accuracy across test datascts (MFT) for model sizes (389K, 18M, 75M) and token counts (200K, 2M, 7.3M) seen during pre-training, using random splits of the pre-training data. (B) Accuracy (MFT) on Coauthor- CS (citation domain) and Amazon-Phott (cu-purchasing uetwork) for the 75M model across different domain-wies pre- pro-training splits.
To evaluate how well the pretrained models generalize to new, unseen data, we applied our lightweight MLP finc- tuning approach (MFT) on a set of nine held-out datasets. These include four homophilic datasets (Coauthor-CSS Coauthor-Phyeee Amazon-Photo, and Amazon- Amazon-Comp)
and five heterophilic datasets (Texas, Wisconsin, Actor, Squirrel, and Chameleon). As illustrated in Figure 44, observe that performance unseen test datasets improves consistently the data size increases. Notably,
8

---

Published in Transactions on Machine Learning Research (12/2025)
the largest model, trained on the full 7.3M tokens, achieves 2.1% improvement in accuracy compared to the smaller models.
We further stratified our pretraining dataset to investigate the effects of cross-domain training by creating three models: (1) "Soc" with social domain graphs (1.3M tokens), (ii) "Soc + Bio" with social and biological graphs (2M tokens), and (iii) "All" with all data, including synthetic graphs (7.3M tokens). As shown in Figure 13, adding biological datasets improved performance on both Coauthor- (citation domain) and Amazon-Photo (co-purchasing network). This suggests that performance continues to scale even if the additional data is from seemingly unrelated domains (refer to Appendix D.1 for additional results from removing synthetic graphs during pretraining).
We want to note that the domains used in this analysis were chosen largely for practical reasons, since they provided the largest sets available datasets for comparison. As such, they should not be viewed as definitive boundarics, but rather as convenient stratification to explore the effects of cross-domain pretraining.
These results underscore the importance of both model scale and data diversity. With more data diversity and larger models, the pretrained model demonstrates stronger generalization capabilities. This scaling analysis provides strong evidence that cross-domain pretraining enables better performance, further validating the benefits of training on diverse datasets. Detailed configurations for each model size are provided in Appendix A.T.
Q2: How does our generalist approach compare with others?
Next we wanted understand how our generalist model compares to specialized single-graph approaches. To do this, we evaluated the performance of our largest model (75M) trained on all of the data on standard node classification benchmarks. We focused on the mean rank across datasets as key metric, providing insight into the model's consistency across diverse graph types. Unlike specialist models that require extensive hyperparameter tuning for each dataset, in MFT we use a single hyperparameter configuration across all evaluations. In the case of NFT, we only have two hyperparameters to tune corresponding to our unfreezing schedule (refer to Appendix A.3.2).
Figure shows the mean rank of our model compared to several common baselines, including pessage-passing ar- chitectures such as GCN [Kipf & Welling 2016) and GAT models such as
Average Rank of Models
6.30
Velickovic et al.] (017). heterophily- H2GCN (Zhu et al. 2020), and transformer-based architec- tures such as SAN Kreuzer et al. 2021 and NAGphormer [Chnn et al.). The NFT fine-tuning strategy achieved the best overall rank, demonstrating its flexibility to adapt to range of graph structures. In addition, the MFT strategy was the second best method and had even lower variance, indicating stable performance of our pretrained models across datasets with varying characteristics.
Mean Mean Rank (aceross 10test datseet
datasets)
4.70
4.00
3,30
2:10
M1
Specialist models like H2GCN and NAGphormer (NAG) exhibit high variability in their ranking because they per- form well on certain datasets but worse on others. H2GCN is designed for hetcrophilic graphs, while NAG is optimized for homophilic ones. A more detailed comparison to specialist models, including per-dataset performance, can be found in Appendix B.6
Model
Figure Mean rank of various models across graph datasets not seen during training (lower better). Error bars indicate the standard deviation the ranks.
These patterns highlight the trade-offs inherent in models tailored for specific graph types, which may impact their consistency across diverse datasets. In contrast, our single model, using a fixed hyperparameter configuration, maintains a competitive ranking across all datasets without requiring dataset-specific tuning.
9

---

Published in Transactions on Machine Learning Research (12/2025)
Actor
Wisconsin
Chameleon
Coauthor-CS
0.8
0.5
0.8
0.3
accuracy
0.6
0.4
0.6
0.2
0.4
0.3
0.4
grephFM (MFT)
0.2
Best GCN model
0.2
0.2
0.1
0
10
20
0
10
20
0
10
20
0
10
20
B
0.8
0.5
accuracy
0.3
0.6
0.4
0.6
0.2
0.4
0.3
0.4
graphFM (MFT) Best NAG model
0.2
0.2
0.2
0.1
0.1
0
20
0
10
20
10
20
10
20
training steps
training steps
training steps
training steps
Figure 6: Analysis of the learning dynamics showing GraphFM achleves faster and more stable convergence compared baseline single dataset models: Learning dynamics for 100 (A) random GCN and (B) NAG (NACphormer) models compared against our lightweight finetuned model GraphFM (MFT) for four datasets. GRAPHFM works out of the box and achieves rapid learning on new datasets with few training steps, while the other approaches less stable and often require early stopping with decreased performance over training.
Q3: How does our model generalize out-of-the-box?
Chameleon
A major challenge in applying graph-based models is the extensive tuning often required to achieve competitive performance. Most models are highly sensitive to hyperparameters like learning rate, depth, and weight decay. Tuning these hyperparameters, especially across datasets with different graph topologies and sizes, requires significant time and computational resources, and even then, finding good configuration can be difficult Guo et al. 2022 Tsitsulin
0.6
distance from optimal hyperparamatinn ryperparrmetttrs
0.5
acouracy
0.3
et al. 2022).
0.000
In contrast, GRAPHFM offers strong out-of-the-box performance without requiring any significant. tuning. To demonstrate this, we evaluated GRAPHFM using the same fixed learning rate and weight decay across multiple datasets (learning rate 10-3, weight decay- 50 and observed stable and high performance across all datasets (Figure G Fine-tuning GRAPHFM with our simple MFT strategy resulted in low variance and rapid convergence, without the need for extensive hyperparameter exploration. This makes GRAPHFM highly efficient and cost-effective compared to models that require substantial tuning.
GCN
GraphFM
Coauthor-CS
1.118
2 Vumaaa
0.9
dislance from aptimaal
aocuracy
hyyerppramelee
0.8
0.6
0.000
To highlight this contrast, compared the performance of GRAPHFM with 100 randomly configured versions of GCN and the best performing transformer-basee NAGphormer (Chen et (Figure G.. Both baseline models exhibit a wide range of performance depending on the hyperparameter choices, with some configurations leading to significant instability or poor results. For instance, in the Texas dataset, GCN required exhaustive exploration of hyperparameter settings to find a stable and effective configura-
GCN
GraphFM
Figure 7: Comparison model sensitivity. The performance GCN and GRAPHFM 100 different random hyperparameters on Chameleon and Coauthor-CSS CS. Star denotes the model with the optimal hyperparameters, and the color indicates the fg-distance between the optimal solution and each model's hyper- parameters.
10

---

Published in Transactions on Machine Learning Research (12/2025)
tion. Similarly, NAGphormer's performance fluctuated greatly depending on the dataset and the selected parameters, further emphasizing the cost. of tuning.
Additionally, GRAPIFM demonstrates quick convergence, reaching near-optimal performance within 10-20 training steps (Figure 6), in stark contrast to GCN which required considerably more iterations to converge. This efficiency is a direct result of leveraging a pretrained model, which allows GRAPHFM start from a robust initialization and quickly adapt to the target task. The reduced need for lypeeraaaameter tuning and faster convergence further solidify the advantages of pretraining in minimizing computational overhead and time-to-solution. Ultimately, our results position FRAPHFM cost-effective and reliable choice for wide range of node classification tasks.
Q4: How stable are the solutions?
Graph-based models are highly sensitive to hyperparameter configurations, where even small deviations from optimal settings can lead to substantial performance degradation. This sensilivily poses significant challenges for ensuring stable and robust deployment. Thus, we wanted to examine the stability of model performance by exploring the performance landscape around the optimal hyperparameter configuration. We analyze the performance of both GCN and GRAPIFM (MFT) on Coauthor-CS (homophilic) and Chameleon (heterophilic) datasets for different hyperparameters around the optimal hyperparameters (Figure 7 This set. of hyperparameters is marked with a star, and other models are colored based on the normalized l-ddistance of their hyperparameter vectors to the optimal hyperparameter vector. For GRAPHFM, obscrve that the distribution is concentrated around the optimal point, suggesting low sensitivity to the choice of the typerparameters used for finetuning. We also observe that the relationship between hyperparameter deviation and performance is linear. On the other hand, for the GCN model, small deviations in hyperparameters can lead to large changes in performance, suggesting instability of the model with respect to the hyperparameters and a much noisier landscape around the optimal model.
6 CO2 Emissions
Training our largest model took 1228.8 A40 hrs, totaling an energy usage of 368kWh. The GPUs were housed and powered al an academic datacenter in Georgia, with a carbon energy intensity of 0.35 kgCO2e/KV) /kWh and Power Usage Effectiveness of 1.15, the resulting emissions were approximately 148kg of CO2eq.
7 Related Work
Graph foundation modeling approaches. Foundation models have achieved significant success for language, vision and time-series data (Radford al.] 2018 Dehghani et al. 2023 Das et al. 2024). These models are pre-trained on large datasets and can be adapted utilizing both prior knowledge from the pre-training stage and data from the downstream tasks to enhance performance (Brown et. al. 2020). The concept of foundation models has recently extended into graph learning, leading to the development of Graph Foundation Models (GFMs) [[barz et al. 2022 Beaini et al.) 2024 Galkin et al. Mao et al. 2024). These models aim to generalize across diverse graph-structured data by leveraging large-scale pretraining, similar to foundation models in vision and language domains.
wide range of downstream tasks, effectively
Initial GFM offorts primarily focused on domain-specific GFMs, where shared structures and feature vocabularies simplify pretraining. For example, Mole-BERT utilizes pretraining improve property prediction specifically for molecules and materials (Xia et al..p Additionally, large-scale models like MatterSim (Yang)
ct al. 2024) designed to predict molccular behaviors across different elements and conditions.
Beyond domain-specific applications, Graph Foundation Models (GFMs) are increasingly being developed for general-purpose tasks across diverse graph domains. Similarly, recent advancements have explored scaling laws in graph models, showing that larger models can lead to improved transfer learning and generalization (Liu) et al. b). consistent with our work which shows that scale improves performance. Models like GMNN [Tbaxx et al. 2022) and ULTRA (Galkin et al.) tackle foundational tasks in algorithmic reasoning and
11

---

Published in Transactions on Machine Learning Research (12/2025)
knowledge graphs respectively, but they are still grounded in narrow domains. Other recent efforts have leveraged LLMs to unify graph inputs via textual representations (Liu et al. a).
Recent work has also begun addressing the more challenging problem of cross-domain generalization, where graphs differ significantly in topology, size, and feature space. For instance, GraphProp (Sun et al., 2025
and GFSE (Chen et al. 2025) focus on structural generalization by pretraining on topological properties such as random walk embeddings. GCOPE (Zhao et al.) 2024a) and MDGPT (Yu et al.) 2024) incorporate domain-aware components like virtual coordinators or learnable domain tokens to encode domain-specific
priors. GraphAny (Zhao et al., 2024b) proposes fully inductive model that ensembles predictions from analytically derived LincarGNNs. While a lot of progress has been made most existing models are constrained by limited dataset scale GraphProp and GFSE were trained on 5 8 datasets, and GraphAny only on 1- making it difficult to assess generalization robustness. In contrast, GraphFM is pretrained on 152 graph datasets (7.4M nodes, 189M edges), offering an unprecodented testbed to study scale and diversity effects.
Scaling up graph transformers. Graph transformers bypass standard local learning rules in GNNs by allowing all nodes on the graph to interact through self-attention (Dwivedi & Bressso 2020). However, duc to the high computational cost and benefits of the inductive bias in message passing, a number of methods have been proposed to move beyond full self-attention or combine transformers with GNNs. One class of methods combine transformer blocks with GNNs, including GraphTrans (Wu et al., 2021), GraphGPS (Rampášek) et al. 2022), and SAT (Chen et al.) 2022). Another strategy is to reduce the computational complexity by using the transformer module on a coarsened compressed graph. For instance, ANS-GT (Zhang et al. 2022) introduced node-sampling baaseddi----- graph transformers, incorporating hierarchical attention and graph 2023) uses k-hops local pooling and global pooling to coarsen the large graph into a smaller set of nodes. Exphormer (Shirzad et al., 2023) coarsens the graph by doing computation through expander graphs (Deac et al.
coarsening, and Gapformer (Liu et al.
2022). This idea of compression has also been studied through the lens
of "skeletonization" (Cao et al.) 2024] where they learn to identify uninformative background nodes and use this information to achieve competitive performance with as little as 1% of the nodes in the graph. Many of these approaches leverage virtual nodes to facilitate message passing across large graphs; however, the compression techniques used in these works are often based on heuristics like pooling layers expander graphs, in contrast to our work where the compression is fully learned through the use of the Perceiver encoder to aggregate information into a set of latent tokens.
8 Conclusion
In this paper, we introduced GRAPHEM, a multi-graph pretraining framework for scalable learning across diverse graph datasets. We curated 152 datasets for pretraining, including 80 real-world graphs and 72 synthetic graphs, totaling more than 7.4M nodes and 163.9M edges. Standard benchmark datasots were excluded from pretraining to enable evaluation on unseen graphs. The syuthetic corpus enriched low-homophily regimes and complemented the real-world distributions, which had a broader range of average degrees. This mixture provided a varied training signal across topology, feature space, and label structure.
Our scaling study shows clear benefits from both model capacity and data diversity. Across three model sizes (389K, 18M, 75M parameters) and increasing numbers of pretraining tokens (200K, 2M, 7.3M), accuracy on held-out datasets improved monotonically, with the largest model trained on 7.3M tokens achieving a 2.1% gain over smaller configurations. Domain-stratified experiments further indicate that adding biological graphs improves performance on citation and co-purchase datasets, and that including synthetic graphs yields the strongest overall results. These findings support the view that diversity, not only size, a key driver of transferable representations.
We evaluated two adaptation strategies on unseen datasets. Low-resource MFT freezes the encoder and node decoder, and adjusts only the fcature MLP; NFT additionally updates the node decoder with gradual unfreezing. NET achieved the best mean rank across ten Lest graphs, while MFT ranked second with lower variance, which indicates stable generalization without dataset-specific tuning. show that GRAPHFM reaches ncar-optimal accuracy within 10 to 20 steps with a single fixed learning rate and weight decay across tasks, while GCN and NAGphormer exhibit broad performance variability and frequent instability under random hyperparameters.
analyses
12

---

Published in Transactions on Machine Learning Research (12/2025)
Sensitivity analyses around the optimal hyperparameters reveal smooth and forgiving landscape for GRAPHFM. Accuracy degrades approximately linearly with distance from the best settings, which suggests robustness to modest deviations and reduces the operational burden of fine-grained tuning. In contrast, GCN cxhibits sharp drops for small changes in configuration, which complicates practical deployment. Together, these results position GRAPHFM as a cost-effective and reliable starting point for new graphs, including challenging low-homophily settings.
Looking ahead, we expect further gains from expanding the pretraining corpus to additional structured domains such as meshes, point clouds, and knowledge graphs, and from broadening the task palette beyond node classification to include link prediction, graph-level prediction, and self-supervised objectives. Progress in synthetic graph generation and principled domain balancing should strengthen scaling trends and improve robustness under distribution shift. unified suite of cross-domain benchmarks, paired with standardized adaptation protocols like MFT and NFT, can help establish repeatable scaling laws for graph learning and accelcrate the development of generalist graph foundation models.
9 Acknowledgements and Funding Disclosure
Thanks to Shivashriganesh P. Mahato, Keertika Saroj and Yifei Sun for insightful discussions and feedback on this manuscript. This project was supported by NSF CAREER Award RI:2146072, NSF award CIF:RI:2212182 as well as generous gifts from the CIFAR Azrieli Global Scholars Program.
References
Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In international conference on machine learning, pp. 21-29. PMLR, 2019.
Mehdi Azabou, Vinam Arora, Venkataramana Ganesh, Ximeng Mao, Santosh Nachimuthu, Michael Mendelson, Blake Richards, Matthew Perich, Guillaume Lajoie, and Eva Dyer. A unified, scalable framework for neural population decoding. Advances Neural Information Processing Systems, 36:44937-44956, 2023a.
Mchdi Azabou, Venkataramana Ganesh, Shantamu Thakoor, Chi-Heng Lin, Lakshmi Sathidevi, Ran Liu, Michal Valko, Petar Veličković, and Eva L Dyer. Half-hop: A graph upsampling approach for slowing down message passing. In International Conference on Machine Learning, pp. 1341-1360 PMLR, 2023b.
Dominique Beaini, Shenyang Huang, Joao Alex Cunha, Zhiyi Li, Gabriela Moisescu-Parejy Oleksandr Dymov, Samuel Maddrell-Mander, Callum McLean, Frederik Wenkel, Luis Müller, et al. Towards foundational models for molecular learning on large-scale multi-task datasets. In ICLR, 2024.
Deyu Bo, Xiao Wang, Chuan Shi, and Huawei Shen. Beyond low-frequency information in graph convolutional networks. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 3950 3957, 2021.
Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? In International Conference on Learning Representations.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Linfeng Cao, Haoran Deng, Yang Yang, Chumping Wang, and Lei Chen. Graph-skeleton: 1% nodes are sufficient to represent billion-scale graph. In Proceedings of the ACM on Web Conference 2024, WWW '24, pp. 570-581, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400701719, doi: 10.1145 /3//333883333455545333 URL https: https://dol. org/10 10. 1145/3589334. 3645452
Cong Chen, Chaofan Tao, and Ngai Wong. Litegt: Efficient and lightweight graph transformers. In Proceedings of the 30th ACM International Conference Information 6 Knowledge Management, pp. 161-170, 2021.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Dexiong Chen, Leslie O'Bray, and Karsten Borgwardt. Structure-aware transformer for graph representation learning. In International Conference on Machine Learning, pp. 3469-34444 3489. PMT.R, 2022.
Jialin Chen, Haolan Zuo, Haoyu Peter Wang, Siqi Miao, Pan Li, and Rex Ying. Towards a universal graph structural encoder. arXiv preprint Xrxi::2504 10917, 2025.
Jinsong Chen, Kaiyuan Gao, Gaichao Li, and Kun He. Nagphormer: A tokenized graph transformer for node classification in large graphs. In The Eleventh International Conference on Learning Represenlations.
Ming Chen, Zhewei Wei, Zengleng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International conference on machine learning, pp. 1725-1735. PMTRR, 2020.
Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. In 9th International Conference on Learning Representations, ICLR 2021, 2021.
Tri Dao. Flashattention-22: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representalions, 2023.
Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for lime-series forecasting. In Forty-first International Conference on Machine Learning, 2024.
Andreca Deac, Marc Lackenby, and Petar Veličković. Expander graph propagation. In NearIPS 2022 Workshop on Symmetrr and Geometry in Newral Representations, 2022. URL htttps///ppenreiee..eee
formm"Id=6cthhq2qqCC,
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Pe- ter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Amurag Arnab, Xiao Wang, Carlos Riquelme Ruiz, Matthias Minderer, Joan Puigcerver, Utku Evei, Manoj Kumar, Sjoerd Van Steenkiste, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Collier, Alexey A. Gritsenko, Vighnesh Birodkar, Cristina Nader Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov, Filip Pavetíc, Dustin Tran, Thomas Kipf, Mario Lucic, Xiaohua Zhai, Daniel Keysers, Jeremiah J. Harmsen, and Neil Houlsby. Scaling vision transformers to 22 billion parameters. In Andreas Krause, Famma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 7480-7512. html
PMLR, 23 23-29 Jul 2023. URL https://proceeddnng: mlr. presss/v222/eehghame
Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. arXiv preprint Xix:2201 2.09699, 2020.
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
Mikhail Galkin, Xinyu Yuan, Hesham Mostafa, Jian Tang, and Zhaocheng Zhu. Towards foundation models for knowledge graph reasoning. In The Twelfth International Conference on Learning Representations.
Shihan Guan, Lei Song, Xin Chen, Yueying Li, Qinghua Si, Laurent Albera, Régine LE BOUQUIN JEANNÉS, Youyong Kong, and Huazhong Shu. Enhancing graph tasks with a dual-block graph transformer: A synergistic approach to local and global altention.
Lingbing Guo, Qiang Zhang, and Huajun Chen. Unleashing the power of transformer for graphs. arXiv preprint ar Xr0::2200. 10581, 2022.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017a.
William L Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: Methods and applications. Xix preprint arXiv 1-9.:x1709.000 05084, 2017b.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Van Thuy Hoang, O Lee, et al. Mitigating degree biases in message passing mechanism by utilizing community structures. Xiv preprint or arXin:2312. (enr:2322.667857 16788, 2023.
Paul W Holland, Kathryn Blackmond Laskey, and Samuel Leinhardt. Stochastic blockmodels: First steps.
Social networks, 5(2):109- 137, 1983.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasels for machine learning on graphs. Advances in newral information processing systems, 33:22118 1122218-22113,, 2020.
Borja Ibarz, Vitaly Kurin, George Papamakarios, Kyriacos Nikiforou, Mehdi Bennani, Róbert Csordás, Andrew Joseph Dudzik, Matko Bošnjak, Alex Vitvitskyi, Yulia Rubanova, Andreea Deac, Beatrice Bevilacqua, Yaroslav Ganin, Charles Blundell, and Petar Veličković. A generalist neural algorithmic learner. In Bastian Rieck and Razvan Pascanu (eds.), Proceedings of the First Learning Graphs Conference, volume 198 of Proceedings of Machine Learning Research, pp. 2:1-2:20 2:23. PMLR, 19-12 Dec 2022. URL fttps::/proceedd1gg mlr. press/v190t 10008///aaae barz22a. html
Andrew Jaegle, Sebastian Borgeaud, Jean- -aptisae Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. In International Conference on Learning Representations.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning, pp. 4651 4664. PMLR, 2021.
Bo Jiang, Ziyan Zhang, Doudon Lin, Jin Tang, and Bin Luo. Semi-supervised lcarning with graph learning- convolutional networks. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pp.
2019.
Shixiong Jing, Lingwei Chen, Quan Li, and Dinghao Wu. H 2 gnn: Graph neural networks with homophilic and heterophilic feature aggregations. In International Conference Dalabase Systems for Advanced Applications, pp.
Springer, 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0880 083611, 2020.
Dongkwan Kim and Alice Oh. How to find your friendly neighborhood: Graph attention design with sclf-supervision. In International Conference on Learning Representations.
Thomas N Kipl and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint 1119: 02907,, 2016.
Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Combining neural networks with personalized pagerank for classification on graphs. In International conference on learning representations, 2019.
Mario Michael Krell, Manuel Lopez, Sreenidhi Anand, Hatem Helal, and Andrew William Fitzgibbon. Tuple packing: Efficient batching of small graphs in graph neural networks. arXiv preprint arXin:2209.06333 2022.
Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral attention. Advances in Neural Information Processing Systems, 34: 21618-2 221182 2021.
Yann LeCun, Yoshua Bongio, and Gooffrey Hinton. Deep learning. nature,
2015.
Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstodt, Jeremy Reizenstein, and Grigory Sizov. xformers: A modular and hackable transformer modelling library. 2022.
15

---

Published in Transactions on Machine Learning Research (12/2025)
Derck Lim, Joshua David Robinson, Lingxiao Zhao, Tess Smidt, Suvrit Sra, Haggai Maron, and Stefanie Jegelka. Sign and basis invariant networks for spectral graph representation learning. In The Fleventh International Conference on Learning Representations.
Derek Lim, Felix Hohne, Xiuyu Li, Sijia Linda Huang, Vaishnavi Gupta, Omkar Bhalerao, and Ser Nam Lim. Large scale learning non-homophilous graphs: New benchmarks and strong simple methods. Advances in neural information processing systems, 14:20887-20902. 2021.
Chuang Liu, Yibing Zhan, Xueqi Ma, Liang Ding, Dapeng Tao, Jia Wu, and Wenbin Hu. Gapformer: Graph transformer with graph pooling for node classification. In Proceedings of the 32nd International Joint Conference on Artificial Intelligence (IJCAT-23), pp. 2196-2205, 2023.
Hao Liu, Jiarui Feng, Lecheng Kong, Ningyue Liang, Dacheng Tao, Yixin Chen, and Muhan Zhang, One for all: Towards training one graph model for all classification tasks. In The Twelfth International Conference Learning Representations, a.
Jingzhe Liu, Haitao Mao, Zhikai Chen, Tong Zhao, Neil Shah, and Jiliang Tang. Towards neural scaling laws on graphs. In The Third Learning on Graphs Conference, b.
Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Represenlations.
Yi Luo, Guangehun Luo, Ke Yan, and Aiguo Chen. Inferring from references with differences for semi- supervised node classification on graphs. Mathemalics, 1068):1262, 2022.
Haitao Mao, Zhikai Chen, Wenzhmo Tang, Jianan Zhao, Yao Ma, Tong Zhao, Neil Shah, Mikhail Galkin, and Jiliang Tang. Position: Graph foundation models are already here. In Forty-first International Conference on Machine Learning, 2024.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-based recommendations styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information relrieval, pp. 43 52, 2015.
John Palowitch, Anton Tsitsulin, Brandon Mayer, and Bryan Perozzi. Graphworld: Fake graphs bring real insights for gnns. In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 3691 1-001, 2022.
Hongbin Pei, Bingzhe Wei, Kevin Chen Chuan Chang, Yu Lei, and Bo Yang. Geom-gen: Geometric graph convolutional networks. In 8th International Conference on Learning Representations, ICLR 2020, 2020.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
Ladislav Rampášek, Michael Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for general, powerful, scalable graph transformer. Advances in Newral Information Processing Systems, 35:14501-14515, 2022.
RA Rossi and NK Ahmed. Networkrepository: An interactive data repository with multi-scale visual analytics, 2014, eprint arxiv, 2014.
Benedek Rozemberezki, Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding. Journal of Complez Networks, 92):cnab0114 2021.
Hamed Shirzad, Ameya Velingker, Balaji Venkatachalam, Danica J Sutherland, and Ali Kemal Sinop. Exphormer: Sparse transformers for graphs. In International Conference on Machine Learning, pp. 31613-31632 PMLR, 2023.
Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-Junc Hsu, and Kuansan Wang. An overview of microsoft academic service (mas) and applications. In Proceedings of the 24th international conference world wide web, pp. 243 246, 2015.
16

---

Published in Transactions on Machine Learning Research (12/2025)
Ziheng Sun, Qi Feng, Lehao Lin, Chris Ding, and Jicong Fan. Graphprop: Training the graph foundation models using graph properties. arXiv preprinl or arXix:2508.00: 04594, 2025.
Jake Topping, Francesco Di Giovanni, Beujamin Paul Chamberlain, Xiaowen Dong, and Michael M Bronstein. Understanding overssquashing and bottlenecks on graphs via curvature. In International Conference on Learning Represenlations.
Anton Tsitsulin, Benedek Rozemberezki, John Palowitch, and Bryan Perozzi. Synthetic graph generation benchmark graph learning. arXiv preprint arXiv:2204.01. 01336, 2022.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Lio, Yoshua Bengio, et al. Graph attention networks. stat, 1050(20):10 48550, 2017.
Zehong Wang, Zheyuan Liu, Tianyi Ma, Jiazheng Li, Zheyuan Zhang, Xingbo Fu, Yiyang Li, Zhengqing Yuan, Wei Song, Yijun Ma, al. Graph foundation models: A comprehensive survey. Xi preprint arXiv:2505.15116, 2025.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Scbastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions Machine Learning Research.
Zhanghao Wu, Paras Jain, Matthew Wright, Azalia Mirhoseini, Joseph E Gonzalez, and lon Stoica. Repre- senting long-range context for graph neural networks with global attention. Advances in Neural Information Processing Systems, 34:13266 13279, 2021.
Jun Xia, Chengshuai Zhao, Bozhen Hu, Zhangyang Gao, Cheng Tan, Yue Liu, Siyuan Li, and Stan Li. Mole-bert: Rethinking pre-training graph neural networks for molecules. In The Fleventh International Conference Learning Representations.
Lianghao Xia and Chao Huang. Anygraph:
Graph foundation model in the wild. arxiv preprint
arXiv:2408. 10700, 2024.
Yujun Yan, Milad Hashemi, Kevin Swersky, Yaoqing Yang, and Danai Koutra. Two sides of the same coin: Heterophily and oversmoothing in graph convolutional neural networks. In 2022 IEEE International Conference on Data Mining (ICDM), pp. 1287-1292. IEEE, 2022.
Han Yang, Chenxi Hu, Yichi Zhou, Xixian Liu, Yu Shi, Jielan Li, Guanzhi Li, Zekun Chen, Shuizhou Chen, Claudio Zeni, et al. Mattersim: A deep learning atomistic model across elements, temperatures and pressures. arXiv preprint or Xrxi::2405.0 04967, 2024.
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie- Yan Liu. Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems, 14:28877-28888 28888, 2021.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference Learning Representations.
Xinglong Yu, Chang Zhou, Yuan Fang, and Xinming Zhang. Text-free multi-domain graph pre-training: Toward graph foundation models. arXio preprint (ti::2205.11933. 2024.
Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. Graphsaint:
Graph sampling based inductive learning method. In International Conference on Learning Representations.
Zaixi Zhang, Qi Liu, Qingyong Hu, and Chee Kong Lee. Hierarchical graph transformer with adaptive node sampling. Advances in Neural Information Processing Systems, 35:21171-2 11:21171-,188,, 2022.
17

---

Published in Transactions on Machine Learning Research (12/2025)
Haihong Zhao, Aochuan Chen, Xiangguo Sun, Hong Cheng, and Jia Li. All in and one for all: A simple yet effective method towards cross-domain graph pretraining. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4443 4454, 2024a.
Jianan Zhao, Hesham Mostafa, Mikhail Galkin, Michael M Bronstein, Zhaocheng Zhu, and Jian Tang. Graphany: A foundation model for node classification any graph. CoRR, 2024b.
Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra, Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in neural information processing systems, 33:7793 7804, 2020.
18

---

Published in Transactions on Machine Learning Research (12/2025)
Appendix
A Additional Model Details
A.1 Model Configuration Details
We used pretrained configuration of models small (398K), medium (18M) and large (75M) for our analysis. Details of the configuration for each model are given Table [A] In the first cross-attention layer, used Flash Attention, whereas for all subsequent attention layers, we used Memory-Efficte Efficient Attention. Both implementations were sourced from the xFormers library [Lefaudeux et al. 2022
Table A1: Architectural details of GraphFM for different parameter sizes used in Section 3.2
Parameter Count
75M
18M
389K
Num Latents (K)
512
256
32
Latent Dimension
512
256
32
Cross-Attention
Heads
4
FFN hidden dim
2048
1024
128
Self-Atee Self-Attention
Depth (L)
12
10
4
Hoads
8
4
FFN hidden dim
2048
1024
128
Node Decoder
Depth (M)
4
4
2
Heads
8
4
4
FFN hidden dim
2048
1024
128
A.2 Rescaling the learning rates for different graph sizes
When training on variable sized graphs, the MLP and linear decoder for each dataset receive updates based on the number of nodes from their respective datasets present in the batch and thus smaller graphs get updated less frequently when compared to large graphs. To mitigate this imbalance, learning rates for the feature MLP and linear decoders. Since they receive updates less frequently, when they do, we use a larger learning rate to update them. Without this adjustment, the weights of the common Perceiver encoder and node decoder would advance more quickly than those of the dataset-specific components, potentially leading to suboptimal learning for smaller datasets.
implemented dataset-spocific
A.3 Fine- TineTTning Strategies
In our evaluation of GraphFM's generalization capability, we employed two fine-tuning strategies aimed at adapting the model to the unseen test datasets.
A.3.1 Low-resource MLP Fine-tuning (MFT)
This approach is designed to assess how well the pretrained model performs out-of-the-box different test graphs without extensive training. In MFT, we freeze the pretrained model and only fine-tune a lightweight multi-layer perceptron (MLP) on top of the learned representations. This strategy allows us to quickly adapt the model to new tasks while relaining the majority of the original learned parameters. MFT is particularly useful in resores settings, where computational power or time is limited, as it requires minimal additional training while still providing insight into how well the pretrained model gencralizes. For all MFT experiments, used a learning rate of 10-3 and a weight decay of 10 optimized using the AdamW optimizer (Loshchilov) & Hutter].
19

---

Published in Transactions on Machine Learning Research (12/2025)
A.3.2 MLP and Node Decoder Fine-tuning (NFT)
In contrast to MFT, the NFT strategy involves fine-tuning part of the model and is recommended when sufficient computational resources are available and the goal is to extract the maximum performance from the model. In NFT, we gradually unfreeze the node decoder, enabling the model to more effectively adapt to the new datasct. Specifically, we set predefined epoch U at which unfreezing begins, starting from the bottom layers of the node decoder. After every S epochs, additional layers are unfrozen in bottom-up manner, facilitating gradual transition to full finetuning of the model. Concurrently, the learning rate is decayed by factor of 1.5 each time new layer is unfrozen, ensuring controlled parameter updates. For all datasets, we tune the hyperparameters U and S, with U set to 10, 20, or 30 epochs and S set to 5 or 10 epochs. This gradual unfreezing mitigates training instability, as smaller perturbations are made to higher-level feature representations. As a result, NFT allows for better adaptation, particularly for unseen testing datascts, and is well suited for cases where exploiting the capacity of pretrained models critical.
B Additional Details on Datasets
B.1 Pretraining datasets
The largest model (75M parameters) was trained on 80 real world and 72 synthetic datasets. The real world datasets and their characteristics are given in Table A3
The synthetic datasets were created using the GraphWorld (Palowitch et al. 2022 using the Stochastic Block Model (Holland et al., 1983). The generator parameters are listed in Table A2 In the graph generation process, the node homophily ratio is varied. The homophily is given by the following formula:
K(e, w) w E N(u) Ay. = swHl [((r)
M
NEV
where denoles the set of all nodes the graph, Nvv) denotes all the neighbors of an arbitrary node v, and ye denotes the class membership of the node V. We classify datasets into homophilic datasets and heterophilic datasets based on the homophily score: datasets with homophily 2 0.5 are classified as homophilic datasets and helerophálic datasets otherwise.
B.2 Details on small and medium scale dataset
The small and medium scale datasets, as discussed in Section 5.2 were created by taking random subset of the large dataset(80 real and 72 synthetic).
Dataset subset for small scale data: The following datasets were used to train models with small scale data: Wiki, BlogCatalog, Roman-empire, Minesweoper, Tolokers, Questions, Twitch-FN,, Twitch-FR, Twitch-PT, Twitch-RU, DeezerBurope, GitHub, LastFMAsia, Airports-USA, Airpotss-Europe, PolBllos and EmailEUCore
Dataset subset for medium scale data: The following datasets were used to train models with medium scale data: Wiki, Bltgoo,,log, Roman-empire, Minesweeper, Tolokers, Questions, Twitch-EN, Twitch-FR, Twitch-PT, Twitch-RU, DeezerEurope, GitHub, LastFMAsia, Airports-USA, Airports-Europe, PolBlogs and EmailEUCore, Reddit, Reddit2, Flickr, Yelp, PPI, Facebook, Amazon-ratings, Minesweeper, Twitch- DE, Twitch-FSS FacebookPage Airports- Arrports-Brazil, penn94, reed98, amherst41, johnshopkins55, genius, CitationFull-Citeeeee EitationFilllCoraee and CitationFalllPubMMee
B.3 Details on social and biology domain datasets
The social and biology datasets, discussed in Section D.1 and Section 5.2 included the following subsets:
20

---

Published in Transactions on Machine Learning Research (12/2025)
Tahle A2: Graphworld generator parameters for synthetic graphs
Parameter Name
Description
Values
nnertex
Number of vertices in the graph.
[32, 500000)
p/q ratio
The ratio of in-cluster edge prob- ability to out-cluster edge proba- bility.
[0.1, 10.0]
avg. degree
The average expected degrees of the nodes.
[1.0, 20.0]
feature center distance
The variance of feature cluster centers, generated from multi- variate Normal.
(0.0, 5.0]
num clusters
The number of unique node la- bels.
[2, 6]
cluster size slope power exponent
The slope of cluster sizes when index-ordered by size.
[0.0, 0.5]
The value of the power law expo- nent. used to generate expected node degrees.
00.5, 1.0]
Dataset subset for social domain: The following datasets were used to train the social-specifie model: fb-CMU-Carnegie49, Yelp, BlogCatalog, Facebook, Twitch-DE, Twitch-EN, Twitch-ES, Twitch-FR, Twitch-PT, Twitch-RU, DeezerEurope, GitHub, FacebookPagePage, MAsia, penn94, reed98, amherst41, johnshopkins55, genius and soc-pokec.
Dataset subset for biology domain: The following datasets were added part of the biology domain to train the combined social and biology model: BZR, DD, DD199, DD21, DD242, DD244, DD349, DD497, DD6, DD68, DD687, DHER, ENZYMES, ENZYMES118, ENZYMES123, ENZYMES295, ENZYMES296, ENZYMES297, ENZYMES8, KKI, OHSU, PROTEINS-full, Peking-1, Tox21 _p3, gene, protcins-all and PPI.
B.4 Finetuning Datasets
For our evaluations, we held out a number of datasets that are used for standard benchmarks in both larger scale node classification and heterophilic graphs.
B.4.1 Homophilic Datasets
We use five real-world datasets, Amazon Computers and Amazon Photos (McAuley et al. 2015), Coauthor CS and Coauthor Physics (Sinha al. 2015) and Obgn-Arxiv (Hu et al.) 2020). Key statistics for the different datasets are listed in Table A3 in the cretuning-sectionn The experimental setup follows that of Luo et al. 2022), where split the dataset into development and test sets. All the hyperparameter tuning is donc on the development set and the best models are evaluated on the test set. The runs are averaged over 20 random splits to minimize noise. We follow a 20:20:20% train/val, /aa//vvllleeeee /lest split for the Amazon and Coauthor datasets. For Obgn-Arxiv we follow the experimental setup used in (Hu et al. 2020). The results for the Coauthor-Physice Coauthor-CS, and Amazon-Photon obtained from Table A8 have been sourced from
Liu et al. 22023.. The results for the Amazon-Come dataset are taken from (Hoang al. 2023) except for
ML.P which was obtained from (Luo al. 2022
21

---

Published in Transactions on Machine Learning Research (12/2025)
Table A3: Pre- Traininnn Datasets and their characteristics
Dataset
Number of Graple
Nodes
Rddge
Homophilyy Ratin
Eiceess Homophilyy
Avernge Degren
Node Prattures
Node Clossee
Tearning Rate
BA-1 10, 93-15
504
66410
6.2
0.0001
115.45
0.0014
BA-2 24, 90-0122
13776
639770
0.5
0.0014
119.03
DU0287
405
35.75
70.71
01:1
000157
0.87
53
0.0055
CL-1000 108 Lo
112622
1772180
0.11
01000
8,00
0.000154
CL. L-5-1-5--8--5
13030
64595
6.2
0.057
8.56
0.00000
DD
1178
254.32
1651.2
0.07
0.0015
0.068
89
0.00055
DM119
511
1932
0.067
00131
1.52
0 0,0055
DU21
5744
14267
0.07
4.56
B
DU342
1284
3335
0.05
0 0.00
DD244
0.0085
DI
2155
DUE
4152
10333
0.07
010136
DD&
775
2008
0.072
00005
0U025
725
2030
0.06
0.0060
7.17
DU022
DAPR
756
12.13
50.00
0.3
010189
0..0
00018
ENZYYIES
DOD
124
0.67
1770
00.00
18
DURGO
LNGYMIESTI
96
121
0.58
4373
2.53
ENZYMESS12 28
90
127
0.62
0.2111
2.52
RL0076
ENZY MISS75
121
135
0.71
83888
2.30
D.0056
125
141
0.73
2.30
0.00057
122
140
0.63
1180000
2.44
ENZYMESS
88
153
0.77
088181
3.60
ER- Degg.A 0-10000--- L.2
05037
400192
0.00
0.50
Filt AagtA (--003C-1.
194153
0.33
00001
0.50
0.0013
KKJ
25.96
96.84
0.33
BL0012
MSTCC-21
77.52
306.05
0.74
MSSCC-21C
209
198.20
VISRCC-
10.58
195-21
0.09
0.36
0.00P
QUSU
52.01
200.22
0L00A22
0.266
PLLC- 40 LS
13025
437777
0.2
75.45
BARING
PLCC-0030-14 30-1.
117672
7046491
0.5
6,2060
119.54
PROTTENS--6ll
1113
59.00
135-63
0.97
1916
0.55
00063
Paking-1
59:51
154-71
0
0.41
150
SW-10000 6-005-1---1222
100001
200000
0.5
SW-10000-4------
10000
50000
0.2
0.0012
SYNTHETIC
300
100
392
0.15
0L0374
0.16
Pre- Trakiin
Ternonstlll
581
8592
0.90
0.3133
19.51
DOISS
Ticza paa
151563
11:0046
0.02
4.00
45
TN-CMUUCC Coomingeet
0537
200967
6.5
75.33
DODDO
1100
1072
0.4
0.6E67
3.60
0,012
peocatno-all
15171
163363
0.06
03710
7.16
0.00075
mollity-rall all
27058
51300
0.9
0.0
15
Hecdit
2021055
114615492
0.70
5653.10
BL000S
2odilc2
202065
20213834
0.75
Flickz
89250
890756
0.91
20.16
710817
13951019
38:33
Will
2416
1788
0.71
4073
BlnaCarako
6190
1728
0.40
0.2680
182.21
5150
D.000
R59LL
0.03
121
Farsbook
1999
89201
0.99
09
1968
193
DO01
Roman empire
22022
000
5,51
310
16
D0074
Amooon eerings
24092
196110
0/46
11 1200
15.2
3001
Minooweppe
10000
76801
0.6
0L00044
15.76
Thlnkans
11736
100500
0.59
11 1900
175.50 54
10
Quoecienx
18921
088
E007500
12.55
301
0L00G
Twitch- DU:
1428
0311
11 111001
68.40
1356
Twild- EN
7120
77774
0.00
01 1171
21.82
195
Twildi- ES
6948
129112
0.00
D1 116644
58.10
Twiidd-FR
6551
231583
0:31
0.0306
20.79
D.0000
Twitccs PT
1912
B4510
0.34
67.47
BU0022
Twitch- IU
G355
THHIS
0.00
0107767
36:02
BLO011
Docope
25251
156504
18.11
BLOONO
GitHob
377/0
22170
912101
0.65
08.05
0.000SS
Lasti
7924
50612
0967
7156
Aiports Brazil
181
1074
0.46
16.20
0U0013
Aaports Enzape
399
2015
0.90
1590
10.05
Airpur-U USA
1100
13599
0.65
0.2571
22.85
00000
Pollliggs
1400
19025
0.91
25. 15.54
10001
100G
25571
0.96
0.2364
60.89
42
R00022
41634
2724458
0.61
111.11
4814
DU1064
medli
902
S7621
0.53
00213
76.22
715
0.002
ambesst41
2235
151916
0.53
362.74 73
3152
0
Jchnecophinnss
51M1
373177
0.55
01825
144.05
2006
421961
052970
0.62
0.08040
4.67
12
0.00040
Ohation Full-CieeSerrr
(25)
10574
0.95
09157
5.65
600
0.001
Casdiio Cull-Co
2046
16516
0.75
10.80
2879
D.022
Ottdton Pubbled
19717
58555
0.80
0.9544
8.50
310)
U.IIIST
soc-polzee
1632303
30522564
0.41
0.00019
Mulii lsled linury clusilication.
Table A4: Pine-Tuning Datasets and Their Characteristics
Datasct
Number of Graphs
Nodes
Edges
Homophily Ratio
Average Degree
Node Fratures
Node Classes
Actor
1 7600
30019
0.21
7.89
932
Amazon- Computers
13752
4421722
0.77
71.51
767
Amazon-Piatte
1
7650
238162
0.82
62:26
745
Coauthur- CS
I
18333
163788
0.50
17.86
6805
15
Coauthor-Plyse
1
34493
495924
0.93
28.75
8415
5
Chameleon
1 2277
36101
0.23
31.70
2325
22

---

Published in Transactions on Machine Learning Research (12/2025)
B.4.2 Heterophilic Datasets
We use five real-world datasets with graphs that have a homophily level < 0.30, Texas, Wisconsin and Actor (Pei et al. 2020) and Chameleon and Squirrel (Rozemberezki et al. 2021 p. Key statisties for the different. datasets are listed in Table A3 in the finetuning-section. We follow the experimental setup in (Pei al., 2020), and use the same 10 train/val/tor
splits that provided. The results for GCN based methods and helerophily based methods in Table A9 have been Laken from (Azabou el al. 2023b and the results for transformer based methods have been taken from (Liu et al. 2023
B.5 Standard hyperparameter search grid for baselines
The hyperparameter search space grid used for tuning baselines for Table A6 is detailed in Table
A5
Table A5: Hyperparameter Search Space
Hyperparameter
Type
Range
Hidden Dim
Categorical
{16, 32, 128)
Depth
Categorical
{1, 2]
Dropout
Uniform
[0.0, 0.9]
Learning Rate
Log uniform
5e-5, 5e-1
Weight Docay
Log uniform
1c-5, 1o-2
B.6 Detailed Comparison with Specialist Models
On both homophilic and heterophilic benchmarks (Table A6 GraphFM performs on par with state-of-the-an specialist models trained from scratch each dataset. While the best-performing baseline model varies across datasets, GraphF'M consistently ranks among the top three: the NFT fine-tuning strategy achieves the highest average rank overall, while MFT is tied for second place with NAG. Furthermore, MFT demonstrates significantly lower variance in rank compared to NAG, whose rankings display a more bimodal distribution across datascts. This indicates that GraphFM provides more stable performance across diverse graph structures.
Specialist models such H2GCN and NAG show variability in performance due to their design focus. H2GCN, tailored for heterophilic graphs, performs strongly on heterophilic datasets but struggles with homophilic ones. Conversely, NAG, optimized for homophilic graphs, excels in homophilic settings but is less effective on heterophilic datasets. These results highlight the trade-offs inherent in models designed for specific graph types, limiting their generalization capabilities across diverse datasets.
In contrast, GraphFM achieves strong performance across all datasets without requiring extensive hyper- parameter tuning, unlike the specialist. models that were fine-tuned for each dataset. By using single
Table A6: Results on a variety of homophilic and heterophilic node classification benchmarks. From left to right, we show different message passing and graph transformer architectures, and then GRAPHFM in both the lightweight MI.P-only finetuning (MFT) and node decoder finetuning (NFT). The top three numbers are bold, with the highest in bright red fading to black. Models are ranked all 10 datasets and the average and standard deviation ranking is at the bottom.
GCN
ML.P
GAT
H2GCN
SAN
NAG
GraghPM-MPT
GrapPMM-NPT
Homopphil
Physics CS
95.3810.20
05.12:1:20
96.83:10.15
96.66110.16
96 96.6120.17
96.77:0.12
94.06+0.16 1616
02.00+0.31 51
03.6140.144
94002-031
94.16+0.36
95.00++0.14
96.19=0.21
95.24-0.16
Photo
$5.94+11.18
88.06+0.85
87.1341.00
01.56
94.17
94.64+10.00
93.11+182
00.37
Computee
$0.4740..66
$1.63
90.78.0.113
89.33.10.27
80.5 80.63.10116
01.22.10.4
00.95.10.53
90.07.0.21
Oghn
70.40=00.10
52.6340.12
07.5640.12
68 08.2070.67
00.17:0.115
08.21+00.2'
69.96:0.21
70.01
Heberophiäe
Texas
55.1445.10 16
80.81::3.11
52, 1616
84.80=7.220 7.25
60.17:6.66
08:37:5.00 27"
80.81:22.76
82.16:13.24
Wisconsin
51.76133.6
85.2913..3
19. 18:11.11.09
87. 07.05544.90
51.37
08.23.5.
83, 13.133...35
83. 83.62.13.21
Actoe
27.32:1..00
36.6310.70
27.44:10.00
36
27.32:1.10
36i.29::0.66
1.00
Chacueleon Squirrel
38. 18..1111992 92
26,2111 99
38. M1.111100
60.11:2.15 2.15
44.32:1175"
57.3910
58, 38.64::::24
59.12:1166
31.5210.71
28.77:11.56
16.48.1:.96
30.922:2.14
49.93:10.07
42.60:11.54
42.98.11.62
Avg Rank (Homophilic)
5.2 2: 2.6
7.8 0.9
6.0 d: 2.2
5.6 d: 0.9
3.4 1.5
2.8 d: 2.0
3.4 1: 0.9
2.0 =: 0.7
Avg (leterophillie)
2.0
Avg Rank (Overall)
5.9 1.9
5.8 d: 2.6
6.3 d: 1.9
4.0 4: 2.2
5.0 2.0
3.4 d: 1.9
3.3 4: 0.7
2.0 0.7
obtained thoough lopernammeter
23

---

Published in Transactions on Machine Learning Research (12/2025)
hyperparameter configuration (learning rate 10- GraphFM consistently achieves competitive rankings. Additionally, the NFT fine-tuning strategy provides significant benefits for challenging datasets such Amazon-Phooe and Actor, as allowing parts of the model to remain learnable enables better adaptation to out-of-distribution datasets.
C
Additional Details on Multi-Graph Training
One key aspect of our work is lesting scale. Thus, to build a model across large amounts of diverse graph data, we developed number of approaches for efficient training and multi-GPU
10a
spoeh timee (minutes)
usage.
Figure A1 shows an ablation study the epoch time for various GPU optimizations we have proposed in Section 33.2 The epoch time was calculated using the medium-sized model with 18M parameters, as detailed in Appendix A.1
Note: Removing chaining made it impossible to run the largest model (75M parameters) with our available computational re- sources (8 A40 GPUs). Therefore,
graphFM
graphFM
graphFM
-sampler
sampllar
performed the ablation using the medium-sized model. This highlights the significance of our optimization techniques, which enabled us to scale up and run such large models efficiently.
Figure A1: Ablation for GPU optimitzations Epoch time in minutes on removing varions gpu optimizations proposed for CRAPaFM.
C.1 DistributedSSampler
In designing this sampler, we prioritized ensuring that it neither introduces bias into the data sampling process nor alters the distribution of the graphs from the datasets. Its primary function is to enhance batch construction and distribution across GPUs.
First, the sampler defines a set of N buckets with a fixed node budget B, where N can be the number of GPUs and is the node-level batch size. The graphs (across all GPUs) are sorted in descending order based upon their size. The sampler then employs bidirectional filling strategy within the buckets. The distribution process, described in Algorithm | involves distributing graphs in snake-like pattern, initially filling from right to left, then switching to left to right and so on. When graph is added to bucket, it uses up part of the budget, equal to its size. This method effectively pairs larger graphs with smaller ones in subsequent passes, preventing the concentration of multiple large graphs on the same GPU, thus achieving efficient load balancing and uniform GPU utilization. Figure A2A shows an overview of how the sampler distributes the graphs into buckets. We find that stability is improved with larger number of buckets (Figure B). When the number of GPUs is fixed, which artificially increases the number of buckets by factor equal to the number of accumulation steps, without biasing the sampling process.
achieve larger N by using gradient accumulation,
A
GPU Memory Usage (%6)
90
80
Larget and wmadlont gryhhcthaee buctain
70
60
50
baseline
N
64
Figure Multi-CPU utilization: A: diagram visualizing our sample distribution strategy. B: GPU memory utilization during distributed training when using the default batch sampler vs. our DistributedSSSampler for and N=64 buckets.
24

---

Published in Transactions on Machine Learning Research (12/2025)
Algorithm Distribute graph nodes into virtual GPU buckets
1: input: Batch size B, Bucket count N, Graphs in the dataset G = (9o, GI....> Subgraphs sampled for this minibatch Gm {96 gim....] precondition: Σ, 9 N initialize:
buckets array of N empty arrays
# will store subgraphs in each bucket # will store number of nodes in each bucket
counts array of N zeroes
6: b+0
# bucket index # direction
7:
d+1
8:
Sort gm according to node-counts in 9, largest graph goes first
for all 9mm in 9mh do
10: while 9TTh do
11:
if counts(b] < B then
12:
# insert part of 9 into bucket 6 n m B counts(b)
13:
14:
counts[b] < counts(b] +
15:
append first n nodes of 9.n to buckets(b) remove first n nodes from G.m end if
16:
17:
18:
# go to the next bucket, switching direction at the boundaries b b b+ d
19:
20:
if b > N or b < 0 then
21:
d + -d < b+
22:
23:
end if
24: end while
25: end for
26: return buckets
25

---

Published in Transactions on Machine Learning Research (12/2025)
C.2 GraphSAINT Random Walk Sampler
Efficient ncighborhood sampling for large graphs is crucial for our node decoder, as traditional methods for k-hop neighborhood sampler often become computationally prohibitive with the increasing size and complexity of the graph data. To overcome these limitations, we have adopted the GraphSAINT Random Walk Sampler (Zeng et al.), specifically designed for efficient sampling in large-scale graphs.
C.3 RAM Optimization in Multi-GPU Environments
In multi-GPU training environments, efficient use of system memory is crucial, especially when handling large graph datasets. Traditional approaches lead to substantial memory redundancy, each GPU process typically loads a complete dataset into system RAM. This results in each process duplicating the dataset in system memory, leading to inefficient memory usage and potential system overload.
To address this, we utilize a shared memory management approach using Python's multiprocessing. Mangessern to coordinate datasct access across multiple GPU processes. This method ensures that each dataset is loaded into RAM only once, regardless of the number of GPUs, thereby avoiding duplication and conserving memory resources.
D Additional Experiments
D.1 Separating pretraining datasets into different domains
We further stratified our pretraining dataset to investigate the effects of cross-domain training, and created three models that contained: (i) graph datasets from "social domains" including product graphs and citation networks (1.3M tokens), (ii) both the social datasets and all bio- logical graphs in the dataset (Bio+Soc, 2M tokens), and
0.76
0.750
0.75
mean accuracy acooss test test datasets
0.74
0.735
0.73
(iii) compare with our model trained on all data including sytnthetic graphs (7.3M tokens).
0.722
0.72
0.714
When comparing graph features across social and bio- logical domains, we found distinct structural differences: biological datasets gencrally exhibited higher levels of het- erophily, lower average degree, and fewer edges, whereas social graphs showed more homophily, higher degrees, and denser connections (Figure A4B). Synthetic graphs added wide range of characteristics, particularly increasing the number of heterophilic graphs used in pretraining, which contributed to broader diversity of features (Figure A4A). To isolate the contribution of synthetic graphs, we further stratified the pretraining data to include only real-world datasets.
0.71
0.70
Soc
Soc + Bio
all data
all data
-Synthetic
Figure A3: Domain Scaling: Average accuracy
unseen testing datasets (using MFT) for models trained different subsets of data
All three models were then fine-tuned on four homophilic datasets (coauthor-CS, coauthor-physics, amazon- photos, and amazon-computers) and five heterophilic datascts (Texas, Wisconsin, Actor, Squirrel, and Chameleon) held out for fine-tuning.
As shown in Figure A3 incorporating biological datasets, despite being seemingly unrelated to the target domains, improves performance on unseen test datascts, with mean accuracy incrcasing from 0.714 (Soc) to 0.722 (Soc + Bio). This suggests that knowledge learned from the biology domain positively impacts performance in other domains. Extending pretraining to all available real-world datasets from Table A3 yields additional improvements, with mean accuracy reaching 0. 0.735. Finally, adding synthetic graphs boostod performance even more, with a mean accuracy of 0.750, indicating that diversity (not just domain specific data) is the key improving generalization.
26

---

Published in Transactions on Machine Learning Research (12/2025)
20
Synthetic Real
30
frequency
15
15
15
20
10
10
10
5
0.5
10
10a
10
homophily ratio
average degree
total number
total number of edges
of nodes
Social Biology
6
frequency
7.5
5.0
0.0
0.0
1.0
10P
10a
10a
10a
homophily ratio
average degree
total number of nodes
total number of edges
Figure A4: Characteristics of graph datasets used train GraphFM: From left right, compute the histograms of the homophily ratio, average degree, number nodes and number edges all 152 graphs used during training. The homophily ratio provides measure of how frequently a directly connected to other nodes the same class.
D.2 Scaling analysis breakdown for different test datasets
The main text reports the average effect of scaling. In Figure A5 provide dataset-level breakdown. While all datasets benefit from increased model and data scale, the magnitude of improvement varies, with more challenging datasets (e.g., Chameleon) showing larger relative gains compared to easier ones (e.g., Coauthor- Coauthor-Physics)
D.3 Ranking of different models
We compare the mean rank of GraphFM against special- ist baselines across the 10 out-of-distributiio datasets (Fig- ure A6 Unlike bascline models that require extensive hyper- parameter tuning for each dataset, we evaluated GraphFM using fixed configuration (learning rate 10 10-3) for both MFT and NFT strategics. For NFT, we additionally applied simple unfreezing schedule (Appendix A.3.2
Models
5.50
Mcan Rank dastaaatnnn
4.70
As shown in Figure A6 NFT achieves the best overall rank, while MFT achieves the second-best rank with the lowest variance. Specialist models such as H2GCN and NAG show higher variability rank due to their specialization for het- erophilic and homophilic graphs, respectively. A detailed per-dataset comparison is provided in Appendix B.6
2,10
M
0AT
CCN
54h
H220X
100
Greay'V
Model
Figure A6: Mean rank various models accross 10 unseen test datasets (lower is better).
D.4 Generalization to Graph Classification
To further evaluate the generalization of GRAPHFM, we conducted additional experiments on graph- and PROTEINS. In this setting, we extend the
level classification using molecular GRAPHFM architecture by adding cross-attention- graph decoder following the Perceiver encoder.
27

---

Published in Transactions on Machine Learning Research (12/2025)
Chameleon
Wisconsin
0.800
accuracy
0.55
0.775
389K
0.50
0.750
75M
0.45
10F
10"
10
109
10F
10
Coauthor Physics
Amazon Photo
0.97
0.90
accuracy
0.85
0.96
0.80
0.95
108
10
10
10
10
107
number tokens seen during pre-training
number tokens
seen during pre-training
Figure A5: Accuracy the model and dataset size are increased. Results shown for four datasets, Chameleon and Wisconsin (heterophilic), and Coauthor Physics and Amazon Photo (homophilic).
This decoder aggregates information from the latent. tokens produce graph-level representations, which are then used for label prediction.
We apply MLP fine-tuning while keeping the cross-attention decoder learnable. Table A7 shows GRAPHFM achieves sub- stantial improvements over existing graph transformer and message-passing baselines, demonstrating strong adaptability to graph-level prediction tasks.
Table A7: Graph classification results for
GRAPHFM (baselincs reported from (Guan
Model
MUTAG
PROTEINS
GCN
716 7.7
73.1
GraphSAGE
8.7
73.8
We note that the current version of GRAPHFM does not handle edge features, which is why MUTAG and PROTEINS were chosen for this experiments. Nevertheless, the strong results these benchmarks indicate that the model architecture is capable of generalizing to graph-level classification.
GT
7.9
68.4
Craphormer
744
68 68.7
SAT
814
7..7
SpecFormer
83.7 8.0
72.0 4.9
GraphFM (ours)
93.8 6.6
76.4 + 3.7
D.5
Additional Baselines
The main text presents comparison of GRAPHFM with baselines that are more consistently reported across provides additional baselines for all the unseen test datasets.
the literature. Table AS and Table A9
D.6 GraphAny Comparisons
GraphAny is primarily designed for zero-shot transfer, whereas focus with GraphFM is supervised and fine-tuned performance across diverse benchmarks. As shown in Tablle A10 GraphAny's absolute accuracy remains significantly below that of fine-tuned methods. On average, GraphFM outperforms GraphAny by 7.11% across datasets. For example, on ogbn-arziv, the best GraphAny model achieves 58.68% accuracy, compared to 70.01% with GraphFM after fine-tuning. These results highlight that while GraphAny demonstrates potential for zero-shot settings, substantial performance gains can still be obtained through multi-graph pretraining and task-specific adaptation with GraphFM.
28

---

Published in Transactions on Machine Learning Research (12/2025)
Table A8: Results node classification tasks for large graph datasets. We report the accuracy (%) with standard deviation 10 splits (OOM indicates Out of Memory).
Method
Photo
Physics
CS
ogbn-arxiv
Comp
GCN-based methods
GCN
Jiang al.
2019
85.94+1.18
95.38++0.20
94.06+0.16
70.40+0.10
89.47 + 0.46
GGCN
Yan
2022
67.84
95.8940.21
89.9412.24
62.71=1.76
APPNP
Kliepera al.
2019
84.71+1.25
95.04+0.31
87.49+0.48
70.20+0.16
90.18 + 0.17
GCNII
Chen al. 2020
67.0641.74
94.8820.32
81.23=0.78
69.78-0.16
GAT
Velickovic cl al
2017
87.13:11.00
95. 14....8
67.56.0.12
90.78 1 0.13
GATv2
Brody et al.
81.52#3.23 23
95.02+0.32
88.46=0.61
68.84=0.13
SuperG AT (Kim & Oh
85.83
95. 06.11110.2
88.1110.43
66.99-00.07
Heterophily-basee
methods
MLP
LeCun et. al.
2015
88.66.10.85
95. 05.12:10.26
0..59
52.63:10.12
84.63
MixHop Abu-El Hajjaa
2019
93.24=0.59
96.34=0.22
93.88=0.63
70.83=0.30
H2GCN
Jing al.
2024
91.56.10.70
96.2810.13
68. 08.2910.67
89.33 1 0.27
FAGCN
Bo et al.
2021
87.53+0.775
95.86++0.12
91.82=0.54
66. 16-1220.02
GPRGNN [Chicn et al.
2021
02.2710.44
96.06.10.21
93 00.60.66
68, 28...1
89.32 +: 0.29
Graph Transformer- basedd based
methods
SAN Kreuzer el al. 2021
94.17.0.665
96.83.10.18
94.16.10.36
69.1710.115 OOM
89.83 1 0.16
Graphormer Ying al.
2021
85.20+4.12
0OM
0OM
OOM
LiteGT CCen et
202
00M
92.16=0.44
0OM
UniMP WWang
2025
92.4910.47
96.8211..33
94.2020.34
73.19-0.18
DET
Gho al.
2022
91.44++0.49
96.30+00.18
93.34+0.31
55.70+0.30
NACphormer [Chen et al.
94.64-0.60
96.6622016
95.00=00114
68.21 = 0.021
91.22 0.14
GRAPHFM -MFT
93.01=1882 82
96.6120.177
95.19+0.21
26.016
89.95 = 0.83
GRAPHFM -NFT
94.37:10.35
96.77:10.12
95.24:10.18
70.01:0.118
90.07 1 0.21
Table A9: Results on node classification tasks for heterophilic graphs. We report the test accuracy acroes many heterophilic graph benchmark datascts. The standard deviation is reported across train/test splits.
Method
Texas
W Wisconsin
Actor
Squirrel
Chameleon
GCN-based
methods
GCN Jiang al.
2019
55.14 5.16
51.76
3.06
27.32 1 1.10
31.52 1 0.71
38.44
1.92
GAT
Velickovie
2017
52.16 6.63
49.41 4.09
27.44
1 0.89
36.77
1 1.68
48.36
1.58
GraphSAGE (aamllton
2017a
82.43 6.14
81.18
5.56
34.23
0.99
41.61 + 0.74
58.73
1.68
Heterophily-bese based methods
MLP
(LeCun et al.
2015
80.81 4.75
85.29 3.31
36.63 + 0.70
28.77 + 1.56
46.21
2.99
HH-GGN
Azabou ct al.
2023b
71.80 1 3.46
79.80
4.30
35.12 1 1.06
47.19
1 1.21
60.24 1: 1.93
HH-GAT
Azabou et al.
2023b
80.54 + 4.80
83.53 + 3.84
36.70 1 0.92
46.35 + 1.86
61.12 : 1.83
HH- TH--GrahhAAGe
Azabou et al
2023h
85.95 + 6.42
85.88 + 3.99
36.82 + 0.77
45.25 + 1.52
62.98 + 3.35
MixHop
Abu-El El Haija et al.
2019
77.84 7.73
76.88 + 4.90
32.22 4 2.34
43,80 + 1.48
60.50 + 2.53
GGCN
Yau el al. 2022
84.86 4.55
86.86
1: 3.29
37.54 1: 1.56
55.17 1 1.58
71.14
1.84
H2GCN
Jung
2024
84.86 7.23
87.65
4.98
35.70
1 1.00
36.48
1 1.86
60.11
2.15
LINKX
2021
74.60 8.37
75.49
5.72
36.10 + 1.55
61.81 + 1.80
68.42
1.38
Graph Transformer based methods
SAN Kreuzer et al. 2021
60.17 6.66
51.37
3.08
27.12 + 2.59
39.92 + 2.14
44.32
1.73
UniMP [Wngg et al.
2025
73.51 8.44
79.60
5.41
35.15 + 0.84
NAGphormun Chen cl ml
63.51 5.85
62.55
6.22
34.33 1 0.94
49.93 1 0.07
57.39
0.02
Gapformer (Liu et al.
80.27 + 4.01
83.53 + 3.42
36.90 + 0.82
GRAPHEM MFT
80.81 2.76
83.13 + 2.35
36.29 + 0.63
42.80 + 1.54
58.64
+ 1.24
GRAPHFM NFT
82. 16 3.24
83.62 + 3.21
38.01 + 1.07
42.98 + 1.62
59.
1.64
29

---

Published in Transactions on Machine Learning Research (12/2025)
Table A10: Comparison GraphFM GraphAny on node classification datasets.
Dataset
GCN
MLP
GAT
GraphAny (Arxiv)
GraphAny (Wisconsin)
GraphFM (MFT)
GraphFM (NFT)
Comp
58.82 + 2.98
85.83 + 0.86
87.01
0.50
83.04 1.24
82.09
1.22
95.13
0.45
95.44
0.47
Photo
68.20 + 0.88
91.88 + 0.79
91.86 = 1.07
90.60 = 0.82
90.18 + 0.91
93.01 - 1.82
94.37 = 0.35
CS
85.88 + 0.93
81.83 + 0.71
88.47 0.79
90.45 + 0.59
90.85 + 0.63
95.10 + 0.21
95.24 0.18
Physics
87.43 +
93.93 + 0.37
93.01 =
92.69 =
92.54 0.43
96.34 0.17
96.77 = 0.12
Arxiv
55.50 + 0.23
71.74 + 0.29
73.65 - 0.11
58.68 + 0.17
57.79 + 0.56
69.96 + 0.21
70.01 0.18
Chameleon
36.62 + 0.87
64.69 + 2.21
67.76 0.72
62.59 + (186
60.09
1.93
58.64 1.24
59.12 1.61
Squirrel
30.36 + 0.78
47.07 + 0.71
46.69 = 1.44
46.70 = 0.95
42.34 + 3.46
42.80 = 1.54
42.98 1.62
Texas
48.65 + 4.01
31.55 + 2.71
50.45 2.11
72.97 = 2.71
73.51 + 1.21
80.51 + 2.76
82.16
3.21
Wisconsin
66.67 +
37.25 + 1.64
52.94 =
71.77 = 71.18 5.08
70.92
1.52
73.63
1.87
Actor
33.95 + 0.80
28.55 +: 0.68
27.30 = 0.22
28.60 + 0.21
29.51 + 0.55
36.29 + 0.63
38.01 1.07
30


Reviews:
[{"review_id": "863XsR87FB", "reviewer": "Reviewer_WaZQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents GraphFM, a Perceiver-based transformer architecture for large-scale multi-graph pretraining across diverse domains. Rather than training a separate GNN for each dataset, the model learns a shared latent space through cross-attention between node embeddings and a fixed set of learnable latent tokens. This design allows the same model to handle graphs of different sizes and topologies efficiently.\n\nTo scale training across many datasets, the authors introduce a multi-graph packing scheme that avoids excessive padding, and a Distributed Snake Strategy Sampler that balances GPU utilization when mixing small and large graphs. The model is pretrained on 152 datasets spanning social, citation, and molecular graphs (7.4M nodes, 189M edges), and evaluated on unseen datasets.\n\nResults show consistent improvements with larger model and dataset sizes, and competitive performance with specialized GNNs and graph transformers, all while requiring minimal hyperparameter tuning. The paper also analyzes model stability, showing that pretrained representations lead to smoother optimization and less sensitivity to hyperparameter choice. Overall, the work demonstrates that cross-domain graph pretraining can yield broadly transferable representations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I believe the claims made are supported by clear and solid empirical evidence.\n\n1. The scaling analysis (Fig. 4) shows a clear monotonic relationship between pretraining scale and downstream accuracy.\n\n2. The efficiency gains from the sampler are substantial (roughly a fivefold speed-up) and well documented.\n\n3. Fine-tuning experiments under both MFT and NFT settings confirm robustness and stable convergence across tasks.\n\n4. Figures 6 and 7 demonstrate markedly lower sensitivity to hyperparameters compared with GCN and NAGphormer.\n\n5. Comparisons with recent graph foundation models, including GraphAny, are given in the appendix and show that GraphFM remains competitive across a range of domains.", "audience_yn": "Yes", "audience_explanation": "This paper aims to determine whether foundation-style pretraining can generalize across graph domains and does so at a scale not previously demonstrated. The work will appeal to researchers in graph learning, transfer learning, and large-scale representation learning. While more of a systems contribution than a theoretical one, the scope and execution make it broadly relevant to TMLR’s audience.", "requested_changes": "A list of my requested changes, as well as whether they are critical for securing my recommendation for acceptance, is given below:\n\n1. I would like the authors to demonstrate generality beyond node classification by including at least one graph-level or link prediction task. (critical for acceptance)\n\n2. Please quantify the contribution of synthetic datasets through an ablation study comparing training with and without them. I think this is a very interesting an important angle to investigate. (critical for acceptance)\n\n3. Add a brief interpretability or probing analysis to illustrate what information the latent tokens capture. (not critical, but recommended)\n\n4. Include a short discussion of scalability limits, i.e. how runtime and memory scale with latent count K and graph size. (not critical, but recommended)\n\n5. I would also like to suggest some minor corrections below:\n\n- (bottom of page 3): \"tthan Ng\" -> \"than Ng\"\n- Clearer captions for Figures 4 and 6 (i.e. adding a brief sentence about the overall purpose) would improve paper quality", "broader_impact_concerns": "This is a methodological paper with no obvious societal risks. The focus is on scalable representation learning and training infrastructure, so a dedicated broader impact statement is not necessary beyond standard comments on compute cost and data provenance.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rRYbPBwK3H", "reviewer": "Reviewer_dSYo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduces a novel pretraining approach that learns from a variety of graph learning datasets to improve generalization capabilities on unseen datasets. GraphFM combines existing architectures from previous works to allow for multi task pretraining. The authors claim that GraphFM as a scalable framework with pretraining, reduces or eliminates the need for training on specific downstream tasks.\n\nFurthermore, the authors introduce a new sampling method to improve GPU utilization and and sampling efficiency during pretraining.\nBoth contributions are supported by experimental results on various of downstream node classification datasets with GraphFM being scaled from 389K up to 75M parameters.", "claims_and_evidence_yn": "No", "claims_explanation": "One of the major contributions claimed by the authors is an improvement in generalization and results on downstream tasks. However, this claim is only partly supported by the experimental results shown in Sections 4, Appendix B.6, and D.5. As Table A6 shows, the proposed GraphFM model does not achieve state-of-the-art (SOTA) results on most datasets. Furthermore, GraphFM's results fall short compared to more recent methods, as seen in Tables A7 and A8.\n\nHowever, the claimed improvements in sampling and GPU utilization are supported by the experimental results.\nBecause of this, it remains unclear whether the proposed improvements are supported by accurate evidence. Overall, the paper seems to be a limited contribution with unclear evidence for the claims made.", "audience_yn": "Yes", "audience_explanation": "As pretraining on graph transformers has only been explored in few works before and few large scale pretraining objectives have been used in other architectures for graph learning, this work introduces a novel approach to pretraining which could be of interest. Additionally, the introduced sampling method could spark the audience's interest, as the experimental results indicate improved memory usage and utilization.\n\nFurthermore, as results from other domains indicate, the importance of pretraining for large scale models is significant and is required for modern architectures.\nTherefore, I believe that there could be suitable interest of TMLR's audience, especially in the graph learning domain.", "requested_changes": "To strengthen the paper I would like the authors to include the following changes:\n\n1. Provide clear intuition and explanation for the advantage on downstream tasks as claimed by the authors. In addition a clear motivation for the selected architecture should be given.\n\n2. Fix unclear notation (eg. $\\tilde{u}_i = MLP(u_i)$) and description of the concatenation of node features as described in section 2\n\n3. Fix spelling mistakes such as “tthan” on the second to last line on page 3\n\n4. Provide more experimental results to differentiate GraphFM from already existing models which do not require extensive pretraining and other graph transformer based methods.", "broader_impact_concerns": "There are no broader impact concerns from this work.", "additional_comments": "Strengths and Weaknesses:\n\nStrengths:\n\nS1 The paper is mostly clearly written and explained, the supporting figures and plots highlight the most important points.\n\nS2 A good combination of sampling, architecture and pretraining into a single framework enabling concise pretraining across many datasets.\n\nS3 Interesting pretraining mixture of datasets based on graph properties and synthetic graphs, one question regarding this: Did you consider other graph properties besides homophily while designing the pretraining precedure?\n\nWeaknesses:\n\nW1 Somewhat unclear reasoning for selection of PE (SignNet) and Encoder (Perceiver), with theoretical observations missing from the paper entirely.\n\nW2 Limited gains from experiments, showcasing that the method is not stronger in empirical evaluation, despite substantial pretraining efforts made.\n\nW3 Only limited selection for large-scale pretraining datasets (eg. no OGB, LRGB or other larger scale datasets) which limits the pretraining to smaller scale datasets and datasets of low quality [1].\n\nW4 Very limited focus only on node classification and lacking experimental results for the claims of a generalist model, limited applicability because of missing tasks from edge- and graph-level tasks.\n\nW5 Finetuning is necessary for each dataset, limiting the usefulness of the pretrained model compared to existing approaches as pretraining takes a significant amount of time and although fine tuning may be faster than training another model. \n\nW6 Since the focus does not lie on novel methods here, the hyperparameter tuning and overall discussion of architecture choices remains very limited, thereby not really supporting the claims made by the authors.\n\n[1] No Metric to Rule Them All: Toward Principled Evaluations of Graph-Learning Datasets, Coupette et al., ICML 2025", "extra_scores": {}, "extra_text": {}}, {"review_id": "Z9tDkadssi", "reviewer": "Reviewer_gPND", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper contributes a scalable graph pre-training approach based on a perceiver-based transformer encoder. The paper demonstrates its benefits on vast pretraining on over 150 diverse datasets including homophilous and heterophilous graphs. While previous works only aspire to create graph foundation models on specific sub domains (e.g. MoleBert), there are no large scale generalist pretrained graph foundation models to date.\n\n## Strengths\n\n1. I appreciate the discussion of heterophilous and homophilous graph types and its usage as a guide for the construction of the dataset, the dataset level statistics, and the thoroughness detailed in appendix B\n2. Figure 1 is well made and served as a helpful aid during my comprehension of this paper\n3. Authors pay delicate attention to training optimizations, innovating using a novel snake allocation procedure and using FlashAttention to speed up their training\n4. The experimental section has a clean recursive structure alternative between well defined questions and experiments that attempt to answer them. The results from Figure 5 provide compelling evidence that their methodology in this paper has great promise\n\n## Weakness\n\n1. There are places where the exposition can be improved, which I have highlighted throughout my requested changes. In particular, this paper would benefit from a background section.\n2. Some figures and tables can be changed to better visualize the experimental results\n3. The categories \"Soc\", \"Bio\", and \"All\" seem a bit arbitrary to study generalization across domains", "claims_and_evidence_yn": "Yes", "claims_explanation": "The core contributions of this paper, in the authors words, are \n1. Scalable Pre-training\n2. Demonstration of Benefits from Across-Graph Pre-training \n3. Scaling Analysis and Impact of Multi-Graph Pre-training \n\nThe authors also pose and answer various subquestions in the experimental section that aid in addressing these contributions. My main concern would be with the claim of the second contribution. I think the categories of \"Soc\", \"Bio\", and \"All\" are slightly arbitrary. In the last paragraph before Q2, I would add this caveat to hedge the result more. Beyond that, the ablations on GPU utilization and training time as well as fine-tuning strategy, the accuracy of their graphs for classification, and thoroughness of dataset curation all support claims made in the paper", "audience_yn": "Yes", "audience_explanation": "This paper is interesting to the graph learning community both for the general pre-training methods and the favorable results on down stream tasks. The comparison of different methods of both heterophilous and homophilous graphs is of further independent interest and elucidates how different inductive bias in different architectures are more favorable on one class of problems than another.", "requested_changes": "## Major / Intermediate\n\n1. This paper would benefit from a short background section that can help the reader understand the design choices and methodology throughout section 2. Please provide a background section on the necessary topics necessary to understand the paper. This would also help clarify what is being contributed in the current paper\n2. The violin plots in figure 3 are hard to compare and can be replaced with overlapping histograms with some opacity threshold and color labeling\n3. In datasets, the authors should compute the straightforward alternative homophily metric in eqns 2 and 3 in [1], which essentially captures the excess homophily above a randomly wired graph\n4. \"the normalized l2-distance of their hyperparameter vectors to the optimal hyperparameter vector.\" Is this just lr and weight decay? The full set of items considered hyperparameters in the vector should be spelled out. It is not immediately clear to me. This may not make sense if the hyperparams tend to be of different order of magnitude as well.\n5. Some more justification for the \"Soc\", \"Bio\", and \"All\" categories is in order, and potentially the authors should qualify their results in this context\n\n[1] https://arxiv.org/pdf/2110.14446\n\n## Minor\n\n1. In section 2.1.1, the authors refer to the positional embedding of the node. On first read, it wasn't clear to me that the authors meant a positional encoding that correspondeded to the order in which the node feature was fed into the model and not cartesian coordinates in 3D space, which might be considered for many atomic datasets. Please make this distinction clearer, or begin the paper with a short background section detailing various aspects of the tokenization process in the literature. \n2. The authors use SignNet [2] to help construct the positional encodings. This is well motivated by Proposition 4 and Figure 2 in [2]. However, the authors do not comment on why they use SignNet here. The authors should add a few words justifying their design choice here, since it may not be readily known to the audience\n3. ***Extremely nitpicky***, but I don't like the use of the word \"Remark\" for the paragraph under Figure 2. That kind of organization helps in the mathematics literature for well defined mathematical statements that don't constitute full theorems. Please start this paragraph differently. The \"remark\" block appears later as well\n4. Table A6 should include also the Avg Rank for Homophilic and Heterophilic graphs specifically, I would value this as a summary statistic\n5. Tables A7 and A8 would benefit from a citation column allowing the interested reader to learn more about the purported baseline methods more easily. Please add this\n\n[2] https://arxiv.org/pdf/2202.13013", "broader_impact_concerns": "Following section 5.7 in [3], it would be great if the authors could estimate the Carbon footprint of large scale graph pretraining and discuss in a broader impacts statement\n\n[3] https://openreview.net/pdf?id=KoFOg41haE", "additional_comments": "1. Would this approach work for graphs that contain both node features and positions? Do any of the datasets include molecule graphs that include 3D coordinates? Is this an avenue for future work?\n2. While not strictly necessary, the authors may consider adoptng the datasets proposed in [1], Lim et al. above\n3. Similarly, while not strictly necessary, I think the authors should include LINKX from [1] as a baseline and potentially also Unitary Convolutions [4], which is also suited for heterophilous graphs. The latter may be expensive to compute all of the exponentiated adjacency matrices, which probably isn't worth the effort considering there are similar baselines already\n4. Why do the authors not consider regression? Is it just to keep the study well scoped and the ability to compute the homophily ratio? It would be nice if a couple of regression tasks were included, even if to much a less extent.\n\n[4] https://arxiv.org/abs/2410.05499", "extra_scores": {}, "extra_text": {}}, {"review_id": "56fC1BnZwV", "reviewer": "Reviewer_SgFC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces GraphFM, a graph-based transformer dedicated to learn from various graph domains (e.g., citation graph, recommendations, molecules, etc.) and to use the resulting model as a pretrained foundation model to achieve node-prediction tasks on graphs from unseen domains. Three contributions are claimed in the paper:\n\n1. A dedicated algorithm to efficiently dispatch graphs on GPUs during the pre-training phase.\n2. The per-se generalist foundation model able to transfer knowledge to unseen graph. Briefly, the main idea is to leverage latent query tokens allowing a communication between distant nodes, which is challenging in standards graph neural network architecture. The remaining of the architecture (transformer with attention layers, encoding and decoding blocks) is relatively standard.\n3. An analysis of the performance of the models on different aspects (comparison with dedicated approaches, stability regarding the hyper-parameters, efficiency of the GPU-dispatch, etc.)\n\n---\n**List of strengths:**\n\n1. The paper addresses a challenging problem to export the idea of foundation models to graph-related tasks. The topic is definitively appropriate for TMLR.\n2. To my knowledge, the methodology is sound and the contribution is well grounded regarding the related work.\n3. The experimental protocol is mostly sound and promising results are reported. I only have a few weaknesses to report concerning the experiments (detailed below).\n4. The paper is well-written and very pleasant to read.\n\n---\n**List of weaknesses:**\n\n1. The paper focuses on node-level prediction and only features node features. As genericity among graph-related tasks is the main goal of the contribution, it is a missed opportunity to do not consider features on the edges, or graph-level/edge-level predictions. On this aspect, future work is still required to reach this full genericity. \n2. I am surprized that there is no source code provided, nor explanation that it is planned to do. Without the implementation available,  the impact of this paper is severely decreased.\n3. It is not clear why other graph foundation modeling models (e.g., Mao et al., 2024, properly referenced in the paper) could not be used for the tasks proposed. It would have been interesting to have a comparison with this approach, or to explain why this comparison is not relevant.\n4. The bibliography needs to be significantly refactored. There are duplicated entries (Fey and Lenssen, 2019), missing information (e.g., Chen et al.), ArXiv references instead of the published one (Dao, 2023), and incorrect capitalization (Gfse).", "claims_and_evidence_yn": "Yes", "claims_explanation": "**The claims are mostly supported.** The methodology, although incremental, is sound. Experiments proposed confirms the hypothesis that we can train a foundation model on graphs. The main weakness is that the architecture only consider node-related features and node-level predictions decreases the genericity of the model, and consequently the main claim. There are also some (minor) missing experiments/information to be fully convinced about the benefit of this architecture.\n\n1. Figure 3: Why not comparing SSSampler with 8 GPUs as well (as for the baseline)? \n2. A comparison with Mao et al. 2024 is missing. Is this comparison is not relevant, please explain why in the paper.\n3. Experiments Q2: It will be interesting to discuss the number of parameters of the specialized approaches. I guess that there are much lower than the ones of the foundation model.", "audience_yn": "Yes", "audience_explanation": "**This needs further clarification.**\n\nIn terms of findings, the main conclusion that it is possible to train foundation models on graph was already known (properly discussed in the paper) – so the main interest of this paper is on the realization/implementation side. \nFor that, this paper will be of interest for the research community related on learning on/from graphs. However, not having the source code available severely decreases the interest/impact that we can have on this paper. This needs to be clarified.", "requested_changes": "**Major changes:**\n1. Refactoring the bibliography (see my previous comments).\n2. Providing the source code or explaining why it is not possible.\n3. Adding a comparison with 8 GPUs on Figure 3 for DistributedSSSampler.\n4. Adding the number of parameters of specialized approaches in experiment Q2.\n5. Adding a comparison with Mao et al. 2024 – or explaining why it is not relevant.\n\n**Typos:**\n1. Page 2: “GraphFM ,” -> “GraphFM,”\n2. Page 4: it’s -> its", "broader_impact_concerns": "I have nothing specific to report here", "additional_comments": "I am not asking to go beyond the node level as it will severely change the paper and as the current paper can still be of interest of the audience. However, I still think that not considering edge features will decrease the global impact of the paper.\n\nIt will be very interesting to see how the model perform on combinatorial problems over graphs (e.g., the Travelling salesman problem). Benchmarks are provided here (https://github.com/graphdeeplearning/benchmarking-gnns)", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)