Bridging Input Feature Spaces Towards Graph Foundation Models
Authors: Moshe Eliasof
Krishna Sri Ipsit Mantri
Beatrice Bevilacqua
Bruno Ribeiro
Carola-Bibiane Schönlieb
Venue: ICLR 2026 Poster (2026)
Keywords: Graph Neural Networks
Graph Foundation Models
Primary area: learning on graphs and other geometries & topologies
TL;DR: We address the lack of a shared input space in graphs. We propose ALL-IN: map node features to a shared random space and build covariance-based representations invariant to feature permutations and orthogonal transforms, enabling zero-shot transfer.

Abstract: Unlike vision and language domains, graph learning lacks a shared input space, as input features differ across graph datasets not only in semantics, but also in value ranges and dimensionality. This misalignment prevents graph models from generalizing across datasets, limiting their use as foundation models.
In this work, we propose ALL-IN, a simple and theoretically grounded method that enables transferability across datasets with different input features. Our approach projects node features into a shared random space and constructs representations via covariance-based statistics, thus eliminating dependence on the original feature space. 
We show that the computed node-covariance operators and the resulting node representations are invariant in distribution to permutations of the input features. We further demonstrate that the expected operator exhibits invariance to general orthogonal transformations of the input features.
Empirically, ALL-IN achieves strong performance across diverse node- and graph-level tasks on unseen datasets with new input features, without requiring architecture changes or retraining. These results point to a promising direction for input-agnostic, transferable graph models.

Published as a conference paper at ICLR 2026
BRIDGING INPUT FEATURE SPACES TOWARDS GRAPH FOUNDATION MODELS
Moshe Eliasof
Krishna Sri Ipsit Mantri
University of Cambridge
Purdue University
Ben-Gurion University of the Negev
mantrik@purrae edu:
me5320 cam www.ac........a....mmmm.. ac.uk uk
Beatrice Bevilacqua
Bruno Ribeiro Purdue University
Carola-Bibiane Schönlieh University of Cambridge cbs 31 cam. ac uk
Purdue University bbevilacEpurdue. edu
#ibeirob@purdue. edu
ABSTRACT
Unlike vision and language domains, graph learning lacks shared input space, input features differ across graph datasets not only in semantics, but also in value ranges and dimensionality. This misalignment prevents graph models from generalizing across datasets, limiting their use foundation models. In this work, we propose ALL-IN, a simple and theoretically grounded method that enables transferability across datasets with different input features. Our approach projects node features into a shared random space and constructs representations via oovariane--based statistics, thus eliminating dependence the original feature space. We show that the computed node-covariance operators and the resulting node representations are invariant in distribution to permutations of the input features. We further demonstrate that the expected operator exhibits invariance to general orthogonal transformations of the input features. Empirically, ALL-IN achieves strong performance across diverse node- and graph-level tasks unseen datasets with new input features, without requiring architecture changes or retraining. These results point to promising direction for input-agnostic, transferable graph models.
INTRODUCTION
Foundation models have shown remarkable success in domains such as language and vision, where large-scale pretraining enables strong performance across wide range of downstream tasks. A similar goal has emerged for graph learning: to develop graph foundation models that generalize across tasks, domains, and datasets (Mao et al., 2024). However, key obstacle in this direction is the lack of transferability across graphs, as knowledge learned from one graph is often difficult to apply another due to fundamental differences in their structure and, critically, their input features.
Unlike vision or language data, graph datasets typically do not share a common input space. Node features often differ significantly not only in distribution and semantics but also in dimensionality from one graph to another. Furthermore, graphs themselves may vary in size, sparsity, and topological patterns. These mismatches break many of the assumptions that underlie successful generalization in other domains, making it difficult to define a common representation space or pretraining strategy.
Existing approaches to graph foundation models fall into broad categories. The first integrates LLMs by serializing graph data into text or designing prompt- based mechanisms (Liu et al., 2024; Zhao et al., 2023; Chen al., 2024b; Fatemi al., 2024; Perozzi al., 2024; Chen et al., 2024a; Zhao et al., 2023; He & Hooi, 2024: Huang et al., 2023; Tang et al., 2024; Kim et al., 2024; Zhao et al., 2024a; Gong et al., 2024; Sun et al., 2022; 2023), leveraging LLM capabilities but often discarding fine-grained graph properties. The second direction aims to explicitly align or adapt feature spaces across datasets using techniques like input projections (Xia & Huang. 2024: Yu et al., 2024; Zhao al., 2024a), specialized encoders (Lachi et al., 2024), structuralization (Frasca et al., 2024), order statistics (Shen et al., 2025). However, these methods often remain specialized to particular settings tasks, may require careful adaptation to scenarios.

---

Published as a conference paper at ICLR 2026
Training
Xada (R1)
NedeCos A)C1)
NodkCros NodeCCol & 101
x
X R
GNNLayer
H(L)
Noleor (te
NodCCost ARR ") SodeCort A'R R Soda A'R
Inference
X1 1gnaa
A1 E Ru
GNNeayer
H
Nodat (R') NakColAR NokCoat (UR)) Nodeliv R
x
X2 R
A B
GNNLayer
(a) Motivation: The challenge varying input features
(b) AL.I-IN consistent node-covariance operators
Figure 1: Addressing feature heterogeneity with ALL-IN's node-covariance operators. (a) When a GNN is trained on graph data with node features X of dimension d, it cannot be directly applied on graphs with features of different dimensionality d'. (b) ALL-IN computes operators, capturing node similarities, providing common space that is independent of the original, heterogeneous, feature spaces. Different node colors indicate distinct node features.
node-covariance
In this work, we propose novel approach, grounded in statistical principles, to overcome input feature heterogeneity (Figure 1). Our method first projects potentially disparate node features into a common, high-dimensional space using a stochastic projection matrix. We then leverage second-order statistics within this space using covariance operators. Specifically, we model feature dimensions as independent and identically distributed samples from an unknown distribution over the nodes, and compute the empirical node-covariance matrix based on these projected representations. This matrix captures pairwise node similarities based on how their projected features co-vary providing representation inherently robust to changes in feature semantics, value, and dimensionality.
We introduce ALL-IN (All Input spaces), a graph learning framework built upon this principle. Instead of directly processing raw node features in downstream layers, ALL-IN utilizes the computed stochastic node-covariance matrix (and its higher-order variants), shown in Figure 1, graph operator within a graph neural network (GNN). This node-covariance matrix captures interactions between nodes, specifically, how similar two nodes are in terms of their feature activations across the feature dimensions. Our theoretical analysis reveals significant robustness properties: (a) The computed operators and, critically, the resulting node representations throughout the GNN are invariant in distribution to arbitrary permutations of the original input features; (b) The expected operator invariant to general orthogonal transformations (basis changes) of the input features; (c) The overall method is inherently insensitive to dimensional mismatches across datasets. We further identify qualitative conditions under which covariance-basee representations retain tesk-relevant information and enable transfer across datasets with different input features.
Our empirical results confirm the efficacy of this approach: ALL-IN achieves strong transfer perfor- mance to new datasets with new input features across diverse node- and graph-level tasks. As result, ALL-IN offers a promising approach toward the development graph foundation models.
2 RELATED WORK
Graph Foundation Models (GFMs). GFMs aim to learn representations that generalize across datasets and tasks, but achieving robust generalization remains challenging, especially when node features change. Some approaches integrate LLMs by converting graphs to text or embedding features through prompt- based designs (Liu et al., 2024; Zhao et al., 2023; Chen et al.. 2024b; Fatemi et al., 2024; Perozzi et al., 2024; Chen et al., 2024a: Zhao et al., 2023; He & Hooi, 2024; Huang et al., 2023; Tang et al., 2024; Kim et al., 2024; Zhao et al., 2024a; Gong et al., 2024; Sun et al., 2022; 2023), or by generating or augmenting graphs with LLM guidance before training a graph encoder (Xia et al., 2024), but this can lead to loss of structural details. Text-altributed GFMs further learn transferable vocabularies or automatically search architectures on such LLr--deivved features (Wang et al., 2024; Chen et al., 2025a), which improves transfer within TAGs but does not directly handle non-textun node attributes. Other works align feature spaces through projections (Xia & Huang, 2024; Yu et al., 2024; Zhao et al., 2024a; Fang et al., 2023) and multi-domain feature structure aligners with
2

---

Published as a conference paper at ICLR 2026
ALI-IN
XL
X
Project (0)
H (0)
GNNLayer
GNNLayer (.,A
NodeCov
K((
GNNLayer (4.0)))) (, :x(())
H
NodeCov
((1))
GNNLayer (c. k(())
A
NodeCov
K((
GNNLayer/ (/. (.KAA)))) (AA)
Figure 2: The ALL-IN Architecture. Input node features X are first randomly projected into R!(") This (0)) serves as initial node representations (0), Concurrently, R()) and its propagated versions (e.g., R(P) = A R( are used to compute a set of node-covariance matrices {K (K(P) Jg=D capturing diverse orders of feature-based node similarities. These matrices are used operators within different GNN (sub-)lagy layers, whose outputs are concatenated to form the updated node representation.
prompts or miitures-offexpeee (Yu et al., 2025; Yuan et al., 2025), perceiver-based encoders (Lachi al., 2024), computing analytical solutions (in the case node classification) (Zhao et al., 2024b), encoding features into the graph structure (Frasca al., 2024; Galkin et al., 2024; Wang & Luo, 2024; Franks et al., 2025) or learning shared structural vocabularies in Riemannian spaces (Sun et al., 2025), encoding feature relationships (Shen et al., 2025). While these methods advance GFM capabilities, they often require task-specific adaptations, leaving gap for truly input-space-agnostie solutions. ALI.-IN offers distinct path: it creates transferable representations by processing arbitrary input features through stochastic projections and mode-covariance operators, enabling frozen-encoder transfer without task- or domain-specific prompts or architectural changes.
Structural and Positional Encodings. Efforts to create universal graph representations include transferable structural and positional encodings (SPEs) (Rampášek al., 2022: Cantürk al., 2024; Chen et al., 2025b; Kim et al., 2024). SPEs aim to capture graph topology in feature-agnosee manner, often within Graph Transformers or GNNs. While such SPEs can complement node features, AL..--IN directly addresses the challenge of heterogeneous node features themselves, transforming them into a robust, transferable format using their covariance structure, irrespective of any additional SPEs.
Covariance networks. Covariance matrices have also informed the design neural networks. For instance, co Variance Neural Networks (VNNs) (Sihag al., 2022) process d d sample covariance matrices, with the input feature dimension, which describe feature inter-correlations. offering benefits like stability to varying sample sizes and inspiring extensions for fairness (Cavallo et al., 2025) and sparsity (Cavallo et al., 2024). Other related efforts focus on transferring principal components derived from data covariance (Hendy & Dar, 2024). While these methods analyze relationships between features using sample covariance matrices, ALL-IN constructs an n x n node- covariance matrix, with n number of nodes. This operator quantifies similarities between pairs of nodes based on how their (randomly projected) features co-vary across dimensions. This distinct formulation is tailored to building transferable representations from graphs with heterogeneous node features, addressing challenge different from that targeted by the aforementioned approaches.
3 METHOD
Our method, ALL-IN, replaces dataset-specific raw node features with covariance-based operators that are better suited for generalization across input feature spaces. The approach comprises three main stages: (1) Random Feature Projection to map input features shared space, (2) Operator computation to capture robust node similarities, and (3) Operator- based Propagation to learn transferable node representations. An overview of AL..--I be found in Figure 2.
3

---

Published as a conference paper at ICLR 2026
Random Feature Projections. Given graph with nodes and node feature matrix X R where the input dimension d may vary across graph datasets, we first apply a random linear trans- formation to project the features into unified ixeeddimensional space h that is shared across datasets:
R 00) =XX,
with vec(C) ~ NO Tax) sampled at each forward pass,
that is, C E gdxh an isotropic Gaussian random weight matrix sampled independently at each forward pass. This step is key to ensuring that our approach is invariant (in distribution) to feature permutations, as we discuss in Section 4.
Operators. We treat each column of R an i.i.d. signal over the nodes and compute the node-covariance matrix to capture second-order relationships (node similarities) based on feature co-variation across the latent dimensions:
K NodeCov(R 1 R
R R!O
E ER
(2)
where R E Ruxh h is the centered projected feature matrix defined by R R 1, with 1 Σ.
th R E gixh the empirical mean of the projected node features, and 1, E _x the all-ones vector. This centering operation is equivalent to pre-multiplynng by the geometric centering matrix II. In 1,, i..., R. I()) The resulting K is n X matrix reflecting node similarities in the projected feature space. An interesting property is that if we consider two nodes u and v with feature vectors X, and X, coincide, but their rows K and K(!) differ in the cross-cos cross-covariance entries with other nodes because their signs flip, so message passing based on ((() still distinguish them.
X.,, then their auto-covariance terms in K())
To integrate structural information with feature similarities, we compute higher-order covariance matrices based on propagated features. Specifically, for each p ........k k. first perform message passing on the initial projected features (0( using the graph's 's adjacency matrix A:
R(a) = A
Then, we compute the covariance matrix on these propagated, centered features R.P) R(P)
K( NodeCov(RRPP) = R! R(m) E _
1
The operator (()) captures node similarities based on features aggregated from neighborhoods up to p hops away, thus encoding increasingly global structural context in the graph.
Node Representations.
We collect a set of graph operators, which includes the identity matrix I, the adjacency matrix A, and the computed node-covariance matrices K. {K ===
O = (I, A, (1)) K')....,K (*))
(4)
Instead of using the original node features, we rely the random projections R(O) potentially augmented with structural encodings, such as random-wall encodings (Dwivedi et al., 2022a). That is, we let (0) be
H (0) R + S
(5)
where ① indicates concatenation and S Ruxh, is structural encoding matrix. We note that, although the node-covariance operators K()) capture second-order statistics, ALL-IN maintains first-order information: the projected features R()) are used directly as part of the initial node representations H in Equation (5). For example, if two nodes u and v have feature vectors X,, and X, = =x,,, then their projected features satisfy R!()) X,C and R! =, they are distinguishable in (0),
At each layer é 1,..., L, we propagate the current node representations using every operator O E O, and concatenate the outputs to obtain the updated representations:
H ( θ GNNLayer((.0)
(H (e-1) O))
(6)
OEO
4

---

Published as a conference paper at ICLR 2026
where GNNLayer!&.) is the GNN layer associated with operator O E O in layer L, taking as input (t-11) and performing message passing using O as the operator, using learnable weights ((.O) ph'l-)xx* and (0)) h + hg-
In the presence of edge features, which, similarly to node features, may vary across datasets, we employ an analogous strategy. Specifically, we first project the edge features into a dimenssiaa space using an isotropic Gaussian random weight matrix, yielding edge representations that are independent of feature dimensionality. Then, we aggregate these projected edge features at the node level (e.g., by averaging features of incoming edges for each node) to obtain node-level representations Rig derived from edges. We then compute node-covariane node-covariance matrices K( based on these aggregated (and potentially propagated) features, similar Equation (3). Finally, we add these edge-derived covariance operators Kedge = do.............................. odge } to the operator set O (Equation (4)). This allows the model (Equation (6)) to incorporate edge information while remaining compatible across datasets with differing edge feature spaces.
edge
4 THEORETICAL INSIGHTS
This section establishes the theoretical foundations underpinning the ability of ALL-IN handle heterogeneous input features and enable generalization across datasets, A core contribution is proving the method's robustness to variations in feature representation. We first demonstrate that the node- covariance operators and the resulting node representations are invariant in distribution to arbitrary permutations of the input features, providing robustness to feature re-ordering. We then show that the expected node-covariance operator is invariant to general orthogonal transformations, ensuring robustness to the choice of orthonormal basis (Section 4. Building on these properties, we validate the stochastic training procedure using Jensen's 's inequality under standard convexity assumptions (Section 4.2). Finally, we discuss conditions supporting transferability, analyzing scenarios where the operator remains stable across graphs with differing feature distributions and proving its consistency for large projection dimensions (Section 4.3). All proofs are provided in Appendix B.
4.1 INVARIANCE TO FEATURE SPACE TRANSFORMATIONS
A primary obstacle to cross-dataset transfer is the lack of feature standardization, leading arbitrary differences in feature ordering and basis choice across datasets. Our approach, centered on node- covariance after random projection, inherently addresses these issues through invariance properties. First, the use of random isotropic Gaussian projections renders the process statistically insensitive to the order of input features. We formalize this by showing that the distribution of the projected feature matrix remains unchanged when the original features are permuted.
Proposition 4.1 (Distributional Invariance of Projected Features to Feature Permutation). Let X E Rgndd be node features, E Rdxd be any permutation matrix, and h be the projection dimension. Let C E ddxh be an isotropic Gaussian random matrix (i.e., vec(C) ~ NO, Ian). Define the projected features as R = XC and the features projected after permutation as R (XP)C.
Then O and R() are equal in distribution:
O)) R(O)
In essence, Proposition 4.1 establishes that random projections effectively "mix" features, rendering their original ordering statistically irrelevant after projection. More importantly, the permutation invariance is characterized in distribution, rather than pointwise: for fixed random projection C, the features in R()) retain sensitivity to input permutations, thereby enabling a neural network to better capture the relationships between node features and topology.
To illustrate this concept, consider three nodes u, v. w E V with features X, = (0. 1). X, = (0. 1), and X O). Under strict (pointwise) permutation invariance, the embeddings of all nodes would be equivalent, obscuring the key distinction that u and v share identical features, whereas w has different feature. In contrast, distributional invariance ensures that the distributions of R(() R and R!!!) are identical, yet individual forward passes yield different outcomes; given C, have R(() = ( + R(O) This property preserves the model's 's ability to distinguish between nodes and (which share the same features) and node (which has different feature), while maintaining
5

---

Published as a conference paper at ICLR 2026
symmetry in the model's statistical behavior, thus striking balance between permutation invariance and expressive power.
Next, we show that the NodeCov operators applied to the sequence {R(P)) Equation (3)) yield features that are also distributionally invariant.
pp=0 (as defined in p==0
Corollary 4.2 (Distributional Invariance Node Covariance Operators to Feature Permutation). Let X E Rnxd be node features, and P RRaxd be any permutation matrix. Let R = XC be the initial projected features. Let K = [K ( yp be the set of node-covariance operators, where K(P) NodeCoví A" AR))) R()) computed using the deterministic function NodeCov (Equation (3)), and is the adjacency matrix. It follows directly from the distributional invariance of R that the entire set of operators K. is also invariant in distribution to permutations of the input features X.
That is, if is the set of operators computed using XP instead of X, then K. R.
The significance of Proposition 4.1 and Corollary 4.2 is substantial: it guarantees that the complete statistical behavior of (2(0) and the operators K central to ALL-IN is fundamentally robust to arbitrary input feature ordering, directly addressing key source of heterogeneity across graph datasets. This distributional invariance also extends to the hidden representations H for all l 1 L. derived from these operators, as shown in Theorem B.1 in Appendix B.
The stochastic projection matrix C' plays a critical role beyond enabling the distributionally invariance properties discussed earlier; its use is intrinsically linked to the expressive capability of the learning framework. Training with node-covariance operators NodeCov (R (R) derived from these stochastic projections offers advantages over relying on single, deterministically computed covariance oper- ator, such as NodeCov( X). While NodeCov( X) provides a stable, pointwise feature-pent permutation invariant view of node similarities, it can obscure subtle but important distinctions between nodes. In contrast, individual stochastic realizations NodeCov (R))) = XodeCov(XC) (XC) (for specific C) can preserve these finer-grinne distinctions, providing richer and more varied signals to the GNN. Theorem 4.3 formalizes this concept by demonstrating that there exist instances where the stochastic operator NodeCov X XC) can distinguish nodes that the deterministic operator NodeCov( (X) cannot.
Theorem 4.3 (Distinguishability through C). There exist node features X E Raxd nodes u E V with X. * X. such that NodeCov(X) makes u, indistinguishable (automorphic), but VodeCov(XXC) (for a.s. all C) makes u. v distinguishable (not automorphic).
Finally, while distributional invariance covers permutations, analyzing the expected operator reveals broader robustness to basis changes and identifies the structure captured on average, as we show next.
Theorem 4.4 (Expected Invariance Orthogonal Transformations). Let X Baudd be node features, Q E Rddd be an orthogonal matrix, and hs be the projection dimension. Consider a random projection matrix Ce Rdxh with vec(C) ~ (O, Iax.. Let (odeCov(R!o)
be the Node Covariance operator Equation (2)), where II, - In 1 is the centering matrix. Then, the expected Node Covariance computed from the stochastically projected features is invariant to the orthogonal transformation Q:
((I,R
featurss
EC CoadeCov(XXQCO = (Noeeoo X = I1,XX
where the expectation Ec[:] is over the random sampling C,, and II, I L of the centered original features.
the Gram matrix
Theorem 4.4 demonstrates that the expected operator agnostic to any choice of orthonormal basis (rotations, reflections, permutations) for the input features. Furthermore, identifying this stable expectation as the Gram matrix of centered original features (II, X III) reveals that ALL-IN, on average, recovers intrinsic, invir-invariant pairwise node similarities directly reflecting the original data structure, irrespective the specific random projection used.
4.2 TRAINING OBJECTIVE UPPER BOUND
ALL-IN computes the feature projection R and node-covariance operator (0) NodeCov(XL using a stochastic projection matrix C' sampled in each forward pass. We now validate this prac- tical training approach by showing its connection to performance on the stable, expected final representation Ee (2)), assuming common convexity conditions for the final prediction layer.

---

Published as a conference paper at ICLR 2026
Table Performance of ALL-IN on pre-training datasets compared ALL-IN-SPECIALIES which is trained separately on each individual dataset. ALL-IN maintains highly competitive performance.
Method
ZINC (MAB.
MOLESOL, (RMSE
MOLHIV (ROC-AUC:)
MOLTOX21
MNIST
CIFARIO MODELNET
CUNEIFORM (ACC
MSRC 21 (ACC
(ACC (ACC (ACC
TRAINED PER DATASET
SLL--NN-PECIAALPER (0 props)
0,1480
1.22
72.65
69.37
94.03
39.96
37.24
85,19
91 65
SPPECIIPPE
0.1195
1.19
73.78
70.04
94.77
40.03
39.81
87.20
94.16
TRAINED ON ALL DATASETS
ALL-IN (0 props) ALL-IN
0.1557
1.28
72.74
68.19
94.57
40.11
37.11
89.88
97.5 51
0.1237
1.29
74.49
68.20
95.22
40.08
39.37
91.17
98.08
Theorem 4.5 (Loss Upper Bound), Let H R be the final node representations computed by ALL-IN, dependent on the initial random projection C. Let é _ + j2nxt be the final prediction layer, and let Li,, Y) be the loss function comparing predictions to ground truth labels Y. Assume that the composite function ((H(L)) L( ((((H))) Y) is convex with respect the final node representations HL)) Then, our stochastic optimization objective provides an upper bound for the loss of the expected representation:
L( [H (L 1),Y) S (c( (1)),
(8)
Loss of Expected Representation
Expected Loss (Training Objective)
where the expectation [E]]:] [:] taken over the random projection matrix C.
This holds, for instance, is linear map or linear plus softmax, and cross-entropy or mean squared error. Theorem 4.5 provides theoretical support for training with stochastic projections. Equation (8) establishes that the expected loss minimized during training (RHS) serves as an upper bound for the loss evaluated the stable, expected final representation (LHS). Thus, minimizing the empirical average loss (approximating the RHS) acts as a theoretically sound surrogate objective, implicitly minimizing the loss associated with the expected representation, validating our stochastic approach.
4.3 CONDITIONS FOR TRANSFERABILITY AND OPERATOR CONSISTENCY
Beyond invariance, achieving transfer across graphs with fundamentally different feature distributions (X(1) (1) x (2) for graphs Gj. G2) relies on the stability of the underlying structure captured by the expected operator, (K)] II, XI. We posit that such stability can arise when graphs share intrinsic properties. Plausible scenarios where such stability in the expected operator might arise include graphs exhibiting similar relational structures tied to node features (e.g., comparable label homophily if features reflect labels), originating from a shared underlying generative process (e.g.. common SBM or graphon influencing features), or possessing similar distributions of node roles (c.g.. hubs, bridges) if features are role-informative. In these cases, even if the specific feature realizations differ, the resulting II, (+) (X(')TII, matrices may capture analogous relational structures.
For this potential transfer be practically realized, the stochastic operator k(O) (0) finite projection dimension h must reliably estimate its expectation. This holds for large h.
computed using
Proposition 4.6 (Consistency of Projected Node Covariance). Let X E Raax be node features. For a projection dimension h, let C E dxh be such that vec( (C) N(O,, Ian). Define the stochastic node-covariance operator k() = NodeCov(XC)) = 1/4 XCII
where II, is the centering matrix. Then, K-(0) converges in probability to its expected value as - Do:
K (0) 2 Ec K
0 XIX
This consistency connects theory to practice. It shows that for a sufficiently large h, the operator accurately reflects the stable expected operator II,X XI Therefore, if two graphs have aligned expected operators (due to shared properties). using a large enough h allows ALL-IN to effectively leverage these shared underlying structures, facilitating transfer across disparate feature spaces.
7

---

Published as a conference paper at ICLR 2026
Table 2: Performance on unseen node classification datasets with new input features. ALL-IN effectively transfers to new datasets with new features, often outperforming or matching SOTA.
Method
CITESEER (ACC f)
(ACC 11)
(ACC f)
NON-PARAMETRIC BASELINES
LABEL PROPAGATION (Zha & Ghahramani, 2002)
69.20
0.00
$1.30 di 0.00
71.40 1: 0.00
SUPERVISED BASELINES
48.56
1.53
GCN (Kipf& & Welling, 2017)
78.86
1.48
64.52 + 0.89
74,49 + 0.99
GIN (Xu et al., 2019)
67.10
3.00
58.80 1: 2.20
68.40 1: 2.70
LLM-AUGMENTED GNNS
2024)
73.04
GLEM-LLM (Chen et ol., 2024h)
67.55 3.53
66.00 + 5.66
62.12 + 0.07
LLM-BASED
GRAPHTEXT (Zhan ef al., 2023)
75.41 d: 2.08 72.29
58.24 +: 0.26
63,70 + 0.29
RWNN-LLAMA3-8B (Kim al., 2024)
N/A
N/A
ANYGRAPI (Xia & Huang, 2024)
62.60 0.14
19.32 + 0.37
70,73 + 4.13
GRAPHANY (Zhan et 2024h)
79.36 0.23
68.42 1 0.39
76.30 1: 0.41
MDGPT (Yu
2024)
43.36 8.92
42.50 - 9.78
51.91
9.00
GCOPE (Zhao ot al., 2024a)
35.54 = 2.09
31.18 - 4.35
32.87 = 4.08
GPPT al.,
11..72
ALL- ONE (Sun et al., 2023)
$2.39 1.10.17
40.41 1: 2,80
45.17 1 6.45
GPROMPT (Gong al., 2024)
53.21
GPF (Pang al. 2023)
38.57 5.41
31.16 8.05
49.99 = 8.86
GPF-PIAS (Fang et al., 2023)
55.77 10.30
59.67 11.87
46.64 18.97
ULTRA (3G) (Galkin
2024)
79.40
67 - 0.00
77.90
0.00
SCORE (Wang & Luo, 2024)
81.0 + 1.02
71.33 + 0.27
82.93 - 0.55
OpenGraph
2024)
RiemannGIM (Sun et al., 2025)
N/A
66.18
76,20
AutoGFM (Chen et al., 2025n)
80.32 11.12
N/A
78.28 1.40
ALI-IN (0 props)
79.26 +: 1.08
65.96 + 1.25
77.30 +: 0.47
ALL-IN
82.13 0.97
69.12 + 0.89
78.03 + 0.82
EXPERIMENTS
In this section, empirically evaluate the ability of ALL-IN learn transferable representations from diverse graph datasets, and, critically, its capability to generalize to new datasets presenting entirely new input features. Our experiments are designed answer two primary research questions:
(Q1) How does single ALL-IN model, pre-trained jointly on diverse collection of graph datasets (each with its own input features and task), perform on these individual source datasets compared to training separate model for each dataser?
(Q2) How effectively do the representations learned by a pre-trained ALL-IN model transfer to new, unseen datasets that may have entirely different input features and downstream tasks?
Next, we report our main experiments and refer to Appendix C for additional results (including time complexity). Implementation details, dataset statisties, and hyperparameter configurations in Appendices D and E.
5.1 PERFORMANCE ON PRE-TRAINING SOURCE DATASETS (A1)
In this subsection, we assess the ability of ALL-IN learn from wide array of source datasets simul- taneously, without significant performance degradation on the individual datasets it was pre-trained on. This is needed for establishing its viability to obtain general-purpose pre-trained representations.
To test this, we pre-train a single ALL-IN encoder on diverse corpus of nine graph datasets, encompassing molecular data (ZINC (Dwivedi et al., 2023), OGBG-MOLIIV (Hu et al., 2020a), OGBG-MOLESOL (Hu et al., 2020a), OGBG-MOLTOX21 (Hu et al., 2020a)), computer vision derived graphs (MNIST (Dwivedi et al., 2023). CIFAR 10 (Dwivedi et al., 2023), CUNEIFORM (Morris et al., 2020), MSRC 21 (Morris et al., 2020)), and 3d shape (MODELNET (Wu et al., 2015)) with varying tasks (classification and regression) and heterogeneous input features (differing dimensionalities. types, value ranges and semantics). For each dataset-task pair, dedicated prediction head is attached
8

---

Published as a conference paper at ICLR 2026
to the shared ALL-IN component and trained to predict the corresponding target. We compare this single, jointly-trained model against its specialist counterparts: nine separate instances the ALL-IN architecture, each trained from scratch only one of the source datasets (ALL-IX-SPEECIALIZEE))
Results and Discussion. Table confirms that ALL-IN not only successfully operates across datasets with heterogeneous features but is also highly effective, achieving performance competitive with, and at times superior to, specialized models. While the individually trained ALL-IN-SPECIAALIZED holds a slight edge on ZINC, MOLESOL, MOLTOX21, and MODELNET, the jointly-trained ALL-IN demonstrates superior performance on the remaining datasets. This advantage is particularly notable on CUNEIFORM (91.17% vs. 87. 97.20%) and MSRC 21 (98.08% vs. 94. 16%), while also outperforming ALL-IN-SEE SPECIALIZED on MOLHIV, MNIST, and CIFAR 10. We also observe clear advantage to using propagated operators, as the full model generally outperforms the (0 props) variant (a version computed without propagated covariance operators) across the tasks.
Overall, these results strongly indicate that a joinle pre-trained ALL.-IN encoder can learn general-purpose representations from diverse data that remain highly competitive with, and in several cases surpass, those obtained when learning on single task.
5.2 TRANSFERABILITY TO UNSEEN DATASETS AND INPUT FEATURES (A2)
This subsection assesses the central hypothesis underlying our research: namely, that a single, pre- trained ALL-IN model can effectively generalize to novel datasets characterized by distinct input features. To evaluate this hypothesis, we maintain the pre-trained ALL-IN encoder frozen, thereby ensuring that its learned representations remain unchanged. For each new dataset, which encompasses a range of node and graph-level tasks and introduces previously unseen input features and target label schemas, we instantiate and train new prediction head using the frozen representations extracted by ALL-IN. This approach enables us to isolate the generalizability of ALL-IN's pre-trained representations, providing test of its ability to adapt to unfamiliar data distributions.
We compare ALL-IN against several cate- gories of baselines: (1) the non- -on-paramtteric baseline Label Propagation (Zhu& & Ghahra- mani, 2002), on node classification tasks where it is applicable; (2) standard super- vised GNNs trained from scratch on the tar- get datasets; (3) -Lgm-negeetted GNNs; (4) LLM-based methods; and (5) other GNN-based foundation models or transfer learning approaches. We adhere their prescribed protocols for adaptation on new datasets. We refer the reader to Appendix C for this categorization.
Table 3: Performance on unseen graph classification datasets with new input features. ALI.-IN demonstrates strong transferability to graph-level tasks with new fea- tures, underscoring its versatility across different tasks and its ability to handle different features.
MUTAG (ACC 1>
PROTEINS (ACC f>
SUPERVISED BASELINES MLP
67.20 1. 1.00
59.20 1: 1.00 76.20
GIN (Xu 2019)
89.40 1 5.60
2.80
LLM-AUGMENTED GNNS
OFA (Liu et 202))
61.04 + 4.71
61.40 + 2.99
GNN-BASED
Results and Discussions. ALL-IN demon- strates robust transfcrability across both node-level (Table 2) and graph-level (Ta- ble 3) tasks unseen datasets with new input features. ALL-IN not only signifi- cantly surpasses the performance of stan- dard supervised GNNs trained from scratch on these target datasets, but also outper- forms recent -fate-of- graph founda- tion models. On node classification benchmarks (Table 2), ALL-IN consistently demonstrates strong transfer capabilities. For instance, on CORA (Kipf & Welling, 2017), it obtains an accuracy of 82. which not only surpasses standard supervised GCN (78.86%) but also exceeds leading baselines like SCORE (Wang& & Luo, 2024) (81.80%) and GRAPIIANY (Zhao et al., 2024b) (79.36%). This strong performance extends to graph classification tasks (Table 3). On MUTAG (Morris et al., 2020), ALL-IN IN achieves 92.90% accuracy, exceeding both the supervised GIN baseline (89.40%) and state of the art methods like SCORE (85.33%) and ALL-IN-ONE (Sun et al., 2023) (79.87%). Furthermore, consistent with observations on the source datasets in Section 5.1, the inclusion of
MDGPT (Yu et al.,
$7.36 144.2
54.35 10.26
GPPT (SUN ET AL., 2022)
60.40 1:15.43
-NL-IN-ONE (Sun et al., 2023)
79.87 1 5.34
66.49 1: 6.26
GPROMPT (Gong et 2024)
73.60 + 4.76
59.17 111.26
GPF (Pang et al., 2023)
68.40 + 5.09
63.91 + 3.26
GPF-PLUS (Fang et al., 2023)
65.20 +: 6.94
62.92 2.78
ULTRA(3G) (Galkin al., 2024)
63.33 4 0.00
58.09
0.00
SCORE (Wang & Luo. 2024)
83.33 4: 2.11
68.54 + 1.47
ALL-IN (0 props)
92.50 + 6.60
76.72 + 3.19
ALL-IN
92.90 + 6.34
78.20 + 3.81

---

Published as a conference paper at ICLR 2026
propagated covariance operators in ALL-IN enhances transfer performance compared ALL-IN (0 props).
These results provide evidence that single pre-trained ALL-IN encoder produces effeettive, general- purpose representations. These representations readily adapt to both node and graph-level tasks on new datasets with new features, maintaining versatility that provides strong advantage over specialized models (GRAPHANY (Zhao et al., 2024b), GRAPHTEXT (Zhao et al., 2023), GCOPE (Zhao et al., 2024a), ANYGRAPII (Xia & Huang, 2024)), only supporting node classification.
CONCLUSION
Input feature heterogeneity critically limits the development of Graph Foundation Models (GFMs). Our ALL-IN offers a theoretically-grounde solution, processing arbitrary node features through stochastic projections and node-covariance operators build robust representations independent of the original feature space. We prove that these representations achieve distributional invariance to input feature permutations, and their underlying expected operator invariant to orthogonal basis changes, thereby helping capture robust intrinsic structures of the data. The empirical transfer performance of ALL-IN across new datasets with disparate features demonstrates its potential to mitigate the challenges posed by feature heterogeneity, contributing to the development of GFMs.
10

---

Published as a conference paper at ICLR 2026
Limitations and Future Work. The scalability of ALL-IN extremely large graphs may be constrained by its dense covariance operators, case direct access to the covariance operators are required, similarly graph transformers; developing sparse approximations presents key avenue for future research. Another promising direction involves exploring structured or learnable input feature projections alternatives the random Gaussian projections. Notably, discussed in Appendix C.17, in common GNNs, we can avoid the storage of dense covariance operators, thereby achieving improved scalability.
Reproducibility Statement. Our code is available at https://gites thu.b.ttthuuu...o com/ MosheE1 /ALLI. We carefully document dataset details in Appendix D and im- plementation details in Appendix E.
Ethics Statement. Our work is primarily methodological and presents minimal direct ethical concerns. All experiments conducted publicly available benchmark datasets widely used in the graph machine learning community, and we have used these datasets in accordance with their established licensing and terms of use. While our contribution foundational, we advocate for the responsible application of transferable graph models. We caution against their use in analyzing sensitive social or personal data without appropriate safeguards and ethical oversight.
Usage of Large Language Models in This Work. LLMs were used in this work for text editing suggestions. All concepts, thcoretical analysis, code development, and original writing were carried out by the authors.
REFERENCES
Beatrice Bevilacqua, Joshua Robinson, Jure Leskovec, and Bruno Ribeiro. Holographic node representations: Pre-training task-agnostic node embeddings. In The Thirteenth International Conference Learning Representations, 2025.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https: wandb. com/. Software Coftwareevaate available from wandb.com.
Semih Cantürk, Renming Liu, Olivier Eepointe-Gagné, Vincent Léuurneau, Guy Wolf, Dominique Beaini, and Ladislav Rampášek. Graph positional and structural encoder. Forty-first International Conference Machine Learning, 2024.
Andrea Cavallo, Zhan Gao, and Elvin Isufi. Sparse covariance neural networks. arXiv preprint arXiv:2410.01669, 2024.
Andrea Cavallo, Madeline Navarro, Santiago Segarra, and Elvin Isufi. Fair covariance neural networks. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2025.
Haibo Chen, Xin Wang, Zeyang Zhang, Haoyang Li, Ling Feng, and Wenwu Zhu. AutoGFM: Automated graph foundation model with adaptive architecture customization. In secty- International Conference on Machine Learning, 2025a. URL https: /wtp//weeeeeeeeeeeeee/ee net/ forrum?dd=CCPBqqRRR.
Jialin Chen, Haolan Zuo, Haoyu Peter Wang, Siqi Miao, Pan Li, and Rex Ying. Towards a universal graph structural encoder. arXiv preprint arXiv:2504.109 2025b.
Runjin Chen, Tong Zhao, Ajay Kumar Jaiswal, Neil Shah, and Zhangyang Wang. LLaGA: Large language and graph assistant. In Forty-first International Conference on Machine Learning, 2024a.
Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Wenqi Fan, Hui Liu, and Jiliang Tang. Exploring the potential of large language models (IIms) in learning on graphs. ACM SIGKDD Explorations Newsletter, 25(2):42-61) 61, 2024b.
Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. In International Conference on Learning Representations, 2021. URL
https: http:////peeeiieeeeee net forum?id=n66 fl.xrP.
11

---

Published as a conference paper at ICLR 2026
Vijay Prakash Dwivedi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Graph neural networks with learnable structural and positional representations. In International Conference Learning Representations, 2022a.
Vijay Prakash Dwivedi, Ladislav Rampášek, Michael Galkin, Ali Parviz, Guy Wolf, Anh Tuan Luu, and Dominique Beaini. Long range graph benchmark. Advances in Neural Information Processing Systems, 35:22326-22340, 2022b.
Vijay Prakash Dwivedi, Chaitanya K Joshi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. Journal of Machine Learning Research, 24 (43):1 48, 2023.
Taoran Fang, Yunchao Zhang, Yang Yang, Chunping Wang, and Lei Chen. Universal prompt tuning for graph neural networks. Advances in Neural Information Processing Systems, 36:52464-52489 2023.
Bahare Fatemi, Jonathan Halerow, and Bryan Perozzi Talk like graph: Encoding graphs for large language models. In The Twelfth International Conference on Learning Representations, 2024.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometrie, 2019.
Billy Joe Franks, Moshe Eliasof, Semih Cantürk, Guy Wolf, Carola-Bibiane Schönlieb, Sophie Fellenz, and Marius Kloft. Towards graph foundation models: A study on the generalization of positional and structural encodings. Transactions on Machine Learning Research, 2025, ISSN 2835-8856. URL httph::///// /atps://openne wptpp/eeeeeeee.eeee/ee/e net/ forum? Certification.
wwwooriiddmSoDieee. id- Reproducibility
Fabrizio Frasca, Fabian Jogl, Moshe Eliasof, Matan Ostrovsky, Carola-Bibiana Schönlieb, Thomas Gärtner, and Haggai Maron. Towards foundation models on graphs: An analysis cross-datases transfer of pretrained gnns, arXiv preprins arXiv::412. 17609, 2024.
Mikhail Galkin, Xinyu Yuan, Hesham Mostafa, Jian Tang, and Zhaocheng Zhu. Towards foundation models for knowledge graph reasoning. In The Twelfth International Conference on Learning Representations, 2024.
Jianfei Gao, Yangze Zhou, Jincheng Zhou, and Bruno Ribeiro. Double equivariance for inductive link prediction for both new nodes and new relation types. arXiv preprint arXiv:2302.01313, 2023.
Vikas K. Garg, Stefanie Jegelka, and T. Jaakkola. Generalization and representational limits of graph neural networks. In International Conference Machine Learning, 2020.
Chenghua Gong, Xiang Li, Jianxiang Yu, Yao Cheng, Jiaqi Tan, and Chengcheng Yu. Self-pro: A self-prompt and tuning framework for graph neural networks. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pp. 197-215. Springer, 2024.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Yufei He and Bryan Hooi. Unigraph: Learning a cross-domain graph foundation model from natural language. ArXiv, abs/2402.13630, 2024.
Sharon Hendy and Yehuda Dar. Tl-pea: Transfer learning principal component analysis. arXiv preprint arXiv:2410. 10805, 2024.
Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang, and Jie Tang. Graphmae: self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pp. 594-604, 2022.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong. Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118 12-2133,,,, 2020a.
12

---

Published as a conference paper at ICLR 2026
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In International Conference on Learning Representations, 2020b. URL https: httpa:///penneevieww http::.nee//ooorreeeee
Qian Huang, Hongyu Ren, Peng Chen, Gregor Kržmanc, Daniel Zeng, Percy S Liang, and Jure Leskovec. Prodigy: Enabling in-context learning over graphs. Advances in Neural Information Processing Systems, 36, 2023.
Xingyue Huang, Pablo Barceló, Michael M Bronstein, ismail flkan Ceylan, Mikhail Galkin, Juan L Reutter, and Miguel Romero Orth. How expressive knowledge graph foundation models? arXiv preprint arXiv:2502.13333 2025.
Jinwoo Kim, Olga Zaghen, Ayhan Suleymanzade, Youngmin Ryou, and Seunghoon Hong. Revisiting random walks for learning on graphs. arXiv preprint arXiv:2407.01224, 2024.
Thomas N. Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
Divyansha Lachi, Mehdi Azabou, Vinam Arora, and Eva Dyer. GraphFM: a scalable framework for multi-graph pretraining. arXiv preprint 2xXv:2247.111977 2024.
Jaejun Lee, Chanyoung Chung, and Joyce Jiyoung Whang. Ingram: Inductive knowledge graph
embedding via relation graphs. In International Conference on Machine Learning, pp. 18796- 18809. PMLR, 2023.
Ron Levie. A graphon-signal analysis of graph neural networks. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Renjie Liao, Raquel Urtasun, and Richard Zemel. A PAC-bayesian approach to generalization bounds for graph neural networks. In International Conference on Learning Representations, 2021.
Hao Liu, Jiarui Feng, Lecheng Kong, Ningyue Liang, Dacheng Tao, Yixin Chen, and Muhan Zhang. One for all: Towards training one graph model for all classification tasks. In The Twelfth International Conference on Learning Representations, 2024.
Haitao Mao, Zhikai Chen, Wenzhuo Tang, Jianan Zhao, Yao Ma. Tong Zhao, Neil Shah, Mikhail Galkin, and Jiliang Tang. Position: Graph foundation models already here. In Forty-first International Conference on Machine Learning, 2024.
Christopher Morris, Nils M. Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. ICML 2020 Workshop Graph Representation Learning and Beyond (GRL+ 2020), 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach De Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019.
Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gen: Geometric graph convolutional networks. In International Conference Learning Representations, 2020.
Bryan Perozzi, Bahare Fatemi, Dustin Zelle, Anton Tsitsulin, Mehran Kazemi, Rami Al-Rfou, and Jonathan Halcrow. Let your graph do the talking: Encoding structured data for llms. arXiv preprint arXiv:2402.05862, 2024.
Oleg Platonov, Denis Kuznedelev, Michael Diskin, Artem Babenko, and Liudmila Prokhorenkova. A critical look evaluation of gnns under heterophily: Are we really making progress? In The Eleventh International Conference on Learning Representations, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 1-67, 2020.
13

---

Published as a conference paper at ICLR 2026
Ladislav Rampášek, Michael Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Do- minique Beaini. Recipe for general, powerful, scalable graph transformer. Advances in Neural Information Processing Systems, 15:14501-145155 2022.
Yangyi Shen, Beatrice Bevilacqua, Joshua Robinson, Charilaos Kanatsoulis, Jure Leskovec, and Bruno Ribeiro. Zero-shot generalization of gnns over distinct attribute domains. In International Conference on Machine Learning, 2025.
Saurabh Sihag, Gonzalo Mateos, Corey McMillan, and Alejandro Ribeiro. covariance neural networks. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 17003-17016. Curran Associates, Inc., 2022.
Li Sun, Zhenhao Huang, Suyang Zhou, Qiqi Wan, Hao Peng, and Philip S. Yu. RiemannGFM: Learning graph foundation model from structural geometry. In THE WEB CONFERENCE 2025, 2025. URL https: http///pppeeeeeieeeeeeeeeeeeeeee net httpt//forrmmdddddaddrroee
Mingchen Sun, Kaixiong Zhou, Xin He, Ying Wang. and Xin Wang. Gppt: Graph pre-training and prompt tuning to generalize graph neural networks. In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 1717-1727, 2022.
Xiangguo Sun, Hong Cheng, Jia Li, Bo Liu, and Jihong Guan. All in one: Multi-task prompting for graph neural networks. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2120-2133 2023.
Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin, and Chao Huang. Graphgpt: Graph instruction tuning for large language models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 491-500, 2024.
Antonis Vasileiou, Ben Finkelshtein, Floris Geerts, Ron Levie, and Christopher Morris. Cov- ered forest: Fine-grained generalization analysis of graph neural networks. arXiv preprint arXiv:2412.07106, 2024.
Antonis Vasileiou, Stefanie Jegelka, Ron Levie, and Christopher Morris. Survey generalization theory for graph neural networks. arXiv preprint M.Xiv:2503.156500 2025.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks. International Conference on Learning Representations, 2018.
Kai Wang and Siqiang Luo. Towards graph foundation models: The perspective of zero-shot reasoning on knowledge graphs. arXiv preprint arXiv:2410. /2609, 2024.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph for learning on point clouds. ACM Transactions on Graphics (tog). 38(5): 1-12, 2019.
Zehong Wang, Zheyuan Zhang, Nitesh V Chawla, Chuxu Zhang, and Yanfang Ye. GFT: Graph foundation model with transferable tree vocabulary. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: http///peeeeeeeeeee/eeeeeeeeeeee wwt.iiiwwweeeeeeeeeeee forum? i.d=0Mxbbv8xxy.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1912-1920, 2015.
Lianghao Xia and Chao Huang. Anygraph: Graph foundation model in the wild. arXiv preprint arXiv:2408. 10700, 2024.
Lianghao Xia, Ben Kao, and Chao Huang. Opengraph: Towards open graph foundation models. In EMNLP, 2024.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference Learning Representations, 2019.
14

---

Published as a conference paper at ICLR 2026
Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting remi-supervised learning with graph embeddings. In International conference on machine learning. pp. 40-48. PMLR, 2016.
Xingtong Yu, Chang Zhou, Yuan Fang, and Xinming Zhang. Text-free multi-domain graph pre- training: Toward graph foundation models. arXiv preprint arXiv:2405.13934 2024.
Xingtong Yu, Zechuan Gong, Chang Zhou, Yuan Fang, and Hui Zhang. SAMGPT: Text-free graph foundation model for multi-domain pre-training and cross-domain adaptation. In 7HE WEB CON- FERENCE 2025, 2025. URL https: wttp//peeeevieeeeeeeeeeeeee fatt/forummddeeeee jallebb...
Haonan Yuan, Qingyun Sun, Junhua Shi, Xingcheng Fu, Bryan Hooi, Jianxin Li, and Philip Yu. How much transfer? BRIDGE: Bounded multi-domain graph foundation model with generalization guarantees. In Forty-second International Conference on Machine Learning, 2025.
net forum? URL htttss://ppenrevettt f 1dkb
Kexin Zhang, Shuhan Liu, Song Wang, Weili Shi, Chen Chen, Pan Li, Sheng Li, Jundong Li, and Kaize Ding. A survey of deep graph learning under distribution shifts: from graph out-of- distribution generalization adaptation. arXiv preprint arXiv:2410.1996. 2024a.
Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE transactions on knowledge and data engineering, 12):5555666609 2021.
Yucheng Zhang, Beatrice Bevilacqua, Mikhail Galkin, and Bruno Ribeiro. TRIX: A more expressive model for zero-shot domain transfer in knowledge graphs. In The Third Learning on Graphs Conference, 2024b.
Haihong Zhao, Aochuan Chen, Xiangguo Sun, Hong Cheng, and Jia Li. All one and one for all: simple yet effective method towards cross-domain graph pretraining. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4443 4454, 2024a.
Jianan Zhao, Le Zhuo, Yikang Shen, Meng Qu, Kai Liu, Michael Bronstein, Zhaocheng Zhu, and Jian Tang. Graphtext: Graph reasoning in text space. arXiv preprint arXiv:2310.01089, 2023.
Jianan Zhao, Hesham Mostafa, Mikhail Galkin, Michael Bronstein, Zhaocheng Zhu, and Jian Tang. Graphany: A foundation model for node classification on any graph. ArXiv, abs/2405.20445,, 2024b.
Jincheng Zhou, Beatrice Bevilacqua, and Bruno Ribeiro. multi-task perspective for link prediction with new relation types and nodes. In NewrIPS 2023 Workshop: New Frontiers in Graph Learning. 2023.
Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems, 33, 2020.
Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propaga- tion. ProQuest number: information to all users, 2002.
15

---

Published as a conference paper at ICLR 2026
A ADDITIONAL RELATED WORK
Generalization Theory of MPNNs. Significant theoretical progress has advanced our under- standing of generalization in Message Passing Neural Networks (MPNNs). As discussed in recent surveys (Vasileiou et al., 2025; Zhang et al., 2024a), these efforts often focus on how architectures and graph properties (such as maximum degree) influence the generalization gap, employing analytical tools like Rademacher complexity and PAC-Bayesian analysis to derive performance bounds (Garg et al., 2020; Liao et al., 2021). Other lines of work, leveraging concepts like covering numbers or graphon theory, investigate model stability and generalization under shifts in graph structure or topology, particularly in large-scale or evolving graph scenarios (Levie, 2023; Vasilciou et al., 2024). While these foundational theories provide important insights into GNN expressivity and their ability to generalize, especially concerning structural variations, they typically assume a consistent definition of the input feature space across different graphs. The cross- datsset generalization chal- lenge that ALL-IN addresses is distinct: we specifically tackle scenarios where graphs present node features from entirely different feature spaces, potentially varying in both the number of available features (dimensionality) and their semantic meaning between train (source) and test (target) graphs. Our theoretical framework (Section 4) therefore focuses on establishing principles for robustness and transferability under such input feature space heterogeneity, aiming to complement existing generalization theories that predominantly address structural changes.
Additional Efforts towards Graph Foundation Models. Another significant challenge in graph transfer learning arises in settings like heterogeneous knowledge graphs, where models must gen- cralize unseen entities and relation types. Approaches such as ISDEA+ (Gao et al., 2023) and MTDEA (Zhou et al., 2023) tackle this by employing set aggregation techniques over representa- tions specific to edge types, aiming for equivariance to permutations of these types, supported by a "double equivariance" theoretical framework. Similarly, methods like InGram (Lee et al., 2023),
ULTRA (Galkin et al., 2024). TRIX (Zhang et al., 2024b), and MOTIF (Huang et al., 2025) construct explicit "relation graphs" to model interactions among different edge types. These works provide valuable solutions for structural and relational heterogeneity. In contrast, ALL-IN primarily addresses the distinet challenge of heterogeneity in input features. that is, varying feature dimensionalities and semantics across graphs. While the aforementioned methods focus on generalizing over graph schema and relation types (often assuming node features are not present), ALL--IN directly processes arbitrary node features to derive transferable node-covariance operators and representations, Other efforts in graph representation learning aim for transferability across diverse graph tasks. For example, HoloGNN (Bevilacqua et al., 2025) proposes a framework to learn node representations that can be applied to various downstream tasks on a given graph graphs. However, such approaches typically assume that the underlying node feature space remains consistent across these tasks. ALL-IN, con- versely, is specifically designed to address the challenge of generalizing to new and unseen datasets where the node features themselves can differ fundamentally in dimensionality and semantics, problem distinct from task-level transfer within a fixed feature domain.
B ADDITIONAL THEORETICAL CONSIDERATIONS AND PROOFS
Proposition 4.1 (Distributional Invariance of Projected Features to Feature Permutation). Let X E Raxd be node features, P E d be any permutation matrix, and h be the projection dimension. Let C E Bddh be an isotropic Gaussian random matrix (i.e., vec(C)) ~ NO, Lan). Define the projected features as R(O) = XC and the features projected after permutation as R = PXP).
Then R))) and R()) are equal in distribution: O) d R()).
Proof. Let C have columns Ci...., C).- Since the entries C& are i.idd Wi 1), each column Cj ~ (0, Ia) and the columns are mutually independent.
Consider the matrix C = pp Since P is a permutation matrix, pT is also a permutation matrix and is orthogonal, that is PT (PTT"" P Ia-
The columns of C are C== = PTc,, cj. Since Cj ~ ~N, Ia) and PT is orthogonal, then
C 3~~((PT 0. PT I' N/ PT PT) - ~V Ia)
(10)
16

---

Published as a conference paper at ICLR 2026
Furthermore, since cI,..., Ch independent, the transformed columns C1..... C2 are also inde- pendent. Thus, the matrix C has the same distribution as C, i.e., C C. Now consider R P)C Since
d C, we can write:
R d (XP)C
Substitute C P'C:
R
(XP) X(PP')C
Since P is orthogonal, PP" La-
R - XIC
Thus, R and ()) are equal in distribution.
Corollary 4.2 (Distributional Invariance of Node Covariance Operators to Feature Permutation). Let X E Ruxd be node features, and P E Rdxd be any permutation matrix. Let R()) = XC be the initial projected features. Let K. (K(P) |ę=o be the set of node-come node-covariance operators, where (P) NodeCovl AP (R) computed using the deterministic function NodeCov (Equation (3)). and is the adjacency matrix. It follows directly from the distributional invariance of R that the entire set of operators K. is also invariant in distribution to permutations of the input features X. That is, if K. is the set of operators computed using X instead of X, then K. & R.
Proof. Let (R)) = NodeCov( (A (") be the deterministic function that computes the p-th order node covariance operator from the initial projected features R From Proposition 4.1, we have O d R Since applying a deterministic function Ip random variables that are equal in distribution results in outputs that equal in distribution, have gp (P())) 9p ()) which means K d (e) for each p 0...k. k. Furthermore, since all operators (()) in K. are derived from the same R(),, and all operators ((r) in K are derived from R()), the distributional equality extends to the joint distribution of the sets: K. _ R.
Theorem B.1 (Distributional Invariance of Hidden Representations to Input Permutation). Ler X E Raxd be node features, and P E Rdddd be any permutation matrix. Let R = XC be the initial projected features, and K. [K 1p=o be the set of node-covariance operators. Let the initial hidden representation be ()) R @ S, where S is a structural encoding matrix independent of X. Subsequent hidden representations ((() for l I,...,L are computed by deterministic GNN layer function.
(m)
The initial hidden representation ()) and all subsequent hidden representations H(() for l 1...., L are invariant in distribution to permutations of the input features X. That is, if H(() are the representations computed using XP instead of X, then (8) H(0) for all l.
Proof. We proceed by induction on the layer index l.
Base Case (E 0). Let ( XC and R (XP)C. The initial hidden representations
are ( ()) + S and ( 0 R @ S. From Proposition 4.1, we know that R(O) & R)) Since the structural encoding S assumed independent of X (and thus fixed with respect to the
permutation P), and the concatenation operation @ is deterministic function, applying this function
@ S d R @ S = (0), The base preserves the distributional equality. Therefore, H R
case holds.
Inductive Hypothesis. Assume that for some layer - > 0, the hidden representations are equal distribution: ( d (1-1))
Inductive Step (Layer f). The hidden representations at layer l are computed as:
H
(f-1))
17

---

Published as a conference paper at ICLR 2026
H = ( ((
where F/ represents the deterministic computation performed by the f-th GNN layer (given fixed learned weights), O = (I, A) U. with K = (NodeCov(AL (AP ()) } and o = [I, A) U.C. with
X [NodeCov( (A" A2RR )) pm
From Corollary 4.2, we know that the set of random operators KC is equal in distribution K, i.e.
K d
K. Since I and A are fixed, the full set of operators used by the layer also satisfies O _ Ó.
Now consider the inputs to the function Fg. The pair (H(6-1), O) determines H and the pair (H O) determines ( Both (t-11) and O are deterministic functions of the ini- tial projection ? (and fixed elements S.A,I, I, and layer weights). Let J be the function representing the computation up layer 6 1 and the computation of operators, such that (H((-1), O) (( S, A, Weights) Similarly, (Hit-1) J(R S, A, Weights).
Since (2(0) T())) (Proposition 4.1) and J is deterministic function, it follows that the joint distribution the outputs preserved:
(H = (H11)1))
This establishes that the inputs to the deterministic layer function Fe are equal in distribution. Applying the deterministic function F/ preserves this equality:
H = (.(H)(-1)) (t (() (()
Thus, the inductive step holds.
Theorem 4.3 (Distinguishability through C). There exist node features X Bexd nodes u, & V with X, & X. such that NodeCov(X X) makes u, indistinguishalle (automorphic). but NodeCov(XC)) (for a.s. all C) makes u, distinguishable (not automorphic).
Proof. We will show that there exists X, such that (1) nodes u and v are automorphic within NodeCov(X) (),, ), and consequently, the GNN, when using NodeCov( (X) as the operator and identical initial embeddings, produces identical final representations for these nodes. (2) For the same X, with probability 1 (over the draw of C), nodes u and v are not automorphic and therefore distinguishable in NodeCov(X ()). We provide constructive example. Let features. Consider the feature matrix X:
nodes {u, v, w) and d
0
X
0 1 1
1
0/
Here, X, (1,0,1)T, (0,1,1) and X (1, (1. Clearly, X. X..
Proof for item (1). The column means of X are X = 2//3.2/3,2/3) The centered feature
matrix X. II. = X 1 X is:
1/3
-2/3
1/3
X, =
2/33
1/3
1/3
1/3
1/3
-2/3
Then
2/9
-1/9
- 1/9
NodeCov(X
1/9
2/9
119
1/9
-1/9
2/9
In the weighted graph defined by NodeCov( ()) all nodes are automorphic to each other. If a GNN uses NodeCov( X) as its feature-deriven derived operator and starts with identical initial embeddings for all nodes, standard message passing layers will preserve this symmetry, leading to identical final representations H!!)) ! H.(l) Thus, such GNN cannot distinguish u from v.
Proof for item (2). Let R XC. The rows of R are R. = X,C. R.C = X,C, R X C. Since X,, * X,, and C' is drawn from continuous distribution (Gaussian entries), X.C
18

---

Published as a conference paper at ICLR 2026
X C with probability 1. Thus, Ra R. almost surely. Let R. R The rows of R. R R R Since R + R follows that Re o ( almost surely (unless II, projects their difference zero, which a measure zero event for a fixed X and random C). The operator is k NodeConv(XCL = Ho R.R... An element (K())); );j = 1 C R Consider the specific symmetry that existed for NodeConv(A (X), e.g., (NodeConv( Am = (ondeConv(X))ae = -1/9.
h
For (()), compare (K()) au R Re, and (K())) vn h R Rec... These equal if (R:3)
02 - R!..) R!!) = O. Since R R * 0 almost surely, and RC,, is random vector (whose distribution depends on C), the event that their dot product is exactly zero has probability O for continuous distributions unless one of them is deterministically zero (which is not the case here a.s.). Therefore, with probability I, (K()) ) f (K) This breaks the specific symmetry that made node u and node have equivalent relational profiles node w in NodeCov( (X). More generally, the matrix (O) will not, with probability exhibit the high degree of symmetry found in NodeCov (X) for this specific X. Thus, nodes u and will generally not be automorphic with respect ((() in the same way they were for NodeCov(XL (X). A GNN using this specific realization ((() (and identical initial embeddings, can now potentially produce H!L)) * H because the operator (0) provides different relational information for and V.
Theorem 4.4 (Expected Invariance to Orthogonal Transformations). Let X Rnux be node features, Q Rd be orthogonal matrix, and h be the projection dimension. Consider random projection matrix Ce E Rd with vec(C') ) ~ N I.n). Let (o (II, (()) R
be the
Node Covariance operator (Equation (2)). where II, = In - 1,,11, is the centering matrix. Then, the expected Node Covariance computed from the stochastically projected features is invariant the orthogonal transformation Q:
Ec (odeNeoov = Eoo
IXXXXIX
(7)
where the expectation Ec|:] is over the random sampling of and II, I is the Gram matrix of the centered original features.
Proof. Let R XCC. Using the definition of the NodeCov operator and properties of the centering matrix II::
NodeCov( (I)
h (I.
(IR (0)
h
-1,C ''II.
h
Taking the expectation over C::
Ec/NodeCov(XC) Ec II x
II..Ee p (by linearity of expectation)
We evaluate E=(CC*). Let C E Rd be the j-th column of C. Since the entries of C are i.i.d. A 1), each column vector ej follows ~ NNO,, Ia). Therefore, Elc,cf]] = la. Using linearity of expectation:
h Bolc,e.1 Σ la hla Σ
Ed|CC EEC
Σ
c
j=1
j=1
Substituting this back:
(oodeCoov(X))
-I II
II.
19

---

Published as a conference paper at ICLR 2026
Now consider the transformed features X XQ. Let R XC XQC. We compute Ec )
NodeCov (R [II ( (II.
(xX(CC)
I1.QQCC Q' I'I.
Taking the expectation over C:
Ec[NodeCov(XQC)) -I 10 X"I.
h +IIxQ(hia'') X'I.
(using E|CC") hIa)
-I.QII''X X'II.
X'X
-IXII (since Q is orthogonal, QQ" -a)
IIX
Thus, O| -I,X'111,
Theorem 4.5 (Loss Upper Bound). Let H E BRxnh be the final node representations computed by ALL-IN, dependent the initial random projection C. Let & BB - B be the final prediction layer, and let L(-, Y) be the loss function comparing predictions to ground truth labels Y. Assume that the composite function (H) = L( (H()))) Y) is convex with respect to the final node representations ( Then, our stochastic optimization objective provides an upper bound for the loss of the expected representation:
EalC(0 H Y)
8
Loss of Expocted Representation Expected Loss (Triining Objective)
where the expectation [E]]:] is taken over the random projection matrix C.
Proof. The proof follows directly from Jensen's 's inequality for vector- or matrix-valued random variables.
Let the random variable be the final hidden representation Z = H14,, which is a function of the random projection matrix C.
By assumption, the function f is convex with respect to its input argument H Jensen's 's inequality states that for convex function I and a random variable with finite expectation, f(E(2) S [Z)( Applying this with Z = (L) and the defined function f. we get:
(( Y) s (c(((((H()) Y)I
which is the desired result.
Proposition 4.6 (Consistency Projected Node Covariance). Let E Baxd be node features. For projection dimension h, let C E dRd be such that vec(C) ~ A(O0, Ia)). Define the stochastic
node-covariance operator K(00) NodeCov(XC) 1 XIIXXIII where II, is the centering matrix. Then, K(() converges in probability its expected value h +
K L. 10
as 00
(9)
Proof. Let C |ct....,
cn] denote the random projection matrix, where each column c, Rd a random vector. Since the entries of C are sampled i.i.d from No0 1), the columns Cj are independent and identically distributed according to Cj (0, Iz).
20

---

Published as a conference paper at ICLR 2026
The stochastic node-covariance operator Ky
(Equation (2)) can be rewritten as:
K
[I, (I),
h
(M
= (IXxe) (using block matrix multiplication definition)
==1
Let us define the random matrix Y, E Rax as:
Y, (IIX))
Since the columns Cj are i..d. and Y, is fixed function of c, (given the fixed matrices X and II.), the random matrices Y1, Y2,.... Y, are also independent and identically distributed (ii.d.).
The operator (() can thus be written as the sample mean of these i.i.d. random matrices:
(0)
Σ
Y
j=1
Now, we compute the expected value of y. Using the linearity of expectation and the property that II and are constant with respect to the expectation over C (and II, II?):
E e| = [
E [ X
I.EEEee X
Since cj (O, Ia). we know that E[c,c]] Cov(c,) Elc,/Ec,,1" Ia 007 Is- Substituting this in:
IIXX E| IXXIX -
Let Kep = II. We have shown that E|y,] Kepp Since X is fixed finite matrix, and the moments of Gaussian variables are finite, the expectation E|(r] exists and finite.
We have K (0) as the sample mean of h i..d random matrices Y,. each with finite expectation Keep- By the Weak Law of Large Numbers, applicable to sums of i.i.d. random vectors or matrices (considering convergence element-wise in matrix norm), the sample mean converges in probability to the expected value as the number samples goes to infinity. Therefore, for each entry (a, b) of the matrices:
KK (0) aab
(Kexp)ab as h (2))ab 2 [(()))
This element-wise convergence implies convergence in probability for the matrix:
(((0) E Kap = IIX as h + ao.
This completes the proof.
C ADDITIONAL RESULTS
C.1 CATEGORIZATION AND DESCRIPTION OF BASELINES
Table 2 compares our approach against diverse families of baselines evaluated on node classification benchmarks. We group methods into four primary categories: (i) SUPERVISED GNNS that are trained
21

---

Published as a conference paper at ICLR 2026
Table 4: Performance of ALL-IN on pre-training source datasets compared to specialized supervised baselines trained individually per dataset (including our ALL-IN-SPECIALIZZR which is trained separately on each individual dataset). ALL-IN maintains highly competitive performance.
Method
ZINC (MAB
MULHIV
MOLESOL (RMSE J)
MOLTON21
MNIST
CIFRRO MODELNET CUNEIPORM
MSRC
(ROC-AUC f)
(ROC-AUC T)
(ACC T>
(ACC f>
(ACC f)
(ACC T>
(ACC f>
TRAINED DATASET
GCN (Kipf & Welling, 2017)
0.3674
76.06
1.11
75.29
90.120
54.142
17.18
45.67
89.53
GAT (Nelicković et 2018)
0.3842
76,00
1,05
75.21
95.535
64.223
65.20
78.60
82.10
GIN (Xu et al., 2019)
0.1630
75.58
1.17
74.91
96,485
55.255
73.13
79.05
86.31
(0 props)
0.1490
72.65
1.22
69.37
94,03
39.96
37.24
85.19
91 .5.65
ALL-IN-SPPCIALIIED
0,1195
73.78
1.19
70.04
94.77
40.03
39.81
87.20
94.16
TRAINED ON ALL DATASETS
ALL-IN props)
0,1557
72.74
1.28
68.19
94.57
40,11
37.11
89.88
97.51
ALL-IN
0.1237
74.49
1.29
68.20
95,22
10.08
39.37
91.17
98.08
from scratch on each dataset, (ii) LLM-AUGMENTED GNNs where the node features are enhanced using language models, (ii) LLM-BASED REASONING that converts the graph into a compatible input to pre-trained LLMs, and (iv) GNN-BASED methods.
SUPERVISED BASELINES include (a) MLP: a multi-layer perceptron directly on the target dataset features without using graph structure: serves as a non-graph baseline. (b) GCN (Kipf & Welling, 2017): trained from scratch on the target dataset (c) GIN (Xu et al., 2019) trained from scratch, included to represent expressive message-passingg GNNs in supervised settings. These fall under supervised baselines as they do not perform pretraining or transfer, and rely solely on training from scratch on each dataset.
LLM-AUGMENTED GNNS include (a) OFA (Liu et al., 2024): constructs a prompt-augmented graph using text nodes and pretrains an RGCN to enable in-context transfer across node/link/graph tasks; falls here for fusing text prompts with GNN structure and relying LLM embeddings. (b) GLEM-LM (Chen et al., 2024b): Enhances GNNs using sentence-level text embeddings from frozen LLM; categorized here due to its augmentation of GNN input via LLM-derived features. These are classified as -Lg-muneettedd GNNs since they incorporate LLMs to enrich graph inputs guide GNN training, but retain a GNN backbone.
LLM-BASED methods include (a) GRAPHTEXT (Zhao et al., 2023) that transforms k-hop neigh- borhoods into textual prompts and performs zero/few-shot classification using frozen LLMs and (b) RWNN (Kim et al., 2024) that converts random walks on graphs to node label anonymized sequences and uses frozen LLMs for prediction. belong to this category due their reliance on prompt-based inference using LLMs without any GNNs.
GNN-BASED methods include (a) ANYGRAPH (Xia & Huang, 2024) that pretrains a graph mix- ture-of-experts model using link prediction objective on diverse graphs that allows transfer to unseen datasets, (b) GRAPHANY (Zhao al., 2024b) that learns permutation- -iniaaattioonnvviannn attention over a bank of pretrained LinearGNNs; (c) MDGPT (Yu et al., 2024) pretrains a GCN multiple datasets with SVD-projected features and prompt vectors; (d) GCOPE (Zhao et al., 2024a) constructs a universal pretraining graph with virtual nodes and uses contrastive learning to train a shared GNN; (c) GPPT (Sun et al., 2022) introduces task-specific graph prompts for node task and link- neeeprrdiition alignment; (I) GPROMPT (Gong al., 2024) utilizes prompt vectors into graph pooling via clement- wise multiplication (g) ANL-IN-ONE (Sun et al., 2023) combines token graphs with original graph as prompts (h) GPF (Fang et al., 2023) introduces prompt tokens and GPF-PLUS trains multiple independent basis vectors and combines them using attention (i) ULTRA (Galkin et al., 2024) learns transferable graph representations by conditioning on relational interactions. (i SCORE (Wang & Luo, 2024) introduces zero-shot reasoning on knowledge graphs using graph topology. All of these are grouped under GNN-N BASED baselines as they rely on pretraining GNNs (often with auxiliary components like prompts or experts) to enable generalization to new graphs
C.2 COMPARISON TO METHOPS TRAINED ON EACH INDIVIDUAL DATASET
In this section, compare the performance of ALL-IN to that of standard supervised GNN baselines (GCN (Kipf & Welling. 2017), GAT (Veličković et al., 2018), GIN (Xu et al., 2019)), trained individually for each dataset, using their original, dataset-specific input features. Contrary to AL..-IN,
22

---

Published as a conference paper at ICLR 2026
which is trained jointly all datasets, these supervised baselines are thus specialized for each respective dataset.
When evaluating the pre-training datasets, it is generally expected that supervised models trained on each individual dataset would achieve strong, if not optimal, performance, particularly as each dataset provides sufficient data for dedicated task-specific learning. The goal for ALL.IN here is therefore to show that its jointly pre-trained shared encoder can support task-specific heads that remain competitive against individually trained models, indicating its ability to learn general-purpose representations without substantial performance degradation on each task.
Table 4 summarizes the performance of ALL.IN (with and without propagated covariance operators, obtained by setting k = 0 in Equation (4), denoted as ALL-IN and ALL-IN (0 props) respectively) against the specialized version of our model PALL-IN-PPECILIZZAI (0 props)), as well as specialized supervised baselines on the pre-training datasets. Our findings indicate that, while specialized baselines maintain an edge on certain datasets (e.g. CIFAR 10 and MODELNET), ALL-IN is broadly competitive. For instance, on the ZINC regression task, ALL-IN achieves MAE of 0. 0.1237, surpassing all listed specialized baselines, including GIN (630).)). Similarly, ALL-IN demonstrates higher accuracy on CUNEIFORM (91.17% vs. GIN 79.05%) and MSRC 21 (98.08% vs. GIN 86.31%). Finally, we highlight the general advantage of the full ALL-IN (which utilizes propagated operators) over ALL-IN (0 props), and similarly of ALL-IN- SPECTALIZED over ALL-IIN-SPECIALIIEED (0 props), suggesting that the richer relational information from propagated operators contributes to more effective representation learning during this phase.
and ALL-IN-SPECIALIZED
Overall, these results indicate that a single, pre-trained ALL-IN encoder can maintain strong, often competitive, performance across a diverse set of source datasets and tasks.
C.3 USING SVM ON THE PRE-TRAINED REPRESENTATIONS
To assess the linear separability and structural quality of the learned graph representations from ALL- IN, we evaluate downstream graph classification accuracy using support vector machines (SVMs) with both linear and radial basis function (RBF) kernels (Table 5). This setup allows to probe how well the learned representations support simple (linear) versus more expressive (nonlinear) decision boundaries.
We compare against several non-learnable baselines that do not involve any representation learning:
(a) Input Features (X): Raw input features of each graph, computed by averaging node features.
(b) Propagated Input Features (AX): Features after one round of neighborhood propagation, capturing local graph structure.
(c) Input Features along with random walk structural encodings (X @ S): Concatenates the raw features with random walk structural encoding (RWSE) (Dwivedi al., 2022a), which encodes graph structure based on transition probabilities of random walks.
These baselines serve as direct input replacements for ALL-IN and are shared across both kernel settings. They provide a strong reference for understanding the inherent structure in the input space, independent of any learning or pretraining.
For ALL-IN, we report results both with and without concatenation of the input features to assess the added value of structural information in the learned embeddings.
Under the RBF kemne,, ALL-IN combined with input features achieves the best performance on four out of six datasets, including PTC, NCII, NCI109, 09, and ENZYMES, highlighting its ability to encode discriminative patterns suitable for nonlinear classification. In contrast, performance under the linear kernel is more mixed, with RWSE showing strong results on datasets like PROTEINS, indicating some inherent linear separability in the structural baseline. Overall, these results demon- strate that ALL-IN learns representations that are expressive and transferable across diverse graph datasets, especially when paired with nonlinear classifiers.
23

---

Published as a conference paper at ICLR 2026
Table Graph classification accuracy (%) using SVMs with Linear and RBF kernels. Baselines are shared across both kernels. Results are reported as mean + standard deviation over 10 runs.
Method
MUTAG (ACC 1>
PROTEINS ()CC
NC1109 (ACC f)
ENZYMES (ACC ()
(ACC
(ACC f)
LINEAR SVM
Input Features
81.87 7.25
60.88 1: 1.83
72.68
0.58
64.59
1.24
63.36
2.22
22.00
4.46
Propagated Input Features
69 14.221
57.34
10.89
59.56
3.94
66
1.22
63.26
1.63
14.33
5.01
Input Features RWSE
80.96 + 0.89
60.14 + 1.15
65.74
0.43
64,30
0.16
63.45
0.20
27.00
4.63
ATL.IN
+
9.09
4.25 62.00
1.36
6:28
ALL-I8 Input Features
74.47 = 7.70
52.84 = 9.03
4.29
64.45
1.48
63.72
1.67
21.50
5.18
RBF SVM
Input Features
72.73 14,
55.88
11.58
71.06
=: 2.93
66.44 = 1.43
66.80 1.35
33.33
4.77
Propagated Input Features
5.42
4.43
Input Fealures RWSE
79.21
10.99
58.71
1 8.76
67.21
6.22
70.68
2.60
67.82
2.79
36.66
5.96
59.28
9.13
5.87
ALL-IN Input Features
84,06
6.61
39.88 1: 7.72
71.42
4.29
67.54 1.33
67.34
1.51
32,16 6.71
C.4 ADDITIONAL RESULTS ON TRANSPERABILIT TO UNSEEN DATASETS
In Table 6, we present comparison with more baselines on our graph classification datasets MUTAG and PROTEINS. We describe below the changes we make the following baselines to make them applicable to this setting:
GLEM-LM (Chen et al., 2024b): This is a method that only supports tasks on text-attributed graphs. Since the TU Datasets (Morris al., 2020) do not have node text attributes, describe the input node features and pass them to ChatGPT.
GCOPE (Zhao et al., 2024a): This method introduces one virtual node for each node classification dataset, connecting it to all the nodes within the dataset. To perform graph classification, we introduce one virtual node for each graph classification dataset and connect it to all the nodes in all the graphs within the dataset.
ANYGRAPH (Xia & Huang, 2024): This method performs node classification by adding one node per class and connecting each training node to its corresponding class node. Classification of unlabeled nodes is performed by computing the dot product between the node's embedding and each class node embedding to rank the classes. To extend this paradigm graph classification, the graph and add one class node per category. For classifying new graphs, we compute the dot product between the virtual node embedding and each class node embedding to rank the classes.
introduce virtual node that connects to all nodes in
We leave out the following methods and provide justification below:
(Zhao al., 2023): While the authors mention that GRAPHTEXT is applicable for graph classification, they do not provide a way to construct graph syntax tree for an entire graph, which can be ambiguous as it could involve introducing a virtual node or averaging results from syntax trees of multiple nodes.
GRAPHTEXT
GRAPHANY (Zhao et al., 2024b): This method is explicitly only designed for node classifi- cation on arbitrary graphs, relies on an analytical solution that is not directly applicable to graph-level tasks.
The results in Table 6 further substantiate ALL-IN's strong performance. These findings reinforce the observations made in the main paper (Table 3): ALL-IN, with its frozen pre-trained encoder and retrained head, effectively generalizes to new graph classification datasets with novel input features, surpassing wide variety of adapted basclines.
C.5 THE IMPORTANCE OF SPES AND RANDOM PROJECTIONS IN EQUATION (5)
In this section, conduct an ablation study investigate the importance of SPEs and random projections within ALL-IN. We compare our ALL-IN with several additional models having the same backbone, loss, and training datasets, namely:
24

---

Published as a conference paper at ICLR 2026
Table 6: Performance on unseen graph-clasiifictioon datasets with new input features. ALL-IN demonstrates strong transferability. underscoring its versatility and ability to handle different fea- ture spaces. indicates these methods were modified to work on these datasets, as explained in Appendix C.4
MUTAG (ACC
PROTEINS (ACC 1)
SUIPERVISDD BASELINES
MLP
67.20 = 1.00
59.20 + 1.00
(Xu al., 2019)
89.40
LLM-AUGMENTED GNNS OFA (Liu et al., 2024)
61.04 + 4.71
61.40 + 2.99
GLEM-LM! (Chen al., 2024h)
72.97 + 0.00
43.22 12201
LLM-BASED
RWNN-DEBERTA (Kim al, 2024)
58.22 + 0.24
67.85 + 0.53
GNN-BASED
GCOPET (Zhao et al., 2024s)
81,87 + 7.26
71.84 + 3.48
ANYGRAPH (Xia Hunng, 2024)
75.61 + 6.94
72.23
4.63
MDGPT (Yo 2024)
57.36 14.26
10.226
GSP et al., 2022)
60.40 15.43
60.92
1:12.47
al., 2023)
79.87
GPROMPT (Gong et al., 2024)
73.60 + 4.76
1117-111.25
GPF (Pang et ol., 2023)
68.40 = 5.09
63.1 3.26
GPF-PLUS (Fang et 2023)
65.20 = 6.94
62.92 2.78
ULTRA(3G) (Galkin al., 2024)
63.33 1 0.00
58.09 1. 0.00
SCORE (Wang Luo, 2024)
85.33 2.11
68.54
ALL-IN props)
92.50 +
76:72 3.19
ALL-IN
92.90 = 6.34
78.20 3.81
Table 7: The impact of SPEs and random projections in Equation (5). ALL-IN with SPEs performs best, while using only SPEs leads to significant drop in performance, highlighting the importance of random feature projections, which cannot be compensated by using SVD.
Method
ZINC [MAR L)
MOLESOL (RMSF 4)
MOLHIV (ROC-AUIC f)
MOLTOX21 (ROC-AUIC 1)
MNIST (ACC 1)
CIFARIO (ACO:T)
MODELNET
CUNEIFORM (ACC f)
MSRC 21
(ACC1)
(ACC f)
ALL-IN (SVD)
0.1445
1443
71.82
65.55
92.97
37.12
36.51
87.28
95.84
SPEs-only
85.89
ALL-IN (SVD SPEs)
0.1318
1.41
72.06
66.
93.40
37.74
36.95
$8.56
96.91
SPEs)
1.31
ALL-IN
0.1237
1.29
74.49
68.20
95.22
40.08
39.37
91.17
98.08
(0)), thus removing both random projections and SPEs, and replacing them with SVD of the input features;
ALL-IN (SVD), where we replace Equation (5) with ()) = SVD(X
SPEs-only variant, where we replace Equation (5) with (0) S, while keeping the same covariance operator set and head, therefore removing R (0) only from Equation (5), but still
using (()) to define the covariance operators.
ALI.IN (SVD SPEs), where we replace Equation (5) with H SVD(X JSS thus removing random projections and replacing them with SVD of the input features (while keeping SPEs);
ALL.IN (no SPEs), where we replace Equation (5) with H
R (0) thus removing SPEs;
The results Table support our claim: the full ALL-IN with SPEs performs best, but only slightly better than the version without SPEs. In contrast, using only SPEs leads to significant drop in performance, highlighting the importance of random feature projections, which provides improved performance also when compared with SVD.
C.6 THE IMPORTANCE OF RANDOM PROJECTIONS
In this section, we demonstrate the impact of random projections by comparing ALL-IN with the baseline obtained by removing random projections from Equation (5) (thus, setting ()) = -) and from Equation (2), thus replacing NodeCov X XC) with NodeCov(X)
25

---

Published as a conference paper at ICLR 2026
Table 8: The impact of using random projections within ALL-IN, obtained by comparing ALL-IN to its counterpart that has no random projections in either Equation (5) or Equation (2).
Method
MNIST (ACC f)
MODELNET
CUNEIFORM (ACC 1)
MSRC
(MAR !)
(RMSE ())
(ROC-ACC 1)
(ROC-AUC T)
(ACC 1)
(ACC 1)
(ACC 1)
ALL-IN (ao random)
0.1475
1.51
71.40
62.85
91.10
35,42
33,91 39.37
85.47 91.17
95.02 98.08
ALL--N
0.1237
1.29
74.49
68.20
95.22
40.08
Table 9: The impact the operators the operator set (Equation (4)). Results improve when considering covariance operators compared to graph (adjacency) only, highlighting their importance in ALL-IN.
Method
ZINC
MOLESOL
MOLHIV (ROC-AUC 1)
MOLTOX21
MNIST (ACC 1)
CIPARTO
MODELNET
CUNEIFORM
MSRC 21 (ACC f)
(MAE 4)
(RMSE L)
(ROC-AUC 1)
(ACC f)
(ACC1)
(ACC f)
Identity Only
0.1535
1.65
67.10
60.33
86.22
29.34
25.15
81.49
91.78
Adjacency Only
0.1378
1.46
71.75
65.17
92.78
35.22
31.40
87.25
95.10
Covariance Only
0,1282
1.34
73,80
67.93
94.30
38.50
36.85
89.44
97.13
ALL-IN
0.1237
1.29
74.49
68.20
95.22
40.08
39.37
91.17
98.08
The results in Table suggest that random projection is critical to bridge input feature spaces. This aligns with our results in Theorem 4.3, which demonstrates the theoretical benefit of using random projections in the covariance operators.
C.7 ABLATION STUDY ON THE OPERATOR SET
In this section, we perform an ablation study isolating the contribution of different operators in ALL- IN. Table reports the performance of ALL-IN (which uses the operators defined in Equation (4)) and compares it with Identity Only, obtained by setting O = {1} in Equation (4), Adjacency Only, obtained by setting O = {A] in Equation (4), and Covariance Only, obtained by setting O = (K()
Equation (4).
Covariance operators enable the neural network to learn shared characteristies in input feature spaces and graph structures, results improve when considering covariance operators compared to graph (adjacency) only operators.
C.8 THE ROLE OF THE FEATURE DIMENSIONALITY
We next evaluate the performance of ALL-IN when varying the hidden dimension h. Results are reported in Table 10.
Across datasets, performance improves from very small h and then plateaus at 256, and gains beyond that are marginal. This trend aligns with Proposition 4.6: grows, the stochastic operator concentrates around its expectation. In practice, a moderate h achieves near-saturated accuracy with a better compute/memorry trade-off than a very large h. Therefore, model performance stabilizes at moderate h and larger h primarily improves stability, matching the proposition's 's claim.
C.9 ADDITIONAL DATASETS
We further evaluate ALL-IN on the larger node-level dataset ogbn-arxiv (Hu al., 2020a) (169,344 nodes, 1,166,243 edges), on heterophilic benchmarks Actor, Chameleon, Squirrel using the splits in Pei al. (2020), and on the AmzRating. Minesweep, Tolokers datasets (Platonov al., 2023). All results, which are reported in Tables 11 to 13, respectively, show that ALL-IN offers consistently better performance. We also investigate the behavior of our covariance operators on heterophilous graphs. Intuitively, the node-covariance matrix computed from the projected input features captures feature similarity across all node pairs, not just along edges. For heterophilous graphs, the base covariance operator (U) can therefore highlight similarities between non-adjacent nodes in the original input graph or dissimilarities between adjacent nodes, which can help GNNs with heterophily, In addition, the propagated operators (r) for p > 0 further help in this setting, because their availability to the GNN allows it view and mix information from multiple neighborhoods, in line
26

---

Published as a conference paper at ICLR 2026
Table 10: Performance of ALL-IN with varying hidden dimension h.
Method
ZINC
MOLESOL
MOLIIIV (ROC-ATIC
MOLTOX21
MNIST (ACC
CIFARIO
MODELNET
CUNEIFORM
MSRC 21
()
(RMSR.J.
(ROC-AUC
(ACC
(ACC
(ACC
(ACC
ALL-IN IN (h = 64)
0.1316
1.43
72.20
65.75
92.14
36.15
34.26
87.89
96.12
ALL-IN IN 128)
0.1264
1.34
73.46
67.91
94.41
38.92
37.88
90.21
97.56
ALL-IN 256)
0.1239
1.30
74.38
68.14
95.03
39.85
39.19
91.05
98.01
ALL-IN (+ 512)
0.1237
1.29
74.49
68,20
95.22
40.08
39.37
91.17
98.08
1024)
0.1236
1.28
68.18
95.24
40.11
39.40
91.22
Table 11: Performance on the ogbn-arxiv (Hu et al., 2020a).
Method
egbu-arxiv
PON-PARAMETIIC BASELINES
LABEL PROPAGATION (Zhu Ghahramani, 2002)
61.04
BASELINES
GCN (Kipf Welling, 2017)
71.74
GAT (Velicković al.. 2018)
71.95
GraphGPS (Rampaäek
2022)
70.97
LLM-AUGMENTED GNNS
OFA (Liu et al., 2024)
73.22
LLM-BASED
GraphTeext (Zhao et al., 2023)
49.47
GNN-BASED
AnyGmpph (Xia Hunng, 2024)
62.33
GraphAny (Zhao et al., 2024b)
58.38
ALL-IN
75.27
with understandings from literature on heterophily in graphs (Zhu al., 2020; Chien al., 2021). Motivated by this discussion, we conduct an ablation study where we vary the number of propagation orders & E {0. 1, 2} used in the covariance operators and evaluate downstream performance on Actor, Chameleon, and Squirell As reported in Table 14, adding propagated operators consistently improves performance.
C.10 FIXED RANDOM PROJECTIONS
In the main experiments, the projection matrix C is sampled each forward pass, which yields the distributional invariance guarantees in Section 4. To isolate the empirical effect this stochasticity. consider variant where C is sampled once and kept fixed for all subsequent training and inference steps denoted ALL-IN (Fixed C'.. We keep all other settings, including the backbone and training budget, identical. Table 15 reports performance on the pre-training source datasets, and Table 16 reports transfer results on representative downstream tasks. Across all pre- tre-training datasets in Table 15, fixing C' leads to a consistent but moderate degradation compared to the stochastic variant. A similar pattern holds on the downstream tasks in Table 16, where ALL-IN (Fixed C)) underperforms the stochastic version on both node and graph classification. These results empirically support the beneficial role of stochastic projections in our framework, while showing that the model remains competitive also when the projection matrix is fixed.
C.11
EDGE FEATURES ABLATION
For datasets with edge features such as ZINC, we follow the strategy described in Section where edge features are first randomly projected, then aggregated to nodes, and used to construct additional node-covariance operators that are added to the operator set. Concretely, the aggregated edge features are converted into an n X n edge covariance operator Keige, whose entries compare the aggregated edge-feature environments of all node pairs, and, the backbone GNN uses the projected edge features. To quantify the empirical contribution of this design, perform ablation on ZINC that compares: (i) a standard GIN without edge features, (ii) GINE (GIN with edge features), (iii) ALL-IN with edge features removed (ALL-IN (no edge features)), and (iv) the full ALL-IN using the edge-derived covariance operator as described above. Results are reported in Table 17. From Table 17. we observe that including the bdse- covariance operator yields substantially better performance
27

---

Published as a conference paper at ICLR 2026
Table 12: Performance on heterophilic datasets, using the splits in Pei al. (2020).
Method
Actor (ACC f)
Chameleon (ACC
Squirrel (ACC f)
NOX-PARAMETRIC BASELINES
LAREL PROPAGATION (Zhu & Ghahramani, 2002)
18.83 0.00
40.89 0.00
33.42 0.00
SUPERVISED
GCN (Kipf& & Welling, 2017)
28.55 0.68
64.69 2.21
47.07 0.71
GNN-BASED
GraphAny (Zhao al., 2024h)
28.60 0.21
62.59
0.86 N/A
49.70 0.95 N/A
ULTRA (Galkin et al., 2024)
22.61 0.00
SCORE (Wang & Luo, 2024)
23.26 0.56
N/A
N/A
ALL-IN
29.47 0.38
67.40 1.29
49.98 0.73
Table 13: Performance on the AmzRating, Minesweep, Tolokers datasets (Platonov et al., 2023).
Method
AmzRatings (xCC
Minesweeper (ACC
Tolokers (ACC f)
GCN (Kipf & Welling, 2017)
47.35 +: 0.26
81.12 + 0.37
79.93 + 0.10 78.24 +
GraphAny (Zhao et al., 2024b)
42.84 0.04
80.46
ALL-IN
49.02 0.11
82.93 + 0.26
81.43 + 0.07
than omitting edge features entirely, and that ALL-IN with edge features not only recovers but surpasses the behavior of an edge-aware GNN such as GINE. contrast, removing edge features in ALL.IN leads performance closer standard GIN, consistent with observations from the supervised GNN literature. This ablation indicates that the aggregation scheme in Section 3 retains and effectively utilizes edge information.
C.12 PRE-TRAINING WITH CITATION NETWORKS
In the main experiments, citation networks are excluded from the pre-training corpus to act as out-of- distribution targets with very tigh-dimensional, sparse features and large graph sizes. We now show that ALL-IN can also benefit from citation networks during pre-training, and consider extended setting where Cora and CiteSeer are added to the pre-training mix. We keep the architecture and training budget fixed, and compare (i) the original pre-training corpus (no citation networks) and (ii) the extended corpus (original + Cora + CiteSeer). Table 18 reports pre-training performance on all source datasets, including Cora and CiteSeer for the extended setting. The results show that adding citation networks leaves performance on the original pre-training corpus stable, further indicating the ability of ALL-IN in learning from multiple sources acting as an input feature space bridge. Table 19 reports downstream performance on OGBN-ARXIV, MUTAG, and PROTEINS for both pre-training regimes, indicating that including citation networks in pre-training maintains or improves downstream performance.
C.13 TRANSFER TO ADDITIONAL. DOMAINS
To further evaluate the generality of ALL-IN beyond citation and bioinformatics datasets, we consider two downstream tasks from distinct domains: (i) 3D shape segmentation ShapeNet, and (ii) social- network classification on MDB--INARE (IMDB-B). For ShapeNet, we use knn graphs over point clouds as is standard with this dataset (Wang al., 2019) and report mean Intersection-over-Umee (mloU); for IMDB-B, we report classification accuracy. In both cases, we use the same ALL-IN encoder in the main experiments and compare with two GNN baselines (GIN and GPS). As shown in Table 20, ALL-IN consistently outperforms the GIN and GPS baselines on both ShapeNet and IMDB-B. This indicates that the input-space bridge from ALL.-N yields representations that are beneficial also in 3D shape graphs and social networks, further highlighting its effectiveness.
C.14 DOWNSTREAM REGRESSION TRANSFER
Our pre- pre-training stage for ALL-IN uses a supervised multi-task objective over several graph-level datasets, including both graph classification and graph regression. This design choice reflects
28

---

Published as a conference paper at ICLR 2026
Table 14: Effect of the number of propagation orders & on heterophilous benchmarks.
Number of propagation orders
Actor (ACC f)
Chamelcon (ACC t)
Squirrel
(ACC f)
28.62 0.45
65.12 1.44 1.35
47.89 0.80
29.00 0.42
66.37
49.14 0.76
2
29.47 = 0.38
67.40 + 1.29
49.98 + 0.73
Table 15: Effect of fixing the projection matrix C' during pre-training.
Method
ZINC
MOLESOL
MNIST CIFARIO (ROC-AUC 1) (ROC-AUC
MOLHIV
MOLTON21
MODELNET
CUNEIFORM
MSRC21
(MAE (RMSE..
(ACC1)
(ACC
(ACC
(ACC
ALL-IN (Fixed C))
0.1369
1.38
74.12
66.72
93.97
39.84
39.02
90.11
96.27
ALL-IN (stochastic
0.1237
1,29
74.49
68.20
95.22
40.08
39.37
91.17
98.08
goal of learning a single encoder that learns across diverse graph modalities and objectives. The motivation for including regression tasks such as ZINC in the pre-training mix is inspired by the broader multi-task and of tasks and objectives is widely used to encourage more general-purposs representations (Zhang & Yang, 2021; Raffel et al., 2020). To directly demonstrate regression-style transfer, we additionally evaluate ALL-IN on held-out graph-level regression benchmark not used during pre-training, PEPTIDES-STRE STRUCT from LRGB (Dwivedi et al., 2022b). We compare GNN baselines (GINE and GPS) with ALL-IN. As shown in Table 21, ALL-IN achieves the lowest mean absolute error PEPTIDES-STRUCT, demonstrating the effectiveness of ALL-IN also in a regression downstream task.
iandattion-oode literature: training shared encoder on diverse collection
C.15 SUPERVISED VS. UNSUPERVISED PRE-TRAINING OF ALL-IN
Our main experiments adopt a supervised multi-task pre-training objective for ALL-IN, combining graph-level classification (e.g., OGBG-MOLHIV MODELNET) and regression tasks (e-g.. ZINC). This design leverages the availability of labels on diverse source datasets to learn input-space agnostic representations that are directly aligned with downstream prediction objectives. Prior work on graph representation learning has shown that, when labels are available, supervised pre-training can yield stronger and more disk-disrimiinative representations than purely self-supervised approaches (Hu al., 2020b), and similar observations hold in large-scale vision studies (He et al., 2022).
To provide an empirical comparison between supervised and unsupervised pre-training on top of input-space bridge, construct unsupervised variant in which replace all supervised losses on the pre-training datasets with masked-feature reconstruction objective of masked graph autoencoders (Hou al., 2022). Concretely, in Hou al. (2022) randomly mask node features and train ALL-IN to reconstruct the original feature values from the node embeddings. The encoder architecture and training budget are kept identical the supervised setting. Then, we benchmark the downstream performance on Cora and MUTAG. As shown in Table 22, both pre-training approaches achieve similar downstream performance, where the supervised variant slightly outperforms the unsupervised. This is consistent with prior observations that supervised objectives can provide particularly strong graph representations when labels are available, and supports our choice to adopt supervised multi-task pre-training for ALL-IN in the setting considered in this work.
1.16 EFFECT OF THE NUMBER OF PROPAGATION ORDERS
ALL-IN constructs node-covariance operators not only on the original projected features, but also on features that have been propagated through the graph up to k times, as discussed in Section 3. Intu- itively, increasing the number of propagation orders l allows the covariance operators to incorporate multi-hop information coupled with the input features, at the cost of additional computations and operators. In the main experiments we set & = 0, 2 as a default choice. Here, we provide an extended ablation over & E {0, 2, 3, 4). In this study we vary the number of propagation orders k, while keeping all other hyperparameters and training settings unchanged. The case = uses only the input features node-covariance operators, whereas larger progressively add operators built from 1-hop, 2-hop, and righer-order propagated features. Table 23 reports the pre-training performance of AL...-IN across all source datasets for different values of k. As can be seen, moving from k 0 to
29

---

Published as a conference paper at ICLR 2026
Table 16: Effect of fixing the projection matrix C on downstream transfer performance.
Method
CORA (ACC 1>
CITESCOR (ACC
MUTAG (ACC
PROTEINS
(ACC: 1)
ALL-IN (Fixed C))
81.95 d: 0.85
68.43
0.92
91.26 :: 5.59
75.86 1: 4.05 78.20 3.81
ALLI-N (stochastic C))
82. d: 0.97
69.12 0.89
92.90 :: 6.34
Table 17: Effect of using edge features and odge-based covariance operators on ZINC (MAE І).
Method
ZINC (MAE !)
GIN GINE (GIN with edge leatures)
0.3870
0.1630
odge features)
0.2583
ALL-IN (with edge features)
0.1195
small positive values of k yields consistent improvements, confirming the benefit of incorporating multi-hop feature information into the covariance operators. Performance largely saturates around 2-3 hops. Thus. we choose to work with k = 2 in the main experiments as a good balance between accuracy and efficiency.
C.17 ASYMPTOTIC COMPUTATIONAL COMPLEXITY
For graph with nodes and m edges, with node feature matrix X Raxd, projecting features using a random linear transformation takes (ndh) time and (nh) memory. where is the projection dimension. Computing ( [R())} p=1 takes (k((m+n)) time, as this is equivalent to peessage-passing layers propagating ()) The centering operation takes (kkn)) time. Notably, when explicitly constructing the node-covariance operators K (p) 1/2 R (R! E R=x= the computational complexity (kn²2h) and memory complexity is O(kr²) (as =, ..... k), resulting in quadratic complexity with respect the number of nodes. This explicit construction may be necessary in certain scenarios such as subgraph GNNs where the full pairwise similarity matrix is required as the graph structure itself (Bevilacqua et al., 2025). However, for standard message passing operations in most MPNNs (Kipf & Welling, 2017; Xu et al., 2019; Rampášek et al., 2022), we can avoid explicitly constructing the covariance matrix. Because message passing can be written left-hand multiplication by propagation matrix (our covariance operator K), and by substituting the definition K = R.R.. we can compute R ((t-1) instead of (RR.T (L-1)) This way, at no point do need to hold the full covariance matrix in memory. This approach has computational complexity Olk(mh + (f-((-)))) and memory complexity (n + 1)(=1))) for the entire layer computation, where (f-1) is the feature dimension of ( avoiding the O(n²) memory bottleneck while producing mathematically identical results. Therefore, the computational complexity of ALLLIN assuming the covariance matrix does to be stored, which is the case in experiments, is Ok((mh - (h)((-1))) time and On + (((-1))) space.
D DATASET INFORMATION
In this section, we describe the datasets used in our experiments. We categorize them based on their use in pretraining and task transferability.
D.1 PRE-TRAINING SOURCE DATASETS (A1)
For pretraining ALL-IN, we use 10 diverse datasets covering molecular graphs, drugs, computer vision, and 3D shapes. The statistics for each dataset are summarized in Table 24. The detailed information as follows:
ZINC (Dwivedi al., 2023) is molecular property prediction dataset where the task is regressing the constrained solubility values of molecules. We report mean absolute error (MAE) the evaluation metric.
30

---

Published as a conference paper at ICLR 2026
Table 18: ALL-IN pre-training performance on different pre-training corpus, with and without citation networks.
Fre training corpus
ZINC (MAE
MOLESOL
MOLIIV
MOLTOX21
MNIST CIIRRO MODELNET
CUNDIPORM
MSRC21 (ACC
CORA, (ACC t)
CITLSEER
(RMSE L)
(ROC ALC
(ROC AUC (> (ACC () (ACC 1)
(ACC
(ACC f)
(ACC
Original
0.1237
1.29
74,49 74.52
68.20 67.99
95.22
6008
39.37
91.17
98.08
Original
Cor Cieeeerr
0.3258
1.30
95.18
40112
39.21
91.08
98.25
82.99
09.33
Table 19: Downstream performance of ALL-IN with and without citation-network in pre-training corpus.
Pre training corpus
OGBN ARXIV (ACC T)
MUTAG (ACC T)
PROTEINS
(ACC T)
Original (no citation networks)
75. 27 75.61
92.90 + 6.34
78.20 + 3.81
Original CiteSeer
+ 78.24 3.77
MOLHIV, MOLESOL, MOLTOX21 (Hu et al., 2020a) is a collection molecular graphs from the OGB benchmark covering drug discovery and toxicity prediction tasks. Depending on the dataset, we perform binary classification (MOLHIV), regression (MOLESOL), or multi- label classification (MOLTOX21). Performance is measured using ROC-AUC RMSE, as appropriate.
MNIST, CIFARIO (Dwivedi et al., 2023) are computer vision datasets converted into graph- structured superpixels. Each image is modeled a fixed-structure graph, with 1 1-dimensional input features and a 10-way classification objective.
MODELNET (Wu et al., 2015) is a 3D object classification benchmark where shapes are represented as fixed-size point cloud graphs. We the 10-class subset.
CUNEIFORM Morris et al. (2020) is graph-baser based OCR dataset derived from ancient script symbols, consisting of 62-node graphs with 150 edges on average and a 30-class prediction target.
MSRC-21 Morris al. (2020) is image segmentation dataset where region adjacency graphs are constructed from visual scenes. Each graph has approximately 212 nodes and 336 edges, with 4-dimensional node features and 21 semantic class labels.
D.2 TRANSFERABILITY TO UNSEEN DATASETS AND INPUT FEATURES (A2)
To evaluate the transferability of ALL-IN to unseen input features, we choose the following datasets summarized in Table 25 and explained below:
CORA, CITESEER, PUBMED Yang et al. (2016): In these datasets, nodes represent aca- demic papers and edges denote citation links. Each node is assigned a class label corre- sponding subject area. The task is to predict the category of paper based on its content features and citation graph. Models are evaluated under transductive learning settings using fixed splits Yang al. (2016).
MUTAG Morris et al. (2020): A binary classification dataset of small molecule graphs Nodes represent atoms with categorical features, and graphs are labeled based on mutagenic effect bacterium.
PROTEINS Morris et al. (2020): A dataset of protein structures modeled graphs where nodes represent secondary structure elements and edges reflect neighborhood in the amino acid sequence. Each graph is labeled enzyme or non-enzyme.
E IMPLEMENTATION DETAILS
We implement ALL-IN using Py Torch (Paszke et al., 2019) (BSD-3 Clause license) and Py PyTcch Geometric (Fey & Lenssen, 2019) (MIT license). For experiment tracking and hyperparameter logging, we utilize the Weights and Biases framework (Biewald, 2020). Experiments were conducted with NVIDIA RTX A6000, RTX 4090, and NVIDIA A100 GPUs.
31

---

Published as a conference paper at ICLR 2026
Table 20: Transfer to 3D shapes (ShapeNet) and social networks (IMDB-B) with ALL-IN, Higher is better for both mloU and accuracy.
Method
ShapeNet
(MTOU ")
(ACC f)
GIN
83.6 +: 0.4
75.1 0..1::55 5.1
GraphGPS
84,9 1: 0.2
76.3 1:
ALL-IN
85.4 0.3
77.2
Table 21: Downstream regression transfer on PEPTIDES-STRUCT (MAE 4).
Method
PEPTIDES-STRRICT (MAE
GINE GPS
0.3547 +: 0.0045
0,2500 = 0.0005
For all experiments, we use the GPS framework (Rampášek al., 2022) with the GIN message passing layer (Xu et al., 2019) for {GNNLayerr (CAA)(,,))) ( )=o= and we use standard message passing layer for other operators.
E.1 PRE-TRAINING ON DIFFERENT SOURCE DATASETS (Q1)
To evaluate large-scale transfer, we pretrain ALI.II on a diverse set of 10 graph datasets spanning multiple domains, described in Appendix D. Each training epoch cycles through all datasets once, optimizing dstasct-specific objectives. We train for 500 epochs and checkpoint every 25 epochs. Hyperparameters are detailed in Table 26. To accelerate training, (I) we use DataPars aaaaaaaalll llel to support multi-GPU runs, (2) cache the random projection matrix C and refresh every 100 steps, (3) sample 10,000 graphs randomly at each epoch for MNIST and CIFAR I 0, and (4) sample 128 nodes with 6-nearest neighbors as edges for MODELNET in each graph.
E.2 EVALUATION ON UNSEEN DATASETS AND INPUT SPACES (Q2)
To evaluate the transferability of ALL-IN to unseen datasets with novel input features, we freeze the pretrained encoder and evaluate its representations by training lightweight classifiers new target datasets. These datasets span both node-level and graph-level classification tasks, with input feature spaces and labels disjoint from those used during pretraining.
For each target dataset, we instantiate prediction head using one of the following: (1) a multi-layer perceptron (MLP) for both node and graph classification tasks; (2) 2-layer GCN Kipf & Welling (2017) applied to node classification benchmarks (CORA, CITESEER, PUBMED); and (3) a 2-layer GIN Xu al. (2019) for graph classification benchmarks (MUTAG, PROTEINS). All prediction heads are trained with frozen ALL-IN features as input. No gradients are backpropagated into the encoder during this stage.
For MIPs, we use single hidden layer of size 128 with ReLU activation, followed by softmax or sigmoid output layer, depending on whether the task single-label or multi-label. We train all classifiers using the Adam optimizer with learning rate of 0.001 and early stopping based on validation loss. Node classification models are trained on the standard 20/30/50 splits Yang et al. (2016) and evaluated using accuracy. For graph classification, we perform 10-fold stratified cross-valiaee cross-validation and report the mean and standard deviation classification accuracy.
All transfer experiments implemented in PyTorch and PyTorch Geometric. Environment and optimization settings match those described in Appendix E.
32

---

Published as a conference paper at ICLR 2026
Table 22: Comparison of supervised unsupervised pre-training of ALL-IN.
Pro-training approach
Cora 1)
MUTAG 1)
ALL-IN (unsupervised)
82.05 0.89
91.96 6.24 92.90 6.31
ALL-IN IN (supervised)
82.13 0.97
Table 23: Effect of the number of propagations is on pre-training performance.
ZINC (MAE
MOLESOI. (RMSE
MOLHIV (ROC-AUC t)
MOLTOX21 (ROC-AUC
MNIST (ACC
CIFARIO
MODELNET
CUNEIFORM
MSRC21
(ACC
(ACC
(ACC
(ACC 1)
0.1557
1.28
72.74
68.19
94.57
40.11
37.11
89.88
97.51
0,1415
1.29
73.60
68.30
94.95
40.20
38.20
90,40
97.85
0.1237
1.29
74.49
68.20
95.22
40.08
39.37
91.17
98.08
39.45
0.1239
1,30
74.65
68.22
95.28
40.18
39.30
91.10
98.05
Table 24: Statistics of pre-training datasets used in ALL-IN. The datasets span molecules, drugs, computer vision-derived graphs and 3D shape point clouds. Our pretraining corpus contains up to 200,558 graphs
Dataset
# Nodes
Edges
# Features
# Classes
Domain / Category
ZINC
23.2 (avg)
(avg)
28
Molocular Graph Regression
OGBG-MOLESOL
13.3 (avg)
13.6 (avg)
Solubility Prediction
OGBG-MOLHIV
25.5 (avg)
27.5 (avg)
Drug Discovery
DGRG-MOLTOX2]
18.6 (avg)
19.4 (avg)
12 (multi-lahel)
Toxicology
MNIST (SUPERPINELS)
75
142
10
Vision (Digits)
CIFAR 10 (SUPERPIXELS)
85
170
10
Vision (Ohjects)
MODELNET
100 (fixed)
(fixed)
3D Shape Classification
CUNEIPORM
62 (avg)
150 (avg)
30
Archacology / OCR Image Segmentation
MSRC
212 (avg)
336 (avg)
Table 25: Statistics of finetuning datasets used in our experiments. For node classification datasets (citation networks), we report the total number nodes and edges. For graph classification datasets (bioinformatics), we report the number of graphs and average graph sizes.
Dataset
Graphs / Nodes
Edges
Features
Classes
Domain
CORA
2,708 nodes
5.429
1.433
Citation Network/ / Node Classification
CITESEER
3,327 nodes
4,732
3,703
Citation Network / Node Classification
PUBMED
19,717 nodes
44,338
500
3
Ciation Network / Node Classification
graphs
17.9 (avg)
Bicinformatics Graph Classificaiion
PROTEINS
1,113 graphs
39.1 (avg)
3
2
Bicinformaties / Graph Classification
Table 26: Hyperparameter Configuration for Pretraining Stage.
Category
Hyperparameter (Valne)
Architecture
Activation Function
Relu
Attention Type in GPS
Per Cerefoorrttente
GPS Heads
Channels (*)
236
Random Projection Dim J
512
Backhone GNNLayer
gps_gine 6
Number of Layers T.
Input ft Use Random Projections
True 0,
Node- Cova--Carriance Operators k
Training Setup
Pretraining Epochs
500
Batch Size
64
Dropout
Leamiing Rate
0.0001
Weight Decay
Normalization Type
bat.chro.m
33


Reviews:
[{"review_id": "SA1juTiMfD", "reviewer": "Reviewer_PJTg", "summary": "Finding a good and general approach for graph foundation models is a relevant, important and actual topic in representation learning for structured data. The present article develops a really interesting approach in this direction, by proposing a novel way to think about misalignement between attributed graph datasets. The idea is to randomly project the features on the nodes and to compute the node-covariances operators though these projections. This ensures that key properties of the features are kept (e.g., strong relations between the features of some nodes) while providing a representation of these features which is invariant to permutations and more generally orthogonal rotations, of the features, and also which has the same dimension whatever the initial dataset.\n\nThe key idea in this work is very clever. The section 3 first details the method using this random projection and the concept of node-covariance operators, and secondly proposes a novel node representation method by using these elements inside a GNN (which encodes the structure, while the random features and node-covariance operators encode the attributes,  possibly supplemented to structural encodings if one wants). Then Section 4 proposes a thorough theoretical analysis of the key properties: invariance of the representation under orthogonal rotations ; cases of distinguishability ; proof that the training provides an upper bound of the training objectives formulated ; some elements on the transferability of the representation. All that are really good elements.\n\nNumerical experiments follow and they are solid. The reader basically will agree to the steps and the conclusions obtained by the authors. The obtained performance are good, both as general purpose representions (Table 1), and for transfer to new datasets (Tables 2 and 3).", "questions": "I have some questions or suggestions:\n\nThe step of equation (1) could be stressed a little bit more, possibly writing already here some insight about why you propose that. Also, could it be generalised to other types of random projections, or of sketching ?\n\n* For theorem 4.5: I would put the remark in parentheses (\"this holds,… \") outside of the theorem. Possibly just underneath.\n\n* In 5.1 : it appears to be only 9 datasets, while D.1 writes 10 (we only have the results for 9 in the paper).\n\n* Table 1: the order of the data is not natural. It would be easier to have it either ordered so that data with the same metric are adjacent, or ordered in the same manner than in the text (ModelNet should then be last). Still, having dataset where smaller is better first (ZINC, MOLSOL), then ROC-AUC (MOLHIV, MOLTOX21) and finally datasets measured with ACC (MOLHIV and the others) would be nice.\n\n* Do you have a comment to compare the proposed ALL-IN to SCORE which appears to be a good competitors. Do you think that ALL-IN can be generally better to transfer in unseen graphs (as in Table 3) or do you currently know no more ?\n\n* page 23, third bullet point: $+ S$ should be $ \\oplus S $, shouldn't it ?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The strengths of the article are : \n\n1. A very good and novel proposition to build a transferable, general and learnable representation of attributed graphs with consistency across datasets, and transferability. Hence, this work is a very good step toward an efficient graph foundation model.\n\n2. There are key theoretical insights, well proven in the article (or its Appendices) that show the impact and the importance of the various elements. \n\n3. Numerical experiments and conducted on several situations. The results are compared to a variety of state-of-the-art methods for graph learning (some supervised baselines ; some GFMs using LLM ; some GFMs built on top of GNN like the present method). The  complementary studies about the separability (linear or non-linear) of the obtained representation, and the ablation study are good also (for ablation: to study the impact of structural and positional encoding ; the impact of random projections ; the impact of using covariance operator ; of propagating it ; \tand so on). This is really a solid work. \n\n4. The writing of the article is globally exemplar.", "weaknesses": "I did not find any weakness in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wwb3DDjxa4", "reviewer": "Reviewer_ptNs", "summary": "This paper tackles a critical problem for graph models: graph datasets have inconsistent node features (like different dimensions or meanings), which stops a model trained on one dataset from working on another. The authors propose ALL-IN, a method that solves this by projecting features into a shared random space. Instead of using the original features, it computes a node-covariance matrix that captures node similarities. The GNN then uses this matrix to learn, making the model independent of the original, messy feature space. This allows ALL-IN to generalize to new, unseen datasets with completely different features, without needing to be retrained.", "questions": "1. How can the $\\mathcal{O}(n^2)$ complexity scale to foundation model-sized graphs? Are the sparse approximations you mentioned practical?\n2. How critical is it to sample the projection matrix $C$ at each forward pass? Your theory (Prop 4.1, Thm 4.3) relies on this stochasticity. Would the method's benefits (especially distributional invariance) be lost if $C$ were sampled only once and then fixed throughout all training and inference?\n3. The method achieves invariance by reducing features to their second-order statistics. This must inherently discard some information. Could you elaborate on what types of task-relevant information might be lost? For example, how would the model distinguish between two nodes with feature vectors $x$ and $-x$, given that they would contribute identically to the expected covariance matrix $\\Pi_{c}XX^{T}\\Pi_{c}$?\n4. The proposed strategy for handling edge features (projecting, aggregating to nodes, then computing a new covariance operator $K_{edge}^{(p)}$) is not empirically ablated. Was this component used for pre-training on datasets like ZINC, which have edge attributes? How does the simple node-level aggregation avoid losing critical relational information, and can you provide an ablation study on this component's impact?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper correctly identifies input feature heterogeneity as a fundamental and persistent obstacle to cross-dataset generalization, and its focus on solving this core problem is a valuable contribution to the field.\n2. The proposed ALL-IN method is supported by theoretical proofs demonstrating that its representations are robust to feature re-ordering (distributional invariance) and basis changes (orthogonal invariance), ensuring it captures stable properties of the data.\n3. The model shows strong empirical results, not only maintaining competitive performance when jointly trained on diverse datasets but also, more importantly, successfully transferring to new, unseen datasets with entirely different features without retraining.", "weaknesses": "1. On the domain difference. The authors should explain how the domain-specific knowledge is modeled and preserved in ALL-IN. \n2. Another main concern is the computational complexity, as a general-purpose foundation model is usually trained on massive data at scale. Given that the covariance is expensive, complexity analysis and comparison against other GFMs or pre-training strategies are strongly encouraged.\n3. This paper lacks some recent advances on GFM, and it will be more interesting if the following papers are well discussed and compared.  \na) NeurIPS24: GFT: Graph Foundation Model with Transferable Tree Vocabulary  \nb) WWW25: SAMGPT: Text-free Graph Foundation Model for Multi-domain Pre-training and Cross-domain Adaptation  \nc) WWW25: Riemanngfm: Learning a graph foundation model from structural geometry   \nd) OpenGraph: Towards open graph foundation models.   \ne) ICML25: How Much Can Transfer? BRIDGE: Bounded Multi-Domain Graph Foundation Model with Generalization Guarantees   \nf) ICML25: AutoGFM: Automated Graph Foundation Model with Adaptive Architecture Customization  \nAlso, the authors are required to specify difference between the aforementioned papers to defend their novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dt30NvFwet", "reviewer": "Reviewer_8FFw", "summary": "The paper introduces ALL-IN, a method for cross-dataset transfer in graph learning that circumvents incompatible node feature spaces by projecting features into a shared random space and building representations from covariance-based statistics. The authors provide theory showing that the resulting node-covariance operators and representations are distributionally invariant to feature permutations, and that the expected operator is invariant to orthogonal transformations, which removes dependence on the original feature semantics, scales, and dimensionality. Practically, the approach requires no architecture changes or retraining and supports both node- and graph-level tasks on unseen datasets with new feature sets.", "questions": "1. What is the pretraining task and loss? Are node classification and graph classification used in pretraining? According to section 5.1, seems that the model is also trained to perform regression. What is the purpose of including regression tasks in the supervised pretraining if the model is not tested for regression during the downstream validation?\n2. Why are supervised tasks chosen for pretraining? Can the authors provide a minimal comparison between supervised and unsupervised pretraining on the proposed method to clarify the underlying principle of the choice of objective?\n3. Can the method support edge level tasks such as link prediction? If so, what changes are needed in the representation or the predictor?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Study the important problem of feature space heterogeneity for GFMs.\n2. Provide theoretical understanding of the proposed method.", "weaknesses": "1. The pretraining process of the proposed method is unclear. The manuscript does not clearly specify the pretraining objective, the supervision signals used, or how equation 8 aligns with the description in Section 5.1.\n2. The data split for pretraining v.s. finetuning is confusing. The pretraining set has feature dimension up to 28, while the three citation datasets used for finetuning have hundreds or thousands of features. Also, the three citation networks have much more nodes and edges than the pretraining graphs. The rationale for this mismatch is not provided, and the effect on transfer is not analyzed.\n3. In addition to bullet 2, the authors mention that the pretraining corpus is drawn from multiple domains, yet citation networks are not included despite being used for evaluation. This leads to some confusion about the choice of the pretraining datasets. I would appreciate if the authors can provide some discussion about the principle of dataset selection for pretraining.\n4. The number of the downstream datasets is limited in the main text. Results focus on citation and bioinformatics datasets, with little evidence from other domains represented in the pretraining corpus. I would love to see the experimental results from domains other than citation networks and bioinformatics.\n5. Table 12 reports results on heterophilic graphs, but the comparison lacks specialized supervised models that are designed for heterophily, which limits the strength of the claim on this setting.\n6. The authors provide the complexity analysis in Appendix C.10. Given the quadratic complexity, it would be better to provide a comparison of the training and inference latency with other GFMs, if applicable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u3D50VmeSi", "reviewer": "Reviewer_txCX", "summary": "The paper proposes ALL-IN, an input-agnostic framework for transferring graph models across datasets with *different node and edge feature spaces*. It enforces both distributional and orthogonal invariance, supported by theoretical and empirical results. Experimental results show that a single instance of ALL-IN, jointly pre-trained on multiple heterogeneous datasets, performs competitively with per-dataset specialists and generalizes well to unseen datasets.", "questions": "- Could the authors provide guidance or heuristics—possibly theory-backed—for selecting the projection dimension $h$ and number of propagation orders $k$?\n- How does ALL-IN perform under strong heterophily benchmarks? Are the covariance operators dominated by local homophily, and do the propagated operators $K^{(p)}$ help mitigate or worsen this effect?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Provides a clear and compelling principle for bridging feature spaces by operating on node–node covariance operators, which are independent of the original feature dimensionality. This method  integrate naturally into standard GNN layers.\n- Introduces an interesting approach to achieve permutation and orthogonal invariance, effectively addressing a key bottleneck in building transferable graph foundation models.\n- The method is simple and architecture-agnostic, making it easy to follow and utilize", "weaknesses": "- The use of  $n \\times n$ covariance operators incurs $\\mathcal{O}(n^2)$ memory and computation cost per operator. The paper could benefit from a more in-depth discussion of this limitation, ideally supported by experiments on larger-scale datasets, such as [OGB](https://ogb.stanford.edu/docs/nodeprop/).\n- The current transfer setup is skewed toward small, homophilous benchmarks (e.g., MUTAG/PROTEINS contain fewer than 1,000 samples). Evaluating the method on larger-scale benchmarks would help demonstrate its viability as a foundation model.\n-  While the method outlines an impact on the edge-feature processing pathway, it remains unclear whether this component was utilized in the experiments. Concrete results or ablation studies to support its effectiveness are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)