Out-of-Distribution Graph Models Merging
Authors: Yidi Wang
Ziyue Qiao
Jiawei Gu
Xubin Zheng
Pengyang Wang
pei Xiaobing
Xiao Luo
Venue: ICLR 2026 Poster (2026)
Keywords: Graph Models Merging
Source-Free Domain Generalization
Graph Neural Networks
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Graph Models Merging for Graph OOD

Abstract: This paper studies a novel problem of out-of-distribution graph models merging, which aims to construct a generalized model from multiple graph models pre-trained on different domains with distribution discrepancy. This problem is challenging because of the difficulty in learning domain-invariant knowledge implicitly in model parameters and consolidating expertise from potentially heterogeneous GNN backbones. In this work, we propose a graph generation strategy that instantiates the mixture distribution of multiple domains. Then, we merge and fine-tune the pre-trained graph models via a MoE module and a masking mechanism for generalized adaptation. Our framework is architecture-agnostic and can operate without any source/target domain data. Both theoretical analysis and experimental results demonstrate the effectiveness of our approach in addressing the model generalization problem.

Published as a conference paper at ICLR 2026
OUT-OF-DISTRIBUTION GRAPH MODELS MERGING
Yidi Wang" 1.
Ziyue Qiao
Jiawei Gu
Xubin Zheng'
Pengyang Wang"
Xiaobing Pei Xiao Luo
1 School of Computing and Information Technology, Great Bay University
"The State Key Laboratory of Internet of Things for Smart City, Department of Computer and Information Science, University of Macau
School of Software Engineering, Huazhong University of Science and Technology "Department of Statistics, University of Wisconsin- Maddison wang. yi di @ooonncntt .mm edu.mo ziyue @mail com
ABSTRACT
This paper studies novel problem of out-of-disrribution graph models merg- ing, which aims to construct a generalized model from multiple graph models pre-trained on different domains with distribution discrepancy. This problem is challenging because of the difficulty in learning domain-invarian knowledge encoded implicitly within model parameters and consolidating expertise from potentially heterogeneous GNN backbones. In this work, propose graph generation strategy that instantiates the mixture distribution of multiple domains. Then, merge and fine-tune the pre- pre-trained graph models via MoE mod- ule and a masking mechanism for generalized adaptation. Our framework is architecture-agnosii and can operate without any source/target domain data. Both theoretical analysis and experimental results demonstrate the effectiveness of our approach in addressing the model generalization problem. The code is available https::/giit com sir wttp:////i//ss////osroo...
INTRODUCTION
As the scale and complexity of observed graph data continue to increase, graph models have become essential tools for extracting insights from real-world scenarios (Zhu al. 2022 Li et al. 2022
Zhang et al.) 2024; Wu et al. (024a).. Recently, Graph Model Generalization (GMG) aims to transcend the limitations of multi-domain datasets with distribution shifts by identifying invariant
features (Arjovsky et al. 2019; Chang et al., 2020 Ahuja et al. 202T), causal relationships (Gui et al.
2024 Chen et (024a), or risk extrapolation (Xu al. 2020 Ziyin et al. 2020 Ye et al. [021]]
et al. 2024c) underlying the graph data distributions. The objective is to maintain robust performance
on unseen, cout-o-ddistrbuttion graphs.
Current research focuses on training generalized model from scratch using graph data from multiple domains with distribution discrepancy. However, a less explored yet practical scenario emerges when graph models have already been trained individually on these different domains-referred to as -itto---Dittiiutionn Graph Models. For instance, in social networks, models trained on user data from different groups or with varying architectures capture diverse behavior patterns. Achieving a unified and generalized model on these datasets usually needs training from scratch, which is complex and wasteful of their learned knowledge.
As presented in Figure T these models are designed for similar tasks but on different datasets, each preserving specialized knowledge. Figure Saaure@iiluutturr illusstate the performance of GNN models pre-trained on one domain and tested on both their own and other domains with distribution shifts (detailed setting is in Sec. 4.1). While models perform well in their domain, their performance degrades in others, and different GNN architectures may excel in different domains. These suggest that by merging these models' intrinsic invariance and complementary expertise, it possible address challenges arising from distribution shifts and achieve generalization on all domains, even without retraining from scratch on the original training datasets labels.
"Corresponding Author.

---

Published as a conference paper at ICLR 2026
Training Domain
Training Damain
Training Domain
A
B
B
Merge
Testing Domain
A
$1.00
$2.00
Testing Domain
15.06
Testing Domain
A
73.00
52.2
42.22
f.
Fex
Pro-train
Domain Generaltzaiio
40.00
C
6..33
Dismibattion Shijts
(a) GCN
(b) GAT
(c) GIN
Figure 2: Comparison of different GNN models' gener- alization performance on PTC between in-distribution and OOD scenarios, with three domains represented as A B C. Values indicate Acc (%). The results within the red dashed box represent the best performance.
Figure I: Illustration of Outtof-Distributtin Graph Models Merging.
Therefore, this paper investigates a novel and practical problem, named Out-of- -iiti---Ditiiittioonn Graph Models Merging: How to consolidate the knowledge of multiple pre-trained GNNs into unified model that generalizes under distribution shifts? Achieving this goal is non-trivial due to the follow- ing challenges: (1) Unlike conventional domain generalization approaches, learning the domain- invariant knowledge from the domain data explicitly, learning from the model parameters in our setting is inherently complicated. (2) Furthermore, the pre-trained models may differ in their architec- tures and hyperparameters, making it difficult to consolidate the expertise of these diverse models into a unified representation.
To address these challenges, we propose novel Out-of-distribution Graph Models Merging (OGMM) framework for domain generalization, which is depicted in Figure Specifically, we explore the theory of multi-domain generalization defining generalization risk in functional space and deriving two-stage objective function. The first stage domain knowledge generation process. We "invert" each pre-trained GNN (expert), to generate a small set of aabel-conditionl graphs starting from random noise. These generative graphs are then aggregated as the training data for the second stage. The second stage involves experts fine-tuning and merging. To effectively retain the source domain knowledge learned by models with different parameters and architectures, we employ Mixture- of-Experts (MoE) module for merging. Meanwhile, based on the mixture distribution assumption, we prove that the fine-tuned MoE with masks serves as an approximation of the generalization risk function. The lightweight sparse gating weights and the masked experts trained with the generative graphs, enabling the allocation logic of "sample-expert" pattern. The main contributions of OGMM are summarized follows:
We propose novel framework named nntoo--distribution graph models merging, which aims to learn a generalized model from multiple graph models pre-trained under domain shifts.
We propose a graph generator for concentrating the model knowledge effectively, and develop an innovative model merging function utilizing fine-tuned MoE to address adaptive integration of multiple pre-trained models, thereby enhancing generalizability to unseen graphs
We validate OGMM on various tasks, demonstrating substantial improvements on out-of- distribution data compared to both individual model and traditional model merging methods.
2 PROBLEM FORMULATION
Graph Neural Networks (GNNs). A graph is represented as G = {4, X), where A E R is the adjacency matrix and X B denotes the node features, with being the number of nodes in G. We consider a basic GNN consisting of two parts: (V, i...e. ʃ(0) 0g o O4 - y, where 0g is parameters in the graph encoder, 0g corresponds to the classifier parameters, and y is the graph-level (or node-level) label space in the downstream tasks. Specifically, Vi represents a multi-layer message aggregation function, where the update mechanism in the L-th layer can be written as follows:
LL+ = a (AGGR (h/ E (()))))
where hg. F,, and h/- is the output representation for node i. or is activation function. AGGR(.) defines the aggregation of nodes and their neighbors M. The classifier & will be trained to assign label for each graph (or node) from the label space y (Yi, Y2, .., Y.) with classes.
2

---

Published as a conference paper at ICLR 2026
Stage 1: Graph Generation
Pretrained GNNs
Generative Graphs
N(0,1)
Gumbel(0,
Generator
GNN fay
label conditionnl
GNN fe
foz
Ieput
Generate
X
A
Merge
Gating Layer
R gate
Trained
Trabi
Noise
Topk
Gates
Expert Loads
Merged GNN
A1,00
Expert
merrge
Distance
oogrut
Product
Train
R mask
Sum
Stage Models Merging
Encoder
Classifier
Distol
Hodamwrrd di Product
Figure 3: Architecture overview: The architecture of OGMM consists of two primary stages: (1) GNN serves as supervisor to train its corresponding generator,
Graph generation. Each which reconstructs labcl-conditional graphs from random noise. (2) Model merging. The generative graphs are aggregated to train a merged GNN using MoE module. It comprises a gating layer and set of fine-tuned masked experts. Gradient updates guided by mask and gating regularization terms alongside classification loss.
Out-of-distribution Generalization on Graphs. The objective of oittt---distibuttion Generalization (also known multi-domain generalization) is to leverage joint data samples from multiple source domains to capture cross-domain invariant knowledge (Crammer et al. 2008 Mansour et al. 2008). Here, present its formulation in the context of graph domains. Suppose
given M sets of
source data, denoted as {G, HeM, where G, = {G1, G2.... G, represents the i-th source dataset. Each G; maps to the label space V. Additionally, of N₂ graphs (N, = for node-level tasks), Gr = [G1, G2.... GN, }. which shares the same label space as the source data but follows different distributions. The goal is optimize a GNN model f()) with parameter θ from scratch to minimize the generalization error under the unseen shifts as:
provided with target dataset consisting
f(0*) = arg min Egr, [e(S(O) (91, (2,,92,-0 92.,., ..r,))))))), (art))rr;>))
Out-of-dissributte Graph Models Merging, Different from the conventional conditions of Out- of-distribution Generalization, Out-of-distribution Models Merging assumes that the task-specific GNNs {f(O,)) have already been trained on different datasets |Gr and aims to learn unified model utilizing the parameters of multiple pre- pre-trained models. The objective is to optimize multi-model merging function to obtain model with higher generalization capabilities. Under the proposed Gruph Models Merging setting, we define an objective function as follows:
r arg min Egr, (a, (01, ......, ...
where I' is the expected model merging funetion, is the combining weights, and E(-) is the loss function that measures the prediction error.
METHODOLOGY
In this section, we present a framework for out-of-distribution graph model merging that constructs a unified model from multiple pre-trained GNNs without access to original data. Building multi- domain generalization theory, we develop a two-stage approach that addresses the fundamental challenge of extracting and consolidating domain-invariant knowledge from model parameters.

---

Published as a conference paper at ICLR 2026
3.1 OVERALL FRAMEWORK
Here we justify Eq. 3 based on multi-domain cut-of-distributiio generalization theory, enabling out-of-distribution models merging. To begin, establish mixture distribution assumption for this problem, stating that the target distribution is a mixture of distributions from multiple sources. Assumption 3.1 (Mixture Distribution). The input the problem is the of M source distributions, denoted as {91, 92..... Gm J. The distribution of the target domain, Gr is assumed to be linear combination of the M source distributions: Gr =; o;9
This assumption is widely accepted in
generalization problems (Crammer et al. 2008 Mansour al. 2008), and leads to the rule of linear combination of functions, expressed as T= Ei a, (O,). Next, we provide the definition of HAAH-divergence define the symmetric difference in hypothesis space H.
Definition 3.2. [HA divvegggee) Let H be hypothesis class. f(O,), functions trained on distributions 9; and G,, respectively. We define the divergence between 9, and 9j in the function space:
are the
dHan(9, 9ÿ) 2 sup s [IS /(9,, Fawe, (I, f(9), G) (4)
By the linear assumption and the definition of divergence, we prove the generalization error bound of I on the target distribution GT the following theorem.
Theorem 3.3. If each f(O,) is an optimal learner trained on the marginal distribution Gú the upper bound of the generalization error for I'(.) on the target domain given by the sum of the cross-validation errors of these sub-learned across different distributions.
The proof is shown in Appendix A.I due to the page limit. To enhance the generalization capability of the I, The overall objective for the merging function can be formulated as:
I.I
introduce fine-tuning weights w' for f(O,) to decrease the cross velidaaiin error.
arg min Σ (Ca-, ((, G)) + Σ (((())))) + Σ (((( KK (5)
where c-(-) denotes the empirical eтror on Gi C(-) is loss function like cross-entropy. represents the minimum sum of errors achievable by the optimal hypothesis across all domains within our hypothesis class H. This value is determined by the design of H itself (like neural network architecture) and is independent of our optimization over r.
Then, we consider the expansion of c (f()))) as starting point for knowledge extraction from f(O,). Consequently, Optimization 5
can be reformulated as a two-stage objective function:
M
arg min
Ca (Di,
Za;s(O), G.)) [Sec. 3.3
0,,a9
(6)
N
s.t. 9 = arg min
Σ
Co, Ca,-a (a I(O;, G2)) Secc. 3.2
9.
where ji are the conditional labels sampled from the label space for samples on G.. 9- = M
a
is the mixture distribution generated from pre-trained GNNs, which will be introduced in Sec. 3.2 The N; represents the number of samples drawn from G,. We use these generative samples to fine-tune a and w in merging function I, which will be introduced in Sec. 3.3 The analysis details are provided in Appendix A.3 This theorem shows that under the mixture distribution assumption, the generalization ability of the merged GNN depends on three factors: the pre-training error of each model, the fine-tuning error these models on the new domains, and the training error of the merged model on the generated samples. Next, we will provide the detailed implementations of OGMM.
3.2 LABEL- CONDIICONDIIIONAL GRAPH GENERATION
In the first stage, we use pre-trained graph models to generate synthetic graphs for subsequent fine-tuning and merging. Instead of using the original graphs, we opt for generated graphs because:
4

---

Published as a conference paper at ICLR 2026
the original datasets may not always be accessible for every model, generating smaller set of graphs is more efficient than using the entire dataset, and the generated graphs may sometimes distill and refine knowledge more effectively, making them more representative than the original data. Still, our method is capable of utilizing the original data, and a comparison is provided in T
As defined by Optimization 6 the goal is to fix all parameters of the pre-trained GNN while optimizing the inputs to minimize the label-conditional posterior error. For graph data, a unique challenge arises due the inputs' composition of both node features X and graph structure A, with A often represented as a discrete variable. This discreteness hinders the direct application of inversion technique (Zaoorykoo & Komodakis 2016, Yin al.) (020). To address this, Deng & Zhang (2021) employs a discrete gradient approximation method tailored for optimizing A. "Zhuang et al. 2022) parameterizes X alone while constructing from the inner product space of X, thus
preserving feature similarity. Better methods like (Liu et al. 20222 Gao et al. (0244) Jin et al.) 2
use edge encoders to retain inherent relationships between node features and edges. Here, we propose discrete edge encoder to handle graph structures.
Specifically, for each pre-trained GNN f(O,), we construct a generator P; to produce label-conditional graphs that maximize s(O,)'s agreement. P, samples feature X E Ra from a standard normal distribution, representing n, generative nodes, as the initial input for every graph Gj For each X', P. samples label y, from a uniform distribution, serving the conditional posterior ground-truth. To generate A from X', we introduce an edge encoder defined as follows:
Graph Generator:
Ais = (MLPo X;)))
(7)
where MLP is a three-layer fully-connected neural network, an activation function, and [:-] denotes the concatenation operator. To enforce discrete edge weights, we assume edges follow a Bernoulli distribution and employ the Gumbel-Softmax to approximate values in [0, 1]:
(log( (A' 11x) + 1
Ajxx = softmax
(8)
T
where μ = - log(- log(e)) and e ~ Uniform(O. 1). Here, T denotes the temperature hyperparameter,
with - leading Ais toward a binary value. By feeding batches of generated samples (X', A', úi)
into the generator P., we can use the label-conditional posterior loss C(j,, f(O,, x',A')) to fit the source domain distribution, obtaining Gü-
Regularizers for Generation. In addition to the label-conditional posterior loss, we leverage priors stored in the batch normalization (BN) layers of the pre-trained models. Following (Deng & Zhang 202I), we enforce the mean and variance values of the generative graph embeddings to match those recorded in the BN layers of the GNNs. Common GNN models perform well with relatively few layers, and correspondingly have a limited number of BN layers (a 2-layer GCN or GAT model typically has only one BN layer while GIN has two). We utilize all BN layers from GNN models to calculate this regularization term:
Roo Σ [(pee(())) B[uz(X') x1] [1))(X*)) B[o2(x')) 1=11
where X denotes the intermediate representations of graph in the BN layers, while X' is the data memorized during training BN layers. JLL o7 denote the feature means and variances, respectively. obtained from the L-th BN layer.
Another regularization term is the model's confidence in classifying the generative graphs, which ensures that graphs are well-calibrated rather than remaining an ambiguous state. We define the confidence regularization as follows:
N Σ G.)log
Reont Eg:ng;
(10)
where 9 is the data generated by the j-th generator and N, is the number of samples. Consequently, the overall loss function for each generator is formulated as follows:
Cgnn Σ
C(i. G,)) - Ron + Reonf-
(11)
GGER
5

---

Published as a conference paper at ICLR 2026
With the parameterized X and 0 learned from the above loss, we can synthesize samples (graphs in graph-level tasks or nodes in node-level tasks) that well-represent the corresponding task data. This process ensures that each generative graph retains structural and feature integrity, without introducing the complexity of gradient approximation methods. Finally, we merge all generated samples to construct the dataset G - :y..,, ... 9is for training the model merging function.
3.3 MODELS PINE-TUNING AND MERGING
In the second stage, we need to find a solution for reusing and integrating heterogeneous GNN backhones. This solution should both fine-tune each pre-trained GNN (expert) to adapt to knowledge from multiple domains and be universally applicable to arbitrary model architectures. The objective function for this stage is rewritten according to Optimization 6
M
arg miim CF
Σ Ca,es; (it. Σ aj (e)((),,'
(12)
where N and M denote the numbers of samples and models. yJ is the generated label of the i-th sample, and aj is the fusion weight, combining different models for different samples, respectively. Indeed, Optimization 12 corresponds to an innovative fine-tuned MoE architecture with Gate Layer (a) and added masks (w). The module's capability is to fine-tune, filter and combine pre-trained experts on mixture distribution to reach a wider generalization plane overall.
Masked Experts. Inspired by mask tuning techniques (Ghanbarzadeh et al., 2023 Li al. 2024a). we aim to identify and re-weight the pre-trained parameters required by new tasks. Given parameters
8: (o...... 01)²² E O, of a trained GNN (1 is the size of subset in module (s)), the mask matrix w² can be optimized downstream-relltte neural pathway:
0: = 0 w"
(13)
where @ denotes Hadamard product, and / replaces 0 in each Masked Expert. According to Optimization r represents distribution-sensitvv function, while the role of w is fine-tune f(O,) to minimize (f((@,,")) )). In shallow networks such as 2-layer GNNs, the position where the mask is added becomes particularly critical. We hypothesize that the weights in the classification head are closely related to downstream tasks, making the model highly susceptible to learning domain- speeiif knowledge from high- digh-diiensional representations. Thus, fine-tuning the parameters of the classification head is reasonable and effective strategy, which is further validated by the experimental results provided in Sec. 4.2 and Appendix CC22
Sparse Gate in MoF. Note that we can directly replace in Optimization 12 with regular MoE Gate layer, which can be written as follows:
A, = a Σ (Gate(X'),
(14)
where a is an activation function, M denotes the number of models (or experts), and Gate(-) is employed to distribute samples to different models. H, and Hig are the outputs of MoE and the j-th pre-trained model, respectively, with respect to sample X For all the masked pre-trained GNNs, the sparse gating strategy is as follows:
Gate(G,)
(QQp k)),
(15)
Q(G.) = G.W, c. softplus(G.W..)
(16)
where G, G* is generated from pre-trained GNNs. TopK (-, k) is a selector to find the first largest (or smallest) members in the sequence. Wg and W,, in Eq. T6 are the learnable weights.
Wy E Rdx M processes clean sample features to get expert selection scores, while W,, E dR adds controlled Gaussian noise e ( 1) to prevent experts from collapsing and ensure load balancing.
Summarizing the above, the loss of Optimization
12 can be re-written as follows:
L
Σ G.Eg-
C(i,, Tw.,W... (G.),,
(17)

---

Published as a conference paper at ICLR 2026
where Tw.,,, (G.) EØ= Gate , is proposed model merging function.
Regularizers for Fine-Tuned MoE. Here we introduce two regularizers to constrain the optimization direction of gates and masks. Following the strategy in (Wang et al. 2024), we utilize an importance loss to prevent single-selection collapse:
Rgate = CV( Σ (Gate(G.)))?,
(18)
GLEg-
where (-) represents the coefficient of variation. This regularizer measures the degree of weight disparity in "sample-expent" pairings, encouraging uniform weight distribution and enforcing all experts to be "load-balancedd.. For masks added to the pre-trained GNNs, it is necessary to minimize changes to the frozen parameters while learning new knowledge to prevent "forgetting" old knowledge. So we design the other regularizer follows:
I wt Rmusk Σ Ca.es- (bc. f(0,, ,.)) Σ
|لii %o) -kilwsh-//T Woll
((
7p)) (19)
where To: 7/p E [0, 1] are two thresholds to control the effects of the masks in terms of their mean values and variances, respectively. |a>|| means the size of w², The first part in Eq. 19 is for learning new knowledge from G* and the second part for controlling the process of fine-tuning. The overall loss function for merging is formulated as follows:
Emerge
Σ C(ui, (G:)) + *Agate Ryate + Amask Rmaskk GLEG"
(20)
where +- {we, Wg g. ),), and Agate and Amask are balanced in Eq. R] To achieve better generalization due the wider plane of the mixed distribution going over the unseen graphs
Recall the objective
4 EXPERIMENTS
In this section, we mainly focus on the graph classification tasks on the widely-used real-world datasets which encompass both observed (training) and unobserved (testing) data. Supplementary experiments (on the large-scale datasets the node-level tasks) are provided in the Appendix C.44 Following common practice, we use the Accuracy (Acc) and Precision (Pre) on the OOD target dataset for measuring the generalization performance.
4.1 EXPERIMENT SETUP
Datasets. We evaluate our method on four datasets: MUTAG, PTC, REDDIT-B, and NCII, following the same configurations
in (Xu et al. 2018). To simulate realistic domain shift scenarios,
partition each dataset based on the edge-to-node ratio, following established domain partitioning
methods (Luo et al.. 2024a Zeng al. 2024 Luo et al. 2024b Wen et al. 2025 Wang et al. 2025). This strategy creates meaningful distributional differences between domains while maintaining the intrinsic properties of each dataset. Summary statisties of these datasets and detailed specifications of partitioning are provided in Appendix B.I In this paper, we distinguish between domains using the notation "A B T". Specifically, "A" represents dataset slices with lower edge density, "B" refers to slices with intermediate edge density values, and "T"" denotes the test set with higher edge density.
Baselines. First we pre-train models on each observable domain, resulting in multiple pre-trained models. Then, we perform graph models merging and evaluate the generalization performance on the unseen testing domain. We use three widely-adopted GNN architectures GCN (Kipt & Welling, 2016), GAT (Veličković et al. 2017), and GIN (Xu et al. 2018) as off-the-shelf models to be merged. Additionally, we use the form of (architecture-A B) to distinguish GNNs trained on different source domains. For example, GCN-A refers to GCN trained on source domain A. For ease of comparison, all GNNs used in our experiments are 2-layer networks with 32 feature dimensions. Since no known methods exist for merging GNN models with diverse architectures, design two baseline approaches for reference: Inverse-X X and Multi-GFKD We compare our method with seven source-free graph domain generalization methods, which be divided into three groups:
7

---

Published as a conference paper at ICLR 2026
Table Performance comparison across four datasets. The form (Architecture-A) B) indicates that this architecture is pre-trained on domain A/B. Highlighted are the top first, second results.
REDDIT-B
MUTAG
NCI1
Methods
fcc/%
Pre/Sit
Acc/%
Pre/Sit
Acc/%
Pre/Sit
Acc/
Pre/Skt
GCN-A
25.03:66.7
35.554332.38
48.97433.59
50.38:4..11
31.25:4.000
37.47431877
49.62466.3
57.85+3.97
GAT-A
24.21110:20
27.051334.1
48.1013.67
53.3018.56
26.8810094
16.22.11.79
49.9113.66
18.0611.17
GIN-A
22.4618.22
18.47420.35
47.9314.42
50.9817.43
28.9146602
27.81431.88
52.5811.98
60.46111.52
GCN-B
66.1013..99
60.64114.6
49.9012.85
49.77
32.0349.79
33.8661.33.3
61.614:3.22
62.4242..53
GAT-B
61.78426.86
26.367466777
19.38:44.16
49.8 10.88
27.8142..6
19.944226. 10
60.75
63.14
GIN-B
18.80118.56
56.74:17.01
50.124:5.59
55.86188.87
42.03111244
49.244126.51
65,02:1166
66.79.1.11
Avg PTM
52.4745.46
51.6648.220 29
50.20+ 195
51.73443.43
31,48422.88 88
32.42441..3
56.58422.1
41.1111113
Ens. Prub
33.65+-25.66
36.12+:39.06
50.17+2.558
56.64-+6.99
29.84+53.6
35.86+335.35
58.05444.36
62.65+1.51
Ens-HighConf
44.46129.00
45 15.86133570
1..1911.87
50.83:13.98
32.34:17.94
47.99133.41
56,44116.7
61.7313.07
Uni-Soup
43.26114.09
31.65&17.22
50.204:2.48
47.3816.09
37.00&12.03
17.97:112.17
48.73418.8
47.25
Groedy-Soup
47.35:18.89
50,70:29.2
50. 17:2.0
42.75:19.43
41,26622
13.91
38 10.43
28.67:11.81
Inverse-X
56.211:27.12
48 38.861330.58
50,4331..50
51.92:13:10
38.75:17.91
40.14:131.57
62.39:19.8
56.3517.32
Mult-GGKD
54.35211.40
37.96111.10
50.7741.3
44.43:1:1.39
44.361..112
29.74:10.37
47.57111.84
36.75:7.12
OGMM
76.98115.19
63.36.10.81
51.2111.774
57.39.16.11
45.62118.67
26.28122.70
66.84110.45
72.90.14.99
Table 2: Ablation study about different modules. Highlighted are the top first, second results.
REDDIT-B
PTC
MUTAG
NC11
Variants
Acc/S
Pre/SE
Acc/St
Pre/S6
Acc/%
Pre/%
Acch%
Pre/#
Given
wlo Mask
43.95:+26.06
72.34:22.14
49,31=2.77
64.4726.97
28. 28:12:1210
33,52:33.46
48.36:33.30
61,78=2246
Source
OGMM
80.98=11.30
78.33-2291
54.31-2.70
59.16=5.04
57.81=7.30
68.79#3.13
68,0442.113
1.19
wlo MoE
50.3915.22
45.01=1.71
50,56.00.4
51.0112.69
39.53.1..0
23.2910.81
60.62.10.22
66.6812.72
Source
wla Mask
18.97
38.99
50.95-2.00
55.36+6.15
28.28+1147
49.39+43.98
51.11=1.066
59.70=0098
wlo Cyre
41.15:22695
39:75=:36:00
48.88-5.23
48.61:5663
45.31
25.81 22.
52.69:11.66
55.82::3446
OGMM
76.98:55.19
63.36=081
51.21-3.74
57.39:6.71
45.62=18.67
56.28:+2670
66.84:00.45
72.90=44599
Ensemble learning methods include averaging the performance of the models (Avg-PTMs), averag- ing the output probabilities of the models (Ens-Prob), and selecting the prediction from the most confident model, defined as the one with the lowest entropy (Ens- HighhH..g....)..
Model merging methods, include computing the clement-wise mean of all models Uni--Supp) (Choshen et al. 2022) and the selective merging approach (Greedy-Soup) (Wortsman et al. 2022).
Generative methods include Inverse-X. X and Multi-GFKD. Inverse-X is a baseline variant of OGMM that uses random graph structures instead of our parameterized edge encoder. Multi-GFKD is an extension of GFKD (Deng & Zhang 2021 multi-teacher distillation.
4.2 EXPERIMENTAL RESULTS
The comparisons of different models under the split-dataset scenarios are shown in Table [ OGMM consistently outperforms individual pre-trained models across all datasets, demon- strating the MoE module's 's effectiveness in capturing distribution shifts and accurately allocating "sample-expert" pairs. Ensemble methods like Avg-PTMs, Ens-Prob, and Ens- Ens-HiihConf show simi- lar precision, suggesting that leveraging multiple models can improve generalization. In contrast, parameter merging methods (Uni-Soup, Greedy-Soup) perform poorly, highlighting that integrating model outputs is more effective for OOD problems. Compared other source-free methods, OGMM sets a new -tate-of-the-art, achieving superior performance across datasets, especially on larger datasets like REDDIT-B and NCII. While data generation-bait based methods (Inverse-X, Multi-GFKD) outperform fusion approaches, OGMM surpasses both, offering significant improvements. Unlike Inverse-X, which only learns node features, OGMM simultaneously learns node features and graph structures, enabling better recovery of aamain-specifie knowledge. Additionally, OGMM preserves more source-domain knowledge, maintaining the diversity of observable distributions.
Main Results.
Analysis of Masks. We apply masks to two parameter groups, MaskCL and MaskNN, across three GNN architectures to analyze mask placement impact. MaskCL applies masks to classifier parameters (0x) while freezing others; MaskNN applies masks to encoder parameters (0z). As shown in Figure 4 models fine-tuned exclusively on classifier parameters achieve competitive performance across datasets. The mask size accounts for only 20% of total parameters in 2-layer GNN on average. This suggests comain-specifii knowledge is concentrated in classifier parameters, making classifier fine- tuning more efficient. See Appendix C. for more results other datasets. Additionally, we analyze
8

---

Published as a conference paper at ICLR 2026
T on RED.
B RED.
(c)
MUT.
(d)
MUT.
Figure 4: Impact of Mask Position on REDDIT-B (RED.) and MUTAG (MUT.). The form (A + T) means that GNN pre-trained on domain A and fine-tuned on the Target domain. The bar chart shows the model performance on the target domain, and the dashed line represents the average performance of masked models with different mask positions on this dataset.
Acc
Pre
Acc
Pre
Acc
Pre
ACc
Pro
301
(a) REDDIT-B (b) PTC
(c) MUTAG
(d) NCI
Figure 5: The Effects of k in Topk Expert Selection on four datasets.
parameter changes after continuous finettunig across multiple domains. Results demonstrate that classifier parameters exhibit stabilizing characteristics after multiple fine-tuning rounds, providing evidence for our mask mechanism's 's effectiveness. The comprehensive parameter evolution analysis and associated visualizations detailed in Appendix C.2
Ablation Studies. To evaluate the efficacy of OGMM's components, we conduct an ablation study comparing five variant configurations, with comprehensive details provided in Appendix [B3 and quantitative results presented in Table 2 The variant "OGMM (under Given Source condition)", which leverages access source domain data and incorporates additional trainable parameters, demonstrates superior performance as expected. The variant "w/o Mask (under Given Source condition)" only optimizes merging weights with fixed pre-traun pre-trained parameters, performing similarly to Ens-Prob / Ens-HighConf from Table I Notably, our proposed OGMM achieves optimal results in the source-free setting, approaching the best performance despite the absence of source domain data, thus validating its capability for effective cross dross-domain knowledge transfer. Removing the MoE module, generator, or masks under the source-free constraint leads to performance declines, underscoring the critical contributions of these components.
Impact of T'opK Expert Selection. To investigate the effects of the hyper-parameter in the TopK selector, evaluate results across four datasets as shown in Figure []] The performance changes reveal that selecting between 2 and generally yields optimal results across all datasets. Most datasets exhibit similar trends, with accuracy improving k increases initially and then stabilizing at higher values. Notably, OGMM consistently outperforms the pre-trained baseline across most settings, confirming the effectiveness of our MoE module. In addition, these results show that the optimal choice requires dataset-specific tuning to accommodate varying dataset characteristics.
Impact of the Number of Synthetic Samples. Theoretically, OGMM can generate unlimited synthetic graphs for training, but their quality and diversity are limited by the pre- rre-trained mod- els, as noted by (Deng Zhang 2021). 1). Figure @ssows the relationship between OGMM's per- formance and the number generated graphs on REDDIT-B and NCII. OGMM achieves high performance even with a small fraction of syn- thetic graphs, these effectively capture high- order domain knowledge, resulting in a concen- trated and informative distribution.
Accuncc
Accrnicy
122
REDDITER
NCTI
Figure 6: Impact of the number of generative graphs. The horizontal axis is the ratio of gen- erative samples to total source domain data.

---

Published as a conference paper at ICLR 2026
Manifold Visualization for Synthetic Graphs. To further validate the effectiveness and diversity of the synthesized graphs, we visualize the data using t-SNE in Figure 7 The results present the class-wise distribution of real and synthetic data specifically reveals that synthetic data aligns with the distribution of real data, indicating that our method successfully extracts dopain- knowledge embedded in the pre-trained models.
Domain A in NCII. The visualization
Boalatte Lond
Jotni Lakel
Resl Lotta Lakel
Fyrk Load
(a) Synthesis by GCN-A / B
(b) Synthesis by GAT-A B
(c) Synthesis by GIN-A /
Figure The t-SNE Visualization of Real Synthetic Samples with Label Distribution on Domain A.
Analysis of Gate Distribution in OGMM. To validate the rationality of our MoE-based merging design, we analyze the gating mechanism's ' behavior on both synthetic training data and real target domain data across four datasets, as shown in Figure We compute the total weight assigned to each expert by aggregating gate assignments across all samples.
During the training phase with synthetic data (gray bars), the load distribution across experts remains relatively balanced, indicating that each expert receives approximately equal amounts of fine-tuning data. This balanced training ensures all experts are adequately optimized without bias toward any particular source domain. In contrast, during inference on real target domain data (purple bars), the gate distribution becomes significantly more discriminative. The gating mechanism effectively captures the distinguishable patterns learned by different experts and adaptively routes test samples to the most suitable expert based on distributional similarity. This shift in distribution reflects the varying relevance of different source domains to the target domain.
be Salis Sapatrra
awd mitiaal Tapit
320
10
(a) REDDIT-B
(b) PTC
(c) MUTAG
(d)
NCII
Figure 8: Gates distribution training data and real target data on four datasets.
CONCLUSION
This paper investigates the problem of Out-of- Distr--Dittiittiionn Graph Models Merging. The primary challenge lies extracting knowledge from pre-trained GNNs and guiding their reuse to address the issue of model generalization. To tackle this challenge, we propose novel out-of-distribution graph models merging framework. Our approach leverages graph generation and fine-tume muned MoE to adaptively optimize the model fusion process, enabling effective generalization under graph OOD scenarios. Extensive experiments on several real-world benchmarks confirm that the proposed approach outperforms state-of-the-ar baselines.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
The work of Ziyue Qiao was partially supported by the Guangdong Basic and Applied Basic Research Foundation (No. 2024A15151400 14) and the National Natural Science Foundation of China (No. 62406056). The work of Pengyang Wang was funded by the Science and Technology Development Fund (FDCT), Macau SAR (File No. 001/2024/SKL).
REFERENCES
Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, ChaisShritt Gagnon-Audet, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of- distribution generalization. Advances in Neural Information Processing Systems, 34:3438-344 2021.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint 117700900..02933 2019.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79:151-175, 2010.
Jinyu Cai, Yunhe Zhang, Pengyang Wang, and See-Kiong Ng. Moegad: mixture-of-experts framework with pseudo-anomaly generation for graph-level anomaly detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
Shiyu Chang, Yang Zhang, Mo Yu, and Tommi Jaakkola. Invariant rationalization. In International Conference Machine Leurning, pp. 1448-1458. PMLR, 2020.
Yongqiang Chen, Yatao Bian, Kaiwen Zhou, Binghui Xie, Bo Han, and James Cheng. Does invariant graph learning via environment augmentation learn invariance? Advances in Neural Information Processing Systems, 36, 2024a.
Zhengyu Chen, Teng Xiao, Kun Kuang, Zheqi Lv, Min Zhang, Jinluan Yang, Chengqiang Lu, Hongxia Yang, and Fei Wu. Learning to reweight for generalizable graph neural network. In Proceedings of the AAAI conference artificial intelligence, volume 38, pp. 8320-8328 8328, 2024b.
Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. Fusing finetuned models for better pretraining. arXiv preprint arXiv:2204.03: 0000440..444.... 2022.
Koby Crammer, Michael Kearns, and Jennifer Wortman. Learning from multiple sources. Journal of machine learning research, 9(8), 2008.
Róbert Csordás, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Are neural nets modular? inspecting functional modularity through differentiable weight masks. arXiv preprint arXiv:2010.02066. 2020.
Xiang Deng and Zhongfei Zhang. Graph-free knowledge distillation for graph neural networks. arXiv preprint 20Xiv:20050075:: 2021.
Jiahua Dong, Zhen Fang, Anjin Liu, Gan Sun, and Tongliang Liu. Confident anchor-induced multi- source free domain adaptation. Advances in neural information processing systems, 34:2848-2860, 2021.
Shaohua Fan, Xiao Wang, Chuan Shi, Peng Cui, and Bai Wang. Generalizing graph neural networks on diitrib-iitiiiioonnnn graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Dongyi Fu, Liri Fang, Zihao Li, Hanghang Tong, Vetle I Torvik, and Jingrui He. What do llms need to understand graphs: A survey of parametric representation of graphs. arXiv preprint arXiv:2410. /2/0/.2/26.22 2024.
Xinyi Gao, Tong Chen, Yilong Zang. Wentao Zhang, Quoe Viet Hung Nguyen, Kai Zheng, and Hongzhi Yin. Graph condensation for inductive node representation learning. In 2024 IEEE 40th International Conference Data Engineering (ICDE), pp. 3056-3069. IEFE, 2024.
11

---

Published as a conference paper at ICLR 2026
Somayeh Ghanbarzadeh, Hamid Palangi, Yan Huang, Radames Cruz Moreno, and Hamed Khanpour. Improving pre-trained language models' generalization. arXiv preprint arXiv:2307.1044 2023.
Shurui Gui, Meng Liu, Xiner Li, Youzhi Luo, and Shuiwang Ji. Joint learning of label and environment causal independence for graph out-of-distribution generalization. Advances in Neural Information Processing Systems, 36, 2024.
Xu Owen He. Mixture of a million experts. arXiv preprint arXiv:2407.04153 2024.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning graphs. Advances in neural information processing systems, 33:22118 18-13133, 2020.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint MrXiv:2212.04089 2022.
Tianrui Jia, Haoyang Li, Cheng Yang, Tao Tao, and Chuan Shi. Graph invariant learning with subgraph co-mixup for out-of-distribution generalization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 8562-85575 8570, 2024.
Wei Jin, Lingxiao Zhao, Shichang Zhang, Yozen Liu, Jiliang Tang, and Neil Shah. Graph condensation for graph neural networks. arXiv preprint arXiv: 10.07580, 2021.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv: www.66699992000 2016.
Haoling Li, Xin Zhang, Xiao Liu, Yeyun Gong, Yifan Wang, Yujiu Yang, Qi Chen, and Peng Cheng. Gradient-mask tuning elevates the upper limits of Ilm performance. arXiv preprint arXiv:2406. 15330, 2024a.
Haoyang Li, Xin Wang, Ziwei Zhang, and Wenwu Zhu. Out-of-distribttion generalization on graphs: A survey. arXiv preprint arXiv: 222hv:202..09887 2022.
Xiner Li, Shurui Gui, Youzhi Luo, and Shuiwang Ji. Graph structure and feature extrapolation for out-of-distribution generalization. arXiv preprint arXiv:2306.0800 08076, 2023.
Xinyao Li, Jingjing Li, Fengling Li, Lei Zhu, and Ke Lu. Agile multi-source-free domain adaptation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 11673-1368 2024b.
Yijiang Li. Sucheng Ren, DENG Weipeng, Yuzhi Xu, Edith CH Ngai, Ying Gao, and Haohan Wang. Beyond finite data: Towards data-free out-of-distributin generalization via extrapolation. 2024c.
Tao Lin, Sebastian U Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. arXiv preprint (xXi:22006.07253, 2020.
Jingzhe Liu, Haitao Mao, Zhikai Chen, Wenqi Fan, Mingxuan Ju, Tong Zhao, Neil Shah, and Jiliang Tang. One model for graph: A new perspective for pretraining with cross-domain graphs. arXiv preprint arXiv:2412.00315, 2024.
Lei Liu, Xingyu Xia, Qianqian Xie, Ben Liu, Wenjie Xu, and Min Peng. Enhanced expert merging for mixture-of-experr in graph foundation models. In The Thirty-ninth Annual Conference Neural Information Processing Systems, 2025.
Mengyang Liu, Shanchuan Li, Xinshi Chen, and Le Song. Graph condensation via receptive field distribution matching. arXiv preprint arXiv:2206. /3697 2022.
Zheyuan Liu, Chunhui Zhang, Yijun Tian, Erchi Zhang, Chao Huang, Yanfang Ye. and Chuxu Zhang. Fair graph representation learning via diverse mixture-of-ee -xxeerrss... In Proceedings of the ACM Web Conference 2023, pp. 28-38, 2023.
Bin Lu, Ze Zhao, Xiaoying Gan, Shiyu Liang, Luoyi Fu, Xinbing Wang, and Chenghu Zhou. Graph out-of- df-tiiibbbuutionn generalization with controllable data augmentation. IEEE Transactions on Knowledge and Data Engineering, 2024.
12

---

Published as a conference paper at ICLR 2026
Junyu Luo, Yiyang Gu, Xiao Luo, Wei Ju, Zhiping Xiao, Yusheng Zhao, Jingyang Yuan, and Ming Zhang. GALA: Graph Diffusion- Based Alignment With Jigsaw for Source-Free Domain Adaptation IEEE Transactions Pattern Analysis & Machine Intelligence, 46(12):9038900)) December 2024a. ISSN 1939-3539.
doi: 10.1 1009/1PAMM.222433167777777
Junyu Luo, Zhiping Xiao, Yifan Wang. Xiao Luo, Jingyang Yuan. Wei Ju, Langechuan Liu, and Ming Zhang. Rank and align: Towards effective source-free graph domain adaptation. arXiv preprint arXiv:2408. / 12/85, 2024b.
Li Ma, Haoyu Han, Juanhui Li, Harry Shomer, Hui Liu, Xiaofeng Gao, and Jiliang Tang. Mixture of link predictors graphs. Advances in Neural Information Processing Systems, 17:16043-16070, 2024.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation with multiple sources. Advances in neural information processing systems, 21, 2008.
Michael S Matena and Colin A Raffel. Merging models with weight- averaging. Advances in Neural Information Processing Systems, 35:17703-17716, 2022.
Aldo Pareja, Giacomo Domeniconi, Jie Chen, Tengfei Ma, Toyotaro Suzumura, Hiroki Kanezashi, Tim Kaler, Tao Schardl, and Charles Leiserson. Evolvegen: Evolving graph convolutional networks for dynamic graphs. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 5115-3333777 2020.
Ziyue Qiao, Xiao Luo, Meng Xiao, Hao Dong, Yuanchun Zhou, and Hui Xiong. Semi-supervised domain adaptation in graph transfer learning. In Proceedings of the Thirty- International Joint Conference on Artificial Intelligence, pp. 2279-2287, 2023.
Ziyue Qiao, Junren Xiao, Qinggiang Sun, Meng Xiao, Xiao Luo, and Hui Xiong. Towards continuous reuse of graph models via holistic memory diversification. In The Thirteenth International Conference on Learning Representations, 2025.
Evani Rixita-Dixit and Xin Wang. How fine can fine-tuning be? learning efficient language models. In International Conference on Artificial Intelligence and Statistics, pp. 2435-2443, PMLR, 2020.
Benedek Rozemberezki, Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding. Journal of Complex Networks, 9())cnab014, 2021.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596-4604, PMLR, 2018.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-expere layer. arXiv preprint arXiv: 70656888,..,.. 2017.
Maohao Shen, Yuheng Bu, and Gregory W Wornell. On balancing bias and variance in unsupervised multi-source-free domain adaptation. In International conference on machine learning, pp. 30976- 30991. PMLR, 2023.
George Stoica, Daniel Bolya, Jakob Bjomer, Pratik Ramesh, Taylor Hearn, and Judy Hoffman. Zipit! merging models from different tasks without training. arXiv preprint 230.5:330000005. 2023.
Yongduo Sui, Xiang Wang, Jiancan Wu, Min Lin, Xiangnan He, and Tat-Seng Chua. Causal attention for interpretable and generalizable graph classification. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1696-1777 1705, 2022.
Xin Sun, Liang Wang, Qiang Liu, Shu Wu, Zilei Wang, and Liang Wang. Dive: subgraph disagreement for graph on Knowledge Discovery and Data Mining, pp. 2805, 2024.
generalization. In Proceedings of the 30th ACM SIGKDD Conference
Amanda L Traud, Peter J Mucha, and Mason A Porter. Social structure of facebook networks. Physica A: Statistical Mechanics and its Applications, (16444461655-1188 2012.
13

---

Published as a conference paper at ICLR 2026
Joachim Utans. Weight averaging for neural networks and local resampling schemes. In Proc. AAAI-96 Workshop on Integrating Multiple Learned Models. AAAI Press, pp. 133-138. Citeseer, 1996.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint 1rxiv:/710010003 2017.
Haotao Wang, Ziyu Jiang, Yuning You, Yan Han, Gaowen Liu, Jayanth Srinivasa, Ramana Kompella, Zhangyang Wang, et al. Graph mixture of experts: Learning on large-scale graphs with explicit diversity modeling. Advances in Neural Information Processing Systems, 36, 2024.
Qiu-Feng Wang, Xin Geng, Shu-Xia Lin, Shi-Yu Xia, Lei Qi, and Ning Xu. Learngene: From open- world to your learning task. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8557-8565. 2022.
Yingxu Wang, Mengzhu Wang, Zhichao Huang, Suyu Liu, and Nan Yin. Nested graph pseudo-label refinement for noisy label domain adaptation learning. arXiv preprint arXiv:2508.007/66 2025.
Tao Wen, Elynn Chen, Yuzhou Chen, and Qi Lei. Bridging domain adaptation and graph neural net- works: A tensor-based framework for effective label propagation. arXiv preprint 0rX5v:2502.08505, 2025.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roclofs, Raphael Gontijo-Lpess, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Korbblith, al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pp. 23965-23998, PMLR, 2022.
Man Wu, Xin Zheng, Qin Zhang, Xiao Shen, Xiong Luo, Xingquan Zhu, and Shirui Pan. Graph learn- ing under distribution shifts: A comprehensive survey on domain adaptation, out-of-distribution, and continual learning. arXiv preprint arXiv:2402. 16374, 2024a.
Shirley Wu, Kaidi Cao, Bruno Ribeiro, James Zou, and Jure Leskovec. Graphmetro: Mitigating complex graph distribution shifts via mixture of aligned experts. Advances in Neural Information Processing Systems, 37:9358-9387. 2024b.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint /
2018.
Keyulu Xu, Mozhi Zhang, Jingling Li, Simon S Du, Ken-ichi Kawarabayashi, and Stefanie Jegelka. How neural networks extrapolate: From feedforward to graph neural networks. arXiv preprint arXiv:2009.11845 2020.
Prateek Yadav, Colin Raffel, Mohammed Mugeeth, Lucas Caccia, Haokun Liu, Tianlong Chen, Mohit Bansal, Leshem Choshen, and Alessandro Sordoni. A survey on model moerging: Recycling and routing among specialized experts for collaborative learning. arXiv preprint arXiv:2408.07057, 2024.
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. arXiv preprint drX.v:2310.0255. 2023.
Haotian Ye, Chuanlong Xie, Tianle Cai, Ruichen Li, Zhenguo Li, and Liwei Wang. Towards theoretical framework of distt--iiittitttion generalization. Advances in Neural Information Processing Systems. 34:23519-22 23531, 2021.
Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 87 15- 8724, 2020.
Haonan Yuan, Qingyun Sun, Xingcheng Fu, Ziwei Zhang, Cheng Ji, Hao Peng, and Jianxin Li. Environment-awwan dynamic graph learning for -ittrf-distribution generalization. Advances in Neural Information Processing Systems, 36, 2024.
14

---

Published as a conference paper at ICLR 2026
Sergey Zagoruyko and Nikos Komodakis. Paying attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint arXiv: 1612.0398, 2016.
Hanqing Zeng, Hanjia Lyu, Diyi Hu, Yinglong Xia, and Jiebo Luo. Mixture of weak & strong experts on graphs. arXiv preprint arXiv:2311.05185 2023.
Zhuo Zeng, Jianyu Xie, Zhijie Yang, Tengfei Ma, and Duanbing Chen. To-ugda: target-oriented unsupervised graph domain adaptation. Scientific Reports, 14(1):9165, 2024.
Kexin Zhang, Shuhan Liu, Song Wang. Weili Shi, Chen Chen, Pan Li, Sheng Li, Jundong Li, and Kaize Ding. A survey of deep graph learning under distribution shifts: from graph out-of- distribution generalization to adaptation. arXiv preprint Stti::2210,192666, 2024.
Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schütze. Masking an efficient alternative to finetuning for pretrained language models. arXiv preprint arXiv:2004. 12406, 2020.
Hongling Zheng, Li Shen, Anke Tang, Yong Luo, Han Hu, Bo Du, and Dacheng Tao. Learn from model beyond fine-tuning: A survey. arXiv preprint arXiv: 10.08/ 84, 2023a.
Kecheng Zheng, Wei Wu, Ruili Feng, Kai Zhu, Jiawei Liu, Deli Zhao, Zheng-Jun Junng-Junnne Zha, Wei Chen, and Yujun Shen. Regularized mask tuning: Uncovering hidden knowledge in pre-trained vision- language models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1666311667,, 2023b.
Yanqiao Zhu, Yuanqi Du, Yinkai Wang, Yichen Xu, Jieyu Zhang, Qiang Liu, and Shu Wu. A survey on deep graph generation: Methods and applications. In Learning on Graphs Conference, pp. 47-1. PMLR, 2022.
Yuanxin Zhuang, Lingjuan Lyu, Chuan Shi, Carl Yang, and Lichao Sun. Data-free adversarial knowledge distillation for graph neural networks. arXiv preprint arXiv:2205.03811 2022.
Liu Ziyin, Tilman Hartwig, and Masahito Ueda. Neural networks fail to learn periodic functions and how to fix it. Advances in Neural Information Processing Systems, 33:1583-1553 2020.
15

---

Published as a conference paper at ICLR 2026
This appendix contains details about mathematical proofs, experimental implementation, supplemen- tary experiments, related works, limitations, and future works.
A
PROOF
A.1 PROOF OF THEOREM 3.3
Proof. Given a domain Gj with two trained classifiers f(O;) and f(0)). These classifiers may have been trained on different source domains or under different conditions, leading potentially divergent prediction behaviors. Based on (Ben-David et al. (0110), we can define the probability according to the distribution 9; that f(O,) disagrees with f(O,):
E, (..) = Ec-a, lIf G) - f(0,, G)[l-
(21)
the classifier f(O;)) is good learner trained on Gi meaning it has achieved low training error and captures the underlying patterns of domain 9; effectively. We will find the generalization enror of f(0,) over G;:
&(((O)), Fo.-, 1 /(0;, G)I, s.ts(O,, =y.
(22)
Following Eq.
21 Definition 3.2 can be formalized as follows:
dHaM(9,, 9j) = sup ((,))()))
E((((O.). (()1)))))))))))((()))))()0))))1
(23)
Substituting Eq. 22 into Eq. 23 yields:
(HAn(Gr,9)) 2 sup
(E, [(j (,)(())),)))
o sup ((0)))(())EE s.t. f(O,, 9r. f()), 9j) = ūj-
log f(O))) log p p(, ((0)))
(24)
which implies that the H of G, and Gj depends the cross-validation results of the respective optimized classifiers. Note that the disparity difference function represented by Eq. [4] symmetric and obeys the triangle incquality. So build cross-domain objective function based on a set of pre-trained models:
M dHAH (9)
arg min wig
JJ
(25)
x arg min
log logr( - 10))
G
i.j
s.t. G: = arg max mox 4G, eg,
where w is learnable parameter for f(O.). Optimization 25 achieves two purposes: (1) sufficient extraction of knowledge from the models to compose more generalized mixture of distributions of the data G. and (2) optimization of the added parameters fine-tune the individual model on the mixture of distributions. Details on solving the Optimization 25 can be found in A.2]
Due to the rule of linear combination, r= Σ." ((,)), we have I' E H. Thus, the optimization objective of Optimization 25 is to identify an appropriate discriminative function that minimizes the generalization enror across arbitrary marginal distributions. Therefore, the upper bound of the generalization error for on the target domain (the mixture distribution according to Assumption 3.1 depends the sum of the cross-validation errors of sub-learners.
16

---

Published as a conference paper at ICLR 2026
A.2 DETAILS ON SOLVING OPTIMIZATION 25
Optimization 25 is non-convex problem, which is hard to solve. We relax it via the triangle inequality. At the same time, replace the log-likelihod function with regular cross-entropy loss, and finally get two-stage target function:
arg min wit"
Ccees (0c, ((),, G)) Sec 3.3
f
(26)
S.t.g. = arg min Caea f(O,, G)), Vg; E E [Sec. 3.2
where (-Ea(-) denotes the cross-entropy loss function on distribution G. O E G is a batch of generated samples. The ideal is for any fine-tuned model have small posteriori error on any sampled data belonging to 9", which is very difficult to achieve. In practice, we simply approximate it using finite number of samples. Meanwhile, substitute r Σx f(,( into Optimization 26
M
min Σ Ca.eg- (Dc. Ea, f(0,, G.)), G,)),
(27)
where N and M denote the number for samples and models. di is the label of G,, and a is the fusion weights, combining different models for different samples.
According to Theorem 4 in (Ben- David et al., 2010), for any 5 E (0, 1), with probability at least (1 8), the error bound of the merged function on the target domain GT can be defined as follows:
M
(T(T) & (T()) + Σ
ar (22g + dHax(O,,
1=1
(28)
M
2dlog(2(N 1)) log (1)
+
N
j=1
where H is a hypothesis space of VC dimension d. ht min 6r (h) is the target error minimizer. N₁ N represents the sum of the number of all samples in all source domains, and 3 y is the ratio of the samples from the j-th domain. a is a fixed weight vector. Aラ = min (er((rr))) + c,())) means the optimal cross-domain generalization error (defined in H), and this term corresponds to expectations for the fine-tuned pre-trained models.
A.3 PROOF OF THE EXPANSION OF e; ((()))))
Proof. First provide the definition of H H-Divergence between G. and Gy: dx(9; 9j) = 2 sup
f G) f(,),, G)1,
(29)
(e,)e,
where Pra-e, (OO, G) means the prediction of ['()) on G;. Suppose that G is the mixed distribution of the { f and G can be defined as follows:
G' G'= Σ a.G. -a.
(30)
where o; is the mixing coefficient. Note that when no training data is available but model parameters are known, we can optimize the inputs by minimizing the empirical error <e(J(O:)) on Ge (i.e., argmazg; log p( &. ff to generate data (Deng & Zhang 2021). This process can be seen as narrowing the H-Divergence between G and 93 according to Eq. 29 When Pra-e, (O,, G) is close enough to úJ (?().) fits well enough on G.), we can assume that 9 samples from g", which still has the AH--iveegencc:
dHAn(97,9) 92)
sup E9 h1) E,(,,
X sup ha EH
p(g 9r, h2) log püg (ū) h:)l
(31)
G arg arg p(p G. f(O.), vg; g
17

---

Published as a conference paper at ICLR 2026
where (h2, h2) is set of functions used to define the lower bound of dHaN (92 93). We the fine-tunable model f(O,) and f(O,) to approximate h; and hy with added parameters -
B IMPLEMENTATION DETAILS
B.1 DATASETS DETAILS
Table shows the summary statistics of the datasets used in Sec.
4.2
Table 3: Summary of datasets.
MUTAG
PTC
REDDIT-B
NC11
#Graphs
344
2,000
4,110
#Classes
2
2
2
#Feature Dim
7
19
37
1
#Nodes
3,371
8,792
859,254
122,747
#Edges
7,442
17,862
1,991,016
265,506
Avg #NNodes
17.93
14.29
429.62
29.87
Avg #Edges
39.59
51.92
995.51
64.60
To simulate realistic out-of-distribution scenarios, partition each graph-level dataset into multiple domains based on graph edge density, following established domain adaptation methods (Luo et al.) 2024 Luo et al. 2024b] Wen et al. 2025) Wang al. (025). Edge density serves as a fundamental structural characteristic that creates meaningful distributional shifts across graph domains (Fu et al.
2024a Zeng et al.
2024). For each graph-level dataset, we calculate the edge density p = M
2|
for every graph and partition the data into domains based on density value, ensuring that each domain contains graphs with similar structural complexity while maintaining sufficient distributional differences between domains. The complete partitioning implementation is available in code repository. This partitioning strategy reflects real-world scenarios where models encounter graph structural variations, such as in molecular datasets like MUTAG, where density variations correspond to distinct chemical families: dense graphs typically represent highly conjugated aromatic systems with extensive double bond networks, while sparse graphs correspond to simpler aliphatic structures that exhibit different toxicity mechanisms and create natural domain boundaries.
Similar density-driven domain shifts appear across various graph learning applications, from social networks where active users generate dense interaction patterns while inactive users create sparse connectivity, to spatiotemporal trajectory analysis where dense urban movement patterns differ significantly from sparse rural trajectories. These distribution shifts lead to model performance degradation, which is precisely the generalization challenge our method aims to address.
B.2 PARAMETERS SETTING
In our experimental setup, the number of generated samples N and the number of source domains M are known priori, not tunable hyperparameters. For the fake graphs generation stage, the number of epochs is set to 200. т 0.2 in Eq. controlls sampling stability. For the model merging stage, the number of epochs is set to 20. The AdamW optimizer (Shazeer & Stern 2018) is used for gradient descent. T = 7p = 0.9 in Eq. 19 control parameter changes from pretrained models to preserve
knowledge. The hyper-parammeerss Agate and Ämast in the merging function, i.e., Eq. 20, chosen
from [10-2, 10-1, 1, 10. 100), and the value of k for the TopKSelector, i.e., Eq. 15 chosen from (1, 2, 4, 5). We report the mean results and standard deviations of ten runs.
B.3 ABLATION VARIANTS
We evaluate six configurations to analyze the contribution of each component:
Given Source variants: (1) Variant "w/o Mask" removes the parameter masks from pre-trained model classifiers while maintaining access to source data. (2) Variant "OGMM" represents our full method with access source domain data, serving as upper bound for performance.
18

---

Published as a conference paper at ICLR 2026
Source Free variants: (3) Variant "w/o MoE" eliminates the MoE module and uses simple averaging of masked GNN predictions. (4) Variant "w/o Mask" removes the masks added to the classifiers of each pre-trained model in the source-free setting. (5) Variant "w/o Egen" removes the graph generation objective (Eq. D., training the model merging stage with randomly generated noise graphs instead of our synthesized graphs. (6) Variant "OGMM" represents full method under the source-free setting.
B.4 ALGORITHM ANALYSIS
We analyze the computational complexity of our model to show its efficiency. Let [V] denote the total number of generated nodes, E' represent the number of generated edges, din and dmmid indicate the dimensions of the initial and intermediate layer features, respectively. The computational complexity of the model during the fake graphs generation stage is given by: O([V dymid + d - (1b) dmidd + [V| ...iid)). Generally speaking. dnn and dmid are significantly smaller than [V] or |E]. So the time complexity of the first stage of OGMM is O(VV + |E| + [V]). The computational complexity where m denotes the number of pre-trained models and IW| represents the scale of masks. Therefore, the time complexity of OGMM O(VI + E + V +m(EE) + [W] -V])). Although our model demonstrates effectiveness, has comparable complexity with the existing baselines.
of the second stage is (ddmsd m + m] ddmid
The algorithm is shown Algorithm During the experiments, we use one NVIDIA GeForce RTX 4090D GPU train and perform inference.
Algorithm 1 Procedure of OGMM
Input: pre-trained graph models (f:))
Output:
The predicting labels the target samples. First stage: Graphs Generation for i to M do
Initialize the domain-specific graph features {X'XX] 1) and arbitrary labels; while not converged do Generate graph structures [ by generators (P (Eq. 8
M Update {P.) and {A*) ==1 end while
by minimizing the generation loss Leen (Eq.
end for
Second stage: Graph Models Merging
Concatenate the generative datasets into g":
Initialize the fine-tuning masks w Eq, 13 and the gating layer Eq, 15 while not converged do Update (w, Wg. W,) by minimizing the merging loss Emerge (Eq. 20
end while
Besides, we further provide the running time comparison between OGMM and SOTA baselines in generating, training and testing phases in Tables 4 to verify effectiveness of the proposed method.
Table 4: Generating Time (in seconds) Comparison on four datasets.
Methods/Datasets
REDDIT-B
PTC
MUTAG
NCI1
Inverse-X
2030.95
88.07
63.17
452.02
Multi-GFKD
4052.60
1640.38
743.02
2043.15
OGMM
3448.28
134.17
126.69
827.76
For generation time, OGMM is significantly faster than Multi-GFKD (1.2-15x speedup across datasets). Compared to Inverse-X, which a simplified version of OGMM without graph structure optimization, OGMM requires additional time but delivers better performance. In the training phase, OGMM achieves the best efficiency, being 2-4x faster than traditional GNN methods and outperforming Multi-GFKI Regarding testing time, OGMM is slightly slower than baselines but still operates at millisecond to second scale, which is acceptable for practical applications. Overall, OGMM shows good computational efficiency, especially in the training phase.
19

---

Published as a conference paper at ICLR 2026
Table 5: Training runtime (in seconds) comparison on four datasets, the results are recorded the time of running 100 epochs for fairness.
Methods/Dataseee
REDDIT-B
PTC
MUTAG
NCT1
GCN
97.55
17.50
9.31
116.89
GIN
112.42
19.09
9.47
175.76
GAT
144.30
14.01
25.26
206.27
Multi-GFKD
51.66
49.02
30.54
51.06
OGMM
41.28
8.25
8.92
45.99
Table 6: Testing runtime (in seconds) comparison on four datasets.
Methods/Datasets
REDDIT-B
PTC
MUTAG
NC11
GCN
0.04
0.01
0.01
0.07
GIN
0.04
0.01
0.01
0.09
GAT
0.06
0.01
0.01
0.08
Uni-Soup
0.05
0.03
0.03
0.09
Greedy-Soup
0.07
0.03
0.02
0.11
Multi-GFKD
0.09
0.02
0.01
0.16
OGMM
0.10
0.07
0.05
0.25
C SUPPLEMENTARY EXPERIMENTS
C.I POSITION MASKS
We conducted additional experiments PTC and NC11 to analyze the impact of mask placement. As shown Figure 9 the results on these datasets further validate the conclusion: incorporating masks into the classifier (MaskCI.) achieves performance comparable to the average of the three mask-tuning methods.
45
400
5
201
40
005
45
(8) (A
T
PTC
(b) (B 1 PTC
(c) (A - T)
NCI1
(d) (B
NCII
Figure 9; Impact of Mask Position on PTC and NCII. The form (A T) means that GNN pre-trained on domain and fine-tuned on the Target domain. The bar chart shows the model performance the target domain, and the dashed line represents the average performance of masked models with different mask positions on this dataset.
C.2 GRADIENT TRENDS
Following the criteria outlined (Wagg et 2022) ) we caleulate the proportion of large gradients
for each model layer, as illustrated in Figures 10 12 The selected criterion is the number of parameters with a variation magnitude exceeding 0.001. Specifically, we partition the dataset into more subsets (10 / 20) based edge density, and sequentially use these subsets continuously train the selected GNNs. Then we use the selected criterion to analyze the degree of parameter variation within each module. Notably, the classifier weights exhibit more pronounced decreasing trend in parameter changes compared to other layers. This further indicates that fine-tuning the classifier parameters facilitates the model's 's ability to learn invariant representations, thereby enhancing the
20

---

Published as a conference paper at ICLR 2026
Laverg wegger Levery piee dering wmuths dcring 2017
corviter weight
clkerter diys
Lever wooete Leyer wiight dermig
Clrride werign
Loyeri snglet
Loynet 2nd1
dereet enight
Clasteed seogie
Clandber
Ceverd wenget
Ceyeno Ale
dermi pogle
darma aua
Clreiite weight
Clessrne Slae
18
19
(a) NCI1 (10 Domains)
(b) NCI1 (20 Domains)
(c) RED. (10 Domains)
(d) RED. (20 Domains)
Figure 10: Gradient trends in 2-layer GCN on NCII and REDDIT-B, (RED). illustrates the
evolution of parameters at each layer of GCN the model is trained with increasing number of domain data, based on a specified criterion.
Lower mereead
100
10
0.8
Demahni
0.8
0.5
0.4
01
(a) NCII (10 Domains)
(b) NCI1 Domains)
(c) RED. (10 Domains)
(d) RED. (20 Domains)
Figure 11: Gradient trends in 2-layer GIN NC11 and REDDIT-B (RED.). illustrates the evolution of parameters at each layer of the GIN as the model is trained with an increasing number of domain data, based specified criterion.
generalization capability of individual sub-learners. The results confirm the insight provided in Sec. the classifier weights gradually stabilize after training on multiple domains, indicating that these parameters capture cross-domain invariant knowledge.
C.3 PERFORMANCE OF HOMOGENBOUS BACKBONES MERGING
Leveraging the MoE architecture, OGMM imposes no explicit constraints on the underlying model architectures. To further assess the generalizability of OGMM within homogeneous GNN backbones, we reduce the number pre-traine traired models and employ only the widely adopted GCN for fusion. As presented in Table 7 even with the integration of just basic GCNs, OGMM outperforms
21

---

Published as a conference paper at ICLR 2026
corgee dhy weigle
sergge Cin Clesti
chp chogget whare
grople Canaie
Coeegate Ae clavotor andiee
Copect Levcc
1.
3.
Domain
0.5
02
522
92
0.
16
0.2
(a) NCII (10 Domains)
(b) NCI1 (20 Domains)
(c) RED. (10 Domains)
(d) RED. (20 Domains)
Figure 12: Gradient trends 2-layer GAT on NCII and REDDIT-B (RED.). illustrates the evolution
of parameters at each layer of the GAT as the model is trained with increasing number of domain data, based on a specified criterion.
existing fusion methods, setting a new state-of- of-e--r--te-rrttt across all four datasets. Notably, compared to the results in Table most performance metrics for OGMM improve upon reducing the number of pre-trained models. This suggests that, while the total knowledge volume remains constant, increased diversity among experts leads to the introduction of additional errors.
Table Data performance comparison across four datasets. The experimental setup was identical to T except that the base models used only two GCNs trained from different domains. Highlighted are the top first, second results.
Methods
REDDIT-B
PTC
MUTAG
NCI
ACC/ET
Pre/S/
ACCME
Pre/S/1
ACC/S/1
Pre/9/1
ACC/S/1
Pre/M/
Avg. PTM
53.85=3.74
59.79:15:42
50.43=2.28
51.0845.14
31.64:8.266
35.67=26.85
55.62:t4.52
60.14:3313
Eas-Prob
42.251:2482
48.291136.58
$1.72-2294
28.12.2.42
34.08133.79
60.6212.61
Cai-HighConf
50. 50.161:7747
54.6=34.05
52.76:2425
56.87:1::44
29.84:17.86 86
28.38:13266
54.87:16.86
60.15::3.65
Uni-Soup
44,46--28.49
43.62-433.01
49.81-4.25
44.27+15.4
35 18.62
16.37+1862
50.88-12.11
56.72+7.42
Greedy Soup
50.066229.28
57.41#28.04
48.71=2.99
19.76:
16.56:15.66
22.94:24.54
19.12
14.57
Inverse-X
64,499.23337
70.15:12.31
13.53=1447
51.16:13.27
37.5
40.41 1.2.02
65.65:10.67
54.0018.31
Muhi-GFKD
14.101.36.88
64.03:136.75
$4.17.222.91
38.87-117.39
46.56:112.53
34.94:22.77
38.02:115:9
54.88121.29
OGMM
80.64:1.71
76.81:99.6
55.34:00.34
59.87:12.60
50.78::20.40
43.58::25.35
66.38:10.04
63.12:22.76
C.4 PERFORMANCE ON LARGE-SCALE DATASETS
Table 8: Additional experiments on five large-scale datasets. The experimental setup was identical to Table Highlighted are the top first, second results.
Task
Graph Classification
Node Classification
ogbg- Molhiv
Lggn-Arxit
Twitch
Facehook- 100
Elliptic
Dutaset
Acc/9itt
Acc/%
Acc/%
Acc/%
Acc/%5
Avg-PTM
93.40:20.14
48.03-20.4
45. 15.43=1.32
51.450.44
60.31=1.45
Ens-Prob
95.46-20.18
42.1222
48.41=2.29
82:10.07
Ens- HighConf
96.52.10.09
14.61
47.89. 2.48
46.66-10.09
81.70
Uni Uni-Soup
94.69-40.03
25.63++290
55.80)
55.0443.755
82.39-0042
Greedy-Soup
78.61 133.11138.16
32.63
$3.29 3.36
55.51.11.46
82.08
0..0..1..30
Inverse-X
56.06=111.5
18.70=18.42
52.73-1.46
53.47:14.78
82.79:0.16
Muli-GFKD
70.43=336.68
15.00=0.54
51.33-1
54.23++1.57
82.65=+0221
OGMM
96.72:00.9
53.38-00011
59.45 0.85
56.89-10.08
82.89::0.00
22

---

Published as a conference paper at ICLR 2026
We extend our method to larger datasets for validation. We conduct graph-level classification tasks on ogbg-Molhiv (Hu et al. 2020). which contains 41,127 molecular graphs where each graph represents chemical compound. Similar to our approach in Sec. 4.2 we edge density as the criterion for domain partitioning, maintaining consistency with the partitioning method and pre-trained models described previously. For node-level classification tasks, we evaluate our approach on ogbn-Arxiv
(Hu et al.] 2020). Twitch (Rozemberezki et al. (021), Facebook- 100 (Traud et al., 2012) and Elliptic PParea al.) 2020) datasets. The ogbn-Arxiv dataset contains 169,343 nodes representing papers
from 40 subject areas. We follow the domain partitioning method (Qiao et al. 2025) based on the temporal shifts and partition the pre-2017 data into two domains (1971-205 11971-2013 and 2014-2017) for pre-training GNNs, and use the 2018 8-2000 data to test our model. The Twitch dataset contains 36,890 nodes representing users across seven regional networks. We pre-train GNNs using two groups of regions: (DE, ENGB, ES) and (FR, PTBR, RU), and evaluate OGMM on the TW region. The Facebook-100 consists of multiple social networks from different regions. The Elliptic is a Bitcoin transactions network dataset, includes graphs from different time steps.
As shown in Table S OGMM consistently establishes new state-of-the-art results across all bench- marks. significantly outperforming existing fusion methods. While traditional generative methods (e.g.. Inverse-X and (ulti struggle on large-scale datasets such as ogbg-Molhin and ogbn- Arxiv often underperforming even simple ensemble baselines, OGMM maintains clear advantage. This can be attributed to its generator design, which enables more stable and expressive expert modeling. In particular, compared with Inverse-X, OGMM yields substantial improvements across all datasets, highlighting its robustness to scale and task variation. The consistent superiority of OGMM underscores its effectiveness in integrating diverse knowledge sources while mitigating the instability typically introduced by generative fusion under large data regimes.
C.5 PARAMETERS ANALYSIS
4.00
42.0
6.16
0
25:44
20:2
20:0
25:39
2623
210
3-
2M14
2757
2029
25
2625
26N
13:9
42344
3M
3830
20,59
59
3355
33.0
37
2612
26:22
22
28:24
#
30:39
3200
22
31.01
3L
A
38755
1959
27:35
33.59
27
2532
26/22
26
11
2824
S2344
32:00
34
20.0
3455
25,75
39
27:18
28.26
2157
0.3L
10
162
465
02
14
148
C1L
04
10
382
(b)
k
(c)
k
Figure 13: Hyper-parameter sensitivity for OGMM MUTAG.
11
6412
41:2
51
41:09
JAN
41:11
0159
0445
41:25
21
47:57
42.00
+2.77
+1.03
+4.44
4743
4743
4133
4.0
47:50
47/02
11:05
14
10:00
67:44
4714
4743
4741
4133
42
Ape
Ape
(a)
1:
2
(b)
k
(c) k
Figure 14: Hyper-parameter sensiiiiity for OGMM on PTC.
Discussion about the Hyper-parameters. We conducted extensive experiments on four datasets mentioned in Sec. 4.2 to analyze the impact of apper-parameters {k. >gates Amask] } on model performance, as shown in Figures B 16 Notably, small datasets like MUTAG and PTC, the influences of {k, Agate) are more pronounced due to the larger variations in pre-trained models caused by limited data. In this case, the fusion process has more significant effect the results. On
23

---

Published as a conference paper at ICLR 2026
20.38
45
0.25
12.11
4004
65.0
7554
mm
<0:13
2312
25.45
53.13
24.00
10.15
44:3
5353
38,00
35
7558
37.34
A-
90.39
67:0
11.33
35.07
40.9
g-
X0.0
15
10.17
4433
3617
50:41
02
PARB
25-1
90:39
07.45
0.8
0.3L
300
108
ESt
19
(a)
2
(b) k
(c)
k
Figure 15: Hyper-parameter sensitivity for OGMM on REDDIT-B.
07.77
57.55
66.94
64.51
$653
66.90
$4.77
95.8
M.72
14.77
9..04
57.55
04
66.90
4452
6A53
66.99
$6.7
2-
04.53
96.54
05.5
95.0
24
74
57
54.28
02:55
7
08:55
96.3
57
95.33
68%
50
04.05
10
19.07
00.11
8+
04.55
08.35
$1.37
963
(a)
2
(b) k
(c) k
Figure 16: Plyper-paranen sensitivity for OGMM on NCII.
larger datasets such as REDDIT-B and NCI1, Amask plays a more crucial role, with the fine-tuning process ultimately determining the performance ceiling of the merged model.
Discussion about the Number of Domains. According to Optimization [2] OGMM can integrate multiple pre-trained models from different domains, with generalization improving as domain diver- sity grows. However, constructing datasets divided into infinite domains is impractical. Consequently, experiments rely on datasets with limited samples, where increasing the number of manually defined domains reduces the sample size per domain, impacting pre-trained model quality. This explains the trend in Figure 17 where OGMM's performance declines, and error rates rise as the number of domains increases.
C.6
VISUALIZATION
We visualize the real and generative graphs obtained from MUTAG and NCII, as shown in Figures 18 19 The visual comparison reveals some similarities between the graphs Iearned by OGMM and the real graphs, highlighting the model's ''s ability capture meaningful domain knowledge.
D RELATED WORKS
D.1 GRAPH DOMAIN GENERALIZATION
growing body of research on Graph Domain Generalization has garnered increasing attention in recent years. Approaches such as (Qiao et al.) 2023 Sun et al., 2024 Chen et al.) (2024b] Yuan et al. (024)) concentrate on learning representations that remain stable and invariant across diverse environments. In parallel, methods like (Sui et al.) 2022, Chen et al.) 2024a Gui et al. 2024
Fan et al. 2023) employ a causal inference framework to uncover relationships between data and labels that are robust to distribution shifts. Other techniques, including (Lu et al. 2024 Li et al.
2023 Jia et al. 2024), focus on improving model generalization by employing data augmentation strategies. Regardless of architectural differences, the effectiveness of these learning strategies is largely contingent on the precise acquisition, partitioning, and labeling of training data. Notably, the majority of existing approaches necessitate access datasets with clearly delineated data from
24

---

Published as a conference paper at ICLR 2026
Precision Accuracy
725
Accuracy Pracision
Performance (%)
70
arrormance
65.0
60
60.0
57.5
525
10
12
16
10 12
16
Number of Domains
Number of Domains
REDDIT-B
(b) NCII
Figure 17: Ablation studies regarding the number of domains. The horizontal axis indicates the number of source domains.
Nodes: 21, Edges: 22, Label:
Nodes: 17, Edges: 18, Labet: (b)
Nodes: 22, Edges: 24, Label: (c)
Nodes: 16, Edges: 17, Label: 1
Real Graphs
Nodes: 11, Edges: 11, Label: (e)
Nodies: 12, Edges: 10, Label: 0
Nodes: 17, Edges: 26, Label: (g)
Nodes: 10, Edges: 10, Label:
(h)
Generative Graphs
Figure 18: Graph visualization on MUTAG, Note that there no correspondence between the graphs in the two rows.
multiple environments, condition that is often impractical for real-world graph data. Additionally, some multi-source--Fi free domain adaptation methods can be easily applied to graphs (Dong al. 2021 Li et al. 2024b Shen et al. 2023); however, these methods require the use of target data in the model training process. In contrast, the source-free graph model generalization method proposed in this work presents more viable solution with broader practical implications.
D.2 MODEL MERGING AND MoE
Model merging and MoE are techniques for reusing pre-trained models to construct aggregation
systems with enhanced performance or generalization capabilities (Yadav et al.) 2024p. Model merging (Zheng et al. 2023a) typically involves the fusion of model parameters, such linear averaging (Utans) 1996. Wortsman et al. 2022), task arithmetic merging (Ilharco et al. 2022). or integration based on hidden representations (Yang et al.) 20233] Matena & Raffel 2022 Stoica et al.) 2023). These methods are primarily applied to vision and language models, which share consistent architectures that allow parameter space operations. However, such approaches are rarely applied to graph models due to their unique structures. Consequently, the MoE framework (Shazeer et al.
2017) has gained more attention in the graph learning field. In general, MoE facilitates ane-grained
fusion of expert outputs, such (Liu et al. 2023
Wang al.
2024
Zeng et al. 2023 Liu et al.
25

---

Published as a conference paper at ICLR 2026
Nodes: 36, Label: (a)
Nodies: 28, Edges: 31, Label: (b)
Nodes: 50, Edges: 55, Label: (c)
Nodes: Label:
(d)
Real Graphs
Edges: 31, (e)
Nodess Edgesi 22,
Edgas:
Edges:
(f)
(g)
(h)
Generative Graphs
Figure 19: Graph visualization on NCII. Note that there is correspondence between the graphs in the two rows.
2024; Ma et al., 2024 Wu et al. 2024b Liu et al. 2025 Cai et al. 2025). However, these works do not address the source-free -iitt--iditiitttionn graph models merging we tackle. In this paper, we
leverage the MoE. framework mechanism select and reuse graph models effectively to achieve domain knowledge fusion.
D.3
MASK TUNING
Radiya-Dixit & Wang 2020), where mask matrix is learned for specific modules of a pre-trained model to cover parameters, thereby avoiding redundant computations during the fine-tuning process. This approach originates from model pruning, which uses binary masks to identify important and sparse parameters (Lin et al.) 2020). In multi-task problems, RMT (Zheng al. 2023b) applies this strategy to facilitate transfer learning in vision-language models under the zero-shot setting. Similarly, GMT (Li et al., 2024a) leverages gradient information to identify key network parts for sparse updates. Regarding efficient utilization of parameter gradients, WWang et al.) (022) introduces judgment criterion to measure the trends of parameters across modules during continual learning, which inspired research mask locations. However, there has been limited exploration of mask tuning in graph models. Unlike vision or language models, graph models typically have fewer layers, and the impact of masks pre-trained GNNs requires further investigation.
Mask tuning is simple yet effective fine-tuning strategy (Zhao al. 2020
2020 Csordás
LIMITATIONS AND FUTURE WORK
Generalization on More Diverse Graph Data: Our work is based on the assumption of a mixture distribution, which has been extensively applied multi-domain generalization problems. For graph data, we have verified this assumption both theoretically and experimentally within the context of graph-level classification tasks. However, the discrepancy among graph domains can be complex, and significant biases can exist across different tasks, graph parametric representations, and scenarios. This variability poses a challenge to the development of graph foundation models (Fu et al. 2024). For the same reason, with new classes. In future work, we aim to further explore how to extend the multi-task learning capabilities of model and adapt more diverse graph data.
approach may generalize well enough to unknown scenarios, like those
26

---

Published as a conference paper at ICLR 2026
Towards Scaling Law: Additionally, the generalization performance of the proposed method is contingent on multiple factors, including the in-distribution performance of pre-trained models, the impact of fine-tuning methods, and the inherent randomness in generative graphs. While the overall computational complexity is relatively low, finding the optimal fitting function remains a challenging task. As the model collection grows, integrating larger number of more diverse and heterogeneous experts may become a significant hurdle for MoE-based techniques (He) 2024). Consequentl,, future efforts will focus on investigating model-centric scaling laws for graph expert systems, such as exploring how generalization performance evolves the number / diversity / capacity of experts grow, and developing automated expert selection and routing mechanisms that can efficiently manage large-scale heterogeneous model pools.
Future Directions: this work, we not only address the novel challenge model generalization for graphs but also highlight several promising directions for future research: (1) Extension to Cross- Transfer Learning: Expanding our approach to cross-task transfer learning by integrating and select- ing graph models trained different objectives. This will enable broader applicability of domain generalization across various graph-related tasks. (2) Model Reuse for Feature Structural Hetero- geneity: Investigating solutions for model reuse that effectively handle feature and structural heterogeneity across different graphs. This would enhance the adaptability of pre-trained models to diverse graph characteristics. (3) Building High- Quality Graph Model Pools: Researching methods for constructing high-quality graph model pools along with effective ranking and selection strategies. This will facilitate efficient adaptation of graph foundation models to new datasets and domains, similar to the successful adaptation in other areas of machine learning.
27


Reviews:
[{"review_id": "3YoVmLEkFj", "reviewer": "Reviewer_pjDW", "summary": "This paper proposes a novel problem of out-of-distribution graph model merging, aiming to merge models pre-trained on graphs from different domains into a model that generalizes under distribution shifts. A two-stage approach is proposed: first, a graph generator is trained to generate synthetic graph data, which is then used in the second stage to fine-tune the MoE module. Experiments demonstrate that the proposed method effectively generalizes to data with distribution shifts.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed problem of Out-of-Distribution Graph Models Merging is novel.\n- The proposed method, OGMM, has a solid theoretical foundation. Generating synthetic data to extract domain knowledge makes sense.\n- The experiments demonstrate the effectiveness of OGMM and the contributions of each component.", "weaknesses": "- The scenarios considered are limited. More results on other node classification graph datasets could be provided. Additionally, the paper only focuses the OOD scenario within a single graph, without considering cross-dataset or cross-domain scenarios.\n- The proposed OGMM relies on a mixture distribution assumption, which is not likely to hold in more complex scenarios.\n- The proposed OGMM seems to rely on many hyperparameters, some of which significantly impact model performance according to the analysis in the paper. This could limit its applicability in practical settings.\n- Although OGMM is theoretically computationally efficient, some experimental results could be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AENkwpCVdn", "reviewer": "Reviewer_1A5i", "summary": "This paper studies a new problem called out-of-distribution graph models merging, which aims to merge multiple pre-trained GNNs from different domains with distribution shifts into a single generalized model. The proposed OGMM is a two-stage framework, leveraging graph generation and a fine-tuned MoE module to enable generalization under graph OOD scenaros. The paper provides theoretical bounds on generalization error, and extensive experiments on multiple datasets demonstrate substantial performance gains over strong baselines.", "questions": "1) How robust is OGMM to the quality of pretrained GNNs? For example, if one model performs poorly or overfits its domain, does the merging process degrade significantly?\n2) Why focus masks primarily on classification heads? Have you ablated masking other layers such as message-passing modules and observed differences in generalization?\n3) How do you handle varying graph sizes or node/edge feature dimensions across domains during generation and merging? Is there preprocessing involved?\n4) Could OGMM be extended to incremental merging, where new pretrained models arrive sequentially?\n5) Would adding a contrastive or alignment loss between expert embeddings further stabilize the merging process?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1) The integration of graph generation and MoE-based model fusion is conceptually coherent, enabling domain knowledge transfer at both data and model levels.\n\n2) The mixture distribution assumption and accompanying error bound provide a formal justification for the merging process.\n\n3) The framework is applicable to heterogeneous GNNs, enhancing its generality and practical relevance.", "weaknesses": "1) The motivation for merging multiple pre-trained GNNs is not sufficiently justified. It remains unclear why model-level merging is preferable to retraining on aggregated data or simply using the best domain-specific model. No empirical or application-level evidence is given to show that scenarios requiring model-level merging without data access are common or practically constrained.\n2) Methodological novelty appears incremental, as the proposed approach largely builds upon existing techniques in graph distillation and mixture-of-experts regularization without a distinct new principle or paradigm.\n3) While the paper qualitatively illustrates synthetic graph realism, it does not quantitatively assess their fidelity to the underlying data manifold, leaving the actual extent of domain knowledge recovery uncertain.\n4) Notation suffers from inconsistencies such as {G_i}_{i\\in M}, and the overall writing is occasionally unclear, making it difficult to follow the theoretical formulation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q5mbY0pYC3", "reviewer": "Reviewer_YmPD", "summary": "This paper addresses a novel problem in the domain of GNNs: Out-of-Distribution Graph Models Merging (OGMM). The authors propose a framework that merges pre-trained GNNs from multiple domains with distribution shifts to create a unified, generalized model. This approach overcomes the challenge of needing to retrain models from scratch by leveraging pre-trained models to preserve specialized domain knowledge. The two-stage process first generates label-conditional graphs using each model and then fine-tunes and merges them using a MoE module. Experimental results show that OGMM outperforms previous methods on multiple datasets, establishing a new state-of-the-art in graph model generalization.", "questions": "1. See weaknesses.\n2. How are these pre-trained GNNs selected? Are they arbitrarily chosen graph models? If not, how is their optimality determined?\n3. Negative transfer is a common issue in generalization domains. I would like to know how OGMM avoids this problem, and whether there is any theoretical or empirical evidence supporting this.\n4. Some graph-MoE works should be compared and discussed in the experiments.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a novel challenge of merging out-of-distribution graph models without needing to retrain from scratch, which is both practical and impactful for real-world applications where data is scarce.\n2. OGMM consistently outperforms previous fusion methods and demonstrates robustness on large-scale datasets like REDDIT-B and NCI1, showing that it can handle diverse graph domains with different GNN architectures.\n3. The authors provide a solid theoretical framework for the problem and support it with comprehensive experiments that demonstrate the framework's effectiveness across multiple domains and datasets.", "weaknesses": "1. The two-stage process for merging out-of-distribution models involves multiple steps, including fine-tuning and the use of the Mixture-of-Experts (MoE) module. While effective, the overall time complexity of this process could be quite high, especially as the number of pre-trained models and the size of the graphs increase.\n2. The experiments primarily focus on datasets such as REDDIT-B and NCI1, which are not necessarily representative of the most commonly encountered graph types in real-world applications. It would strengthen the paper to include additional tests on more widely used or complex datasets, such as social network graphs or biological networks, to better understand the model's applicability across various domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "97NE4kst5x", "reviewer": "Reviewer_GZwp", "summary": "This paper tackles the problem termed Out-of-Distribution Graph Models Merging, aiming to build a generalized graph model by merging multiple pre-trained GNNs trained on different domains, without requiring access to source data. The authors propose a two-stage framework: (1) a label-conditional graph generation phase that inverts each pre-trained GNN to synthesize pseudo-graphs representing its domain knowledge, and (2) a model merging phase that employs a fine-tuned Mixture-of-Experts with learnable masks and sparse gates to integrate the knowledge of heterogeneous GNNs. Theoretical analyses are provided based on a mixture distribution assumption, and experiments o graph classification benchmarks show improvements over ensemble and model soup baselines.", "questions": "1. How sensitive is the method to the number and diversity of pre-trained GNNs? Would OGMM still perform well when experts are highly redundant or of poor quality?\n\n2. How stable is the label-conditional graph generation? Are there cases of mode collapse or low diversity?\n\n3. How does OGMM scale with the number of experts (in terms of computation and performance)?\n\n4. Could the authors provide a qualitative or quantitative analysis of the learned gating and mask patterns to better interpret the merging mechanism?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The idea of merging multiple pre-trained graph models without source data is interesting and practically relevant in the context of model reusability and privacy-aware learning.\n\n2. The combination of graph generation and MoE-based merging is conceptually coherent and could inspire further exploration of source-free graph model integration.\n\n3. The paper is well-organized and provides technical clarity, including equations, regularizers, and ablation settings.", "weaknesses": "1. The proposed task largely overlaps with graph-free knowledge distillation [1] and model soup [2], making the originality less substantial. The contribution seems more like a synthesis of existing ideas than a fundamentally new paradigm.\n\n2. The mixture distribution assumption $G_{T} = \\sum_{i} \\alpha_{i} G_{i}$ is overly strong and lacks empirical justification, especially in discrete graph spaces. Theoretical analysis is abstract and disconnected from the implemented MoE mechanism.\n\n3. All experiments are conducted on small, low-diversity datasets. No results are shown on large-scale or node-level tasks (e.g., OGB benchmarks). Hence, the claimed “OOD generalization” may not hold in realistic settings.\n\n4. The “domain” split based solely on edge-to-node ratio may not meaningfully reflect true distribution shifts. The evaluation therefore risks testing within-distribution variations rather than genuine OOD generalization.\n\n5. There is no investigation into how experts or masks behave, how gating decisions distribute across domains, or what knowledge is actually merged.\n\n6. Competing baselines are mainly adapted from non-graph domains; missing comparisons to recent source-free domain generalization or prompt-based GNN adaptation methods weakens the empirical argument.\n\n[1]  Graph-free knowledge distillation for graph neural networks.\n\n[2] Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)