G-Merging: Graph Models Merging for Parameter-Efficient Multi-Task Knowledge Consolidation
Authors: Jun Chen
Ziyue Qiao
Qin Zhang
Kaize Ding
Xiao Luo
Venue: ICLR 2026 Poster (2026)
Keywords: Model Merging
Parameter Efficient Fine-Tuning
Multi-task Learning
Primary area: learning on graphs and other geometries & topologies

Abstract: The pretrain-finetuning paradigm has achieved notable success in graph learning. Moreover, merging models fine-tuned on different tasks to enable a parameter-efficient model with multi-task capabilities is gaining increasing attention for its practicality. However, existing model merging methods, such as weight averaging and task arithmetic, struggle to generalize well to graph structures and Graph Neural Network (GNN) models due to the unique structural heterogeneity of graph data. In this paper, we propose an innovative graph model merging framework called G-Merging for merging multiple task-specific fine-tuned GNN models. G-Merging first employs task arithmetic to coarsely merge graph models, capturing shared cross-task knowledge. Second, it introduces a Topology-aware Wasserstein Distance (TWD) loss to train lightweight task adapters, preserving domain-specific graph patterns via aligning the embeddings of merged and fine-tuned models. Third, G-Merging integrates the adapters into a training-free, topology-aware router within a mixture-of-experts (MoE) architecture, dynamically routing input graphs to task-specific adapters based on structural similarity, thereby mitigating conflicts and enhancing knowledge sharing. Extensive experiments on 8 graph downstream datasets demonstrate the effectiveness of G-Merging, showing impressive performance close to or exceeding individual finetuned models while improving parameters and training efficiency. Our code is available at https://github.com/cjcj46262/G-Merging.

Published as a conference paper at ICLR 2026
G- G-MERGING: GRAPH MODEL MERGING FOR CTTAMETER-ERFICIENT MULTI-TASK KNOWLEDGE CONSOLIDATION
Jun Chen Ziyue Qiao².1 Qin Zhangt- Kaize Ding Xiao Luo
1 College of Computer Science and Software Engineering, Shenzhen University
2 School of Computing and Information Technology, Great Bay University
" "Department of Statistics and Data Science, Northwestern University 'Department of Statistics, University of Wisconsin-Madison 022433011@stu... wtt.su...b.edu.o. zyqiao@gbu.edu.
qinzhangeszu. edu. en
kaize. ding@northwester.. edu
xiao. luo@wl wi sc.edu
f Corresponding authors
ABSTRACT
The pvetrain-finetuni paradigm has achieved notable success in graph learning. Moreover, merging models fine-tuned on different tasks to enable a parameter- efficient model with multi-task capabilities is gaining increasing attention for its practicality. However, existing model merging methods, such as weight averaging and task arithmetic, struggle generalize well to graph structures and Graph Neural Network (GNN) models due to the unique structural heterogeneity of graph data. In this paper, we propose an innovative graph model merging framework called G-Merging for merging multiple task-specific fine-tuned GNN models. G-Merging first employs task arithmetic to coarsely merge graph models, capturing Wasserstein shared tross-task knowledge. Second, it introduces Topology-awar Distance (TWD) loss to train lightweight task adapters upon the merged model, preserving domain-specific graph patterns via aligning the embeddings of merged and fine-tuned models. Third, G-Merging integrates the adapters into a training- topology-awan aware router within mixture-of-expert (MoE) architecture, dynamically routing input graphs to tusk-specific adapters based on structural similarity, thereby mitigating conflicts and enhancing knowledge sharing. Extensive experiments 8 graph downstream datasets demonstrate the effectiveness of the merged model, showing impressive performance close to or exceeding individual finetuned models while improving parameters and training efficiency. Our code is available at https://gitee com c2mmm/jcjj42226233 (-
INTRODUCTION
With the gradual development of graph learning, various model architectures have been proposed (Scarselli et al., 2008; Wu et al., 2020; Xu et al., 2018; Wu et al., 2022), especially Graph Neural Networks (GNNs). Pre-trained models and pre-training strategies (Hu et al., 2020; Qiu al., 2020; Xu et al., 2018; Gu et al.; Ju et al., 2024) on graph data have gained attention due to their strong generalization ability. Meanwhile, fine-tuning these pre-trained models on downstream tasks has become standard paradigm (Sun et al., 2024; Zhang et al., 2022; Zhili et al., 2024; Sun et al., 2022; Gu et al., 2025), particularly scenarios where data labels are limited or out-of-distribution, such as in the domains of chemistry (Kim et al., 2023; Mayr et al., 2018; Wu et al., 2018; Sterling & Irwin, 2015) or biology (Veličković et al., 2018; Ingraham et al., 2019; Zitnik et al., 2019). Fine- specific downstream tasks, they trained
tuned models usually achieve good performance on task- specifc datasets in a targeted manner. However, when we need to perform multiple tasks simultaneously. applying individual fine-tuned models to different tasks results in high storage and

---

Published as a conference paper at ICLR 2026
Weight Average GoMergingoou))
Fine lsnel
Fre: Funelumd
Free Junee
Tuce
Merge
Task
Multi-task Model
BACZ
Tesk
Input groph from oifienan losks
Tesk
Figure I: From left to right: (1) The illustration of the graph model merging task. (2) A 2D I-SNE visualization of the final-layer embeddings from eight molecular graph datasets, encoded using the graph pre-training model (Hu et al., 2020). (3) The cross-domain validation performance of task-specific graph models, each fine-tuned on one of the eight datasets from a shared pre-trained model. (4) The performance comparison between models merged via the Weight Average and our proposed methods.
deployment costs. Multi-Task Learning (MTL) offers way to train unified model on an aggregated multi-task dataset, but it usually learns from scratch and does not leverage the knowledge already encapsulated in fine-tuned models.
Thus, model merging seeks to construct a unified multi- task model by combining the parameters of fine-tuned models instead of joint training from scratch, as shown in Figure (1). Ideally, it can retain the performance of each fine-tuned model while reducing parameter overhead and avoiding the need for extensive re- re-training on all task data. Until now, multiple model merging methods have been proposed, showing both practical significance and wide applicability. These range from the simple weight averaging strategy (Wortsman et al., 2022) to advanced ones, including Task Arithmetic (Ilharco al., 2022), Ties-Merging (Yadav et al., 2023), AdaMerging (Yang et al., 2023), DARE (Yu et al., 2024), and Wwin--Meggig (Lu et al.). While model merging has been explored extensively in domains like vision and language models, its applicability in the context of graph models still remains a challenge. For example, as shown in Figure (2), embeddings from eight similar graph datasets exhibit distinct clustering patterns, and in Figure (3), the models from the same pre-training graph model, fine-tuned on one of these domains (row), usually fail to generalize to others (column) when replacing the backbones of the corresponding fine-tuning models. These observations indicate that graphs, inherently with heterogeneous structural patterns, lead to complex representations that are highly -emaai--peccfic. These can result in a significant performance gap between fine-tuned models and their merged model when applying existing merging strategies (Yang et al., 2024a; Huang et al., 2025), due to task conflict and knowledge sparsity (Yadav et al., 2023).
These observations motivate the need for novel model merging strategies tailored to graph data. In this paper, we propose an innovative framework called G-Merging for merging multi-task fine- tuned GNN models. Specifically, we first conduct coarse merging multiple fine-tuned GNN models via task arithmetic to obtain a unified GNN model. Then, to capture nuanced task-specific knowledge, we introduce the Topology-awan -awar Wasserstein Distance (TWD) as a feature alignment loss. This loss trains lightweight, task-specific adapters upon the merged model, which supplement knowledge and alleviate representation bias. Furthermore, to mitigate task conflicts and encourage task knowledge sharing, we further integrate these adapters into Mixture of Experts (MoE) framework with a training-free. topology-awar router, which, together with the merged model, forms the final multi-task model to handle various downstream tasks. We demonstrate the effectiveness and generalization ability of G-Merging through extensive experiments on eight graph datasets, encompassing various GNN architectures and pre-training strategies. As an experimental example shown in Figure 1 (4), our method achieves a clear performance boost over naive weight averaging. Meanwhile, our approach offers efficient storage and fast inference, requiring storage space nearly equivalent to a single GNN model. The main contributions can be summarized as follows:
This paper proposes G-Merging, a novel approach to merging fine-tuned graph models via task arithmetic and TWD-based adapter routing, resolving cross-domain structural heterogeneity and consolidating task-specific knowledge while enabling cross-task knowledge sharing.
This paper proposes a topology-aware and training-free MoE that dynamically selects adapters at inference, enabling efficient cross-task knowledge transfer and multi-task generalization.
0 Extensive experiments demonstrate that G-Merging not only maintains or exceeds the perfor- mance of individual fine-tuned models but also improves storage and training efficiency. The framework is also model-aggosticc, supporting integration with various graph models.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Model Merging. In recent years, model merging has become rapidly evolving technique applied in two dominant scenarios: (i) merging multiple models trained on the same dataset, aiming to improve model generalization (Wortsman al., 2022; Wang al., 2022; Cha et al., 2021; Gupta et al., 2020; Wang et al., 2025) or support federated learning (Liu et al., 2022a; Wang et al., 2020) (ii) merging multiple models trained on different datasets or for different tasks to perform MTL (Matena & Raffel; Jin et al., 2022; Yang et al., 2023; Huang et al., 2025). This paper primarily focuses on the latter scenarios. Consequently, numerous works propose advanced merging techniques improve performance or efficiency (Huang et al., 2025; Yadav et al., 2023; Lu et al.; Ilharco et al., 2022). The baselines in our experiments provide specific examples.
Fine-tuning in Graph Transfer learning. Supervised Fine-tuning (SFT) from pre-trained models on downstream tasks is becoming standard paradigm in both NLP and vision fields (Paul & Chen, 2022; Dodge et al., 2020; Devlin et al., 2019; Bommasani et al., 2021; Dosovitskiy et al., 2020; Ma et al.; Qiao et al., 2023) and is gaining increasing popularity graph learning. SFT in graph transfer learning demonstrates effective performance while alleviating the burden of collecting labels regarding new tasks. Depending on whether all the parameters of models are adjusted, SFT can be divided into conventional full fine-tuning (FFT) and parameter-efficien fine-tuning (PEFT). PEFT aims to reduce the number of trainable parameters for downstream tasks by either inserting additional modules (e.g. adapters, learnable prompts) or training only a subset of parameters while freezing the rest (Hu et al.; Liu et al., 2022b; Houlsby et al., 2019a).
Mixture of Experts. The Mixture of Experts (MoE) paradigm introduces adaptive routing method that allocates experts dynamically to handle different inputs. Sparse MoE models activate only subset of experts for each input improve computational efficiency, while dense MoE models combine the outputs of all experts to achieve superior performance. The concept was first introduced by (Jacobs et al., 1991) with gating mechanisms select the experts. Recent studies have focused on challenges such as load balancing of experts (Clark et al., 2022; Zhou et al., 2022), training instability (Zoph al., 2022), expert specialization (Dai et al., 2024; Tang et al., 2024), and synchronization reduction (Sukhbaatar et al., 2024) for tasks in CV and NLP fields. Despite their success, the requirement for substantial multi-task data and the high cost of joint training remain significant challenges for these methods. In contrast, our MoE module is completely train-free and orchestrates task-specific experts based on TWD.
PRELIMINARIES
Notation. Let G(V,E) ) be a graph with vertices V and edges E. An input graph can be expressed as G === (A. X). where X E RVVixdaode is the node feature matrix and A E [V] is the adjacency matrix. Aij otherwise 0 if there is edge between nodes n,, and ny. Under the fintrainfiieeuue paradigm, let fG.) be a pre-training GNN model with parameter 6pre, which is pre-trained on large, general-purpose graph dataset. Then, we consider collection of K downstream graph tasks, indexed by
{1. 2, .... K}. Each task is associated with a private dataset Dj = {(G,, y ).))," No, where y/ is the label of the graph G; and N& is the number of samples in task k. For each task, the model is initialized with prr and then fine-tuned on Dr. to obtain task-specific parameters 6, with an additional task-specific prediction head.
Problem Definition. The graph model merging problem aims obtain a unified graph model fo.. with reduced parameter overhead by consolidating knowledge from a collection of fine-tuned models (fa, foz: fox and pre-trained model for... eliminating the need to maintain a full set of parameters for each task-specific model. Then, fo., as a shared graph encoder, combined with the task-specific classification heads from the fine-tuned models, is used to perform multi-task inference. The objective is to achieve this without retraining new multi-task model from scratch, which can be computationally expensive and require access to all labeled data, or relying on naive parameter averaging strategies, which often result in degraded performance due to parameter misalignment and lack of task-specific nuance. The goal is to strike a balance by efficiently merging multiple task-specific fine-tuned models into unified model that reduces storage and computational cost by
3

---

Published as a conference paper at ICLR 2026
eliminating the overall number of parameters, maintains even improves individual performance all downstream tasks, and possibly leverages shared knowledge to enable cross-task generalization.
Wasserstein Distance. Wasserstein Distance (WD) (Peyré et al., 2019)a.k.a Earth Mover's Dis- tance, Optimal Transport Distance) quantifies the similarities between objects such probability distributions, either discrete or continuous, by computing the minimum cost of an optimal transport plan from one the other (Bécigneul et al., 2020). We now describe the definition of WD between two discrete distributions as follows.
Definition 1. Consider two discrete probability measures E P(X) and V E P(Y), represented respectively as μ Σi=1 u,Ox, and شÿ=1 Vjoy, where dg denotes the Dirac measure centered at x. A coupling of μ(z) and vly) can be expressed as Tx., y) = Di= bド= (ayō(ma in which T R fulfills the marginal conditions T1, and T 1m respective weight vectors of p and v, and 1, E Rh' the all-ones vector of length n. The Wasserstein distance between y and is then given by:
Here, and v are the
T ΣΣ Ts c(E,,)g).
D Dwa(p,v)
min TEll(u,v)
(1)
where the feasible set II(u,v) = {T R = v} includes all joint distributions with marginals p and v. The function e( Vÿ) denotes the ground cost associated with transporting mass from x; to Vj. The matrix T, referred to as the transport plan or transport map. defines the quantity Tsj of mass relocated from to Vj.
4
METHODOL.OGY
4.1 COARSE PARAMETERS MERGING VIA TASK ARITHMETIC
We begin with a coarse merging process to reduce parameter redundancy. Specifically, from previous investigations and discussions on model merging research, knowledge modulariza- tion is effective and reasonable technique that decomposes the knowledge possessed by experts into 1 Shared knowledge and @ Task-specific exclusive knowledge (Lu et al.; Huang et al., 2025). As the name suggests, shared knowl- edge represents common and generalized knowl- edge across different tasks, e-g., in the pretrain- finetune paradigm, the base model or pre-trainet model possesses shared knowledge across down- stream tasks. Furthermore, task specifc knowl- edge can be compressed into shared knowledge by merging the parameters of fine-tuned mod- simple
Algorithm 1 G-Merging
Require:
fine-tuned model {fa, Seg. fex ). pre- trained model fepme- loss function Lrwd p Las D. non panameaaettric router R. and pre-specified weight X
1:
Parameters merging:
Shared knowledge
Compute unified model fe...:
3: Ouni + Opre + AE Opre)
4: Training:
b Exclusive knowledge
5:
for each task & do
6: Transfer knowledge of fe, into task-specific
adapters fadap, ;;
7:
0% + train forne LIWD, LMDY
8:
end for
els (e.g., direct weight averaging approach). Based on the above observation, merge fine-tuned GNN models into a unified model with shared knowledge, leveraging es- tablished merging technique called Task Arith-
9: Inference:
b. Main inference loop
10: perform task A
11: input a graph G with embedding matrix H
12:
for each layer model do
metic (Ilharco al., 2022).
13:
Update embeddings by unified model:
14:
H + fe... (H) Calculate rotter weights:
Specifically, for K tasks, the corresponding task vectors of parameters are defined т O2 - Opre. where & E {1, 21,2,... K}. Further- more, multiple task vectors {Tx jk=1 are added and merged into the pre-trained parameters Ppre. formulated as 8uns 6prc + > >k= Tke where the Own is the parameters of unified model with
15:
16:
K/ {
R([fudop.eg (H) (
17:
Merge for final output H();
18:
H H Σk_ Wk Sadap, (H)
19: end for
Ensure: Output H for input graph G.
4

---

Published as a conference paper at ICLR 2026
O frozen portmeters from unified medil
phose( parameters merging
input graph
model
ik tooke
0
0 Ouii
(0 ppm
0e)
Tk
MaE adopters
0 assi node embeddling
0 pre
8k frozen porometers from fineturre model
macel
0
6NN conv
input gnaph
McE Adapters
Finetuned model
node embedding
0x
Ad Adjoceccy Matrix
und
node embedding
+55 kyyers
5 kayers
H
und
GNN
GNIN
batchrorm
NodeAdapter
pool
McE Adopters
bot chnorm
batchnorm
CTWD Topology owore Wasserstein distance
GrophAdspher
pool
(+))
routee
GrophAdapter
LMD distance
classificaatin head
8k
0%
clessification head
6% clossification
L= CTWD a LMD
output logits
OuTput for each task phase(Ill): inference
phase{ ): Incorporating task-specific adapters
Figure 2: Overall framework of G-Merging, containing three main phases. In phase we coarsely merge GNN models fine-tuned different tasks into a unified model using task arithmetic. In phase(II), we train multiple task-specific adapters, lightweight modules used to solve the represen- tation bias between the unified model and the fine-tuned model. Moreover, Topology-Awand Wasserstein Distance (denoted as Crww D in the figure) and the LI distance (denoted LMDD in the figure) promote representation alignment the node and graph levels, respectively. In Phase (III), we add MoE adapters, composed of task-specific adapters and a router method, at each layer of the unified GNN model and before the prediction head for test-time inference.
apply the graph
shared knowledge and À is a scaling hyperpa-
rameter to control the balance between fundamental knowledge in pre-trained model and task-related knowledge in fine-tuned models. Setting À as A/ recovers naive weight averaging, while larger/smaller values amplify or suppress task-related adjustments. From the empirical results in Appendix F, value bigger than K may be more effective for downstream tasks. shown in Phase([) of Figure this coarse merging process offers foundational backbone that captures commonalities across tasks, serving as an initialization for the subsequent process.
4.2 INCORPORATING TASK ASE-CCECCIIC ADAPTERS
While coarse merging provides a foundation for capturing shared knowledge, it may be insufficient for capturing nuanced, task- sssk-specffic knowledge. Empirically, we notice that the single unified GNN model we obtained exhibits a performance gap compared to the fine-tuned GNN models on downstream graph tasks. Previous research has demonstrated that this discrepancy attributed to representation bias, which refers to substantial difference in the representation distribution between the unified and fine-tuned models (Yang et al., 2024a;b). To address this problem, we further introduce additional adapter-base based modules, which inserted after the graph convolution layers and before the classification head. We trained sask-specific adapters to minimize representation bias at both the node level (i.e., Adaters) and the graph level (i.e., GraphAdapters).
5

---

Published as a conference paper at ICLR 2026
Specifically, given an imput graph data G = {A, X3, the node embedding matrices H R (H) X) updated in 1-th GNN layers with without NodeAdapters respectively formulated as
Ho seo of (A, H H, Sudap.o .- (f (A, H²(
(2)
g Leannoo ( H
where O and 0 denote the parameters of GNN model and adapters, respectively, frow is the graph convolution function for aggregating messages and updating embeddings, which varies with different GNN backbones like GIN or GCN, and fodap is the lightweight adapter module, which can be an arbitrary implementation (such as multiple fully connected layers). Without loss of generality, in our method, we follow the original work on Adapters (Houlsby et al., 2019b) and set fadap as follows:
(3) where W downn Rddr and Wap up Rexd are two trainable matrices, i.e., 0" ( downs W Waph. ReLU(-) is a nonlinear activation function. The parameter r represents the internal rank of the adapters, controlling the scale. Similarly, the final graph embedding vectors R2d extracted by the Pooling layer with or without GraphAdapter can be formulated as
fadap.0 (H)- ReLU(H- W down) Wapt
h ( - Sadap.0 o- (f+ (fr((H, D)), he = (( }).
where fr is readout function such as averaging or summarizing the node embeddings.
For each task k. we denote the task-specific adapter parameters as 0z. We train these adapters on the corresponding task graph data to align the distribution of embeddings extracted by the merged and the previous fine-tuned GNN models. To achieve this, we employ Wasserstein Distance in Section quantify the similarity between two sets of node embeddings in a graph, which can be regarded as discrete distributions (Zhang et al., 2022; Chen et al., 2020). Motivated by the earlier observation of comaain-speciffe structural patterns inherent in graph data, we further incorporate graph topology to enhance structural awareness. Specifically, given the node embeddings H!!) 9u encoded from
the 1-th layer of the previously derived unified model with NodeeAaaterss and ! from the k-th fine-tuned model, we introduce an alignment loss based on Topology-aware Wasserstein Distance:
M
M
CTw = TWD(H U Bu oz
H
(1) Or
A)
= min TETT(A)
TS
c(h
(1) 1
(5)
where II(A)= IT E D! IIXIIII
Tip
1 11
T M
M 1py A T C (1MMM) A)
[ is the set of transport plans, h
H
Puns 02 [i. and hn0 H By b. denote two
embedding vectors, and c(., .) is the cost function. Compared to the original WD, the transport plan T now constrained by the graph adjacency matrix A. Therefore, this loss function is computationally aware of graph topology.
Notably, in our task, we calculate the cosine distance c(a, b) = (1- cos(a, b)) as the cost function, following many prior works based on the application of Optimal Transport problems (Zhang et al., 2022; Xu et al., 2020; Chen et al., 2020). Additionally, we set as the I-hop adjacency matrix with self- i.e., A, = 1 if and only if i j there exists reducing the cosine distance between h; and any h/. where j E N ( (with (((hh)(() denotes the set of neighbors of node i), contributes to the reduction of TWD. This is intuitively reasonable because the embeddings of neighboring nodes extracted by GNNs are expected to be similar-a property often referred as the smoothness of GNNs (Li et al., 2018). Similar to the standard WD, calculating TWD requires finding out the optimal transport plan T. This problem has been extensively studied in prior work, with efficient solutions such as Sinkhorn algorithm variants (Cuturi, 2013; Peyré et al., 2019; Dvurechensky et al., 2018; Zhang et al., 2022). We provide a detailed introduction including theoretical background and computational complexity of TWD Appendix A.
edge between and j. Conceptually,
For the final graph embeddings ho....0;
0z and ho,, we define the alignment loss as Manhattan Distance
(also called LI distance) between them, i.e,, LMD l above, the final optimization problem
b Il1. Combining the losses
L a L--LDD Σ LTWD
min 0%
Σ GEDA
(6)
Do

---

Published as a conference paper at ICLR 2026
where a is the hyperparameter to balance two losses.
As shown in Figure 2, in Phase(ll), we train without task labels and obtain K sets of adapters corresponding K tasks. These adapters serve as supplemental task-specific knowledge to the shared knowledge the unified model and improve the performance on downstream tasks. Notably, training process is efficient because the majority of parameters Juis frozen while only parameters in adapters 0% remain for training.
4.3 INFERENCE PROCEDURE
In the inference phase, we aim to obtain an enhanced model by integrating the knowledge in adapters from different tasks. Inspired by the perspective that knowledge from similar tasks can be mutually beneficial (Ilharco et al., 2022), we compose the task-specific adapters into MoEAdapters based on a parameter-free Mixture-of-Expee -eepertss architecture (Cai et al., 2024) during inference. Specifically, we compose fadap.0 to fmoe.f09, mooe..09.05..... og.....0kk] in Equations 2 and 4. Inside the MoEAdapter, each expert is set as the task-specific adapter, well-trained for particular task, and the output is weighted sum of the outputs of the experts. The process can be formulated as
Imoe.101.05 00....0..0.................. 0g (H)
= Σ Wk Soday.0; (H) k=1
where {0i, 02...,,0 are parameters trained in phase(II), {wi, W2,..., wK} are MoE weights cal- culated by a train-free router module based on the similarity between different tasks. Precisely, assume that have node embeddings H E RiVixd from given graph G and obtain two outputs, fadap.0z, (H) and Sadap.0;, (H), from two task-specific adapters. We can consider task A and task to be similar if fadap.0; (H) and Sadap.0;, (H) are similar, and the similarity can be measured by TWD. On this basis, during inference on task k, {w), W2, wx are calculated as follows:
(W1, 12, wk softmax ( (-TWD(Jadoppo (adop.0; (H), Sadap.o (H)))
(8)
This mechanism amplifies the contribution of experts trained on similar tasks when addressing the target task. Additionally, the similarity between two graph embeddings h R2 can be measured by Manhattan Distance, and the {W1, ..., wK) on graph level are calculated as:
(1 W2 wk} softmax
(1-1lfadap.e) softmax(1-11
(h) Sadon. o7 (
In this way, the MoEAdapters are tailored for each graph instance and effectively utilize knowledge while mitigating knowledge conflict across different tasks. As result, we only deploy puns and 10t ...... 00) during inference for all tasks, which is significantly smaller than the total size of 0,, 02 ..., 0g from all finetuned models, according to the experimental data Table
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Datasets. We use 8 binary graph classification datasets for molecule property prediction down- stream tasks, which are widely used as benchmarks for evaluating pretrain-finetume strategy in previous work (Zhang et al., 2022; Kim et al., 2023; Zhili et al., 2024; Sun et al., 2024). Data statistics and preprocessing are detailed Appendix B and C.
Baselines. Since we have not found related work in merging GNN models, we compare our G- Merging with several typical and advanced model merging methods from the CV or NLP field, including: Weight Averaging, Task Arithmetic (Ilharco et al., 2022), Ties-Merging (Yadav et al., 2023), and AMR-Merging (Huang et al., 2025), AdaMerging (Yang et al., 2023), Twin-merging (Lu al.) (detailed in Appendix D). Furthermore, models, and the pre- trained model as additional baselines beyond the merging strategy. The fine-tuned and pre-trained models serve the upper and lower performance bounds, respectively.
include Multi-tass tulti-task Learning, individual fine-tuned
7

---

Published as a conference paper at ICLR 2026
Table Test ROC-AUC score (%) of GIN models (contextpred) on downstream molecular property prediction tasks after merging fine-tuned models. (" denotes performance surpassing that of the fine-tuned model.)
Methods
Tox21
Toxcast
SIDER
ClinTox
BBBP
BACE
Average
Full Fine-Tuned
78.0
64.8
62.5
74.0
69.6
86.8
79.6
83,9
74.9
Pretrained
68.9
63.3
58.1
61.9
55.3
78.5
59.1
72.3
64.7
Multi-Task Leariing
75.5
63.4
62.8
64.9
66.4
84.7
74.8
77.5
71.2
Weight Average
74.7
64.5
60.4
70.7
63.5
78.8
66.5
77.5
69.6
Task Arithmetic
74.2
64.6
60.4
77.6
66.6
70.8
68.5
74.8
69.7
Ties-Merging
69.2
63.4
57.8
62.1
55.5
79.0
61.6
77.5
65.8
M
77.6
63.5
62.2
72.8
69.1
80.9
74.8
71.3
71.5
AdaMerging
69.2
62.0
57.6
60.2
70.5
64.4
Twin-Merging
69.9
63.4
59.0
63.0
59.0
57.5
59.8
75.7
63.4
G- G-Merging-s (Ours)
77.2
65.7"
64.6"
76.0
67.0a0a
86.6
76.0203
80.8199
74.2
G-Merging (Ours) 77.4
65.8"
10.1
64.8"
20.5
74.2" 10.
67.1
86.8
74.2.004
81.9
74.0
Table 2: Test ROC-AUC score (%) of GIN models (edgepred) on downstream molecular property
prediction tasks after merging fine-tuned models.
Methods
Tox21
Toxcast
SIDER
ClinTox
BBBP
BACE
IIV
MUV
Average
Full Fine-Tuned
76.1
64.8
70.0
70.5
86.1
77.6
79.7
73.9
Pretrained
71.6
64.9
60.0
61.6
54.6
76.4
64.4
65.5
64.9
Multi-Task Learning
73.8
64.0
63.3
71.2
68.8
81.1
74.5
72.8
71.2
Weight Average
74.1
66.1
62.2
65.9
63.6
67.0
68.5
68.3
Task Arithmetic
74.1
65.8
63.1
71.6
66.1
75.7
68.9
66.8
69.0
Ties-Merging
71.7
65.0
59.7
61.5
54.8
78.3
64.4
66.6
65.2
EMR-Merging
76.6
65.3
64.1
67.8
70.4
82.1
70.2
67.1
70.4
AdaMerging
71.3
63.6
59.7
72.7
57.3
72.4
68.6
60.8
65.8
Twin-Merging
71.8
64.6
58.8
67.6
56.4
56.7
61.6
69.0
63.3
G-Merging-s (Ours)
77.3" #0.4
66.0+n.
64.6292
74.1" =0.5
69.22e0
83. 100
74,9
75.5+0.
73.1
G-Merging (Ours)
76.9
66.0su.1
64.8
71.8"
=0.4
68.9
84.6:0.
74.2=0.
77.4s06
73.1
Settings. To evaluate the effectiveness of G-Merging in various scenarios, we use models with range of GNN backbones. Specifically, we reuse the pretrained models provided by Weihua et al. (Hu al., 2020) with two GNN architectures: GIN (Xu et al., 2018) and GCN (Kipf Welling, 2016), and two pretrain strategies: contextpred and edgepred. All models are self-supervised and pretrained on the chemistry dataset 7INC15 (Sterling & Irwin, 2015) (containing over million molecules). We obtain the fine-tuned models by fully fine-tuning pretrained models 8 downstream tasks. We run our method with different random seeds and report the mean and standard deviation of the performance. In addition, we also validate simplified version of our method, G-Merging-s, which directly uses task-specific adapters but not MoEAdapters during inference. More details pretrained models and hyperparameters provided in Appendix B.
5.2 EXPERIMENTAL RESULTS AND ANALYSIS
Main Results. The ROC-AUC scores in Table Table and Table 6 (in Appendix E) show the results for all tasks with various pretrained GNN models. The total average scores across 8 datasets indicate the overall capability of merging methods. We observe that our method significantly outperforms the baseline method in the vast majority of tasks, regardless of the GNN architectures and pretraining strategies. The performance is competitive with finetuned models, and outperforms them in some tasks, indicating that our merging method maintains or even improves individual performance, which is perhaps attributed to knowledge transfer or knowledge complementarity across tasks. Comparing G-Merging with G-Merging-s, G-Merging can perform better on certain tasks, indicating that one task can benefit from others the MoE structure. Additionally, for tasks with more pronounced topological heterogeneity, such HIV and MUV (datasets with numerous and structurally complex molecules), our method achieves a more significant performance improvement the baselines. This suggests that the TWD loss effectively extracts topological information from graph data, which is an essential element of the knowledge in the merged GNN model.
Ablation Studies. To examine the effect of each component on the final performance, we conduct ablation studies eight downstream tasks using the GIN (contextpred) pretrained model. We design
8

---

Published as a conference paper at ICLR 2026
five variants of G-Merging by removing certain components, and their overall performances shown in Figure 3 (more results are shown in Appendix E Figure 6).
Effect of parameter merging. When we directly use the original pretrained model as the unified model (Buns = ppre) and skip the phase of parameter merging, the per-
74.9
Aug. ROC scores 66
74
73.2
formance significantly drops. This suggests that shared knowledge is an indispensable fundamental part of model before incorporating task-specific exclusive knowledge.
36.2
en
64.7
Effect of MoE adapters. The performance of the vari- ant methods without MoEAdapters at either the node or graph level is worse than that of G-Merging, and the model performs the worst when MoEAdapters are removed at the graph level. This demonstrates the importance of MoEAdapters as task-specific knowledge suppliers and their effectiveness in alleviating representation bias.
Pretrained
wio TWD
wio parameter merging
wio dstance
lovel
(Ours)
Table 3: Ablation Studies.
Average
ROC-AUC scores across tasks for five ablated variants of G-Merging
Effect of loss functions. We further validate the effectiveness of LTWD and LMD by disabling one of them during training. From the results, can be seen that CTwD improves performance more significantly as it effectively leverages graph topologica information. In addition, LMD further boosts performance by enhancing graph embeddings alignment based on this.
Routing Analysis. Here we per- form an analysis of the TWD-based router method in our MoE struc- ture. We extract the average of ex- pert weights during inference, shown in Figure 4 and Appendix E, Figure 5. is evident that each sample receives maximum weights its task expert adapters. Furthermore, the router tends to allocate a greater weight to expert adapters when there is higher similarity between the test task and the source task of the expert, and this relationship bidirectional. For instance, considering the two tasks, Clin'ox and SIDER, which focus on the toxicity of drug compounds and diverse adverse drug reactions, respec- tively, they intuitively share high degree of similarity. Consequently, the router tends to allocate greater weights to expert_ expert_ClinTTox Clin'Tox when per- forming the SIDER task, and vice versa. This demonstrates the capabil- ity of our method to effectively inte- grate knowledge across tasks. We further conduct experiments on the top-k expert selection strategy our routing mechanism, which is a common design consideration in MoE models (see Appendix B for detailed settings). The overall results presented in Figure 3(b) (detailed in Appendix E Figure 7) show that as the number K of selected experts increases from 8, the model performance first decreases and then increases. When relatively few task experts are selected, the source task cannot receive sufficient support from other tasks, and due to the hetcrogeneity of adapters, the effectiveness of the original adapter may also be affected, leading to performance degradation. In contrast, when more task experts are selected, the router can better fulfill its role by assigning appropriate balancing weights, allowing large pool of similar task experts to collaboratively support the source task, thereby improving the performance. As a result, the above results suggest that G-Merging possibly leverages shared knowledge promote generalization across tasks.
Upper Bound (Full Fine-Tuned!
Upper Bound (Full Fine-Tuned)
74
74
ROC scores
22
M.Marinng
72
G Merging
70
ROC
70
68
68
66
Lower Bound (Pretrained)
66
Lower Bound (Pretraines) 80
10
30
top-k Figure 3: (a) Average scores 8 tasks with varying ranks of the adapters. (b) Average scores over 8 tasks of top-k expert selection in our routing mechanism.
rank
Tax21
Toxcast
Toxcast
SIDER
SIDER
ClinTxx
Clinlox
886P
BACE
BACE
HIV
HIV
MUV
experts:
Tok21
Trccast
SIDER
6O0P
BACE
HV
experts:
Tox21
Tracast
stoen
Rox
GR9P
BACE
WiV
Figure 4: The MoB weight heatmaps illustrate the expert al- location patterns during inference a target task, using two pre-trained models, GIN (contextpred) and GIN (edgepred).

---

Published as a conference paper at ICLR 2026
Efficiency and Storage Analysis. examine how G-Merging can reduce storage and computational cost. As shown in Fig- ure 3(a), our method remains effective under various settings of rank r, which is a hyperparameter related to the scale of adapters (see equation 6). It can be seen that, as the rank gradually in- creases, performance improves obviously when the rank is below 30, but decreases slightly when the rank exceeds 30. Empirically,
Table The parameter cost of the MoE adapters module.
The total number of parameters
Rank
MoE adapters
Ratio(%)
48000
2.58
20
96000
5.17
30
144000
7.75
our training time is approximately 1/8 of finetuning a whole model (see Table 7 in Appendix E). Additionally, the number of parameters in Table 4 indicates that the scale of MoEAAapters is considerably smaller compared to full GNN model.
192000
10.33
50
240000
12.92
one full GNN model: 1857900
CONCLUSIONS
In this paper, we investigate the problem of graph model merging for the first time. We propose method called G-Merging, which consolidates knowledge from task-specific fine-tuned models and achieves high performance while requiring lower storage and training costs. The effectiveness of G-Merging is validated by extensive experiments on 8 downstream datasets, which also indicate that graph model merging can be successfully achieved, like in models in the CV and NLP fields. More- over, we demonstrate that fully utilizing the graph structure can significantly impact the performance of the merged model. Finally, we conclude that knowledge sharing and mutual benefit between tasks are feasible and promising for models on graph data. In the future, we will extend our work to more model merging and graph learning scenarios, such as graph continual learning.
ACKNOWLEDGMENTS
The work of Ziyue Qiao partially supported by the National Natural Science Foundation of China (No. 62406056) and the Guangdong Basic and Applied Basic Research Foundation (No. The work of Qin Zhang was partially supported by the National Natural Science Foundation of China (No. 62576221) and the Guangdong Provincial Natural Science Foundation (No. 2025A1515010288).
REFERENCES
Han Altae-Tran, Bharath Ramsundar, Aneesh Pappu, and Vijay Pande. Low data drug discovery with one-shot learning. ACS central science, 3(4):283-293, 2017.
Jason Altschuler, Jonathan Jiles-Weed and Philippe Rigollet. Near-linear time approximation algorithms for optimal transport via sinkhorn iteration. Advances in neural information processing systems, 30, 2017.
Artem V Artemov, Evgeny Putin, Quentin Vanhaelen, Alexander Aliper, Ivan V Ozerov, and Alex Zhavoronkov. Integrated deep learned transcriptomic and structure-based predictor of clinical trials outcomes. BioRxiv, pp. 095653, 2016.
Gary Bécigneul, Detavian-Eugen Ganea, Benson Chen, Regina Barzilay, and Tommi S Jaaakola. Optimal transport graph neural networks. 2020.
Guy W Bemis and Mark A Murcko. The properties of known drugs. 1. molecular frameworks. Journal of medicinal chemistry, 39(15))2887-289) 1996.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
Weilin Cai, Juyong Jiang, Fan Wang. Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts. arXiv preprint arXiv:2407.06200 2024.
10

---

Published as a conference paper at ICLR 2026
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 1--22405-22411 2021.
Liqun Chen, Zhe Gan, Yu Cheng, Linjie Li, Lawrence Carin, and Jingjing Liu. Graph optimal transport for cross-domain alignment. In International Conference on Machine Learning, pp- 1542-1553. PMLR, 2020.
Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. Fusing finetuned models for better pretraining, arXiv preprint arXiv:2204.0300 2022.
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, al. Unified scaling laws for routed language models. In International conference on machine learning, pp. 4057-4086. PMLR, 2022.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture- of-experts language models. arXiv preprint arXiv:2401.06066 2024.
Jacob Devlin, Ming-Wes Mei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannanch Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Heigold, Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2020.
Pavel Dvurechensky, Alexander Gasnikov, and Alexey Kroshnin. Computational optimal transport: Complexity by accelerated gradient descent is better than by sinkhorn's 's algorithm. In International conference on machine learning. pp. 1367-1376. PMLR, 2018
Kaitlyn M Gayvert, Neel S Madhukar, and Olivier Elemento. A data-driven approach to predicting successes and failures of clinical trials. Cell chemical biology, 23(10); 2016.
Jiawei Gu, Ziyue Qiao, and Zechao Li. Refining norms: A post-hoc framework for ood detection in graph neural networks. In The Thirty-ninth Annual Conference on Neural Information Processing Systems.
Jiawei Gu, Ziyue Qiao, and Zechao Li. Activation shape matters: Ood detection with norm-entropy fusion. In Proceedings of the 33rd ACM International Conference on Multimedia, pp. 3076-3084, 2025.
Vipul Gupta, Santiago Akle Serrano, and Dennis DeCoste. Stochastic weight averaging in parallel: Large-batch training that generalizes well. arXiv preprint arXiv:2001.023: 2020.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
Neil Houlsby, Andrei Giurgiu. Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficiem transfer learning for nlp. In International conference machine learning, pp. 2790-2799. PMLR, 2019a.
11

---

Published as a conference paper at ICLR 2026
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan. and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference machine learning. pp. 2790- 2799. PMLR, 2019b.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.
W Hu, B Liu, Gomes, M Zitnik, Liang. V Pande, and J Leskovee. Strategies for pre-training graph neural networks. In International Conference on Learning Representations (ICLR), 2020.
Chenyu Huang, Peng Yc, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. Emr-merging: Tuning-free high-performanne model merging. Advances in Neural Information Processing Systems, 37:122741-122711 2025.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint 20X00 2022.
John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph- based protein design. Advances in neural information processing systems, 32, 2019.
Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-77, 1991.
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. arXiv preprint arXiv:22122 1et.:2222..99999 2022.
Wei Ju, Zheng Fang, Yiyang Gu, Zequn Liu, Qingging Long, Ziyue Qiao, Yifang Qin, Jianhao Shen, Fang Sun, Zhiping Xiao, et al. A comprehensive survey on deep graph representation learning. Neural Networks, 173:106207, 2024.
Suyeon Kim, Dongha Lee, SeongKu Kang, Seonghyeon Lee, and Hwanjo Yu. Learning topology- specific experts for molecular property prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 8291-8299, 2023.
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.00 02907, 2016.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Chang Liu, Chenfei Lou, Runzhong Wang, Alan Yuhan Xi, Li Shen, and Junchi Yan. Deep neural network fusion via graph matching with applications to model ensemble and federated learning. In International Conference on Machine Learning, pp. 13857-13869. PMLR, 2022a.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot sarfmeter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950-1966 1965,,996,..0 2022b.
Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng. Twin-merging: Dy- namic integration of modular expertise model merging. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
Xin Ma, Yifan Wang, Siyu Yi, Wei Ju, Bei Wu, Ziyue Qiao, Chenwei Tang, and Jiancheng Lv. Pala: Class-imbalanced graph domain adaptation via prototype-anchore learning and alignment.
Ines Filipa Martins, Ana I. Teixeira, Luis Pinheiro, and Andre Falcao. A bayesian approach to in silico blood-brain barrier penetration modeling. Journal of chemical information and modeling, 52 (6):1686-1697, 2012.
Michael Matena and Colin Raffel. Merging models with fisher weighted averaging, 2021. arXiv preprint arXiv:2/ 109832.
12

---

Published as a conference paper at ICLR 2026
Andreas Mayr, Günter Klambauer, Thomas Unterthiner, Marvin Steijaert, Jörg K Wegner, Hugo
Ceulemans, Djork-Amé Clevert, and Sepp Hochreiter. Large-scale comparison of machine learning methods for drug target prediction chembl. Chemical science, 9(24))544115451, 2018.
Sayak Paul and Pin- Yu Chen. Vision transformers are robust learners. In Proceedings of the AAAI conference on Artificial Intelligence, volume 36, pp. 2071-2081, 2022.
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends@ Machine Learning, 11(5-6)3355-600 2019.
Ziyue Qiao, Xiao Luo, Meng Xiao, Hao Dong, Yuanchun Zhou, and Hui Xiong. Semi-supervised domain adaptation in graph transfer learning. arXiv preprint arXiv:2309. 100,7733091177, 2023.
Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang. Ming Ding, Kuansan Wang, and Jie Tang. Gee: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD international conference knowledge discovery & data mining, Pp- 1150-1160, 2020.
Bharath Ramsundar, Peter Eastman, Pat Walters, and Vijay Pande. Deep learning for the life sciences: applying deep learning to genomics, microscopy, drug discovery, and more. O'Reilly Media. 2019.
Ann M Richard, Richard S Judson, Keith A Houck, Christopher M Grulke, Patra Volarath, Inthirany Thillainadarajah, Chihae Yang, James Rathman, Matthew T Martin, John F Wambaugh, et al. Toxcast chemical landscape: paving the road to 21st century toxicology. Chemical research in toxicology, 29(8):1225-1255) 2016.
Sebastian G Rohrer and Knut Baumann. Maximum unbiased validation (muv) data sets for virtual screening based on pubchem bioactivity data. Journal of chemicul information and modeling, 49 (2):169-184, 2009.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks, 20(1):61 10.180,02010 2008.
Teague Sterling and John J Irwin. Zinc 15-ligand discovery for everyone. Journal of chemical information and modeling, 55(11))2324-2337, 2015.
Govindan Subramanian, Bharath Ramsundar, Vijay Pande, and Rajiah Aldrin Denny. Computational modeling of 3-secretase 1 (bace-1) 1) inhibitors using ligand based approaches. Journal of chemical information and modeling, 56(10):1936-144) 2016.
Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Rozière, Jacob Kahn, Daniel Li, Wen-tau Yih, Jason Weston, et al. Branch-trait taaneh--rii-mi: Mixing expert 1lms 2024.
into a mixture-of-experts 1lm. arXiv preprint arXiv:2403.078
Mingehen Sun, Kaixiong Zhou, Xin He, Ying Wang, and Xin Wang. Gppt: Graph pre-training and prompt tuning to generalize graph neural networks. In Proceedingy of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1717-1727, 2022.
Yifei Sun, Qi Zhu, Yang Yang, Chunping Wang, Tianyu Fan, Jiajun Zhu and Lei Chen. Fine-tuning graph neural networks by preserving graph generative patterns. Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 0053-9061. 2024.
Anke Tang, Li Shen, Yong Luo, Nan Yin, Lefei Zhang, and Dacheng Tao. Merging multi-task models via weight-ensembling mixture of experts. arXiv preprint 3xXi:22402.0033: 2024.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
Hongyi Wang, Mikhail Yurochkin, Yuckai Sun. Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. arXiv preprint arXiv:2002.06-44 2020.
Yidi Wang, Jiawei Gu, Xubin Zheng, Xiao Luo. Pengyang Wang, Ziyue Qiao, et al. distribiittinttion graph models merging. arXiv preprint arXiv:2506.0300 03674, 2025.
13

---

Published as a conference paper at ICLR 2026
Zhenyi Wang, Xiaoyang Wang. Li Shen, Qiuling Suo, Kaiqiang Song, Dong Yu, Yan Shen, and Mingchen Gao. Meta-learning without data via wasserstein distributionally-robuss model fusion. In Uncertainty in Artificial Intelligence, pp. 2045-2055. PMLR, 2022.
David Weininger. Smiles, a chemical language and information system. 1. introduction to methodol- ogy and encoding rules. Journal of chemical information and computer sciences, 28(1):31-36, 1988.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes Ari Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pp. 13965-23998, PMLR, 2022.
Lingfei Wu, Peng Cui, Jian Pei, Liang Zhao, and Xiaojie Guo. Graph neural networks: foundation, frontiers and applications. In Proceedings of the 28th ACM SIGKDI Conference on Knowledge Discovery and Data Mining, pp. 4840 4841, 2022.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: benchmark for molecular machine learning. Chemical science, 9()):513-330, 2018.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32(1))4-24, 2020.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826, 2018.
Renjun Xu, Pelen Liu, Liyan Wang. Chao Chen, and Jindong Wang. Reliable weighted optimal transport for unsupervised domain adaptation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 4394 4403, 2020.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36:7093-71155 2023.
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. arXiv preprint arX2v:2310.02275. 2023.
Enneng Yang, Li Shen, Zhenyi Wang, Guibing Guo, Xiaojun Chen, Xingwei Wang, and Dacheng Tao. Representation surgery for multi-task model merging. In International Conference on Machine Learning, pp. 56332-56356. PMLR, 2024a.
Enneng Yang, Li Shen, Zhenyi Wang, Guibing Guo, Xingwei Wang. Xiaocun Cao, Jie Zhang, and Dacheng Tao. Surgeryv2: Bridging the gap between model merging and multi-task learning with deep representation surgery. arXiv preprini arXiv:2410. 2024b.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning, 2024.
Jiying Zhang, Xi Xiao, Long-Kat Kai Huang, Yu Rong, and Yatao Bian. Fine-tuning graph neural networks via graph topology induced optimal transport. arXiv preprini arXiv:2203. 10453, 2022.
WANG Zhili, DI Shimin, CHEN Lei, and ZHOU Xiaofang. Search to fine-tune pre- pre-trained graph neural networks for graph-level tasks. In 2024 IEEE 40th International Conference on Data Engineering (ICDE), pp. 2805-2819 19. IEEF, 2024.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture- of-eeetss with expert choice routing. Advances in Neural Information Processing Systems, 35:7103-7114 2022.
14

---

Published as a conference paper at ICLR 2026
Marinka Zitikk, Rok Sosié, Marcus W Feldman, and Jure Leskovee. Evolution of resilience in protein interactomes across the tree of life. Proceedings of the National Academy of Sciences, 116(10): 1426-4433, 2019.
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. Designing effective sparse expert models, arXiv preprint 222::2202.00906 2(3): 17, 2022.
15

---

Published as a conference paper at ICLR 2026
Appendix
In this appendix, we provide supplementary materials for this work. In Section A, we present detailed introduction of Topology-aware Wasserstein distance with related theoretical analysis. Sections B to G offer comprehensive description of our experimental settings, including additional results and analyses. The limitations and furture works are in Appendix I
A TOPOLOGY-AWARE WASSERSTEIN DISTANCE
A.1 BACKGROUND AND DEFINITION
Recall the definition of original Wasserstein Distance (Definition 1), we firstly discuss how to the Wasserstein distance to measure the similarity between two sets of node embeddings in graph. Given graph G(V, E) with adjacency matrix and two different node emmbedding matrix X and
X", corresponding to the two sets of node embeddings {x;"
[V] Ji=1 and {x [P] we characterize these two sets using two unified distributions μ = EM [ 5x2 and EM 1 & If we replace the distributions with ii and y in Eq. 1, we obtain the Wasserstein distance for node embeddings as follows:
M
M
D Dwdnode (u,
min Telluu.)
ΣΣ
E J=1
Tu cxx; xx]).
(10)
However, Eq. 10 depends solely on the node embeddings and not on the graph's topological structure. As discussed above, mass in x can be transported to any xJ E {xx] 1x] by transport plan Tıj- Here, we intuitively assume that mass in x² can only be transported to x if there is an edge between node n, and ny (i.e A.j = 1). This constraint can be enforced by setting Túj = 0 whenever A.j O. In this way, incorporate the graph's topological information into the WD, namely, the Topology-aware Wasserstein Distance (TWD). The definition given as follows:
Definition (Topology-awar) aware Wasserstein distance). Let G(V, be graph with adjacency matrix S A and two sets of node embeddings {x, Ji=1 and (x: ) represented by two unified distributions = I Ox and v
ΣM1 M Oxr. The Topology-aware Wasserstein distance between μ and [V]
is then defined
Dowa(A, p,v)
min TEII(A)
ΣΣ To c(x;, xxx]x)])
(11)
where (T
Tip -/- A IM - A (1M A)
1p1
O and c(...) ) is the cost function. Compared to the original WD, the transport plan T is now constrained by graph adjacency matrix A. Different matrix A. node embedding x,, and cost function lead to different WD, and its obvious that Dewel ( (A,
H4, v) > Dwd(H, v).
We define the matrix inner product (.-) for U, V E mx as (U,V) It U.yV Furthermore, we introduce its more intuitive formulation, which serves as the alignment loss function in Section 4.2:
M v
TWD XT_ A) Diwa(A, min TEN(A)
To c(x;,
min (T, C) TEl(A)
(12)
i=1==1
where C E R M||||| is the cost matrix with C₁ = c(x² xxxxx77) A widely used option of the cost function is cosine distance c(x; xx) [1 cos(x², x))) (Zhang et al., 2022; Xu al., 2020; Chen et al., 2020), while others are not elaborated here. Additionally, we originally set A as the 1-hop adjacency matrix with self-loops, i.e., Auj if and only if = or there exists an edge between and j. The 1-hop adjacency matrix A is easy extend to k-hop. like A AA, which may represent more global information of the graph structure. We conduct a supplementary experiment to examine this aspect in more detail, see Appendix G.
16

---

Published as a conference paper at ICLR 2026
A.2 COMPUTATION AND TIME COMPLEXITY
The computation of both WD and TWD requires solving for the optimal transport plan P, task that has been widely examined in the literature. In practice, approximate solution to WD can be obtained in polynomial time by applying the Sinkhorn algorithm with an entropic regularization term.
Computing both WD and TWD involves solving for the optimal transport plan P. problem that has been extensively studied in prior work. In practice, an approximate solution to WD can be obtained in polynomial time by applying the Sinkhorn algorithm utlizing an entropic regularization term(Cuturi, 2013; Peyré et al., 2019; D Dvurechensky et al., 2018).
For TWD, there exists an essential theoretical result for the iterative algorithm:
Proposition 1. Let e be hypter-paramettr HV-) be the entropy function, and assume that 0 Olog 0. The solution to definition with entropic regularization e H(A @ T) is unique and has the form
Túj = U,A.AK.yy
(13)
where K,j= = -exp( and (u, v) E RT x I are two unknown scaling variables.
Based this, the Sinkhorn algorithm useful for computing the TWD by iteratively approximating the dual variables u and v. However, the sparsity of A may lead to numerical instability or overflow during the iterative process. To mitigate this issue, adopt improved version of the Sinkhorn algorithm that performs computation in the log domain, which effectively alleviates such instability. The entire process involves a finite number of iterations. The complete algorithm, along with the theoretical justification of its correctness and feasibility, can be found in (Zhang et al., 2022).
We now analyze the computational complexity of the algorithm used to compute the TWD. Suppose that after certain iterations we get an approximate solution T satisfying:
(T, C) S TWD(X X A) +
According (Altschuler et al., 2017), when T 4e log([V)), the computational complexity of traditional Sinkhorn iterations to obtain T for WD is:
O 2 7T log(s) + a0 log(n)))
where Oc == maxi Cij and 8 = Eis Kij.
In TWD computation, K replaced with A O K, so is replaced with:
= A. Ss
Thus, the time complexity of the algorithm becomes:
O(n² (n2| - log(s') + |C| log(n)))
This smaller than the time complexity of original WD computation. conclusion, the time complexity of our method is positively correlated with the number of nodes, the magnitude of the cost values, and the sparsity of the adjacency matrix. Therefore, the proposed method is theoretically efficient and feasible, especially on relatively small and sparse graph data (e.g.. molecular graphs). As for the inference time, the TWD needs to be computed times K is the number of tasks), with each computation differing only in the cost matrix (varying node features the same graph). Therefore, the overall complexity is approximately linear with respect to K.
B EXPERIMENTAL SETUP DETAILS
Data Preprocessing and Splitting. All input molecules in downstream task datasets are represented as SMILES strings (Weininger, 1988). We follow the preprocessing procedure described in Hu et al. (Hu et al., 2020), embedding the SMILE-formated molecules into 120-dimensional node features and 3-dimensional edge features. We use scaffold (Bemis & Murcko, 1996), a splitting scheme based on the molecular graph structure, to divide the datasets into training, validation, and test sets following an 8:1:1 ratio. This scaffold split results in unseen structures in the validation and test sets, while common structures appear in the training set, which is closer to realistic model training and performance (Ramsundar et al., 2019).
17

---

Published as a conference paper at ICLR 2026
Training Settings and Computational Environment. In our experiments, employ two repre- sentative self-supervised pretraining strategies for graphs. One of them, Context Prediction, leverages subgraphs to infer their surrounding structures, with the goal of pretraining GNN that places nodes occurring in similar structural contexts close to each other in the embedding space (Hu et al., 2020). Edge prediction leverages observed edge connections and node representations to predict whether a masked edge exists, encouraging the GNN to learn node embeddings that place connected nodes closer in the representation space (Hamilton et al., 2017). We run all our experiments on Nvidia 4090 GPUs equipped with 24GB RAM. We fine-tuned all parameters of the pre-trained models for 100 epochs, using binary cross-entropy loss. The learning rate was manually tuned from the set (0.01, 0.005, 0.001, 0,0005), and the batch size was selected from (16, 32, 64). During the merging stage, we train task-specific adapters for 30 epochs with the Adam optimizer. The learning rate was consistently set to 0.01, and the batch size was selected from (16, 64, 256, 512).
Hyperparameter Strategies. In the final test evaluation, the loss balancing parameter in Eq. 6 is set to a = 1. The scaling factor X in phase(l) of G-Merging (see Section 4. 1. is searched to achieve optimal performance. The sensitivity of the aforementioned hyperparameters is further analyzed in Appendix F. For the TWD loss computation procedure, we set the hyperparameter 0.1, the threshold 7 0.1, and the maximum number of iterations 100.
Description of the Top-k Selection Experiments. Top-k selection means that the MoE router sorts the assigned expert weights in descending order, keeps the top k weights, and sets all other weights to zero. Formally, let w {wi, W2, wk} denotes the original MoE weights for experts, the top-k selected weights are defined as
Wi, if wt is among the top-k largest weights in w, i= 1= K
w top-k
0, otherwise,
Then we compute the final MoE weights as finall softmax (w/ 1op-k For this experiment, we use the GIN (contextpred) pretrained model, and keep all other setting same as in the main experiments.
Other Details. For the numerical stability of TWD computation, we normalize the elements of the cost matrix Ci = (1 - cos(x; xx")) )) to the range [0, 1] using max normalization. In most experiments, set the adapter rank (Eq. 6) 30, which provides good trade-off between efficiency and performance.
C THOROUGH DESCRIPTION OF THE DATASETS
The downstream tasks are derived from the public MoleculeNet benchmark (Wu et al., 2018), a widely used collection for molecular machine learning. MoleculeNet comprises more than 700,000 compounds evaluated on diverse properties, which are mainly grouped into four categories: quantum mechanics, physical chemistry, biophysics, and physiology. For our experiments, we focus eight datasets restricted to binary classification tasks within the biophysics and physiology domains. The statistical characteristics of these datasets summarized in Table 5.
This dataset provides IC50 values and binary binding labels for 1,522 compounds targeting human f-secretase 1 (BACE-1) (Subramanian et al., 2016). Data are collected from published studies, with some molecules having crystal structures. Scaffold splitting is recommended for generalization.
BACE.
BBBP. The BBBP dataset contains binary annotations for more than 2,000 chemical compounds, specifying whether they can penetrate the blood-brain barrier (Martins et al., 2012). It is commonly used in CNS drug development, with the scaffold split recommended.
Tox21, From the Tox21 initiative, this dataset offers qualitative toxicity labels for 8,014 compounds across 12 biological targets. It was featured in the 2014 Tox21 Data Challenge'.
ToxCast. ToxCast contains high thghuhhrouuhhutt in vitro toxicity data for 8,615 compounds from over 600 bioassays (Richard al., 2016), processed by MoleculeNet.
Tox21 Challenge, http:://tiioo...i..goooo////////
18

---

Published as a conference paper at ICLR 2026
Table Statistics the downstream graph datasets, including median, maximum, minimum, mean. and standard deviation of node numbers.
dataset
Categories
Tasks
Molecules
median
max
min
mean
std
Tox21
Physiology
7831
14
114
16.5
Toxcast
Physiology
617
8575
14
103
2
16.7
9.7
SIDER
Physiology
27
1427
23
483
30.0
39.7
ClinTox
Physiology
I
1478
23
121
1
25.5
15.3
BBBP
Physiology
2039
22
63
2
22.5
8.1
BACE
Biophysies
1513
32
33.6
MUV
Biophysics
93087
24
24.0
HIV
Biophysics
41127
23
222
2
25.3
12.0
SIDER. SIDER catalogs side effects of 1,427 marketed drugs, grouped into organ system classes
based on MedDRA (Altae-Tran et al., 2017)².
Clin'Tox. ClinTox compares 1,491 compounds approved by the FDA vs. those that failed clinical trials due to toxicity (Artemov et al., 2016; Gayvert et al., 2016). It includes two tasks: predicting toxicity and FDA approval.
MUV. The MUV dataset consists of 90,000 compounds across 17 tasks, designed to reduce screening bias (Rohrer Baumann, 2009). It serves benchmark for virtual screening models.
HIV. From the DTP AIDS Antiviral Screen, this dataset includes over 40,000 compounds tested for HIV inhibition "Active" and "moderately active" labels are merged; scaffold split is recommended.
D BASELINES DETAILS
Here will elaborate the baselines utilized in main comparison experiment, outlined in Table 1, 2, and 6.
Full Fine-Tuned refers directly performing the downstream task using a model that has been fully fine-tuned on the task-specific dataset.
Pretrained means using the pre-trained model fixed graph representation encoder, equipped with task-specific classification head taken from the corresponding fine-tuned model.
Multi-task learning aggregates datasets from multiple tasks into unified training set and trains single model to improve generalization and achieve better performance across tasks.
Weight Average (Choshen al., 2022) straightforwardly averages the parameters of multiple fine- tuned models, which is computationally efficient but often results in inferior performance.
Task Arithmetic (Ilhareo et al., 2022) first introduces the concept of task vectors and merges them into the pre-trained model with a hand-tuned scalar.
Ties-Merging (Yadav et al., 2023) Improves merging stability by resolving parameter conflicts based on tied weights across models.
EMR-Merging (Huang et al., 2025) training muining-free and gigh-performanc model merging method, containing three steps: Elect, Mark, and Rescale.
AdaMerging (Yang et al., 2023) employs output entropy minimization to learn the merging coef- ficients without label supervision for each task vector (Task-wise AdaMerging) or for each layer (Layer-wise AdaMerging). AdaMerging++ an enhanced version that applies Ties-Merging before learning the merging coefficients.
Tein-ng (Lu et al.) decomposes the knowledge into shared and task-specific knowledge, where the exclusive knowledge can be compressed to enhance efficiency. Consequently, the router is trained to dynamically merge shared and task-specific knowledge based on the input. This approach significantly narrows the performance gap between pre-trained and fine-tuned models. However, it tends to perform suboptimally on graph-structured data.
Medicall Dictionary for Regulatory Activities, http://wwww.mmddrr.rr
hhtps://wiki.nci.nt http:///ww...ttteceeooe/eeeeeets
19

---

Published as a conference paper at ICLR 2026
E ADDITIONAL RESULTS
We provide additional experimental results that could not be included the main text due to space constraints. Table 6 reports the outcomes of the primary comparison experiment based the GCN (contextpred) pre-trained model. Figure illustrates the detailed findings of the ablation study introduced in Section 5.2. Table 7 lists the complete results corresponding to Figure 3(a), supplemented with training time statistics. Figure 7 provides the detailed results corresponding to Figure 3(b). Figure shows two additional heatmaps of the MoE weights corresponding to different pre-trained models beyond GIN (contextpred). Note that we compute the MoE weights for each input by summing the weights from the GraphAdapter and the ayer-wise Adapers..
Table 6: Test ROC-AUC score (%) of GCN models (contextpred) downstream molecular property prediction tasks after merging fine-tuned models. (" denotes performance surpassing that of the fine-tuned model.)
Methods
Tox21
Toxcast
SIDER
ClinTox
BBBP
BACE
HIV
MUV
Average
Full Fine-Tuned
75.8
64.7
60.2
65.2
71.2
76.7
77.0
81.0
71.5
Pretrained
70.4
58.5
56.9
45.4
61.7
70.8
54.7
70.0
61.1
Multi-Task Learning
73.1
62.9
62.0
61.4
68.6
76.1
74.5
73.7
69.0
Weight Average
71.5
63.0
59.8
46.3
66.3
68.5
62.9
71.9
63.8
Task Arithmetic
71.7
63.0
59.9
47.1
66.2
69.2
62.4
72.1
63.9
Ties-Merging
70.4
58.7
57.9
42.6
61.1
72.3
57.8
72.7
61.7
EMR-Merging
73.8
61.3
60.8
53.2
70.3
73.2
72.7
66.1
66.4
AdaMerging
68.4
59.1
56.3
34,4
61.2
63.7
61.6
65.1
58.7
Twin-Merging
71.4
59.8
58.5
53.3
62.4
57.4
57.4
72.0
61.5
G-Merging-s (Ours)
73.020.2
63.1
61.9"
20.3
62.247m
69.8+02
73,4=03
68.5-1.5
78.8:005
68.8
G-Merging (Ours)
73.020.2
63.2...1
62.0"
10.3
62.6142
69.8102
73.3106
68.6-15
78.8
68.9
Table 7: Efficiency and Storage Analysis of GIN (contextpred) models.
Methods
Tox21 Toxcast SIDER Clin'Tox BBBP BACE HIV MUV
Average
times
Full Tuned
78.0
64.8
62.5
74.0
69.6
86.8
79.6
83.9
74.9
about 400+ min
Multi-Task Learning
75.5
63.4
62.8
64.9
66.4
84.7
74.8
77.5
71.2
144 min 45 min
Pretrained
68.9
63.3
58.1
61.9
55.3
78.5
59.1
72.3
64.7
G-Merging(r=10)
76.0
65.6
63.1
78.9
66.0
83,8
73.3
75.9
72.9
59 min 4 (single 4090 GPU)
(-Merging(r=20)
77.0
65.8
64.2
71.7
66.7
85.6
73.3
78.8
73.0
58 min 28 s (single 4090 GPU)
G-Merging(r=308 30)
77.4
65.8
64.4
74.1
67.1
86.9
74.5
81.2
73.9
57 min 56 s (single 4090 GPU)
(0))
77.5
65.8
64.4
72.9
67.5
87.2
74.1
81.7
73.9
min 49 (single 4090 GPU)
G-Merging(r=50)
77.5
64.3
71.2
67.6
86.6
75.4
82.7
73.8
min 10 (single 4090 GPU)
Tax21
Tox21
Tox21
21
SIDER
SIDER
SIDER
Cinisx
888P
88BP
R88P
BACE
BACE
HI
HIV
HIV
MUV
MUv
MUV
exports:
Tox21
Toccast
cairax
Bear
BACE
Her
experts
Ta2l
Taecest
SIDER
cairtox
BR3P
BACE
HW
MUy
experts:
To21
Taecast
sam
mear
BACK
knv
MUY
Figure MoE Weights Heatmap. The heatmap illustrates the weights assigned by the routing
mechanism of G-Merging between different testing tasks (rows) and task-specific expert adapters in the MoE framework (columns). Each value represents the weight distribution, indicating the contribution of cach expert to given task. Darker colors correspond to higher weights, highlighting more influential expert-task relationships. From left to right, the three figures correspond to three pre- trained model settings: GIN (ContextPred)), GIN (EdgePreed). and GCN (ContextPred), respectively.
20

---

Published as a conference paper at ICLR 2026
Tox21
775
77,0
21.2
585
66.8
SIDER
14.
Avg. RDC scores
76.2
65.0
55.7
65.5
65.0
Aer ROC
62
64.5
611
64.0
42.4
18.2
S8.3
58
64
BACE
77.5
76.3
70.0
615
$5.5
Wrg ROC scoves
75.0
74.2
60.3
67.0
Aeg: RDCC Scores
B4-8
72.5
65.0
84
921
66.00
78.5
08.9
42.5
25.0
HIV
MUV
80
79.5
84
83.
Pretrained
76.0
32.0
We parameter merging
Aeg RDC scores
73.8
23.7
73.0
Arg ROC scores
85.7
20.3
node moe adapters
76.
graph level moe adapters
weo TWD
distance
Merging (Ours)
Full Fine-Tuned
56.
/5./
Figure 6: The detailed results ablation studies Section 5.2. We evaluate five ablated variants of our method, each with a specific component removed, across downstream tasks.
Lipprr Bound (Tul Fmnetaree))
Moerriing
Salleringg
Avg. RDC scors
AVO. RCC scores
Upper Bound (Full Fine Tuned!
upper Bound Fine-Tated)
AVG.
Lower Bound (Pretrinne)
LOWer
ClinTox
Upper EieTinnd)
Upper
Avg. ROC scores
2vr. BCC scores
Cr S-Maroang
AVD.
8
Loper Bound (retrained
Bound
Uoper
Upper Sound (Pull Fine Tanedt
Upper Bound (Full Rne Tunedt
Aur. BOC scores
Aterr no
Avg. ROC scares
Marging
MaManana
forg ROC
Lower Dound Proto pedd
topk
Figure 7: The impact of Top-k selection in the MoE structure the performance of G-Merging, tested on tasks and their average.
21

---

Published as a conference paper at ICLR 2026
F HYPERPARAMETER SENSITIVITY ANALYSIS IN G-MERGING
We further conduct experiments about two inevitable hyperparameters in G-Merging: task vectors scalar À and loss balance a. As shown in Figure and 9, We present the performance variety of G-M G-Merging with À ranging from 0.05 to 0.25 and ranging from 10-6 to 10.
To2:
21
MIV
72-
ACC 2009
BACE
To21
5izoh
BACE
MIV
6,755 0.00 E.D5 5.15 0.302 5225 Taramooe
81552 E005 0.00 6,125 6155 6.15 0.002 0.25 0.252 Farameous
ILESE 6,75 WIDE E.15 8.15 6125 0.302 0.25 6253 Faremabus
Figure 8: Effect hyperparameter Lines of different colors represent the performance on each task. From left right, the three figures correspond to three pre- trained model settings: GIN (ContextPred), GIN (EdgePred), and GCN (ContextPred), respectively
34-
78-
75
HV
MIV
B00P BOCI
M20H
HV
MUV
Hpper Past
(pper Parsmveee
Figure 9: Effect hyperparameter a. Lines different colors represent the performance on each task. From left to right, the three figures correspond to three pre- trained model settings: GIN (ContextPred)), GIN (EdgePred), and GCN (ContextPred), respectively
G VARIOUS ADIACENCY MATRICES IN CALCULATING TWD
Recalling Eq. 5, the topology information incorporated into TWD is based on the standard 1-hop adjacency matrix A. However, this matrix can be easily extended to k-hop adjacency matrix, which may encode different aspects of the topological strueture. To this end, we investigate the impact of using different adjacency matrices the performance of TWD. As shown in Figure 10, the 1- 1-hop adjacency matrix generally performs better overall, while other adjacency matrices achieve competitive performance certain specific tasks.
22

---

Published as a conference paper at ICLR 2026
Performance on Different Adjacency Matrices
G Merging(A)
66.5
Merging(ssign(A1)
G Merginngtsign(A")
$2.4
74.5/44
75.2
ROC
57A
65.4 M.ASS&855 55.7
65
54.2
60
Tox21
Torcast
SIDER
ClinTox
BBBP
BACE
HIv
MUV
eeraaee
Figure 10: Investigating the Impact of Adjacency Matrices on TWD. The GIN (contextpred) pre- trained model is used in this experiment.
H G-MERGING ON HETEROGENEOUS AND DOMAINS.
GRAPH
To further evaluate the generalization of G-Merging beyond molecular property prediction, conduct experiments on nine heterogeneous downstream tasks, covering transportation networks, academic graphs, movie actor networks, social networks, and Reddit interaction graphs. These tasks include both node-level and graph-level classification and therefore provide a comprehensive assessment of performance outside the molecular domain. First, presents the experimental setup here.
Pretraining. Following the protocol of GCC (Qiu et al., 2020), we adopt publicly available pretrained GNN model trained jointly on mixture of heterogencous graph datasets, including: Academic networks (Academia, DBLP-SNAP, DBLP-NeIRep), Entertainment networks (IMDB), and Social networks (Facebook, LiveJournal). These datasets differ substantially in structure and semantics, providing suitable basis for evaluating the generalizability of our method beyond molecular domains.
Downstream Datasets. We fine-tune the pretrained model on nine downstream datasets covering both node-level and graph-Ievel prediction tasks. For node-level classification, we use four datasets: USA Airport, Europe Airport, Brazil Airport, three Airline transportation networks (over 1K nodes), where labels correspond to airport activity levels (4 classes); and H-index, co-authorship subgraph from Open Academic Graph, where labels indicate whether an author's 's h-index is above below the median. For graph-level classification, we use five datasets: IMDB-Binan and IMDB-Multi, movie actor collaboration graphs (small, dense, 2-3 classes); COLLAB. medium-siie sized scientific collaboration networks; Reddit- Reddit-Binary and Reddit-5K, Large-scale sparse Reddit discussion graphs with diverse interaction patterns. These datasets downstream tasks to evaluate the generalization of GNN models pretrained using their proposed strategy. And we believe that this setting already spans both node-level and graph-level tasks over diverse and heterogeneous graph topologies.
also used in GCC (Qiu et al., 2020) as
Merging Procedure. For each dataset, we fine-tune one model from the same pretrained checkpoint. Then we apply G-Merging and two baseline methods (Weight Average and Task Arithmeti) to merge nine finetuned models. All merged models evaluated their corresponding tasks using Fl-score. The results are presented below:
23

---

Published as a conference paper at ICLR 2026
Table 8: Performance comparison on diverse non-molccular graph benchmarks.
Method
airport
curope_airport
brazil, airport
b-index
imdb-binary
imdb-multi
rdt-5k
Pre trained
41.68
37.85
56.54
56.84
59.60
39.27
61.44
59.70
32.07
Weight Averae
56.49
56.32
65,29
60.40
39.86
62.12
Task Ariihmetic
58.20
$8.31
42.90
69.10
59.80
42,60
60.80
67.80
27.80
G-Merging (Ours)
59.01
57.24
60.52
72.87
61.29
47.30
62.22
70.57
39.11
We observe that G-Merging achieves the best performance on most of datasets and improves sub- stantially over baselines both node-Ievel and graph-level tasks. These results demonstrate that G-Merging generalizes well to non-molecular graph domains such as transportation, academic, and social networks, and remains effective on diverse task types, including large sparse graphs and multi-class graph classification.
Overall, this study confirms that G-Merging restricted molecular property prediction. Our method successfully merges models fine-tuned on broad variety of graph domains and task settings, which confirming its applicability under heterogeneous and non-molecular scenarios.
LIMITATIONS AND FUTURE WORKS
Our proposed G-Merging framework demonstrates promising results in merging fine-tuned graph models and transferring knowledge across tasks. However, our setting assumes the task-specific models used for merging are all fine-tuned from a shared pretrained GNN model. A more general scenario, where these models may originate from different checkpoints or even differ in architecture, remains an open direction for future exploration. Also, the use of fixed number of fine-tuned models may limit adaptability to evolving unseen tasks.
In future work, we plan to study a more generalized graph model merging framework by developing adapter alignment strategies that can reconcile representations from different backbones, and also explore unified routing mechanisms that remain effective across diverse model types. Furthermore, plan to extend the framework with continual learning capabilities enable dynamic adapter composition, improving the model's practicality in real-world scenarios.
24


Reviews:
[{"review_id": "VMjIs52Yux", "reviewer": "Reviewer_J2df", "summary": "The paper introduces G-Merging, a three-phase framework to merge multiple task-specific GNNs into a single multi-task model. Phase I performs task arithmetic to coarsely combine fine-tuned models into a unified encoder. Phase II trains lightweight adapters (node- and graph-level) on each task to align the unified model’s representations to those of the original fine-tuned models via a Topology-aware Wasserstein Distance (TWD) plus an L1 graph-level loss. Phase III composes all task adapters into a training-free MoE with a topology-aware router that computes per-input soft weights from representation similarities, enabling test-time expert selection. Across eight molecular property prediction datasets with GIN/GCN backbones and two pretraining strategies, G-Merging matches or exceeds many weight-merging baselines and approaches per-task fine-tuned upper bounds while using far fewer trainable parameters during adapter training.", "questions": "See weakness. I'll adjust my rating according to the authors' rebuttal.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The TWD loss constrains optimal transport by the input graph’s adjacency, encouraging local smoothness in node embeddings and addressing structural heterogeneity that undermines naïve merging. This is a principled graph-specific twist on WD and is well-motivated by prior observations of task/domain clustering.\n\n2. The three phases, task arithmetic (shared knowledge), adapter training (exclusive knowledge), and train-free MoE routing, are clearly specified with equations and an algorithmic sketch, making the method easy to reimplement or ablate.\n\n3. While being, to my knowledge, the first study of model merging in the graph domain is noteworthy, the paper should better justify why such a method is necessary. Given how long model merging has been around, its absence in the graph domain likely reflects factors beyond the task difficulty, for example a lack of clear necessity.", "weaknesses": "1. This method still appears computationally intensive, because a task-specific adapter must be trained for each model involved in the merging. If the computational cost of this training is comparable to, or not much lower than, training a GNN from scratch, it seems to run counter to the original purpose of merging. This concern rests on two points: first, GNN training is itself quite lightweight; second, merging is intended to reduce the training burden when facing new tasks. In addition, how transferable are the trained adapters? Do we need to retrain a new set of adapters for every configuration (number/combination of models)?\n\n2. Compared with the baselines, this method appears to require more GPU memory at inference. The baselines’ memory footprint is essentially that of a single model, whereas this approach involves an MoE component, which requires a router to decide whether and how each adapter is applied during inference. Moreover, the results in Tables 1 and 2 suggest that introducing the MoE architecture has little to no positive effect on overall performance and seems largely dispensable.\n\n3. Figure 3(b) is only shown in the main text but is barely discussed in the experimental description. The paper mentions on lines 473–473 that “as the set of selected experts becomes larger, indicating access to more diverse knowledge, the model performance improves accordingly.” However, this statement does not match the results in Figure 3(b): as the set of selected experts grows, performance first drops and then rises, rather than monotonically increasing. In the appendix, Figure 7 presents more fine-grained results but still lacks detailed analysis. I recommend that the authors provide a thorough analysis of this experiment. Based on Figure 7, the Top-K strategy appears to have markedly different effects across datasets. How do the authors balance these differences in practice, and what explains these divergent impacts?\n\n4. It seems that all tasks are molecular property prediction. It is unclear whether TWD and routing generalize to non-molecule domains (social, knowledge, large sparse graphs). Please add some non-molecular datasets or discuss constraints.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a8pllNmPig", "reviewer": "Reviewer_YDHz", "summary": "This work primarily proposes a graph model fusion framework that first performs coarse fusion of graph models using task-specific operations, and then trains task adapters via TWD loss, leveraging an MoE-based routing mechanism to reduce conflicts after merging and enhance knowledge sharing.", "questions": "please refer to weaknesses.\n\nOverall, I think this is a good paper, but there are some unclear aspects that prevent me from fully evaluating its contribution. If the authors could clarify my concerns, I would be happy to raise my score.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Originality.The authors' work is original.\n\n2. Quality. The quality of the authors' work is excellent; the assumptions and experiments mutually support each other.\n\n3. Clarity. The authors' work is clear. In particular, the figures fully explain what they are doing.\n\n4. Significance. Given the well-known characteristics of graph data, obtaining a unified cross-modal model in the graph domain is an important problem.", "weaknesses": "1. Data Selection: The data selection in this work is confusing to me—why do the experiments only use graph-level datasets and not discuss more widely adopted node-level benchmarks (e.g., Cora)?\n\n2. Lack of Discussion on Related Works: The paper lacks sufficient discussion of related work, which makes it hard to follow. To my understanding, the authors’ method can be summarized as parameter merging combined with knowledge distillation (please correct me if I’m wrong). Even within the graph learning community, this type of approach has been extensively explored—for instance, ParetoGNN [1] (during training) and WAS [2] (during fine-tuning). Could you please clarify how your design differs from or builds upon these prior works?\n\n---\n\n[1] ICLR '23, https://arxiv.org/abs/2210.02016  \n[2] ICLR '24, https://arxiv.org/abs/2403.01400", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FTWGNN2nGb", "reviewer": "Reviewer_tomN", "summary": "The paper introduces a strategy for model merging in graph neural networks (GNNs). In particular, the considered setting includes a pre-trained GNN, trained on some general task, which then gets adapted into multiple specialized versions through fine-tuning for specific tasks. The goal is then to merge the fine-tuned versions together to obtain a single model that can perform all tasks with comparable performance. The proposed method involves first merging parameters with task vectors, then adding adapter modules in between graph convolutions (and training them), and finally a training-free mixture of expert approach to mix the contributions from different adapters at inference time. A novel loss based on Wasserstein distance is used to train the adapters with the goal of reducing the difference in the distribution of the representations between the unified and fine-tuned models.\nThe experiments consider 8 binary graph classification datasets regarding molecule property prediction, and compare against existing model averaging techniques (Weight Averaging, Task Arithmetic, Ties-Merging, and EMR-Merging, AdaMerging, Twin-merging). Additional baselines are given by multi-tasks learning, individual fine-tuning, and the pre-trained model. Results show that the merged model has similar performance to the individual fine-tuned ones.\nAn ablation study shows the importance of all components in the proposed method.", "questions": "-  Oversmoothing is a well known problem of GNNs. The Wasserstein loss introduced in this paper seems to actually encourage oversmoothing (it would be minimized if all representations were the same). Could the authors please comment on this?\n- The proposed Wasserstein loss has a complexity which is quadratic in the number of nodes (as shown in the appendix). How does this behave for large graphs? Would it be possible to add some runtime values for some of the experiments?\n- One of the properties of the proposed Wasserstein loss is that it is \"aware of graph topology\" (i.e., constrained by the graph adjacency matrix). Have the authors tried removing this constraint and seeing how performance would change? I agree that this seems like a principled choice, but at the same time giving more freedom to the transport plan may provide benefits?\n- Could the authors expand on the practicality of the proposed method? It requires a pre-trained model, and several fine-tuned versions of it. While this is a somehow realistic scenario for large language models, is this something that happens when deploying GNNs in real-world applications?\n- If possible, as the experiments have been run over 5 random seeds, could you add numbers for the variance/standard deviation? I think this is an important factor to validate the effectiveness of the method.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Extensive experimental section with many baselines.\n- Informative ablation experiments to showcase the inner workings of the method (router heatmaps and top-k expert selection plots)", "weaknesses": "- The graphs considered in the experimental evaluation mostly have a very small number of nodes. \n- The proposed Wasserstein loss can be computationally prohibitive for large graphs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zZMeaoR4qe", "reviewer": "Reviewer_VpK5", "summary": "This paper introduces G-Merging, a novel framework for consolidating multiple task-specific fine-tuned Graph Neural Network (GNN) models into a unified multi-task model with parameter efficiency. The proposed approach consists of three key components: (1) Task Arithmetic for coarse parameter merging, (2) Topology-aware Wasserstein Distance (TWD) for aligning embeddings and training lightweight task-specific adapters, and (3) a training-free MoE router that dynamically selects task-specific adapters based on structural similarity. The authors position this work as the first systematic approach to address the model merging problem specifically for graph neural networks.", "questions": "1. The paper assumes all tasks use the same pre-trained GNN backbone. How would G-Merging handle cases where different tasks employ different GNN architectures? If not currently supported, what modifications would be needed to extend the framework?\n2. In the MoE router, how is \"structural similarity\" formally defined and computed? Is there theoretical justification for this similarity metric's effectiveness? Were different similarity measures experimentally compared?\n3. How does G-Merging perform when merging models trained on graph datasets with significantly different structural distributions (e.g., social networks vs. molecular graphs)? Are there specific mechanisms to handle such task heterogeneity?\n4. The paper mentions that TWD leverages graph topological information, but the ablation study only shows results with different adjacency matrix powers. Could you provide more detailed analysis on how different topological aspects (e.g., degree distribution, community structure) affect the merging performance?\n\nIf the authors provide satisfactory responses to these concerns during the rebuttal phase, I would be prepared to revise my recommendation to a stronger accept.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This is the first work to systematically address the model merging problem specifically for graph neural networks. The proposed three-stage framework (Task Arithmetic, TWD-based alignment, MoE router) is logically sound with each component complementing the others.\n2. By only fine-tuning a small number of adapter parameters rather than the entire model, G-Merging significantly reduces computational and storage costs for multi-task deployment, aligning with current trends in parameter-efficient fine-tuning.\n3. The evaluation across multiple molecular property prediction tasks demonstrates G-Merging's effectiveness in consolidating knowledge from multiple task-specific models. Ablation studies validate the contribution of each component.\n4. The paper is well-structured with clear methodology descriptions and intuitive visualizations that make complex technical content accessible.", "weaknesses": "1. The current approach assumes all tasks share the same output dimension, preventing it from handling tasks with different output dimensions (e.g., binary vs. multi-class classification). This represents a significant limitation since real-world applications often involve diverse task types with varying output structures.\n2. The method involves multiple hyperparameters (task vector scaling factor, loss balancing weights, adapter rank, etc.). While the paper provides some sensitivity analysis, there's insufficient guidance on how to select these hyperparameters based on task characteristics.\n3. Current experiments focus exclusively on molecular property prediction tasks (all graph-level binary classification), lacking validation on other graph task types (e.g., node classification, link prediction).\n4. The paper primarily compares against model merging approaches but lacks comprehensive comparison with parameter-efficient fine-tuning (PEFT) methods for multi-task learning (e.g., LoRA-based approaches). \n5. While the topology-aware Wasserstein distance is proposed, there's minimal theoretical analysis of its properties, relationship to graph structural characteristics, or convergence guarantees. Additional theoretical insights would strengthen the methodological contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)