Deep Graph Mating
Authors: Yongcheng Jing
Seok-Hee Hong
Dacheng Tao
Venue: NeurIPS 2024 poster (2024)
Keywords: Efficient Learning
Graph Neural Networks
Primary area: graph_neural_networks

Abstract: In this paper, we introduce the first learning-free model reuse task within the non-Euclidean domain, termed as Deep Graph Mating (Grama). We strive to create a child Graph Neural Network (GNN) that integrates knowledge from pre-trained parent models without requiring re-training, fine-tuning, or annotated labels. To this end, we begin by investigating the permutation invariance property of GNNs, which leads us to develop two vanilla approaches for Grama: Vanilla Parameter Interpolation (VPI) and Vanilla Alignment Prior to Interpolation (VAPI), both employing topology-independent interpolation in the parameter space. However, neither approach has achieved the anticipated results. Through theoretical analysis of VPI and VAPI, we identify critical challenges unique to Grama, including increased sensitivity to parameter misalignment and further the inherent topology-dependent complexities. Motivated by these findings, we propose the Dual-Message Coordination and Calibration (DuMCC) methodology, comprising the Parent Message Coordination (PMC) scheme to optimise the permutation matrices for parameter interpolation by coordinating aggregated messages, and the Child Message Calibration (CMC) scheme to mitigate over-smoothing identified in PMC by calibrating the message statistics within child GNNs. Experiments across diverse domains, including node and graph property prediction, 3D object recognition, and large-scale semantic parsing, demonstrate that the proposed DuMCC effectively enables training-free knowledge transfer, yielding results on par with those of pre-trained models.

Deep Graph Mating
Yongcheng Jing'
Seok-Hee Hong' Dacheng Tao
'University of Sydney (yongcheng. jing, seokhee hong) @sydney. edu.au, au, dacheng. tacentu. edu. sg
Nanyyngg Technological University
Abstract
In this paper, we introduce the first learning-free model reuse task within the non-Buclidean domain, termed Deep Graph Mating (GRAMA). We strive create child Graph Neural Network (GNN) that integrates knowledge from pre- trained parent models without requiring re-training, fine-tuning, or annotated labels. To this end, we begin by investigating the permutation invariance property of GNNs, which leads us to develop two vanilla approaches for GRAMA: Vanilla Parameter Interpolation (VPI) and Vanilla Alignment Prior to Interpolation (VAPI), both employing
interpolation in the parameter space. How- ever, neither approach has achieved the anticipated results. Through theoretical analysis of VPI and VAPI, we identify critical challenges unique to GRAMA, cluding increased sensitivity to parameter misalignment and further the inherent topology-dependent complexities. Motivated by these findings, we propose the Dual-Message Coordination and Calibration (DuMCC) methodology, comprising the Parent Message Coordination (PMC) scheme to optimise the permutation ma- trices for parameter interpolation by coordinating aggregated messages, and the Child Message Calibration (CMC) scheme to mitigate over- mmo-shig identified PMC by calibrating the message statistics within child GNNs. Experiments across diverse domains, including node and graph property prediction, 3D object recog- nition, and large-scale semantic parsing, demonstrate that the proposed DuMCC effectively enables training- knowledge transfer, yielding results on par with those of pre-trained models.
Introduction
The remarkable progress made in deep neural networks has resulted in growing number of pre- trained models being made publicly available for the purpose of performance reproducibility and As such, there is a mounting interest in the community on the reusability of existing pre-trained neural networks for the sake of strengthening performance, reducing model size, or alleviating training efforts, with abundant inspiring works proposed [16] [42] 46 40, 47 45| 63 67]. Despite the growing interests in model reuse, current research endeavors have predominantly focused on the Fuclidean domain, which are specifically designed to handle image
further development [64] 12 27
23
61
24.
data with regular grid structures [39] 64
22
On the other hand, the study of reusing pre-trained Graph Neural Networks (GNNs) to tackle non- Euclidean and irregular graph data is still in its early stage and remains limited in scope. Almost all existing research on GNN reuse is established upon non-Euclidean Knowledge Distillation (KD)
pioneered by Yang al. [60] where a favourable student GNN is learned from a single pre-trained
teacher [1] 49 9 66 29). Subsequent research has extended the scope of Yang et al. [60 from
single-teacher to a
context, introducing the novel task of non- -uncliidean Knowledge
Amalgamation (KA) 25 15 36). However, all these existing approaches to GNN reuse necessitate
resource-intensive re-training the student model, imposing substantial computational burdens and
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Graph-Centrel Model Reuse Tasks
Multi- model Reuse Annotation Free Freiiniainnn//iin -uninn Free
Knowledge Distillation [60]
11
49
9
66
29
Knowledge Amalgamation 25
15
36
Deep Graph Mating (GRAMA)
Table 1:
Comparison of various model reuse tasks in the non-Euclidean domain, tailored for GNNs.
memory costs. This challenge from re- re-tranni is particularly pronounced when dealing with large
models and large-scale graphs [31 7 37 20
8 26 17
In this paper, we strive to push the boundaries of rescurce-effiiiet GNN reuse by introducing the first training-free model reuse task in the non-Euclidean domain, termed as Deep Graph Mating (GrAMA). Our objective is to derive a child GNN, withoul re-training or fine-tuning, from pre-trained parent GNNs, each possessing unique knowledge from different datasets, all while operating without access to human-annotated labels-a common constraint using publicly available models. The child model born from GRAMA is expected to scamlessly integrate the diverse expertise of its parent models in a completely learning-free free manner. A comparative analysis of GRAMA with existing non-Euclidean model reuse approaches is presented in Tab. I As pilot study of this novel task, this paper focuses on homogeneous GRAMA scenarios with identical parent architectures, and sets the stage for future investigations into the more complex cross- crosstacchieectree heterogeneous GRAMA.
To achieve the ambitious goals of GRAMA, we first investigate the permutation invariance property of GNNs to establish correspondence between neurons in pre-trained models. This investigation guides the development of two vanilla methods: Vanilla Parameter Interpolation ( (VPI) and Vanilla Alignment Prior to Interpolation (VAPI). VPI involves straightforward linear interpolation of parent GNN parameters, while VAPI incorporates data-indep inte-eindeenn parameter matching before interpolation. However, the performance of both VPI and VAPI has not been as promising as expected. By analysing the mechanisms underlying these methods, we have identified and theoretically demonstrated unique challenges associated with GRAMA, including increased susceptibility to parameter misalignment and deeeloyg-deeeeeentt complexity in GNNs. These findings underscore the necessity for tailored methods that are specifically adapted to GRAMA.
To this end, we introduce the Dual-Message Coordination and Calibration (DuMCC) methodology, specifically tailored for GRAMA to incorporate the topological characteristics of graphs. DuMCC consists of two distinct schemes: Parent Message Coordination (PMC) and Child Message Calibration (CMC). PMC seeks to identify optimal permutation matrices for parameter matching in a topology- aware manner, by coordinating aggregated messages from parent GNNs. Although PMC shows promising results, our empirical and theoretical analyses indicate that the child GNN derived from this coordination is more prone to over-smoothing. To mitigate this issue, we propose the CMC scheme, which calibrates the message statisties of the child GNN using specialised, learning-free message normalisation (LFNorm) layer, drawing on the statistics of the parent GNNs. Together, these two schemes contribute to training-free and label-free GRAMA process, enabling the derivation of child GNNs that effectively embody the knowledge from their pre-trained parent models.
In summary, our contribution is a novel non-Euclidean model reuse paradigm that allows for the creation of student GNN, which integrates the capabilities of pre-trained parent GNNs without requiring muman-annotate labels re-training. We evaluated our proposed approach seven benchmarks across various tasks, including node/graph classification, object recognition, and large- scale indoor semantic segmentation, and across five different GNN architectures: Graph Convolutional Network (GCN) 30 GraphSAGE 13], Graph Attention Network (GAT) [48). Graph Isomorphism Networks (GIN) (56], and Dynamic Graph CNN (DGCNN) [53]. Experimental results demonstrate that the generated child GNN is competent to handle the pre-trained tasks of parent models. We also present discussion of the limitations, highlighting potential future research directions that can be explored based on GRAMA.
2 Related Work
This section briefly reviews topics relevant to GRAMA, including existing GNN reuse techniques and model merging, which, while conceptually akin to GRAMA, is confined to the Fuclidean domain. Extended related work is provided in Appendix A.
2

---

Non- Non-Euclidean Model Reuse. Recent advancements in graph-centric intelligence have led to the widespread availability of pre-trained models for reproducibility. Despite their availability, the exploration of their reuse for downstream tasks, particularly in the non-Euclidean domain, still in its infaney. Existing research primarily revolves around knowledge distillation (KD) [60, [9 666 11 49 [9] and knowledge amalgamation (KA) [25 15, 36), aimed at single-GNN and multi-GNI reusing, respectively. In particular, the foundational KD research [60] introduced a method specifically designed for GNNs that conserves local structural integrity. The scheme was later expanded by KA (25) to adapt to multi-teacher settings without the need for labels, thereby enhancing GNN reuse capabilities. Extending these concepts, this paper launches GRAMA, new GNN reuse paradigm that further boosts resource efficiency by obviating the need for re- re-training.
Model Merging. Recent years have seen a surge in interest in merging the weights of CNNs or
transformers into a single model [54 43
19
59
35
62
38 21]. Model merging is
typically categorised into two types: fine-tuned model merging and variably initialised model merging Merging models fine-tuned from the same initialisation is generally straightforward as these models often reside within the same error basin, allowing for simple weight interpolation [54]. Conversely, merging models from different initialisations is more challenging due to the randomness in network channels and components [33
33
43 []. A key issue is aligning neurons between models to establish correspondences before weight interpolation. To address this issue, Git Re-basin M proposes to minimise the L2 distance between weight vectors. Singh and Jaggi 43 propose an Optimal Transport Fusion (OTFusion) method that uses the Wasserstein distance to align weight matrices prior performing parameter fusion. The subsequent work [19] extends the application of OTFusion to Transformer- based architectures, aiming to enhance efficiency and performance through fusion. Also, Liu et al. 33 approach the challenging task of model fusion as a graph matching problem, incorporating second-order parameter similarities for improved fusion performance.
Addressing the fusion of pre-trained models trained on disparate tasks, Stoica et al. [44] develop Ziplt!, a novel method that utilises a "zip" operation for layer-wise fusion based on feature redundancy, creating a versatile multi-task model without additional training. More recently, Xu et al. (57) present the Merging under Sual-Space Constraints (MuDSC) framework. This approach optimises permutatioon matrices by mitigating inconsistencies in unit matching across both weight and activation spaces, targeting effective model fusion in multi-task scenarios. Furthermore, Jordan et al. [28] introduce REPAIR, method that tackles the issue of variance collapse by rescaling the hidden units in the interpolated network to restore the statistical properties of the original networks.
However, model merging has not yet been explored within the context of GNNs. This paper introduces GRAMA, the first formulation for weight space model merging tailored for the non- non-Euulidean domain. It represents the initial investigation into merging GNNs, addressing the unique challenges that arise with graph tasks.
3 Motivation and Problem Definition
In this section, we begin by introducing Knowledge Amalgamation (KA), the sole existing task in multi-GNN reuse, exploring its inherent limitations, and present our novel GRAMA paradigm for resource-efficient multi-model reuse with practical applications in real-world scenarios.
To the best of our knowledge, KAA represents the only muultimodee reuse task in the non- non-Euclidean domain. KA is defined in the literature follows: its objective is to learn single, compact student GNN that integrates the diverse expertise of pre-trained teacher GNNs, without accessing human annotations (25] Despite the promising performance, KA is inherently limited by its resource- intensive nature, requiring the re- re-training of a student GNN to amalgamate knowledge from existing GNNs. Furthermore, while KA ostensibly operates without ground-truth labels, it instead relies on soft labels generated by teachers, making it susceptible to the misclassification errors of teachers.
To mitigate these constraints of KA, this paper proposes:
Task 3.1 (Deep Graph Mating). Deep Graph Mating (GRAMA) is a fully learning-free free model reuse task where a child GNN is derived from pre-trained parent GNNs without re-training or fine-tuning, integrating their expertise without requiring Human-annotater labels.
In aggregate, GRAMA advances beyond KA by eliminating the need for any training or label dependency, paving the way for more widespread and versatile model reuse applications. Given the
3

---

novelty and complexity of GRAMA, our initial investigation in this paper is confined to scenarios where pre-trained GNNs possess identical architectures yet are trained on separate datasets, termed as homogenous GRAMA. We reserve the exploration of more challenging heterogeneous GR (RAMMA A scenarios, where pre-trained parent models either have diverse architectures or are designed for different domain tasks, as topic for future studies, potentially incorporating solutions like partial GRAMA inspired by the work of Stoica et al. [44].
The applications of homogenous GRAMA are especially crucial in contexts where full data access for training is restricted due to privacy concerns and regulatory requirements. This common in sectors like healthcare or retail, where organisations operate across different regions, each gathering data that cannot be centrally aggregated due local privacy regulations. The proposed homogenous GRAMA paradigm enables the seamless integration of knowledge from these isolated datasets, thereby safeguarding against the risks of privacy violations and the disclosure of sensitive data.
4 Vanilla Methodologies and Challenge Pre-analysis
4.1 Two Vanilla GRAMA Methods
Vanilla Parameter Interpolation (VPI). To achieve the ambitious goal of GRAMA outlined in Sect. 3 the initial naïve approach employs vanilla weight averaging [50] 54]. This method involves a straightforward linear interpolation of weights Wa and W. from two pre-trained GNNs: W((-- =
( + +1 ( where represents the interpolation weight and W( denotes the network
weights at layer
However, the vanilla averaging approach requires the pre-trained models to share portion of their training trajectory and remain sufficiently close in the parameter space [}] typically achieved by fine-tuning from the same initial model. This is not applicable in our GRAMA context, where parent GNNs are trained on distinct datasets. This mismatch leads to empirically poor performance for GRAMA, as observed in our experiments.
Vanilla Alignment Prior to Interpolation (VAPI). To address this issue in vanilla interpolation, previous research in the Euclidean domain, based on the conjecture of the permutation invariance property of typical neural networks proposes aligning the neurons between pre-trained models by permuting parameter matrices before performing linear interpolation [I] The alignment and interpolation process can be formulated as:
W = ( + (1
( p(o E P""
(1)
where P [P((]
[E[r] represents the set of all permutation matrices p(0) for each layer E of the GNN. Here, [L] refers to the set of indices corresponding to all layers in the GNN.
However, to apply Eq. [ to GNN-based GRAMA, it is essential to first discuss whether the permuta- tion invariance property extends to GNNs. This property has been extensively studied in existing literature for various architectures 10 5 43
1 33 19] including multi-layer perceptrons (MLPs), convolutional neural networks (CNNs), and Transformers. Supported by these prior studies, and
given that GNNs are fundamentally built upon MLPs 58 30
we propose:
Conjecture 4.1 (Permutation Invariance in GNNs). Permutation invariance for parameters in GNNs exists if and only if there exists a set of permutation matrices p(f) for each layer such that applying these permutations to the parameters does not alter the oulcome of graph-bused learning task, regardless of the structure of the adjacency matrix.
A key subsequent issue involves searching the optimal permutation matrix P for GNNs. One possible data-independent solution is to minimise the T-2 distance between the weight vectors of the pre-trained models by solving a sum of bilinear assignments problem, similar to weight matching techniques described in M. This method, which does not consider data distributions, could be adapted as a baseline vanilla method for our GRAMA task in the non-Euclidean domain, which is evaluated in our experiments.
Neurons in each layer of neural networks can be permuted without altering network functionality 10

---

4.2 Challenges Towards GRAMA
However, we empirically observed that even the Vanilla Aligument Prior to Interpolation method yielded unfavourable results for our GRAMA task. To elucidate the underlying cause of this phe- nomenon, we theoretically demonstrated that GNNs typically exhibit greater sensitivity to parameter mismatches than neural networks in the Euclidean domain:
Lemma 4.1 (Amplified Sensitivity of GNNs Parameter Misalignment). GNNs exhibit greater sensitivity to mismatches in parameter alignment compared to CNNs, amplified by the degree of connectivity and heterogeneity of the node features in the gruph topology.
complete theoretical proof of Lemma 4.1 is provided in Appendix G.I. Here, we present only the final formulation of the approximated output changes resulting from weight perturbations due to mismatching, based on Taylor series approximation:
AF, R
Σ W. xx (EN(()
X3
Σ c. JEN(()
(2)
where AF; refers to the change in output at node i due to the perturbations c in the weights W, and o' represents the derivative of the activation function. X, denotes the features of the nodes within
the neighbourhood (1) of node i. Detailed, model-specific formulations of Eq. 2 are provided in Sect. B of the appendix.
Eq. 2 implies that the effect of can exacerbated by the potentially large and diverse neighbour- hoods (i). thereby making the output highly sensitive to changes in W. In other words, the effect of weight perturbation can vary dramatically based on the node's connectivity and the characteristics of its neighbours. Such variability leads to significant and less predictable changes in output, illustrating the particular vulnerability of GNNs to parameter mismatches.
The integration of Eq. 1 with Lemma 4.1 and Eq. further gives rise to the following conjecture: Conjecture 4.2 (Topology-depenn))
Complexity GNNs). The identification of optimal permu- tation matrices P for GNNs presents increased complexity compared to the Euclidean domain, contingent upon the topological characteristics inherent to each graph.
Conjecture 4.2 highlights the essential need for developing GRAMA methods that are specifically tailored to accommodate the unique topologies of graphs, motivating the design of the proposed approach in Sect. 5)
Proposed Approach: Dual-Message Coordinator and Calibrator
5.1 Overview
Motivated by Conjecture 4.2 we introduce in this section the proposed Dual-Message Coordina- tion and Calibration (DuMCC) C)) methodology, which is specifically designed to harness the unique topological features of input graphs for achieving GRAMA without relying on human annotations.
The proposed DuMCC is composed of two strategic schemes. In particular, the first Parent Message Coordination (PMC) scheme effectively integrates topological information by deriving optimal permutation matrices from layer-specific aggregation results. However, both empirical and theoretical analyses reveal reduction in node feature variance in child GNNs, suggesting that models derived through this coordination are more susceptible to over-smoothing compared to their parent GNN counterparts.
To address this issue, we further propose the Child Message Calibration (CMC) scheme as our second strategic component. This scheme aims to maintain message variance consistency from the parent models, ensuring the retention of feature diversity essential for robust GNN performance. Further elaboration on each component provided in subsequent sections.
5.2 Parent Message Coordination Scheme
Motivated by Conjecture 4.2 which highlights the significance of incorporating topology information in the GRAMA process, propose a Parent Message Coordination (PMC) scheme for identifying
5

---

optimal topology- aware permutation matrices P described in Eq. I Unlike the vanilla method of VAPI in Sect. that minimises the distance between weight vectors without considering the input graphs topologies, PMC optimises P* by leveraging the topology information embedded in aggregated messages from two pre-trained parent GNN models.
Assume we have two pre-trained parent GNNs, denoted by Ga and G.. both sharing identical architectures. Their corresponding weight matrices are denoted by W. and W6, respectively. To establish correspondence between neurons in Wa and W, as described in Eq. [ our PMC optimises the permutation matrix p(() at layer é by aligning the aggregated messages of the two parent GNNs. To align with the notations [ 33 used in model merging within the Fuclidean domain, the associated optimisation process can be formulated as follows:
p(0) arg min pleEP
E, Agg @ (Wie-D) xx(((-11 ,es)) jEN((1
(3)
-P(O)
Agg @ (w x( .eis) 11:
JEN(1)
where & denotes the message function that encodes both node features X and edge features e, and Agg represents the message aggregation function that accumulates incoming messages from Φ. acting on each node i. (i) denotes the set of neighbours of node i. The minimisation problem in the form of Eq. can be typically transformed into maximisation problem maximise an inner product (as derived from expanding Eq. P). thereby fitting it within the framework of standard linear assignment problem, as also done the works of [T 35 44].
Eq. is based on the rationale that aggregated messages inherently encapsulate essential graph topologies, and that structurally similar GNNs typically generate analogous aggregated messages when tasked with similar graph operations and topologies. As such, through Eq. 3 P can be determined in topology-aware manner, matching aggregated messages reflect the topological characteristics of the graphs. Here, we clarify that while our approach involves passing the graph data to the pre-trained model to capture the graph-specific topological characteristics by utilising X, it requires only a single forward pass of the unlabelled graph data to extract messages for aligmment- seauuentt-limmiaiinn.. the need for iterative training or ground -ttuh labels. Subsequently, child GNNs can be derived through linear parameter interpolation.
Despite the encouraging performance, observe that the child GNN, derived from the proposed PMC, exhibits a reduction in the variance of node embeddings. We conjecture that this reduction stems from an averaging effect, which may smooth out the distinctive features captured by each parent model, particularly when these models have learned different structural aspects of the graph:
Lemma 5.1 (Variance Reduction in Interpolated Graph Embeddings). The variance of the graph embeddings in an interpolated child GNN is rypically smaller than the variances of the embeddings from the individual pre-trained parent GNNs.
The full proof of Lemma 5. is detailed in Appendix G.2 providing evidence of feature homogeni- sation within child GNNs. In the context of GNNs, we further explore and establish the following proposition:
Proposition 5.1 (Increased Susceptibility to Over-Smoothing in Child GNNs). Interpolated child GNNs exhibit increased susceptibilityy to over-smoothing compared their parent networks, as measured by Dirichlet energy.
The detailed proof of Proposition 5.1 is provided in Appendix G.3, utilising the quantitative over- smoothing measurement based on Dirichlet energy [41]. where a lower value indicates greater homogeneity or smoothness among node features. In particular, our theoretical analysis in Ap- pendix G.3 demonstrates that:
E(X') S max (E(X))
(4)
where E(X*) denotes the Dirichlet energy for the node features X at layer É of the child GNN.
In GRAMA, the parameter a typically set 0.5 to ensure unbiased knowledge integration from both pre-trained models. This setting promotes a balanced contribution from each model and prevents any bias toward the characteristics of one over the other. As detailed in Appendix G.3, with a at this level, the Dirichlet energy of the interpolated child GNN, E(X"), significantly decreases

---

compared to the Dirichlet energies of the individual par- ent models, indicating higher susceptibility to over- smoothing.
Child GNN (PMC)
Parent GNN
0.80
0,46
Corresponding to this theoretical analysis, empirical ev- idence is presented in Fig. where 60 parent models are pre-trained on distinct partitions of the dataset with different random seeds. Additional imple- mentation details are provided the appendix. Fig. Fig. fur- ther demonstrates the increased smoothing effect, which can potentially diminish the model's expressive power and discriminative capability.
0.75
0.44
0.70
0.42
0.7
0.8
Parent GNN
Figure 1: Comparison of Dirichlet en- ergies between pre-trained parent GNNs and the corresponding child from PMC.
5.3 Child Message Calibration Scheme
To mitigate the smoothing issue identified in Proposition 5.1 one potential solution involves leveraging established methods designed to address over-smoothing, such as PairNorm [65] and residual connections [32] However, to the best of our knowledge, all these existing solutions typically require re-training the model. This requirement contradicts the fundamental principle of our GRAMA approach, which aims for training-free -fre model reuse.
To address this issue, we introduce Child Message Calibration (CMC) scheme designed to refine the message statistics of the obtained child GNN without the need for re-training ground-truth labels. Central to this scheme is our Learning-Free Free Message Normalisation (I.FNorm) layer specifically tailored for our GRAMA task, inspired by 28). This layer is intended to enhance the diseriminative power and representational capacity of the child GNN by promoting more diverse node feature distribution.
According to Proposition 5.1 while the shift in mean node features is typically less problematic than variance reduction, it is noted in [6] that mean statisties in GNNs also carry vital graph structural information. Therefore, our approach aims to simultaneously refine both mean and variance statisties in the child GNN, ensuring a comprehensive enhancement of topological representation.
Towards this end, we first process the raw graph through the parent GNNs to compute the target message mean and variance intended for alignment in the child GNN. Subsequently, we integrate the LFNorm layer into the child GNN to refine the message statistics using the derived mean and variance from the parent models:
Var(m)) + Var(m!j)
B(mf)) ) + B(m!))
mmj (mi--NNG) KNG (N(())
m4j
(5)
2
2
where may 2 dieNto (w(t-)) 1); x(xx((11) ,,y) represents the messages from node j to node i, as
specified in Eq. 3 In this setup, mij denotes the statistically calibrated message within the child
GNN, while (s)(s) and (N(() represent the mean and standard deviation, respectively, of the messages directed to node in the child GNN.
ensures that the normalised messages in the interpolated child model maintain a balanced representation of central tendencies from both pre-trained parent models. This method effectively reduces the risk of over-smoothing in child GNNs by preserving essential topological statistics from the parent models.
more detailed algorithmic procedure is outlined in Alg. [ practice, we find that incorporating single LFNorm layer and aligning the overall message mean and variance of the parent GNNs is typically sufficient achieve favourable performance while minimising computational costs.
6 Experiments
We evaluate the performance of DuMCC across seven benchmarks spanning five GNN architectures. More ablation studies and sensitivity analyses, additional results and implementation details, as well as more visualisations, are detailed Secs. D and of the appendix.
7

---

Algorithm The proposed Dual-Message Coordinator and Calibrator (DuMCC) for GRAMA.
Input:
Pre-trained Parent GNNs Ga and 9.. interpolation factor Output: Child GNN & that integrates the expertise of Ga and 9 learnin-frree manner.
// Parent Message Coordination foreach layer & from 1 to L do
Extract weights W!4) from Gui Extract weights W² from Go
// Compute permutation matrix for current layer with aggregated messages ; .es)
p(0) arg min p.
E,
Agg jEN(()
p().
Agg (e
xx( (Wi x((-1) , ||²
// Interpolate weights for current layer (p
W <- aWa (1 aa)aa(lw (pit-1)yT
end
G [W(0)
// Child Message Calibration foreach layer É from (L- n) to L do
foreach edge (i, in the graph do
// Compute the messages in Go and
m, + b3 (W!"
:x/("") x jaa exy): my + (W "ix"l"") 1 eeg)
// Compute the message for g
my + (jje 1) ";x((-), .ey)
// Compute the scale and shift parameters
3 (Var(mm))) + Var(m()))/2211 + (B(m))) + H(mgg))
// Learning-free message calibration for g
mig + B ((m;j Hw(o)/axo) +7
end foreach node i the graph do
// Aggregate the calibrated messages and update features X + Agg
end end
Implementation Details. Detailed dataset descriptions and statistics provided Appendix C. For multi- class classification tasks ogbn- arxiv 17]. ogbn-products [4), and ModelNet40 (55),
Table 2: Multi-class molecule property prediction results for parent GNNs, each pre-trained on disjoint partitions of the agnn-arxiv and -eonuutss datasets [8]
Re- train?
ogba- arxiv
ogba- products
Methods
Dataset A Dataset
Dataset C Datasel D
adopt the dataset partition strategy widely used in model merging within the Eu-
Parent GCN A
30
0.7193
0.5516
N/A
N/A
Purent GCN B
30
0.6564
0.7464
N/A
N/A
Parent GraphSAGE
13
N/A
N/A
0.7982
0.7308
clidean domain 1 28 Specifically, each dataset randomly split into two disjoint subsets: the first subset com- prises 20% of the data with odd labels and 80% with even labels, while the second subset arranged vice versa. For the semantic segmentation task on S3DIS 3 we train the two parent models using Areas 1, 2,
Parent GraphSAGE
13
N/A
N/A
0.7626
0.7904
KA 25
(Section 3
V
0,7150
0.6687
0,7973
0.7775
VPT 54
(Section
0,3486
0.4361
0.6568
0.6546
VAPI 0
(Section
0.6140
0.5752
0.5425
0.5779
Ours (w/o CMC)
0.6531
0.5957
0.7374
0.7414
CMC)
0.6645
0.6382
0.7647
0.7515
3 and Areas 2, 3, 4, 6, respectively, with Area 5 designated for testing, as also done [32). In the multi-label classification task on bebneproteins [17] one parent model is trained on nodes with odd labels and the other on nodes with even labels. Implementation follows the official codes provided by the Deep Graph Library (DGL) [52) and the original authors, including detailed architectures and hyperparameter settings. We set interpolation factor a Eq. T to 0.5 for all experiments, with sensitivity analysis provided in Sect. D of the appendix. For models originally equipped with

---

4
42
(a) KA 255
(b) VPI S4
(c) VAPI !
(d) Ours (w/o CMC) (e) Ours (w/ CMC)
Figure 2: The I-SNE visualisations of various methods on a subset comprising the first 10 classes of ogbn-arxin Additional visualisations for the remaining classes are available in Appendix E.
Methods
Re-train"
Dataset
Dataset
Parent DGCNN 53
0.9159
0.8151
Parent DGCNN 53
0.8862
0.9275
KA 25
(Section
V
0.9250
0.9283
VPI 54
Sectionn
0.4518
0.4096
VAPI
(Section
X
0.6538
0,5482
Ours (w/o CMC)
0.8326
0,8088
Ours (w/ CMC)
0.8920
0.8574
KA
VPI
VAPI
Ours
Near
Far
Table 4: Results of the point cloud classification task on ModelNet40 [55] using DGCNN, with two parent models trained on disjoint partitions.
Figure 3: Visualisations of feature space struc- tures, depicted by the distances between the red point and all other points.
normalisation layers, we recompute the running mean and running variance for the student GNN. In particular, the recomputation of statistics is performed concurrently with that of the LFNorm layer for the child GNN in CMC. More implementation details are elaborated in Sect. E of the appendix.
Comparison Methods. Considering the limited exploration multi-GNN reuse in existing literature, focus our comparison of the proposed DuMCC approach on the seining-dependent KAA (23 and training-free VPI [54] and VAPI [1] methods, as introduced in Sect. 4 Furthermore, in Sect. of the appendix, also provide the results of retraining child model using combined parent datasets with ground-truth labels, which establish an upper bound for GRAMA'S performance.
Node Property Prediction. Tab. presents the results for the multi-class node classification task. The proposed DuMCC framework, as shown in the table, achieves a more balanced performance across all datasets. It notably outperforms the parent models datasets that were not used for their training. Additionally, the last two lines of Tab. detail an ablation study on the proposed CMC scheme, demonstrating its ability to enhance performance beyond PMC. While lags behind KA in performance, KA involves a complex re- re-training process, whereas our approach is completely training- free. A sample t-SNE visualisation of the results is provided in Fig. 2
method slightly
We further show in Tab. the results of multi-label molecule property prediction. Notably, our approach slightly outperforms KA in Tab. 3 underscoring the limitations of KA discussed in Sect. 3
These limitations stem from KA's reliance on soft labels produced by the teacher GNNs, making vulnerable to their misclassification errors. Furthermore, we explore the potential of multi-model GRAMA by concurrently reusing three pre-trained node classification GNNs, as discussed Sect. of the appendix.
Graph Property Prediction. Tab. also presents results for the graph classification task using GAT architectures. The proposed DuMCC demonstrates enhanced equilibrium in performance relative to the parent models. Notably, DuMCC slightly outperforms KA on egbg-molbas further illustrating
Table 3: Results for multi-label node classification and graph classification, indicating KA's vulnera-
bility to misclassification errors from pre-trained models.
Architectures
GIN [56]
Architectures
GAT [48]
Methods
Parent E Parent F
KA
Ours
Methods
Parent G
Parent II
KA
Ours
ogbg-molban
0.7247
0.4067
0.6135
0.6296
ogbn-proteins
0.7478
0.7222
0.7215
0.7341
ogbg-molbbbp
0.4681
0.6366
0.6446
0.5275

---

Table 5: Results of the 3D semantic segmentation task on the S3DIS dataset B3. with detailed per-class results provided. Architecture details can be found in Sect. E of the appendix.
Re- train?
Structural Elements
Methods
ceiling
floor
wall
heam
column
window
door
mcan
Parent DGCNN K
53
0.9655
0,9947
0.9355
0.0079
0.0557
0,4529
0.1430
0.5079
Parent DGCNN L
53
0.9529
0.9927
0.9546
0.0573
0.0661
0.3555
0.1335
0.5018
KA [25] (Section
V
0,9580
0.9943
0.9003
0.0681
0.1835
0,5154
0.7048
0.6178
VPI [54 (Section
0.6909
0,9871
0.3612
0.0000
0.0044
0.0000
0.0037
0.2925
VAPI
(Section
X
0,5338
0.8766
0.6825
0.0382
0.0046
0.0284
0.0019
0,3094
Ours (wo CMC)
0.9804
0.9967
0,9186
0.0302
0.0000
0.0008
0.0752
0,4289
Ours (w/ CMC)
0.9695
0.9962
0.9290
0.1466
0.0004
0.0344
0.1047
0.4544
Re-
Furniture
Others
Overall
Methods
train?
table
chair
sofa
bookcase
board
mean
clutter
mean
Parent DGCNN K 53
0.7182
0.7746
0.0216
0.4980
0.4022
0.4829
0.6515
0.8181
Parent DGCNN L
53
0.7447
0.9255
0.1301
0.5267
0.1774
0.5009
0.6038
0.8174
KA 25
(Section
0.7358
0.8649
0.0295
0.5377
0.4211
0.5178
0.6844
0.8382
VPI 54
(Section
X
0.0159
0.2802
0.0019
0.0135
0.0001
0.0623
0,4188
0.4791
VAPI
(Section
0.0117
0.1984
0.0142
0.1235
0.0025
0.0701
0.2326
0.5060
Ours (wo CMC)
0.5953
0.8485
0.0015
0.1160
0.0009
0.3125
0.5279
0.7497
Ours (w/ CMC)
X
0.6351
0.8089
0.0086
0.2867
0.0182
0.3515
0.5274
0.7676
that KA is vulnerable to the errors of pre-trained models. In contrast, our approach does not rely on soft labels, thus avoiding this limitation.
3D Object Recognition and Semantic Parsing. Tab. and Fig. Billustrate the quantitative results and qualitative visualisations for the point cloud classification task, respectively. The proposed DuMCC outperforms model on dataset and model J on Dataset I without requiring re-training. Moreover, Tab. 4 demonstrates that our approach significantly outperforms two vanilla methods. Fig. 3 further illustrates the structure of the feature space, revealing that our method produces semantically similar structures to those achieved by KA with re-training. We also show in Tab. 5 the results for the large-scale indoor semantic segmentation task. Our method notably surpasses other learning-fre free GNN reuse methods VPI and VAPI. Further qualitative and quantitative results across various dataset splits and network architectures are provided in Sect. E of the appendix.
7 Conclusions and Limitations
In this paper, we explore a novel GRAMA task for learning-free free GNN reuse. The child model from GRAMA is expected to functionally merge knowledge from pre-trained parent models. Uniquely, GRAMA establishes the first paradigm in GNN reuse that operates entirely without re-training or fine- tuning, while also climinating the need for ground-truth labels. To this end, we start by developing two vanilla GRAMA approaches, which reveal specific challenges inherent to GRAMA. These challenges motivate us to develop DuMCC framework for topology-awar model reuse, leveraging parent message coordination scheme followed by child message calibration. Experiments on node- and graph-level tasks across various domains demonstrate the effectiveness of the proposed approach for annotation-free knowledge transfer without additional learning.
Despite its strengths, the proposed DuMCC is primarily designed for homogeneous GRAMA, as discussed in Sect. 3 Currently, the framework does not support cross-architccture heterogencous GRAMA, where parent models have different architectures, such as combination of GCN and GraphSAGE. Additionally, it does not handle scenarios where parent models address tasks at differ- ent levels, such as node-level versus graph-level tasks another aspect of heterogeneous GRAMA. These limitations primarily arise from the absence of direct correspondence between the differing architectural layers of the parent models, an issue we plan to explore in our future work. We will also explore the possibility of a fully data-independent GRAMA scheme and investigate broader applications beyond training-free model reuse, such as its use as a pre-processing step to facilitate graph-based knowledge amalgamation. Further discussions on limitations and potential solutions are provided in Sect. H of the appendix.
10

---

Acknowledgement
This research project is supported by the National Research Foundation, Singapore, and Cyber Security Agency of Singapore under its National Cybersecurity R&D Programme and CyberSG R&D Cyber Research Programme Office, as well as Australian Research Council Discovery Project DP190103301.
Any opinions, findings and conclusions or recommendations expressed in these materials are those of the author(s) and do not reflect the views of National Research Foundation, Singapore, Cyber Security Agency of Singapore as well as CyberSG R&D Programme Office, Singapore.
References
[1] Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In ICLR, 2023.
[2] Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. arXiv preprint arXIv:2403.13187, 2024.
[3] Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In CVPR, 2016.
[4] K. Bhatia, K. Dahiya, H. Jain, P. Kar, A. Mittal, Y. Prabhu, and M. Varma. The extreme classification repository: Multi-label datasets and code, 2016.
[5] Johanni Brea, Berfin Simsek, Bernd Illing, and Wulfram Gerstner. Weight-space symmetry in deep networks gives rise permutation saddles, connected by equal-loss valleys across the loss landscape. arXiv preprint arXiv: 0000100...1.......000000...... 2019.
[6] Tianle Cai, Shengjie Luo, Keyulu Xu, Di He, Tie-yan Liu, and Liwei Wang. Graphnorm: A principled approach to accelerating graph neural network training. In ICMI., 2021.
[7] Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In ICML, 2020.
[8] Avery Ching, Sergey Edunov, Maja Kabiljo, Dionysios Logothetis, and Sambavi Muthukrishnan. One trillion edges: Graph processing at facebook-scale. Proceedings of the VLDB Endowment, 2015.
[9] Xiang Deng and Zhongfei Zhang. Graph-free knowledge distillation for graph neural networks. In IJCAI, 2021.
[10] Rahim Entezari, Hanie Sedghi. Olga Saukh, and Behnam Neyshabur. The role of permutation invariance linear mode connectivity of neural networks. In ICLR, 2022.
[11] Kaituo Feng, Changsheng Li, Ye Yuan, and Guoren Wang. Freekd: Free-direction knowledge distillation for graph neural networks. In KDD, 2022.
[12] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: survey. IJCV, 2021.
[13] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NeurIPS, 2017.
[14] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, al. A survey vision transformer. TPAMI, 2022.
[15] Yunzhi Hao, Yu Wang, Shunyu Liu, Tongya Zheng, Xingen Wang, Xinyu Wang, Mingli Song. Wenqi Huang, and Chun Chen. Attribution guided layerwise knowledge amalgamation from graph neural networks. ICONIP, 2023.
[16] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015.
11

---

[17] Weihua Hu, Matthias Fey. Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. Ogb-Isc: A large-scald scale challenge for machine learning on graphs. In NeurIPS Datasets and Benchmarks, 2021.
[18] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXhv:2005.00653 2020.
[19] Moritz Imfeld, Jacopo Graldi, Marco Giordano, Thomas Hofmann, Sotiris Anagnostidis, and Sidak Pal Singh. Transformer fusion with optimal transport. In ICLR, 2024.
[20] Wei Jin, Lingxiao Zhao, Shichang Zhang, Yozen Liu, Jiliang Tang, and Neil Shah. Graph condensation for graph neural networks. In ICLR, 2022.
[21] Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In ICLR, 2023.
[22] Yongcheng Jing. Efficient representation learning with graph neural nelworks. PhD thesis, 2023.
[23] Yongcheng Jing. Yining Mao, Yiding Yang. Yibing Zhan, Mingli Song, Xinchao Wang, and Dacheng Tao. Learning graph neural networks for image style transfer. In ECCV, 2022.
[24] Yongcheng Jing, Xinchao Wang, and Dacheng Tao. Segment anything in non-euclidean domains: Challenges and opportunities. arXiv preprint arXiv 1155, 2023.
[25] Yongcheng Jing, Yiding Yang, Xinchao Wang, Mingli Song, and Dacheng Tao. Amalgamating knowledge from heterogeneous graph neural networks. In CVPR, 2021.
[26] Yongcheng Jing, Yiding Yang, Xinchao Wang, Mingli Song, and Dacheng Tao. Meta-aggregator: Learning to aggregate for 1-bit graph neural networks. In ICCV, 2021.
[27] Yongcheng Jing, Chongbin Yuan, Li Ju, Yiding Yang, Xinchao Wang, and Dacheng Tao. Deep graph reprogramming. In CVPR, 2023.
[28] Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. Repair: Renormalizing permuted activations for interpolation repair. In ICLR, 2023.
[29] Chaitanya K Joshi, Fayao Liu, Xu Xun, Jie Lin, and Chuan Chuan-Sheng Foo. On representation knowledge distillation for graph neural networks. arXiv preprint arXiv:21 1.04111..4444444 2021.
[30] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
[31] Guohao Li, Matthias Müller, Guocheng Qian, Itzel C Delgadillo, Abdulellah Abualshour, Ali Thabet, and Bemard Ghanem. Deepgens: Making gens go as deep as cnns. TPAMI, 2021.
[32] Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem. Deepgens: Can gens go deep as cnns? In ICCV, 2019.
[33] Weishi Li, Yong Peng. Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey, arXiv preprint arXivv2309. 2023.
[34] Sihao Lin, Hongwei Xie, Bing Wang, Kaicheng Yu, Xiaojun Chang, Xiaodan Liang, and Gang Wang. Knowledge distillation via the target-aware transformer. In CVPR, 2022.
[35] Chang Liu, Chenfei Lou, Runzhong Wang, Alan Yuhan Xi, Li Shen, and Junchi Yan. Deep neural network fusion via graph matching with applications to model ensemble and federated learning. In ICML, 2022.
[36] Haibo Liu, Di Zhang, Liang Wang, and Xin Song. Multi-teacher local semantic distillation from graph neural networks. ADMA, 2023.
[37] Xin Liu, Mingyu Yan, Lei Deng, Guoqi Li, Xiaochun Ye, Dongrui Fan, Shirui Pan, and Yuan Xie. Survey on graph neural network acceleration: An algorithmic perspective. In IJCAI, 2022.
12

---

[38] Michael S Matena and Colin A Raffel. Merging models with fisher- averaging. In NeurIPS, 2022.
[39] Yu Pan, Ye Yuan, Yichun Yin, Zenglin Xu, Lifeng Shang, Xin Jiang, and Qun Liu. Reusing pretrained models by multi-linear operators for efficient training. In NeurIPS, 2023.
[40] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550 2014.
[41] T Konstantin Rusch, Michael Bronstein, and Siddhartha Mishra. A survey on oversmoothing in graph neural networks. SAM Research Report, 2023.
[42] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 1606. 04671, 2016.
[43] Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. In NeurIPS, 2020.
[44] George Stoica, Daniel Bolya, Jakob Brandt Bjomner Pratik Ramesh, Taylor Hearn, and Judy Hoffmaan. Zipit! merging models from different tasks without training. In ICLR, 2024.
[45] Xiu Su, Shan You, Fei Wang, Chen Qian, Changshui Zhang, and Chang Xu. Benet: Searching for network width with bilaterally coupled network. In CVPR, 2021.
[46] Xiu Su, Shan You, Jiyang Xie, Fei Wang, Chen Qian, Changshui Zhang, and Chang Xu. Searching for network width with bilaterally coupled network. TPAMI, 2022.
[47] Xiu Su, Shan You, Jiyang Xie, Mingkai Zheng, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Vitas: Vision transformer architecture search. In ECCV, 2022.
[48] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In ICLR, 2018
[49] Can Wang, Zhe Wang, Defang Chen, Sheng Zhou, Yan Feng, and Chun Chen. Online adversarial distillation for graph neural networks. arXiv preprint arXiv:2112.13366 2021.
[50] Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In ICLR, 2020.
[51] Lin Wang and Kuk-Jin Yoon. Knowledge distillation and student-teacher learning for visual intelligence: review and new outlooks. TPAMI, 2021.
[52] Minjie Wang, Lingfan Yu, Da Zheng, Quan Gan, Yu Gai, Zihao Ye. Mufei Li, Jinjing Zhou, Qi Huang, Chao Ma, et al. Deep graph library: Towards efficient and scalable deep learning on graphs. In ICLR Workshop. 2019.
[53] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay F. Sarma, Michael M. Bronstein, and Justin M. Solomon. Dynamic graph cnn for learning on point clouds. TOG, 2019.
[54] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML, 2022.
[55] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In CVPR, 2015.
[56] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR, 2019.
[57] Zhengqi Xu, Ke Yuan, Huiqiong Wang, Yong Wang, Mingli Song, and Jie Song. Training- free pretrained model merging. In CVPR, 2024.
[58] Chenxiao Yang, Qitian Wu, Jiahua Wang, and Junchi Yan. Graph neural networks are inherently good generalizers: Insights by bridging gnns and mips. In ICLR, 2023.
13

---

[59] Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang. and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning, In ICLR, 2024.
[60] Yiding Yang, Jiayan Qiu, Mingli Song, Dacheng Tao, and Xinchao Wang. Distilling knowledge from graph convolutional networks. In CVPR, 2020.
[61] Jingwen Ye, Zunlei Feng, and Xinchao Wang. Flocking birds of a feather together: Dual-step gan distillation via realer-fake samples. VCIP, 2022.
[62] Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as free lunch. arXiv preprint arXiv:2311.0309) 2023.
[63] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv: 16/2.00993 2016.
[64] Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In CVPR, 2018.
[65] Lingxiao Zhao and Leman Akoglu. Pairnorm: Tackling oversmoothing in gnns. In ICLR, 2020.
[66] Sheng Zhou, Yucheng Wang, Defang Chen, Jiawei Chen, Xin Wang, Can Wang, and Jiajun Bu. Distilling holistic knowledge with graph neural networks. In ICCV, 2021.
[67] Xiatian Zhu, Shaogang Gong, al. Knowledge distillation by on-the-fly native ensemble, In NeurlPS, 2018
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitation discussions are provided Sect.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
The full set of assumptions and a complete proof are provided in Appendix G.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yss]
Justification:
The information reproduce the experiments is provided in Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is a new model (e.g., large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer:
[Yes]
Justification:
The code and model are provided in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
ooiiioooooyyllloool for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
httppp/lllctggiieeee///eeeaaaain CoodeSubeeiieeeeeeeeee ssi asionnolle onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Implementation details are elaborated in Sect. E of the appendix. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
The statistical analysis is provided in Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Computer resources needed are detailed in Sect. E of the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //t::://nnuis/eeeeeeeeee wwweeeris.cccppppeeeeeee https///cccccccc/ccccccccoddeled
Answer: [Yes]
Justification:
The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Potential societal impacts of the work are discussed in Appendix I.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
18

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper studies GNN reuse for resource-efficient graph representation learning and thereby, to our knowledge, poses no such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The existing assets are cited and properly respected with the discussions provided Appendix
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi www.wwtththc....c com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
19

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Details of the code and model are provided in the supplementary material. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20


Reviews:
[{"review_id": "fDCiBMHLi7", "reviewer": "Reviewer_rPnJ", "summary": "The paper is interested in re-using Graph Neural Networks trained from one task to another task (e.g., transfer learning). This motivation is nice. We've seen such trends in computer vision (e.g., network trained on ImageNet used for CIFAR), or more recently, everyone is using LLMs for a variety of tasks they were not trained on (by prompt engineering, LoRA, etc). The paper proposes to do the same for graphs. Their specific methodology starts by combining multiple GNNs. To do so, they want to permute the internal channels of the graph neural network so that the networks are best-aligned, before combing them.", "questions": "- Does the exact form of Equation 2 depends on the architecture of the GNN model? I would expect to see different formulas depending on the model (e.g., GraphSAGE vs GCN of Kipf vs GAT vs MixHop etc)\n\n- Please respond to the main math weakness, which might cause you to be explicit about the notation", "limitations": "I did not spot a Limitations section.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Using Pre-trained networks can save compute resources\n* If the initial trained dataset is so large and/or private, it might not be practical to retrain the network (even with much compute resources). Using pre-trained networks would work well here.\n* Paper proposes to use multiple GNN (pre-trained) networks for a new task. The method is simple: just average the latent vectors of the GNNs. However, for this averaging to make sense, authors propose permuting the channels of the GNN latent vectors. This permutation is optimized using an objective (and algorithm).", "weaknesses": "While the paper is well-motivated and well-written, it nicely builds-up the reader's excitment to anticipate for the actual method. Once reader arrives to the well-anticipated Equation (3), reader discovers that the equation is not properly-written (\"buggy\"). I've written many papers with probably subtle bugs in them (e.g., perhaps some footnote or some text in some side-section has a bug). However, a bug in the main equation qualifies for an immediate rejection.\n\n## Math incorrectness\nAfter Eq 1:\n\n\"where $\\mathbf{P}^∗$ represents the permutation matrices\" -- what does that mean? The expression $\\mathbf{P}^∗$  is not used in the Equation. I **think** it can imply one of two things. Option 1: $\\mathbf{P}^∗ = [P^{(\\ell)} ]_{\\ell \\in [L]}$; or Option 2: $\\mathbf{P}^∗$ refers to the space of permutation matrices (e.g., a reshuffle of the identity matrix or its continuous relaxation which is probably any basis?)\n\nEquation 3 has several issues:\n\n1. Where does $i$ come from? do you mean to add $\\sum_i$ between the $\\arg\\min$ and the Frobenius norm?\n2. The aggregation function \"Agg\" is not introduced. While the exact definition is unnecessary, however, the domain and range should be clearly specified. Does it take a many vectors and output one vector? If yes, then $P \\cdot \\textrm{Agg}$ should be a vector and therefore you should be minimizing the L2 norm (not Frobenius norm). If it acts on the whole graph (i.e., $\\textrm{Agg}$ outputs a matrix with shape `NumNodes x FeatureDim`), then you are multiplying the permutation matrix from the wrong side.\n3. In general, where do the $X$s come from? I was expecting data-independent scheme. In my reading, at this point, I think paper **is \"learning** to align\" (because you are doing gradient decent on the permutation matrix by looping over data, no?)\n\n-- with all honesty, I stopped reading the paper after arriving at Eq3 and skimming over the algorithm. Please address the above and I am happy to take another look during the rebuttal. That being said, I do trust that your implementation is correct, but the paper may not indeed reflect the implementation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Response to Reviewer rPnJ (Part 1/2)**\n\nWe appreciate the reviewer's constructive comments and thoughtful suggestions, and we sincerely apologise for any confusion or ambiguity caused by our use of notations. \n\nWe are committed to addressing each of the reviewer's concerns as outlined below. Due to character constraints, we have to divide our responses into two parts. The second part will be presented as a comment appended to our initial rebuttal.\n\nAdditionally, we warmly welcome any further questions or comments the reviewer may wish to share.\n\n\n`W1.` **Clarity on $\\mathbf{P}$***\n\n>\"After Eq 1: 'where $\\mathbf{P}^*$ represents the permutation matrices' -- what does that mean? The expression $\\mathbf{P}^*$ is not used in the Equation. I *think* it can imply one of two things. Option 1: $\\mathbf{P}^* = \\left[\\mathbf{P}^{(\\ell)}\\right]_{\\ell \\in [L]}$ or Option 2: $\\mathbf{P}^*$ refers to the space of permutation matrices (e.g., a reshuffle of the identity matrix or its continuous relaxation which is probably any basis?)\n\n`Response:` We apologise for not having clearly defined the symbol $\\mathbf{P}^*$, which may have led to confusion and inadvertently extended the review process. We appreciate the reviewer's patience and constructive comments. We would like to clarify that $\\mathbf{P}^*$ denotes the set of all permutation matrices corresponding to each layer $\\ell$ of the graph neural network (GNN), which is exactly Option 1 suggested by the reviewer. \n\nWe would like to further clarify that while $\\mathbf{P}^*$ does not appear in Eq. 1, it is utilised in line 158, Eq. 3, Conjecture 4.1, and lines 208, 210, and 222. We intended to use this notation to simplify the discussions related to the collection of permutation matrices across different layers.\n\nAdmittedly, we sincerely apologise for the oversight in not explicitly linking Eq. 1 with $\\mathbf{P}^*$ and for not adequately defining and elaborating on this symbol, which resulted in ambiguity. To address this issue, we will revise Eq. 1 in line with the reviewer's constructive suggestion and enhance the description of $\\mathbf{P}^*$ in our revision by incorporating the following:\n\n\"$$\nW^{(\\ell)} = \\alpha W_{a}^{(\\ell)} + (1 - \\alpha) P^{(\\ell)} W_{b}^{(\\ell)} (P^{(\\ell-1)})^{T}, \\quad P^{(\\ell)} \\in \\mathbf{P}^*, \\tag{1}\n$$\nwhere $\\mathbf{P}^* = \\left[{P}^{(\\ell)}\\right]_{\\ell \\in [L]}$ represents the set of all permutation matrices $P^{(\\ell)}$ for each layer $\\ell$ of the graph neural network (GNN). Here, $[L]$ refers to the set of indices corresponding to all layers in the GNN.\"\n\n---\n\n`W2.` **Issues on Eq. 3**\n\n`W2.1. & W2.2.` *Clarity on $i$ and \"Agg\"* \n\n>\"Where does $i$ come from? do you mean to add $\\sum_i$ between the $\\arg \\min$ and the Frobenius norm?\"\n\n>\"The aggregation function 'Agg' is not introduced. While the exact definition is unnecessary, however, the domain and range should be clearly specified. Does it take many vectors and output one vector? If yes, then $P \\cdot \\text{Agg}$ should be a vector and therefore you should be minimizing the L2 norm (not Frobenius norm). If it acts on the whole graph (i.e., Agg outputs a matrix with shape NumNodes $\\times$ FeatureDim), then you are multiplying the permutation matrix from the wrong side.\"\n\n`Response:` We sincerely appreciate the reviewer's constructive feedback and apologise once again for the lack of clarity in our notations and the rigor of our equations. Since the issues concerning $i$ and $\\text{Agg}$ are interrelated, we are combining our responses to W2.1 and W2.2 to address these two issues collectively.\n\nOriginally, our intention was to use the symbol $i$ to universally represent the set of all nodes, rather than a single node. Consequently, $\\text{Agg}$ was meant to act on the entire graph, outputting a matrix. Here, we would like to clarify that this output matrix was shaped as FeatureDim $\\times$ NumNodes. This configuration was deliberately chosen to align with the conventions used in model merging within the Euclidean domain (e.g., Eq. 1 in [r1] and Eq. 13 in [r2], where the corresponding matrices are denoted as Dim $\\times$ Num). This alignment was intended to facilitate a unified formulation and simplify conceptually intuitive comparisons across different domains. As a result, in our original Eq. 3, we multiplied the permutation matrix from the left side to match this dimension. This choice also explains our use of the Frobenius norm, as the operation is performed on the matrix.\n\nAdmittedly, exactly as the reviewer kindly suggested, the definition and use of the symbol $i$ in this context lacks mathematical rigor, correctness, and explicitness, which subsequently led to confusion concerning $\\text{Agg}$ and the Frobenius norm. We sincerely appreciate the reviewer's thorough advice on this issue. In our revision, we will address this issue by exactly following the reviewer's suggestion: defining $i$ as a single node identifier and introducing $\\sum_i$ after $\\arg \\min$ to explicitly iterate over all nodes. This adjustment will result in the output of $\\text{Agg}$ being a vector. Accordingly, we will replace the original Frobenius norm with the L2 norm to ensure consistency with these changes.\n\nWe sincerely thank the reviewer once again for the insightful comments. We will revise Eq. 3 as described above, along with including the corresponding detailed descriptions for rigorous and explicit formulation and symbol definitions. Additionally, we will also update Eq. 3 in Alg. 1 accordingly.\n\n\n[r1] Ainsworth, et al. Git re-basin: Merging models modulo permutation symmetries. In ICLR, 2023.\n\n[r2] Li, et al. Deep model fusion: A survey. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q1QIo5bzjS", "reviewer": "Reviewer_856w", "summary": "The paper tackles pre-trained model fusion for graph-centric tasks. The pre-trained models share the same architecture, but differ in the graph datasets they are trained on. The pipeline involves two core approaches. The first one matches parameters in pre-trained parent models by aligning the aggregated messages of the pre-trained parent models. The second one modifies the message statistics for the child model to correspond with the overall statistics of the pre-trained models. The authors have validated two approaches on diverse datasets and models and show nice visualization results.", "questions": "Address weaknesses as mentioned above.", "limitations": "Yes. There are no obvious negative society impacts I can notice.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "S1. As far as I know, this is the first work to study model fusion for graph tasks. The idea is novel and has potential applicability to other networks like Transformers.\n\nS2. Throughout the paper, the author examines alternative approach and the challenges associated with each to more effectively justify their choices. \n\nS3. The authors show quantitative and qualitative results on diverse datasets for convincing validation throughout their experiments. I found the visualization results in the experiments to be nice and show the effectiveness of the approach.", "weaknesses": "W1. Discussions on existing model fusion approaches for CNNs are not adequate. The presentation will benefit from a more comprehensive literature review.\n\nW2. Lack of sensitivity analysis on the interpolation coefficient alpha.\n\nW3. It seems that each task uses only one network architecture and dataset partition. More experiments are needed with more network architectures.\n\nW4. It will be great to showcase more applications of the proposed approach besides training-free model reuse.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Response to Reviewer 856w**\n\nWe would like to thank the reviewer for the very helpful feedback. We will address each of the reviewer's comments in detail as follows.\n\n`W1.` **Inadequate literature review**\n\n>\"Discussions on existing model fusion approaches for CNNs are not adequate. The presentation will benefit from a more comprehensive literature review.\"\n\n`Response:` We appreciate the reviewer for the advice. We will elaborate on the *Model Merging* section in Sect. 2 of the main paper's related work, and add a new section in Sect. E (extended related work) of the appendix to offer a more comprehensive discussion on existing model fusion techniques for CNNs and transformers. Specifically, we plan to broaden the discussions on model fusion for CNNs and transformers in the revision, covering, but not limited to, the following:\n- Singh and Jaggi [r1] frame the task of one-shot neural network merging based purely on model weights as model fusion. They propose an Optimal Transport Fusion (OTFusion) method that uses the Wasserstein distance to align weight matrices prior to performing parameter fusion, eliminating the need for re-training;\n- The subsequent work [r2] extends the application of OTFusion to Transformer-based architectures, aiming to enhance efficiency and performance through fusion;\n- Liu et al. [r3] approach the challenging task of model fusion as a graph matching problem, incorporating second-order parameter similarities to improve the effectiveness of model fusion;\n- Addressing the fusion of pre-trained models trained on disparate tasks, Stoica et al. [r4] develop ZipIt!, a novel method that utilises a \"zip\" operation for layer-wise fusion based on feature redundancy, creating a versatile multi-task model without additional training;\n- More recently, Xu et al. [r5] present the Merging under Dual-Space Constraints (MuDSC) framework. This approach optimises permutation matrices by mitigating inconsistencies in unit matching across both weight and activation spaces, targeting effective model fusion in multi-task scenarios.\n\n[r1] Singh and Jaggi. Model fusion via optimal transport. In NeurIPS, 2020.\n\n[r2] Imfeld, et al. Transformer fusion with optimal transport. In ICLR, 2024.\n\n[r3] Liu, et al. Deep neural network fusion via graph matching with applications to model ensemble and federated learning. In ICML, 2022.\n\n[r4] Stoica, et al. Zipit! merging models from different tasks without training. In ICLR, 2024.\n\n[r5] Xu, et al. Training-free pretrained model merging. In CVPR, 2024.\n\n\n---\n\n`W2.` **Sensitivity analysis on $\\alpha$**\n\n>\"Lack of sensitivity analysis on the interpolation coefficient alpha.\"\n\n`Response:` We appreciate the reviewer's comments. In fact, the results of sensitivity analysis and the corresponding detailed discussions have already been reported in Sect. C of the appendix. We have also referenced these results of sensitivity analysis in line 292-293 of the main paper. We will further highlight these results in the revision.\n\n\n---\n\n`W3.` **Experiments with more network architectures**\n\n>\"It seems that each task uses only one network architecture and dataset partition. More experiments are needed with more network architectures.\"\n\n`Response:` We appreciate the constructive feedback from the reviewer. Following the suggestions provided, we conducted additional experiments with various network architectures and dataset splits on the large-scale ModelNet40 dataset. The results, along with detailed descriptions of the architectures used, are shown below. We will incorporate these results into the revised version of the paper.\n\n\n| Architectures | Layers | Feature Map Channels | MLP | \n|:----- |:----- | :----- | :----- | \n| Architecture-Main |   8   | [64, 64, 128, 256, 1024] | [512, 256, 40] |\n| Architecture-Rebuttal |   7   | [32, 32, 64, 128, 512] | [256, 40] |\n\n\n| Architectures | Partition | Datasets | Parent 1 | Parent 2 | Ours (w/o CMC) | Ours (w/ CMC)\n|:----- |:----- |:----- | :----- | :----- | :----- | :----- |\n| Architecture-Main | 10%/90% | Dataset 1 | 0.9246 | 0.8393 | 0.8451 | 0.8782 |\n| Architecture-Main | 10%/90% | Dataset 2 | 0.7621 | 0.9294 | 0.8374 | 0.8422 |\n\n\n| Architectures | Partition | Datasets | Parent 1 | Parent 2 | Ours (w/o CMC) | Ours (w/ CMC)\n|:----- |:----- |:----- | :----- | :----- | :----- | :----- |\n| Architecture-Rebuttal | 20%/80% | Dataset 1 | 0.9184 | 0.8920 | 0.8236 | 0.8846 | \n| Architecture-Rebuttal | 20%/80% | Dataset 2 | 0.8175 | 0.9299 | 0.8279 | 0.8550 | \n\n\n---\n\n`W4.` **Additional applications**\n\n>\"It will be great to showcase more applications of the proposed approach besides training-free model reuse.\"\n\n`Response:` We would like to thank the reviewer for the constructive comments. Here, we demonstrate an additional application of the proposed DuMCC beyond its primary role in training-free model reuse, specifically as a pre-processing step that facilitates more effective graph-based knowledge amalgamation (KA). In the table below, \"KA\" refers to re-training a student model from random initialisations by amalgamating knowledge from two teachers pre-trained on distinct partitions of the ModelNet40 dataset. \"KA + GRAMA\" denotes the process of performing KA by fine-tuning from the child model obtained via the proposed DuMCC method. The results indicate that \"KA + GRAMA\" leads to improved knowledge amalgamation performance. Additionally, since GRAMA provides a superior initialisation compared to random initialisation, \"KA + GRAMA\" empirically achieves faster convergence speed.\n\n\n| Methods | Performance: Dataset 1 | Performance: Dataset 2 | Performance: Average | \n|:----- | :----- | :----- | :----- | \n| KA | 0.9246 | 0.9270 | 0.9258 |\n| KA + GRAMA | 0.9221 | 0.9318 | 0.9269 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5wBco8AlKr", "reviewer": "Reviewer_Uv5e", "summary": "This paper proposes Deep Graph Mating, a novel task for model reuse in non-Euclidean domains, specifically focusing on GNNs. The goal is to create a child GNN that combines knowledge from pre-trained parent GNNs without requiring re-training, fine-tuning, or ground-truth labels. The process firstly identifies the permutation invariance properties of GNNs, and accordingly explores two naïve methods: Vanilla Parameter Interpolation (VPI) and Vanilla Alignment Prior to Interpolation (VAPI), both of which were shown to be insufficient due to challenges such as parameter misalignment and topological dependencies. \n\nThe contribution of this paper is to address this issue by proposing a Dual-Message Coordination and Calibration (DuMCC) methodology that optimizes the permutation matrices for parameter interpolation (PMC scheme) by incorporating topological information, and solves over-smoothing by calibrating the message statistics for child GNNs (CMC scheme). This results in performance on par with traditional re-training methods but without the associated training cost, which is validated on seven datasets across node and graph classification as well as semantic parsing tasks. Code and proofs are given in the appendix and the supplementary material.", "questions": "See weaknesses. Overall, I believe this paper addresses a significant problem and proposes a novel approach for model reuse in GNNs. The motivation is strong, supported by solid theoretical analysis. Despite the merits, I see several weaknesses. Therefore, I rate the paper as borderline at this stage and look forward to the authors' responses and further discussions regarding my concerns in the weaknesses section.", "limitations": "Yes, limitations and broader impacts have been discussed. An additional limitation to consider would be the scenario where two pre-trained models address tasks at different levels, such as node and graph-level tasks.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "I see several merits of this paper:\n\nAddressing the challenge of model reuse in GNNs is a significant topic, especially given the increasing scale of graph data and models in this era. The proposed GRAMA bypasses the need for re-training or fine-tuning, which is commonly required in existing model reuse approaches. GRAMA also removes the need for ground-truth labels, enhancing its generalizability in scenarios where such labels are unavailable. These properties make GRAMA suitable for graph analysis scenarios where resources are limited. \n\nThe authors present a couple of effective schemes PMC and CMC to deal with the challenges in GRAMA, specifically focusing on parent models and child model. These proposed methods seem overall straightforward yet supported by strong motivations and solid theoretical analysis, such as the amplified sensitivity to parameter misalignment and the increased susceptibility to over-smoothing in child models. The paper also conducts a thorough analysis of these methods across seven graph benchmarks and demonstrates good performance. I see competitive values in Tables 4,5 for the large-scale point cloud benchmarks, without needing to re-training.\n\nOverall, the paper is well-written and the main ideas, arguments, and algorithm are very easy to follow, with ample details provided in the appendix.", "weaknesses": "I see the following weaknesses in this paper:\n\nWhile I appreciate the authors’ attempt to present the complex trajectory of the paper (from defining the problem and elaborating the task motivation, to developing two naïve methods and analyzing challenges, and finally proposing two schemes) in a narrative style for easier understanding, there is an imbalance in the organization of content. The current organization of the paper only allocates less than two pages to the experiment section, leaving many experimental details to the appendix. I suggest the authors move more experimental details from the appendix into the main body of the paper, reducing the current extensive focus on the motivations of the method, and instead, expanding more information about the datasets used, enhancing the details and settings of the comparative methods, and elaborating on the experimental procedures. Also, the connection between the main paper and the appendix is very limited. The authors vaguely direct readers to Section D for further details, without specifying specific tables or paragraphs, which makes it hard to locate relevant information.\n\nAlso, the explanation regarding the heterogeneous GRAMA in Line 123 actually makes me confused. The statement is too vague. What exactly are the key differences between the homogeneous and heterogeneous cases of GRAMA? Does it specifically relate to the same or different architectures of the pre-trained models, or does it also involve models specializing in different tasks? For example, if there are two pre-trained models with the same architecture but trained for different tasks, would this scenario be classified as homogeneous or heterogeneous GRAMA? Furthermore, while the authors claim that “our initial investigation in this paper is confined to scenarios where pre-trained GNNs possess identical architectures yet are trained on separate datasets”, the authors should at least provide some insights or potential solutions for solving heterogeneous GRAMA, or perhaps consider removing this statement of heterogeneous GRAMA since this is not the contribution of this paper.\n\nFollowing this discussion of heterogeneous GRAMA, I think it is at least worth a try to apply GRAMA to two different GNN variants, such as GCN and GraphSage, since the key learnable parameters in these GNNs are all MLPs. The existing literature has demonstrated the importance of MLPs to GNN performance [1]. Exploring this avenue could be very interesting; if successful, the contributions of this paper would be significantly enhanced. Furthermore, the results may shed light on the connections between different GNN variants in the weight space.\n\nIn Tables 2 to 5, the authors overlook an important comparative result: the performance of retraining a multi-dataset model that can jointly combine the expertise of both parent models. While this approach involves retraining, it can at least serve as an upper bound for GRAMA, showing the potential space for further improvement.\n\nIt is also not very clear to me why the authors chose a 20%/80% splitting ratio in the experiments, as there are too few explanations about the splitting protocol. For example, is the split random? More details should be provided, as this concerns the pre-trained models and can reveal how knowledge from the two pre-trained models is distributed and when the proposed method is effective. Would other splitting, such as 10/90, work for GRAMA? It is necessary to conduct more experiments with more splitting ratios to make the authors’ claims more convincing.\n\nThe current discussion of limitations in this paper appears somewhat constrained. The authors should consider expanding this section, either in the main body of the paper or in the appendix. Another limitation of this work, compared to previous approaches such as KA, is the combination of models working on different levels of tasks, such as node classification and graph classification. Weight-space combination for models addressing various tasks has recently been explored in the Euclidean domain such as ZipIt [2], making it intriguing to see whether similar results can also be achieved in the non-Euclidean domain.\n\nMinor issues in the paper:\n\nLine 26: \"reduing\" $\\rightarrow$ \"reducing\";\n\nLine 106: \"presents\" $\\rightarrow$ \"present\";\n\nLine 193: \"are\" $\\rightarrow$ \"is\".\n\n[1] Han, Xiaotian, et al. MLPInit: Embarrassingly Simple GNN Training Acceleration with MLP Initialization. ICLR 2023.\n\n[2] Stoica, George, et al. ZipIt! Merging Models from Different Tasks without Training. ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Response to Reviewer Uv5e (Part 1/2)**\n\nWe truly appreciate the reviewer's insightful comments, and would like to address them as follows. Due to character limitations, we have to split our response into two parts. The second part will be provided as a comment following our initial response.\n\n`W1.` **Imbalanced organisation**\n\n>\"The current organization of the paper only allocates less than two pages to the experiment section, leaving many experimental details to the appendix.\"\n\n`Response:` We appreciate the reviewer for the suggestion. Given the novelty of the studied GRAMA task and the sophisticated rationale behind the proposed DuMCC, our organisational approach was intended to provide detailed explanations, thus facilitating easier comprehension for the readers. These include:\n- `Sect. 3`: Motivation, definition, and applications of the GRAMA task;\n- `Sect. 4.1`: Vanilla solutions for the GRAMA task described in `Sect. 3`;\n- `Sect. 4.2`: Challenges associated with the vanilla solutions from `Sect. 4.1`, leading to the motivations for our proposed DuMCC in `Sect. 5`;\n- `Sect. 5.1`: An overview of our proposed DuMCC, motivated by the challenges outlined in `Sect. 4.2`;\n- `Sect. 5.2`: The first component of DuMCC, PMC, including an analysis of its shortcomings, which subsequently motivates the introduction of CMC;\n- `Sect. 5.3`: The second component of DuMCC, CMC, motivated by the findings in `Sect. 5.2`.\n\nThe sophisticated logic detailed above may require some pages to clarify potential confusion arising from our novel task and method. Admittedly, our preference for clarity, on the other hand, made us have to relocate some details to the appendix to adhere to page limits, exactly as the reviewer suggested.\n\nIn the revision, we will strive to achieve a better balance by incorporating as many details as possible within the scope of the additional content page allowed. Our detailed plan is as follows:\n- Relocating portions of Sect. B and Sect. C to Sect. 6;\n- Reorganising the architectural details in Sect. D into narrative texts and transferring the associated experimental details from Sect. D to Sect. 6.\n\n\n---\n\n`W2.` **Limited connection between the main paper and the appendix**\n\n>\"The authors vaguely direct readers to Section D for further details, without specifying specific tables or paragraphs, which makes it hard to locate relevant information.\"\n\n`Response:` We apologise for any inconvenience caused by the insufficient linkage between the main paper and the appendix, which may have extended the time required for reviewing our paper. Following the transfer of experimental details from the appendix to the main paper in `W1`, we will endeavour to enhance the connectivity of the remaining implementation details within the appendix to the main paper in our revision.\n\n---\n\n`W3.` **Vague heterogeneous GRAMA descriptions**\n\n>\"The explanation regarding the heterogeneous GRAMA in Line 123 actually makes me confused. The statement is too vague. What exactly are the key differences between the homogeneous and heterogeneous cases of GRAMA?\"\n\n`Response:` We apologise for not having made our heterogeneous GRAMA clearer, which may have confused the reviewer. We would like to clarify that the heterogeneous case of GRAMA serves as a complementary extension to the proposed homogeneous GRAMA framework in our paper, where pre-trained parent models share similar architectures and purposes. We will explicitly highlight and detail this distinction in Sect. 3 of our revised version.\n\n---\n\n`W4.` **Potential solutions for heterogeneous GRAMA**\n\n>\"The authors should at least provide some insights or potential solutions for solving heterogeneous GRAMA, or perhaps consider removing this statement of heterogeneous GRAMA since this is not the contribution of this paper.\"\n\n`Response:` We appreciate the advice from the reviewer. A potential solution could involve Partial GRAMA, which entails first identifying shared features between two parent models that have different architectures or handle varied tasks. Subsequently, this possible method would yield a multi-head child GNN that selectively integrates elements of the pre-trained parent GNNs. Although this method would increase the model size compared to full GRAMA, which combines the entire models, it would be expected to offer a more favourable balance and trade-off between model size and performance. We will incorporate this discussion in the revised paper.\n\n---\n\n`W5.` **Applying GRAMA to variants of GNNs**\n\n>\"I think it is at least worth a try to apply GRAMA to two different GNN variants, such as GCN and GraphSage, since the key learnable parameters in these GNNs are all MLPs.\"\n \n`Response:` The reviewer's point is very well taken. Our immediate-next goal is, exactly as the reviewer suggested, to adapt our proposed method for cross-architecture model reuse scenarios, including the combination of GCN and GraphSAGE as outlined in lines 348-349 of our paper.\n\nHere, to echo the reviewer's concern, we conducted a pilot study by performing additional experiments on GRAMA using both a GCN and a GraphSAGE model on non-overlapping partitions of the ogbn-arxiv dataset. These two parent models share a similar overall architecture but differ in their respective GCN and GraphSAGE layers. For the GraphSAGE model, we employed the official SAGEConv implementation from the Deep Graph Library (DGL), setting the 'aggregator_type' to 'gcn' for this preliminary study. This adjustment is due to the design of our homogeneous GRAMA, which requires alignment of parameters with the same shape and number. The results, as shown below, highlight the promising potential and feasibility of cross-architecture GRAMA applications.\n\n| Parent Models | Architectures | Performance: Dataset 1 | Performance: Dataset 2 |  \n|:----- | :----- | :----- | :----- | \n| Parent 1 | GCN | 0.6908 | 0.5752 |\n| Parent 2 | GraphSAGE | 0.6181 | 0.7508 |\n| Child | GCN | 0.6082 | 0.5746 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TmBMkXIuAz", "reviewer": "Reviewer_tRzs", "summary": "In this paper, the authors present a method for learning-free and label-free model reuse for graph neural networks. They dub the task Deep Graph Mating (Grama). Without relying on costly fine-tuning or re-training, Grama aims to generate a child model by reusing and fusing knowledge from pre-trained parent models, particularly by managing pre-trained parameters. Compared with conventional model reuse tasks (knowledge distillation, and knowledge amalgamation) in the GNN domains, the authors claim in Tab 1 that Grama simultaneously enables multi-model reuse, annotation free, and training/fine-tuning free, leading to more resource-efficient model reuse. To achieve Grama, the authors begin by deriving two vanilla Grama methods, dubbed VPI and VAPI, motivated by the permutation invariance proposition in GNNs. However, both VPI and VAPI don’t work well, and the authors observe unique challenges for Grama, namely the increased sensitivity to weight misalignment and the accompanying topology-dependent complexities, substantiated by a series of propositions, lemmas, and conjectures. Motivated by the identified challenges, the authors introduce the proposed DuMCC framework, which includes a PMC and CMC. PMC is designed to identify topology-aware permutation matrices but suffered from increased susceptibility to over-smoothing. To solve this problem, the authors further designs CMC, which is to refine message statistics of the child network through a learning-free message normalization layer. Extensive experiments (7 benchmarks, 4 tasks, 5 architectures) in various graph applications have been performed to validate the effectiveness of the proposed framework, covering popular GNNs including GCN, GraphSAGE, GAT, GIN, and DGCNN. The authors also provide extensive supplementary information with detailed proofs in the appendix.", "questions": "1. Can you clarify if the statistics in the original normalization layer are recomputed during Grama? If so, when would they be recomputed?\n\n2. Have you experimented with using three or more pre-trained models in Grama?\n\n3. Consider improving the experimental section to more closely relate the texts to the validated claims.\n\nAlthough the authors missed some key information, I feel these could be easily addressed. On balance, the paper is both novel and useful. The concept of training-free and label-free model reuse for GNN had not been done before.", "limitations": "The paper has thoroughly discussed limitation in Sec 7 and social impact in appendix.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The positive traits of this paper include:\n\n- Well-organized paper with a very clear logic flow. The authors first give a well-defined problem of Grama with reasonable motivations. Then they present two vanilla methods as initial solutions to solve the Grama problem. Based on these methods, the authors identify the unique challenges from Grama with elaborated analysis and clear thoughts. Motivated by the identified challenges, a DuMCC framework is proposed. The way the authors present the entire paper make it very easy for following its development.\n\n- Interesting idea. To the best of my knowledge, Grama is the first to simultaneously enable multi-model reuse, annotation-free, and training/fine-tuning free capabilities. The authors clearly demonstrate their idea by comparing Grama with previous model reuse methods in Tab 1, which is very useful for understanding the main benefits of Grama. It is also interesting to explore the correspondence in the weight space for different GNNs, which also appears to be the first exploration in this field of GNN.\n\n- Solid intuitions and well-established algorithm. At first sight, the proposed approach appears a bit ad-hoc. However, the authors provide sufficient motivations for each component in DuMCC, making their method well-justified. This includes extensive discussions on vanilla methods and the over-smoothing issues from PMC, with a motivating example shown early in Fig 1, making the proposed method technically sound.\n\n- Experiments are conducted thoroughly. It is clear that the authors try to cover a wide range of tasks and various GNN architectures in their experiments. The tasks include node property prediction, graph property prediction, 3D object recognition, and 3D semantic segmentation, and the architectures include GCN, GraphSAGE, GAT, GIN, and DGCNN. The effectiveness of the proposed method and nearly all claims in the paper are thoroughly verified.\n\n- Good reproducibility. Extensive supplementary details are provided in the appendix, with source code and models provided in the supplementary material. Algorithm 1 is clear and useful for capturing the main procedure of the entire framework.", "weaknesses": "- In Sec 5.3, the authors introduce a learning-free message normalization (LFNorm) layer for statistics calibration in CMC, which is technically sound. However, the authors miss the discussions on the Grama case where pre-trained models already contained normalization layers, which is common in the area of graph neural networks. In that case, it is unclear whether the batch statistics are also recomputed in CMC. If they are, it should be specified when the recomputation occurs: simultaneously with LFNorm or after?\n\n- Because the authors have claimed multimodel reuse as a contribution of this work, I would advise the authors to include more experiments for the Grama cases with three or more pre-trained models. Otherwise, clear statements should be added to limit the scope of the study to the reuse of two pre-trained models.\n\n- The descriptions and explanations in the experimental results are not closely matched with the authors’ assumptions and propositions from the previous sections. It is suggested that the authors more closely relate the texts in the experimental section to the methods section, particularly by specifying which results validate which proposition or claim. This would make the paper more readable.\n\n- In the appendix, there is a minor problem where Eq 30 extends beyond the margin of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Response to Reviewer tRzs**\n\nWe appreciate the reviewer for the positive support and constructive comments. \n\n`W1. & Q1.` **Statistics recomputation details**\n\n>\"The authors miss the discussions on the Grama case where pre-trained models already contained normalization layers, which is common in the area of graph neural networks. In that case, it is unclear whether the batch statistics are also recomputed in CMC. If they are, it should be specified when the recomputation occurs: simultaneously with LFNorm or after?\"\n\n>\"Can you clarify if the statistics in the original normalization layer are recomputed during Grama? If so, when would they be recomputed?\"\n\n`Response:` We apologise for not having sufficiently highlighted the details regarding the recomputation of statistics, which might have led to them being overlooked by the reviewer. In fact, as detailed in line 313-314, we have indeed discussed the scenario where models are equipped with normalisation layers. We have explicitly explained that the running mean and variance of the original normalisation layers are recalculated for the child GNN. In our revised version, we will enhance the clarity of this detail further by also providing additional explanations in Sect. 5.3.\n\nBut admittedly, we apologise for not having specified when the recomputation of statistics occurs for the original normalisation layers in our paper. We appreciate the reviewer's insightful comments on this issue. As specified in the source code provided in the supplementary material, we clarify that the recomputation of statistics is performed concurrently with that of the LFNorm layer for the child GNN. In the revision, we will provide a detailed clarification and add the discussion on the normalisation-related procedures described above in Sect. 5.3 and Sect. 6.\n\n---\n\n`W2. & Q2.` **Multi-model GRAMA results**\n\n>\"Because the authors have claimed multimodel reuse as a contribution of this work, I would advise the authors to include more experiments for the Grama cases with three or more pre-trained models. Otherwise, clear statements should be added to limit the scope of the study to the reuse of two pre-trained models.\"\n\n>\"Have you experimented with using three or more pre-trained models in Grama?\"\n\n`Response:` We appreciate the reviewer's constructive suggestion. As suggested, we have conducted additional experiments on disjoint partitions of the ogbn-arxiv dataset using the architecture described in Tab. 10 of the main paper, specifically for the proposed GRAMA in scenarios involving the simultaneous reuse of three pre-trained models. The results, presented below, demonstrate that our proposed DuMCC approach is also effective in multi-model GRAMA contexts. These results will be incorporated into the revised version of the paper.\n\n| Models | Performance: Dataset 1 | Performance: Dataset 2 |  Performance: Dataset 3 |  \n|:----- | :----- | :----- | :----- | \n| Parent 1 | 0.6645 | 0.6476 | 0.6040 |\n| Parent 2 | 0.4796  | 0.7044 | 0.4651 |\n| Parent 3 | 0.5805 | 0.6078 | 0.7728 |\n| Child    | 0.5574 | 0.6360 | 0.5478 |\n\n\n---\n\n`W3. & Q3.` **Linking experiments to the claims to validate**\n\n>\"The descriptions and explanations in the experimental results are not closely matched with the authors' assumptions and propositions from the previous sections. It is suggested that the authors more closely relate the texts in the experimental section to the methods section, particularly by specifying which results validate which proposition or claim. This would make the paper more readable.\"\n\n>\"Consider improving the experimental section to more closely relate the texts to the validated claims.\"\n\n`Response:` We would like to thank the reviewer for the constructive suggestion. In our revised version, we will strive to expand the text in Sect. 6. This will include more detailed descriptions that closely and explicitly connect the results presented in Figs. 2-3 and Tabs. 2-5 with the validated conjectures, propositions, and claims discussed in Sects. 4-5.\n\n---\n\n`W4.` **Minor problem**\n\n>\"In the appendix, there is a minor problem where Eq 30 extends beyond the margin of the paper.\"\n\n`Response:` Thank you for pointing out this formatting issue. We will address this issue in the revision by breaking Eq. 30 into three separate lines to ensure it fits within the margins.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The aim of the paper is the reuse of GNN models for transfer to new tasks without re-training, fine-tuning or annotated labels. It proposes a new Dual-Message Coordination and Calibration methodology to solve the problems of vanilla permutation-based approaches.

The reviewer consensus is that the paper is well motivated addressing an interesting and important problem in a novel way. It contributes to both the model fusion and GNN domains. The method is well presented and the experiments document convincingly its good performance. 

The submitted version of the paper raised numerous questions and requests for clarifications or improvements. The authors addressed most of these in their extensive and meticulous rebuttal to the satisfaction of the reviewers. As it was these additional pieces of information - e.g. new experimental results, corrected notation, improved explanations - that convinced the reviewers about the quality of the contribution, we urge the authors to update the final version of the paper accordingly.
Decision: Accept (poster)