Deep Graph Mating

Authors: yongcheng  jing
seok-hee  hong
dacheng  tao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-02 03:06:00

Abstract: In this paper, we introduce the first learning-free model reuse task within the non-Euclidean domain, termed as Deep Graph Mating (Grama). We strive to create a child Graph Neural Network (GNN) that integrates knowledge from pre-trained parent models without requiring re-training, fine-tuning, or annotated labels. To this end, we begin by investigating the permutation invariance property of GNNs, which leads us to develop two vanilla approaches for Grama: Vanilla Parameter Interpolation (VPI) and Vanilla Alignment Prior to Interpolation (VAPI), both employing topology-independent interpolation in the parameter space. However, neither approach has achieved the anticipated results. Through theoretical analysis of VPI and VAPI, we identify critical challenges unique to Grama, including increased sensitivity to parameter misalignment and further the inherent topology-dependent complexities. Motivated by these findings, we propose the Dual-Message Coordination and Calibration (DuMCC) methodology, comprising the Parent Message Coordination (PMC) scheme to optimise the permutation matrices for parameter interpolation by coordinating aggregated messages, and the Child Message Calibration (CMC) scheme to mitigate over-smoothing identified in PMC by calibrating the message statistics within child GNNs. Experiments across diverse domains, including node and graph property prediction, 3D object recognition, and large-scale semantic parsing, demonstrate that the proposed DuMCC effectively enables training-free knowledge transfer, yielding results on par with those of pre-trained models.

Main paper: 
 introduction: 
 The remarkable progress made in deep neural networks has resulted in a growing number of pretrained models being made publicly available for the purpose of performance reproducibility and further development [64, 12, 27, 23, 61, 24] . As such, there is a mounting interest in the community on the reusability of existing pre-trained neural networks for the sake of strengthening performance, reducing model size, or alleviating training efforts, with abundant inspiring works proposed [16, 42, 46, 40, 47, 45, 63, 67] . Despite the growing interests in model reuse, current research endeavors have predominantly focused on the Euclidean domain, which are specifically designed to handle image data with regular grid structures [39, 64, 34, 12, 51, 14, 22] .On the other hand, the study of reusing pre-trained Graph Neural Networks (GNNs) to tackle non-Euclidean and irregular graph data is still in its early stage and remains limited in scope. Almost all existing research on GNN reuse is established upon non-Euclidean Knowledge Distillation (KD) pioneered by Yang et al. [60] , where a favourable student GNN is learned from a single pre-trained teacher [11, 49, 9, 66, 29] . Subsequent research has extended the scope of Yang et al. [60] from a single-teacher to a multi-teacher context, introducing the novel task of non-Euclidean Knowledge Amalgamation (KA) [25, 15, 36] . However, all these existing approaches to GNN reuse necessitate resource-intensive re-training of the student model, imposing substantial computational burdens andMulti-model Reuse Annotation Free Training/Fine-tuning Free Knowledge Distillation [60, 11, 49, 9, 66, 29] × × × Knowledge Amalgamation [25, 15, 36] √ √ × Deep Graph Mating (GRAMA) √ √ √Table 1 : Comparison of various model reuse tasks in the non-Euclidean domain, tailored for GNNs.memory costs. This challenge from re-training is particularly pronounced when dealing with large models and large-scale graphs [31, 7, 37, 20, 8, 26, 17] .In this paper, we strive to push the boundaries of resource-efficient GNN reuse by introducing the first training-free model reuse task in the non-Euclidean domain, termed as Deep Graph Mating (GRAMA). Our objective is to derive a child GNN, without re-training or fine-tuning, from pre-trained parent GNNs, each possessing unique knowledge from different datasets, all while operating without access to human-annotated labels-a common constraint in using publicly available models. The child model born from GRAMA is expected to seamlessly integrate the diverse expertise of its parent models in a completely learning-free manner. A comparative analysis of GRAMA with existing non-Euclidean model reuse approaches is presented in Tab. 1. As a pilot study of this novel task, this paper focuses on homogeneous GRAMA scenarios with identical parent architectures, and sets the stage for future investigations into the more complex cross-architecture heterogeneous GRAMA.To achieve the ambitious goals of GRAMA, we first investigate the permutation invariance property of GNNs to establish correspondence between neurons in pre-trained models. This investigation guides the development of two vanilla methods: Vanilla Parameter Interpolation (VPI) and Vanilla Alignment Prior to Interpolation (VAPI). VPI involves straightforward linear interpolation of parent GNN parameters, while VAPI incorporates data-independent parameter matching before interpolation. However, the performance of both VPI and VAPI has not been as promising as expected. By analysing the mechanisms underlying these methods, we have identified and theoretically demonstrated unique challenges associated with GRAMA, including increased susceptibility to parameter misalignment and topology-dependent complexity in GNNs. These findings underscore the necessity for tailored methods that are specifically adapted to GRAMA.To this end, we introduce the Dual-Message Coordination and Calibration (DuMCC) methodology, specifically tailored for GRAMA to incorporate the topological characteristics of graphs. DuMCC consists of two distinct schemes: Parent Message Coordination (PMC) and Child Message Calibration (CMC). PMC seeks to identify optimal permutation matrices for parameter matching in a topologyaware manner, by coordinating aggregated messages from parent GNNs. Although PMC shows promising results, our empirical and theoretical analyses indicate that the child GNN derived from this coordination is more prone to over-smoothing. To mitigate this issue, we propose the CMC scheme, which calibrates the message statistics of the child GNN using a specialised, learning-free message normalisation (LFNorm) layer, drawing on the statistics of the parent GNNs. Together, these two schemes contribute to a training-free and label-free GRAMA process, enabling the derivation of child GNNs that effectively embody the knowledge from their pre-trained parent models.In summary, our contribution is a novel non-Euclidean model reuse paradigm that allows for the creation of a student GNN, which integrates the capabilities of pre-trained parent GNNs without requiring human-annotated labels or re-training. We evaluated our proposed approach on seven benchmarks across various tasks, including node/graph classification, object recognition, and largescale indoor semantic segmentation, and across five different GNN architectures: Graph Convolutional Network (GCN) [30] , GraphSAGE [13] , Graph Attention Network (GAT) [48] , Graph Isomorphism Networks (GIN) [56] , and Dynamic Graph CNN (DGCNN) [53] . Experimental results demonstrate that the generated child GNN is competent to handle the pre-trained tasks of parent models. We also present a discussion of the limitations, highlighting potential future research directions that can be explored based on GRAMA. 
related work: 
 This section briefly reviews topics relevant to GRAMA, including existing GNN reuse techniques and model merging, which, while conceptually akin to GRAMA, is confined to the Euclidean domain. Extended related work is provided in Appendix A.Non-Euclidean Model Reuse. Recent advancements in graph-centric intelligence have led to the widespread availability of pre-trained models for reproducibility. Despite their availability, the exploration of their reuse for downstream tasks, particularly in the non-Euclidean domain, is still in its infancy. Existing research primarily revolves around knowledge distillation (KD) [60, 9, 66, 11, 49, 29] and knowledge amalgamation (KA) [25, 15, 36] , aimed at single-GNN and multi-GNN reusing, respectively. In particular, the foundational KD research [60] introduced a method specifically designed for GNNs that conserves local structural integrity. The scheme was later expanded by KA [25] to adapt to multi-teacher settings without the need for labels, thereby enhancing GNN reuse capabilities. Extending these concepts, this paper launches GRAMA, a new GNN reuse paradigm that further boosts resource efficiency by obviating the need for re-training.Model Merging. Recent years have seen a surge in interest in merging the weights of CNNs or transformers into a single model [54, 43, 1, 19, 44, 57, 33, 59, 35, 62, 38, 2, 21] . Model merging is typically categorised into two types: fine-tuned model merging and variably initialised model merging [33, 44] . Merging models fine-tuned from the same initialisation is generally straightforward as these models often reside within the same error basin, allowing for simple weight interpolation [54] .Conversely, merging models from different initialisations is more challenging due to the randomness in network channels and components [33, 43, 1] . A key issue is aligning neurons between models to establish correspondences before weight interpolation. To address this issue, Git Re-basin [1] proposes to minimise the L 2 distance between weight vectors. Singh and Jaggi [43] propose an Optimal Transport Fusion (OTFusion) method that uses the Wasserstein distance to align weight matrices prior to performing parameter fusion. The subsequent work [19] extends the application of OTFusion to Transformer-based architectures, aiming to enhance efficiency and performance through fusion. Also, Liu et al. [35] approach the challenging task of model fusion as a graph matching problem, incorporating second-order parameter similarities for improved fusion performance.Addressing the fusion of pre-trained models trained on disparate tasks, Stoica et al. [44] develop ZipIt!, a novel method that utilises a "zip" operation for layer-wise fusion based on feature redundancy, creating a versatile multi-task model without additional training. More recently, Xu et al. [57] present the Merging under Dual-Space Constraints (MuDSC) framework. This approach optimises permutation matrices by mitigating inconsistencies in unit matching across both weight and activation spaces, targeting effective model fusion in multi-task scenarios. Furthermore, Jordan et al. [28] introduce REPAIR, a method that tackles the issue of variance collapse by rescaling the hidden units in the interpolated network to restore the statistical properties of the original networks.However, model merging has not yet been explored within the context of GNNs. This paper introduces GRAMA, the first formulation for weight space model merging tailored for the non-Euclidean domain.It represents the initial investigation into merging GNNs, addressing the unique challenges that arise with graph tasks.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Limitation discussions are provided in Sect. 7.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
motivation and problem definition: 
 In this section, we begin by introducing Knowledge Amalgamation (KA), the sole existing task in multi-GNN reuse, exploring its inherent limitations, and present our novel GRAMA paradigm for resource-efficient multi-model reuse with practical applications in real-world scenarios.To the best of our knowledge, KA represents the only multi-model reuse task in the non-Euclidean domain. KA is defined in the literature as follows: its objective is to learn a single, compact student GNN that integrates the diverse expertise of pre-trained teacher GNNs, without accessing human annotations [25] . Despite the promising performance, KA is inherently limited by its resourceintensive nature, requiring the re-training of a student GNN to amalgamate knowledge from existing GNNs. Furthermore, while KA ostensibly operates without ground-truth labels, it instead relies on soft labels generated by teachers, making it susceptible to the misclassification errors of teachers.To mitigate these constraints of KA, this paper proposes: Task 3.1 (Deep Graph Mating). Deep Graph Mating (GRAMA) is a fully learning-free model reuse task where a child GNN is derived from pre-trained parent GNNs without re-training or fine-tuning, integrating their expertise without requiring human-annotated labels.In aggregate, GRAMA advances beyond KA by eliminating the need for any training or label dependency, paving the way for more widespread and versatile model reuse applications. Given the novelty and complexity of GRAMA, our initial investigation in this paper is confined to scenarios where pre-trained GNNs possess identical architectures yet are trained on separate datasets, termed as homogenous GRAMA. We reserve the exploration of more challenging heterogeneous GRAMA scenarios, where pre-trained parent models either have diverse architectures or are designed for different domain tasks, as a topic for future studies, potentially incorporating solutions like partial GRAMA inspired by the work of Stoica et al. [44] .The applications of homogenous GRAMA are especially crucial in contexts where full data access for training is restricted due to privacy concerns and regulatory requirements. This is common in sectors like healthcare or retail, where organisations operate across different regions, each gathering data that cannot be centrally aggregated due to local privacy regulations. The proposed homogenous GRAMA paradigm enables the seamless integration of knowledge from these isolated datasets, thereby safeguarding against the risks of privacy violations and the disclosure of sensitive data.4 Vanilla Methodologies and Challenge Pre-analysis 4.1 Two Vanilla GRAMA Methods Vanilla Parameter Interpolation (VPI). To achieve the ambitious goal of GRAMA outlined in Sect. 3, the initial naïve approach employs vanilla weight averaging [50, 54] . This method involves a straightforward linear interpolation of weights W a and W b from two pre-trained GNNs:W ( ) = αW ( ) a + (1 -α)W ( )b , where α represents the interpolation weight and W ( ) denotes the network weights at layer . However, the vanilla averaging approach requires the pre-trained models to share a portion of their training trajectory and remain sufficiently close in the parameter space [33] , typically achieved by fine-tuning from the same initial model. This is not applicable in our GRAMA context, where parent GNNs are trained on distinct datasets. This mismatch leads to empirically poor performance for GRAMA, as observed in our experiments.Vanilla Alignment Prior to Interpolation (VAPI). To address this issue in vanilla interpolation, previous research in the Euclidean domain, based on the conjecture of the permutation invariance property of typical neural networks 1 , proposes aligning the neurons between pre-trained models by permuting parameter matrices before performing linear interpolation [1] . The alignment and interpolation process can be formulated as:EQUATIONwhereP * = P ( ) ∈[L]represents the set of all permutation matrices P ( ) for each layer of the GNN. Here, [L] refers to the set of indices corresponding to all layers in the GNN. However, to apply Eq. 1 to GNN-based GRAMA, it is essential to first discuss whether the permutation invariance property extends to GNNs. This property has been extensively studied in existing literature for various architectures [10, 5, 43, 1, 33, 19] , including multi-layer perceptrons (MLPs), convolutional neural networks (CNNs), and Transformers. Supported by these prior studies, and given that GNNs are fundamentally built upon MLPs [58, 30] , we propose: Conjecture 4.1 (Permutation Invariance in GNNs). Permutation invariance for parameters in GNNs exists if and only if there exists a set of permutation matrices P ( ) for each layer such that applying these permutations to the parameters does not alter the outcome of graph-based learning task, regardless of the structure of the adjacency matrix.A key subsequent issue involves searching the optimal permutation matrix P * for GNNs. One possible data-independent solution is to minimise the L 2 distance between the weight vectors of the pre-trained models by solving a sum of bilinear assignments problem, similar to weight matching techniques described in [1] . This method, which does not consider data distributions, could be adapted as a baseline vanilla method for our GRAMA task in the non-Euclidean domain, which is evaluated in our experiments.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
challenges towards grama: 
 However, we empirically observed that even the Vanilla Alignment Prior to Interpolation method yielded unfavourable results for our GRAMA task. To elucidate the underlying cause of this phenomenon, we theoretically demonstrated that GNNs typically exhibit greater sensitivity to parameter mismatches than neural networks in the Euclidean domain: Lemma 4.1 (Amplified Sensitivity of GNNs to Parameter Misalignment). GNNs exhibit greater sensitivity to mismatches in parameter alignment compared to CNNs, amplified by the degree of connectivity and heterogeneity of the node features in the graph topology.A complete theoretical proof of Lemma 4.1 is provided in Appendix G.1. Here, we present only the final formulation of the approximated output changes resulting from weight perturbations due to mismatching, based on Taylor series approximation:EQUATIONwhere ∆F i refers to the change in output at node i due to the perturbations in the weights W , and σ represents the derivative of the activation function. X j denotes the features of the nodes within the neighbourhood N (i) of node i. Detailed, model-specific formulations of Eq. 2 are provided in Sect. B of the appendix.Eq. 2 implies that the effect of can be exacerbated by the potentially large and diverse neighbourhoods N (i), thereby making the output highly sensitive to changes in W . In other words, the effect of weight perturbation can vary dramatically based on the node's connectivity and the characteristics of its neighbours. Such variability leads to significant and less predictable changes in output, illustrating the particular vulnerability of GNNs to parameter mismatches.The integration of Eq. 1 with Lemma 4.1 and Eq. 2 further gives rise to the following conjecture: Conjecture 4.2 (Topology-dependent Complexity in GNNs). The identification of optimal permutation matrices P * for GNNs presents increased complexity compared to the Euclidean domain, contingent upon the topological characteristics inherent to each graph.Conjecture 4.2 highlights the essential need for developing GRAMA methods that are specifically tailored to accommodate the unique topologies of graphs, motivating the design of the proposed approach in Sect. 5.5 Proposed Approach: Dual-Message Coordinator and Calibrator 
overview: 
 Motivated by Conjecture 4.2, we introduce in this section the proposed Dual-Message Coordination and Calibration (DuMCC) methodology, which is specifically designed to harness the unique topological features of input graphs for achieving GRAMA without relying on human annotations.The proposed DuMCC is composed of two strategic schemes. In particular, the first Parent Message Coordination (PMC) scheme effectively integrates topological information by deriving optimal permutation matrices from layer-specific aggregation results. However, both empirical and theoretical analyses reveal a reduction in node feature variance in child GNNs, suggesting that models derived through this coordination are more susceptible to over-smoothing compared to their parent GNN counterparts.To address this issue, we further propose the Child Message Calibration (CMC) scheme as our second strategic component. This scheme aims to maintain message variance consistency from the parent models, ensuring the retention of feature diversity essential for robust GNN performance. Further elaboration on each component is provided in subsequent sections.Motivated by Conjecture 4.2, which highlights the significance of incorporating topology information in the GRAMA process, we propose a Parent Message Coordination (PMC) scheme for identifying optimal topology-aware permutation matrices P * described in Eq. 1. Unlike the vanilla method of VAPI in Sect. 4 that minimises the distance between weight vectors without considering the input graphs' topologies, PMC optimises P * by leveraging the topology information embedded in aggregated messages from two pre-trained parent GNN models.Assume we have two pre-trained parent GNNs, denoted by G a and G b , both sharing identical architectures. Their corresponding weight matrices are denoted by W a and W b , respectively. To establish correspondence between neurons in W a and W b as described in Eq. 1, our PMC optimises the permutation matrix P ( ) at layer by aligning the aggregated messages of the two parent GNNs.To align with the notations [1, 33] used in model merging within the Euclidean domain, the associated optimisation process can be formulated as follows:EQUATIONwhere φ denotes the message function that encodes both node features X and edge features e, and Agg represents the message aggregation function that accumulates incoming messages from φ, acting on each node i. N (i) denotes the set of neighbours of node i. The minimisation problem in the form of Eq. 3 can be typically transformed into a maximisation problem to maximise an inner product (as derived from expanding Eq. 3), thereby fitting it within the framework of a standard linear assignment problem, as also done in the works of [1, 33, 35, 44] .Eq. 3 is based on the rationale that aggregated messages inherently encapsulate essential graph topologies, and that structurally similar GNNs typically generate analogous aggregated messages when tasked with similar graph operations and topologies. As such, through Eq. 3, P * can be determined in a topology-aware manner, matching aggregated messages to reflect the topological characteristics of the graphs. Here, we clarify that while our approach involves passing the graph data to the pre-trained model to capture the graph-specific topological characteristics by utilising X, it requires only a single forward pass of the unlabelled graph data to extract messages for alignment-eliminating the need for iterative training or ground-truth labels. Subsequently, child GNNs can be derived through linear parameter interpolation.Despite the encouraging performance, we observe that the child GNN, derived from the proposed PMC, exhibits a reduction in the variance of node embeddings. We conjecture that this reduction stems from an averaging effect, which may smooth out the distinctive features captured by each parent model, particularly when these models have learned different structural aspects of the graph: Lemma 5.1 (Variance Reduction in Interpolated Graph Embeddings). The variance of the graph embeddings in an interpolated child GNN is typically smaller than the variances of the embeddings from the individual pre-trained parent GNNs.The full proof of Lemma 5.1 is detailed in Appendix G.2, providing evidence of feature homogenisation within child GNNs. In the context of GNNs, we further explore and establish the following proposition: Proposition 5.1 (Increased Susceptibility to Over-Smoothing in Child GNNs). Interpolated child GNNs exhibit increased susceptibility to over-smoothing compared to their parent networks, as measured by Dirichlet energy.The detailed proof of Proposition 5.1 is provided in Appendix G.3, utilising the quantitative oversmoothing measurement based on Dirichlet energy [41] , where a lower value indicates greater homogeneity or smoothness among node features. In particular, our theoretical analysis in Appendix G.3 demonstrates that:EQUATIONwhere E(X ) denotes the Dirichlet energy for the node features X at layer of the child GNN.In GRAMA, the parameter α is typically set to 0.5 to ensure unbiased knowledge integration from both pre-trained models. This setting promotes a balanced contribution from each model and prevents any bias toward the characteristics of one over the other. As detailed in Appendix G.3, with α at this level, the Dirichlet energy of the interpolated child GNN, E(X ), significantly decreases compared to the Dirichlet energies of the individual parent models, indicating a higher susceptibility to oversmoothing.Corresponding to this theoretical analysis, empirical evidence is presented in Fig. 1 , where 60 parent models are pre-trained on distinct partitions of the ogbn-products dataset with different random seeds. Additional implementation details are provided in the appendix. Fig. 1 further demonstrates the increased smoothing effect, which can potentially diminish the model's expressive power and discriminative capability.To mitigate the over-smoothing issue identified in Proposition 5.1, one potential solution involves leveraging established methods designed to address over-smoothing, such as PairNorm [65] and residual connections [32] . However, to the best of our knowledge, all these existing solutions typically require re-training the model. This requirement contradicts the fundamental principle of our GRAMA approach, which aims for training-free model reuse.To address this issue, we introduce a Child Message Calibration (CMC) scheme designed to refine the message statistics of the obtained child GNN without the need for re-training or ground-truth labels.Central to this scheme is our Learning-Free Message Normalisation (LFNorm) layer specifically tailored for our GRAMA task, inspired by [28] . This layer is intended to enhance the discriminative power and representational capacity of the child GNN by promoting a more diverse node feature distribution.According to Proposition 5.1, while the shift in mean node features is typically less problematic than variance reduction, it is noted in [6] that mean statistics in GNNs also carry vital graph structural information. Therefore, our approach aims to simultaneously refine both mean and variance statistics in the child GNN, ensuring a comprehensive enhancement of topological representation.Towards this end, we first process the raw graph through the parent GNNs to compute the target message mean and variance intended for alignment in the child GNN. Subsequently, we integrate the LFNorm layer into the child GNN to refine the message statistics using the derived mean and variance from the parent models:EQUATIONwhereEQUATION, e ij ) represents the messages from node j to node i, as specified in Eq. 3. In this setup, mij denotes the statistically calibrated message within the child GNN, while µ N (i) and σ N (i) represent the mean and standard deviation, respectively, of the messages directed to node i in the child GNN.Eq. 5 ensures that the normalised messages in the interpolated child model maintain a balanced representation of central tendencies from both pre-trained parent models. This method effectively reduces the risk of over-smoothing in child GNNs by preserving essential topological statistics from the parent models.A more detailed algorithmic procedure is outlined in Alg. 1. In practice, we find that incorporating a single LFNorm layer and aligning the overall message mean and variance of the parent GNNs is typically sufficient to achieve favourable performance while minimising computational costs. 
experiments: 
 We evaluate the performance of DuMCC across seven benchmarks spanning five GNN architectures. More ablation studies and sensitivity analyses, additional results and implementation details, as well as more visualisations, are detailed in Secs. D and E of the appendix. P ( ) ∈P * i Agg j∈N (i) φ W ( -1) a ; X ( -1) j,a , e ij -P ( ) • Agg j∈N (i) φ W ( -1) b ; X ( -1) j,b, e ij 2 // Interpolate weights for current layer W ( ) ← αWEQUATION, e ij ) // Compute the scale and shift parameters [1, 28] . Specifically, each dataset is randomly split into two disjoint subsets: the first subset comprises 20% of the data with odd labels and 80% with even labels, while the second subset is arranged vice versa.β ← Var(m a ij ) + Var(m b ij ) /2; γ ← E(m a ij ) + E(m b ij ) /2 // Learning-free message calibration for G mij ← β (m ij -µ N (i) )/σ N (i) +For the semantic segmentation task on S3DIS [3] , we train the two parent models using Areas 1, 2, 3 and Areas 2, 3, 4, 6, respectively, with Area 5 designated for testing, as also done in [32] . In the multi-label classification task on ogbn-proteins [17] , one parent model is trained on nodes with odd labels and the other on nodes with even labels. Implementation follows the official codes provided by the Deep Graph Library (DGL) [52] and the original authors, including detailed architectures and hyperparameter settings. We set the interpolation factor α in Eq. 1 to 0.5 for all experiments, with a sensitivity analysis provided in Sect. D of the appendix. For models originally equipped with normalisation layers, we recompute the running mean and running variance for the student GNN. In particular, the recomputation of statistics is performed concurrently with that of the LFNorm layer for the child GNN in CMC. More implementation details are elaborated in Sect. E of the appendix.Comparison Methods. Considering the limited exploration of multi-GNN reuse in existing literature, we focus our comparison of the proposed DuMCC approach on the training-dependent KA [25] and training-free VPI [54] and VAPI [1] methods, as introduced in Sect. 4. Furthermore, in Sect. E of the appendix, we also provide the results of retraining a child model using combined parent datasets with ground-truth labels, which establish an upper bound for GRAMA's performance.Node Property Prediction. Tab. 2 presents the results for the multi-class node classification task.The proposed DuMCC framework, as shown in the table, achieves a more balanced performance across all datasets. It notably outperforms the parent models on datasets that were not used for their training. Additionally, the last two lines of Tab. 2 detail an ablation study on the proposed CMC scheme, demonstrating its ability to enhance performance beyond PMC. While our method slightly lags behind KA in performance, KA involves a complex re-training process, whereas our approach is completely training-free. A sample t-SNE visualisation of the results is provided in Fig. 2 .We further show in Tab. 3 the results of multi-label molecule property prediction. Notably, our approach slightly outperforms KA in Tab. 3, underscoring the limitations of KA discussed in Sect. 3. These limitations stem from KA's reliance on soft labels produced by the teacher GNNs, making it vulnerable to their misclassification errors. Furthermore, we explore the potential of multi-model GRAMA by concurrently reusing three pre-trained node classification GNNs, as discussed in Sect. F of the appendix.Graph Property Prediction. Tab. 3 also presents results for the graph classification task using GAT architectures. The proposed DuMCC demonstrates enhanced equilibrium in performance relative to the parent models. Notably, DuMCC slightly outperforms KA on ogbg-molbace, further illustrating that KA is vulnerable to the errors of pre-trained models. In contrast, our approach does not rely on soft labels, thus avoiding this limitation.3D Object Recognition and Semantic Parsing. Tab. 4 and Fig. 3 illustrate the quantitative results and qualitative visualisations for the point cloud classification task, respectively. The proposed DuMCC outperforms model I on dataset J and model J on Dataset I without requiring re-training. Moreover, Tab. 4 demonstrates that our approach significantly outperforms two vanilla methods. Fig. 3 further illustrates the structure of the feature space, revealing that our method produces semantically similar structures to those achieved by KA with re-training. We also show in Tab. 5 the results for the large-scale indoor semantic segmentation task. Our method notably surpasses other learning-free GNN reuse methods VPI and VAPI. Further qualitative and quantitative results across various dataset splits and network architectures are provided in Sect. E of the appendix. 
conclusions and limitations: 
 In this paper, we explore a novel GRAMA task for learning-free GNN reuse. The child model from GRAMA is expected to functionally merge knowledge from pre-trained parent models. Uniquely, GRAMA establishes the first paradigm in GNN reuse that operates entirely without re-training or finetuning, while also eliminating the need for ground-truth labels. To this end, we start by developing two vanilla GRAMA approaches, which reveal specific challenges inherent to GRAMA. These challenges motivate us to develop a DuMCC framework for topology-aware model reuse, leveraging a parent message coordination scheme followed by child message calibration. Experiments on node-and graph-level tasks across various domains demonstrate the effectiveness of the proposed approach for annotation-free knowledge transfer without additional learning.Despite its strengths, the proposed DuMCC is primarily designed for homogeneous GRAMA, as discussed in Sect. 3. Currently, the framework does not support cross-architecture heterogeneous GRAMA, where parent models have different architectures, such as a combination of GCN and GraphSAGE. Additionally, it does not handle scenarios where parent models address tasks at different levels, such as node-level versus graph-level tasks-another aspect of heterogeneous GRAMA. These limitations primarily arise from the absence of direct correspondence between the differing architectural layers of the parent models, an issue we plan to explore in our future work. We will also explore the possibility of a fully data-independent GRAMA scheme and investigate broader applications beyond training-free model reuse, such as its use as a pre-processing step to facilitate graph-based knowledge amalgamation. Further discussions on limitations and potential solutions are provided in Sect. H of the appendix. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper studies GNN reuse for resource-efficient graph representation learning and thereby, to our knowledge, poses no such risk. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The existing assets are cited and properly respected with the discussions provided in Appendix C. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: Details of the code and model are provided in the supplementary material. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The main claims in the abstract and introduction accurately reflect the paper's contributions and scope. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Neurons in each layer of neural networks can be permuted without altering network functionality[10,1]. 


Decision: The aim of the paper is the reuse of GNN models for transfer to new tasks without re-training, fine-tuning or annotated labels. It proposes a new Dual-Message Coordination and Calibration methodology to solve the problems of vanilla permutation-based approaches.

The reviewer consensus is that the paper is well motivated addressing an interesting and important problem in a novel way. It contributes to both the model fusion and GNN domains. The method is well presented and the experiments document convincingly its good performance. 

The submitted version of the paper raised numerous questions and requests for clarifications or improvements. The authors addressed most of these in their extensive and meticulous rebuttal to the satisfaction of the reviewers. As it was these additional pieces of information - e.g. new experimental results, corrected notation, improved explanations - that convinced the reviewers about the quality of the contribution, we urge the authors to update the final version of the paper accordingly.

Reviews: review: paper_summary: paper_summary: The paper is interested in re-using Graph Neural Networks trained from one task to another task (e.g., transfer learning). This motivation is nice. We've seen such trends in computer vision (e.g., network trained on ImageNet used for CIFAR), or more recently, everyone is using LLMs for a variety of tasks they were not trained on (by prompt engineering, LoRA, etc). The paper proposes to do the same for graphs. Their specific methodology starts by combining multiple GNNs. To do so, they want to permute the internal channels of the graph neural network so that the networks are best-aligned, before combing them.
strength_weakness: strength_weakness: While the paper is well-motivated and well-written, it nicely builds-up the reader's excitment to anticipate for the actual method. Once reader arrives to the well-anticipated Equation (3), reader discovers that the equation is not properly-written ("buggy"). I've written many papers with probably subtle bugs in them (e.g., perhaps some footnote or some text in some side-section has a bug). However, a bug in the main equation qualifies for an immediate rejection.

## Math incorrectness
After Eq 1:

"where $\mathbf{P}^∗$ represents the permutation matrices" -- what does that mean? The expression $\mathbf{P}^∗$  is not used in the Equation. I **think** it can imply one of two things. Option 1: $\mathbf{P}^∗ = [P^{(\ell)} ]_{\ell \in [L]}$; or Option 2: $\mathbf{P}^∗$ refers to the space of permutation matrices (e.g., a reshuffle of the identity matrix or its continuous relaxation which is probably any basis?)

Equation 3 has several issues:

1. Where does $i$ come from? do you mean to add $\sum_i$ between the $\arg\min$ and the Frobenius norm?
2. The aggregation function "Agg" is not introduced. While the exact definition is unnecessary, however, the domain and range should be clearly specified. Does it take a many vectors and output one vector? If yes, then $P \cdot \textrm{Agg}$ should be a vector and therefore you should be minimizing the L2 norm (not Frobenius norm). If it acts on the whole graph (i.e., $\textrm{Agg}$ outputs a matrix with shape `NumNodes x FeatureDim`), then you are multiplying the permutation matrix from the wrong side.
3. In general, where do the $X$s come from? I was expecting data-independent scheme. In my reading, at this point, I think paper **is "learning** to align" (because you are doing gradient decent on the permutation matrix by looping over data, no?)

-- with all honesty, I stopped reading the paper after arriving at Eq3 and skimming over the algorithm. Please address the above and I am happy to take another look during the rebuttal. That being said, I do trust that your implementation is correct, but the paper may not indeed reflect the implementation.
questions: questions: - Does the exact form of Equation 2 depends on the architecture of the GNN model? I would expect to see different formulas depending on the model (e.g., GraphSAGE vs GCN of Kipf vs GAT vs MixHop etc)

- Please respond to the main math weakness, which might cause you to be explicit about the notation
limitations: limitations: I did not spot a Limitations section.
paper_summary: paper_summary: The paper tackles pre-trained model fusion for graph-centric tasks. The pre-trained models share the same architecture, but differ in the graph datasets they are trained on. The pipeline involves two core approaches. The first one matches parameters in pre-trained parent models by aligning the aggregated messages of the pre-trained parent models. The second one modifies the message statistics for the child model to correspond with the overall statistics of the pre-trained models. The authors have validated two approaches on diverse datasets and models and show nice visualization results.
strength_weakness: strength_weakness: W1. Discussions on existing model fusion approaches for CNNs are not adequate. The presentation will benefit from a more comprehensive literature review.

W2. Lack of sensitivity analysis on the interpolation coefficient alpha.

W3. It seems that each task uses only one network architecture and dataset partition. More experiments are needed with more network architectures.

W4. It will be great to showcase more applications of the proposed approach besides training-free model reuse.
questions: questions: Address weaknesses as mentioned above.
limitations: limitations: Yes. There are no obvious negative society impacts I can notice.
paper_summary: paper_summary: This paper proposes Deep Graph Mating, a novel task for model reuse in non-Euclidean domains, specifically focusing on GNNs. The goal is to create a child GNN that combines knowledge from pre-trained parent GNNs without requiring re-training, fine-tuning, or ground-truth labels. The process firstly identifies the permutation invariance properties of GNNs, and accordingly explores two naïve methods: Vanilla Parameter Interpolation (VPI) and Vanilla Alignment Prior to Interpolation (VAPI), both of which were shown to be insufficient due to challenges such as parameter misalignment and topological dependencies. 

The contribution of this paper is to address this issue by proposing a Dual-Message Coordination and Calibration (DuMCC) methodology that optimizes the permutation matrices for parameter interpolation (PMC scheme) by incorporating topological information, and solves over-smoothing by calibrating the message statistics for child GNNs (CMC scheme). This results in performance on par with traditional re-training methods but without the associated training cost, which is validated on seven datasets across node and graph classification as well as semantic parsing tasks. Code and proofs are given in the appendix and the supplementary material.
strength_weakness: strength_weakness: I see the following weaknesses in this paper:

While I appreciate the authors’ attempt to present the complex trajectory of the paper (from defining the problem and elaborating the task motivation, to developing two naïve methods and analyzing challenges, and finally proposing two schemes) in a narrative style for easier understanding, there is an imbalance in the organization of content. The current organization of the paper only allocates less than two pages to the experiment section, leaving many experimental details to the appendix. I suggest the authors move more experimental details from the appendix into the main body of the paper, reducing the current extensive focus on the motivations of the method, and instead, expanding more information about the datasets used, enhancing the details and settings of the comparative methods, and elaborating on the experimental procedures. Also, the connection between the main paper and the appendix is very limited. The authors vaguely direct readers to Section D for further details, without specifying specific tables or paragraphs, which makes it hard to locate relevant information.

Also, the explanation regarding the heterogeneous GRAMA in Line 123 actually makes me confused. The statement is too vague. What exactly are the key differences between the homogeneous and heterogeneous cases of GRAMA? Does it specifically relate to the same or different architectures of the pre-trained models, or does it also involve models specializing in different tasks? For example, if there are two pre-trained models with the same architecture but trained for different tasks, would this scenario be classified as homogeneous or heterogeneous GRAMA? Furthermore, while the authors claim that “our initial investigation in this paper is confined to scenarios where pre-trained GNNs possess identical architectures yet are trained on separate datasets”, the authors should at least provide some insights or potential solutions for solving heterogeneous GRAMA, or perhaps consider removing this statement of heterogeneous GRAMA since this is not the contribution of this paper.

Following this discussion of heterogeneous GRAMA, I think it is at least worth a try to apply GRAMA to two different GNN variants, such as GCN and GraphSage, since the key learnable parameters in these GNNs are all MLPs. The existing literature has demonstrated the importance of MLPs to GNN performance [1]. Exploring this avenue could be very interesting; if successful, the contributions of this paper would be significantly enhanced. Furthermore, the results may shed light on the connections between different GNN variants in the weight space.

In Tables 2 to 5, the authors overlook an important comparative result: the performance of retraining a multi-dataset model that can jointly combine the expertise of both parent models. While this approach involves retraining, it can at least serve as an upper bound for GRAMA, showing the potential space for further improvement.

It is also not very clear to me why the authors chose a 20%/80% splitting ratio in the experiments, as there are too few explanations about the splitting protocol. For example, is the split random? More details should be provided, as this concerns the pre-trained models and can reveal how knowledge from the two pre-trained models is distributed and when the proposed method is effective. Would other splitting, such as 10/90, work for GRAMA? It is necessary to conduct more experiments with more splitting ratios to make the authors’ claims more convincing.

The current discussion of limitations in this paper appears somewhat constrained. The authors should consider expanding this section, either in the main body of the paper or in the appendix. Another limitation of this work, compared to previous approaches such as KA, is the combination of models working on different levels of tasks, such as node classification and graph classification. Weight-space combination for models addressing various tasks has recently been explored in the Euclidean domain such as ZipIt [2], making it intriguing to see whether similar results can also be achieved in the non-Euclidean domain.

Minor issues in the paper:

Line 26: "reduing" $\rightarrow$ "reducing";

Line 106: "presents" $\rightarrow$ "present";

Line 193: "are" $\rightarrow$ "is".

[1] Han, Xiaotian, et al. MLPInit: Embarrassingly Simple GNN Training Acceleration with MLP Initialization. ICLR 2023.

[2] Stoica, George, et al. ZipIt! Merging Models from Different Tasks without Training. ICLR 2024.
questions: questions: See weaknesses. Overall, I believe this paper addresses a significant problem and proposes a novel approach for model reuse in GNNs. The motivation is strong, supported by solid theoretical analysis. Despite the merits, I see several weaknesses. Therefore, I rate the paper as borderline at this stage and look forward to the authors' responses and further discussions regarding my concerns in the weaknesses section.
limitations: limitations: Yes, limitations and broader impacts have been discussed. An additional limitation to consider would be the scenario where two pre-trained models address tasks at different levels, such as node and graph-level tasks.
paper_summary: paper_summary: In this paper, the authors present a method for learning-free and label-free model reuse for graph neural networks. They dub the task Deep Graph Mating (Grama). Without relying on costly fine-tuning or re-training, Grama aims to generate a child model by reusing and fusing knowledge from pre-trained parent models, particularly by managing pre-trained parameters. Compared with conventional model reuse tasks (knowledge distillation, and knowledge amalgamation) in the GNN domains, the authors claim in Tab 1 that Grama simultaneously enables multi-model reuse, annotation free, and training/fine-tuning free, leading to more resource-efficient model reuse. To achieve Grama, the authors begin by deriving two vanilla Grama methods, dubbed VPI and VAPI, motivated by the permutation invariance proposition in GNNs. However, both VPI and VAPI don’t work well, and the authors observe unique challenges for Grama, namely the increased sensitivity to weight misalignment and the accompanying topology-dependent complexities, substantiated by a series of propositions, lemmas, and conjectures. Motivated by the identified challenges, the authors introduce the proposed DuMCC framework, which includes a PMC and CMC. PMC is designed to identify topology-aware permutation matrices but suffered from increased susceptibility to over-smoothing. To solve this problem, the authors further designs CMC, which is to refine message statistics of the child network through a learning-free message normalization layer. Extensive experiments (7 benchmarks, 4 tasks, 5 architectures) in various graph applications have been performed to validate the effectiveness of the proposed framework, covering popular GNNs including GCN, GraphSAGE, GAT, GIN, and DGCNN. The authors also provide extensive supplementary information with detailed proofs in the appendix.
strength_weakness: strength_weakness: - In Sec 5.3, the authors introduce a learning-free message normalization (LFNorm) layer for statistics calibration in CMC, which is technically sound. However, the authors miss the discussions on the Grama case where pre-trained models already contained normalization layers, which is common in the area of graph neural networks. In that case, it is unclear whether the batch statistics are also recomputed in CMC. If they are, it should be specified when the recomputation occurs: simultaneously with LFNorm or after?

- Because the authors have claimed multimodel reuse as a contribution of this work, I would advise the authors to include more experiments for the Grama cases with three or more pre-trained models. Otherwise, clear statements should be added to limit the scope of the study to the reuse of two pre-trained models.

- The descriptions and explanations in the experimental results are not closely matched with the authors’ assumptions and propositions from the previous sections. It is suggested that the authors more closely relate the texts in the experimental section to the methods section, particularly by specifying which results validate which proposition or claim. This would make the paper more readable.

- In the appendix, there is a minor problem where Eq 30 extends beyond the margin of the paper.
questions: questions: 1. Can you clarify if the statistics in the original normalization layer are recomputed during Grama? If so, when would they be recomputed?

2. Have you experimented with using three or more pre-trained models in Grama?

3. Consider improving the experimental section to more closely relate the texts to the validated claims.

Although the authors missed some key information, I feel these could be easily addressed. On balance, the paper is both novel and useful. The concept of training-free and label-free model reuse for GNN had not been done before.
limitations: limitations: The paper has thoroughly discussed limitation in Sec 7 and social impact in appendix.

Comments: title: Final rating
Response to Reviewer 856w
Response to the rebuttal
Response to Reviewer Uv5e
Quesiton about homogeneous GRAMA
Response to Reviewer rPnJ (Part 2/2)
Response to Reviewer Uv5e (Part 2/2)
comment: Since the reviewers have responded to all my comments, I am increasing my score.

Thank you for your good work!
I thank the authors for sending their detailed further clarification. I've updated my rating accordingly, considering that the responses have resolved all of my concerns satisfactorily. I am also pleased that the authors are open to further discussions. I hope the authors attend to the main points raised in the reviews as they prepare an updated paper version.
>"There is one minor issue remaining. When reviewing again the related work on model fusion, I noticed several works in the reference list have the wrong pubilcation year. For instance, the well-known git rebasin work ([1] in the paper) was actually published at last year's ICLR, NOT 2022; Zipit [57] was at this year's ICLR, not in 2023. Double check the reference list for such errors."

`Response:` We sincerely appreciate the reviewer for the positive support and the constructive comment regarding the issues in our reference entries. We have thoroughly reviewed all the entries in the reference section and will correct the entries accordingly in our revision as follows:

*Original references:*

"

[1] Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In ICLR, 2022.

[26] Moritz Imfeld, Jacopo Graldi, Marco Giordano, Thomas Hofmann, Sotiris Anagnostidis, and Sidak Pal Singh. Transformer fusion with optimal transport. In ICLR, 2023.

[57] George Stoica, Daniel Bolya, Jakob Brandt Bjorner, Pratik Ramesh, Taylor Hearn, and Judy
Hoffman. Zipit! merging models from different tasks without training. In ICLR, 2023.

[61] Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In ICLR, 2019.

"

*Revised references:*

"

[1] Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In ICLR, 2023.

[26] Moritz Imfeld, Jacopo Graldi, Marco Giordano, Thomas Hofmann, Sotiris Anagnostidis, and Sidak Pal Singh. Transformer fusion with optimal transport. In ICLR, 2024.

[57] George Stoica, Daniel Bolya, Jakob Brandt Bjorner, Pratik Ramesh, Taylor Hearn, and Judy
Hoffman. Zipit! merging models from different tasks without training. In ICLR, 2024.

[61] Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In ICLR, 2020.

"
Thanks for the well-structured rebuttal. I'm happy with the authors' reply and particularly appreciate the well-prepared and ready-to-incorporate text in the rebuttal, which strongly convinces me that the authors will make the corresponding necessary changes to improve the manuscript. 

Generally speaking, there is no doubt that the paper makes a clear contribution to both the fields of model fusion and gnn, with novel insights like the amplified parameters sensitivities due to topology, which adds new knowledge to the field. After careful consideration for my final justification, I have decided to raise my score to 6: weak acceptance.

There is one minor issue remaining. When reviewing again the related work on model fusion, I noticed several works in the reference list have the wrong pubilcation year. For instance, the well-known git rebasin work ([1] in the paper) was actually published at last year's ICLR, NOT 2022; Zipit [57] was at this year's ICLR, not in 2023. Double check the reference list for such errors.
>"I want to thank the authors for their detailed responses and the extensive efforts put into conducting new experiments. Most of my questions have been answered satisfactorily. However, I still find the clarification on heterogeneous GRAMA somewhat vague. As I mentioned in my review, in the case of two pre-trained GNN with exactly the same architecture but trained for different domain tasks, should this scenario be classified as homogeneous or heterogeneous GRAMA? I would appreciate further elaboration on this point."

`Response:` We appreciate the reviewer's follow-up discussion and would like to apologise for not explicitly defining and explaining heterogeneous GRAMA in our paper and the rebuttal.

We would like to clarify that heterogeneous GRAMA refers to scenarios where pre-trained parent models have either diverse architectures or are designed for different domain tasks. Indeed, the case mentioned by the reviewer falls under heterogeneous GRAMA.

In our revision, we will address the reviewer's comments by explicitly defining heterogeneous GRAMA. Specifically, we will replace the sentence in lines 123-124 of the main paper, "The exploration into varied architectures and tasks (i.e., heterogeneous GRAMA) remains a topic for subsequent future studies", with the following:

"We reserve the exploration of more challenging heterogeneous GRAMA scenarios, where pre-trained parent models either have diverse architectures or are designed for different domain tasks, as a topic for future studies."
I want to thank the authors for their detailed responses and the extensive efforts put into conducting new experiments. Most of my questions have been answered satisfactorily. However, I still find the clarification on heterogeneous GRAMA somewhat vague. As I mentioned in my review, in the case of two pre-trained GNN with exactly the same architecture but trained for different domain tasks, should this scenario be classified as homogeneous or heterogeneous GRAMA? I would appreciate further elaboration on this point.
`W2.3(a)` *Clarity on $\mathbf{X}$s*

>"In general, where do the $\mathbf{X}$s come from? I was expecting data-independent scheme."

`Response:` The reviewer's point is very well taken. In our paper, we initially introduced a fully data-independent approach for GRAMA, referred to as the vanilla VAPI method, described in lines 158-161: "One possible data-independent solution is to minimise the L2 distance between the weight vectors of the pre-trained models by solving a sum of bilinear assignments problem, similar to weight matching techniques described in [1]." 

However, as discussed in Sect. 4.2, our subsequent analysis reveals that GNNs are particularly sensitive to mismatches in parameter alignment, which are "contingent upon the topological characteristics inherent to each graph" (Conjecture 4.1). This sensitivity renders the data-independent solution less effective. Motivated by this, we propose integrating the topological characteristics inherent to each graph. To achieve this, we have to resort to passing the graph data to the pre-trained model to capture these graph-specific topological characteristics, utilising $\mathbf{X}$.

Nevertheless, we clarify that our PMC and CMC methodologies require only a single forward pass of the unlabelled graph data to extract messages for alignment and calibration, respectively—eliminating the need for iterative training or ground-truth labels. Our immediate-next goal is, exactly as the reviewer suggested, to explore the possibility of an entirely data-independent GRAMA scheme, for example, by initially generating fake graphs as described in [r3]. We will include these discussions in the revised version.


[r3] Deng and Zhang. Graph-free knowledge distillation for graph neural networks. In IJCAI, 2021


`W2.3(b)` *Details of aligning*
>" In my reading, at this point, I think paper is 'learning to align' (because you are doing gradient descent on the permutation matrix by looping over data, no?)"

`Response:` We apologise for any confusion caused by our previous lack of clarity and the omission of essential details. We would like to clarify that our method does not employ gradient descent for alignment. In the field of model merging within the Euclidean domain, the minimisation problem in the form of Eq. 3 is typically transformed into a maximisation problem to maximise an inner product (as derived from expanding Eq. 3), thereby fitting it within the framework of a standard linear assignment problem [r1, r2, r3, r4]. In our revision, we will include these crucial details and introduce a preliminary section on Euclidean model merging to ensure our paper is self-contained.


[r1] Ainsworth, et al. Git re-basin: Merging models modulo permutation symmetries. In ICLR, 2023.

[r2] Li, et al. Deep model fusion: A survey. 2023.

[r3] Liu, et al. Deep neural network fusion via graph matching with applications to model ensemble and federated learning. In ICML, 2022.

[r4] Stoica, et al. Zipit! merging models from different tasks without training. In ICLR, 2024.


---

`Q1.` **Model-specific variants of Eq. 2**

>"Does the exact form of Equation 2 depends on the architecture of the GNN model? I would expect to see different formulas depending on the model (e.g., GraphSAGE vs GCN of Kipf vs GAT vs MixHop etc)"

`Response:` We appreciate the constructive comments provided by the reviewer. To maintain clarity and simplicity, Eq. 2 in our paper is based on the most basic form of GNNs. Due to the short rebuttal period, we have only been able to develop specific formulas for GraphSAGE and Kipf's GCN as suggested by the reviewer. Our model formulation presented here follows the unified mathematical framework detailed in [r5], albeit with symbols adapted to those used in our paper. In the revised version, we are committed to developing more detailed, model-specific formulas.

(We apologise for having to split each of our equations below into two lines for display purposes, as OpenReview does not support single-line presentation of long equations.)

*Eq. 2 tailored for GCN of Kipf is as follows:*
$$
\Delta F_i \approx \sigma'\left(W  \sum_{j \in \mathcal{N}(i)} \frac{X_j}{\sqrt{\deg}_i \sqrt{\deg}_j}\right) 
$$

$$
\cdot \left(\epsilon  \sum_{j \in \mathcal{N}(i)} \frac{X_j}{\sqrt{\deg}_i \sqrt{\deg}_j}\right)
$$

*Eq. 2 tailored for GraphSAGE is as follows:*
$$
\Delta F_i \approx \sigma' \left(W \ \text{Concat}\left(X_i, \text{Mean}_{j \in \mathcal{N}(i)} X_j\right)\right)  
$$

$$
\cdot \left(\epsilon \ \text{Concat}\left(X_i, \text{Mean}_{j \in \mathcal{N}(i)} X_j\right) \right)
$$


[r5] Dwivedi, et al. Benchmarking graph neural networks. In JMLR, 2023.


---

`L1.` **Limitations**

>"I did not spot a Limitations section."

We apologise for not making the limitations section more explicit in our paper. In fact, we have included a section on limitations in Sect. 7, titled "Conclusions and Limitations". We will highlight this section further in the introduction.
`W6.` **Comparative upper bound results of re-training**

>"In Tables 2 to 5, the authors overlook an important comparative result: the performance of retraining a multi-dataset model that can jointly combine the expertise of both parent models."

`Response:` We appreciate the reviewer's constructive suggestion. Following the reviewer's advice, we conducted additional experiments by re-training a model using combined parent datasets with ground-truth labels. The results are presented below.

As the reviewer pointed out, these results can indeed establish an upper bound for GRAMA's performance. We will incorporate these results into Sect. 6 and provide the corresponding discussion in the revision.

| Tables | Datasets  | Parent 1 | Parent 2 | Re-training |
|:----- | :----- | :----- | :----- | :----- |
| Tab. 2 | ogbn-arxiv | 0.7193 / 0.5516 |  0.6564 / 0.7464 | 0.6903 / 0.7268 |
| Tab. 2 | ogbn-products | 0.7982 / 0.7308 | 0.7626 / 0.7904 | 0.7981 / 0.7787 | 
| Tab. 3 | ogbg-proteins | 0.7478 | 0.7222 | 0.7514 | 
| Tab. 3 | ogbn-molbace, ogbg-molbbbp | 0.7247 / 0.4681 | 0.4067 / 0.6366 | 0.6954 / 0.6087 |
| Tab. 4 | ModelNet40 | 0.9159 / 0.8151 | 0.8862 / 0.9275 | 0.9390 / 0.9243 | 
| Tab. 5 | S3DIS | 0.8181 | 0.8174 | 0.8428 | 

---

`W7.` **Experiments with additional splitting ratios**

>"It is also not very clear to me why the authors chose a 20%/80% splitting ratio in the experiments, as there are too few explanations about the splitting protocol. For example, is the split random? Would other splitting, such as 10/90, work for GRAMA?"

`Response:` We would like to thank the reviewer for the constructive comments. In fact, we have indeed addressed the choice of our dataset partition strategy in lines 299-302 of the main paper, following the well-established methodology of "Git Re-basin" by Ainsworth et al. [1], which involves random partitioning of the dataset.

Following the reviewer's suggestion, we conducted further experiments with an additional 90%/10% random split on the relatively large-scale ModelNet40 dataset, used for 3D object recognition tasks. The results, presented below, further substantiate the effectiveness of our proposed method.


Datasets | Parent 1 | Parent 2 | Ours (w/o CMC) | Ours (w/ CMC)
|:----- |:----- | :----- | :----- | :----- |
| Dataset 1 | 0.9246 | 0.8393 |  0.8451 | 0.8782 |
| Dataset 2 | 0.7621 | 0.9294 |  0.8374 | 0.8422 |


---

`W8.` **Constrained limitation discussions**

>"The current discussion of limitations in this paper appears somewhat constrained. The authors should consider expanding this section, either in the main body of the paper or in the appendix. An additional limitation to consider would be the scenario where two pre-trained models address tasks at different levels, such as node and graph-level tasks."

`Response:` The reviewer's point is well taken. Indeed, our current DuMCC framework does not accommodate scenarios where the parent models tackle tasks at different levels, a challenge that falls within the scope of heterogeneous GRAMA as outlined in `W3`. To address this, we will enhance our discussion on limitations by introducing a new Sect. G titled "Limitation Discussion" in the appendix. This section will offer a detailed analysis of this limitation and provide the corresponding potential solutions for such cases, as exemplified in `W4`.

---

`W9.` **Typos**
>"Line 26: 'reduing' - 'reducing';
Line 106: 'presents' - 'present';
Line 193: 'are' - 'is'."

`Response:` We appreciate the reviewer's thorough examination of our paper. We will address these typographical errors in our revisions by correcting "reduing" to "reducing" in line 26, removing the extraneous "s" from "presents" in line 106, and adjusting "are" to "is" in line 193 to ensure grammatical correctness.