Scalable Multi-Source Pre-training for Graph Neural Networks

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-25 05:44:36

Abstract: Graph Neural Networks (GNNs) have been shown as powerful tools in various scenarios, such as multimodal and multimedia. A fundamental approach, pre-training on available graphs and subsequently transferring the acquired knowledge to optimize downstream tasks with limited labels, was widely exploited to mitigate the demand for extensive labeled training data. However, previous works commonly assumed that pre-training and fine-tuning occur in the same or closely related domains that share similar feature/label spaces and graph distributions. A limitation is that for each individual graph without accessible pre-training data, a GNN must be trained from scratch, imposing high training overhead and hindering the ability of generalization. In this paper, we address the \emph{GNN multi-domain pre-training problem}, which intends to pre-train a transferable GNN model from heterogeneous multi-source graph domains and then apply it in an unseen one with minor fine-tuning costs. To this end, we propose a sca\underline{LA}ble \underline{M}ulti-source \underline{P}re-training (LAMP) method. For pre-training, LAMP presents a graph dual-distillation approach to distill massive knowledge from various graph domains to form synthetic homogeneous graphs. Simultaneously, high-level meta-knowledge from the synthetic graphs is extracted to train the GNN model, whose capability can be adjusted according to target graph contexts through a co-training modulation architecture. For fine-tuning, LAMP respectively aligns the target graph distribution, graph context, and graph task with the pretext so that the downstream task in the unseen domain can be reshaped to leverage the transferable knowledge efficiently. Extensive experiments on four real-world graph domain datasets demonstrate the superiority of LAMP, showcasing notable improvements in various downstream graph learning tasks. Our codes are publicly available on GitHub.

Main paper: 
 introduction: 
 Graph neural networks (GNNs) have emerged as powerful models in handling graph-related data with rich relational information [19, 24, 58] . Hence, GNNs exhibited significant potential in many real-world networking systems [35, 46, 50, 52, 54, 56] . By employing neighborhood aggregation and message passing among graph nodes, GNNs can output informative node representations, which play a crucial role in diverse machine learning tasks [4, 31, 45] .To alleviate the burden of substantial handcraft annotations and the cost of model retraining from scratch, graph pre-training has become a fundamental approach to enable knowledge transfer among various graph learning tasks. Conventional GNN pre-training follows a two-stage process that involves pre-training and fine-tuning. The pre-training stage captures transferable graph patterns from accessible graph information, and the fine-tuning stage tunes the pre-trained model to generalize the pre-trained knowledge to the downstream tasks. In line with the process, many works designed pre-training tasks to capture the latent knowledge [12, 14, 26] more effectively. They leveraged substantial unlabeled structural data, evolving from mechanically tailored tasks (e.g., masking techniques [12, 14] and contrastive learning [30, 37, 51] ) to intentionally designed strategies (e.g., generative pre-training [13] and learning to pre-train [28] ). However, there is a clear gap between the pretraining and downstream tasks when their training objectives are far more different, leading to failure knowledge elicitation (a.k.a negative transfer [41, 44] ). To this end, recent works have introduced graph prompt [27, 32, 33] in the fine-tuning stage to bridge the gap by aligning downstream objectives with that of the pre-training.However, previous approaches mainly work in settings where pre-training and fine-tuning occur in the same or closely related domains with heterogeneous feature/label spaces and graph distributions, making it unlikely to transfer knowledge from heterogeneous domains. In this paper, we study a more general graph pre-training problem called GNN multi-domain pre-training, where a GNN model is trained on multi-source graphs with heterogeneous feature/label spaces and distributions, and then applied to an unseen domain without train-from-scratch. A motivation scenario of our work is illustrated in Fig. 1(a) , where a general GNN model is trained based on multiple heterogeneous graphs (i.e., citation, social, and comment networks) to learn the graph-related knowledge, and then applied to a co-purchasing network for product recommendation with the minor cost of fine-tuning by a few labeled samples. To the best of our knowledge, this problem was not addressed in the literature, and it poses the following challenges.• Heterogeneous domains: Accessible domains for pre-training have massive knowledge, and different graph domains may have heterogeneous feature/label space and data distributions. Thus, a scalable mechanism for co-training on various graph domains is necessary to alleviate domain heterogeneity. • Transferable knowledge: In the situation when pre-training and fine-tuning domains exhibit substantial divergence, how to extract informative transferable knowledge from source domains and adapt it to the unseen domain efficiently requires sophisticated framework design. • Pretext and downstream task gap: The gap between the constructed pretext and the dedicated downstream task from unseen domains is more pronounced. Mitigating this gap to avoid "negative transfer" is a key challenge.To address the aforementioned challenges, we propose a novel scaLAble Multi-source Pre-training (LAMP) framework for GNN learning. LAMP is specifically designed to pre-train a GNN model on multi-source domains scalably, aiming to extract common transferable knowledge that can be effectively generalized to the graphs in target domains. As shown in Fig. 1 (b), the LAMP framework comprises three major modules.(1) Synthetic graph distillation. This module aims to transform the massive knowledge from multi-source heterogeneous domains into a set of synthetic small graphs with unified feature spaces and distributions. To achieve this, we propose a graph dualdistillation method which can jointly distill the intrinsic semantic knowledge and external graph distribution from multiple domains into synthetic graphs.(2) Modulated meta pre-training. This module aims to extract transferable knowledge which can be exploited by unseen domain graphs. Thus simultaneously with the graph synthesis, we derive high-level meta knowledge by sampling link prediction meta-tasks for pre-training. Moreover, we co-train an additional modulation architecture to adapt the pre-trained GNN model according to different graph contexts so that the model's representational capacity can be improved for better fine-tuning.(3) Knowledge transfer for downstream tasks. This module aims to enable the GNN to seamlessly utilize the pre-trained knowledge in downstream tasks. Therefore, leveraging the intermediate results from pre-training, alignment methods are proposed by matching downstream graph distribution with the synthetic graphs, employing the co-training modulation architecture to adjust the pre-trained GNN model, and reshaping the downstream task in line with the form of the pretext task.The contributions of our work are summarized as follows.• We are the first to address the multi-domain GNN pre-training problem. Our study can overcome the limit of "one-graph-onemodel" for conventional GNNs and shed a light on building foundation models with multiple heterogeneous graph domains.  
related work: 
 Graph Neural Networks (GNNs) have prevailed for learning graph data representations and were found wide applications in diverse fields such as social networks [5, 46, 54] , knowledge graphs [34, 35, 50] , recommendation systems [8, 52, 56] , and multimedia [39, 42, 47] . Utilizing neighboring information aggregation and message passing mechanisms, many effective GNN structures were proposed including graph conventional network (GCN) [19] , graph attention network (GAT) [36] and graph isomorphism network (GIN) [48] .They have proven effective for node classification [9, 19] , graph classification [1, 21] , and link prediction [23, 31] . Inspired by the successful pre-training techniques employed in neural language processing (NLP) [6, 20, 22] and computer vision (CV) [3, 17, 25] , graph pre-training presented significant prospects of the swift and efficient training of GNN models with reducing annotation costs and alleviating the requirement of training from scratch. It acquired intrinsic graph knowledge by training GNN models on easily accessible graphs and transferring such knowledge to downstream tasks by updating the pre-trained GNN weights. Consequently, the conventional graph pre-training method adhered to a pre-train/finetune procedure, typically conducted within a single graph domain.We summarize the state-of-the-arts into two categories:(1) Graph self-supervised learning methods. Initially, various pre-training approaches employed distinct self-supervised tasks based on diverse graph attributes, including node features [12, 14] , graph edges [12, 14] , and graph contexts [12, 37] . These tasks were inspired by conventional self-supervised methods, which involve the masking of partial patterns (e.g., node properties and edges) in the graph and then train a GNN model on the remaining graph to predict the masked attribute. Recently, contrastive learning was used to capture universal network topological properties across multiple networks [30] . Meanwhile, GraphCL [51] minimized the distance between pairs of graph-level representations for the same graph. Additionally, GPT-GNN [13] adopted the concept of generative language model pre-training and established an autoregressive framework to iteratively perform reconstruction on given graphs. L2P-GNN [28] simulated fine-tuning during pre-training to allow the pre-trained model to be adapted to downstream tasks quickly.(2) Graph prompt learning methods. The gap between different pre-training and downstream objectives can potentially lead to negative transfer [32, 41, 44] . To address this issue, increasing attention was directed towards graph prompt-tuning. Prompting originated from large language models [55] , aiming to integrate pre-training and downstream tasks within a common task template. For example, GPPT [32] reframed node classification as the pretext of edge prediction by introducing labeled tokens to the original graph. GraphPrompt [27] employed a learnable prompt to assist a downstream task in identifying the most relevant knowledge from the pretext. ProG [33] standardized the format of graph and language prompts, and utilized meta-learning to acquire a better initialization for the prompt efficiently. These works introduced graph prompts in the fine-tuning stage to bridge the gap by aligning downstream objectives with that of the pre-training.In contrast to the existing works that confine the pre-training and fine-tuning processes within the same or similar domains, we are the first to address the heterogeneous multi-domain GNN pretraining problem. We proposed a novel GNN pre-training framework to generalize latent knowledge from multi-source domains and transfer it to downstream tasks in unseen domains.  
preliminaries: 
 To simplify the presentation, we primarily focus on undirected graphs. Let G = (𝑋, 𝐴) denote an undirected graph, where 𝑋 ∈ R 𝑁 ×𝑑 represents the 𝑑-dimensional node feature for a total of 𝑁 nodes; 𝐴 ∈ R 𝑁 ×𝑁 is the adjacency matrix of the graph. A GNN model iteratively updates the presentation of a node by aggregating the representations from its neighbors across multiple layers. Specifically, the operation in each layer can be formulated as: 1) , 𝐴; 𝜃 (𝑙 ) ),𝑍 (𝑙 ) = M (𝑍 (𝑙 -where 𝑙 = 1, 2 . . . , 𝐿 is the layer indicator of the GNN model, and 𝑍 (0) = 𝑋 . In the equation, M (•) represents the aggregation function, and 𝜃 = {𝜃 (1) , 𝜃 (2) , . . . , 𝜃 (𝐿) } denotes the model parameters.The general objective of GNN learning tasks can be formulated as:EQUATIONwhere ℓ (•, •) is loss function to measure the difference between the predictions Pred(𝑍 (𝐿) ) and corresponding true labels 𝑌 . For a clearer discussion, we use GNN(G; 𝜃 ) to represent the final node representations 𝑍 (𝐿) for the GNN model and exploit 𝑓 𝜓 (•) with parameter 𝜓 to serve as an overall prediction function, combining the prediction operation in Pred(•) and the loss function ℓ. Thus 𝜃,𝜓 are the parameters to be optimized during task training. The objective of our proposed multi-domain pre-training is to extract diverse transferable knowledge and effectively transfer it to the downstream tasks in an unseen domain. Specifically, assume that we have 𝐾 different domains, which are massive in knowledge and heterogeneous in features and label spaces.{G 𝑘 |(𝑘 = 1, 2, • • • , 𝐾)}are subgraphs sampled from them respectively. Also, a target graph represented by G * for the downstream tasks is from another distinct unseen domain rather than the multi-source domains. Compared to the conventional single-domain pre-training process, the proposed multi-source pre-training can produce a more uniform GNN model with better generalization. The detailed formulations of the conventional and our proposed pre-training approaches, as well as the notations summarization, can be found in Appendix A. 
the lamp method: 
 The LAMP framework is illustrated in Fig. 1 Then, it extracts link prediction meta-tasks from these synthetic graphs to pre-train a GNN model and a modulation architecture to adapt GNN models to different graph contexts. During fine-tuning, it aligns the downstream graph with the pretext from the views of graph distribution, graph context, and graph task to mitigate "negative transfer" as much as possible. The details of the proposed method are as follows.The common practice of pre-training within a single domain limits the consideration of more diverse patterns. However, graphs originating from diverse domains often exhibit heterogeneous feature spaces and graph distributions, posing challenges to processing them within a single GNN model. To address the heterogeneity, we transform the multiple graph domains into synthetic homogeneous graphs with a unified feature space via knowledge distillation. Specifically, we propose a graph dual-distillation method to approximate knowledge and match graph distribution between the multi-domains and the synthetic graphs.(1) Knowledge Approximation. We denote 𝑇 synthetic graphs as{S 𝑡 |𝑡 = 1, 2, • • • ,𝑇 } generated by parameters {𝜙 S 𝑡 |𝑡 = 1, 2, • • • ,𝑇 }.Meanwhile, we employ a set of deep neural networks (DNNs){𝑔 𝑘 |𝑘 = 1, 2,• • •, 𝐾 } with parameters {𝜙 G 𝑘 |𝑘 = 1, 2,• • •, 𝐾 } to trans- form the 𝐾 multi-source graphs {G 𝑘 } into a unified node feature space with dimension 𝑑. Thus, {𝑔 𝑘 (G 𝑘 )|𝑘 = 1, 2,• • •, 𝐾 }means the transformed multi-source subgraphs. The knowledge approximation operation distills the self-supervised intrinsic semantic knowledge into the synthetic graphs. Therefore, the objective is to form a set of small synthetic graphs so that GNN parameters (initialized randomly and trained on synthetic graphs {S 𝑡 }) can achieve competitive performances to the parameters (initialized randomly and trained on original multi-source domain graphs {𝑔 𝑘 (G 𝑘 )}) in specific self-supervised tasks.The objective can be achieved in a bi-level manner [43] to train the GNNs by minimizing an approximation loss L appr :L appr = 𝑓 𝜓 (GNN( {𝑔 𝑘 (G 𝑘 )} ; 𝜃 syn ), {𝑌 kwg 𝑘 }),whereEQUATIONwhere 𝑌 kwg is the specific labels of certain self-supervised knowledge and 𝜂 is the inner learning rate. This equation intuitively suggests that GNN model parameters 𝜃 initialized randomly and then trained on the synthetic graphs as 𝜃 syn should achieve a minimized loss when they are applied to the transformed graph 𝑔 𝑘 (G 𝑘 ). Hence, 𝜃 syn is derived as a function of {S 𝑡 } with parameters {𝜙 S 𝑡 }, and thereby the objective loss L appr is differentiable w.r.t both {𝜙 S 𝑡 } and {𝜙 G 𝑘 }, allowing for the joint optimization using standard gradient-based methods. In this way, this process can preserve intrinsic semantic information across multi-source domains into multiple small graphs. The semantic information can be derived from any self-supervised task, and we leverage the Graph Autoencoder (GAE) [18] in this paper. However, preserving intrinsic semantics alone is insufficient for knowledge distillation since external graph distribution is also crucial for graph domains.(2) Distribution Matching. Graph distribution encompasses node features and graph structures, which is hard to calculate qualitatively. Hence, we propose to represent graph distribution by viewing it as a collection of walk distributions, where zero-walk distribution is equivalent to node distribution, and one-walk distribution represents edge distribution until the 𝑟 -th (𝑟 = 0, 1, • • • , 𝑅) walk distribution related to 𝑟 + 1 neighboring nodes. These distributions can be represented by the empirical matrices 𝑃 G = {𝑃 0 G , 𝑃 1 G , . . . , 𝑃 𝑅 G , } = {𝑋, Ā1 𝑋, . . . , Ā𝑅 𝑋, }, where { Ā𝑟 } indicates the average normalized powered adjacency matrics and 𝑋 is the node feature vectors. With the empirical matrices for graph distribution, the distance between the ground truth and the synthetic graph distributions can be estimated through the commonly used Wasserstein Distance [38] .Wasserstein distance is developed to measure the distance between two discrete distributions. Formally, Wasserstein distance with the first moment can be written as:EQUATIONwhere P src , P tgt are the real distributions for source and target domains respectively, and ∥ 𝑓 𝜔 ∥ 𝐿 ≤ 1 is the Lipschitz norm of 𝑓 𝜔 . The equation indicates the existence of an optimal 1-Lipschitz function 𝑓 𝜔 that separates P src and P tgt , with its maximum expectation being the first Wasserstein distance. Thus, a trained neural network 𝑓 𝜔 * parameterized by 𝜔 * can serve as a Wasserstein discriminator to fit such a 1-Lipschitz function. The discriminator can be obtained by maximizing the formulaE 𝑥∼P src [𝑓 𝜔 (𝑥)] -E 𝑥∼P tgt [𝑓 𝜔 (𝑥)],where the expectation can be approximated by empirical average value. Thus, Eq.( 4) takes all samples as input and outputs a real number.Distribution matching intends to minimize the Wasserstein distance between two graph distributions. We use 𝑃 (G) to empirically estimate the Wasserstein distance between the multi-domain graphs and the synthetic graphs. Therefore, distribution matching can be achieved by minimizing the distribution distance loss L dis :L dis = 𝑅 ∑︁ 𝑟 =0 𝑊 1 (𝑃 𝑟 { S 𝑡 } , 𝑃 𝑟 {𝑔 𝑘 (𝐺 𝑘 } ) = 𝑅 ∑︁ 𝑟 =0 [ 𝑓 𝜔 * (𝑃 𝑟 { S 𝑡 } ) -𝑓 𝜔 * (𝑃 𝑟 {𝑔 𝑘 (𝐺 𝑘 ) } )],whereEQUATIONIt quantifies the graph distribution distance between the synthetic and multi-source graphs by amalgamating empirical Wasserstein distances for up to 𝑅-walk distributions. Minimizing L dis is also a bi-level process. Firstly, the distance function 𝑓 𝜔 with trainable parameters 𝜔 can be maximized to the supremum, which is the Wasserstein distance of the graph distributions. Secondly, L dis is then minimized by freezing the trained 𝜔 * and optimizing {𝜙 S 𝑡 } and {𝜙 G 𝑘 } so that the distribution distance can be narrowed down.Knowledge approximation ensures that knowledge from multiple source domains is integrated into synthetic graphs. Distribution matching helps synthetic graphs retain the distribution from source graphs. In the implementation of bi-level process for L appr , L dis , the prepositive optimizations of 𝜃 syn and 𝜔 * can be conducted within a fixed number of steps as in the literatures [15, 16, 49, 57] .To adapt the pre-trained GNN model to an unseen graph domain, the divergence in graph domains can be tackled in two aspects. The first involves domain-specific graph patterns. For instance, a citation network may exhibit sparser connections than a co-purchasing network. A single GNN model initialization may struggle to handle varying graph patterns from different domains. Thus, we co-train an additional modulation architecture to adjust the pre-trained model capability to different downstream graph patterns without changing the model's definition and introducing extra inference overhead. The second concerns domain-specific semantic information. For instance, information from a citation network may not be useful for a co-purchasing network. We argue that meta-knowledge, encompassing learning experiences, is high-level and more transferable across diverse graph domains than knowledge from basic tasks. Hence, meta-tasks are explicitly designed for GNN pre-training.We first introduce the modulation process. Recall that a GNN model is parameterized by 𝜃 = {𝜃 (1) , 𝜃 (2) , . . . , 𝜃 (𝐿) } and the outputs of its layers are {𝑍 (1) , 𝑍 (2) , . . . , 𝑍 (𝐿) }. The overall modulation process can be formulated as:EQUATIONwhere 𝜙 𝑚 represents all the involved parameters for modulation. It adapts the GNN model from 𝜃 to θ based on the context information 𝑐 (G) of the objective graph G, which is extracted by a graph context encoder. The encoder can be any model, and we implement it with a GCN followed by a mean pooling operation. Inspired by the gating method [2] , the 𝑙-th layer of the GNN model can be modulated by:EQUATIONwhere MLP is the multilayer perceptron; MEAN denotes the averaging operation; ⊙ denotes the broadcastable element-wise multiplication; 𝜎 is the sigmoid function; 𝑊 𝑙 is a learnable gating parameter with the same shape of 𝜃 (𝑙 ) and I is a matrix of ones. The whole process learns a scalar 𝜅 𝑙 that adaptively shapes the magnitude of the layer weights to a proper level. The sigmoid gating term 𝛾 𝑙 decides whether to exploit the scalar or not. The modulation operation is applied in each layer of the GNN model and is designed to be lightweight to prevent overfitting to the synthetic graphs.The above modulation architecture is co-trained with meta pretraining. Without loss of generality, we adopt the masked edge prediction, a widely applicable pretext task, to extract meta-tasks for GNN pre-training. Specifically, following the task-based metalearning (MAML) setup [7] EQUATIONwhere 𝜆 is the inner learning rate and the labels are the set of support edges S spt 𝑡 ; and θ spt 𝑡 serves as a composite parameter to be integrally updated. Once meta-training is completed, the obtained meta-trained parameter θ spt 𝑡 * is applied to predict the edges of a query graph S qry 𝑡 to minimize the prediction loss, which serves as task-level update signals. Thus, we have the following link prediction loss function L pre for all synthetic graphs:EQUATIONwhere θ spt 𝑡 * involves the GNN parameter 𝜃 , modulation parameter 𝜙 𝑚 , and synthetic {𝜙 S 𝑡 }. They can be optimized during meta pretraining and further used to help fine-tune the downstream task.Throughout the fine-tuning process to transfer the pre-trained metaknowledge into an unseen domain task for target graph G * , three key considerations are identified to mitigate the risk of negative transfer: graph distribution alignment, graph context alignment, and graph task alignment.(1) Graph Distribution Alignment. The feature spaces of an unseen graph may be heterogeneous and unapplicable for the pretrained GNN model. Although some feature reduction methods can be employed to adjust the feature dimensions, they cannot deal with graph distribution mismatches. To overcome this limit, we regard the synthetic graphs from pre-training as anchors and propose a deep neural network (DNN) 𝑔 * with parameters 𝜙 G * to transform the node features so that the downstream graph distribution can be aligned with that of the synthetic graphs. This goal can be achieved by reusing the bi-level distribution matching of Eq. ( 5), treating the downstream graph as the source and the synthetic graphs as the target to optimize transformation parameters of 𝑔 * :L ′ dis = 𝑅 ∑︁ 𝑟 =0 [ 𝑓 𝜔 * (𝑃 𝑟 𝑔 * ( G * ) ) -𝑓 𝜔 * (𝑃 𝑟 { S 𝑡 } ) ],whereEQUATIONThe distribution alignment is conducted during preprocessing and the obtained 𝑔 * is further fine-tuned together with the pre-trained GNN model for the downstream task.(2) Graph Context Alignment. A single pre-trained GNN model can hardly handle graphs with different contexts from diverse domains. To cope with it, we train a modulation architecture 𝑓 𝜙 pre 𝑚 to adapt the pre-trained GNN model to the target graph G * . Thus, the pre-trained parameters 𝜃 pre should be modulated as:EQUATIONAccording to the equation, the alignment can adjust the GNN model's representation capability without changing the model definition and introducing extra inference overhead.(3) Graph Task Alignment. During fine-tuning, different task objectives from the pretext may lead to negative knowledge elicitation, therefore it is necessary to align the downstream graph tasks. Since the proposed multi-domain pre-training is based on the link prediction task, following the method in [32] , we can reshape the node classification task to resemble link prediction. Firstly, each class is regarded as a trainable virtual node, initialized with the mean representation of training nodes labeled with the same class. These virtual nodes and the target graph jointly act as the model inputs. In this way, the node classification task can be viewed as predicting the existence of a link between the unclassified nodes and the virtual class nodes. Secondly, for the reformulated link prediction task, positive examples involve links between the training nodes and the classes they belong to, while negative examples are the rest. Finally, applying the orthogonal constraint on virtual node embeddings, the classifier executes node classification by querying the highest probability of link existence between an unclassified node and every virtual node.We present the overall pre-training and fine-tuning processes. Their pseudo-codes and complexity analysis are shown in Appendix B. where 𝛼, 𝛽 are both hyperparameters to trade off the importance. The outputs of pre-training are the parameters 𝜃 pre ,𝜓 pre of the pre-trained GNN model, 𝜙 pre 𝑚 of the modulation architecture and the synthetic graphs {S 𝑡 }.Target Domain Fine-tuning. On this stage, given the unseen domain graph G * , the pre-trained parameters 𝜃 pre for GNN, the prediction function𝜓 pre for pretext task, the modulation parameters 𝜙 pre 𝑚 and the synthetic graphs {S 𝑡 }, three alignments are processed to alleviate negative transfer.• Match the graph distribution between target graph G * and synthetic graphs {S 𝑡 } during preprocess to get 𝜙 G * [Eq. (10) ]. • Modulate pre-trained GNN model to get θ pre [Eq. (11) ].• Reshape the downstream task in line with the pretext of 𝜓 pre .After these three alignments, the GNN model can be fine-tuned with θ pre ,𝜓 pre , 𝜙 G * on the target unseen domain graph G * . 
evaluation 5.1 experimental settings: 
 Datasets. We evaluate the proposed framework based on four realworld graphs with massive knowledge from different domains. ( 1) Academic (A) [11] is a citation network consisting of papers indexed by MAG [40] ; (2) Product (P) [11] is an Amazon product co-purchasing network. ( 3) Reddit (R) [9] is a comment graph constructed from Reddit. ( 4) Yelp (Y) [53] is a social network formed by users and their friendship of the Yelp website. The detailed information is summarized in Tab. 1. In experiments, we follow the leave-one-domain-out protocol. Specifically, we choose three domains as the multi-source domains for pre-training and the remaining one as the unseen target domain. For example, PRY-A denotes PRY for pre-training and A for fine-tuning. We use the full unlabeled graphs for pre-training and randomly extract ten different graphs of 2000 nodes from the target domain for testing. Baseline Algorithms. We use twelve baselines for comparison, which include the state-of-the-art from three different categories.• Graph supervised learning methods. (1) GCN [19] , (2) GAT [36] ,(3) GIN [48] . They all operate neighborhood aggregation in an end-to-end manner. • Graph self-supervised learning methods. (4) EdgePred [12] and ( 5)AttrMask [14] randomly masked partial edges or node attributes and then trained GNNs to predict them. ( 6) DGI [37] and ( 7) GCC [30] leveraged contrastive learning to capture the latent graph properties. ( 8) GPT-GNN [11] and (9) L2P-GNN [28] exploited intentionally designed strategies of generative model pre-training and learning to pre-training. • Graph prompt learning methods. (10) GPPT [32] , (11) Graph-Prompt, (12) ProG [33] . They bridge the task gap by aligning the downstream objectives with those of the pre-training through designed learnable prompt tokens.Default settings. In this work, we focus on node-level tasks of link prediction and node classification for the fine-tuning stage, with AUC and F1-score as their evaluation indicators. Up to 20% of the known labels or edges are utilized for training and 10% for validation. We exploit a two-layer GCN as the GNN backbone, with the inner product decoder [18] serving as the prediction function for link prediction and an MLP-based logistic regression classifier for node classification. We fine-tune all the models for 500 epochs as the final results, using Adam optimizer with a learning rate of 0.005. Each experiment is conducted five times, and the mean results are reported. It's important to note that graph classification task can also be reshaped to the pretext task. Due to the page limit, we show the corresponding results in Appendix F. Besides, for our proposed LAMP, we use the basic MLP as the transformation structure {𝜙 G 𝑘 }. We also choose to sample 𝑇 subgraphs from the source domains beforehand and use MLPs, namely {𝜙 S 𝑡 }, to recode their node features as the synthetic graph generation. The number of the synthetic graphs is𝑇 = 10 with unified node feature dimension 𝑑 = 128. We set 𝑅 = 1 for distribution matching. The step number of prepositive training for the bi-level processes in Eqs. (3), ( 5) is set to 10 with a learning rate 𝜂 of 0.005. The steps for meta training in Eq. ( 8) is also 10 with a learning rate 𝜆 of 0.01. The hyper-parameters for the loss function are 𝛼 = 𝛽 = 1. For baselines, since the graph prompting methods have their own prediction functions, we made no modifications to them. The baseline algorithms cannot handle graphs with different feature dimensions in pre-training, thus we use the Autocoder [10] to adjust them to a fixed dimension (i.e., 128) with minimum information loss.Training Efficiency. It indicates that the model can be trained quickly in downstream tasks with the pre-trained parameters, which corresponds to the fact that the initialized model achieves good performance with faster convergence and fewer training epochs. We present results for 10% labels or edges known in Fig. 2 .For link prediction, LAMP stands out as visibly superior to the baselines. LAMP represented with the purple line starts at a relatively higher value, indicating a narrower gap between the pretext and the downstream tasks. As training progresses, LAMP shows a faster convergence. This efficiency dues to meta pre-training, which guides the model to learn more efficiently. Notably, prompting methods yield poorer results for two main reasons. Firstly, general pre-training methods cannot capture common patterns from multi-source graphs. Secondly, prompting-tuning methods typically freeze the pre-trained GNN parameters, which is usually ineffective for an unseen domain graph.For node classification, LAMP patterns are a little different. In the beginning, LAMP is close to the other baselines. As training progresses, the superiority of LAMP becomes clear. This is due to the successful knowledge extraction from multi-source graphs and the efforts to minimize the risk of negative transfer. In summary, LAMP provides GNN models with an effective parameter initialization, demonstrating better training efficiency. Final Results. We examine the final performances with 10% and 20% edges or labels known. The results are in Tab. 2.For link prediction, LAMP consistently outperforms the other baselines, except for APY-R with 10% known edges. Our solution generally enhances the capability of GNN models and achieves significant performance improvements up to 30% in PRY-A, 18% in APR-Y, 21% in APY-R, and 32% in ARY-P. Compared with the second-best baselines, the improvement exceeds 2% in the ARY-P setting. This suggests that using pre-trained knowledge from different graph domains can help adapt a GNN model to an unseen domain, even though they have heterogeneous semantic contexts. For node classification, LAMP also consistently outperforms the baselines to showcase improvements up to 45% in PRY-A, 25% in APR-Y, 45% in APY-R, and 43% in ARY-P. Compared with the second-best methods, LAMP has a better performance of about 2% improvement, except for the cases of APR-Y with 10% known labels, where GCC and GCN exhibit slightly higher accuracy. However, when there are 20% known labels in APR-Y, LAMP takes the lead. None of the baselines excels in both tasks. However, our proposed LAMP can not only achieve good efficiency but also work well in various graph learning tasks.We conduct ablation experiments to investigate the effectiveness of the proposed three modules by removing their subcomponents. We compare their performances with the full LAMP framework for all dataset settings with 20% labels or edges known. We present both the 50-epoch results and the final results in Tab. 3, where M1-M3 represent Synthetic Graph Distillation, Modulated Meta Pre-training and Knowledge Transfer for Downstream Tasks respectively.In the table, each module plays a distinct role in multi-source pretraining. For M1, distribution matching emerges as more crucial, as its absence leads to a significant drop in performances, especially in node classification. Therefore, considering distribution matching as walk distributions proves highly meaningful. Knowledge approximation is generally beneficial but not as critical as other components. In some cases, it may have a minor impact on final results, as observed in APY-R for link prediction and ARY-P for node classification. For M2, the importance of meta-knowledge surpasses common knowledge since high-level knowledge can be transferred across datasets. Modulation yields noticeable improvements in model efficiency, particularly in node classification. This is due to its efforts to consider different graph contexts. For M3, we find it necessary to mitigate the risk of negative transfer. Graph distribution alignment is beneficial as a preprocessing operation on the unseen graph to match its distribution with synthetic graphs. It gives the target graph a suitable distribution for the pre-trained GNN model. It also implies diverse knowledge and patterns have been extracted into the synthetic common graphs and can be exploited for pre-training. The context and task alignments are both important since visible declines in LAMP can be observed when removing either. In summary, all modules are necessary and work collaboratively to enhance the overall performance.In this section, we conduct a hyperparameter analysis on the ARY-P setting with 20% know labels or edges. We present the 50-epoch results to illustrate fine-tuning efficiency as well as the final results.Parameters 𝛼, 𝛽, 𝑑,𝑇 , 𝑅. We first analyze five main hyperparameters, as shown in Fig. 3 . The parameters 𝛼, 𝛽 ∈ [0.1, 0.5, 1, 2, 3, 4] respectively adjust the importance of the distribution matching loss L dis and the pre-training loss L pre in the loss function. When the values of 𝛼 and 𝛽 are either too small or too large (i.e., smaller than 0.5 and larger than 3), the model efficiency declines. The parameter 𝑑 ∈ [16, 32, 64, 128, 256, 512] is the dimension of the synthetic graphs, thereby influencing the input dimension of the GNN model. As 𝑑 increases, both model efficiency and final results improve slightly, attributed to the enhanced model representation ability. But when 𝑑 reaches 128, the improvements become less noticeable and may negatively affect the final results in link prediction. The parameter 𝑇 ∈ [2, 4, 6, 8, 10, 12] is the number of synthetic graphs, where a larger 𝑇 implies more graphs used for pre-training. Thus, the increase in 𝑇 leads to improvements in both model efficiency and final results. But when 𝑇 ≥ 10, model efficiency does not show further improvement in both link prediction and node classification. In addition, when 𝑇 is small, the pre-trained model shows low robustness, as indicated by the large error bar. The parameter 𝑅 ∈ [0, 1, 2, 3, 4, 5] is the walks considered in the distribution matching. Visibly, the best result is achieved when 𝑅 = 1 and the other performances are obviously worse. It means that it's practically effective to match node and edge distributions merely.Source Domain Settings. Experiments were conducted with different source domain settings. The results, displayed in Tab. 4, indicate that different source domain settings impact model efficiency more than the final results. The influence on the node classification task is larger than that of link prediction because it faces greater challenges in leveraging link prediction pre-trained knowledge. Moreover, the quality of transferable knowledge from different source domain settings varies. For example, A-P outperforms R-P, Y-P, and RY-P. Though it is acknowledged that some source domains may be capable of transferring the most positive knowledge (i.e., A-P), such fact cannot be known in advance. In general, incorporating more multisource domain graphs (i.e., ARY-P) aids in integrating additional knowledge, resulting in more generalized and robust performances for unseen domains. In essence, LAMP can be considered as a comprehensive method to explore the correlations among different graph domains.  
conclusion: 
 This paper addressed the multi-source GNN pre-training problem with the difficulties of heterogeneity of feature/label space, misalignment of learning tasks, and negative transfer of knowledge. We proposed a novel framework called LAMP to generalize latent knowledge from multi-source domains to pre-train a general GNN model. It introduced a dual-distillation method to generate synthetic graphs for meta pre-training. A modulation architecture was proposed to modulate the pre-trained GNN model based on different graph contexts, with which the downstream task can be aligned to the pre-trained task to achieve multi-domain knowledge transfer. We conducted extensive experiments on four different-scale graph datasets, which showed that LAMP significantly outperforms the state-of-the-art graph pre-training methods on various tasks.  
 
 Appendix:  


Decision: According to all the review comments, rebuttals, discussions and final ratings, the majority of the reviewers gave positive ratings to this paper and the concerns were well addressed. Although reviewer b6eR didn't give positive response after rebuttal, I think most of  his concerns have been addressed. I am happy to recommend to accept this paper. Please carefully revise the final manuscript according to the comments and discussions.

Reviews: review: paper_summary: paper_summary: Summary
This paper aims to the GNN multi-domain pre-training problem. The authors propose a scalable multi-source pre-training method (LAMP). Concretely, at the pre-training process, a graph dual-distillation module is proposed to distill massive knowledge from different graph domains for synthetic homogeneous graphs. At the same time, the high-level meta-knowledge is learned from the synthetic graphs. Then, at the fine-tuning stage, it aligns the target graph distribution, graph context, and graph task with the pretext. Experiments demonstrate the superiority and effectiveness.
strength_weakness: strength_weakness: Strength
- The presentation is excellent and the figure 1 is clear and cool.
- The motivation is clear and the research problem is meaningful and practical. 
- The experiments are extensive and the performance of the proposed method is promising.
limitations: limitations: Weakness
- I consider the used graphs are not from different domains, maybe they are from different sub-domains in one domain. The different domains should contain the very different knowledge, e.g., social network and molecular graph. The used graphs may can be represented as the simple knowledge graphs or the heterogenous graphs.

- Although the performance is promising, the method seems to be more complex than the baselines. The time and space cost experiments are missing. 

- How can the proposed method process some purely unsupervised downstream tasks, such as node clustering?

- The code of this method is unavailable.
paper_summary: paper_summary: The paper presents a novel framework for pre-training Graph Neural Networks (GNNs) across multiple heterogeneous domains and efficiently fine-tuning them on unseen domains. This approach aims to overcome the limitations of conventional GNN pre-training methods that require pre-training and fine-tuning to occur within the same or closely related domains. LAMP consists of three major modules: Synthetic Graph Distillation, Modulated Meta Pre-training, and Knowledge Transfer for Downstream Tasks. The effectiveness of LAMP is demonstrated through extensive experiments on four real-world datasets.
strength_weakness: strength_weakness: - The paper provides a detailed description of the three modules of LAMP, including synthetic graph distillation, modulated meta pre-training, and knowledge transfer mechanisms.
- The experimental evaluation is thorough, using multiple datasets and comparing LAMP with various state-of-the-art methods, demonstrating its superiority in performance and efficiency.
limitations: limitations: - The focus on link prediction and node classification tasks is somewhat narrow. The framework’s generalizability to other GNN tasks, such as graph classification or regression, is not thoroughly explored.
- The framework involves many hyperparameters, and the sensitivity analysis suggests that the performance can vary significantly with different settings, potentially complicating practical adoption.
- While the paper compares LAMP against several baselines, some recent advanced methods in the field of GNN pre-training might be missing from the comparison: Self-supervised Graph-level Representation Learning with Adversarial Contrastive Learning. TKDD'23; Multi-task Self-supervised Graph Neural Networks Enable Stronger Task Generalization. ICLR'23
- As a research direction in machine learning and data mining on graphs, it is necessary to have some up-to-date surveys and related works. A Comprehensive Survey on Graph Neural Networks. 2024; A Comprehensive Survey on Deep Graph Representation Learning. 2024; Graph Neural Networks- Taxonomy, Advances and Trends. 2022; Deep Learning on Graphs: A Survey. 2022
paper_summary: paper_summary: The paper "Scalable Multi-Source Pre-training for Graph Neural Networks" introduces a novel framework called LAMP (scaLAble Multi-source Pre-training) aimed at addressing the limitations of existing Graph Neural Network (GNN) pre-training methods. Traditional approaches typically require that pre-training and fine-tuning occur within the same or closely related domains, which is impractical for graphs from entirely different domains. LAMP is designed to pre-train a GNN model on multiple heterogeneous graph domains and effectively transfer this knowledge to unseen domains with minimal fine-tuning. The framework consists of Synthetic Graph Distillation, 
Modulated Meta Pre-training, and Knowledge Transfer for Downstream Tasks. Extensive experiments demonstrate that LAMP significantly outperforms existing state-of-the-art methods in accuracy and training efficiency across various graph learning tasks.
strength_weakness: strength_weakness: 1. Innovative Framework: The introduction of LAMP addresses a significant gap in GNN pre-training by enabling cross-domain knowledge transfer, which is a substantial advancement over existing methods.

2. Comprehensive Approach: The framework's multi-module design effectively tackles the challenges of domain heterogeneity and negative transfer, offering a robust solution for real-world applications.

3. Strong Experimental Results: Extensive experiments on diverse datasets validate the framework's superiority in performance and efficiency, showcasing its practical utility.
limitations: limitations: 1.  The multi-module architecture of LAMP, while effective, introduces significant complexity, which might pose challenges in implementation and require extensive computational resources. The lack of complexity analysis.

2. I acknowledge that this paper proposes an important problem, addressing a significant gap in GNN pre-training by enabling cross-domain knowledge transfer, however, it seems that if we increase the data scales or model scales, this problem may not be existed. I want to know the advantages of the large language models? these large foundation models may have good generation ability.

3. The paper could benefit from a more detailed analysis of the fine-tuning costs and performance trade-offs, providing deeper insights into the practical deployment of the framework.

Overall, I think this paper is to tackle an important domain problem, the methods, results seems good.
paper_summary: paper_summary: The paper introduces LAMP, a novel method to pre-train Graph Neural Networks (GNNs) on heterogeneous graph domains and efficiently transfer this pre-trained knowledge to unseen domains with minimal fine-tuning. LAMP comprises three main modules: Synthetic Graph Distillation, Modulated Meta Pre-training, and Knowledge Transfer for Downstream Tasks. The proposed method is validated through extensive experiments on four real-world graph domain datasets, demonstrating superior performance in various downstream graph learning tasks.
strength_weakness: strength_weakness: 1. The LAMP framework presents a novel approach to address the multi-domain pre-training problem in GNNs, enhancing generalization across heterogeneous graph domains.
2. The paper thoroughly explains the components of LAMP, including synthetic graph distillation, modulated meta pre-training, and knowledge transfer, adding clarity and depth to the proposed method.
3. The method is validated through experiments on four real-world datasets, demonstrating its effectiveness in improving GNN performance across different tasks.
limitations: limitations: 1. While the combination of multi-source pre-training and knowledge transfer is innovative, the general concept of pre-training and fine-tuning GNNs is not entirely new. The novelty of LAMP might be perceived as incremental due to similarities with existing techniques in other domains [1-3].
2. The paper lacks visualizations that illustrate the multi-source pre-training process and the synthetic graph representations, which could provide more intuitive insights into the effectiveness of LAMP.
3. While the experiments cover several datasets, the method's generalization to other graph-based tasks (e.g., clustering, community detection) is not explored, limiting the broader applicability of the findings.

[1] Gpt-gnn: Generative pre-training of graph neural networks
[2] Pre-training on large-scale heterogeneous graph
[3] Learning to pre-train graph neural networks