Title: Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modeling
Venue: NeurIPS 2023
Accepted: yes
Keywords: graph neural networks
Abstract: Graph neural networks (GNNs) have found extensive applications in learning from graph data. However, real-world graphs often possess diverse structures and comprise nodes and edges of varying types. To bolster the generalization capacity of GNNs, it has become customary to augment training graph structures through techniques like graph augmentations and large-scale pre-training on a wider array of graphs. Balancing this diversity while avoiding increased computational costs and the notorious trainability issues of GNNs is crucial. This study introduces the concept of Mixture-of-Experts (MoE) to GNNs, with the aim of augmenting their capacity to adapt to a diverse range of training graph structures, without incurring explosive computational overhead. The proposed Graph Mixture of Experts (GMoE) model empowers individual nodes in the graph to dynamically and adaptively select more general information aggregation experts. These experts are trained to capture distinct subgroups of graph structures and to incorporate information with varying hop sizes, where those with larger hop sizes specialize in gathering information over longer distances. The effectiveness of GMoE is validated through a series of experiments on a diverse set of tasks, including graph, node, and link prediction, using the OGB benchmark. Notably, it enhances ROC-AUC by $1.81\%$ in ogbg-molhiv and by $1.40\%$ in ogbg-molbbbp, when compared to the non-MoE baselines. Our code is publicly available at https://github.com/VITA-Group/Graph-Mixture-of-Experts.

Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modeling

Abstract

Graph neural networks (GNNs) have found extensive applications in learning from graph data. However, real-world graphs often possess diverse structures and comprise nodes and edges of varying types. To bolster the generalization capacity of GNNs, it has become customary to augment training graph structures through techniques like graph augmentations and large-scale pre-training on a wider array of graphs. Balancing this diversity while avoiding increased computational costs and the notorious trainability issues of GNNs is crucial. This study introduces the concept of Mixture-of-Experts (MoE) to GNNs, with the aim of augmenting their capacity to adapt to a diverse range of training graph structures, without incurring explosive computational overhead. The proposed Graph Mixture of Experts ( GMoE ) model empowers individual nodes in the graph to dynamically and adaptively select more general information aggregation experts . These experts are trained to capture distinct subgroups of graph structures and to incorporate information with varying hop sizes, where those with larger hop sizes specialize in gathering information over longer distances. The effectiveness of GMoE is validated through a series of experiments on a diverse set of tasks, including graph, node, and link prediction, using the OGB benchmark. Notably, it enhances ROC-AUC by 1 . 81% in ogbg-molhiv and by 1 . 40% in ogbg-molbbbp, when compared to the non-MoE baselines. Our code is publicly available at https: //github.com/VITA-Group/Graph-Mixture-of-Experts .

1 Introduction

Graph learning has found extensive use in various real-world applications, including recommendation systems [1], traffic prediction [2], and molecular property prediction [3]. Real-world graph data typically exhibit diverse graph structures and heterogeneous nodes and edges. In graph-based recommendation systems, for instance, a node can represent a product or a customer, while an edge can indicate different interactions such as view, like, or purchase. Similarly in biochemistry tasks, datasets can comprise molecules with various biochemistry properties and thherefore various graph structures. Moreover, purposefully increasing the diversity of graph data structures in training sets has become a crucial aspect of GNN training. Techniques such as graph data augmentations [4, 5] and large-scale pre-training on diverse graphs [6, 7, 8, 9, 10] have been widely adopted to allow GNNs for extracting more robust and generalizable features. Meanwhile, many real-world GNN applications, such as recommendation systems and molecule virtual screening, usually involve processing a vast number of candidate samples and therefore demand computational efficiency. That invites the key

question: Can one effectively scale a GNN model's capacity to leverage larger-scale, more diverse graph data, without compromising its inference efficiency ?

A common limitation of many GNN architectures is that they are essentially 'homogeneous' across the whole graph, i.e., forcing all nodes to share the same aggregation mechanism, regardless of the differences in their node features or neighborhood information. 2 That might be suboptimal when training on diverse graph structures, e.g, when some nodes may require information aggregated over longer ranges while others prefer shorter-range local information. Our solution is the proposal of a novel GNN architecture dubbed Graph Mixture of Experts ( GMoE ). It comprises multiple 'experts' at each layer, with each expert being an independent message-passing function with its own trainable parameters. The idea establishes a new base to address the diversity challenges residing in graph data.

Throughout the training process, GMoE is designed to intelligently select aggregation experts tailored to each node. Consequently, nodes with similar neighborhood information are guided towards the same aggregation experts. This fosters specialization within each GMoE expert, focusing on specific subsets of training samples with akin neighborhood patterns, regardless of range or aggregation levels. In order to harness the full spectrum of diversity, GMoE also incorporates aggregation experts with distinct inductive biases. For example, each GMoE layer is equipped with aggregation experts of varying hop sizes. Those with larger hop sizes cater to nodes requiring information from more extended ranges, while the opposite holds true for those with smaller hop sizes.

We have rigorously validated GMoE's effectiveness through a range of comprehensive molecular property prediction tasks, underscoring our commitment to deliberate diversity modeling. Moreover, our analysis demonstrates that GMoE surpasses other GNN models in terms of inference efficiency , even when they possess similar-sized parameters, thanks to the dynamic expert selection. This efficiency proves crucial in real-world scenarios, such as virtual screening in libraries of trillion-scale magnitude or beyond. The potency of our approach is corroborated by extensive experiments on ten graph learning datasets within the OGB benchmark. For instance, GMoE enhances the ROC-AUC by 1 . 81% on ogbg-molhiv, 1 . 40% on ogbg-molbbbp, 0 . 95% on ogbn-proteins, and boosts Hits@20 score by 0 . 89% on ogbl-ddi, when compared to the single-expert baseline. To gain deeper insights into our method, we conduct additional ablation studies and comprehensive analyses.

2 Related Work

Graph Neural Networks Graph neural networks (GNNs) [11, 12, 13] have emerged as a powerful approach for learning graph representations. Variants of GNNs have been proposed [11, 12, 13], achieving state-of-the-art performance in different graph tasks. Under the message passing framework [14], graph convolutional network (GCN) adopts mean pooling to aggregate the neighborhood and updates embeddings recursively [11]; GraphSAGE [15] adopts sampling and aggregation schemes to eliminate the inductive bias in degree; and graph attention network (GAT) utilizes the learnable attention weights to adaptively aggregate. To capture long-range dependencies in disassortative graphs, Geom-GCN devises a geometric aggregation scheme [16] to enhance the convolution, benefiting from a continuous space underlying the graph. Lately, Graphormer [17] proposes a novel graph transformer model that utilizes attention mechanisms to capture the structural information.

Mixture of Experts The concept of Mixture of Experts (MoE) [18] has a long history, tracing its origins back to earlier work s[19, 20, 21]. Recently, spurred by advancements in large language models, sparse MoE [22, 23, 24, 25, 26] has re-gained prominence. This variant selectively activates only a small subset of experts for each input, significantly enhancing efficiency and enabling the development of colossal models with trillions of parameters. This breakthrough has revolutionized the learning process, particularly on vast language datasets [25, 27]. Subsequent studies have further refined the stability and efficiency of sparse MoE [28, 29]. The remarkable success of sparse MoE in the realm of language has spurred its adoption in diverse domains, including vision [30, 31], multi-modal [32], and multi-task learning [33, 34, 35].

MoE for GNNs In the domain of graph analysis, prior research has assembled knowledge from multiple ranges by combining various GNNs with different scopes [36, 37], akin to a fixed-weight

Expert 1

Expert 2

Expert 3

Graph Mixture of Experts

(hop-1)

(hop-1)

(hop-2)

(Mix hop-1&hop-2)

Mixture of Experts (MoE). Pioneering efforts [38, 39] have also investigated the application of MoE to address the well-known issue of imbalance and to develop unbiased classification or generalization algorithms. However, none of these approaches harnessed the potential advantages of sparsity and adaptivity. Recent work [40] introduced the use of a mixture of experts for molecule property prediction. They employed a GNN as a feature extractor and applied a mixture of experts, where each expert is a linear classifier, on top of the extracted features for graph classification. In contrast, each layer of GMoE constitutes a mixture of experts, with each expert being a GCN/GIN layer featuring different aggregation step sizes. Another distinction from [40] lies in their utilization of domain-specific knowledge (specifically, molecule topology) for expert routing, whereas our approach is designed to operate on general graphs without relying on domain-specific assumptions. One more concurrent study by [41] employs MoE to achieve fairness in predictions for GNNs.

Our study takes a significant stride forward by introducing sparse MoE to scale graph neural networks in an end-to-end fashion, enabling efficient learning on datasets featuring diverse graph structures. We incorporate experts with varying scopes, allowing the gating function to dynamically select neighbors with the desired range.

Adapting Deep Architectures for Training Data Diversity Several prior studies have delved into enhancing the capacity of generic deep neural networks to effectively leverage a wide array of training samples without incurring additional inference costs. For instance, [42] suggested employing two distinct batch normalization (BN) layers for randomly and adversarially augmented training samples, based on the observation that these two sets of augmented samples originate from different distributions. Building upon this concept, [43] extended it by introducing an auxiliary instance normalization layer to further reduce the heterogeneity of input features before reaching the BN layers. More recently, [44] demonstrated that normalizer-free convolutional networks (CNNs) [45, 46] exhibit significantly greater capability in accommodating diverse training sets compared to conventional BN-based CNNs. However, these prior works have primarily concentrated on devising improved normalization strategies for CNNs, while Graph Neural Networks (GNNs) often do not rely on (batch) normalization as heavily.

3 Method

Preliminaries: Graph Neural Networks Taking the classical Graph Convolutional Network (GCN) [11] as an example, the propagation mechanism can be formulated as

where W ( i ) ∈ R s × s is a trainable weight and σ is an element-wise non-linear activation function. h j ∈ R b × s denotes the input feature of j th node while h ′ i ∈ R b × s is its output feature in i th node. b and s are batch size and hidden feature size, respectively. N i denotes the collection of neighbors for i th node including self-connection. The output feature is normalized by 1 √ | N i || N j | . A canonical GCN layer only aggregates the information from immediately adjacent neighbors (hop-1).

3.1 Graph Mixture of Experts

The general framework of GMoE is outlined in Figure 1. The GMoE layer comprises multiple experts, each utilizing either the hop-1 or hop-2 aggregation function. To determine which experts to use for a given node, a gating function is employed. This allows for similar nodes to be assigned to the same experts when learning with diverse graph structures, thereby enabling each expert to specialize in a particular structure type. By doing so, the model can more effectively capture diverse graph structures present within the training set. The GMoE layer's adaptive selection between the hop-1 and hop-2 experts enables the model to dynamically capture short-range or long-range information aggregation for each node. Formally, a GMoE layer can be written as:

where m and n denote the hop-1 and total experts number, respectively. Hench the number of hop-2 experts is n -m . E o and e ij denote the message function and edge feature between i th and j th nodes, respectively. It can represent multiple types of message-passing functions such as one employed by GCN [47] or GIN [13]. G is the gating function that generates multiple decision scores with the input of h i while G ( h i ) o denotes the o th item in the output vector of G . We employ the noisy topk gating design for G following [22], which can be formalized with

where k denotes the number of selected experts. ϵ ∈ N (0 , 1) denotes standard Gaussian noise. W g ∈ R s × n and W n ∈ R s × n are learnable weights that control clean and noisy scores, respectively.

The proposed GMoE layer can be applied to many GNN backbones such as GIN [13] or GCN [11]. In practice, we replace every layer of the backbone with its corresponding GMoE layer. For simplicity, we name the resultant network GMoE-GCN or GMoE-GIN (for GCN and GIN, respectively).

Additional Loss Functions to Mitigate GMoE Collapse Nonetheless, if this model is trained solely using the expectation-maximization loss, it may succumb to a trivial solution wherein only a single group of experts is consistently selected. This arises due to the self-reinforcing nature of the imbalance: the chosen experts can proliferate at a much faster rate than others, leading to their increased frequency of selection. To mitigate this, we implement two additional loss functions to prevent such collapse [22]. The first one is importance loss:

where the importance score Importance( H ) is defined as the sum of each node's gate value g across the whole batch. CV represents the coefficient of variation. The importance loss L importance ( H ) hence measures the variation of importance scores, enforcing all experts to be 'similarly important".

̸

̸

While the importance score enforces equal scoring among the experts, there may still be disparities in the load assigned to different experts. For instance, one expert could receive a few high scores, while another might be selected by many more nodes ye t all with lower scores. This situation can potentially lead to memory or efficiency issues, particularly on distributed hardware setups. To address this, we introduce an additional load-balanced loss to encourage a more even selection probability per expert. Specifically, G ( h i ) = 0 if and only if Q ( h i ) o is greater than the k -th largest element of Q ( h i ) excluding itself. Consequently, the probability of G ( h i ) = 0 can be formulated as:

where k th_ ex () denotes the k -th largest element excluding itself. P ( h i , o ) can be simplified as

where Φ is the CDF of standard normal distribution. The load is then defined as ( p is the node-wise probability in the batch):

The final loss employs both the task-specific loss and two load-balance losses, leading to the overall optimization target ( λ is a hand-tuned scaling factor):

where L EM denotes the task-specific MoE expectation-maximizing loss.

Pre-training GMoE We further discover that GMoE could be combined with and strengthened by the self-supeervised graph pre-training techniques. We employ GraphMAE [10] as the self-supervised pre-training technique, defined as

where f and d denote the encoder and the decoder networks, and M represents the mask for the input graph H . f and d collaborative conduct the reconstruction task from the corrupted input, whose quality is measured by the distance metric D . We later will experimentally demonstrate and compare GMoE performance with and without pre-training.

3.2 Computational Complexity Analysis

We show that GMoE-GNN brings negligible overhead on the inference cost compared with its GNN counterpart. We measure computational cost. using the number of floating point operations (FLOPs).

The computation cost of a GMoE layer can be defined as

where F maps functions to its flops number. C GMoE denotes the computation cost of the whole layer in GMoE-GCN. Given there exists an efficient algorithm that can solve hop-1 and hop-2 functions with matching computational complexity [36], we can further simplify C GMoE as

Given ∑ n o =1 1 ( G ( h i ) o ) = k , C GMoE can be further simplified to

Here, C is the computation cost of a single message passing in GMoE-GCN. Denote the computation cost of a single GCN message passing as C 0 , and the total computational cost in the whole layer as C GCN . By setting C = C 0 k in GMoE-GCN, we have C GMoE = ∑ h i ∈ H ∑ j ∈ N i C 0 = C GCN .

In traditional GMoE-GCN or GMoE-GIN, the adjustment of C can be easily realized by controlling the hidden feature dimension size s . For instance, GMoE-GCN and GMoE-GIN with hidden dimension size of s = s 0 √ k can have similar FLOPs with its corresponding GCN and GIN with dimension size of s 0 . The computation cost of gating functions in GMoE is meanwhile negligible compared to the cost of selected experts, since both W g ∈ R n × s and W n ∈ R n × s is in a much smaller dimension than W ( i ) ∈ R s × s given n ≪ s .

In practice, on our NVIDIA A6000 GPU, the inference times for 10 , 000 samples are 30 . 2 ± 10 . 6 ms for GCN-MoE and 36 . 3 ± 17 . 2 ms for GCN. The small variances in GPU clock times align with their nearly identical theoretical FLOPs.

4 Experimental Results

In this section, we first describe the detailed settings in Section 4.1. We then show our main results on graph learning in Section 4.2. Ablation studies and analysis are provided in Section 4.3.

4.1 Experimental Settings

Datasets and Evaluation Metrics We conduct experiments on ten graph datasets in the OGB benchmark [48], including graph-level (i.e., ogbg-bbbp, ogbg-hiv, ogbg-moltoxcast, ogbg-moltox21, ogbg-molesol, and ogbg-freesolv), node-level (i.e., ogbn-protein, ogbn-arxiv), and link-level prediction (i.e., ogbl-ddi, ogbl-ppa) tasks. Following [48], we use ROC-AUC (i.e., area under the receiver operating characteristic curve) as the evaluation metric on ogbg-bbbp, ogbg-hiv, ogbg-moltoxcast, ogbg-moltox21, and ogbn-protein; RMSE (i.e., root mean squared error) on ogbg-molesol, and ogbg-freesolv; classification accuracy (Acc) on ogbn-arxiv; Hits@100 score on ogbl-ppa; Hits@20 score on ogbl-ddi.

Model Architectures and Training Details We use the GCN [11] and GIN [13] provided by OGB benchmark [48] as the baseline models. All model settings (e.g., number of layers, hidden feature dimensions, etc.) and training hyper-parameters (e.g., learning rates, training epochs, batch size, etc.)

are identical as those in [48]. We show the performance gains brought by their GMoE counterparts: GMoE-GCN and GMoE-GIN. For GMoE models, as described in Section 3, we select k experts out of a total of n experts for each node, where m out of n experts are hop-1 aggregation functions and the rest n -m are hop-2 aggregation functions. All three hyper-parameters n, m, k , together with the loss trade-off weight λ in Eq. (9), are tuned by grid searching: n ∈ { 4 , 8 } , m ∈ { 0 , n/ 2 , n } , k ∈ { 1 , 2 , 4 } , and λ ∈ { 0 . 1 , 1 } . The hidden feature dimension would be adjusted with k (Section 3.2) to ensure the same flops for all comparisons. The hyper-parameter values achieving the best performance on validation sets are selected to report results on test sets, following the routine in [48]. All other training hyper-parameters on GMoE (e.g., batch size, learning rate, training epochs) are kept the same as those used on the single-expert baselines. All experiments are run for ten times with different random seeds, and we report the mean and deviation of the results following [48].

Pre-training Settings Following the transfer learning setting of [49, 6, 8, 10], we pre-train the models on a subset of the ZINC15 dataset [50] containing 2 million unlabeled molecule graphs. For training hyperparameters, we employ a batch size of 1024 to accelerate the training on the large pre-train dataset for both baselines and the proposed method. We follow [10] employing GIN [13] as the backbone, 0.001 as the learning rate, adam as the optimizer, 0 as the weight decay, 100 as the training epochs number, and 0.25 as the masking ratio.

4.2 Main Results

Our evaluation primarily centers around comparing GMoE-GCN with the baseline single-expert GCN using six graph property prediction datasets. These datasets encompass four graph classification tasks and two regression tasks within a supervised learning framework. In all cases, models are trained from scratch, utilizing only the labeled samples in the training set. The classification and regression results are outlined in Table 1 and 2, respectively.

It is worth noting that GMoE-GCN consistently outperforms the baseline across all six datasets. Notably, there are substantial improvements in ROC-AUC, with increases of 1 . 81% on the molhiv dataset and 1 . 41% on the molbbp dataset. While these enhancements may seem modest, they represent significant progress. Additionally, lifts of 1 . 40% on ogbg-moltox21, 1 . 43% on ogbg-molbbbq, and 1 . 18% on ogbg-moltoxcast have been observed. The uniformity of these improvements across diverse tasks and datasets underscores the reliability and effectiveness of GMoE-GCN.

Leveraging large-scale pretraining on auxiliary unlabeled data has notably enhanced the generalization capabilities of GNNs even further. Expanding on this, our GMoE consistently improves performance when integrated with large-scale pretraining methods. Following the methodology outlined in [10], we employ GIN [13] as the baseline model for comparison with GMoE-GIN. As illustrated in Table 3, GMoE-GIN outperforms GIN on 3 out of 4 datasets, enhancing the average performance by 0 . 54% , even without pretraining. This underscores the versatility of GMoE in enhancing various model architectures. When coupled with the pretraining, GMoE-GIN further widens the performance gap with the GIN baseline, achieving a slightly more pronounced improvement margin of 0 . 6%

Additionally, GMoE showcases its potential in node and link prediction tasks. Experiments conducted on ogbn-protein and ogbn-arxiv for node prediction, as well as ogbl-ddi and ogbl-ppa for link prediction, further validate this observation. The results, detailed in Table 4 and Table 5, demonstrate GMoE-GCN's superiority over the single-expert GCN. Notably, it enhances performance metrics like ROC-AUC by 0 . 95% and Hits@20 by 0 . 89% on the ogbn-protein and ogbl-ddi datasets.

4.3 Ablation Study and Analysis

Observation 1: Larger graphs prefer larger hop sizes We conducted an ablation study on two prominent molecule datasets, namely ogbg-molhiv and ogbg-molfreesolv, which exhibit the largest and smallest average graph sizes, respectively, among all molecular datasets in the OGB benchmark. As illustrated in the third column of Table 6, the average size of molecular graphs in ogbg-molhiv is approximately three times greater than that of the ogbg-molfreesolv dataset. Remarkably, on ogbgmolfreesolv, if all experts utilize hop-2 aggregation functions (i.e., when m = 0 ), the performance substantially lags behind the scenario where all experts employ hop-1 aggregations (i.e., when m = n ). In contrast, on ogbg-molhiv, characterized by significantly larger graphs, leveraging all hop-2 experts (i.e., when m = 0 ) yields superior performance compared to employing all hop-1 experts (i.e., when m = n ). This observation suggests that larger graphs exhibit a preference for aggregation experts with greater hop sizes. This alignment with intuition stems from the notion

that larger molecular graphs may necessitate more extensive long-range aggregation information in contrast to their smaller counterparts.

Observation 2: Sparse expert selection helps not only efficiency, but also generalization As depicted in Table 6, the optimal performance on both datasets is attained with sparse MoEs (i.e., when k < n ), as opposed to a full dense model (i.e., when k = n ). This seemingly counter-intuitive finding shows another benefit of sparsity besides significantly reducing inference computational cost: that is, sparsity is also promising to improve model generalization , particularly when dealing with multi-domain data, as it allows a group of experts to learn collaboratively and generalize to unseen domains compositionally. Our finding also echoes prior work, e.g., [51], which empirically shows sparse MoEs are strong domain generalizable learners.

Observation 3: GMoE demonstrates performance gains, though converging over more epochs We compared the convergence curves of GMoE-GCN and the single-expert GCN, illustrated in Figure 2. Specifically, we plotted the validation and test ROC-AUC at different epochs while training on the protein dataset. As observed, the performance of GMoE-GCN reaches a plateau later than that of GCN. However, GMoE-GCN eventually achieves a substantial performance improvement after a sufficient number of training epochs.

We provide an explanation for this phenomenon below. In the initial stages of training, each expert in GMoE has been updated for k times fewer iterations compared to the single expert GNN. Consequently, all experts in GMoE are relatively weak, leading to GMoE's inferior performance compared to GCN during this early phase. However, after GMoE has undergone enough epochs of training, all experts have received ample updates, enabling them to effectively model their respective subgroups of data. As a result, the performance of GMoE surpasses that of the single-expert GCN.

Observation 4: Load balancing is essential

We conducted an investigation into the scaling factor for the load balancing loss, denoted by λ . As shown in Table 7, utilizing λ = 0 . 1 resulted in significantly improved performance, exceeding 2 . 58% in terms of ROC-AUC compared to when λ = 0 . This underscores the crucial role

75

70

65

(%)

60

DC-AUC

55

RO

50

45

40

GMoE-GCN

GCN

35

0

250

500

75010001250 1500 17502000

Epochs

of implementing load-balancing losses. Conversely, the choice of λ exhibits less sensitivity; opting for λ = 1 would yield a similar performance of 75 . 27% .

Observation 5: GMoE improves other state-of-the-art GNN methods Finally, we explore whether our proposed GMoE can yield an improvement in the performance of other state-of-theart models listed on the OGB leaderboard. We selected Neural FingerPrints [52] as our baseline. As of the time of this submission, Neural FingerPrints holds the fifth position in the ogbg-molhiv benchmark. It is noteworthy that this methodology has gained significant acclaim, as it serves as the foundation for the top four current methods [53, 54, 55, 56].

After integrating the GMoE framework into the Neural FingerPrints architecture, we achieved an accuracy of 82.72% ± 0.53%, while maintaining the same computational resource requirements. This performance outperforms Neural FingerPrints by a margin of 0.4%, underlining the broad and consistent adaptability of the GMoE framework.

5 Conclusion

In this work, we propose the Graph Mixture of Experts (GMoE) model, aiming at addressing the challenges posed by diverse graph structures. By incorporating multiple experts at each layer, each equipped with its own trainable parameters, GMoE introduces a novel approach to modeling graph data. Through intelligent expert selection during training, GMoE ensures nodes with similar neighborhood information are directed towards the same aggregation experts, promoting specialization within each expert for specific subsets of training samples. Additionally, the inclusion of aggregation experts with distinct inductive biases further enhances GMoE's adaptability to different graph structures. Our extensive experimentation and analysis demonstrate GMoE's notable accuracy-efficiency trade-off improvements over baselines. These advancements hold great promise for real-world applications, particularly in scenarios requiring the efficient processing of vast amounts of candidate samples.

Limitations: As an empirical solution to enhance the capability of GNNs to encode diverse graph data, we primarily assess the effectiveness of the proposed method using empirical experimental results. It is important to note that our initial comparison primarily focused on fundamental backbones such as GCN/GIN to elucidate our concept. Recognizing the crucial importance of a comprehensive evaluation, we are actively broadening our comparative scope to include state-of-the-art models derived from GCN and other MoE models. Future iterations will thus incorporate a more exhaustive evaluation. Moreover, given the prevalence of GCN in semi-supervised learning, there is potential in exploring the benefits of our proposed approach in such tasks - an avenue we are eager to explore further. Another open question stemming from this work is whether we can apply GMoE on GNNs with heterogeneous aggregation mechanisms, such as GAT and GraphTransformer, which may potentially further improve the performance on diverse graph data.

References

Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. Graph neural networks in recommender systems: A survey. ACM Computing Surveys , 55(5):1-37, 2022.

Weiwei Jiang and Jiayun Luo. Graph neural network for traffic forecasting: A survey. Expert Systems with Applications , page 117921, 2022.

Oliver Wieder, Stefan Kohlbacher, Mélaine Kuenemann, Arthur Garon, Pierre Ducrot, Thomas Seidel, and Thierry Langer. A compact review of molecular property prediction with graph neural networks. Drug Discovery Today: Technologies , 37:1-12, 2020.

Xiaotian Han, Zhimeng Jiang, Ninghao Liu, and Xia Hu. G-Mixup: Graph data augmentation for graph classification. arXiv preprint arXiv:2202.07179 , 2022.

Songtao Liu, Rex Ying, Hanze Dong, Lanqing Li, Tingyang Xu, Yu Rong, Peilin Zhao, Junzhou Huang, and Dinghao Wu. Local augmentation for graph neural networks. In ICML , 2022.

Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems , 33:5812-5823, 2020.

Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. When does self-supervision help graph convolutional networks? In International Conference on Machine Learning , pages 10871-10880, 2020.

Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In International Conference on Machine Learning , pages 12121-12132, 2021.

Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. Bringing your own view: Graph contrastive learning without prefabricated data augmentations. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining , pages 1300-1309, 2022.

Zhenyu Hou, Xiao Liu, Yuxiao Dong, Chunjie Wang, Jie Tang, et al. GraphMAE: Selfsupervised masked graph autoencoders. arXiv preprint arXiv:2205.10803 , 2022.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 , 2017.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 , 2018.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In ICML , pages 1263-1272, 2017.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in neural information processing systems , 2017.

Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-GCN: Geometric graph convolutional networks. arXiv preprint arXiv:2002.05287 , 2020.

Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation? In NeurIPS , 2021.

Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation , 3(1):79-87, 1991.

Michael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation , 6(2):181-214, 1994.

Ke Chen, Lei Xu, and Huisheng Chi. Improved learning algorithms for mixture of experts in multiclass classification. Neural networks , 12(9):1229-1252, 1999.

Seniha Esen Yuksel, Joseph N Wilson, and Paul D Gader. Twenty years of mixture of experts. IEEE Transactions on Neural Networks and Learning Systems , 23(8):1177-1193, 2012.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538 , 2017.

Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668 , 2020.

William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.

Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In ICML , pages 4057-4086, 2022.

Stephen Roller, Sainbayar Sukhbaatar, Jason Weston, et al. Hash layers for large sparse models. In Advances in Neural Information Processing Systems , pages 17555-17566, 2021.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022.

Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In International Conference on Machine Learning , pages 6265-6274, 2021.

Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. Designing effective sparse expert models. arXiv preprint arXiv:2202.08906 , 2022.

Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. In Advances in Neural Information Processing Systems , volume 34, pages 8583-8595, 2021.

Xin Wang, Fisher Yu, Lisa Dunlap, Yi-An Ma, Ruth Wang, Azalia Mirhoseini, Trevor Darrell, and Joseph E Gonzalez. Deep mixture of experts via shallow embedding. In Uncertainty in Artificial Intelligence , pages 552-562, 2020.

Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Multimodal contrastive learning with limoe: the language-image mixture of experts. arXiv preprint arXiv:2206.02770 , 2022.

Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 1930-1939, 2018.

Jinguo Zhu, Xizhou Zhu, Wenhai Wang, Xiaohua Wang, Hongsheng Li, Xiaogang Wang, and Jifeng Dai. Uni-Perceiver-MoE: Learning sparse generalist models with conditional moes. arXiv preprint arXiv:2206.04674 , 2022.

Hanxue Liang, Zhiwen Fan, Rishov Sarkar, Ziyu Jiang, Tianlong Chen, Kai Zou, Yu Cheng, Cong Hao, Zhangyang Wang, et al. M 3 vit: Mixture-of-experts vision transformer for efficient multi-task learning with model-accelerator co-design. In Advances in Neural Information Processing Systems , 2022.

Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. MixHop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In ICML , pages 21-29, 2019.

Sami Abu-El-Haija, Amol Kapoor, Bryan Perozzi, and Joonseok Lee. N-GCN: Multi-scale graph convolution for semi-supervised node classification. In Uncertainty in Artificial Intelligence , pages 841-851, 2020.

Fenyu Hu, W Liping, L Qiang, Shu Wu, Liang Wang, and Tieniu Tan. Graphdive: graph classifcation by mixture of diverse experts. In IJCAI , 2022.

Liguang Zhou, Yuhongze Zhou, Tin Lun Lam, and Yangsheng Xu. CAME: Context-aware mixture-of-experts for unbiased scene graph generation. arXiv preprint arXiv:2208.07109 , 2022.

Suyeon Kim, Dongha Lee, SeongKu Kang, Seonghyeon Lee, and Hwanjo Yu. Learning topology-specific experts for molecular property prediction. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 8291-8299, 2023.

Zheyuan Liu, Chunhui Zhang, Yijun Tian, Erchi Zhang, Chao Huang, Yanfang Ye, and Chuxu Zhang. Fair graph representation learning via diverse mixture of experts. In The Web Conference , 2023.

Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan L Yuille, and Quoc V Le. Adversarial examples improve image recognition. In IEEE Conference on Computer Vision and Pattern Recognition , pages 819-828, 2020.

Haotao Wang, Chaowei Xiao, Jean Kossaifi, Zhiding Yu, Anima Anandkumar, and Zhangyang Wang. AugMax: Adversarial composition of random augmentations for robust training. In Advances in Neural Information Processing Systems , volume 34, pages 237-250, 2021.

Haotao Wang, Aston Zhang, Shuai Zheng, Xingjian Shi, Mu Li, and Zhangyang Wang. Removing batch normalization boosts adversarial training. In International Conference on Machine Learning , pages 23433-23445, 2022.

Andrew Brock, Soham De, and Samuel L Smith. Characterizing signal propagation to close the performance gap in unnormalized resnets. arXiv preprint arXiv:2101.08692 , 2021.

Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. In ICML , pages 1059-1071, 2021.

Guohao Li, Chenxin Xiong, Ali Thabet, and Bernard Ghanem. DeeperGCN: All you need to train deeper GCNs. arXiv preprint arXiv:2006.07739 , 2020.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00687 , 2020.

Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. arXiv preprint arXiv:1905.12265 , 2019.

Teague Sterling and John J Irwin. Zinc 15-ligand discovery for everyone. Journal of Chemical Information and Modeling , 55(11):2324-2337, 2015.

Bo Li, Yifei Shen, Jingkang Yang, Yezhen Wang, Jiawei Ren, Tong Che, Jun Zhang, and Ziwei Liu. Sparse mixture-of-experts are domain generalizable learners. arXiv preprint arXiv:2206.04046 , 2022.

Weibin Li, Shanzhuo Zhang, Lihang Liu, Zhengjie Huang, Jieqiong Lei, Xiaomin Fang, Shikun Feng, and Fan Wang. Molecule representation learning by leveraging chemical information. Technical report, 2021.

Lanning Wei, Huan Zhao, Quanming Yao, and Zhiqiang He. Pooling architecture search for graph classification. In Proceedings of the 30th ACM International Conference on Information &Knowledge Management , pages 2091-2100, 2021.

Yan Wang, Hao Zhang, Jing Yang, Ruixin Zhang, and Shouhong Ding. Technical report for ogb graph property prediction. In Technical Report , 2021.

Zhuoning Yuan, Yan Yan, Milan Sonka, and Tianbao Yang. Large-scale robust deep auc maximization: A new surrogate loss and empirical studies on medical image classification. In CVPR , pages 3040-3049, 2021.

Hao Zhang, Jiaxin Gu, and Pengcheng Shen. Gman and bag of tricks for graph classification, 2021.

Peer reviews:
comments: SUMMARY:
This paper considers the problem of graph neural networks. After discussing the drawback of existing methods, this paper presents a new model, called  Graph Mixture of Expert (GMoE), which can dynamically select its own optimal information aggregation experts. Extensive experiments conducted on several graph datasets show the benefit of the proposed method.

STRENGTHS:
+ The authors have carefully reviewed the drawback of existing graph models and then propose a novel and effective model, called Graph Mixture of Expert (GMoE). The GMoE enables the model to dynamically select its own optimal information aggregation experts.

+ The paper is well-written and easy to follow. The reviewer can easily understand the motivation of this paper and the details of the proposed method.

+ Experiments have shown the benefit of the proposed method over the traditional GCN.

WEAKNESSES:
- Although the authors have shown the benefit of the proposed method over GCN, the improvement seems somewhat limited. For example, in several tables, the improvement over GCN is only 0.*%.

- It seems that the comparison to other methods is limited. The authors have only compared to the traditional GCN. However, the comparison to other state-of-the-art models modified based on GCN is missed. In addition, it is very important to compare with other MoE models.

- As GCN is also popular in semi-supervised learning. I wonder if the proposed can also benefit the semi-supervised learning task.

- The authors stated that the proposed method is efficient. However, the computation cost about training time and testing time is not provided.

QUESTIONS:
Please see the weaknesses.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Graph Mixture of Expert (GMoE) model, enhancing Graph Neural Networks' (GNNs) ability to handle diverse graph structures without additional computational costs. GMoE allows nodes to select their information aggregation experts dynamically, catering to different subgroups of graph structures. Additionally, it incorporates experts with varying aggregation hop sizes for broader information capture. Its effectiveness is validated through various tasks in the OGB benchmark, with improved results compared to non-MoE baselines.\n\nSTRENGTHS:\nThis article uses MoE in GNN, which is a relatively new idea.\n\nWEAKNESSES:\n1. After reading the entire article, the motivation behind it remains unclear. What issue pertaining to GNN does the article aim to address? It would be beneficial if the author could include an \"preliminary\" section that clearly delineates the problem this article seeks to solve.\n\n2. Certain assertions in the article are questionable. For instance, the statement: “forcing all nodes to share the same aggregation mechanism” does not align with the functioning of GAT and GraphTransformer, where each node's aggregation is adaptively carried out via the attention mechanism.\n\n3. The article doesn't clarify how the model's improved generalization ability is assured. How can the model learn a more diverse graph structure based on observation data from a single domain? This is an important point that requires further clarification.\n\n4. The algorithm's design seems more akin to an engineering solution or a skill rather than a scientifically robust methodology. The article lacks theoretical discussion explaining why the algorithm is designed in its particular way. This absence of theoretical framework underpinning the algorithm design makes it difficult to fully appreciate or understand the design choices.\n\n5. There are also many articles [1-3] that use MoE to GNN, and the author does not add these works to the discussion or compare them.\n\n6. According to the guidelines provided by the NIPS official website, authors are required to include sections on 'Limitations' and 'Broader Impacts'. However, these essential components appear to be missing in this article.\n\n[1] Explore Mixture of Experts in Graph Neural Networks\n\n[2] Graph Classification by Mixture of Diverse Experts, IJCAI 2022\n\n[3] Fair Graph Representation Learning via Diverse Mixture-of-Experts, WWW 2023\n\nQUESTIONS:\nplease refer to weaknesses.\n\nLIMITATIONS:\nplease refer to weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper claims that increasing the diversity of graph structures plays an important role in GNN training, which makes GNNs more robust and generalizable. To make GNN models leverage the structural diversity better, the paper proposes the Graph Mixture of Experts (GMoE). In each layer of GMoE, the representation of each node will be computed based on multiple experts with independent aggregation functions. Each expert will utilize either the hop-1 or hop-2 aggregation function to increase the diversity further. The paper conducts extensive experiments on OGB benchmark datasets and improves the performance compared to the single-expert baselines.\n\nSTRENGTHS:\n1. The paper claims the importance of graph data diversity in GNN training. \n2. The paper proposes a new architecture called GMoE, which combines MoE in neighborhood aggregation. \n3. The paper also tests whether the GMoE can help the graph self-supervised learning.\n\nWEAKNESSES:\n1. In the abstract, the paper argues that GMoE enables each node to select its own optimal information aggregation experts. But we do not see evidence why GMoE will find optimal experts. \n2. From my view, the formulation of the proposed GMoE is similar to GAT with 2-hop neighbors, limiting the work's novelty. \n3. Although GMoE improves the results compared to single-expert GCN/GIN models, the results are still far from the state-of-the-art results on the OGB leaderboard (e.g., top-1 result of ogbg-molhiv is 84.20).\n\nQUESTIONS:\n1. Is there any evidence that GMoE will find the optimal experts?\n2. From Equation (3), it seems that the selection of experts only relies on the target node i and do not rely on the node j. Is it reasonable?\n3. The authors might give more discussions of the difference or connections between GMoE and GAT with 2-hop neighbors.\n\nLIMITATIONS:\nThere is no negative societal impact of the work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel Graph Mixture of Experts (GMoE) architecture that enhances the ability of Graph Neural Networks (GNNs) to leverage and benefit from the structural diversity present in training graphs. The GMoE model comprises multiple \"experts\" at each layer, with each expert being an independent aggregation function with its own trainable parameters. During training, the model learns to select the most appropriate aggregation experts for each node, resulting in nodes with similar neighborhood information being routed to the same aggregation experts.\n\nSTRENGTHS:\nThe proposed Graph Mixture of Experts (GMoE) architecture is a new approach to enhancing the ability of Graph Neural Networks (GNNs) to leverage and benefit from the structural diversity present in training graphs.\n\nThe proposed GMoE model is evaluated on several benchmark datasets and achieves state-of-the-art performance on node classification and graph classification tasks. This demonstrates the effectiveness of the proposed architecture in improving the performance of GNNs on diverse graph structures.\n\nThe proposed GMoE model is flexible and can be easily integrated with existing GNN architectures. This makes it easy to incorporate the proposed model into existing GNN-based applications and frameworks.\n\nWEAKNESSES:\nWhile the proposed GMoE model is designed to avoid additional computational overhead, the paper does not provide a detailed analysis of the computational cost of the model compared to other GNN architectures. This makes it difficult to assess the practicality of the proposed model in real-world applications.\n\nWhile the paper describes how the model learns to select the most appropriate aggregation experts for each node, it does not provide a detailed explanation of the selection mechanism. This makes it difficult to understand how the model makes decisions and how it can be improved.\n\nOne suggestion for improving this paper would be to include more details of parameter tuning for the compared baselines. This would help to ensure a fair comparison between the proposed GMoE model and other state-of-the-art models. The paper could also describe the process used to tune these hyperparameters, such as grid search or random search.\n\nQUESTIONS:\nThe paper lacks a detailed analysis of the computational cost of the proposed GMoE model compared to other GNN architectures. How does the computational cost of the model compare to other state-of-the-art models, and how practical is it for real-world applications?\n\nThe paper does not provide a detailed explanation of the selection mechanism that the proposed GMoE model uses to choose the most appropriate aggregation experts for each node. How does the model make these decisions, and how can the selection mechanism be improved?\n\nThe paper could be improved by including more details on the process used to tune the hyperparameters of the compared baselines. What process was used to tune these hyperparameters, and how can a fair comparison be made between the proposed GMoE model and other state-of-the-art models?\n\nLIMITATIONS:\nThe paper lacks a detailed analysis of the computational cost of the proposed GMoE model compared to other GNN architectures and does not provide a thorough explanation of the selection mechanism used by the model to choose the most appropriate aggregation experts for each node. To improve the paper, the authors could include more details on the process used to tune the hyperparameters of the compared baselines, enabling a fair comparison with other state-of-the-art models. Additionally, providing a more in-depth explanation of the selection mechanism used by the GMoE model would facilitate a better understanding of the model's decision-making process and enable its improvement.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tracks the problem of training a good GNN. To enhance the generalization ability of GNNs, it has become common to increase the diversity of training graph structures by incorporating graph augmentations and/or performing large-scale pre-training on more graphs. Therefore, it is essential for a GNN to simultaneously model diverse graph structures. However, naively increasing the GNN model capacity will suffer from both higher inference costs and the notorious trainability issue of GNNs. To address this issue, this paper proposes Graph mixture of experts, borrowing the idea from recent development of mixture of experts. Based on this new method, each node can select their own optimal information aggregation experts. Experiments are solid and verify the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. This paper focuses on an important problem: learning under graphs, which is a common scenario in the real world. \n\n2. This paper designs a new and interesting framework to explore how to use mixture of experts on graphs. The way to mixture graphs is interesting and make a lot of sense. For example, for different node, its optimal local information should be different. This paper considers a novel way to address this problem.\n\n3. The experiments are conducted on many datasets, and the proposed method performs well. It is very hard to explore a useful way to mixture graphs. This paper makes the first try, and the proposed method is novel and verified by solid experiments, which is appreciated.\n\nWEAKNESSES:\n1. This paper can be improved further in terms of writing. You introduce two interesting modules but without good motivations. You can verify your motivation via conducting experiments to show that considering different location for each node will make things different. Thus, you can motivate your idea: we need to choose optimal local information (hop 1 or hop 2). \n\n2. The paper structure should be improved. I think the preliminaries should be considered as a separate section, instead of subsection. It can make readers obtain essential knowledge before the method.\n\n\n3. For formula, there should always be “,”, “.” or “;” in the end of it. Both of formula and text consist of your paper, so formula is just text to be displayed. Besides, text inside the formula should be with /textnormal{}.\n\n4. There is one plus point if the authors can address it well. Because you select experts, how could you ensure that the algorithm converges? The selection results will always converge to a certain selection result? It is very interesting to discuss this point. What is the main drawback to obtain a corresponding convergence result?\n\n5. Some in-text citation is not completed. For example, in line 197, reference 10.\n\nQUESTIONS:\n1. This paper can be improved further in terms of writing. You introduce two interesting modules but without good motivations. You can verify your motivation via conducting experiments to show that considering different location for each node will make things different. Thus, you can motivate your idea: we need to choose optimal local information (hop 1 or hop 2). \n\n2. The paper structure should be improved. I think the preliminaries should be considered as a separate section, instead of subsection. It can make readers obtain essential knowledge before the method.\n\n\n3. For formula, there should always be “,”, “.” or “;” in the end of it. Both of formula and text consist of your paper, so formula is just text to be displayed. Besides, text inside the formula should be with /textnormal{}.\n\n4. There is one plus point if the authors can address it well. Because you select experts, how could you ensure that the algorithm converges? The selection results will always converge to a certain selection result? It is very interesting to discuss this point. What is the main drawback to obtain a corresponding convergence result?\n\n5. Some in-text citation is not completed. For example, in line 197, reference 10.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}