SlowGuess/NeurIPS_2023_OCR huggingface dataset:
GN block. The full GN block, introduced in Sec. 2.1, is implemented as a graph full network (GFuN) layer. This layer exhibits a highly configurable within-block structure, enabling it to express a variety of other architectures (see Sec. 4.2 of [6]), like GCN, GIN, GAT, GatedGCN. Thus, modifying the within-block structure of GFuN is akin to plugging in different GNN cores. Further details can be found in App. C.3.
Encoder and decoder. Most preprocessing methods (including positional encodings and graph rewiring), encoding (input embedding) and decoding (readout functions) schemes applicable to GNNs can also be applied to MeGraph. We give implementation details in App. C.4.
3.4 Computational Complexity and Discussion
The overall complexity of the MeGraph model is contingent on the height  $h$ , the number of Mee layers  $n$ , the chosen modules, and the corresponding hyperparameters. Let  $D$  be the embedding size,  $V$  the number of nodes, and  $E$  the number of edges in the input graph  $\mathcal{G}$ . The time complexity of S-Edgepool is  $O(ED + E\log E)$ , and that of a GFuN layer is  $O(VD^2 + ED)$ . Assuming both the pooling ratios of nodes and edges are  $\eta$ , the total time complexity to construct the mega graph  $\mathcal{M}\mathcal{G}$  becomes  $O((ED + E\log E) / (1 - \eta))$ , where  $\sum_{i = 0}^{h - 1}\eta^i < 1 / (1 - \eta)$ . Similarly, the total time complexity of an Mee layer is  $O((VD^2 + ED) / (1 - \eta))$ . This complexity is equivalent to a typical GNN layer if we consider  $1 / (1 - \eta)$  as a constant (for instance, it is a constant of 2 when  $\eta = 0.5$ ).
Theoretically, when using the same number of layers, MeGraph is better at capturing LRIs than standard message-passing GNNs owing to the hierarchical structure (see App. D.1 for details). On the other hand, MeGraph can degenerate into standard message-passing GNNs (see App. D.2 for details), indicating it should not perform worse than them on other tasks.
We conduct extensive experiments to evaluate the MeGraph's ability to capture long-range interactions (LRIs) and its performance in general graph learning tasks.
Baselines. We compare MeGraph model to three baselines as follows: 1) MeGraph  $h = 1$  variant does not use the hierarchical structure and falls back to standard GNNs. 2) MeGraph  $n = 1$  variant gives up repeating information exchange over the mega graph. 3) Graph U-Nets [22] uses a U-shaped design and only traverses the multi-scale graphs once.
Due to page limits, statistics of the datasets are provided in App. B.1, hyper-parameters are reported in Table 9, and the training and implementation details are reported in App. E.
Table 1: Results on Graph Theory Benchmark (medium size). For each task, we report the MSE regression loss on test set, averaged over different graph generation methods. Darker blue cells denote better performance and the bold denotes the best one. We provide detailed results on each type of graphs in App. F.7.
Category	Model	\(SP_{sssd}\)	MCC	Diameter	\(SP_{ss}\)	ECC	Average
Baselines (h=1)	n=1	11.184	1.504	11.781	22.786	20.133	13.478
n=5	3.898	1.229	5.750	12.354	18.971	8.440
n=10	2.326	1.264	5.529	7.038	18.876	7.006
MeGraph (h=5)	n=1	1.758	0.907	4.591	5.554	14.030	5.368
EdgePool (\(τ_c\)=2)	n=5	0.790	0.767	2.212	0.712	6.593	2.215
MeGraph S-EdgePool Variants (h=5, n=5)	\(\tau_c\)=3	0.660	0.747	0.719	0.459	0.942	0.705
\(\eta_v\)=0.3	2.225	0.778	1.061	3.591	2.009	1.933
\(\eta_v\)=0.3,\(τ_c\)=4	0.615	0.702	0.651	0.434	0.975	0.675
\(\eta_v\)=0.5,\(τ_c\)=4	1.075	0.769	0.945	1.204	1.992	1.197
\(\eta_v\)=0.3,\(τ_c\)=4 (X-Pool)	0.935	0.751	0.864	1.462	2.003	1.203
\(\eta_v\)=0.3,\(τ_c\)=4 (w/o pw)	0.632	0.730	0.864	0.765	2.334	1.065
Graph-UNets	h=5,n=9,\(\eta_v\)=0.3,\(τ_c\)=4	1.118	1.008	2.031	1.166	2.584	1.581
Table 2: Results on Graph Theory Benchmark (large size).
Category	Model	SPssd	MCC	Diameter	SPss	ECC	Average
Baseline	h=1,n=5	328.014	39.4772	189.577	324.033	219.746	220.169
MeGraph	h=5,n=5,ηv=0.3,τc=4	23.8963	16.8321	19.2185	14.9676	44.9234	23.9676
Graph-UNets	h=5,n=9,ηv=0.3,τc=4	101.009	30.3711	39.8708	100.070	75.1185	69.2879
Table 3: Results on LRGB [17], numbers are taken from corresponding papers. All methods use around 500K parameters for a fair comparison. Message-passing-based models have 5 layers, while the Transformer-based models have 4 layers [17]. PE indicates positional encoding, which makes it easier to distinguish different nodes.
Methods	Use PE	Peptide-func ↑	Peptide-struct ↓
GCN [17]	59.30 ±0.23	0.3496 ±0.0013
GINE [17]	55.43 ±0.78	0.3547 ±0.0045
GatedGCN [17]	58.64 ±0.77	0.3420 ±0.0013
GatedGCN+RWSE [17]	✓	60.69 ±0.35	0.3357 ±0.0006
GatedGCN+RWSE+VN [10]	✓	66.85 ±0.62	0.2529 ±0.0009
Transformer+LapPE [17]	✓	63.26 ±1.26	0.2529 ±0.0016
SAN+LapPE [17]	✓	63.84 ±1.21	0.2683 ±0.0043
SAN+RWSE [17]	✓	64.39 ±0.75	0.2545 ±0.0012
GPS [46]	✓	65.35 ±0.41	0.2500 ±0.0005
MGT+WavePE [43]	✓	68.17 ±0.64	0.2453 ±0.0025
GNN-AK+ [27]	64.80 ±0.89	0.2736 ±0.0007
SUN [27]	67.30 ±0.78	0.2498 ±0.0008
GraphTrans+PE [27]	✓	63.13 ±0.39	0.2777 ±0.0025
GINE+PE [27]	✓	64.05 ±0.77	0.2780 ±0.0021
GINE-MLP-Mixer+PE [27]	✓	69.21 ±0.54	0.2485 ±0.0004
MeGraph (h=9,n=1)	67.52 ±0.78	0.2557 ±0.0011
MeGraph (h=9,n=4)	69.45 ±0.77	0.2507 ±0.0009
4.2 Performance on LRI Tasks
To test MeGraph's ability to capture long-range interactions, we establish a Graph Theory Benchmark, of which four tasks related to shortest path distance, i.e., Single Source Shortest Path  $(\mathrm{SP}_{\mathrm{ss}})$ , Single Source Single Destination Shortest Path  $(\mathrm{SP}_{\mathrm{sdd}})$ , Graph Diameter (Diameter) and Eccentricity of nodes (ECC); and 1 task related to connected component, i.e., Maximum Connected Component of the same color (MCC). To generate diversified undirected and unweighted graphs for each task, we adopt the ten methods used in PNA [12] and add four new methods: cycle graph, pseudotree, SBM, and geographic threshold graphs. The details of the dataset generation can be found in App. B.2.
As depicted in Table 1, the MeGraph model with  $h = 5$ ,  $n = 5$  significantly outperforms both the  $h = 1$  and  $n = 1$  baselines in terms of reducing regression error across all tasks. It is worth noting that even
Table 4: Results on GNN benchmark. Regression tasks are colored with blue.  $\downarrow$  indicates that smaller numbers are better. Classification tasks are colored with green.  $\uparrow$  indicates that larger numbers are better. Darker colors indicate better performance.  $\dagger$  denotes the results are reported in [15].
Model	ZINC ↓	AQSOL ↓	MNIST ↑	CIFAR10 ↑	PATTERN ↑	CLUSTER ↑
GCN†	0.416 ±0.006	1.372 ±0.020	90.120 ±0.145	54.142 ±0.394	85.498 ±0.045	47.828 ±1.510
GIN†	0.387 ±0.015	1.894 ±0.024	96.485 ±0.252	55.255 ±1.527	85.590 ±0.011	58.384 ±0.236
GAT†	0.475 ±0.007	1.441 ±0.023	95.535 ±0.205	64.223 ±0.455	75.824 ±1.823	57.732 ±0.323
GatedGCN†	0.435 ±0.011	1.352 ±0.034	97.340 ±0.143	67.312 ±0.311	84.480 ±0.122	60.404 ±0.419
Graph-UNets	0.332 ±0.010	1.063 ±0.018	97.130 ±0.227	68.567 ±0.339	86.257 ±0.078	50.371 ±0.243
MeGraph (h=1)	0.323 ±0.002	1.075 ±0.007	97.570 ±0.168	69.890 ±0.209	84.845 ±0.021	58.178 ±0.079
MeGraph (n=1)	0.310 ±0.005	1.038 ±0.018	96.867 ±0.167	68.522 ±0.239	85.507 ±0.402	50.396 ±0.082
MeGraph	0.260 ±0.005	1.002 ±0.021	97.860 ±0.098	69.925 ±0.631	86.507 ±0.067	68.603 ±0.101
MeGraphbest	0.202 ±0.007	1.002 ±0.021	97.860 ±0.098	69.925 ±0.631	86.732 ±0.023	68.610 ±0.164
Table 5: Results on OGB-G.  $\dagger$  indicates that the results are reported in [28].
Model	molhiv ↑	molbace ↑	molbbbup ↑	molclintox ↑	molsider ↑
GCN†	76.06 ±0.97	79.15 ±1.44	68.87 ±1.51	91.30 ±1.73	59.60 ±1.77
GIN†	75.58 ±1.40	72.97 ±4.00	68.17 ±1.48	88.14 ±2.51	57.60 ±1.40
Graph-UNets	79.48 ±1.06	81.09 ±1.66	71.10 ±0.52	91.67 ±1.69	59.38 ±0.63
MeGraph (h=1)	78.54 ±1.14	71.77 ±2.15	67.56 ±1.11	89.77 ±3.48	58.28 ±0.51
MeGraph (n=1)	78.56 ±1.02	79.72 ±1.24	67.34 ±0.98	91.07 ±2.21	58.08 ±0.59
MeGraph	77.20 ±0.88	78.52 ±2.51	69.57 ±2.33	92.04 ±2.19	59.01 ±1.45
MeGraphbest	79.20 ±1.80	83.52 ±0.47	69.57 ±2.33	92.06 ±1.32	63.43 ±1.10
Model	moltox21 ↑	moltoxcast ↑	molesol ↓	molfreesolv ↓	mollipo ↓
GCN†	75.29 ±0.69	63.54 ±0.42	1.114 ±0.03	2.640 ±0.23	0.797 ±0.02
GIN†	74.91 ±0.51	63.41 ±0.74	1.173 ±0.05	2.755 ±0.34	0.757 ±0.01
Graph-UNets	77.85 ±0.81	66.49 ±0.45	1.002 ±0.04	1.885 ±0.07	0.716 ±0.01
MeGraph (h=1)	75.89 ±0.45	64.49 ±0.46	1.079 ±0.02	2.017 ±0.08	0.768 ±0.00
MeGraph (n=1)	77.01 ±0.93	66.89 ±1.21	0.896 ±0.04	1.892 ±0.06	0.730 ±0.01
MeGraph	78.11 ±0.47	67.67 ±0.53	0.886 ±0.02	1.876 ±0.05	0.726 ±0.00
MeGraphbest	78.11 ±0.47	67.90 ±0.19	0.867 ±0.02	1.876 ±0.05	0.688 ±0.01
the  $h = 5$ ,  $n = 1$  baseline outperforms the  $h = 1$ ,  $n = 10$  baseline, indicating that adopting a multi-scale graph hierarchy is crucial in these tasks. The improvement is also substantial when compared with our reproduced Graph-UNets using S-EdgePool ([MeGraph] 0.675 vs. [Graph UNets] 1.581). The improvements are more significant when the size of graphs becomes larger (as shown in Table 2). These results collectively demonstrate the superior ability of MeGraph to capture LRIs.
Furthermore, we evaluated MeGraph model and compared it with other recent methods on the Long Range Graph Benchmark (LRGB) [17] that contains real-world tasks that require capturing LRIs. As depicted in Table 3, the  $h = 9$ ,  $n = 4$  variant of MeGraph achieves superior results on the Peptide-function task, and comparable performance on the Peptide-struct task, relative to state-of-the-art models. It is worth noting that the  $n = 1$  variant already surpasses other methods except the recent MLP-Mixer [27] in the Peptide-function task.
4.3 Generality of MeGraph
To verify the generality of MeGraph model, we evaluate MeGraph on widely adopted GNN Benchmark [15], Open Graph Benchmark [28] and TU Dataset [42]. Results on TU Datasets are available in App. F.3. In addition to the standard model that shares hyper-parameters in similar tasks, we also report MeGraphbest with specifically tuned hyper-parameters for each task.
GNN Benchmark. We experiment on chemical data (ZINC and AQSOL), image data (MNIST and CIFAR10) and social network data (PATTERN and CLUSTER). As shown in Table 4, MeGraph outperforms the three baselines by a large margin, indicating the effectiveness of repeating both the local and hierarchical information aggregation.
Open Graph Benchmark (OGB). We choose 10 datasets related to molecular graphs from the graph prediction tasks of OGB. The task of all datasets is to predict some properties of molecule graphs
Figure 4: Node classification accuracy (averaged over 10 random repetitions) for MeGraph on TreeCycle (left) and TreeGrid (right) datasets by varying the number of Mee layers  $n$ , the height  $h$ , and the graph pooling methods (EdgePool, Louvain, Random). Clear gaps can be observed among heights 1, 2, and 3 for EdgePool and Louvain [8] methods, while the accuracy is almost invariant among different heights for randomized pooling.
based on their chemical structures. As shown in Table 5, MeGraph outperforms the  $h = 1$  baseline by a large margin, suggesting that building a graph hierarchy is also essential in molecule graphs. The performance of MeGraph,  $n = 1$  baseline, and the reproduced Graph U-Nets are comparable. This observation may be because the information obtained from multi-hop neighbors offers only marginal improvements compared to the information aggregated hierarchically.
Hierarchy vs. Locality. We study the impact of the height  $h$  and the number of Mee layers  $n$  on four synthetic datasets introduced in [61], which are BASHape, BACommunity, TreeCycle, and TreeGrid. Each dataset contains one graph formed by attaching multiple motifs to a base graph. The motif can be a 'house'-shaped network (BAShape, BACommunity), six-node cycle (TreeCycle), or 3-by-3 grid (TreeGrid). The task is to identify the nodes of the motifs in the fused graph.
As shown in Fig. 4, we can observe clear improvements in performance as the height  $h$  and the number of layers  $n$  increase when using EdgePool. Although increasing height shows better improvements, both hierarchy and locality are indispensable in TreeCycle and TreeGrid tasks. In App. F.5, we show that the conclusion also holds on BASHape and BACommunity datasets, except that the accuracy is already saturated with height  $h = 2$ . The significance of integrating locality with hierarchy is also demonstrated in the CLUSTER task, as presented in Table 4. Here, MeGraph reaches  $68.6\%$  accuracy, which is markedly higher than the  $50.4\%$  accuracy achieved by both the  $n = 1$  baseline and Graph-UNets.
Varying the Pooling Method. We varied the graph pooling with two non-learnable pooling methods, Louvain [8] and Random. On the TreeCycle and TreeGrid datasets, as depicted in Fig. 4, Louvain achieves comparable accuracy to EdgePool, while MeGraph using random pooling matches the performance of the  $h = 1$  variant of MeGraph. These observations indicate that: 1) a well-structured hierarchy crafted by suitable graph pooling methods enhances MeGraph's ability to harness hierarchical information; and 2) despite the disruption from a randomly constructed hierarchy, the MeGraph model effectively taps into the local structure of the original graph (also discussed in App. D.2).
We further studied the impact of the node pooling ratio  $\eta_v$  and the maximum cluster size  $\tau_c$  in S-EdgePool by perturbing these parameters. As indicated in Table 1, the best variant  $(\eta_v = 0.3, \tau_c = 4)$  achieved a regression error about 3x smaller (0.675) compared to the original EdgePool ( $\tau_c = 2$  with an error of 2.215). This suggests the benefit of having a flexible pooling stride. Moreover, the mega graph produced by S-EdgePool can vary significantly with different parameters. In App. F.2, we visualized the resulting graph hierarchy to illustrate the difference between different S-EdgePool variants. However, irrespective of the parameter set used, MeGraph consistently outperforms  $h = 1$  baselines. This suggests that MeGraph exhibits robustness against different architectures of the mega graph.
Varying GN Block. We varied the aggregation function of the GN block as attention (w/ ATT) and gated function (w/ GATE). We observe similar results as in Sec. 4.2 and 4.3, verifying the robustness of MeGraph towards different types of GN blocks. Detailed results can be found in Tables 16, 17 and 18 in App. F.6.
Changing cross update function (X-UPD). The unpool operation is frequently used by other hierarchical architectures that build upon graph pooling. As illustrated in Table 1, we substituted the X-Conv implementation with the X-Pool implementation of X-UPD, which resulted in a performance decline from 0.675 to 1.203 (smaller is better). This finding suggests that other hierarchical GNNs might also benefit from replacing the unpool operation with a convolution over the inter-graph edges.
Disabling bidirectional pathway. We verify the effectiveness of the bidirectional pathway design by replacing steps 2 and 3 of the Mee layer as a standard message-passing along the inter-graph edges (denoted w/o pw). As shown in Table 1, the performance degrades from 0.675 to 1.065 (smaller is better), which indicates the contribution of the bidirectional pathway.
Long-Range Interactions (LRIs). Various methods have been proposed to address the issue of LRIs, including making the GNNs deeper [40]. Another way is to utilize attention and gating mechanism, including GAT [52], jumping knowledge (JK) network [59], incorporating Transformer structures [35, 60, 56, 46] and MLP-Mixer [27]. Another line of research focuses on multi-scale graph hierarchy using graph pooling methods [62, 22, 47], or learning representation based on subgraphs [5, 66]. Recently, Long Range Graph Benchmark [17] has been proposed to better evaluate models' ability to capture LRIs.
Feature Pyramids and Multi-Scale Feature Fusion. Multi-scale feature fusion methods on image feature pyramids have been widely studied in computer vision literature, including the U-Net [48], FPN [39], UNet++ [67], and some recent approaches [63, 41, 38, 37]. HRNet [53] is a similar method compared to MeGraph. HRNet alternates between multi-resolution convolutions and multi-resolution fusion by stridden convolutions. However, the above methods are developed for image data. The key difference compared to these approaches is that the multi-scale feature fusion in MeGraph is along the inter-graph edges, which is not as well structured as the pooling operation in image data. For graph networks, the GraphFPN [65] builds a graph feature pyramid according to the image feature pyramid and superpixel hierarchy. It applies GNN layers on the hierarchical graph to exchange information within the graph pyramid. Existing works [22, 20, 47, 30] have also explored similar ideas in graph-structured data. Our approach aligns with the broader concept of multi-scale information fusion, but it is the first method that builds a mega graph using graph pooling operations and alternates local and hierarchical information aggregation.
Graph Pooling Methods. Graph pooling is an important part of hierarchical graph representation learning. There have been some traditional graph pooling methods like METIS [32] in early literature. Recently, many learning-based graph pooling methods have been proposed, including the DiffPool [62], TopKPool [22], SAG pool [36], EdgePool [13], MinCutPool [7], Structpool [64], and MEWISPool [44], etc. In this work, we utilize S-EdgePool improved from EdgePool to build the mega graph, while this module can be substituted with any of the above-mentioned pooling methods.
Graph Neural Network (GNN) Layers. The GNN layer is the core module of graph representation learning models. Typical GNNs include the GCN [34], GraphSage [24], GAT [52, 9], GIN [58], PNA [12]. MeGraph adopts the full GN block [6] by removing part of links in the module as an elementary block, and similarly this can be replaced by any one of the popular GNN blocks.
The MeGraph model suffers from some limitations. The introduced mega graph architecture inevitably increases both the number of trainable parameters and tuneable hyper-parameters. The flexible choices of many modules in MeGraph post burdens on tuning the architecture on specific datasets. For future research, MeGraph encourages new graph pooling methods to yield edge features in addition to node features, when mapping the input graph to the pooled graph. It is also possible to improve MeGraph using adaptive computational steps [49]. Another direction is to apply some expressive but computationally expensive models like Transformers [51] and Neural Logic Machines [14, 57] (only) over the pooled small-sized graphs.
We thank Qizheng He for the discussions on the Graph Theory Dataset, Kirsty You for improving the illustrations, and Jiafei Lv and the Maddison group for their helpful discussions or feedback on the paper draft; This work was supported by the STI 2030-Major Projects under Grant 2021ZD0201404.
[54] Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis, Jinyang Li, and Zheng Zhang. Deep graph library: A graph-centric, highly-performant package for graph neural networks. arXiv preprint arXiv:1909.01315, 2019.
[55] Duncan J WATTS. Networks, dynamics and the small world phenomenon. American Journal of Sociology, 105(2):50-59, 2003.
[56] Zhanghao Wu, Paras Jain, Matthew Wright, Azalia Mirhoseini, Joseph E Gonzalez, and Ion Stoica. Representing long-range context for graph neural networks with global attention. Advances in Neural Information Processing Systems, 34:13266-13279, 2021.
[57] Guangxuan Xiao, Leslie Pack Kaelbling, Jiajun Wu, and Jiayuan Mao. Efficient training and inference of hypergraph reasoning networks, 2022.
[58] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2018.
[59] Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In International conference on machine learning, pages 5453–5462. PMLR, 2018.
[60] Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems, 34:28877-28888, 2021.
[61] Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. Gnnexplainer: Generating explanations for graph neural networks. Advances in neural information processing systems, 32, 2019.
[62] Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. Advances in neural information processing systems, 31, 2018.
[63] Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2403-2412, 2018.
[64] Hao Yuan and Shuiwang Ji. Structpool: Structured graph pooling via conditional random fields. In Proceedings of the 8th International Conference on Learning Representations, 2020.
[65] Gangming Zhao, Weifeng Ge, and Yizhou Yu. Graphfpn: Graph feature pyramid network for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2763-2772, 2021.
[66] Lingxiao Zhao, Wei Jin, Leman Akoglu, and Neil Shah. From stars to subgraphs: Uplifting any gnn with local structure awareness. In International Conference on Learning Representations.
[67] Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang. Unet++: A nested u-net architecture for medical image segmentation. In Deep learning in medical image analysis and multimodal learning for clinical decision support, pages 3-11. Springer, 2018.
A Code and Reproducibility 16
B Dataset Details 16
B.1 Dataset Statistics and Metrics 16
B.2 Graph Theory Benchmark 16
C Method Details 18
C.1 Cross Update Function 18
C.2 S-EdgePool 19
C.3 GFuN 20
C.4Encoder and Decoder 21
C.5 Architecture Variants 22
D Theoretical Discussions 22
D.1 Smaller Number of Aggregation Steps for Capturing Long-Range Interactions 22
D.2 MeGraph can degenerate to standard GNNs 22
E Implementation and Training Details 23
F Additional Experiment Results 23
F.1 Experimental Protocol 23
F.2 Visualization 24
F.3 Other Real-Wrold Datasets 24
F.4 GFuN 24
F.5 Synthetic Datasets 25
F.6 Varying GN block 25
F.7 Graph Theory Dataset 27
A Code and Reproducibility
The code along with the configuration of hyper-parameters to reproduce our experiments can be found at https://github.com/dhh1995/MeGraph.
We set the random seed as 2022 for all experiments to enable reproducible results. We provide dataset statistics in Table 6 and details for the proposed graph theory benchmark in Appendix B.2. Details of the hyper-parameters are reported in Table 9. Configuration of all hyper-parameters and the command lines to reproduce the experiments have been included in the code repository.
B.1 Dataset Statistics and Metrics
We provide the statistics of all datasets used in our experiments in Table 6 and introduce the evaluation metrics for each dataset.
For Synthetic datasets, we use classification accuracy (ACC) as the evaluation metric. We use Mean Square Error (MSE) as the evaluation metric for all datasets in our Graph Theory Benchmark. For GNN Benchmark, we follow the original work [15] for evaluation, i.e., Mean Absolute Error (MAE) for ZINC and AQSOL, classification accuracy for MNIST and CIFAR10, and balanced classification accuracy for PATTERN and CLUSTER. For OGB Benchmark, we follow the original work [28] and use the ROC-AUC for classification tasks and Root Mean Square Error (RMSE) for regression tasks. For TU datasets, we follow the setting used by [11] and use classification accuracy as the evaluation metric.
B.2 Graph Theory Benchmark
In this section, we provide the details about the tasks and how the graph features and the labels are generated given a base graph  $\mathcal{G} = (\mathcal{V},\mathcal{E})$ :