SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Single source single destination shortest path  $(\mathrm{SP}_{\mathrm{sssd}})$ : a source node  $s \in \mathcal{V}$  and a destination node  $t \in \mathcal{V}$  are selected uniform randomly. The feature of each node  $v$  contains three numbers: (1, whether the node  $v$  is  $s$ , whether the node  $v$  is  $t$ ). The label of a graph is the length of the shortest path from  $s$  to  $t$ .
- A maximum connected component of the same color (MCC): each node of the graph is colored with one of three colors. The feature for each node is the one-hot representation of its color. The label of graph is the size of the largest connected component of the same color for each color.
- Graph diameter (Diameter): the label of the graph is the diameter of the graph. The diameter of a graph  $\mathcal{G}$  is the maximum of the set of shortest path distances between all pairs of nodes in the graph. The feature of each node is a uniform number 1.
- Single source shortest path  $(\mathrm{SP}_{\mathrm{ss}})$ : a source node  $s$  is selected uniformly randomly. The feature of each node contains two numbers: (1, whether the node is  $s$ ). The label of each node is the length of the shortest path from  $s$  to this node.
- Graph eccentricity (ECC): the label of each node  $v$  is node's eccentricity in the graph, which is the maximum distance from  $v$  to the other nodes. The feature of each node is a uniform number 1.
For each task and graph generation method, We generate the dataset by the following steps:
- Sample  $N$  (number of nodes) from an interval with a total number of graphs. These numbers can be configured. For the medium size setting, the interval is [20, 50], a total of 300 graphs. For the large size setting, the interval is [100, 200], a total of 500 graphs.
- Use the graph generation method to generate a graph of  $N$  nodes.
- Create graph features and labels according to the task.
Table 6: The statistics of the datasets used in experiments. Some statistics (like the average number of edges) of the Graph Theory datasets may vary depending on different random graph generation methods. The regression tasks are marked with  $\checkmark$  in a separate column. The tasks of 4 synthetic datasets are transductive, where the same graph is used for both training and testing. We do not use the node labels as features during the training time. The train-val-test split is over nodes. All other datasets in the table are inductive, where the testing graphs do not occur during training, and the train-val-test split is over graphs.
Collection	Dataset	# Graphs	Avg # Nodes	Avg # Edges	# Node Feat	# Edge Feat	# Classes	Task	Reg.
Synthetic	BaShape	1	700	1761	1	-	4	Trans-Node
Synthetic	BaCommunity	1	1400	3872	10	-	8	Trans-Node
Synthetic	TreeCycle	1	871	970	1	-	2	Trans-Node
Synthetic	TreeGrid	1	1231	1705	1	-	2	Trans-Node
GraphTheory	SPsssd	300	35.0	-	3	-	-	Graph	✓
GraphTheory	Diameter	300	35.0	-	1	-	-	Graph	✓
GraphTheory	MCC	300	35.0	-	3	-	-	Graph	✓
GraphTheory	SPss	300	35.0	-	2	-	-	Node	✓
GraphTheory	ECC	300	35.0	-	1	-	-	Node	✓
LRGB	Peptides-func	15535	150.94	307.30	9	3	10	Graph
LRGB	Peptides-struct	15535	150.94	307.30	9	3	-	Graph	✓
GNNBenchmark	ZINC	12000	23.16	49.83	28	4	2	Graph	✓
GNNBenchmark	AQSOL	9823	17.57	35.76	65	5	2	Graph	✓
GNNBenchmark	MNIST	70000	70.57	564.53	3	1	10	Graph
GNNBenchmark	CIFAR10	60000	117.63	941.07	5	1	10	Graph
GNNBenchmark	PATTERN	14000	118.89	6078.57	3	-	2	Node
GNNBenchmark	CLUSTER	12000	117.20	4301.72	7	-	6	Node
OGB Graph	molhiv	41127	25.51	80.45	9	3	2	Graph
OGB Graph	molbace	1513	34.09	107.81	9	3	2	Graph
OGB Graph	molbbbp	2039	24.06	75.97	9	3	2	Graph
OGB Graph	molclintox	1477	26.16	81.93	9	3	2	Graph
OGB Graph	molsider	1427	33.64	104.36	9	3	2	Graph
OGB Graph	moltox21	7831	18.57	57.16	9	3	2	Graph
OGB Graph	moltoxcast	8576	18.78	57.30	9	3	2	Graph
OGB Graph	molesol	1128	13.29	40.64	9	3	-	Graph	✓
OGB Graph	molfreesolv	642	8.7	25.50	9	3	-	Graph	✓
OGB Graph	mollipo	4200	27.04	86.04	9	3	-	Graph	✓
TU	MUTAG	188	17.93	19.79	7	-	3	Graph
TU	NCI1	4110	29.87	32.30	37	-	2	Graph
TU	PROTEINS	1113	39.06	72.82	4	-	2	Graph
TU	D&D	1178	284.32	715.66	89	-	2	Graph
TU	ENZYMES	600	32.63	62.14	21	-	6	Graph
TU	IMDB-B	1000	19.77	96.53	10	-	2	Graph
TU	IMDB-M	1500	13.00	65.94	10	-	3	Graph
TU	RE-B	2000	429.63	497.75	10	-	2	Graph
TU	RE-M5K	4999	508.52	594.87	10	-	5	Graph
TU	RE-M12K	11929	391.41	456.89	10	-	11	Graph
We then provide the details about the random graph generation methods we used to create our Graph Theory datasets.
Following [12], we continue to use undirected and unweighted graphs from a wide variety of types. We inherit their 10 random graph generation methods and quote their descriptions here for completeness (the percentage after the name is the approximate proportion of such graphs in the mixture setting).
- Erdős-Rényi (ER)  $(20\%)$  [18]: with a probability of presence for each edge equal to  $p$  where  $p$  is independently generated for each graph from  $\mathcal{U}[0,1]$
- Barabási-Albert (BA)  $(20\%)$  [2]: the number of edges for a new node is  $k$ , which is taken randomly from  $\{1, 2, \dots, N - 1\}$  for each graph
- Grid (5%):  $m \times k$  2d grid graph with  $N = mk$  and  $m$  and  $k$  as close as possible
- Caveman (5%) [55]: with  $m$  cliques of size  $k$ , with  $m$  and  $k$  as close as possible
- Tree (15%): generated with a power-law degree distribution with exponent 3
- Ladder graphs (5%)
- Line graphs (5%)
- Star graphs (5%)
- Caterpillar graphs (10%): with a backbone of size  $b$  (drawn from  $\mathcal{U}[1,N)$ ), and  $N - b$  pendent vertices uniformly connected to the backbone
- Lobster graphs (10%): with a backbone of size  $b$  (drawn from  $\mathcal{U}[1, N)$ ),  $p$  (drawn from  $\mathcal{U}[1, N - b]$ ) pendant vertices uniformly connected to the backbone, and additional  $N - b - p$  pendant vertices uniformly connected to the previous pendant vertices.
Additional, we add three more graph generation methods:
- Cycle graphs
- Pseudotree graphs: A tree graph plus an additional edge. The graph is generated by first generating a cycle graph of size  $m = \text{sample}(0.3N, 0.6N)$ . Then  $n - m$  remaining nodes are sampled to  $m$  parts, where  $i$ -th part represents the size of the tree hanging on the  $i$ -th node on the cycle. The trees are randomly generated with the given size.
- Stochastic Block Model (SBM) graphs: graphs with clusters. We randomly sample the size of each block to be random from (5, 15), and the probability of edge within the block to be random from (0.3, 0.5) and those for other edges to be random from (0.005, 0.01). To make all the tasks well-defined, we filtered out the unconnected graphs during the generation.
- Geographic (Geo) graphs: geographic threshold graphs, but with added edges via a minimum spanning tree algorithm, to ensure all nodes are connected. This graph generation method is introduced by [6] in their codebase<sup>2</sup>. We use the geographic threshold  $\theta = 200$  instead of the default value  $\theta = 1000$ .
Note that we do not have randomization after the graph generation as in [12]. Therefore, very long diameter is preserved for some type of graphs.
C.1 Cross Update Function
The cross update function  $(\mathbf{X}_j', \hat{\mathbf{X}}_j', \mathbf{X}_{j+1}') = \mathbb{X}\text{-UPD}(j, \mathbf{X}_j, \hat{\mathbf{X}}_j, \mathbf{X}_{j+1})$  performs information exchange in consecutive hierarchies.
The  $X$ -Conv realization contains the following steps:
1. Merge the node features of  $\mathbf{X}_j$  and  $\mathbf{X}_{j + 1}$  with the inter-graph feature  $\hat{\mathbf{X}}_j$ , results in  $\bar{\mathbf{X}}_j$
2. Apply GN blocks on inter-graph  $\hat{\mathcal{G}}_j$ :  $\hat{\mathbf{X}}_j' = \mathrm{GN}_{\mathrm{inter}}^{i,j}(\hat{\mathcal{G}}_j, \bar{\mathbf{X}}_j)$ .
3. Retrieve  $\mathbf{X}_j^{\prime}$  and  $\mathbf{X}_{j + 1}^{\prime}$  from the node features of inter-graph features  $\hat{\mathbf{X}}_j^\prime$
$^{2}$ https://github.com/deepmind/graph_nets, the shortest path demo
C.2 S-EdgePool
In this subsection, we introduce the details of S-EdgePool. We first introduce the score generation method, then give details about the SELECT, CONNECT,REDUCE and EXPAND functions, and lastly provide pseudocode of the algorithm.
C.2.1 Edge Score Generation
Both S-EdgePool and EdgePool methods compute a raw edge score  $\mathbf{r}_k$  for each edge  $k$  using a linear layer:
$$
\mathbf {r} _ {k} = \mathbf {W} \cdot \left(\mathbf {V} _ {s _ {k}} | | \mathbf {V} _ {t _ {k}} | | \mathbf {E} _ {k}\right) + \mathbf {b}
$$
where  $s_k$  and  $t_k$  are the source and target nodes of edge  $k$ ,  $\mathbf{V}$  is node features,  $\mathbf{E}$  is edge features,  $\mathbf{W}$  and  $\mathbf{b}$  are learned parameters. The raw edge scores are further normalized by a local softmax function over all edges of a node:
$$
\mathbf {w} _ {k} = \exp (\mathbf {r} _ {k}) / \sum_ {k ^ {\prime}, t _ {k ^ {\prime}} = t _ {k}} \exp (\mathbf {r} _ {k ^ {\prime}}),
$$
and biased by a constant 0.5 [13].
C.2.2 Select, Connect, Reduce and Expand
SELECT step. S-EdgePool shares the same computations as in EdgePool to generate learnable edge scores, as detailed above. Then, we use a clustering procedure to determine the subset of nodes to be reduced.
Let  $I_{\mathbf{v}}$  be the identifier of the cluster containing a set of nodes  $\mathbf{v}$ . Initially, we let  $\mathbf{v} = \{v\}$  for every single node  $v$ . A contraction of an edge merges a pair of nodes  $(v, v')$  connected by this edge (where  $v \in \mathbf{v}$ ,  $v' \in \mathbf{v}'$  and  $\mathbf{v} \neq \mathbf{v}'$ ), and thus unifies the cluster identifiers, i.e.,  $I_{\mathbf{v}} = I_{\mathbf{v}'} = I_{\mathbf{v}_{\text{merge}}}$  and  $\mathbf{v}_{\text{merge}} = \mathbf{v} \cup \mathbf{v}'$ . That is, once an edge connecting any pair of nodes from two distinct clusters is contracted, we merge the two clusters and unify their identifiers. Edges are visited sequentially by a decreasing order on the edge scores, and contractions are implemented if valid. We set the maximum size of the node clusters to be a parameter  $\tau_c$ , where  $\tau_c = 2$  degenerates to the case of EdgePool [13]. We further introduce the pooling ratio  $\eta_v$  to control the minimal number of remaining clusters after edge contractions to be  $N^v * \eta_v$ . Contractions that violate the above two constraints are invalid and will be skipped. Both parameters control the number of nodes in the pooled graph. In our implementation, the cluster of nodes is dynamically maintained using the disjoint-set data structure [21].
Then each node cluster  $i$  collapses into a new node  $\tilde{v}$  of the pooled graph (i.e.  $S_{\tilde{v}} = \{v|I_v = i\}$ ), with inter-graph edges connect the nodes in the cluster to the new node  $\tilde{v}$ .
CONNECT step. The CONNECT function rebuilds the edge set  $\hat{\mathcal{E}}$  between the nodes in  $\tilde{\nu}$ . As aforementioned, we build the pooled graph's nodes according to node clusters. We call this mapping function from node clusters to new nodes as  $c2n$ . After that, we build the pooled graph's edges following three steps: First, for all edges in the original graph, we find out the corresponding node cluster(s) of its two endpoints (using a disjoint-set's find index operation). Then, we find out the corresponding new nodes by using the mapping function  $n$ . Last, we add a new edge between the new nodes.
REDUCE and EXPAND step. The REDUCE and EXPAND are generalized from the method mentioned in [13]. The REDUCE function computes new node features and edge features. We follow their method to compute new node features by taking the sum of the node features and multiplying it by the edge score. Specifically, we generalize the computation between two nodes to a node cluster. The node clusters are maintained with a disjoint-set data structure and a cluster  $S_{\tilde{v}}$  consists of  $|S_{\tilde{v}}|$  nodes. We define  $\mathcal{E}_{\tilde{v}}^{ds}$  as a set of  $|S_{\tilde{v}}| - 1$  edges, where the edges are the selected edges to be contracted in the SELECT step. Then,
$$
c _ {\tilde {v}} = \frac {1 + \sum_ {e _ {k} \in \mathcal {E} _ {\tilde {v}} ^ {d s}} \mathbf {w} _ {k}}{| S _ {\tilde {v}} |}
$$
$$
\mathbf {V} _ {\tilde {v}} = c _ {\tilde {v}} \sum_ {v \in S _ {\tilde {v}}} \mathbf {V} _ {v}
$$
To integrate the edge features between two node clusters, we first find all the connected edges between the two node clusters (the edges between node clusters are edges that connect two nodes from different node clusters). Then, we use the sum of all the connected edges' features between the two node clusters as the new edge's features.
The EXPAND function is also referred as unpool operation. It computes node features of the input graph  $\mathbf{V}_v$  given the node features of the pooled graph  $\mathbf{V}_{\tilde{v}}$  as following:
$$
\mathbf {V} _ {v} = \frac {\mathbf {V} _ {\tilde {v}}}{c _ {\tilde {v}}}
$$
C.2.3 Pseudo Code
The pseudo-code includes two parts, where Algorithm 1 describes how to maintain the clusters using a disjoint-set data structure, and Algorithm 2 describes the procedure of S-EdgePool that generates a pooled graph  $\tilde{\mathcal{G}}$  with configurable node pooling ratio  $\eta_v$  and maximum of cluster sizes  $\tau_c$ .
Algorithm 1 Get Cluster Index And Cluster Size of a Node (Using disjoint-set data structure)
function InitializeDisjointSet.graph  $\mathcal{G}(\mathcal{V},\mathcal{E})$    
for  $v\in \mathcal{V}$  do index  $[v] = v$  {the identifier of the cluster the node  $v$  belongs to} end for   
end function   
function FindIndex(node  $v$  1 if index  $[v] = v$  then return  $v$  else index[v]  $\leftarrow$  FindIndex(index[v]) return index[v]   
end if   
end function   
function FindIndexAndSize(node  $v$  1  $i\gets$  FindIndex(v) s  $\leftarrow$  size[i] return i,s   
end function   
function MERGE(cluster index  $x$  ,cluster index  $y$  1 size[y]  $\leftarrow$  size[x] + size[y] index[x]  $\leftarrow$  index[y]   
end function
C.3 GFuN
We first realize the  $\phi_e, \phi_v, \phi_u$  functions in the full GN block (Sec 2.1 and [6]) as neural networks:
$$
\mathbf {E} _ {k} ^ {\prime} = \mathrm {N N} _ {e} \left(\mathbf {E} _ {k}, \mathbf {V} _ {s _ {k}}, \mathbf {V} _ {t _ {k}}, \mathbf {u}\right), \tag {2}
$$
$$
\mathbf {V} _ {i} ^ {\prime} = \mathrm {N N} _ {v} \left(\bar {\mathbf {E}} _ {i} ^ {\prime}, \mathbf {V} _ {i}, \mathbf {u}\right), \tag {3}
$$
$$
\mathbf {u} ^ {\prime} = \mathrm {N N} _ {u} \left(\overline {{\mathbf {E}}} ^ {\prime}, \overline {{\mathbf {V}}} ^ {\prime}, \mathbf {u}\right), \tag {4}
$$
respectively, where
$$
\bar {\mathbf {E}} _ {i} ^ {\prime} = \rho^ {e \rightarrow v} \left(\left\{\mathbf {E} _ {k} ^ {\prime} \right\} _ {k \in [ 1 \dots N ^ {e} ], t _ {k} = i}\right), \tag {5}
$$
$$
\bar {\mathbf {E}} ^ {\prime} = \rho^ {e \rightarrow u} \left(\mathbf {E} ^ {\prime}\right), \tag {6}
$$
$$
\bar {\mathbf {V}} ^ {\prime} = \rho^ {v \rightarrow u} \left(\mathbf {V} ^ {\prime}\right). \tag {7}
$$
We further decompose the neural networks according to the features in the function:
$$
\mathrm {N N} _ {e} \left(\mathbf {E} _ {k}, \mathbf {V} _ {s _ {k}}, \mathbf {V} _ {t _ {k}}, \mathbf {u}\right) = \mathrm {N N} _ {e \leftarrow e} \left(\mathbf {E} _ {k}\right) + \mathrm {N N} _ {e \leftarrow v _ {s}} \left(\mathbf {V} _ {s _ {k}}\right) + \mathrm {N N} _ {e \leftarrow v _ {t}} \left(\mathbf {V} _ {t _ {k}}\right) + \mathrm {N N} _ {e \leftarrow u} (\mathbf {u}), \tag {8}
$$
$$
\mathrm {N N} _ {v} \left(\bar {\mathbf {E}} _ {i} ^ {\prime}, \mathbf {V} _ {i}, \mathbf {u}\right) = \mathrm {N N} _ {v \leftarrow e} \left(\bar {\mathbf {E}} _ {i} ^ {\prime}\right) + \mathrm {N N} _ {v \leftarrow v} \left(\mathbf {V} _ {i}\right) + \mathrm {N N} _ {v \leftarrow u} (\mathbf {u}), \tag {9}
$$
$$
\mathrm {N N} _ {u} \left(\overline {{\mathbf {E}}} ^ {\prime}, \overline {{\mathbf {V}}} ^ {\prime}, \mathbf {u}\right) = \mathrm {N N} _ {u \leftarrow e} \left(\overline {{\mathbf {E}}} ^ {\prime}\right) + \mathrm {N N} _ {u \leftarrow v} \left(\overline {{\mathbf {V}}} ^ {\prime}\right) + \mathrm {N N} _ {u \leftarrow u} (\mathbf {u}) \tag {10}
$$
Algorithm 2 Strided EdgePool
input graph  $\mathcal{G} = (\mathcal{V},\mathcal{E})$  edge scores w, node pooling ratio  $\eta_v$  , maximum cluster sizes  $\tau_{c}$    
output pooled graph  $\tilde{\mathcal{G}} = (\hat{\mathcal{V}},\tilde{\mathcal{E}})$  and inter graph  $\hat{\mathcal{G}} = (\hat{\mathcal{V}},\hat{\mathcal{E}})$  InitializeDisjointSet(G) remains  $\leftarrow N^v\{N^v\}$  is the number of nodes in graph  $\mathcal{G}\}$ $\bar{\mathcal{E}}\gets$  Sort the edges  $\mathcal{E}$  according to the edge scores w decreasingly.   
for  $e\in \bar{\mathcal{E}}$  do  $x,y\gets$  the two endpoints of the edge e  $rx,sx\gets$  FindIndexAndSize(x)  $ry,sy\gets$  FindIndexAndSize(y) if  $rx\neq ry$  and  $(sx + sy\leq \tau_c)$  then Merge(x,y) remains  $\leftarrow$  remains-1 if remains  $\leq N^v *\eta_v$  then break end if end if   
end for   
 $\tilde{\nu},\tilde{\varepsilon},\hat{\nu},\hat{\varepsilon}\gets \{\},\{\},\{\},\{\}$    
create empty mapping  $c2n$  from cluster index to nodes   
for  $v\in \mathcal{V}$  do if FindIndex(v)=  $v$  then create new node  $\tilde{v}$ $c2n[v] = \tilde{v}$ $\tilde{\nu}\gets \tilde{\nu}\cup \{\tilde{\nu}\}$    
end if   
end for   
for  $e\in \mathcal{E}$  do  $x,y\gets$  the two endpoints of the edge e  $\tilde{x}\gets c2n[\mathrm{FindIndex}(x)]$ $\tilde{y}\gets c2n[\mathrm{FindIndex}(y)]$ $\tilde{\mathcal{E}}\gets \tilde{\mathcal{E}}\cup \{(\tilde{x},\tilde{y})\}$    
end for   
for  $v\in \mathcal{V}$  do  $\tilde{v}\gets c2n[\mathrm{FindIndex}(v)]$ $\hat{\mathcal{E}}\gets \hat{\mathcal{E}}\cup \{(v,\tilde{v})\}$    
end for   
 $\hat{\nu}\gets \nu \cup \tilde{\nu}$
However, such GN block uses 10 times the number of parameters as the standard GCN [34] layer when the node, edge and global embedding dimensions are all equivalent. In practice, we disable all computations related to global features  $\mathbf{u}$ , as well as the neural networks  $\mathrm{NN}_{e\leftarrow e}$  and  $\mathrm{NN}_{e\leftarrow v_t}$ . We also set  $\mathrm{NN}_{v\leftarrow e}$  to be Identity.
In practice, we use the summation function as the aggregator function  $\rho^{e\rightarrow v}$  by default. But other choices like MEAN, MAX, gated summation, attention or their combinations can also be used.
Overall, we call such GN block as graph full network (GFuN).
C.4 Encoder and Decoder
Encoder. For input embedding, we use the Linear layer or Embedding layer to embed input features. For example, we follow [15] and use the Linear layer on MNIST and CIFAR10 datasets, and use the Embedding layer on ZINC and AQSOL datasets. For the molecular graph in OGB, we use the same embedding method as in the original work [28]. Besides, we can adopt positional encoding methods like Laplacian [15] and Random Walk [16] to further embed global and local graph structure information. The embedding of positional encoding can be combined into (like concatenation, addition, etc.) input features and form new embeddings.
Decoder. We can freely choose from the multi-scale features computed during the process stage as inputs to the decoder module. Empirically, we use the features on the original graph for prediction in all experiments. For node-level tasks, we apply a last GNN layer on the original graph to get logits for every node. For graph-level tasks, we first use global pooling functions to aggregate features. We can use common global pooling methods like SUM, MEAN, MAX, or their combination. After the global pool, we use MLP layer(s) to generate the prediction.
C.5 Architecture Variants
We can replace some GN blocks within Mee layers as an Identity block to reduce the time complexity. We call the height  $j$  is reserved if the intra GN block of height  $j$  is not replaced by an Identity block. We prefer to reserve an interval of consecutive heights for the Mee layers. (The inter GN blocks between these heights remain unchanged while others are replaced as identities) By varying the heights reserved in each Mee layers, we can create a large number of variants of MeGraph model including U-Shaped, Bridge-Shaped and Staircase-Shaped.
U-Shaped. This variant is similar to Graph U-Net [22]. In this U-Shaped variant, the relationship between the number of layers  $n$  and height  $h$  is  $n = 2h + 1$ , and there is only one GN block in each layer. We keep the GN block at height  $j = i$  for each layer  $i$  at the first half layers and keep the GN block at height  $j = n - i + 1$  for each layer  $i$  at the later half layers. In the middle layer, only the last height  $j = h = (n - 1)/2$  has a GN block.
Bridge-Shaped. In this variant, all GN blocks are combined like an arch bridge. Describe in detail, in the first and last layers, there are GN blocks in each height. In other layers, there are GN blocks at a height of 1 to  $j$  (where  $1 < j < h$ ).
Staircase-Shaped. There are four forms in this variant, and the number of layers  $n$  is equal to the height  $h$  in all forms. The first form is like the 'downward' staircase. In each layer  $i$  of this form, there are GN blocks at the height of  $j$  to  $h$  (where  $j = i$ ). The second form is the inverted first form. In each layer  $i$  of this second form, there are GN blocks at height of 1 to  $h - i + 1$  (where  $j = i$ ). The last two forms are the mirror of the first and second forms.
D Theoretical Discussions
D.1 Smaller Number of Aggregation Steps for Capturing Long-Range Interactions
We rephrase the analysis provided in [47] as following:
We analyze the number of aggregation steps required to capture long-range interactions between nodes in the original graph while assuming the node representation capacity is large enough.
Standard message-passing GNNs require  $n$  aggregation steps to capture long-range interactions of  $n$  hops away, therefore requiring a stack of  $n$  layers, which could be expensive when  $n$  is large.
We also assume the height  $h$  of the hierarchy is large enough so that all nodes of the original graph are pooled into a single node. In that case, the information aggregation along the hierarchy captures all pairs of LRIs into the embedding of the single node. Which means the number of aggregation steps of MeGraph is  $h$ . When we adopt a pooling method that coarsens the graph at least half,  $h$  is at most  $O(\log(|V|))$  where  $|V|$  is the number of nodes of the input graph. Therefore, the height  $h$  is significantly smaller than the diameter of the graph (which could be  $O(|V|)$ ) in most cases.
D.2 MeGraph can degenerate to standard GNNs
MeGraph can learn a gating function (within the X-UPD function) that only reserves the features of the same scale while performing cross-scale information exchanging. In that case, there will be no information exchange across multi-scale graphs, and features other than those in the original scale will not be aggregated. We provide a proof sketch below, while the results of the random pooling method ablation study in Sec. 4.4 also provide empirical evidence.
Proof: The cross update function is  $(X_{j}^{\prime},\hat{X}_{j}^{\prime},X_{j + 1}^{\prime}) = \mathbb{X}\text{-UPD}(j,X_{j},\hat{X}_{j},X_{j + 1})$ . There is a residual function applied here, and we assume it is implemented as a gated residual:  $X_{j}^{\prime \prime} = \sigma (\alpha)X_{j} + \sigma (\beta)X_{j}^{\prime}$ , where  $\sigma$  is the sigmoid function and  $\alpha ,\beta$  are learnable parameters. Theoretically, it is possible that
Table 7: Running time (s) for one epoch on the GNN benchmark. See Sec. E for more implementation details.
ZINC	AQSOL	CIFAR10	MNIST	PATTERN	CLUSTER
Megraph (h=5)	25.69	20.22	336.63	307.23	101.52	69.65
Megraph (h=1)	2.41	1.67	51.74	38.60	9.21	6.52
Table 8: Running time (s) for one epoch on the OGBG datasets. See Sec. E for more implementation details.
molhiv	molbace	molbbbpl	molclintosh	molsider
Megraph (h=5)	393.42	14.70	20.36	14.03	14.12
Megraph (h=1)	22.50	1.43	1.58	1.26	1.41
moltox21	moltoxcast	molesol	molfreesolv	mollipo
Megraph (h=5)	70.15	78.77	11.68	6.24	44.77
Megraph (h=1)	5.27	8.17	0.76	0.41	2.77