Title: MeGraph: Capturing Long-Range Interactions by Alternating Local and Hierarchical Aggregation on Multi-Scaled Graph Hierarchy
Venue: NeurIPS 2023
Accepted: yes
Keywords: Long-Range Interactions

Hierachical Structure

Multi-Scale

Graph Pooling

Graph Neural Networks(GNNs)
Abstract: Graph neural networks, which typically exchange information between local neighbors, often struggle to capture long-range interactions (LRIs) within the graph. Building a graph hierarchy via graph pooling methods is a promising approach to address this challenge; however, hierarchical information propagation cannot entirely take over the role of local information aggregation. To balance locality and hierarchy, we integrate the local and hierarchical structures, represented by intra- and inter-graphs respectively, of a multi-scale graph hierarchy into a single mega graph. Our proposed MeGraph model consists of multiple layers alternating between local and hierarchical information aggregation on the mega graph. Each layer first performs local-aware message-passing on graphs of varied scales via the intra-graph edges, then fuses information across the entire hierarchy along the bidirectional pathways formed by inter-graph edges. By repeating this fusion process, local and hierarchical information could intertwine and complement each other. To evaluate our model, we establish a new Graph Theory Benchmark designed to assess LRI capture ability, in which MeGraph demonstrates dominant performance. Furthermore, MeGraph exhibits superior or equivalent performance to state-of-the-art models on the Long Range Graph Benchmark. The experimental results on commonly adopted real-world datasets further demonstrate the broad applicability of MeGraph.

MeGraph: Capturing Long-Range Interactions by Alternating Local and Hierarchical Aggregation on Multi-Scaled Graph Hierarchy

Abstract

Graph neural networks, which typically exchange information between local neighbors, often struggle to capture long-range interactions (LRIs) within the graph. Building a graph hierarchy via graph pooling methods is a promising approach to address this challenge; however, hierarchical information propagation cannot entirely take over the role of local information aggregation. To balance locality and hierarchy, we integrate the local and hierarchical structures, represented by intra- and inter-graphs respectively, of a multi-scale graph hierarchy into a single mega graph. Our proposed MeGraph model consists of multiple layers alternating between local and hierarchical information aggregation on the mega graph. Each layer first performs local-aware message-passing on graphs of varied scales via the intra-graph edges, then fuses information across the entire hierarchy along the bidirectional pathways formed by inter-graph edges. By repeating this fusion process, local and hierarchical information could intertwine and complement each other. To evaluate our model, we establish a new Graph Theory Benchmark designed to assess LRI capture ability, in which MeGraph demonstrates dominant performance. Furthermore, MeGraph exhibits superior or equivalent performance to state-of-the-art models on the Long Range Graph Benchmark. The experimental results on commonly adopted real-world datasets further demonstrate the broad applicability of MeGraph . 1

1 Introduction

Graph-structured data, such as social networks, traffic networks, and biological data, are prevalent across a plethora of real-world applications. Recently, Graph Neural Networks (GNNs) have emerged as a powerful tool for modeling and understanding the intricate relationships and patterns present in such data. Most existing GNNs learn graph representations by iteratively aggregating information from individual nodes' local neighborhoods through the message-passing mechanism. Despite their effectiveness, these GNNs struggle to capture long-range interactions (LRIs) between nodes in the graph. For instance, when employing a 4-layer vanilla GNN on the 9-node ( A to I ) graph (as shown in Fig. 1), the receptive field of node A is limited to 4-hop neighbors, making the aggregation of information from nodes G , H , and I into node A quite challenging. While GNNs could theoretically incorporate information from nodes n -hops away with n -layers of message passing, this often leads to over-smoothing and over-squashing issues [17, 3] when n is large.

Connect

Select

Reduce

B

E

H

A

D

G

C

F

I

3

2

1

X

𝒉

1

2

3

…

B

E

H

A

D

G

C

F

I

3

2

1

X

IntraEdges

InterEdges

2

One mainstream solution to this problem involves constructing a multi-scale graph hierarchy through graph pooling methods. Previous efforts, such as Graph UNets [22] and HGNet [47], have attempted to broaden the receptive field using this strategy. They downsample and upsample the graph, aggregating information along the hierarchy. However, hierarchical information propagation cannot take over the role of local information aggregation. To illustrate, consider the graph hierarchy depicted in Fig. 1. The information propagated along the hierarchy from node B to nodes D , E , and F tends to be similar since they share the common path B-1-X-2 . However, in the original graph, node B holds different degrees of importance to nodes D , E , and F as they are 2, 1, and 3 hops away respectively.

To balance the importance of locality and hierarchy, we amalgamate the local and hierarchical structures of a multi-scale graph hierarchy into a single mega graph as depicted in Fig. 1, where we refer to the local structure as intra-graph edges and hierarchical structure as inter-graph edges. Based on this mega graph, we introduce our MeGraph model consisting of n Mee layers. Each layer first performs local-aware message-passing on graphs of varied scales via the intra-graph edges and then fuses information across the whole hierarchy along the bidirectional pathways formed by inter-graph edges. This method enables hierarchically fused information to circulate within local structures and allows locally fused information to distribute across the hierarchy. By repeating this fusion process, local and hierarchical information could intertwine and complement each other. Moreover, to support flexible graph pooling ratios when constructing the multi-scale graph hierarchy, we propose a new graph pooling method S-EdgePool that improves from EdgePool [13].

In our experiments, We first evaluate MeGraph 's capability to capture Long Range interactions (LRIs). We establish a Graph Theory Benchmark comprising four tasks related to shortest paths and one related to connected components. MeGraph demonstrates superior performance compared with many competitive baselines. MeGraph also achieves comparable or superior performance than the state-of-the-art on the Long Range Graph Benchmark (LRGB) [17]. In addition, we perform extensive experiments on widely-used real-world datasets that are not explicitly tailored for assessing the capacity to capture LRIs. These include the GNN benchmark [15] and OGB-G datasets [28]. In these tests, MeGraph demonstrates superior or equivalent performance compared to the baseline models, suggesting its broad applicability and effectiveness.

The main contributions of this work are summarized as follows: 1) Mega graph and novel architecture : we propose the mega graph, a multi-scale graph formed by intra- and inter-graph edges, where the message-passing over the mega graph naturally balances locality and hierarchy. On this basis, we introduce a novel architecture MeGraph , which alternates information aggregation along the intraand inter-edges of the mega graph. This fusion process intertwines local and hierarchical information, leading to mutual benefits. 2) Hierarchical information fusion : we design a bidirectional pathway to facilitate information fusion among the hierarchies. 3) S-EdgePool : we enhance EdgePool into S-EdgePool, allowing an adjustable pooling ratio. 4) Benchmark and Evaluations : we establish a new graph theory benchmark to evaluate the ability of models to capture LRIs. In these evaluations, MeGraph exhibits dominant performance. Additionally, MeGraph achieves new SOTA in one task of LRGB and shows better or comparable performance compared with baselines on popular real-world datasets.

2 Notations, Backgrounds and Preliminaries

Let G = ( V , E ) be a graph with node set V (of cardinality N v ) and edge set E (of cardinality N e ). The edge set can be represented as E = { ( s k , t k ) } k =1: N e , where s k and t k are the indices of the source and target nodes connected by edge k . We define X G as features of graph G , which is a combination of global (graph-level) features u G , node features V G , and edge features E G . Accordingly, we use V G i to represent the features of a specific node v i , and E G k denotes the features of a specific edge ( s k , t k ) . We may abuse the notations by omitting the superscript G when there is no context ambiguity.

2.1 Graph Network (GN) Block

We adopt the Graph Network (GN) block design in accordance with the GN framework [6]. In our notation, a GN block accepts a graph G and features X = ( u , V , E ) as inputs, and produces new features X ′ = ( u ′ , V ′ , E ′ ) . A full GN block [6] includes the following update steps. In each of these steps, ϕ denotes an update function, typically implemented as a neural network:

Edge features : E ′ k = ϕ e ( E k , V s k , V t k , u ) , ∀ k ∈ [1 , N e ] .

Node features : V ′ i = ϕ v ( ρ e → v ( { E ′ k } k ∈ [1 ,N e ] ,t k = i ) , V i , u ) , ∀ i ∈ [1 , N v ] , where ρ e → v is an aggregation function taking the features of incoming edges as inputs.

Global features : u ′ = ϕ u ( ρ e → u ( E ′ ) , ρ v → u ( V ′ ) , u ) , where ρ e → u and ρ v → u are two global aggregation functions over edge and node features.

Given a fixed graph structure G and the consistent input and output formats outlined above, GN blocks can be seamlessly integrated to construct complex, deep graph networks.

2.2 Graph Pooling

Graph pooling operation downsamples the graph structure and its associated features while ensuring the preservation of structural and semantic information inherent to the graph. Drawing from the SRC framework [23], we identify graph pooling as a category of functions, POOL , that maps a graph G = ( V , E ) with N v nodes and features X G to a reduced graph ˜ G = ( ˜ V , ˜ E ) with N ˜ v nodes and new features X ˜ G . Here, N ˜ v ≤ N v and ( ˜ G , X ˜ G ) = POOL ( G , X G ) .

The SRC framework deconstructs the POOL operation into SELECT , REDUCE , and CONNECT functions, which encompass most existing graph pooling techniques. We reinterpret these functions in our own notation as follows:

As shown in Fig. 1, the SELECT establishes N ˜ v nodes for the pooled graph, and each node ˜ v corresponds to a subset of nodes S ˜ v ⊆ V in the input graph. This creates an undirected bipartite graph ˆ G = ( ˆ V , ˆ E ) , with ˆ V = V ∪ ˜ V and ( v, ˜ v ) ∈ ˆ E if and only if v ∈ S ˜ v . We refer to this graph ˆ G as the inter-graph , a larger graph that links nodes in the input graph G with nodes in the pooled graph ˜ G . The SELECT function can be generalized to include inter-graph features X ˆ G . As an example, edge weights can be introduced for some edge (ˆ s k , ˆ t k ) in graph ˆ G to gauge the importance of node ˆ s k from the input graph contributing to node ˆ t k in the pooled graph.

The CONNECT function reconstructs the edge set ˜ E between the nodes in ˜ V of the pooled graph ˜ G based on the original edges in E and the inter-graph edges in ˆ E . The REDUCE function calculates the graph features X ˜ G of graph ˜ G by aggregating input graph features X G , taking into account both the inter-graph ˆ G and features X ˆ G . In a similar vein to the relationship between graph lifting and coarsening, we define the EXPAND function for graph features, which serves as the inverse of the REDUCE function: X G = EXPAND ( X ˜ G , ˆ G , X ˆ G ) .

3 Methods

We begin with the introduction of the mega graph (Sec.3.1), which amalgamates the local (intraedges) and hierarchical (inter-edges) structures of a multi-scale graph hierarchy into a single graph.

Depth

h Inputs

Layer 1

Layer 2

Layer 3

Layern

Outs

Graph

1

X

X1

X1

X2

X2

X

Xn

X

Features

X9

Xn

Intra GN

InterGN

2

X2

X2

X2

X2

n.

X2

Pooling

X9

X2

Mee

3

X3

EX

X3

n

X

Layer

X3

X

Readout

Xout

Logits

4

X4

X2

n.

uX

Xn

Last Layer(s)

Encode

Process

Decode

Following this, we present the MeGraph model (Sec.3.2), which alternates between the aggregation of local and hierarchical information along the intra- and inter-edges of the mega graph. We then discuss the specific choices made for the core modules of the MeGraph , along with the innovations (Sec.3.3). Finally, we delve into the computational complexity of the MeGraph model (Sec.3.4).

3.1 Connecting Multi-scale Graphs into a Mega Graph

Similar to the concept of an image pyramid [1], a graph pyramid is constructed by stacking multiscale graphs, which are obtained through iterative downsampling of the graph using a graph pooling technique. Formally, in alignment with the definition of an image feature pyramid [39], we define a graph feature pyramid as a set of graphs G 1: h := {G i } i =1 , ··· ,h and their corresponding features X G 1: h := { X G i } i =1 , ··· ,h . Here, G 1 represents the original graph, X G 1 signifies the initial features, h stands for the height , and ( G i , X G i ) = POOL ( G i -1 , X G i -1 ) for i > 1 .

By iteratively applying the POOL function, we can collect the inter-graphs ˆ G 1: h := { ˆ G i } i =1 , ··· ,h -1 and their features X ˆ G 1: h := { X ˆ G i } i =1 , ··· ,h -1 (since there are h -1 inter-graphs for h intra-graphs), where ( ˆ G i , X ˆ G i ) = SELECT ( G i , X G i ) for i < h . The bipartite inter-graph ˆ G and its features X ˆ G essentially depict the relationships between the graphs before and after the pooling process (see Sec. 2.2).

Finally, as illustrated in Fig. 1, we wire the graph pyramid G 1: h using the edges found in the bipartite graphs ˆ G 1: h . This results in a mega graph MG = ( MV , ME ) , where MV = ⋃ h i =1 V i and ME = ⋃ h i =1 E i ∪ ⋃ h -1 i =1 ˆ E i . The structure of the mega graph would vary as the graph pooling method trains. We denote MG intra = ⋃ h i =1 G i as the intra-graph of MG , and refer to the edges therein as intra-edges. Correspondingly, MG inter = ⋃ h -1 i =1 ˆ G i is referred to as the inter-graph of MG , with its corresponding edges termed as inter-edges. The features X MG of the mega graph MG is a combination of intra-graph features X G 1: h and inter-graph features X ˆ G 1: h .

3.2 Mega Graph Message Passing

We introduce the MeGraph architecture, designed to perform local and hierarchical aggregations over the mega graph alternately. As shown in Fig.2, the architecture follows the encode-process-decode design [6, 25] and incorporates GN blocks (refer to Sec. 2.1) as fundamental building blocks.

During the encode stage, initial features are inputted into an intra-graph GN block, which is followed by a sequence of graph pooling operations to construct the mega graph MG and its associated features

𝑋 1

𝑖 -

𝑋 2

𝑖 -

𝑋 3

𝑖 -

𝑋3

′

𝑋 4

𝑖 -

𝑋4

′

𝑋1

′′

𝑋2

′′

𝑋3

′′

𝑋4

′′

Inter GN

Intra GN

Residual

Same

Inputs

Outputs

Vertical Updates

𝑋𝑗+

𝑋𝑗

𝑋 𝑗

′

𝑋 𝑗 +

′

Cross Update

𝑋1

′

𝑋2

′

𝑋 𝑗

𝑖

Graph

Features

𝒉

1

2

3

4

…

…

……

𝑋1

𝑖

𝑋2

𝑖

𝑋3

𝑖

𝑋4

𝑖

…

…

……

𝑋 1

𝑖 -

𝑋1

′

𝑋1

𝑖

𝑋 𝑗

′

𝑋 2

𝑖 -

𝑋 3

𝑖 -

𝑋2

𝑖

𝑋3

𝑖

𝑋 𝑗

𝑖

Step 1

Step 2

Step 3

𝑋1

𝑋2

′

𝑋2

𝑋1

′′

𝑋1

′

𝑋𝑗

𝑋3

′

𝑋3

𝑋2

′′

𝑋2

𝑋3

′′

𝑋3

′

( X 0 ) MG . In the process stage, the Mee layer, which performs both local and hierarchical information aggregation within the mega graph, is stacked n times. The i -th Mee layer receives ( X i -1 ) MG as input and outputs ( X i ) MG . Through the stacking of Mee layers, a deeper architecture is created, enabling a more profound fusion of local and hierarchical information. Lastly, in the decode stage, the features ( X n ) MG are transformed into task-specific representations using readout functions.

Mee Layer. The Mee layer is designed to aggregate local and hierarchical information within the mega graph. A detailed structure of the Mee layer is depicted in Fig. 3.

For the i -th Mee layer, we consider inputs denoted by ( X i -1 ) MG = { ( X i -1 ) G 1: h , ( X i -1 ) ˆ G 1: h } . For simplicity, we omit the superscript and denote the features of intraand inter-graphs as { X i -1 j } j =1 , ··· ,h := ( X i -1 ) G 1: h and { ˆ X i -1 j } j =1 , ··· ,h -1 := ( X i -1 ) ˆ G 1: h respectively.

The first step applies GN blocks on intra-graph edges, performing message passing on the local structure of graphs at each scale: X ′ j = GN i,j intra ( G j , X i -1 j ) . Here, X ′ j represents the updated intra-graph G j features.

The second and third steps focus on multi-scale information fusion. The second step applies crossupdates across consecutive heights from 1 to h , while the third step reverses the process, forming a bidirectional pathway for the information flow across the hierarchy. The cross-update between consecutive heights j and j +1 is denoted by a function ( X ′ j , ˆ X ′ j , X j +1 ′ ) = X-UPD ( j, X j , ˆ X j , X j +1) . The prime notation indicates the updated value, and residual links [26] are used in practice.

This cross-update can be implemented via an inter-graph convolution with GN i,j inter , referred to as XConv (detailed in App.C.1). Alternatively, it can be realized using the REDUCE and EXPAND operations of POOL (refer to Sec.2.2) by X ′ j +1 = REDUCE ( ˆ G j , ˆ X 0 j , X j ) and X ′ j = EXPAND ( ˆ G j , ˆ X 0 j , X j +1 ) , where ˆ G j is the j -th inter-graph. We denote this implementation as X-Pool .

The Mee layer outputs features { X i j } j =1 , ··· ,h and { ˆ X i j } j =1 , ··· ,h -1 . Residual links [26] can be added from X i -1 j to X i j and from ˆ X i -1 j to ˆ X i j empirically, creating shortcuts that bypass GN blocks in the Mee layer. It's worth noting that the intra and inter GN blocks can share parameters across all heights j to accommodate varying heights, or across all Mee layers to handle varying layer numbers.

3.3 Module Choice and Innovation

MeGraph incorporates two fundamental modules: the graph pooling operator and the GN block. This architecture can accommodate any graph pooling method from the POOL function family (refer to Sec. 2.2). Furthermore, the GN block is not strictly confined to the graph convolution layer found in standard GCN, GIN, or GAT.

Graph Pooling. There are a number of commonly used graph pooling methods, including DiffPool [62], TopKPool [22], EdgePool [13], etc. We opt for EdgePool due to its simplicity, efficiency,

and ability to naturally preserve the graph's connectivity through edge contraction. However, edge contraction is applied only to the edges in a specific maximal matching of the graph's nodes [13], thereby setting a lower limit of 50% to the pooling ratio η v . This constraint implies that a minimum of log 2 N pooling operations is required to reduce a graph of N nodes to a single node. To address this limitation, we propose the Stridden EdgePool (S-EdgePool), which allows for a variable pooling stride.

The principle behind S-EdgePool involves dynamically tracking the clusters of nodes created by the contraction of selected edges. Similar to EdgePool, edges are processed in descending order based on their scores. When an edge is contracted, if both ends do not belong to the same node cluster, the two clusters containing the endpoints of the edge merge. The current edge can be contracted if the resulting cluster contains no more than τ c nodes after this edge's contraction. The iteration stops prematurely once a pooling ratio, η v , is achieved. During pooling, each node cluster is pooled as a new node. When τ c = 2 , S-EdgePool reverts to the original EdgePool. The algorithm's details and pseudocode are available in App. C.2.

For efficiency, we employ the disjoint-set data structure to dynamically maintain the node clusters, which has a complexity of O ( Eα ( E )) , where E is the number of edges and α ( E ) is a function that grows slower than log( E ) [50]. The total time complexity of S-EdgePool is equivalent to EdgePool and is calculated as O ( ED + E log E ) , where D is the embedding size, O ( ED ) from computing edge scores and O ( E log E ) from sorting the edges.

GNblock. The full GN block, introduced in Sec. 2.1, is implemented as a graph full network (GFuN) layer. This layer exhibits a highly configurable within-block structure, enabling it to express a variety of other architectures (see Sec. 4.2 of [6]), like GCN, GIN, GAT, GatedGCN. Thus, modifying the within-block structure of GFuN is akin to plugging in different GNN cores. Further details can be found in App. C.3.

Encoder and decoder. Most preprocessing methods (including positional encodings and graph rewiring), encoding (input embedding) and decoding (readout functions) schemes applicable to GNNs can also be applied to MeGraph . We give implementation details in App. C.4.

3.4 Computational Complexity and Discussion

The overall complexity of the MeGraph model is contingent on the height h , the number of Mee layers n , the chosen modules, and the corresponding hyperparameters. Let D be the embedding size, V the number of nodes, and E the number of edges in the input graph G . The time complexity of S-Edgepool is O ( ED + E log E ) , and that of a GFuN layer is O ( V D 2 + ED ) . Assuming both the pooling ratios of nodes and edges are η , the total time complexity to construct the mega graph MG becomes O (( ED + E log E ) / (1 -η )) , where ∑ h -1 i =0 η i < 1 / (1 -η ) . Similarly, the total time complexity of an Mee layer is O (( V D 2 + ED ) / (1 -η )) . This complexity is equivalent to a typical GNN layer if we consider 1 / (1 -η ) as a constant (for instance, it is a constant of 2 when η = 0 . 5 ).

Theoretically, when using the same number of layers, MeGraph is better at capturing LRIs than standard message-passing GNNs owning to the hierarchical structure (see App. D.1 for details). On the other hand, MeGraph can degenerate into standard message-passing GNNs (see App. D.2 for details), indicating it should not perform worse than them on other tasks.

4 Experiments

Weconduct extensive experiments to evaluate the MeGraph 's ability to capture long-range interactions (LRIs) and its performance in general graph learning tasks.

4.1 Experimental Settings

Baselines. We compare MeGraph model to three baselines as follows: 1) MeGraph h =1 variant does not use the hierarchical structure and falls back to standard GNNs. 2) MeGraph n =1 variant gives up repeating information exchange over the mega graph. 3) Graph U-Nets [22] uses a U-shaped design and only traverses the multi-scale graphs once.

Due to page limits, statistics of the datasets are provided in App. B.1, hyper-parameters are reported in Table 9, and the training and implementation details are reported in App. E.

4.2 Perfomance on LRI Tasks

To test MeGraph 's ability to capture long-range interactions, we establish a Graph Theory Benchmark, of which four tasks related to shortest path distance, i.e. , Single Source Shortest Path (SP ss ), Single Source Single Destination Shortest Path (SP sssd ), Graph Diameter (Diameter) and Eccentricity of nodes (ECC); and 1 task related to connected component, i.e. , Maximum Connected Component of the same color (MCC). To generate diversified undirected and unweighted graphs for each task, we adopt the ten methods used in PNA [12] and add four new methods: cycle graph, pseudotree, SBM, and geographic threshold graphs. The details of the dataset generation can be found in App. B.2.

As depicted in Table 1, the MeGraph model with h =5, n =5 significantly outperforms both the h =1 and n =1 baselines in terms of reducing regression error across all tasks. It is worth noting that even

the h =5, n =1 baseline outperforms the h =1, n =10 baseline, indicating that adopting a multi-scale graph hierarchy is crucial in these tasks. The improvement is also substantial when compared with our reproduced Graph-UNets using S-EdgePool ([ MeGraph ] 0.675 vs. [Graph UNets] 1.581). The improvements are more significant when the size of graphs becomes larger (as shown in Table 2). These results collectively demonstrate the superior ability of MeGraph to capture LRIs.

Furthermore, we evaluated MeGraph model and compared it with other recent methods on the Long Range Graph Benchmark (LRGB) [17] that contains real-world tasks that require capturing LRIs. As depicted in Table 3, the h =9, n =4 variant of MeGraph achieves superior results on the Peptide-func task, and comparable performance on the Peptide-struct task, relative to state-of-the-art models. It is worth noting that the n = 1 variant already surpasses other methods except the recent MLP-Mixer [27] in the Peptide-func task.

4.3 Generality of MeGraph

To verify the generality of MeGraph model, we evaluate MeGraph on widely adopted GNN Benchmark [15], Open Graph Benchmark [28] and TU Dataset [42]. Results on TU Datasets are available in App. F.3. In addition to the standard model that shares hyper-parameters in similar tasks, we also report MeGraph best with specifically tuned hyper-parameters for each task.

GNN Benchmark . We experiment on chemical data (ZINC and AQSOL), image data (MNIST and CIFAR10) and social network data (PATTERN and CLUSTER). As shown in Table 4, MeGraph outperforms the three baselines by a large margin, indicating the effectiveness of repeating both the local and hierarchical information aggregation.

Open Graph Benchmark (OGB) . We choose 10 datasets related to molecular graphs from the graph prediction tasks of OGB. The task of all datasets is to predict some properties of molecule graphs

1

2

3

4

5

6

7

8

9

10

0.60

0.64

0.68

0.72

0.76

0.80

0.84

0.88

0.92

0.96

1.00

TreeCycle

EdgePool h=2

EdgePool h=3

Louvain h=2

Louvain h=3

Random h=2

Random h=3

h=1

1

2

3

4

5

6

7

8

9

10

0.80

0.82

0.84

0.86

0.88

0.90

0.92

0.94

0.96

0.98

1.00

TreeGrid

EdgePool h=2

EdgePool h=3

Louvain h=2

Louvain h=3

Random h=2

Random h=3

h=1

based on their chemical structures. As shown in Table 5, MeGraph outperforms the h =1 baseline by a large margin, suggesting that building a graph hierarchy is also essential in molecule graphs. The performance of MeGraph , n = 1 baseline, and the reproduced Graph U-Nets are comparable. This observation may be because the information obtained from multi-hop neighbors offers only marginal improvements compared to the information aggregated hierarchically.

4.4 Ablation Study

Hierarchy vs. Locality. We study the impact of the height h and the number of Mee layers n on four synthetic datasets introduced in [61], which are BAShape, BACommunity, TreeCycle, and TreeGrid. Each dataset contains one graph formed by attaching multiple motifs to a base graph. The motif can be a 'house'-shaped network (BAShape, BACommunity), six-node cycle (TreeCycle), or 3-by-3 grid (TreeGrid). The task is to identify the nodes of the motifs in the fused graph.

As shown in Fig. 4, we can observe clear improvements in performance as the height h and the number of layers n increase when using EdgePool. Although increasing height shows better improvements, both hierarchy and locality are indispensable in TreeCycle and TreeGrid tasks. In App. F.5, we show that the conclusion also holds on BAShape and BACommunity datasets, except that the accuracy is already saturated with height h = 2 . The significance of integrating locality with hierarchy is also demonstrated in the CLUSTER task, as presented in Table 4. Here, MeGraph reaches 68.6% accuracy, which is markedly higher than the 50.4% accuracy achieved by both the n =1 baseline and Graph-UNets.

Varying the Pooling Method. We varied the graph pooling with two non-learnable pooling methods, Louvain [8] and Random. On the TreeCycle and TreeGrid datasets, as depicted in Fig. 4, Louvain achieves comparable accuracy to EdgePool, while MeGraph using random pooling matches the performance of the h = 1 variant of MeGraph . These observations indicate that: 1) a well-structured hierarchy crafted by suitable graph pooling methods enhances MeGraph 's ability to harness hierarchical information; and 2) despite the disruption from a randomly constructed hierarchy, the MeGraph model effectively taps into the local structure of the original graph (also discussed in App. D.2).

We further studied the impact of the node pooling ratio η v and the maximum cluster size τ c in S-EdgePool by perturbing these parameters. As indicated in Table 1, the best variant ( η v =0.3, τ c =4) achieved a regression error about 3x smaller (0.675) compared to the original EdgePool ( τ c =2 with an error of 2.215). This suggests the benefit of having a flexible pooling stride. Moreover, the mega graph produced by S-EdgePool can vary significantly with different parameters. In App. F.2, we visualized the resulting graph hierarchy to illustrate the difference between different S-EdgePool variants. However, irrespective of the parameter set used, MeGraph consistently outperforms h = 1 baselines. This suggests that MeGraph exhibits robustness against different architectures of the mega graph.

Varying GN Block. We varied the aggregation function of the GN block as attention (w/ ATT) and gated function (w/ GATE). We observe similar results as in Sec. 4.2 and 4.3, verifying the robustness of MeGraph towards different types of GN blocks. Detailed results can be found in Tables 16, 17 and 18 in App. F.6.

Changing cross update function ( X-UPD ). The unpool operation is frequently used by other hierarchical architectures that build upon graph pooling. As illustrated in Table 1, we substituted the X-Conv implementation with the X-Pool implementation of X-UPD , which resulted in a performance decline from 0.675 to 1.203 (smaller is better). This finding suggests that other hierarchical GNNs might also benefit from replacing the unpool operation with a convolution over the inter-graph edges.

Disabling bidirectional pathway. We verify the effectiveness of the bidirectional pathway design by replacing steps 2 and 3 of the Mee layer as a standard message-passing along the inter-graph edges (denoted w/o pw). As shown in Table 1, the performance degrades from 0.675 to 1.065 (smaller is better), which indicates the contribution of the bidirectional pathway.

5 Related Work

Long-Range Interactions (LRIs). Various methods have been proposed to address the issue of LRIs, including making the GNNs deeper [40]. Another way is to utilize attention and gating mechanism, including GAT [52], jumping knowledge (JK) network [59], incorporating Transformer structures [35, 60, 56, 46] and MLP-Mixer [27]. Another line of research focuses on multi-scale graph hierarchy using graph pooling methods [62, 22, 47], or learning representation based on subgraphs [5, 66]. Recently, Long Range Graph Benchmark [17] has been proposed to better evaluate models' ability to capture LRIs.

Feature Pyramids and Multi-Scale Feature Fusion. Multi-scale feature fusion methods on image feature pyramids have been widely studied in computer vision literature, including the U-Net [48], FPN [39], UNet++ [67], and some recent approaches [63, 41, 38, 37]. HRNet [53] is a similar method compared to MeGraph . HRNet alternates between multi-resolution convolutions and multi-resolution fusion by stridden convolutions. However, the above methods are developed for image data. The key difference compared to these approaches is that the multi-scale feature fusion in MeGraph is along the inter-graph edges, which is not as well structured as the pooling operation in image data. For graph networks, the GraphFPN [65] builds a graph feature pyramid according to the image feature pyramid and superpixel hierarchy. It applies GNN layers on the hierarchical graph to exchange information within the graph pyramid. Existing works [22, 20, 47, 30] have also explored similar ideas in graphstructured data. Our approach aligns with the broader concept of multi-scale information fusion, but it is the first method that builds a mega graph using graph pooling operations and alternates local and hierarchical information aggregation.

Graph Pooling Methods. Graph pooling is an important part of hierarchical graph representation learning. There have been some traditional graph pooling methods like METIS [32] in early literature. Recently, many learning-based graph pooling methods have been proposed, including the DiffPool [62], TopKPool [22], SAG pool [36], EdgePool [13], MinCutPool [7], Structpool [64], and MEWISPool [44], etc. In this work, we utilize S-EdgePool improved from EdgePool to build the mega graph, while this module can be substituted with any of the above-mentioned pooling methods.

Graph Neural Network (GNN) Layers. The GNN layer is the core module of graph representation learning models. Typical GNNs include the GCN [34], GraphSage [24], GAT [52, 9], GIN [58], PNA [12]. MeGraph adopts the full GN block [6] by removing part of links in the module as an elementary block, and similarly this can be replaced by any one of the popular GNN blocks.

6 Limitations and Future Work

The MeGraph model suffers from some limitations. The introduced mega graph architecture inevitably increases both the number of trainable parameters and tuneable hyper-parameters. The flexible choices of many modules in MeGraph post burdens on tuning the architecture on specific datasets. For future research, MeGraph encourages new graph pooling methods to yield edge features in addition to node features, when mapping the input graph to the pooled graph. It is also possible to improve MeGraph using adaptive computational steps [49]. Another direction is to apply some expressive but computationally expensive models like Transformers [51] and Neural Logic Machines [14, 57] (only) over the pooled small-sized graphs.

References

Edward H Adelson, Charles H Anderson, James R Bergen, Peter J Burt, and Joan M Ogden. Pyramid methods in image processing. RCA engineer , 29(6):33-41, 1984.

Réka Albert and Albert-László Barabási. Statistical mechanics of complex networks. Reviews of modern physics , 74(1):47, 2002.

Uri Alon and Eran Yahav. On the bottleneck of graph neural networks and its practical implications. In International Conference on Learning Representations , 2020.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Jinheon Baek, Minki Kang, and Sung Ju Hwang. Accurate learning of graph representations with graph multiset pooling. In International Conference on Learning Representations .

Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261 , 2018.

Filippo Maria Bianchi, Daniele Grattarola, and Cesare Alippi. Spectral clustering with graph neural networks for graph pooling. In International Conference on Machine Learning , pages 874-883. PMLR, 2020.

Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment , 2008(10):P10008, 2008.

Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? In International Conference on Learning Representations , 2021.

Chen Cai, Truong Son Hy, Rose Yu, and Yusu Wang. On the connection between mpnn and graph transformer. arXiv preprint arXiv:2301.11956 , 2023.

Ting Chen, Song Bian, and Yizhou Sun. Are powerful graph neural nets necessary? a dissection on graph classification. arXiv preprint arXiv:1905.04579 , 2019.

Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veliˇ ckovi´ c. Principal neighbourhood aggregation for graph nets. Advances in Neural Information Processing Systems , 33:13260-13271, 2020.

Frederik Diehl, Thomas Brunner, Michael Truong Le, and Alois Knoll. Towards graph pooling by edge contraction. In ICML 2019 workshop on learning and reasoning with graph-structured data , 2019.

Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, and Denny Zhou. Neural logic machines. In International Conference on Learning Representations , 2018.

Vijay Prakash Dwivedi, Chaitanya K Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. arXiv preprint arXiv:2003.00982 , 2020.

Vijay Prakash Dwivedi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Graph neural networks with learnable structural and positional representations. In International Conference on Learning Representations , 2021.

Vijay Prakash Dwivedi, Ladislav Rampasek, Mikhail Galkin, Ali Parviz, Guy Wolf, Anh Tuan Luu, and Dominique Beaini. Long range graph benchmark. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022.

Paul Erd˝ os, Alfréd Rényi, et al. On the evolution of random graphs. Publ. Math. Inst. Hung. Acad. Sci , 5(1):17-60, 1960.

Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv:1903.02428 , 2019.

Matthias Fey, Jan-Gin Yuen, and Frank Weichert. Hierarchical inter-message passing for learning on molecular graphs. arXiv preprint arXiv:2006.12179 , 2020.

Bernard A Galler and Michael J Fisher. An improved equivalence algorithm. Communications of the ACM , 7(5):301-303, 1964.

Hongyang Gao and Shuiwang Ji. Graph u-nets. In international conference on machine learning , pages 2083-2092. PMLR, 2019.

Daniele Grattarola, Daniele Zambon, Filippo Maria Bianchi, and Cesare Alippi. Understanding pooling in graph neural networks. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

Jessica B Hamrick, Kelsey R Allen, Victor Bapst, Tina Zhu, Kevin R McKee, Joshua B Tenenbaum, and Peter W Battaglia. Relational inductive bias for physical construction in humans and machines. arXiv preprint arXiv:1806.01203 , 2018.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Xiaoxin He, Bryan Hooi, Thomas Laurent, Adam Perold, Yann LeCun, and Xavier Bresson. A generalization of vit/mlp-mixer to graphs, 2022.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems , 33:22118-22133, 2020.

Yuanming Hu, Tzu-Mao Li, Luke Anderson, Jonathan Ragan-Kelley, and Frédo Durand. Taichi: a language for high-performance computation on spatially sparse data structures. ACM Transactions on Graphics (TOG) , 38(6):201, 2019.

Truong Son Hy and Risi Kondor. Multiresolution equivariant graph variational autoencoder. Machine Learning: Science and Technology , 4(1):015031, 2023.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning , pages 448-456. PMLR, 2015.

George Karypis and Vipin Kumar. A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM Journal on scientific Computing , 20(1):359-392, 1998.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster) , 2015.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. 2016.

Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral attention. Advances in Neural Information Processing Systems , 34:21618-21629, 2021.

Junhyun Lee, Inyeop Lee, and Jaewoo Kang. Self-attention graph pooling. In International conference on machine learning , pages 3734-3743. PMLR, 2019.

Xiangtai Li, Houlong Zhao, Lei Han, Yunhai Tong, Shaohua Tan, and Kuiyuan Yang. Gated fully fusion for semantic segmentation. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 11418-11425, 2020.

Di Lin, Dingguo Shen, Siting Shen, Yuanfeng Ji, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. Zigzagnet: Fusing top-down and bottom-up context for object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7490-7499, 2019.

Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2117-2125, 2017.

Meng Liu, Hongyang Gao, and Shuiwang Ji. Towards deeper graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages 338-348, 2020.

Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia. Path aggregation network for instance segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 8759-8768, 2018.

Christopher Morris, Nils M Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. arXiv preprint arXiv:2007.08663 , 2020.

Nhat Khang Ngo, Truong Son Hy, and Risi Kondor. Multiresolution graph transformers and wavelet positional encoding for learning long-range and hierarchical structures. The Journal of Chemical Physics , 159(3), 2023.

Amirhossein Nouranizadeh, Mohammadjavad Matinkia, Mohammad Rahmati, and Reza Safabakhsh. Maximum entropy weighted independent set pooling for graph neural networks. arXiv preprint arXiv:2107.01410 , 2021.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Ladislav Rampášek, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a general, powerful, scalable graph transformer. arXiv preprint arXiv:2205.12454 , 2022.

Ladislav Rampášek and Guy Wolf. Hierarchical graph neural nets can capture long-range interactions. In 2021 IEEE 31st International Workshop on Machine Learning for Signal Processing (MLSP) , pages 1-6. IEEE, 2021.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention , pages 234-241. Springer, 2015.

Hao Tang, Zhiao Huang, Jiayuan Gu, Bao-Liang Lu, and Hao Su. Towards scale-invariant graph-related problem solving by iterative homogeneous gnns. Advances in Neural Information Processing Systems , 33:15811-15822, 2020.

Robert E Tarjan and Jan Van Leeuwen. Worst-case analysis of set union algorithms. Journal of the ACM (JACM) , 31(2):245-281, 1984.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations , 2018.

Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning for visual recognition. IEEE transactions on pattern analysis and machine intelligence , 43(10):3349-3364, 2020.

Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis, Jinyang Li, and Zheng Zhang. Deep graph library: A graph-centric, highly-performant package for graph neural networks. arXiv preprint arXiv:1909.01315 , 2019.

Duncan J WATTS. Networks, dynamics and the small world phenomenon. American Journal of Sociology , 105(2):50-59, 2003.

Zhanghao Wu, Paras Jain, Matthew Wright, Azalia Mirhoseini, Joseph E Gonzalez, and Ion Stoica. Representing long-range context for graph neural networks with global attention. Advances in Neural Information Processing Systems , 34:13266-13279, 2021.

Guangxuan Xiao, Leslie Pack Kaelbling, Jiajun Wu, and Jiayuan Mao. Efficient training and inference of hypergraph reasoning networks, 2022.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations , 2018.

Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In International conference on machine learning , pages 5453-5462. PMLR, 2018.

Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems , 34:28877-28888, 2021.

Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. Gnnexplainer: Generating explanations for graph neural networks. Advances in neural information processing systems , 32, 2019.

Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. Advances in neural information processing systems , 31, 2018.

Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 24032412, 2018.

Hao Yuan and Shuiwang Ji. Structpool: Structured graph pooling via conditional random fields. In Proceedings of the 8th International Conference on Learning Representations , 2020.

Gangming Zhao, Weifeng Ge, and Yizhou Yu. Graphfpn: Graph feature pyramid network for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2763-2772, 2021.

Lingxiao Zhao, Wei Jin, Leman Akoglu, and Neil Shah. From stars to subgraphs: Uplifting any gnn with local structure awareness. In International Conference on Learning Representations .

Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang. Unet++: A nested u-net architecture for medical image segmentation. In Deep learning in medical image analysis and multimodal learning for clinical decision support , pages 3-11. Springer, 2018.

Appendix

Table of Contents

A Code and Reproducibility

The code along with the configuration of hyper-parameters to reproduce our experiments can be found at https://github.com/dhh1995/MeGraph .

We set the random seed as 2022 for all experiments to enable reproducible results. We provide dataset statistics in Table 6 and details for the proposed graph theory benchmark in Appendix B.2. Details of the hyper-parameters are reported in Table 9. Configuration of all hyper-parameters and the command lines to reproduce the experiments have been included in the code repository.

B Dataset Details

B.1 Dataset Statistics and Metrics

We provide the statistics of all datasets used in our experiments in Table 6 and introduce the evaluation metrics for each dataset.

For Synthetic datasets, we use classification accuracy (ACC) as the evaluation metric. We use Mean Square Error (MSE) as the evaluation metric for all datasets in our Graph Theory Benchmark. For GNN Benchmark, we follow the original work [15] for evaluation, i.e., Mean Absolute Error (MAE) for ZINC and AQSOL, classification accuracy for MNIST and CIFAR10, and balanced classification accuracy for PATTERN and CLUSTER. For OGB Benchmark, we follow the original work [28] and use the ROC-AUC for classification tasks and Root Mean Square Error (RMSE) for regression tasks. For TU datasets, we follow the setting used by [11] and use classification accuracy as the evaluation metric.

B.2 Graph Theory Benchmark

In this section, we provide the details about the tasks and how the graph features and the labels are generated given a base graph G = ( V , E ) :

Single source single destination shortest path (SP sssd ): a source node s ∈ V and a destination node t ∈ V are selected uniform randomly. The feature of each node v contains three numbers: ( 1 , whether the node v is s , whether the node v is t ). The label of a graph is the length of the shortest path from s to t .

A maximum connected component of the same color (MCC): each node of the graph is colored with one of three colors. The feature for each node is the one-hot representation of its color. The label of graph is the size of the largest connected component of the same color for each color.

Graph diameter (Diameter): the label of the graph is the diameter of the graph. The diameter of a graph G is the maximum of the set of shortest path distances between all pairs of nodes in the graph. The feature of each node is a uniform number 1 .

Single source shortest path (SP ss ): a source node s is selected uniformly randomly. The feature of each node contains two numbers: ( 1 , whether the node is s ). The label of each node is the length of the shortest path from s to this node.

Graph eccentricity (ECC): the label of each node v is node's eccentricity in the graph, which is the maximum distance from v to the other nodes. The feature of each node is a uniform number 1 .

For each task and graph generation method, We generate the dataset by the following steps:

Sample N (number of nodes) from an interval with a total number of graphs. These numbers can be configured. For the medium size setting, the interval is [20 , 50] , a total of 300 graphs. For the large size setting, the interval is [100 , 200] , a total of 500 graphs.

Use the graph generation method to generate a graph of N nodes.

Create graph features and labels according to the task.

We then provide the details about the random graph generation methods we used to create our Graph Theory datasets.

Following [12], we continue to use undirected and unweighted graphs from a wide variety of types. We inherit their 10 random graph generation methods and quote their descriptions here for completeness (the percentage after the name is the approximate proportion of such graphs in the mixture setting).

Erdös-Rényi (ER) (20%) [18]: with a probability of presence for each edge equal to p , where p is independently generated for each graph from U [0 , 1]

Barabási-Albert (BA) (20%) [2]: the number of edges for a new node is k , which is taken randomly from { 1 , 2 , . . . , N -1 } for each graph

Grid (5%): m × k 2d grid graph with N = mk and m and k as close as possible

Caveman (5%) [55]: with m cliques of size k , with m and k as close as possible

Tree (15%): generated with a power-law degree distribution with exponent 3

Ladder graphs (5%)

Line graphs (5%)

Star graphs (5%)

Caterpillar graphs (10%): with a backbone of size b (drawn from U [1 , N ) ), and N -b pendent vertices uniformly connected to the backbone

Lobster graphs (10%): with a backbone of size b (drawn from U [1 , N ) ), p (drawn from U [1 , N -b ] ) pendent vertices uniformly connected to the backbone, and additional N -b -p pendent vertices uniformly connected to the previous pendent vertices.

Additional, we add three more graph generation methods:

· Cycle graphs

Pseudotree graphs : A tree graph plus an additional edge. The graph is generated by first generating a cycle graph of size m = sample(0 . 3 N, 0 . 6 N ) . Then n -m remaining nodes are sampled to m parts, where i -th part represents the size of the tree hanging on the i -th node on the cycle. The trees are randomly generated with the given size.

Stochastic Block Model (SBM) graphs : graphs with clusters. We randomly sample the size of each block to be random from (5, 15), and the probability of edge within the block to be random from (0.3, 0.5) and those for other edges to be random from (0.005, 0.01). To make all the tasks well-defined, we filtered out the unconnected graphs during the generation.

Geographic (Geo) graphs : geographic threshold graphs, but with added edges via a minimum spanning tree algorithm, to ensure all nodes are connected. This graph generation method is introduced by [6] in their codebase 2 . We use the geographic threshold θ = 200 instead of the default value θ = 1000 .

Note that we do not have randomization after the graph generation as in [12]. Therefore, very long diameter is preserved for some type of graphs.

C Method Details

C.1 Cross Update Function

The cross update function ( X ′ j , ˆ X ′ j , X ′ j +1 ) = X-UPD ( j, X j , ˆ X j , X j +1 ) perform information exchange in consecutive hierarchies.

The X-Conv realization contains the following steps:

Merge the node features of X j and X j +1 with the inter-graph feature ˆ X j , results in ¯ X j .

Apply GN blocks on inter-graph ˆ G j : ˆ X ′ j = GN i,j inter ( ˆ G j , ¯ X j ) .

Retrieve X ′ j and X ′ j +1 from the node features of inter-graph features ˆ X ′ j .

C.2 S-EdgePool

In this subsection, we introduce the details of S-EdgePool. We first introduce the score generation method, then give details about the SELECT , CONNECT , REDUCE and EXPAND functions, and lastly provide pseudocode of the algorithm.

C.2.1 Edge Score Generation

Both S-EdgePool and EdgePool methods compute a raw edge score r k for each edge k using a linear layer:

where s k and t k are the source and target nodes of edge k , V is node features, E is edge features, W and b are learned parameters. The raw edge scores are further normalized by a local softmax function over all edges of a node:

and biased by a constant 0 . 5 [13].

C.2.2 Select, Connect, Reduce and Expand

SELECT step. S-EdgePool shares the same computations as in EdgePool to generate learnable edge scores, as detailed above. Then, we use a clustering procedure to determine the subset of nodes to be reduced.

̸

Let I v be the identifier of the cluster containing a set of nodes v . Initially, we let v = { v } for every single node v . A contraction of an edge merges a pair of nodes ( v, v ′ ) connected by this edge (where v ∈ v , v ′ ∈ v ′ and v = v ′ ), and thus unifies the cluster identifiers, i.e., I v = I v ′ = I v merge and v merge = v ∪ v ′ . That is, once an edge connecting any pair of nodes from two distinct clusters is contracted, we merge the two clusters and unify their identifiers. Edges are visited sequentially by a decreasing order on the edge scores, and contractions are implemented if valid. We set the maximum size of the node clusters to be a parameter τ c , where τ c = 2 degenerates to the case of EdgePool [13]. We further introduce the pooling ratio η v to control the minimal number of remaining clusters after edge contractions to be N v ∗ η v . Contractions that violate the above two constraints are invalid and will be skipped. Both parameters control the number of nodes in the pooled graph. In our implementation, the cluster of nodes is dynamically maintained using the disjoint-set data structure [21].

Then each node cluster i collapses into a new node ˜ v of the pooled graph (i.e. S ˜ v = { v | I v = i } ), with inter-graph edges connect the nodes in the cluster to the new node ˜ v .

CONNECT step. The CONNECT function rebuilds the edge set ˜ E between the nodes in ˜ V . As aforementioned, we build the pooled graph's nodes according to node clusters. We call this mapping function from node clusters to new nodes as c2n . After that, we build the pooled graph's edges following three steps: First, for all edges in the original graph, we find out the corresponding node cluster(s) of its two endpoints (using a disjoint-set's find index operation). Then, we find out the corresponding new nodes by using the mapping function n . Last, we add a new edge between the new nodes.

REDUCE and EXPAND step. The REDUCE and EXPAND are generalized from the method mentioned in [13]. The REDUCE function computes new node features and edge features. We follow their method to compute new node features by taking the sum of the node features and multiplying it by the edge score. Specifically, we generalize the computation between two nodes to a node cluster. The node clusters are maintained with a disjoint-set data structure and a cluster S ˜ v consists of | S ˜ v | nodes. We define E ds ˜ v as a set of | S ˜ v | -1 edges, where the edges are the selected edges to be contracted in the SELECT step. Then,

To integrate the edge features between two node clusters, we first find all the connected edges between the two node clusters (the edges between node clusters are edges that connect two nodes from different node clusters). Then, we use the sum of all the connected edges' features between the two node clusters as the new edge's features.

The EXPAND function is also referred as unpool operation. It computes node features of the input graph V v given the node features of the pooled graph V ˜ v as following:

C.2.3 Pseudo Code

The pseudo-code includes two parts, where Algorithm 1 describes how to maintain the clusters using a disjoint-set data structure, and Algorithm 2 describes the procedure of S-EdgePool that generates a pooled graph ˜ G with configurable node pooling ratio η v and maximum of cluster sizes τ c .

C.3 GFuN

We first realize the ϕ e , ϕ v , ϕ u functions in the full GN block (Sec 2.1 and [6]) as neural networks:

We further decompose the neural networks according to the features in the function:

respectively, where

̸

However, such GN block uses 10 times the number of parameters as the standard GCN [34] layer when the node, edge and global embedding dimensions are all equivalent. In practice, we disable all computations related to global features u , as well as the neural networks NN e ← e and NN e ← v t . We also set NN v ← e to be Identity.

In practice, we use the summation function as the aggregator function ρ e → v by default. But other choices like MEAN, MAX, gated summation, attention or their combinations can also be used.

Overall, we call such GN block as graph full network (GFuN).

C.4 Encorder and Decoder

Encoder . For input embedding, we use the Linear layer or Embedding layer to embed input features. For example, we follow [15] and use the Linear layer on MNIST and CIFAR10 datasets, and use the Embedding layer on ZINC and AQSOL datasets. For the molecular graph in OGB, we use the same embedding method as in the original work [28]. Besides, we can adopt positional encoding methods like Laplacian [15] and Random Walk [16] to further embed global and local graph structure information. The embedding of positional encoding can be combined into (like concatenation, addition, etc.) input features and form new embeddings.

Decoder . We can freely choose from the multi-scale features computed during the process stage as inputs to the decoder module. Empirically, we use the features on the original graph for prediction in all experiments. For node-level tasks, we apply a last GNN layer on the original graph to get logits for every node. For graph-level tasks, we first use global pooling functions to aggregate features. We can use common global pooling methods like SUM, MEAN, MAX, or their combination. After the global pool, we use MLP layer(s) to generate the prediction.

C.5 Architecture Variants

We can replace some GN blocks within Mee layers as an Identity block to reduce the time complexity. We call the height j is reserved if the intra GN block of height j is not replaced by an Identity block. We prefer to reserve an interval of consecutive heights for the Mee layers. (The inter GN blocks between these heights remain unchanged while others are replaced as identities) By varying the heights reserved in each Mee layers, we can create a large number of variants of MeGraph model including U-Shaped, Bridge-Shaped and Staircase-Shaped.

U-Shaped . This variant is similar to Graph U-Net [22]. In this U-Shaped variant, the relationship between the number of layers n and height h is n = 2 h +1 , and there is only one GN block in each layer. We keep the GN block at height j = i for each layer i at the first half layers and keep the GN block at height j = n -i +1 for each layer i at the later half layers. In the middle layer, only the last height j = h = ( n -1) / 2 has a GN block.

Bridge-Shaped . In this variant, all GN blocks are combined like an arch bridge. Describe in detail, in the first and last layers, there are GN blocks in each height. In other layers, there are GN blocks at a height of 1 to j (where 1 < j < h ).

Staircase-Shaped . There are four forms in this variant, and the number of layers n is equal to the height h in all forms. The first form is like the 'downward' staircase. In each layer i of this form, there are GN blocks at the height of j to h (where j = i ). The second form is the inverted first form. In each layer i of this second form, there are GN blocks at height of 1 to h -i +1 (where j = i ). The last two forms are the mirror of the first and second forms.

D Theoretical Discussions

D.1 Smaller Number of Aggregation Steps for Capturing Long-Range Interactions

We rephrase the analysis provided in [47] as following:

We analyze the number of aggregation steps required to capture long-range interactions between nodes in the original graph while assuming the node representation capacity is large enough.

Standard message-passing GNNs require n aggregation steps to capture long-range interactions of n hops away, therefore requiring a stack of n layers, which could be expensive when n is large.

We also assume the height h of the hierarchy is large enough so that all nodes of the original graph are pooled into a single node. In that case, the information aggregation along the hierarchy captures all pairs of LRIs into the embedding of the single node. Which means the number of aggregation steps of MeGraph is h . When we adopt a pooling method that coarsens the graph at least half, h is at most O (log( | V | )) where | V | is the number of nodes of the input graph. Therefore, the height h is significantly smaller than the diameter of the graph (which could be O ( | V | ) ) in most cases.

D.2 MeGraph can degenerate to standard GNNs

MeGraph can learn a gating function (within the X-UPD function) that only reserves the features of the same scale while performing cross-scale information exchanging. In that case, there will be no information exchange across multi-scale graphs, and features other than those in the original scale will not be aggregated. We provide a proof sketch below, while the results of the random pooling method ablation study in Sec. 4.4 also provide empirical evidence.

Proof : The cross update function is ( X ′ j , ˆ X ′ j , X ′ j +1 ) = X-UPD ( j, X j , ˆ X j , X j +1 ) . There is a residual function applied here, and we assume it is implemented as a gated residual: X ′′ j = σ ( α ) X j + σ ( β ) X ′ j , where σ is the sigmoid function and α, β are learnable parameters. Theoremetically, it is possible that

σ ( α ) = 1 and σ ( β ) = 0 after training. In that case, X ′′ j = X j , which means X j is not changed over steps 2 and 3 of the Mee layer. Therefore, X i 0 = GN i, 0 intra ( G , X i -1 0 ) , this is equivalent to a simple GNN layer that X i = GNN i ( G , X i -1 ) as X i 0 is the features of the original graph and GN intra is a GNN layer. Therefore, MeGraph degenerates to standard message-passing GNNs in this case. ■

E Implementation and Training Details

We use PyTorch [45] and Deep Graph Library (DGL) [54] to implement our method.

We implement S-EdgePool using DGL, extending from the original implementation of EdgePool in the Pytorch Geometric library (PYG) [19]. We did Constant optimization over the implementation to speed up the training and inference of the pooling. We further use Taichi-Lang [29] to speed up the dynamic node clustering process of S-EdgePool. The practical running time of MeGraph model with height h > 1 after optimization is about 2 h times as the h = 1 baseline. This is still slower than the theoretical computational complexity due to the constant in the implementation and the difficulty of paralleling the sequential visitation of edges (according to their scores) in the EdgePool and S-EdgePool. This process could be further speed up by implementing the operations with the CUDA library. We provide the practical running time for h > 1 and h = 1 in GNN benchmark and OGB-G datasets in Tables 7 and 8. The speed is slower than the theoretical one partially due to the pooling ratio of edges being larger than the nodes, making the number of edges decrease slowly over the hierarchy. To further speed up, we could use the variants of MeGraph introduced in App. C.5 by skipping some computation modules.

We run all our experiments on V100 GPUs and M40 GPUs. For training the neural networks, we use Adam [33] as the optimizer. We report the hyper-parameters of the Megraph in Table 9.

For models using GFuN layer as the core GN block, we find it benefits from using layer norms [4]. However, for models using GCN layer as the core GN block, we find it performs best when using batch norms [31].

F Additional Experiment Results

F.1 Experimental Protocol

We evaluate MeGraph on public real-world graph benchmarks. To fairly compare MeGraph with the baselines, we use the following experimental protocols. We first report the public baseline results and our reproduced standard GCN's results. We then replace GCN layers with GFuN layers (which is equivalent to MeGraph ( h = 1 )) to serve as another baseline. We tune the hyper-parameters (such as learning rate, dropout rate and the readout global pooling method, etc.) of MeGraph ( h = 1 ) and choose the best configurations. We then run other diversely configured MeGraph candidates by tuning

other hyper-parameters that only matters for h > 1 , and these hyper-parameters are referred to as the MeGraph hyper-parameters. Detailed configurations are put in Table 9 in App. E.

F.2 Visualization

We plotted the graph hierarchies discovered by MeGraph in the shortest path tasks of Graph Theory Benchmark. In Figure 5, the S-EdgePool with η v = 0 . 3 , τ c = 4 well preserves the structure of the graph after pooling, while the S-EdgePool with η v = 0 . 3 (no cluster size limit) sometimes pooled too many nodes together, breaking the graph structure. The former S-EdgePool leads to better performance as indicated in Table 1. We also plot the hierarchy for SBM-generated graphs in Figure 6, indicating that EdgePool can handle graphs that naturally contains clusters of different size.

F.3 Other Real-Wrold Datasets

TU dataset consists of over 120 datasets of varying sizes from a wide range of applications. We choose 10 datasets, 5 of which are molecule datasets (MUTAG, NCI1, PROTEINS, D&D and ENZYMES) and the other 5 are social networks (IMDB-B, IMDB-M, REDDIT-BINARY, REDDITMULTI-5K and REDDIT-MULTI-12K). They are all graph classification tasks. For more details of each dataset, please refer to the original work [42].

Our Megraph uses the same network structure and hyper-parameters for the same type of dataset. As shown in Table 10, our Megraph achieves about 1% absolute gain than the h = 1 Baselines.

F.4 GFuN

We show our GFuN results on real-world datasets compared to our reproduced GCN in Table 11, 12 and 13. Both GCN and GFuN have the same hyper-parameters except the batch norm for GCN and layer norm for GFuN as stated in Appendix E.

1 0

2

3 4 5

6

7

8

9

10

11

12

13

14

15 16

17

18

19 20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

40 39

41

42

43

45 44

46

47

48

49

50

51

52

53 54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97 98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

160 159

161

162

163

164

165

166

167

168

169

170

171

172

0

1

2

3

4

5 6

7

8

9

1112 10

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50 51

52

53

54

55

56

57

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

0

1

2

3

4

5

6

1 0

2

3 4 5

6

7

8

9

10

11

12

13

14

15 16

17

18

19 20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

40 39

41

42

43

45 44

46

47

48

49

50

51

52

53 54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97 98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

160 159

161

162

163

164

165

166

167

168

169

170

171

172

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

0

1

2

3

0 1 23

5 4 6

7

8

10 11 9

12

13

15 14

16

17

18

19

20

21

22 23

25 24

26

27 28 9

30

31

32

33

34

35

36

37 38

39

40

41

42

43

44

45

46 47

48

49

50

51

52

53

54

55

56

57

58 59

61 60

62

63

64 5

66

67

6869 70

71

72

73

77 76 75 74

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

4 95

96

97

98

99

100

101

102

103

104

106 105

107

108

109

110 111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

127 126

8

129

130

131

132

133

134

135

136

138 137

139

140

141

142

143

144

145 146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168 169

170 171

172

173

174

175

176 177

178

179

180

181

182

183

184

185 186

187

0

1

2

3

4 5

6

8 7

9

10

11

12

13

14

15

17 16

18

19

20

21

22

23

24

25

26

27 28

29

30

31

32

33

34

35

36

37

39 38

41 40

42

43

44

45

46 47

48

49

50

51 52

53

54

55

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

0

1

2

3

4

5

0 1 23

5 4 6

7

8

10 11 9

12

13

15 14

16

17

18

19

20

21

22 23

25 24

26

27 28 9

30

31

32

33

34

35

36

37 38

39

40

41

42

43

44

45

46 47

48

49

50

51

52

53

54

55

56

57

58 59

61 60

62

63

64 5

66

67

6869 70

71

72

73

77 76 75 74

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

4 95

96

97

98

99

100

101

102

103

104

106 105

107

108

109

110 111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

127 126

8

129

130

131

132

133

134

135

136

138 137

139

140

141

142

143

144

145 146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168 169

170 171

172

173

174

175

176 177

178

179

180

181

182

183

184

185 186

187

0

1

2

34

5

6

7 8

9

10

11

12

13

14

15

16 17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

33 32

34

35

36 37

38

39

40

4142

43 44

45

46

47

48

49

50

51

52

53

54

55

0

1

2

3

4 5

6

7

8

9

10

1112

13

14

15

0

1

2

3

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29 30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29 30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

0

1

2

3

4

5

6

7

8

9

10

F.5 Synthetic Datasets

Figures 7 and 8 show the influence of the height h and the number of Mee layers n for MeGraph model on the BAShape and BACommunity datasets. The trends are similar while less significant on easier datasets BAShape and BACommunity.

F.6 Varying GN block

We vary the aggregation function of the GN block as attention (w/ ATT) and gated function (w/ GATE). We observe similar results as in Sec. 4.2 and 4.3, verifying the robustness of MeGraph over different GN blocks. Results are shown in Tables 16, 17 and 18.

1

2

3

4

5

6

7

8

9

10

0.80

0.84

0.88

0.92

0.96

1.00

TreeCycle

height=1

height=2

3

4

5

6

1

2

3

4

5

6

7

8

9

10

0.90

0.92

0.94

0.96

0.98

1.00

TreeGrid

height=1

height=2

3

4

5

6

1

2

3

4

5

6

7

8

9

10

0.980

0.984

0.988

0.992

0.996

1.000

BAShape

height=1

height=2

3

4

5

6

1

2

3

4

5

6

7

8

9

10

0.95

0.96

0.97

0.98

0.99

1.00

BACommunity

height=1

height=2

3

4

5

6

F.7 Graph Theory Dataset

We provide a list of tables (from Table 19 to 30) showing the individual results of Table 1 for each possible graph generation method. Each table contains a list of variants of models and 5 tasks. Some graph generation methods and task combinations are trivial so we filter them out.

Category

Model

MCC

ECC

mix

ER

Star

mix

ER

Baselines

(

h

=1)

n

=1

3.46

±

0

.

211

2.91

±

0

.

206

0.015

±

0

.

004

0.144

±

0

.

031

0.316

±

0

.

003

0.346

±

0

.

006

n

=5

3.29

±

0

.

261

3.35

±

0

.

205

0.014

±

0

.

003

0.078

±

0

.

021

0.228

±

0

.

008

0.289

±

0

.

008

n

=10

3.51

±

0

.

323

3.53

±

0

.

375

0.018

±

0

.

006

0.065

±

0

.

005

0.212

±

0

.

008

0.414

±

0

.

102

h

=5)

EdgePool(

τ

c

=2)

n

=1

1.25

±

0

.

167

0.749

±

0

.

058

0.018

±

0

.

005

0.135

±

0

.

055

0.150

±

0

.

011

0.320

±

0

.

071

n

=5

1.11

±

0

.

143

0.723

±

0

.

073

0.017

±

0

.

005

0.052

±

0

.

017

0.125

±

0

.

010

0.345

±

0

.

064

MeGraph

S-EdgePool

Variants

(

h

=5,

n

=5)

τ

c

=3

1.07

±

0

.

034

0.714

±

0

.

039

0.017

±

0

.

002

0.072

±

0

.

016

0.137

±

0

.

013

0.232

±

0

.

035

η

v

=

0

.

3

0.908

±

0

.

153

0.627

±

0

.

090

0.026

±

0

.

007

0.125

±

0

.

026

0.128

±

0

.

014

0.248

±

0

.

012

η

v

=

0

.

3

,

τ

c

=4

1.10

±

0

.

085

0.709

±

0

.

092

0.019

±

0

.

004

0.073

±

0

.

012

0.129

±

0

.

009

0.224

±

0

.

053

η

v

=

0

.

5

,

τ

c

=4

1.12

±

0

.

219

0.722

±

0

.

128

0.026

±

0

.

008

0.058

±

0

.

010

0.147

±

0

.

017

0.219

±

0

.

042

η

v

=

0

.

3

,

τ

c

=4 (X-Pool)

1.01

±

0

.

166

0.838

±

0

.

078

0.029

±

0

.

007

0.107

±

0

.

021

0.119

±

0

.

008

0.213

±

0

.

027

η

v

=

0

.

3

,

τ

c

=4 (w/o pw)

1.13

±

0

.

059

0.622

±

0

.

073

0.019

±

0

.

003

0.075

±

0

.

01

5

0.126

±

0

.

016

0.307

±

0

.

062

Graph-UNets

h

=

5

,

n

=

9

,

η

v

=

0

.

3

,

τ

c

=

4

1.06

±

0

.

171

0.859

±

0

.

092

0.041

±

0

.

007

0.057

±

0

.

010

0.153

±

0

.

012

0.34 5

±

0

.

133

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel methodology for capturing long-range interactions in graph-based learning models, that is based on mult-scale graph construction and merging. Specifically, following the Select-Reduce-Connect framework [1], the authors introduce a novel pooling method called Edgepool  that enables refinement of the original graph structure in an arbitrary continuum. They then perform graph pooling in order to create multi-scale graph instances of the original graph, and merge the inter-graph and intra-graph connections through the proposed MeGraph framework. \n\nThe authors perform extensive experimentation to over 10 datasets. They, also, show a limited (compared with the number of trainable parameters) ablation study on four synthetic datasets. They, also, propose a rather novel benchmark, called Graph Theory Benchmark (testing properties: graph diameter, single source shortest path, graph eccentricity, maximum connected component), where they evaluate the prediction capabilities of their model. The results suggest a solid performance of MeGraph framework for tasks that require both shallow and deeper architectures.\n\n[1] Daniele Grattarola, Daniele Zambon, Filippo Maria Bianchi, and Cesare Alippi. Understanding pooling in graph neural networks. IEEE \n Transactions on Neural Networks and Learning Systems, 2022.\n\nSTRENGTHS:\n1. The proposed framework is fairly novel. It shows some dependencies on previous frameworks (i.e. SRC framework needed for graph pooling).\n2. The idea and methodology of the hierarchical pooling and merging, although it's not new (e.g. DiffPool), its quite interesting and intuitive for the problem of capturing long-range interactions.\n3. The extensive experimentations to a broad set of datasets suggest a strong performance of the MeGraph framework. \n4. There is a newly introduced benchmark, called Graph Theory Benchmark that can be used for various methods of evaluation. \n5. The written text is fairly clearly written. However, it can be improved, mainly due to the fact that it's hard to follow the main contribution section. A more clear summarization of the contributions (methodology, and experimentation) would significantly improve the clarity of the paper.\n6. The problem that MeGraph is approaching is quite important, mainly due to the known incapabilities of various graph neural network architectures to perform on par in deeper settings.\n\nWEAKNESSES:\n1. I am concerned about the computational complexity of the model. It seems that in the average case the combined computational cost of pooling, merging, and message passing is high, with respect to simple graph neural network models. The authors propose some variants to alleviate the high computational complexity, however it becomes more unclear how each one of the choices impact the behavior of the model. It seems (based on Tables 2, 4 ) that either the best performance comes from the computationally heavy variant of MeGraph or that there is no clear impact of the model choices. \n2. The model consists of a high number of trainable parameters, and module choices. Although I value the effort of the authors for an ablation study, I think that a further investigation of the parameters of the modules ( especially with respect to the encoder-decoder choices, and the graph pooling methods) would be quite helpful.\n3. There is no safe conclusion that long-range interactions can be captured because of the multi-scale method that the authors propose rather than the increased parameter space in MeGraph. The lack of any theoretical insight on why the specific choices contribute to capturing long-range interactions hinders convincing the reader that the multi-scale approach is appropriate for LRI.\n\nQUESTIONS:\n1. Can the authors pose any theoretical motivations on why the combination of SRC framework with the MeGraph framework can contribute to more effectively capture long-range interactions?\n\nLIMITATIONS:\n1. As I mentioned, I think that there is a lack of theoretical motivation for the choice of the model's modules. It would be quite helpful to provide any information on positive benefits of the SRC framework,as well as the representations obtained from the intra-graph and inter-graph interactions. \n2. The computational complexity is dependent on the choice of pooling methods, and encoder-decoder methods (as well as the message passing steps). It would be quite interesting to investigate on any potential improvement that would balance the increased parameter space with the training time. \n3. The written text can be further be clarified. The complex architecture of MeGraph requires a clear, and easy-to-follow description, so that the readers can grasp its mechanics.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn summary, the paper proposes a mechanism to learn on graph in a multiscale and hierarchical manner. This is one of the approaches that can address the long-range problem, in which the graph has a large diameter, i.e. the length of maximum shortest paths between all pairs of nodes is long, in graph learning.\n\nGiven the following strengths and weaknesses, I vote this paper in the borderline. There are certainly more works to do.\n\nSTRENGTHS:\nI believe this paper asks the right question: Long-range problem in graph learning is certainly important. Existing GNNs based on the conventional message passing have severe limitations with long-range graphs (i.e. graphs with \"long\" maximum shortest paths). The reason is message passing only allows exchanging of information among neighboring nodes, and consequently we would need many layers / iterations of message passing (one can argue that the number of layers needed is proportional to the size of the graph) so that two distant nodes can reach each other. However, it is computationally infeasible to have many layers for large graphs and it will cause the problem of over-smoothing and potentially other problems with gradients and training. \n\nTherefore, we need a method that allows distant nodes can exchange information via a small number of hops. Constructing a hierarchical and multiscale structure is one of the reasonable choices.\n\nWEAKNESSES:\n* Novelty: The long-range problem has attracted increasing attention from the graph learning community. There are similar ideas / works about multiscale and hierarchical structure learning being developed in parallel with this work.\n\nFor example, this work \"Multiresolution equivariant graph variational autoencoder\" https://iopscience.iop.org/article/10.1088/2632-2153/acc0d8/pdf proposed a similar multiscale / multiresolution method, but the construction of the hierarchy is done via a flexible learning-to-cluster algorithm in data-driven manner. The follow-up paper \"Multiresolution Graph Transformers and Wavelet Positional Encoding for Learning Long-Range and Hierarchical Structures\" https://arxiv.org/pdf/2302.08647.pdf (accepted at Journal of Chemical Physics) proposed an extension with Graph Transformers and a new positional encoding motivated from multiresolution analysis and wavelet theory, that can efficiently learn long-range and hierarchical graphs.\n\nFurthermore, a recent ICML 2023 paper \"On the Connection Between MPNN and Graph Transformer\" https://arxiv.org/pdf/2301.11956.pdf suggested that a simple method of adding a virtual node (i.e. a node connecting to all other nodes) can significantly boost the performance of the conventional message passing for the long-range graphs. This work has theoretical analysis that proves the equivalence between adding a virtual node and graph transformer.\n\n* Theoretical analysis: The paper got rejected from ICLR 2023 https://openreview.net/forum?id=Oz0npxjLAsI. One of the main reasons is the lack of theoretical analysis in analyzing the expressive power of their proposal. I suggest the authors to investigate theoretically their proposed model.\n\nQUESTIONS:\nN / A\n\nLIMITATIONS:\nI have written about other limitations of this work in the \"Weaknesses\" section. \n\nI think the authors should extend the Graph Theory Benchmark further. For example, the graphs should be much larger, it is interesting to see how each method behaves on a very large and very long-range graphs in terms of efficiency and efficacy. In particular, since this work is about multiscale representation of a graph, the benchmark should have Stochastic Block Model (SBM) or some graphs with clustering or hierarchical patterns.\n\nIf the benchmark is sophisticated enough, the authors can submit to the data & benchmark track. Recently, there is a separate track for data & benchmark in main ML conferences.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes MeGraph, a GNN architecture that interleaves local and hierarchical structural information in a graph at multiple-scales, to capture long-range interactions (LRI). The authors propose S-EdgePool that generalizes EdgePool by allowing more than two nodes to be clustered in order to achieve a desired pooling ratio. Using graph pooling, they build a \"mega graph\", which includes inter-edges connecting nodes at one height to the corresponding super-nodes at the next height. Furthermore, they propose a message passing scheme based on \"MeeLayers\", which first propagates information locally at each height, then propagates upwards and then backwards in the hierarchy. A complexity analysis is provided. Authors conduct an extremely extensive set of experiments on graph theory benchmarks, 10 standard LRI tasks plus three others proposed in the paper, LRGB, GNN benchmarks and OGB. Results show the superior performance of MeGraph in comparison with GCN, GIN, GAT, GatedGCN, Graph U-Net and a large variety of baselines specific to LRGB. An ablation study demonstrates gains from larger height and number of layers, as well as some other components of the method.\n\nSTRENGTHS:\nS1. The paper investigates a fundamental problem: how to capture longe-range interactions in a graph using GNNs without incurring over-smoothing or over-squashing issues. \nS2. The proposed solution is clean, well-justified and introduces some novel components (e.g., S-EdgePool, MeeLayer).\nS3. The proposed solution matches or exceeds the performance of reference methods in a large variety of diverse benchmarks.\nS4. Excellent presentation; writing is very clear despite the sophistication of the method.\nS5. Extremely extensive set of experiments (9 out of 16 pages in the appendix contain additional tables/plots).\nS6. Code and details of the experimental setup allow great reproducibility.\n\nWEAKNESSES:\nW1. There is no discussion regarding the hierarchies discovered by MeGraph for specific datasets/tasks, which could shed some light into the way the method is working.\nW2. HGNet also builds a multi-scale hierarchy via graph pooling and was shown to slightly outperform Graph U-Net, but is not included in the comparisons.\n\nQUESTIONS:\nQ1. While perturbation-based graph explainability frameworks can be applied with MeGraph, they would not provide immediate insights into how the hierarchy comes into play. Is there any graph explainability frameworks suitable for this task or are there fundamental challenges in deriving explanations for outputs predicted by MeGraph?\nQ2. Was there a particular reason not to include a comparison with HGNet?\nQ3. Does S-EdgePool tend to create groups of nodes with roughly the same size? If so, how would the performance of MeGraph be affected in graphs where clusters of different sizes arise naturally?\n\nLIMITATIONS:\nYes, the authors have discussed some obvious limitations in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-17"}