SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A common characteristic of these methods is that the number of supernodes  $K$  cannot be directly specified. Graclus and ECPool achieve a pooling ratio of approximately 0.5 by roughly reducing each time the graph size by  $50\%$ . On the other hand,  $k$ -MISPool can control the coarsening level by computing the maximal independent set from  $\mathcal{G}^k$ , which is the graph where each node of  $\mathcal{G}$  is connected to its  $k$ -hop neighbors. As the value of  $k$  increases, the pooling ratio decreases.
3.3 Criticism on graph pooling
Recently, the effectiveness of graph pooling has been questioned using as an argument a set of empirical results aimed at exposing the weaknesses of certain pooling operators [28]. The experiments showed that using a randomized cluster assignment matrix  $\mathbf{S}$  (followed by a softmax normalization) gives comparable results to using the assignment matrices learned by Diffpool [39] and MinCut-Pool [8]. Similarly, applying Graclus [13] on the complementary graph would give a performance similar to using the original graph.
We identified potential pitfalls in the proposed evaluation, which considered only pooling operators that are expressive and that, even after being modified, retain their expressive power. Clearly, even if expressiveness ensures that all the information is preserved in the pooled graph, its structure is corrupted when using a randomized S or a complementary graph. This hinders the effectiveness of the MP layers that come after pooling, as their inductive biases no longer match the data structure they receive. Notably, this might not affect certain classification tasks where the goal is to detect small structures, such as a motif in a molecule graph [25, 21], that are already captured by the MP layers before pooling.
To address these limitations, first, we propose to corrupt a pooling operator that is not expressive. In particular, we design a Top- $k$  pooling operator where the nodes are ranked based on a score that is sampled from a Normal distribution rather than being produced by a trainable function of the vertex features. Second, we evaluate all the modified pooling operators in a setting where the MP layers after pooling are essential for the task and show that the performance drop is significant.
To empirically confirm the theoretical results presented in Section 3, we designed a synthetic dataset that is specifically tailored to evaluate the expressive power of a GNN. We considered a GNN with MP layers interleaved with 10 different pooling operators: DiffPool [39], DMoN [36], MinCut [8], ECPool [14], Graclus,  $k$ -MISPool [4], Top- $k$  [18], PanPool [26], ASAP [31], and SAGPool [24]. For each pooling method, we used the implementation in Pytorch Geometric [17] with the default configuration. In addition, following the setup used to criticize the effectiveness of graph pooling [28], we considered the following pooling operators: Rand-Dense, a dense pooling operator where the cluster assignment is a normalized random matrix; Rand-Sparse, a sparse operator that ranks nodes based on a score sampled from a Normal distribution; Cmp-Grclus, an operator that applies the Graclus algorithm on the complement graph.
4.1 The EXPWL1 dataset
Our experiments aim at evaluating the expressive power of MP layers when combined with pooling layers. However, existing real-world and synthetic benchmark datasets are unsuitable for this purpose as they are not specifically designed to relate the power of GNNs to that of the WL test. Recently, the EXP dataset was proposed to test the capability of special GNNs to achieve higher expressive power than the WL test [1], which, however, goes beyond the scope of our evaluation. Therefore, we introduce a modified version of EXP called EXPWL1, which comprises a collection of graphs  $\{\mathcal{G}_1,\dots ,\mathcal{G}_N,\mathcal{H}_1,\dots ,\mathcal{H}_N\}$  representing propositional formulas that can be satisfiable or unsatisfiable. Each pair  $(\mathcal{G}_i,\mathcal{H}_i)$  in EXPWL1 consists of two non-isomorphic graphs distinguishable by a WL test, which encode formulas with opposite SAT outcomes. Therefore, any GNN that has an expressive power equal to the WL test can distinguish them and achieve approximately  $100\%$  classification accuracy on the dataset. Compared to the original EXP dataset, we increased the size of the dataset to a total of 3000 graphs and we also increased the size of each graph from an average of 55 nodes to 76 nodes. This was done to make it possible to apply an aggressive pooling without being left with a trivial graph structure. The EXPWL1 dataset and the code to reproduce the experimental results are publicly available<sup>2</sup>.
4.2 Experimental procedure
To empirically evaluate which pooling operator maintains the expressive power of the MP layers preceding it, we first identified a GNN architecture without pooling layers, which achieves approximately  $100\%$  accuracy on the EXPWL1. We tried different baselines (details in Appendix C.1) and we found that a GNN with three GIN layers [38] followed by a global_sum_pool reaches the desired accuracy. Then, we inserted a pooling layer between the  $2^{\mathrm{nd}}$  and  $3^{\mathrm{rd}}$  GIN layer, which performs an aggressive pooling by using a pooling ratio of 0.1 that reduces the graph size by  $90\%$ . The details of the GNN architectures are in Appendix C.2. Besides 0.1, we also considered additional pooling ratios and we reported the results in Appendix C.3. To ensure a fair comparison, when testing each method we shuffled the datasets and created 10 different train/validation/test splits using the same random seed. We trained each model on all splits for 500 epochs and reported the average training time and the average test accuracy obtained by the models that achieved the lowest loss on the validation set. To validate our experimental approach, we also measured the performance of the proposed GNN architecture equipped with the different pooling layers on popular benchmark datasets for graph classification [30, 10].
Table 1 reports the performances of different pooling operators on EXPWL1. These results are consistent with our theoretical findings: pooling operators that satisfy the conditions of Th. 1 achieve the highest average accuracy and, despite the aggressive pooling, they retain all the necessary information to achieve the same performance of a GNN without a pooling layer. On the other hand, non-expressive pooling operators achieve a significantly lower accuracy as they are not able to correctly distinguish all graphs.
$^{2}$ https://github.com/FilippoMB/The-expressive-power-of-pooling-in-GNNs
Pooling	s/epoch	GIN layers	Pool Ratio	Test Acc	Expressive
No-pool	0.33s	3	-	99.3±0.3	✓
DiffPool	0.69s	2+1	0.1	97.0±2.4	✓
DMoN	0.75s	2+1	0.1	99.0±0.7	✓
MinCut	0.72s	2+1	0.1	98.8±0.4	✓
ECPool	20.71s	2+1	0.2	100.0±0.0	✓
Graclus	1.00s	2+1	0.1	99.9±0.1	✓
k-MIS	1.17s	2+1	0.1	99.9±0.1	✓
Top-k	0.47s	2+1	0.1	67.9±13.9	X
PanPool	3.82s	2+1	0.1	63.2±7.7	X
ASAPool	1.11s	1+1	0.1	83.5±2.5	X
SAGPool	0.59s	1+1	0.1	79.5±9.6	X
Rand-dense	0.41s	2+1	0.1	91.7±1.3	✓
Cmp-Graclus	8.08s	2+1	0.1	91.9±1.2	✓
Rand-sparse	0.47s	2+1	0.1	62.8±1.8	X
Table 1: Classification results on EXPWL1.
Table 1 also shows that employing a pooling operator based on a normalized random cluster assignment matrix (Rand-dense) or the complement graph (Cmp-Graclus) gives a lower performance. First of all, this result disproves the argument that such operators are comparable to the regular ones [28]. Additionally, we notice that the reduction in performance is less significant for Rand-Dense and Cmp-Graclus than for Rand-sparse. This outcome is expected because, in terms of expressiveness, Rand-dense and Cmp-Graclus still satisfy the conditions of Th. 1. Nevertheless, their performance is still lower than their regular counterparts. The reason is that even if a badly-behaved CON function does not compromise the expressiveness of the pooling operator, the structure of the pooled graph is corrupted when utilizing a randomized S or a complementary graph. This, in return, reduces the effectiveness of the last GIN layer, which is essential to correctly classify the graphs in EXPWL1.
There are two remarks about the experimental evaluation. As discussed in Section 3.2, it is not possible to explicitly specify the pooling ratio in Graclus, ECPool, and  $k$ -MISPool. For  $k$ -MISPool, setting  $k = 5$  gives a pooling ratio of approximately 0.09 on EXPWL1. However, for Graclus, Cmp-Gracus, and ECPool, the only option is to apply the pooling operator recursively until the desired pooling ratio of 0.1 is reached. Unfortunately, this approach is demanding, both in terms of computing time and memory usage. While in EXPWL1 it was possible to do this for Graclus and Cmp-Gracus, we encountered an out-of-memory error after a few epochs when running ECPool on an RTX A6000 with 48GB of VRAM. Thus, the results for ECPool are obtained with a recursion that gives a pooling ratio of approximately 0.2. While this simplifies the training in ECPool we argue that, due to its expressiveness, ECPool would have reached approximately  $100\%$  accuracy on EXPWL1 if implementing a more aggressive pooling was feasible.
The second remark is that in EXPWL1 when using too many MP layers, at least one node ends up containing enough information to accurately classify the graphs. This was demonstrated a model with 3 GIN layers followed by global_max_pool, which achieved an accuracy of  $98.3 \pm 0.6$  (more details in Appendix C.1). Note that the baseline model in Tab. 1 with 3 GIN layers equipped with the more expressive global_sum_pool achieves a slightly higher accuracy of  $99.3 \pm 0.3$ . In contrast, a model with only 2 GIN layers and global_max_pool gives a significantly lower accuracy of  $66.5 \pm 1.8$ . Therefore, to ensure that the evaluation is meaningful, no more than 2 MP layers should precede the pooling operator. Since ASAP and SAGPool implement an additional MP operation internally, we used only 1 GIN layer before them, rather than 2 as for the other pooling methods.
Finally, Fig. 3 shows the average accuracy and the average run-time obtained on several benchmark datasets by a GNN equipped with the different pooling methods (the detailed results are in Appendix C.5). These benchmarks are not designed to test the expressive power and, thus, a GNN equipped with a non-expressive pooling operator could achieve good performance. Specifically, this happens in those datasets where all the necessary information is captured by the first two GIN layers that come before pooling or in datasets where only a small part of the graph is needed to determine the class. Nevertheless, this second experiment serves two purposes. First, it demonstrates the soundness of the GNN architecture used in the first experiment, which achieves results comparable to those of GNNs carefully optimized on the benchmark datasets [16]. Second, and most importantly, it shows that the performances on the benchmark datasets and EXPWL1 are aligned; this underlines the
Figure 3: Average accuracy (and std.) v.s. average runtime on the benchmark datasets.
relevance of our theoretical result on the expressiveness in practical applications. It is worth noting that on the benchmark datasets, it was not possible to obtain a pooling ratio of 0.1 for both Graclus and ECPool. Using a pooling ratio of 0.5 gives Graclus and ECPool an advantage over other methods, which makes the comparison not completely fair and shows an important limitation of these two methods.
As a concluding remark, we comment on the training time of the dense and sparse pooling methods. A popular argument in favor of sparse pooling methods is their computational advantage compared to the dense ones. Our results show that this is not the case in modern deep-learning pipelines. In fact, ECPool, Graclus, PanPool, and even ASAPool are slower than dense pooling methods, while the only sparse method with training times lower than the dense ones is  $k$ -MIS. Even if it is true that the sparse methods save memory by avoiding computing intermediate dense matrices, this is relevant only for very large graphs that are rarely encountered in most applications.
In this work, we studied for the first time the expressive power of pooling operators in GNNs. We identified the sufficient conditions that a pooling operator must satisfy to fully preserve the expressive power of the original GNN model. Based on our theoretical results, we proposed a principled approach to evaluate the expressive power of existing graph pooling operators by verifying whether they met the conditions for expressiveness.
To empirically test the expressive power of a GNN, we introduced a new dataset that allows verifying if a GNN architecture achieves the same discriminative power of the WL test. We used such a dataset to evaluate the expressiveness of a GNN equipped with different pooling operators and we found that the experimental results were consistent with our theoretical findings. We believe that this new dataset will be a valuable tool as it allows, with minimal effort, to empirically test the expressive power of any GNN architecture. In our experimental evaluation, we also considered popular benchmark datasets for graph classification and found that the expressive pooling operators achieved higher performance. This confirmed the relevance in practical applications of our principled criterion to select a pooling operator based on its expressiveness. Finally, we focused on the computational time of the pooling methods and found that most sparse pooling methods not only perform worse due to their weak expressive power but are often not faster than the more expressive pooling methods.
We hope that our work will provide novel insights into the relational deep learning community and help to debunk misconceptions about graph pooling. We conclude by pointing to three limitations of our work. Firstly, the conditions of Th. 1 are sufficient but not necessary, meaning that there could be a non-expressive pooling operator that preserves all the necessary information. A similar consideration holds for EXPWL1: methods failing to achieve  $100\%$  accuracy are non-expressive, but the opposite is not necessarily true. In fact, reaching  $100\%$  accuracy on EXPWL1 is a necessary condition for expressiveness, though not sufficient. Secondly, condition 1 is not guaranteed to hold for continuous node features, which is a theoretical limitation not necessarily relevant in practice. Finally, our investigation focused on the scenario where the MP operation before pooling is as powerful as the 1-WL test. Even if layers more powerful than 1-WL test are rarely used in practice, it would be interesting to extend our approach to investigate the effect of pooling in these powerful architectures.
This research was partially funded by the MUR PNRR project "THE - Tuscany Health Ecosystem". We gratefully acknowledge the support of Nvidia Corporation with the donation of the RTX A6000 GPUs used in this work. Finally, we thank Daniele Zambon, Caterina Graziani, and Antonio Longa for the useful discussions.
[1] R. Abboud, I. I. Ceylan, M. Grohe, and T. Lukasiewicz. The surprising power of graph neural networks with random node initialization. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI), 2021.
[2] L. Babai. Graph isomorphism in quasipolynomial time. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing, pages 684-697, 2016.
[3] D. Bacciu and L. Di Sotto. A non-negative factorization approach to node pooling in graph convolutional neural networks. In AI* IA 2019-Advances in Artificial Intelligence: XVIIIth International Conference of the Italian Association for Artificial Intelligence, Rende, Italy, November 19–22, 2019, Proceedings 18, pages 294–306. Springer, 2019.
[4] D. Bacciu, A. Conte, and F. Landolfi. Graph pooling with maximum-weight  $k$ -independent sets. In Thirty-Seventh AAAI Conference on Artificial Intelligence, 2023.
[5] J. Baek, M. Kang, and S. J. Hwang. Accurate learning of graph representations with graph multiset pooling. In Proceedings of the 9th International Conference on Learning Representations, 2021.
[6] S. T. Barnard and H. D. Simon. Fast multilevel implementation of recursive spectral bisection for partitioning unstructured problems. *Concurrency: Practice and experience*, 6(2):101-117, 1994.
[7] B. Bevilacqua, F. Frasca, D. Lim, B. Srinivasan, C. Cai, G. Balamurugan, M. M. Bronstein, and H. Maron. Equivariant subgraph aggregation networks. In International Conference on Learning Representations, 2022.
[8] F. M. Bianchi, D. Grattarola, and C. Alippi. Spectral clustering with graph neural networks for graph pooling. In International conference on machine learning, pages 874-883. PMLR, 2020.
[9] F. M. Bianchi, D. Grattarola, L. Livi, and C. Alippi. Hierarchical representation learning in graph neural networks with node decimation pooling. IEEE Transactions on Neural Networks and Learning Systems, 33(5):2195-2207, 2020.
[10] F. M. Bianchi, C. Gallicchio, and A. Micheli. Pyramidal reservoir graph neural network. volume 470, pages 389-404. Elsevier, 2022.
[11] J. Bruna, W. Zaremba, A. Szlam, and Y. LeCun. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203, 2013.
[12] M. Defferrard, X. Bresson, and P. Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems, 29, 2016.
[13] I. S. Dhillon, Y. Guan, and B. Kulis. Weighted graph cuts without eigenvectors a multilevel approach. IEEE transactions on pattern analysis and machine intelligence, 29(11):1944-1957, 2007.
[14] F. Diehl. Edge contraction pooling for graph neural networks. arXiv preprint arXiv:1905.10990, 2019.
[15] V. P. Dwivedi, C. K. Joshi, T. Laurent, Y. Bengio, and X. Bresson. Benchmarking graph neural networks. 2020.
[35] S. A. Tailor, F. Opolka, P. Lio, and N. D. Lane. Do we need anisotropic graph neural networks? In International Conference on Learning Representations, 2021.
[36] A. Tsitsulin, J. Palowitch, B. Perozzi, and E. Müller. Graph clustering with graph neural networks. J. Mach. Learn. Res., 24:127:1-127:21, 2023.
[37] B. Weisfeiler and A. Leman. The reduction of a graph to canonical form and the algebra which appears therein. nti, Series, 2(9):12-16, 1968.
[38] K. Xu, W. Hu, J. Leskovec, and S. Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019.
[39] Z. Ying, J. You, C. Morris, X. Ren, W. Hamilton, and J. Leskovec. Hierarchical graph representation learning with differentiable pooling. Advances in neural information processing systems, 31, 2018.
[40] H. Yuan and S. Ji. Structpool: Structured graph pooling via conditional random fields. In Proceedings of the 8th International Conference on Learning Representations, 2020.
[41] M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. J. Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
[42] D. Zambon, C. Alippi, and L. Livi. Graph random neural features for distance-preserving graph representations. In International Conference on Machine Learning, pages 10968-10977. PMLR, 2020.
Proof. Let  $\mathbf{S} \in \mathbb{R}^{N \times K}$  and  $\mathbf{T} \in \mathbb{R}^{M \times K}$  be the matrices representing the cluster assignments generated by  $\mathrm{SEL}(\mathcal{G}_1^L)$  and  $\mathrm{SEL}(\mathcal{G}_2^L)$ , respectively. When condition 2 holds, we have that the entries of matrices  $\mathbf{S}$  and  $\mathbf{T}$  satisfy  $\sum_{j=1}^{k} s_{ij} = \lambda, \forall i = 1, \dots, N$  and  $\sum_{j=1}^{K} t_{ij} = \lambda, \forall i = 1, \dots, M$ .
If condition 3 holds, then the  $j$ -th row of  $\mathbf{X}_P$  is  $\mathbf{x}_{P_j} = \sum_{i=1}^{N} \mathbf{x}_i^L \cdot s_{ij}$ . The same holds for the  $j$ -th row of  $\mathbf{Y}_P$ , which is  $\mathbf{y}_{P_j} = \sum_{i=1}^{M} \mathbf{y}_i^L \cdot t_{ij}$ . Suppose that there exists a rows' permutation  $\pi: \{1, \ldots, K\} \to \{1, \ldots, K\}$  such that  $\mathbf{x}_{P_j} = \mathbf{y}_{P_{\pi(j)}} \forall i = 1, \ldots, M$ , that is:
$$
\sum_ {i = 1} ^ {N} \mathbf {x} _ {i} ^ {L} \cdot s _ {i j} = \sum_ {i = 1} ^ {M} \mathbf {y} _ {i} ^ {L} \cdot t _ {i \pi (j)} \quad \forall j = 1, \dots , K
$$
$$
\begin{array}{l} \sum_ {j = 1} ^ {K} \sum_ {i = 1} ^ {N} \mathbf {x} _ {i} ^ {L} \cdot s _ {i j} = \sum_ {j = 1} ^ {K} \sum_ {i = 1} ^ {M} \mathbf {y} _ {i} ^ {L} \cdot t _ {i \pi (j)} \Leftrightarrow \sum_ {i = 1} ^ {N} \mathbf {x} _ {i} ^ {L} \cdot \sum_ {j = 1} ^ {K} s _ {i j} = \sum_ {i = 1} ^ {M} \mathbf {y} _ {i} ^ {L} \cdot \sum_ {j = 1} ^ {K} t _ {i \pi (j)} \Leftrightarrow \\ \stackrel {\Leftrightarrow} {\sum_ {i = 1} ^ {N}} \mathbf {x} _ {i} ^ {L} \cdot \lambda = \sum_ {i = 1} ^ {M} \mathbf {y} _ {i} ^ {L} \cdot \lambda \Leftrightarrow \sum_ {i = 1} ^ {N} \mathbf {x} _ {i} ^ {L} = \sum_ {i = 1} ^ {M} \mathbf {y} _ {i} ^ {L} \\ \end{array}
$$
which contradicts condition 1.
Note that there are no restrictions on the cardinality of the original sets of nodes,  $|\mathcal{V}_1| = N$  and  $|\mathcal{V}_2| = M$ . Indeed, since the proof does not depend on the number of nodes in the original graphs,  $N$  can either be equal to  $M$  or not. Additionally, we only focused on pooled graphs with the same number of nodes, i.e.,  $|\mathcal{V}_{1_P}| = |\mathcal{V}_{2_P}| = K$ . The case where  $|\mathcal{V}_{1_P}| \neq |\mathcal{V}_{2_P}|$  is trivial since two graphs with different numbers of nodes are inherently not WL equivalent.
B Examples from the EXPWL1 dataset
In Figure 4 we report four graph pairs from the EXPWL1 dataset. Each pair contains graphs with a different SAT outcome, which are WL-1 distinguishable. In the Figure, we used a different color map for each pair but the node features always assume a binary value in  $\{0,1\}$  in each graph.
C.1 Baseline models
To identify a GNN architecture that achieves approximately  $100\%$  accuracy on EXPWL1, we tried configurations with a different number of GIN layers followed by a global_max_pool or global_sum_pool. For the sake of comparison, we also considered a GNN with GCN layers, which are not expressive. The results are shown in Table 2.
As expected, the architectures with GIN layers outperform those with GCN layers, especially when the layers are two. This is due to the fact that GCN implements mean pooling as an aggregator, which is a well-defined multiset function due to its permutation invariance, but it lacks injectiveness, leading to a loss of expressiveness. Similarly, the GNNs with global_sum_pool perform better than those with global_max_pool, since the former is more expressive than the latter. An architecture with 3 GIN layers followed by global_sum_pool achieves approximately  $100\%$  accuracy on EXPWL1, making it the ideal baseline for our experimental evaluation. Perhaps more importantly, there is a significant difference in the performance of a 2-layers GNN followed by a global pooling layer that is more or less expressive. For this reason, the node embeddings generated by 2 GIN layers are a good candidate to test the expressiveness of the pooling operators considered in our analysis.
Figure 4: Four pairs of graphs from the EXPWL1 dataset. Each pair consists of two graphs with different classes that are 1-WL distinguishable.
MP layers	Global Pool	Test Acc
2 GIN	global_max_pool	66.5±1.8
2 GIN	global_sum_pool	92.1±1.0
2 GCN	global_max_pool	62.3±2.4
2 GCN	global_sum_pool	76.7±2.4
3 GIN	global_max_pool	98.3±0.6
3 GIN	global_sum_pool	99.3±0.3
3 GCN	global_max_pool	97.4±0.5
3 GCN	global_sum_pool	98.7±0.6
Table 2: Performance of baseline architectures on EXPWL1.
C.2 Hyperparameters of the GNN architectures