SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- OGBG-Molhiv [47] is a molecular dataset with binary graph labels according to the inhibition effect on HIV virus replication, where nodes are atoms, and edges are chemical bonds. As there are no ground-truth explanations, we merely evaluate the prediction performance.
B.1 Statistical Characteristics
We show the detailed statistics of the datasets in Table 5.
Table 5: Detailed statistics of graph classification datasets in main experiment.
BA-2Motifs	Spurious-Motif	Mutag
Train	Val	Test	Train	Val	Test	Train	Val	Test
#Classes	2	3	2
#Graphs	400	400	200	9,000	3,000	6,000	1000	1000	951
Avg. #Nodes	25.0	25.0	25.0	25.4	26.1	88.7	30.1	30.1	30.2
Avg. #Edges	50.9	51.0	50.9	35.4	36.2	131.1	61.3	60.2	61.2
MNIST-75sp	Graph-SST2	OGBG-Molhiv
Train	Val	Test	Train	Val	Test	Train	Val	Test
#Classes	10	2	2
#Graphs	20,000	5,000	10,000	28,237	3,147	12,305	32,901	4,113	4,113
Avg. #Nodes	66.8	67.3	67.0	17.7	17.3	3.45	25.3	27.8	25.3
Avg. #Edges	539.3	545.9	540.9	33.3	33.5	4.89	54.1	61.1	55.6
We show the detailed statistics of datasets that used in the inter-task evaluation in Table 6.
Table 6: Detailed statistics of node classification datasets in inter-task experiment.
BA-Shapes	BA-Community	Tree-Cycles	Tree-Grid
#Classes	4	8	2	2
#Nodes	700	1,400	871	1,231
#Edges	4,110	8,920	1,950	3,410
B.2 Structural Patterns
Degree Distribution. As shown in Figure 4, we visualize the degree distribution of the four real-world datasets, i.e., Mutag, MNNIST-75sp, Ogbg-Molhiv, and Graph-SST2, and our synthetic scale-free dataset PT-Motifs. The power-law like degree function  $D(x) = cx^{-\alpha}$  of each real-world dataset is formally represented as follows, optimized by the ordinary least square error. Here  $D(x)$  represents the number of  $x$ -degree nodes and  $R^2$  is the coefficient of determination.
- Mutag:  $D(x) = 54447x^{-1.537}$ ,  $R^2 = 0.9015$ .
- MNIST-75sp:  $D(x) = 10^{7}x^{-5.913}$ ,  $R^2 = 0.6713$ .
- Ogbg-Molhiv:  $D(x) = 3 \times 10^{6}x^{-6.039}$ ,  $R^2 = 0.6828$ .
- Graph-SST2:  $D(x) = 8 \times 10^{6}x^{-4.988}$ ,  $R^2 = 0.8491$ .
- PT-Motifs:  $D(x) = 10^{6}x^{-1.501}, R^{2} = 0.9677$ .
In Table 7, we report the average value of the four structural patterns over each individual graph in the datasets, including
- Transitivity is defined as the fraction of all possible triangles present in the given graph  $G$ . It measures the tendency of connections or relationships between nodes to form triangles or triplets and captures the clustering characteristics in a given graph. Let triads be the structure that two edges with a shared node, then the transitivity can be formally represented as follows,
$$
\text {T r a n s i t i v i t y} = 3 \frac {\# \text {T r i a n g l e s}}{\# \text {T r i a d s}}, \tag {14}
$$
where #Triangles and #Triads are the number of triangles and triads in  $G$ , respectively.
(a) Degree distribution of Mutag
(b) Degree distribution of MNIST-75sp
(c) Degree distribution of Ogbg-Molhiv
(d) Degree distribution of Graph-SST2
(e) Degree distribution of PT-Motifs
Figure 4: The degree distribution of four real-world graph datasets (Mutag, MNIST-75sp, Ogbg-Molhiv, and Graph-SST2) and our synthetic dataset PT-Motifs. The  $x$  axis is the node degree.
- Assortativity is defined as the similarity of connections in the given graph  $G$  with respect to the node degree. It measures the preference of nodes to connect with nodes of similar or different characteristics. Assortativity helps uncover the underlying patterns of connectivity and provides insights into the organization and functioning of complex networks. The average assortativity of the given graph  $G$  is the mean value over all nodes.
- Efficiency of a pair of nodes is defined as the multiplicative inverse of the shortest path distance between the endpoints in the given graph  $G$ . The average global efficiency of a graph is the average efficiency of all pairs of nodes.
- Clustering of the node  $v$  is defined as the fraction of possible triangles through that node. The corresponding clustering coefficient  $c_{u}$  can be formulated as,
$$
c _ {v} = 2 \frac {\# \text {T r i a n g l e s} _ {u}}{\deg_ {u} (\deg_ {u} - 1)}, \tag {15}
$$
where  $\# \mathrm{Triangles}_u$  is the number of triangles through node  $u$  and  $\deg_u$  is degree of node  $u$ . The average clustering coefficient  $C$  of the given graph  $G$  with  $n$  nodes is defined as follows,
$$
C = \frac {1}{n} \sum_ {v \in G} c _ {v}. \tag {16}
$$
Table 7: Structural patterns of the PT-Motifs dataset and four real-world datasets.
PT-Motifs	Mutag	MNIST-75sp	Molhiv	Graph-SST2
Transitivity	0.3620	0.0013	0.5005	0.0021	0
Avg Assortativity	0.1525	-0.4196	0.3235	-0.2678	-0.6269
Avg Efficiency	0.5044	0.3261	0.3977	0.3243	0.5491
Avg Clustering	0.4185	0.0010	0.5408	0.0020	0
For an intuitive understanding of the graph structural patterns, we visualize the distribution of node pair efficiency and node assortativity in Figure 5 and Figure 6, respectively. One can notice that the efficiency distributions of PT-Motifs, Mutag, MNIST-75sp, and Molhiv all follow a bell-shaped curve, while that of Graph-SST2 is a little different. Similarly, we found that the node assortativity distributions of Mutag, MNIST-75sp, Molhiv, and PT-Motifs all have one main peak which largely surpasses the adjacent values, while the node assortativity of Graph-SST2 dataset has four peaks.
Overall, the structural patterns above, including the degree distribution, the transitivity, the assortativity, the efficiency, and the clustering, are universal and generalizable across different datasets. However, the expression degree of different structural patterns may differ largely in different datasets. As shown above, the Graph-SST2 dataset strictly follows the power law shaped degree distribution, but its node pair efficiency and node assortativity distributions are different from the other three real-world datasets. Therefore, we propose the structural pattern learning module in  $\pi$ -GNN, to capture multiple structural patterns and integrate them for a more universal and generalizable representation.
We first present the details of the pre-training phase over the synthetic PT-Motifs dataset. The PT-Motifs dataset is split into the training set of 50,000 graphs, the validation set of 10,000 graphs, and the testing set of 20,000 graphs. Each graph class has equal number of instances in the three sets. During the pre-training phase, the batchsize is set as  $\{32,64,128,256\}$  and the learning rate is set as  $\{10^{-3},5\times 10^{-3},10^{-4},10^{-5},10^{-6}\}$ . The pre-training epoch is set as  $\{20,40,60,80\}$ . We select the pre-trained  $\pi$ -GNN model according to the validation performance. During the pre-training and the fine-tuning phases, we use the Adam optimizer.
As shown in Tables 8 and 9, we present both the downstream predictor architecture and the fine-tuning details of the graph classification datasets and the node classification datasets, respectively. For all the graph classification datasets, we use the global mean pooling as the pooling function. Specially, in the Molhiv predictor, we introduce the virtual node and weighted loss tricks to mitigate the class-imbalance issue (39,684 negative examples and only 1,443 positive examples). All experiments are conducted on a single NVIDIA GeForce 3090 GPU (24GB).
D Explanatory Visualization
For an intuitive understanding on the  $\pi$ -GNN explanation, we present a few visualized explanation results of the Graph-SST2 dataset in Figure 7. Overall,  $\pi$ -GNN has the ability to highlight the influential phrases that directly express the positive or negative sentiments in the sentence. Specifically,  $\pi$ -GNN correctly allocates large weights to the positive words "a legend" in Figure 7(a), as well as "bewilderingly brilliant" and "entertaining" in Figure 7(c). Furthermore, the negative word, such
(a) Node pair efficiency distribution of Mutag
(b) Node pair efficiency distribution of MNIST-75sp
(c) Node pair efficiency distribution of Ogbg-Molhiv
(d) Node pair efficiency distribution of Graph-SST2
(e) Node pair efficiency distribution of PT-Motifs
Figure 5: The node pair efficiency distribution of four real-world graph datasets (Mutag, MNIST-75sp, Ogbg-Molhiv, and Graph-SST2) and the synthetic dataset PT-Motifs.
Table 8: Predictor architecture and fine-tuning details of graph classification datasets.
BA-2Motifs	Spurious-Motif	Mutag	MNIST-75sp	Molhiv	Graph-SST2
Backbone	GCN	GCN	GCN	GIN	GIN	GCN
Layers	1	1	1	2	2	1
Batchsize	256	128	64	256	128	32
Learning rate	8 × 10-4	4 × 10-4	1 × 10-4	1 × 10-3	1 × 10-3	1 × 10-4
Epochs	30	30	30	50	40	40
as "absurd lengths" and "skip dreck" in Figure 7(b) and Figure 7(d), are identified by  $\pi$ -GNN for a transparent sentiment prediction. The visualized explanation again demonstrates the effectiveness of  $\pi$ -GNN: (1) the pre-trained  $\pi$ -GNN interpreter can faithfully extract the most vital subgraph that contains the label-relevant information; and (2) the pre-trained  $\pi$ -GNN interpreter is able to cooperate with specific downstream predictor for both accuracy and interpretability. See Figure 7 for more visualized results of the explanations.
E Supplement Experiment
To further investigate the impact of the pre-training dataset on the final results, we conduct empirical studies on the size and the imbalance degree of the pre-training dataset.
(a) Node assortativity distribution of Mutag
(b) Node assortativity distribution of MNIST-75sp
(c) Node assortativity distribution of Ogbg-Molhiv
(d) Node assortativity distribution of Graph-SST2
(e) Node assortativity distribution of PT-Motifs
Figure 6: The node assortativity distribution of four real-world graph datasets (Mutag, MNIST-75sp, Ogbg-Molhiv, and Graph-SST2) and our synthetic dataset PT-Motifs.
Table 9: Predictor architecture and fine-tuning details of node classification datasets.
BA-Shapes	BA-Community	Tree-Cycles	Tree-Grid
Backbone	GCN	GCN	GCN	GCN
Layers	3	3	3	3
Learning rate	1 × 10-1	1 × 10-1	1 × 10-2	5 × 10-2
Epochs	20	20	20	20
First, compared with PT-Motifs (80,000 graphs) in the main experiment, we generate another two datasets with different sizes. In detail, PT-Motifs-M with 50,000 graphs and PT-Motifs-S with 10,000 graphs represent the middle-level and the small-level pre-training datasets, respectively. The split ratio of the training, validation, and testing set is  $\{0.7, 0.1, 0.2\}$  in PT-Motifs-M and PT-Motifs-S. As shown in Table 10, the results show that even the PT-Motifs-S is able to outperform the model without pre-training. Moreover, we can notice that a large pre-training dataset can indeed improve the performance more significantly than that with small size.
Furthermore, it is possible that if the synthetic pre-training dataset is imbalanced, the performance improvement on downstream tasks will degrade, in terms of both the interpretation and prediction. To further investigate this issue, we have added supplement experiments on imbalanced pre-training dataset. Following existing works [13, 11], to generate the imbalanced datasets, we sample the
(a) Explanation subgraph: positive sentiment
(b) Explanation subgraph: negative sentiment
(c) Explanation subgraph: positive sentiment
(d) Explanation subgraph: negative sentiment
(e) Explanation subgraph: positive sentiment
(f) Explanation subgraph: negative sentiment
(g) Explanation subgraph: positive sentiment
(h) Explanation subgraph: negative sentiment
(i) Explanation subgraph: positive sentiment
(j) Explanation subgraph: negative sentiment
(k) Explanation subgraph: positive sentiment
(1) Explanation subgraph: negative sentiment
Figure 7: Visualization of  $\pi$ -GNN explanation on the Graph-SST2 dataset. Each graph represents a comment, where the explanations are highlighted by bule boxes and bolder lines.
Table 10: The impact of pre-training datasets with different sizes.
Task	Dataset	PT-Motifs	W/O Pre-train	PT-Motifs-M	PT-Motifs-S
Interpretation	BA-2Motifs	99.33	93.19	98.91	97.07
Mutag	99.81	95.29	99.06	96.28
Prediction	Ogbg-Molhiv	80.86	79.71	80.77	79.82
Graph-SST2	88.05	83.48	87.59	85.23
explanatory  $G_{e}$  from uniform distribution, while the base  $G_{b}$  is determined by the following formula,
$$
P \left(G _ {b}\right) = b \times I \left(G _ {b} = G _ {e}\right) + \frac {1 - b}{4} \times I \left(G _ {b} \neq G _ {e}\right). \tag {17}
$$
Therefore, we can manipulate  $b$  to control the imbalance degree and the imbalance degree is defined as  $i = 4b / (1 - b)$ . The corresponding result is reported in Table 11. The results demonstrate that when pretraining on an imbalanced dataset, the performance improvement is less significant than that on the balanced one, but still better than that without pretraining. Moreover, an overly imbalanced pre-training dataset ( $b = 0.7$ ,  $i \approx 9.3$ ) may cause negative transfer issue (on Muatg, Molhiv, and Graph-SST2). Therefore, the pre-training dataset ought to be balanced and thus can mitigate the negative transfer issue to some extent.
Table 11: The impact of pre-training datasets with different sizes.
Task	Dataset	PT-Motifs	W/O Pre-train	PT-Motifs-M	PT-Motifs-S
Interpretation	BA-2Motifs	99.33	93.19	98.91	97.07
Mutag	99.81	95.29	99.06	96.28
Prediction	Ogbg-Molhiv	80.86	79.71	80.77	79.82
Graph-SST2	88.05	83.48	87.59	85.23
F Hyper-parameter Analysis
In Figure 8, we conduct analysis on the number of basic pattern-learners, to evaluate the tendency of  $\pi$ -GNN performance with the increase of the pattern-learner number. Additionally, we report the performance of the variant without pre-training phase, which is marked by the subscript "DFT". One can observe that as the number of basic learners increasing, both the interpretation and prediction performance generally improves, which demonstrates the co-existence of multiple structural patterns. For the Mutag dataset, when we increase the number to 2, the performance improvement (5.76%) is the most significant. But when the number increases to 16 and 32, the performance is inferior to that of 8 basic pattern-learners. We ascribe this degradation to the difficulty of integrating more and more patterns when the basic learner number increasing. For the Graph-SST2 dataset, the prediction performance consistently improves along with the number of basic learners. This may indicate that the structural patterns in Graph-SST2 are more intricate than those in Mutag dataset. Moreover, the induced variant without the pre-training phase is superior to the  $\pi$ -GNN model consistently, which again verifies the effectiveness of the explainer pre-training phase, by 3.32% ROC-AUC score on Mutag dataset and 4.08% accuracy on Graph-STT2 dataset on average.
G Derivation
First, we restate and prove Theorem 1 [44].
Theorem 1. Let  $\Sigma_{n}$  be the set of all adjacency matrix  $\mathbf{A}$  with  $n$  nodes. Given a graph  $G = (\mathcal{V},\mathcal{E})\in \Sigma_{n},n\geq 2,$  let  $\Gamma^{*}(S,\mathbf{A})$  be a most-expressive structural representation of nodes set  $S\subseteq \mathcal{V}$  in  $G$ .  $\forall \mathbf{A}\in \Sigma_{n}$ , there exists a most-expressive node representation  $\mathbf{Z}^{\ast}|\mathbf{A}$  satisfies the relationship as follows,
$$
\Gamma^ {*} (S, \mathbf {A}) = \mathbb {E} _ {\mathbf {Z} ^ {*}} [ f ^ {(| S |)} ((\mathbf {Z} _ {v} ^ {*}) _ {v \in S}) | \mathbf {A} ], \forall S \subseteq \mathcal {V}, \tag {18}
$$
Proof. Given graph  $G = (\mathcal{V}, \mathcal{E})$  with  $n$  nodes, we construct an equivalent set of the most-expressive structural representation  $\Gamma^{*}(S, \mathbf{A})$ , with permutations on node indices:
$$
\Pi (\mathbf {A}) = \left\{\Gamma^ {*} (v, \mathbf {A}, \pi (1, 2, \dots , n)) _ {\forall v \in \mathcal {V}} \mid \pi \in \Pi_ {n} \right\} \tag {19}
$$
Define  $\mathbf{Z}^*|\mathbf{A}$  as the random variable with a uniform measure over  $\Pi(\mathbf{A})$ . Assume the node subset  $S$  has no other joint isomorphic set  $S'$ . Then, for any such  $S$  and any element  $\gamma_{\pi} \in \Pi(\mathbf{A})$  as follow,
$$
\gamma_ {\pi} = \Gamma^ {*} (v, \mathbf {A}, \pi (1, 2, \dots , n)) _ {\forall v \in \mathcal {V}}, \tag {20}
$$
there exists a bijective measurable map between the nodes in  $S$  and their positions in the representation vector  $\gamma_{\pi}$ . Next, we consider the representation set  $\mathcal{O}_S(\mathbf{A})$  restricted to the node subset  $S$  as follows,
$$
\mathcal {O} _ {S} (\mathbf {A}) := \left\{\Gamma^ {*} (v, \mathbf {A}, \pi (1, 2, \dots , n)) _ {\forall v \in S} \mid \pi \in \Pi_ {n} \right\} = \left\{\left(\left(\mathbf {Z} _ {v} ^ {*}\right) _ {v \in S} | \mathbf {A}\right) \right\}, \tag {21}
$$
and prove that there exists an surjection between  $\mathcal{O}_S(\mathbf{A})$  and  $\Gamma^{*}(S,\mathbf{A})$ . The surjection exists if,  $\forall$  non-isomorphic node subset  $S_{1}, S_{2}$ , it implies  $\mathcal{O}_{S_1}(\mathbf{A}) \neq \mathcal{O}_{S_2}(\mathbf{A})$ . This condition naturally holds if  $|S_{1}| \neq |S_{2}|$ . When  $|S_{1}| = |S_{2}|$ , we prove by contradiction and assume  $\mathcal{O}_{S_1}(\mathbf{A}) = \mathcal{O}_{S_2}(\mathbf{A})$ . Since the node indices is unique and  $\Gamma^{*}$  is most-expressive, the representation  $\Gamma^{*}(v,\mathbf{A},\pi (1,2,\dots ,n))$  of node  $v$  and permutation  $\pi$  is unique too. As  $S_{1}$  is non-isomorphic to  $S_{2}$ , there must exist at least one node  $u \in S_{1}$  that has no isomorphic equivalent element in  $S_{2}$ . Hence,  $\exists \pi \in \Pi_n$  that provides a representation  $\Gamma^{*}(u,\mathbf{A},\pi^{\prime}(1,2,\dots ,n))$  and  $\exists \pi^{\prime} \in \Pi_{n}, v \in S_{2}$  that the corresponding representation
(c) Prediction performance on Graph-SST2
Figure 8: The hyper-parameter analysis on the number of the basic pattern-learners.
(d) Prediction performance on Spurious-Motif
$\Gamma^{*}(u,\mathbf{A},\pi^{\prime}(1,2,\dots ,n))$  matches. Finally, we conclude a contradiction from the original assumption  $\mathcal{O}_{S_1}(\mathbf{A}) = \mathcal{O}_{S_2}(\mathbf{A})$  and we know the surjection between  $\mathcal{O}_S(\mathbf{A})$  and  $\Gamma^{*}(S,\mathbf{A})$  does exist.
Furthermore, it has been proved that for finite multisets with real number elements, a most-expressive multiset function can be defined as the expectation of a function  $f^{(|S|)}$  over the multiset. Therefore, there exists some subjective function  $f^{(|S|)}$  whose expectation over  $\mathcal{O}_S(\mathbf{A})$  gives  $\Gamma^*(S, \mathbf{A})$ .
Next, we restate and prove Theorem 2.
Proof. According to Theorem 1,  $(\mathbf{Z}_v)_{v\in \{i,j\}}|\mathbf{A}$  can be represented as follows,
$$
\left(\mathbf {Z} _ {v}\right) _ {v \in \{i, j \}} | \mathbf {A} = \varphi \left(\Gamma (v, \mathbf {A}) _ {v \in \{i, j \}}, \epsilon_ {\{i, j \}}\right), \tag {22}
$$
where the noise  $\epsilon_{\{i,j\}}$  is marginalized from an independent noise distribution. With an assumption of  $f^{(2)}$  that is able to capture the structural dependencies within the adjacent matrix  $\mathbf{A}$ , we can compute the expectation of  $(\mathbf{Z}_v)_{v\in \{i,j\}}|\mathbf{A}$  and eliminate the noise.
H Algorithm
We present the algorithm of graph-hypergraph transformation as following.
Algorithm 1 Graph-Hypergraph Transformation
Input: Edge index  $\Omega \in \mathbb{R}^{2\times |\mathcal{E}|}$  of raw graph  $G$   
Output: Hyperedge index  $\Omega_h$  of hypergraph  $G_h$   
1: initialize hyperedge set  $\mathcal{E}_h\Leftarrow \emptyset$   
2: initialize hyperedge index  $\Omega_h\Leftarrow \emptyset$   
3: for edge  $e_i = (u,v)$  in  $\Omega$  do  
4:  $\mathcal{E}_h^u\Leftarrow \mathcal{E}_h^u\cup \{i\},\mathcal{E}_h^v\Leftarrow \mathcal{E}_h^v\cup \{i\} //$  Record the endpoints of each edge  
5: end for  
6: for the  $p$  th hyperedge  $\mathcal{E}_h^p$  in  $\mathcal{E}_h$  do  
7: for item  $q$  in  $\mathcal{E}_h^p$  do  
8:  $\Omega_h\Leftarrow \Omega_h\cup \{(p,q)\} //$  Allocate the same index  $p$  for each individual hypernode  $q$   
9: end for  
10: end for  
11: return  $\Omega_h$
We also provide an illustrative example of the transformation process in Figure 2 of Section 3.1.
At last, we provide open discussion about the limitations of the  $\pi$ -GNN model.
Efficiency. If the number of basic pattern-learners need to be increased to a large amount for some intricate graph datasets, the computational efficiency of  $\pi$ -GNN will become the bottleneck. Although the efficiency can be improved by introducing multi-thread computation, the time complexity of  $\pi$ -GNN with multiple basic pattern-learners is higher than the current interpretable GNNs.
Feature Dimension. The feature dimensions of the downstream graph datasets are usually different from that of the PT-Motifs dataset in pre-training phase. Therefore, we have to additionally introduce a linear layer to align the dimension differences between PT-Motifs and the downstream datasets. Though this linear layer can be merged into the hypergraph refining module, it indeed increases the optimization difficulty when fine-tuning.
Node Feature.  $\pi$ -GNN focuses on identifying the influential subgraphs by computing the edge contribution score, but it is unable to select the important fraction of the node features that leads to the model prediction. As a future direction, we consider to systematically extend  $\pi$ -GNN to the interpretation problem in node classification and link prediction task, where the node features are more informative and influential than those in the graph classification task.
Unseen Structural Patterns. As shown in Appendix B.2, several structural patterns are universal and generalize to the synthetic PT-Motifs dataset and the real-world datasets (e.g., Mutag, MNIST-75sp, Molhiv, Graph-SST2). During the pre-training phase over PT-Motifs dataset,  $\pi$ -GNN extracts these universal structural patterns and combines them with the local structural interactions to achieve generalizable interpretation. But for some unseen structural patterns that does not exist in PT-Motifs,  $\pi$ -GNN is unable to capture and then employ them to identify explanations.