Train Once and Explain Everywhere: Pre-training Interpretable Graph Neural Networks
Authors: Jun Yin
Chaozhuo Li
Hao Yan
Jianxun Lian
Senzhang Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Intrinsic Interpretability
Graph Neural Networks
Pre-training and Fine-tuning

Abstract: Intrinsic interpretable graph neural networks aim to provide transparent predictions by identifying the influential fraction of the input graph that guides the model prediction, i.e., the explanatory subgraph. However, current interpretable GNNs mostly are dataset-specific and hard to generalize to different graphs. A more generalizable GNN interpretation model which can effectively distill the universal structural patterns of different graphs is until-now unexplored. Motivated by the great success of recent pre-training techniques, we for the first time propose the Pre-training Interpretable Graph Neural Network ($\pi$-GNN) to distill the universal interpretability of GNNs by pre-training over synthetic graphs with ground-truth explanations. Specifically, we introduce a structural pattern learning module to extract diverse universal structure patterns and integrate them together to comprehensively represent the graphs of different types. Next, a hypergraph refining module is proposed to identify the explanatory subgraph by incorporating the universal structure patterns with local edge interactions. Finally, the task-specific predictor is cascaded with the pre-trained $\pi$-GNN model and fine-tuned over downstream tasks. Extensive experiments demonstrate that $\pi$-GNN significantly surpasses the leading interpretable GNN baselines with up to 9.98\% interpretation improvement and 16.06\% classification accuracy improvement. Meanwhile, $\pi$-GNN pre-trained on graph classification task also achieves the top-tier interpretation performance on node classification task, which further verifies its promising generalization performance among different downstream tasks. Our code and datasets are available at https://anonymous.4open.science/r/PI-GNN-F86C

Train Once and Explain Everywhere: Pre-training Interpretable Graph Neural Networks
Jun Yin' Central South University yinjun2000@csu. edu. on
Chaozhuo Li* Microsoft Research Asia i @mi crosoft. com
Hao Yan Central South University CSUyh19990cu. edu. on
Jianxun Lian Microsoft Research Asia j jaaxuun.. l aian@mi crosoft. com
Senzhang Wang
Central South University szwangScsu. edu. cn
Abstract
Intrinsic interpretable graph neural networks aim to provide transparent predictions by identifying the influential fraction of the input graph that guides the model prediction, i.c., the explanatory subgraph. However, current interpretable GNNs mostly are dataset-specific and hard generalize different graphs. A more generalizable GNN interpretation model which can effectively distill the universal structural patterns of different graphs until-now unexplored. Motivated by the great success of recent pre- pre-training techniques, we for the first time propose the Pre-training Interpretable Graph Neural Network (7-GNNN) to distill the universal interpretability of GNNs by pre-training over synthetic graphs with ground-truth ex- planations. Specifically, we introduce structural pattern learning module to extract diverse universal structure patterns and integrate them together to comprehensively represent the graphs different types. Next, a hypergraph refining module proposed to identify the explanatory subgraph by incorporating the universal struc- ture patterns with local edge interactions. Finally, the task-specific predictor is cascaded with the pre-trained N-GNN model and fine-tuned over downstream tasks. Extensive experiments demonstrate that π GNNN significantly surpasses the leading interpretable GNN baselines with up to 9 9.98% interpretation improvement and 16.06% classification accuracy improvement. Meanwhile, r-GNN pre-trained graph classification task also achieves the top-tier interpretation performance on node classification task, which further verifies its promising generalization performance among different downstream tasks.
Introduction
Although graph neural networks (GNNs) 2 have achieved remarkable success in various applications [6.
7 g 9]. their black-box nature prevents humans from understanding the inner decision-making mechanism 1T0 MTI. This issue calls for the development of intrinsic interpretable GNNs 12] [13 14), which can reveal the mystery of "Which fraction of the input graph is the most vital and leads the model prediction?" Intrinsic interpretable GNNs aim identify an influential subgraph of the input graph, i.c., the explanation, and make final predictions under the guidance of the Constructing interpretable GNNs makes possible investigate
explanatory subgraph [2
14
"Equal contribution.
i Corresponding author.
Our code and datasets are available at http://aaooyynoosssoooruuseeeeea F86C
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

the decision maakin mechanism and justify model predictions, which is critically important to develop trustworthy artificial intelligence.
However, a major issue of the existing interpretable GNNs 12 13 is that they are mostly
dataset-specific, which means interpretable GNN model trained certain graph dataset (e.g., the semantic networks [15] [6 (7] usually does not work well on another graph dataset (c.g., the molecular graphs (18). It also difficult for an interpretable GNN trained on certain task (e.g.. graph classification 7 to generalize to another task (e.g., node classification [6]. Compared to the text or image data [99 20]. graphs with -un-iuille structure are often associated with substantial node features of various domains, making the pre-training method in NLP and CV hard be directly applied [21]. Recently graph representation learning based pre- pre-training methods have been investigated [22 21 [3 24). however, they target at downstream prediction tasks but ignore the interpretability the GNNs. is widely acknowledged that the topological structure of various graphs generally follows some universal structural patterns or properties which are transferable [22] such as the scale-free property (25), the motif distribution [26]. and the core-periphery structure 277.. Therefore, in this paper we argue that the intrinsic GNN interpretation contains some universal structural patterns, which are independent of the downstream tasks and generalizable to different types of graphs. Motivated by the great success of pre-training technique [19 20 22, 21, we for the first time study: whether we can and how to construct a pre-training interpretable GNN that is general enough work well different rypes of graphs and downstream tasks?
The challenges of designing a pre-training interpretable GNN are three-fold. First, labeling ground- truth explanation in the real-world graphs is extremely resource- and time-comsuming (28). The lacking of ground-truth explanation makes it hard distill the universal interpretabílity in the pre- training phase. Second, multiple structural patterns usually co-exist in one graph dataset, such as the scale-free pattern and the motif distribution pattern in chemical molecule graphs [29]. How to extract and integrate multiple structural patterns for more comprehensive and general graph representation during pre-training is also challenging. Third, the local structural interactions such as the neighbor edges interaction (30 should be considered when identifying explanations. However, due to the structural diversity of different local neighborhoods, it is non-trivial to incorporate the global structural patterns with the local structural interactions.
In this papaer, we propose a Pre-training Interpretable Graph Neural Network model (T-GNN for short), which is first pre-trained over large synthetic graph dataset with ground-truth explanations and then fine-tuned on different downstream datasets and tasks. Specifically, we first construct a synthetic graph dataset named PT-Motifs that contains various structural patterns and ground truth explanations, over which the π- GNNN model is pre-trained. Considering the co-existence of multiple structural patterns, structural pattern learning module is introduced to extract and integrate multiple structural patterns to make them generalizable to diverse graph datasets. To better identify the explanatory subgraph, hypergraph refining module is also proposed to capture the local structural interaction and incorporate it with the universal structural patterns. It is also proved the structural representation ability of T-GNN can approach the theoretical upper bound through the hypergraph refining process. Our main contributions are summarized as follows.
We for the first time propose a pre-training interpretable GNN model T-GNN, which can be generalized to different graph datasets and diverse downstream tasks.
To extract the universal interpretability of G-NNN, we construct a synthetic graph classification dataset PT-Motiis with tround-truth explanations for pre-training.
Two innovative modules, i.c., the structural pattern learning module and the hypergraph refining module, are designed and integrated into GNGNN The former captures and integrates multiple universal structural patterns for generalizable graph representation. The latter incorporates the universal patterns with local structural interactions to identify the explanation.
Compared with the SOTA baselines, T-GNN achieves up to 9.98% ROC-AUC improvement in interpretation and 16.06% accuracy improvement in prediction. Moreover, T-GNN pre-trained on graph classification dataset is able to achieve comparable performance with the leading baselines on the node classification task.

---

2 Background
In this section, we briefly introduce the graph neural networks and the GNN explanation methods The key notations are summarized in Appendix A for clarity.
Graph Neural Networks. Graph structure data can be denoted as G (V, with the node set V and the edge set &. The node features are represented as the matrix X E R and the edge features (if exist) are represented as X R (e)x,, The topologiaa structure is usually represented as an adjacency matrix A E I R where the element A.j indicates the edge (i.j) exists and Aij- = otherwise. Graph neural networks (GNNs) aim to learn expressive representation on graphs such as graph classification 7 node classification Typically, to learn the representation of node uy GNNs aggregate the information from its neighborhood (u;, and then combine it with 2 's own features. For example, the operation of the k-th GCNs layer can be formulated as follows [T]
for the downstream tasks 1 2 3
32 33
[4] and link prediction 35 36]
X = -D
(1)
where Xk and Xk+1 are the input and output of the k-th layer. Å A= A i the adjacent matrix with self-loops, and D is a diagonal matrix whose element Di, represents the degree of us W is trainable matrix and F()) is non-linear activation function.
GNN Explanation Methods. GNN explanation methods can be catogrized into the post-hoe hoc
explanation methods [I 38 39 and the intrinsic interpretable methods B3 40 12 [3 14).
The post-hoc methods target at explaining the black-box models which are fixed or unaccessible. The interpretable methods devote making transparent prediction from scratch, including not only the predicted label but also the influential subgraph that guides the prediction 14). In both the
post-hoc [10
37 39] and the interpretable methods [ 13 14]. the GNN explainer learns a contribution function h which maps each feature of the input graph into the contribution score to the predicted label. One insight in GNN explanation methods is that, the edge contribution function
is more essential to GNN explanation compared with that of the node [11 13 14). For example, when some nodes are selected, is non-trivial to identify the explanatory subgraph. On the contrary, when the important edges are selected, the correlated endpoints are naturally selected as well. We can naturally identify the explanatory subgraph or further explore the important subset of node features. In this work, we follow the previous works [[]] 39 13
and focus the contribution of structure features (i.e., edges). Formally, we learn the contribution function h in terms of each edge in graph G = EV as follows,
- -= hG),
(2)
where و RI and each element in ő is the contribution score of the edge the task label. Next, selection module S as follows is employed to select the edges of the explanatory subgraph g, such as the top-k selector 113, the threshold selector 100 and the probabilistic selector [30 [4]
= p)
(3)
3 Methodology
The framework of the proposed TT-NNN is shown in Figure which contains an explainer pre-training phase and a conjoint fine-tuning phase. In the explainer pre-training phase, we pre-train the r-GNN explainer over the synthetic dataset PT-Motifs with ground-truth explanations, by taking the binary edge classification the pretext task. Afterwards, the pre-trained r-GNN explainer is incorporated with task-specific predictor to identify explanatory subgraphs and provide transparent predictions on different tasks. During the fine-tuning phase, the explainer and the predictor are conjointly optimized. Next, we introduce the 7 GNNN model in detail.
3.1 Explainer Pre-training Phase
Due to the lack of ground-truth explanation in real-world graph datasets, we first construct large synthetic dataset with ground-truth explanation called PT-Motif's to support the explainer pre-training.
In what follows, we use the term explainer and interpretor interchangeably
311
3

---

Syrtheicggapps
Explaier pre-traiiinggaaae
Granddrrthespkmaiin
Structural Pattern
Lammnng Mochlee
Hypargaahh Refining
Module
91
92
8
84
95
Explainne
tratwed
explaratioon
Structural Pattern
Taske Tiskk-Speciliic
Lemming Mochile
Predictor
Hlpergraph Refining
Explaines
geal-warddgrapls
Conjeint fime-utnnnsnttnnnngyhehh
Explaration
Predict Loss
Figure 1: Framework of the r-GNN model. In the explainer pre-training phase, we use the synthetic graphs with ground-truth explanations to pretrain the r-GNN explainer. In the fine-tuning phase towards downstream tasks in real-world graphs, the pre-trained explainer and the task specifc predictor are conjointly fine-tuned for transparent and accurate predictions.
Following previous works on generating synthetic graphs [T
each graph in PT-Motifs dataset consists of one base subgraph G2 and explanation subgraph Ge (also known the motif)
and the ground -oound-truth truth task label which determined by Ge solely [10 [3]. The shapes of the
explanatory subgraphs in PT-Motifs include Diamond, House, Crane, Cycle, and Star and the basic shapes are Clique, Tree, Wheel, Ladder, and the Barabási-Alberl Net [25). Each combination of base and motif has the same number of samples in P'T-Motifs. As shown in Figure 4(e) of Appendix
[..2, the degree distribution of PT-Motifs follows the power law function = 10 x a -1.001 See
Appendix for more detailed structural patterns of PT-Motif's The proposed T-GNN model distills interpretability from the PT-Motifss dataset during the pre- pre-training phase.
Structural Pattern Learning Module. To capture the multiple structural patterns, we propose to parallelize multi-threat of basic pattern-leamer = BB { pattern-lend -eaterr-learerr aggregates them for more comprehensive and general representation of various graph structural patterns. Specifically, each basic learner B, identifies a vectorized representation of the structural patterns (such the degree distribution) and the integrated learner t provides a combination of each individual representation. We formally define the basic pattern-learner as follows.
1,2. ................................ Then, an integrated
Definition (Basic Pattern-Learner). Consider a graph G with n nodes, whose adjacency matrix is A E R=x" The basic pattern-learner B; projects the adjacency matrix A into low-dimensional pattern matrix Z₁ E Ruxd d < n, as follows,
Z: (A).
The basic pattern-learner B; approaches the low-dimensional pattern matrix Z₃ by maximizing the likelihood of preserving the graph topological structure.
As widely adopted technique which provides global views of a graph [4] 42 node embedding can serve as a simple yet effective basic pattern-learner. Inspired by the multi-head attention mechanism for improving the expressive power [4). parallelize N-thread basic pattern-leamers to achieve the structural pattern tensor 2 IZ, Zz, Zn] E R Xxxx which contains multiple universal structural patterns. For each basic pattern-leamen the adjacency matrix A is randomly permuted 42 44. Afterwards, the integrated learner & defined as follows aggregates the pattern tensor for a more expressive and generalizable pattern representation Zent [45)
Definition 2 (Integrated Ceatern-ILearner. Given the structural pattern tensor E R xxed the integrated pattern-learner & moves forward convex combination Zint E R ux of each

---

G (V,8)
Gp = (Vr, En)
Graph-hypergraph Transformation
B
V (A,
V/ [1,2,3,4,5}
5
2
3 = 31,5,3,4,5)
ε (A,,,C,
3
3
Figure 2: An example of the graph-hypergraph transformation. Considering the 3-degree node A in
graph G, its corresponding hyper-view is the hyperedge A which connects the hypernodes {1, 4, 53.
individual patlern matrix Z; as
N
Zint = ((2) Zi:
(5)
where O is an integration function to comprise appropriate structural patterns for diverse graphs.
Subsequently, Zim is fed into the hypergraph refining module for computing the edge contribution score and exploring explanatory subgraph.
Hypergraph Refining Module. With the universal structural patterns extracted from the graphs, we next incorporate with the edge interactions identify the explanatory subgraph. Specifically, we first learn the edge structural representation Zg from the integrated pattern matrix Zint Taking the edge j) as an example, the corresponding structural representation Z can be approached from the endpoint representations in Z as follows,
Z (2) (Zi (m
(6)
where ((2) is 2-variable function and Z Zj are the representations of nodes 04 My respectively.
Since the edge representation ZE directly determines the edge contribution score in the explanation, should express as much information possible. Theoretically, the expressive power Zg is guaranteed by the following Theorem 1.
Theorem 1. Let En be the of all adjacency matrix A with nodes. Given graph G (V,E) E,,n n 2 2, let [S A) be a most- expressive structural representation of nodes set S S V in G. AA E.. there exists mosi-expressive node representation Z A satisfies the relationship as follows,
(*( = 1:000=----- (F(1s))((22))))))) (())) VV
for appropriate k-variable function (-)
Theorem 1 defines the upper bound of Zæ representation ability 44 The upper bound can be approached according to the Theorem 2 as follows.
Theorem 2. The structural representation of edge (v,, U3) can be learnt by simply approaching function f(2) which satisfies [(e, A) ( (E( AA).
In the hypergraph refining module, we adopt the integrated pattern matrix Zags as an approximation of the expectation embedding matrix IE (Z,))eee(()) A| and 2-layer MLP to fit the transition
function (2) during pre-training phase 44 See Appendix
G for the proofs Theorems I and 2.
Based on the edge structural representation Zg a straightffowwad way is to directly estimate the con- tribution score of each individual edge 313 However, such procedure ignores the incorporation of universal patterns with local structural interactions, leading to the missing of the dependencies among the edges. The edges in the explanation supposed to interact with each other [46), form the coalition, and guide the downstream prediction task better than individuals. We next introduce how to capture the local edge interactions and incorporate them with the universal structural patterns.
Given the graph (V, E), we define its corresponding hypergraph as G, (Vn, En). For dedderee node E V. the corresponding hyper-view view is hyperedge connecting & hypernodes.
5

---

Similarly, the edge e E & becomes hypernode in the corresponding hypergraph. An example of graph- hypperrapg transformation is shown in Figure See Appendix Horr the detailed algorithm of wraph-hypergraph transformation. During the graph-hypergraph transformation, the edge structural representation is naturally converted to the hypernode structural representation. Therefore, we can capture the local structural interaction by conducting hyperedge message passing. In our imple- mentation, a 2-layer hypergraph convolutional network is employed to capture the edge interaction information and further map the edge structural representations to the edge contribution scores Finally, the hypergraph refining is formally represented as,
(2) Tanh Tanh(HyyerComy (Z))),
(8)
I o (HyperConv(2) (1) ))
(9)
where the Tanh(-) function is used for zero-mean activate value and the af-) function normalizes the output to probability value within the range of [0, 1]. The normalized contribution score in measures the probability of each edge belongs to the explanatory subgraph.
Under the supervision of the ground-truth explanation in PT-Motifs, T-GNN distills the universal interpretability during the pre-training phase. Specifically, for each synthetic graph G (V,8), the edges in & that belong to the explanation subgraph are assigned with positíve labels and the complementary part EN receives negative label [TO] I1. We denote the ground-truth explanation as
/ E {0, 1) Jel, T-GNN takes G as input and outputs the predicted explanation [0, 1](81 which is optimized by the binary cross-entropy loss as follows,
L(O,p)
p logI: + (1 -pi) -log(1 p.)].
(10)
3.2 Conjoint Fine-tuning Phase
During the conjoint fine-tur suee-tuning phase, we combine the pre-trained π T-GNN explainer with task- specific predictors to identify the explanatory subgraph and make final prediction simultaneously for real-world datasets. Note that the pre- pre-trained T-GNN explainer is orthogonal to the post post-positional predictor. That is, we can implement a predictor with arbitrary architecture as long it deal with the graph structure data. Even though the pre-training dataset PT-Motifs belongs to graph classification task, the pre-trained r-GNN model can be easily generalized to other tasks, such as nodel classification, by simply implementing a node classifier.
Following existing works ||| 13 14, given the input graph G and the corresponding task label y. we introduce a probabilistic sampler S to comprise the explanatory subgraph g according to the predicted edge probability á as follows,
= S(,
(11)
Going beyond the probabilistic sampling procedure, the post-p post-positional predictor takes the explanatory subgraph g input and fits the mapping function to the predicted label ÿ. by optimizing task- specific loss function Lrask (Ø, y). In addition, we introduce an entropy regularizer [T] in the fine-tuning phase amplify the gap of each value in ý for sparser explanations follows.
[El
M(p) = -ie
lā= log;: + (1 pc) log(1 - -A)] + 1löll
(12)
The overall objective of the fine-tuning phase is to jointly optimize the the task term and the entropy regularizer as follows,
L(g. P. v) = Leaste (0. + L(p).
(13)
4 Experiment
In this section, we conduct extensive experiments to evaluate the performance of r-GNN by answering the following two questions.
RQ1: How effective --GNN when it is generalized different graph datasets?
RQ2: How effective is π T-GNN when it is generalized different graph tasks?

---

Table 1: Interpretation Performance (ROC-AUC) Comparison. The underlined results highlight the best baselines. The bold results mean the N-GNN or T-GNNDFT outperform the best baselines.
Spurious- Motif b = 0.7
Model
BA-2Motifs
Mulag
MNIST-75sp
b 0.5
b = 0.9
GNNExplainer
67.35 +: 3.29
61.8 + 5.45
59.01 + 2.04
62.62 + 1.35
62.25 = 3.61
58.86 + 1.93
PGExplainer
84,59 9.09
60.91 = 17.10
69.34 4.32
69.54 + 5.64
72.33 9.18
72.34 = 2.91
GraphMask
92.54 +
62.23 9.01
73.10 6.41
72.06 +5.58
73.06 4.91
66.68 6.96
IB-Subgraph
86.06 +: 28.37
91.04
11.20:5.111
57.29 1 14.35
62.89 + 15.59
47.29 13.39
DIR
82.78 + 10,97
64.44 = 28.81
32.35 9.39
78.15 + 1.32
77.68 = 1.22
49.08 = 3.66
GIN-GSAT
98.74 0.55
99.60 +
83.36 1.02
78.45 + 3.12
74.07 = 5.28
71.97 = 4.41
INA-GSAT
93.77 + 3.90
99.07 0.50
84.68
1.06
83.34 + 2.17
86.94 4.05
88.66 2.44
g-GNN
99.33 + 0.63
99.81 0.17
92.77 = 0.80
93.24 + 0.72
96.92 = 0.85
96.39 = 0.92
T-GNNDFT
93.19 1.48
95.29 0.67
85.18 1.08
86.29 + 2.22
87.43 2.47
89.64 - 2.26
4.1 Experimental Settings
In the experiment, we use two popular synthetic datasets [1
[4] and four real-world datasets of graph classification tasks. The details of dataset characteristics and statistics are summarized in Appendix B A brief introduction to the datasets is follows.
13
Synthetic Datasets. BA-2Mottffs 10 and Spurious- Motif I3 are two widely-used synthetic
datasets to evaluate the interpretation performance of the GNN explanation methods.
Real-world Datasets. We use four real-world datasets, the superpixel graph dataset MNIST-75sp 440, the sentiment analysis dataset Graph-SST2 [16), and two chemical molecule datasets Mutag and Ogbg-Molhiv 47] Note that, in the MNIST-75sp dataset, the subgraph with nonzero pixel values is regarded as the ground -oound-truth explanation [14] in the Mutag dataset, -NO₂ and -NH₂ functional groups in mutagen graphs are labelled as the ground- truth explanation (28). Hence, we use the MNIST-75sp and Mutag datasets for both the interpretation and the prediction evaluations.
We extensively compare 7- T-GNN with the following two types of baselines:
Interpretation Baselines. We compare the interpretation performance with both the post-hoc explanation methods including GNNExplainer [10]. PGExplainer [IT], and GraphMask (39) and the intrinsic interpretable methods including DIR [33, IB-subgraph [12] GIN-GSAT, and PNA- GSAT [1]. Following the standard setting, the evaluation metric is the explanation ROC-AUC
13 14
Prediction Baselines. We compare the perdiction performance with the powerful GNN models including GIN 4 and PNA (32] and the intrinsic interpretable methods including DIR, IB-subgraph, GIN-GSAT, and PNA-GSAT. For the OGBG-Molhiv dataset, we use the classification ROC-AUC as the prediction metric 47). For all the other dataset, we report the classification accuracy [3
Additionally, report the performance of 7-GNN which directly fine-tunes on downstream datasets without pre-training (denoted as the x R-GNNDFT), to investigate the effectiveness of pre-training phase. All the results are averaged over 10-times evaluation with different random seeds. Architecture of the downstream GNN predictors used in π r-GNN and π D- are reported in Appendix C
4.2 Main Results (RQ1)
To investigate the effectiiveness of GGNN, we compare the interpretation and the prediction perfor- mance with the SOTA interpretation and prediction baselines. See Appendix C the pre-training and fine-tuning details. The overall interpretation performance and prediction performance are summarized in Table I and Table respectively. We conclude the following observations:
TT-NNN significantly outperforms the leading GNN explanation methods. Specifically, for the Spurious-Mon Motif dataset, π GNNN surpasses DIR by 27.21% % on average and by 47.31% at most. Compared with the best baselines, i.e., the GSAT with 4-layer PNA encoder, n-GNN improves the interpretation performance by 9.20% on average. However, we merely employ the combination of truncated SVD embedding and 2-layer MI.P as the encoder, which convincingly demonstrates the effectiveness of our proposed pre-training phase. Moreover, the degree of spurious correlation in
7

---

Table 2: Prediction Performance (Acc) Comparison. The underlined results highlight the best baselines. The bold results mean the π N-GNN or л GNGNNrT outperform the best baselines.
Spurious-Modd Motit b = 0.7
Model
Molliv(AUC)
Graph-SST2
MNIST-75sp
b= 0.5
b 0.9
GIN
76.69 + 1.25
82.73 = 0.77
95.74 1 0.36
39.87 +: 1.30
39.04 + 1.62
38.57 : 2,31
PNA
78.91 + 1.04
79.87 = 1.02
87.20
5.61
68.15 2.39
66.35 = 3.34
61.40 3,56
IB-bubgraph
76.43 + 2.65
82.99 0.67
93.10 1.32
48.51 5.76
46.19 5.63
DIR
76.34 1.01
82.32 0.85
88.51 2.57
45,49 3.81
41.13 2.62
37.61 2.02
GIN-GSAT
76.47 + 1.53
82.95 = 0.58
9
$2.74
49.12 3.29
44.22 = 5.57
PNA-GSAT
80.24 + 0.73
80.92 = 0.66
93.96 0.92
68.74 (8.74
64.38 = 3.20
57.01 = 2.95
T-GNN
80,86 + 0.61
88,05
0.43
96.89 + 0.20
74.67 0.63
77.52 1 0.77
77.46 = 0.96
R-GNNDFT
79.71 = 1.08
83.48
1.20
92.89 + 0.95
70.78 + 1.63
71.02 = 1.43
72.61 = 1.75
Spurious-Motif (i.e., the parameter b) increasing, the performance of T-GNN does not decrease as most the baselines. The post-hoc explainers, including GNNExplainer, PGExplainer, and GraphMask seems to be robust in terms of the spurious correlation, but their interpretability is limited by the fixed prediction model. Although DIR and GSAT introduce complex mechanism for mitigating the spurious correlation, r--GNN still performs better. We ascribe this superiority to the generalizable knowledge which is distilled from the pre-training phase over large dataset with ground-truth explanations. For the BA-2Motif, and the Mutag datasets, т T-GNN using more simpler encoder architecture also achieves comparable performance. For the MNIST-75sp dataset, T-GNN surpasses the best baseline PNA-GSAT by 4.50%. Such top-tier performance strongly validates the effectiveness of G-NNN explainer.
T-GNN also achieves better prediction performance than the baselines. Overall, π GNGNN outperforms all the prediction baselines consistently by significant margin. Specifically, for the Spurious-Moti Motif dataset, T-GNN outperforms the best baselines by 11.05% on average, which is a significant improvement. If we restrict the baselines into interpretable GNNs, the performance boost is much more significant (by 13. 3.17% on average and up to 20. 4.)...). For the Graph-SST2 dataset, T-GNN outperforms the black-box predictor (i.e., GIN and PNA), as well the SOTA interpretable methods IB-Subgraph, which optimizes the explanatory subgraph based on the information bottleneck principle. Note that, we only implement a 2-layer GCN with global mean pooling function as the Graph-SST2 predictor. We credit such outperformance to the pre-training phase, from which the universal patterns of the graphs are potentially distilled. For the MNIST- 75sp dataset, rNGNN achieves comparable prediction accuracy with the GSAT methods, but our interpretation ROC-AUC exceeds GSAT by 8.09%. Additionally, one can notice that the black-box predictors are insensitive to the spurious correlation while the interpretable baselines deteriorate obviously. This phenomena may indicate that insufficient interpretability conflicts with the chase of prediction accuracy, but a powerful interpretor is quite helpful to the subsequent predictor.
The pre-training phase advances the interpretor in terms of both interpretation and perdiction performance. As shown in Table and Table the =-GNN with pre-training phase significantly improves both the interpretation and the prediction performance, compared with the reduced variant т T-GNNDFT. For the interpretation ROC-AUC, --NN outperforms the reduced variant by 6.91 % on average. This suggests that the universal structural patterns distilled in the pre-training phase can indeed generalize to various downstream tasks and improve the interpretability. Compared with the reduced variant, T-GNN consistently provides much stabler interpretation with smaller variance. Additionally, one can observe that the reduced variant π- 1-GNNDFT still surpasses the best baselines on some real-world datasets, such the interpretation performance MNIST-75sp dataset and the prediction performance on Graph-SST2 dataset. This may imply that the edge interaction captured by r-GNNDFT is able identify the influential subgraphs more accurately.
Moreover, we present the explanatory visualization, the investigate of different pre- vee-trannig datasets and the
analysis in Appendix D and respectively.
4.3 Inter-Tase Generalization Performance (RQ2)
To further investigate whether the universal structural patterns behind different tasks is common, we next study the generalization ability across different tasks. Specifically, we evaluate the r-GNN model that is pre- mreitrained over graph classification dataset PT-Motifs on the explanation task of node
8

---

Table 3: Inter-Tast Interpretation Performance (ROC-AUC) Camparison. The bold font highlights the best results and the underlined results highlight the second best method.
Model
BA-Shapes
BA CA-Coomunity
Tree-Cycles
Tree-Grid
GRAD
88,20
75.00
90.50
61.20
Attention
81,50
73.90
82.40
66.70
GNNExplainer
92.50
83.60
94.80
87.50
PGExplainer
96.30 1: 1.10
94.50
1.90
98.70
0.70
90.70 1.40
x *-GNN
94.78 + 0.32
94.67
1,50
95.19
0.88
90.11
1.10
x GNNDDN
93.17 0.40
92.15
1.61
92.53
88.62
1.87
classification datasets. The pre-trained R-GNN model and the reduced variant are both equipped with node classifier. Following existing works, we use four widely-used synthetic node classification datasets [10], namely BA -hapees BA- -AAmmummun Tree-Cycles and Tree-Grid, whose detailed statistics are presented in Appendix B The main result of inter-task explanation in Table 3 that π T-GNN achieves comparable performan with SOTA node classification explainers. This evidence is accordant with our basic premise that the universal structural patterns can generalize across datasets of different tasks. For the BA-Community dataset, T-GNN even supasses PGExplainer with smaller varience. This may be because the community structure in the BA-Coomunity graph conforms more to the universal patterns embedded in r-GNN.
4.4 Ablation Study
As shown in Figure 3 we conduct ablation study on TTGNN, by evaluate the performance of its three variants. First, π G-GNN-SPL substitutes the structural pattern learning module with N-thread GNN encoders. Second, π T-GNN-HPR removes the hypergraph refining module and directly calculates the edge contribution score. Third, we simultaneously conduct the two ablations above and denote it as n T-GNN-ALL. Additionally, we report the performance of the variant without pre-training phase.
Specifically, when removing the structural pattern learning module, the interpretation performance on Mutag decreases by 3.85% on average and the prediction performance on Graph-SST2 decreases by 1.08% on average. For the variant π T-GNN--HI -PRR, the interpretation and prediction performance decreases by 6.56% and 1.84%, respectively. When we remove the two modules simultaneously. the interpretation and prediction performance decreases by 7.56% and 2. 2.16%, respectively. The ablation study on the two modules demonstrates their effectiveness in capturing universal structural patterns and identifying the explanatory subgraphs. Moreover, for all the variants, -NNNN consistently outperforms the variant without pre-training phase by 5.59% in terms of ROC-AUC on Mutag and 2.84% in terms of accuracy on Graph-SST2.
5 Related Work
Intrinsic interpretable GNNs. The leading interpretable GNNs 12 13 14] usually consist of an explainer module and a predictor module. The prepositional explainer takes the raw graph as input and outputs the explanation subgraph. The subsequent predictor calculates the prediction strictly relying on the explanation. Graph neural networks with attention mechanism are regarded as the initial interpretable GNNs 13),4 such as graph attention network 3. self-attention graph pooling (40) where the learned weights can interpreted as the importance certain features. Recently, invariant learning is introduced to construct intrinsic interpretable GNNs [48] [4], such as DIR [[]] argues that augmenting training data with causal intervention may assist explainer to distinguish the causal and non-causal parts. Besides, interpretable GNNs based on the information bottleneck principle [50], such as IB-Subgraph [12] and GSAT 14, are proposed to constraint the information flow from the input graph the prediction, where the label-relevant graph components will be kept while the label-irrelevant ones are reduced.
Pre-training on Graphs. The research focus of current graph pre- pre-training is the graph representation learning problem, whose objective is to learn generic encoder f(A, X) for various downstream tasks [1 52 The development of graph pre-training can be broadly divided into pre-trained graph
embeddings 53 42 and pre-trained graph encoders 23 22 21). Pre-trained graph embedding
models aim to provide good graph embeddings for various tasks, while the models themselves

---

25
R-GNN T-GNN-HPR
N-GNN--PLL GNNN-HPR
97.5
R-GNN-ALL
20
R-GNN-ALL
ROC-AUC
92.5
ROC-AUC
85
50.0
80-
87.5
85.0
75-
80.0
70-
With Pre-traing
Without Pre-training
With Pre-traing
Without Pre-training
Interpretation performance Mutag
Interpretation performance Spurious-Motif
GNN
R-GNN-SPL
74-
R-GNNN-PL
R-GNN-HPR
G- ALL
T. GNN--LLL
Accuracy
85-
68
With Pre-traing
Without Pre-training
With Pre-traing
Without Pre-training
(c) Prediction performance Graph-SST2
(d) Prediction performance on Spurious-Motii
Figure 3: The ablation study of 7-GNN Mutag, Graph-SST2, and Spurious-Motif datasets.
longer needed to the tasks. WeepWalk [4]] explores the graph embeddings by conducting random walks over graphs to generate node sequences which contain the co-occurrence relationship. Further, Node2vec 42 defines a flexible node neighborhood and proposes a biasd random walk process. The goal of pre-trained graph encoder models is generic encoder model which can deal with different tasks. Gpt-GNN [3] pre-trains a 5-layer GIN encoder across the graph-level and node-level tasks. GCC [22] introduces the contrastive learning framework pre-train the encoder
subgraph discrimination task. However, the precursor graph pre-training works are not designed to the graph explanation problem and not be directly applied the pre-training interpretable GNNs.
6 Conclusion
In this work, we for the first time investigated the universal interpretation problem in graph data and proposed the Pre-training Interpretable Graph Neural Network named r-GNN r-GNN is able to work well on different types of graphs and downstream tasks. r-GNN was pre-trained over constructed large synthetic graph dataset with ground-truth explanations to distill the generalizable interpretability. Then, TNGNN was fine-tuned on downstream tasks. Technically, we proposed an intergrated embedding module to capture and integrate multiple graph structural patterns for more generalizable representations. A hypergarph refining module was aslo proposed to incorporate universal patterns with local interaction for more faithful explanatory subgraphs identification. Extensive experiments different datasets and tasks demonstrated the promising generalizable interpretability as well as prediction performance of 7-GNN.
10

---

Acknowledgement
This research was funded by the National Science Foundation of China (No. 62172443), Open Project of Xiangjiang Laboratory (22XJ03010, 22XJ03005), the Science and Technology Major Project of Changsha (No. kh2202004), Human Provincial Natural Science Foundation China (No. 2022J130053), and the High Performance Computing Center of Central South University.
References
[1] Thomas N. Kipf and Max Welling. Semi-Supervised Classification with Graph Convolu- tional Networks. In Proceedings of ICIR, 2017.
[2] William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive Representation Learning on Large Graphs. In Proceedings of NIPS, 2017.
[3] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liò, and Yoshua Bengio. Graph Attention Networks. In Proceedings ICLR. 2018.
[4] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How Powerful are Graph Neural Networks? In Proceedings of ICLR, 2019.
[5] Hao Chen, Yue Xu, Feiran Huang, Zengde Deng, Wenbing Huang, Senzhang Wang, Peng He, and Zhoujun Li. Label-aware graph convolutional networks. In Proceedings of KDD, 2020.
[6] Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard S. Zemel. Gated Graph Sequence Neural Networks. In Proceedings of ICLR, 2016.
[7] Zhitao Ying. Jiaxuan You, Christopher Morris, Xiang Ren, William I.. Hamilton, and Jure Leskovec. Hierarchical Graph Representation Learning with Differentiable Pooling. In Pro- ceedings of NeurIPS, 2018.
[8] Yue Xu, Hao Chen, Zefan Wang. Jianwen Yin, Qijic Shen, Dimin Wang, Feiran Huang, Lixiang Lai, Tao Zhuang, Junfeng Ge, and Xia Hu. Multi-factor sequential re-ranking with perception- aware diversification. Proceedings of KDD, 2023.
[9] Senzhang Wang, Hao Yan, Jinlong Du, Jun Yin, Junxing Zhu, Chaozhuo Li, and Jianxin Wang, Adversarial hard negative generation for complementary graph contrastive learning, In Proceedings of SDM, 2023.
[10] Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitmik, and Jure Leskovec. GNNEx- plainer: Generating Explanations for Graph Neural Networks. In Proceedings of NeurIPS, 2019.
[1]] Dongsheng Luo, Wei Cheng, Dongkuan Xu, Wenchao Yu, Bo Zong, Haifeng Chen, and Xiang Zhang. Parameterized Explainer for Graph Neural Network. In Proceedings of NeurlPS, 2020.
[12] Junchi Yu, Tingyang Xu, Yu Rong, Yatao Bian, Junzhou Huang, and Ran He. Graph Information Bottleneck for Subgraph Recognition. In Proceedings of ICLR, 2021.
[13] Yingxin Wu, Xiang Wang, An Zhang, Xiangnan He, and Tat-Seng Chua. Discovering Invariant Rationales for Graph Neural Networks. In Proceedings of ICLR, 2022.
[14] Siqi Miao, Mia Liu, and Pan Li. Interpretable and Generalizable Graph Learning via Stochas- tic Attention Mechanism. In Proceedings of ICML, 2022.
[15] Nils Chr Stenseth, Wilhelm Falck, Ottar N Bjørnstad, and Charles J Krebs. Population regulation snowshoe hare and canadian lynx: asymmetric food web configurations between hare and lynx. In Proceedings of NAS, 1997.
[16] Yuan Luo Liang Yao, Chengsheng Mao. Graph Convolutional Networks for Text Classification. In Proceedings of AAAI, 2019.

---

[17] Zhoujin Tian, Chaozhuo Li, Zhiqiang Zuo, Zengxuan Wen, Lichao Sun, Xinyue Hu, Wen Zhang. Haizhen Huang. Senzhang Wang, Weiwei Deng, et al. Pass: Personalized advertiser-awer sponsored search. In Proceedings of KDD, 2023.
[18] Jeroen Kazius, Ross McGuire, and Roberta Bursi. Derivation and validation of toxicophores for mutagenicity prediction. Journal of Medicinal Chemistry, 48, 2005.
[19] Jacob Devlin, Ming- wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT, 2019.
[20] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick, Momentum contrast for unsupervised visual representation learning. In Proceedings of CVPR, 2020.
[21] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph Contrastive Learning with Augmentations. In Proceedings of NewrIPS, 2020.
[22] Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. GCC: Graph Contrastive Coding for Graph Neural Network Pre- Tre-Training. In Proceedings of KDD, 2020.
[23] Ziniu Hu, Yuxiao Dong, Kuansan Wang. Kai-Wei Chang, and Yizhou Sun. GPT-GNN: Genera- tive Pre-Training of Graph Neural Networks. In Proceedings of KDD, 2020.
[24] Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Supervised Graph Transformer on Large-Scale Molecular Data. In Proceedings of NeurlPS, 2020.
[25] Réka Albert and Albert-László Barabási. Statistical mechanics of complex networks. Reviews of modern physics, 74, 2002.
[26] Ron Milo, Shalev Itzkovitz, Nadav Kashtan, Reuven Levitt, Shai Shen-Orr, Inbal Ayzenshtat, Michal Sheffer, and Uri Alon. Superfamilies of Evolved and Designed Networks. Science, 303(5663), 2004.
[27] Stephen Borgatti and Martin G Everett. Models of core/periphery structures. Social networks, 21, 2000.
[28] Juntao Tan, Shijie Geng, Zuohui Fu, Yingqiang Ge, Shuyuan Xu, Yunqi Li, and Yongfeng Zhang. Learning and Evaluating Graph Neural Network Explanations based on Counterfac- tual and Factual Reasoning. In Proceedings of WebConf, 2022.
[29] Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. Motif-based Graph Self-Supervised Learning for Molecular Property Prediction. In Proceedings of NeurlPS, 2021.
[30] Xiang Wang, Ying-Xin Wu, An Zhang, Xiangnan He, and Tat-Seng Chua. Towards Multi- Grained Explainability for Graph Neural Networks. In Proceedings of NeurlPS, 2021.
[31] Hao Yuan, Haiyang Yu, Shurui Gui, and Shuiwang Ji. Explainability in Graph Neural Networks: Taxonomic Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 2023.
[32] Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Velickovic. Principal Neighbourhood Aggregation for Graph Nets. In Proceedings of NeurlPS, 2020.
[33] Peiyan Zhang, Yuchen Yan, Chaozhuo Li, Senzhang Wang, Xing Xie, Guojie Song, and Sunghun Kim. Continual learning on dynamic graphs via parameter isolation. In Proceedings of SIGIR, 2023.
[34] Jianan Zhao, Meng Qu, Chaozhuo Li, Hao Yan, Qian Liu, Rui Li, Xing Xie, and Jian Tang. Learning on large-scale text-attributed graphs via variational inference. In Proceedings of ICLR, 2022.
[35] Muhan Zhang and Yixin Chen. Link Prediction Based on Graph Neural Networks, In Proceed- ings of NeurIPS, 2018.
12

---

[36] Zhongyu Huang, Yingheng Wang, Chaozhuo Li, and Huiguang He. Going deeper into permutation-senan graph neural networks. In Proceedings of ICML, pages 9377-9409, 2022.
[37] Xiaoqi Wang and Han-Wei Shen. GNNInterpreter: A Probabilistic Generative Model-Level Explanation for Graph Neural Networks. In Proceedings of ICLR, 2023.
[38] Yifei Liu, Chao Chen, Yazheng Liu, Xi Zhang, and Sihong Xie. Multi-objective Explanations of GNN Predictions. In Proceedings of ICDM, 2021.
[39] Michael Sejr Schlichtkrull, Nicola De Cao, and Ivan Titov. Interpreting Graph Neural Networks for NLP With Differentiable Edge Masking. In Proceedings of ICLR, 2021.
[40] Junhyun Lee, Inyeop Lee, and Jaewoo Kang. Self-Attention Graph Pooling, In Proceedings of ICMI., 2019.
[41] Bryan Perozzi, Rami AI-Rfou, and Steven Skiena. DeepWalk: online learning of social representations. In Proceedings KDD, 2014.
[42] Aditya Grover and Jure Leskovee. node2vec: Scalable Feature Learning for Networks. In Proceedings of KDD, 2016.
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Proceedings of NIPS, 2017.
[44]
Balasubramaniam Srinivasan and Bruno Ribeiro. On the Equivalence between Positional Node Embeddings and Structural Graph Representations. In Proceedings of ICLR, 2020.
[45] Thomas G. Dietterich. Ensemble Methods in Machine Learning. In Proceedings of MCS, 2000.
[46] Christopher Frye, Damien de Mijolla, Tom Begley, Laurence Cowton, Megan Stanley, and Ilya Feige. Shapley explainability on the data manifold. In Proceedings of ICLR, 2021.
[47] Weihua Hu, Matthias Fey, Marinka Zitiik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open Graph Benchmark: Datasets for Machine Learning on Graphs. In Proceedings of NeurIPS, 2020.
[48] Shiyu Chang, Yang Zhang, Mo Yu, and Tommi S. Jaakkola. Invariant Rationalization. In Proceedings of ICMI., 2020.
[49] David Krueger, Ethan Caballero, Jömn-Herrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Rémi Le Priol, and Aaron C. Courville. Out-of-Distribution Generalization via Risk Extrapolation (REx). In Proceedings of ICML, 2021.
[50] Tailin Wu, Hongyu Ren, Pan Li, and Jure Leskovec. Graph Information Bottleneck. In Proceedings NeurIPS, 2020.
[51] Jun Xia, Yanqiao Zhu, Yuanqi Du, and Stan Z. Li. A Survey of Pretraining on Graphs: Taxonomy, Methods, and Applications. In Proceedings of IJCAI, 2022.
[52] Hao Yan, Senzhang Wang, Jun Yin, Chaozhuo Li, Junxing Zhu, and Jianxin Wang. Going deeper into permutation-sensitive graph neural networks. In Proceedings of ECML/PKDD, 2023.
[53] Jian Tang, Meng Qu, Mingzhe Wang. Ming Zhang, Jun Yan, and Qiaozhu Mei. LINE: Large- scale Information Network Embedding. In Proceedings of WWW, 2015.
13

---

A
Notations
In Table
we give the mathematical symbols and their descriptions used in this paper for clarity.
Table Symbols and their descriptions.
Symbol
Description
G, V,E
Graph instance, node set, edge set Node feature matrix
X
X
Edge feature matrix Adjacency matrix, element at the i-th row, j-th column of A
A, Ay
0, Ug e
Node instance; edge instance Neighborhood of node
((e)
W
Trainable parameter matrix
P()a()
Non-linear activation function, Sigmoid function
h
Edge contribution function Ground-truth explanation, predicted explanation
p.p
S
Selection module
g
Explanatory subgraph
B, B,
Set of basic pattern-learners, basic pattern-learner
N
Number of basic pattern- learnerrs
Z. Z
Pattern matrix, pattern tensor Integrated pattern-learne
+
Zin
Integrated pattern representation
Z
Edge structural representation Subset of node
S
f(2)
Node to edge transition function
G1. VA, En
Hypergraph instance, hypernode set, hyperedge set Positive (Negative) edge set
Ep b
Degree spurious correlation in Spurious- dataset
B Datasets
We introduce the datasets used in the main experiment follows.
BA-2Motifs 10]
a synthetic dataset with binary graph classes for evaluating the interpretation performance. House motifs and cycle motifs decide the graph labels and serve as the ground-truth explanations the two classes, respectively.
Spurious-Motii [13 is synthetic dataset with three graph classes for evaluating both the in- terpretation and the prediction performance. Each class corresponds to particular motif which is regarded the ground-trute explanation. During the synthesis process, spurious correlations, including the unbalanced base sampling and the scale drift, are injected into the training dataset. A hyper-paramest b controls the degree of spurious correlation, usually set (0.5, 0.7, 0.9).
Mutag 18] is a chemical molecule dataset with binary graph classes which represents mutagenic property. Benefiting from domain knowledge, -NO₂ and -NH, functional groups in mutagen graphs are labelled as the ground-truth explanation. Therefore, we employ the Mutag dataset for the evaluation of interpretation and prediction performance.
converts the MNIST image dataset into superpixel graph dataset with ten graph classes, The nodes are superpixels, while edges the spatial distance between the related endpoints. The subgraph with nonzero pixel values is regarded as the ground- -tunh explanation. Hence, the M5ist-75sp is used for both the interpretation and the prediction evaluations.
MNIST-75sp 40
Graph-SST2 16 is sentiment analysis dataset with binary graph labels, where each sequence SST2 dataset is transformed graph. Since no ground-truth explanations are available, we evaluate the prediction performance only.
14

---

OGBG-Molhiv 47 is a molecular dataset with binary graph labels according to the inhibition effect on HIV virus replication, where nodes are atoms, and edges are chemical bonds. As there are ground-truth explanations, we merely evaluate the prediction performance.
B.1
Statistical Characteristics
We show the detailed statistics of the datasets in Table
Table 5: Detailed statistics of graph classification datasets in main experiment.
BA AMoMMsss
Spurious- Motif
Mutag Val
Train
Val
Test
Train
Val
Test
Train
Test
#Classes
2
3
2
#Graphs
400
400
200
9,000
3,000
6,000
1000
1000
951
Avg. #Nodes
25.0
25.0
25.0
25.4
26.1
88.7
30.1
30.1
30.2
Avg. #Edges
50.9
51.0
50.9
35.4
36.2
131.1
61.3
60.2
61.2
MNIST-75sp
Graph-SST2
OGBG-Molhiv Val
Train
Val
Test
Train
Val
Test
Train
Test
#Classes
10
2
2
#Graphs
20,000
5,000
10,000
28,237
3,147
12,305
32,901
4,113
4,113
Avg. #Nodes
66.8
67.3
67.0
17.7
17.3
3.45
25.3
27.8
25.3
Avg. #Edges
539.3
545.9
540.9
33.3
33.5
4.89
54.1
61.1
55.6
We show the detailed statistics of datasets that used in the inter-task evaluation in Table 6
Table 6: Detailed statistics of node classification datasets in inter-task experiment.
BA- AA-Shapes
BA-Community
Tree-Cycles
Tree-Grid
#Classes
#Nodes
700 4,110
1,400
871 1,950
1,231
#Edges
8,920
3,410
B.2 Structural Patterns
Degree Distribution. As shown in Figure we visualize the degree distribution of the four real- world datasets, Le., Mutag, MNNIST-75sp. Ogbg-Molhiv, Molhiv, and Graph-SST2, and our synthetic scale-free dataset PT-Motifs. The power-law like degree function D(x) = cr of each real-worlh dataset is formally represented as follows, optimized by the ordinary least square error. Here D(x)) represents the number of x-degree nodes and 182 is the coefficient determination.
Mutag: (z)) 54447x R² 0.9015.
MNIST-75sp: D(z) = 10 559113 R² 0.6713.
Ogbg-Molhiv: D(x) = 3 x 10 6.039 R² = 0.6828.
Graph-SST2: D(x) = 8 10 - 1.968 R² 0.8491.
PT-Motifs: ((x)= -1.501 R2 0.9677.
In Table 7 we report the average value of the four structural patterns over each individual graphs in the datasets, including
Transitivity defined as the fraction all possible triangles present in the given graph G. measures the tendency of connections relationships between nodes to form triangles triplets and captures the clustering characteristics in a given graph. Let triads be the structure that two edges with shared node, then the transitivity can be formally represented as follows,
Triangles
Transitivity
(14)
# Trirddd
where #Triangles and # #Triads are the number of triangles and triads in G, respectively.
15

---

10000
#Nodes
Nodes
20000
10000
(a) Degree distribution of Mutag
(b) Degree distribution MNIST-755pp
500002
400000
350000
400002
300003
#odes
200203
202000
100202
10
(c) Degree distribution of Ogbg Molhiv
(d) Degree distribution of Graph-SST2
0.6
0.4
8.2
(c) Degree distribution PT-Mosifs
Figure 4:
The degree distribution of four real-world graph datasets (Mutag, MNIST-75sp, Ogbg- Molhiv, and Graph-SST2) and our synthetic dataset PT-Motifs. The axis is the node degree.
Assortativity is defined as the similarity of connections in the given graph G with respect to the node degree. measures the preference of nodes to connect with nodes of similar or different characteristics. Assortativity helps uncover the underlying patterns of connectivity and provides insights into the organization and functioning of complex networks. The average assortativity of the given graph G is the mean value over all nodes.
Efficiency of a pair of nodes is defined as the multiplicative inverse of the shortest path distance between the endpoints in the given graph G. The average global efficiency of graph is the average efficiency of all pairs of nodes.
16

---

Clustering of the node is defined as the fraction of possible triangles through that node. The corresponding clustering coefficient Cn can be formulated as,
# Triangles, Cv 2 degu -))
(15)
where #Triangles, is the number of triangles through node and deg,, is degree of node u. The average clustering coefficient C' of the given graph with n nodes is defined as follows,
1 Σ C
(16)
VEG
Table 7: Structural patterns of the PT-Motifs dataset and four real-world datasets.
PT-Motifs
Mutag
MNIST-75sp
Molhiv
Graph-SST2
Transitivity
0.3620
0.0013
0.5005
0,0021
0
Avg Assortativity
0.1525
-0.4196
0.3235
-0,2678
-0.6269
Avg Efficiency
0.5044
0.3261
0,3977
0.3243
0.5491
Avg Clustering 0.4185
0.0010
0.5408
0.0020
0
For an intuitive understanding of the graph structural patterns, we visualize the distribution of node pair efficiency and node assortativity in Figure [5 and Figure 6 respectively. One can notice that the efficiency distributions of PT-Motifss Mutag. MNIST-75sp, and Molhiv all follow a bell-shaped curve, while that of Graph-SST2 is little different. Similarly, we found that the node assortativity distributions of Mutag, MNIST-75spp Molhiv, and PT-Motifs all have one main peak which largely surpasses the adjacent values, while the node assortativity of Graph-SST2 dataset has four peaks.
Overall, the structural patterns above, including the degree distribution, the transitivity, the assor- tativity, the efficiency, and the clustering, are universal and generalizable across different datasets. However, the expression degree of different structural patterns may differ largely different datasets. As shown above, the Graph-SST2 dataset strictly follows the power law shaped degree distribution, but its node pair efficiency and node assortativity distributions are different from the other three real- world datasets. Therefore, we propose the structural pattern learning module in r-GNN, to capture multiple structural patterns and integrate them for more universal and generalizable representation.
C Experimental Details
We first present the details of the pre-training phase the synthetic PT--Motifs dataset. The PT-Motifs dataset is split into the training set of 50,000 graphs, the validation set of 10,000 graphs, and the testing of 20,000 graphs. Each graph class has equal number of instances the three sets. During the pre-training phase, the batchsize is set as {32, 64, 128, 256} and the learning rate is set as {10-3,5 5 10-3 10-4 10 10- The pre-training epoch set as {20, 40, 60, 80). We select the pre-trained - GNN model according to the validation performance. During the pre-traiee pre-training and the fine-tuning phases, we use the Adam optimizer.
As shown in Tables Tadle [ we present both the downstream predictor architecture and the fine-tuning details of the graph classification datasets and the node classification datasets, respectively. For all the graph classification datasets, we use the global mean pooling as the pooling function. Specially, in the Molhiv predictor, we introduce the virtual node and weighted loss tricks mitigate the class- imbalance issue (39,684 negative examples and only 1,443 positive examples). All experiments are conducted on single NVIDIA GeForce 3090 GPU (24GB).
DD Explanatory Visualization
For an intuitive understanding on the π -NNN explanation, we present a few visualized explanation results of the Graph-SST2 dataset in Figure 7 Overall, T-GNN has the ability to highlight the influential phrases that directly express the positive or negative sentiments in the sentence. Specifically, T-GNNN correctly allocates large weights to the positive words legend" Figure((() as well as "bewilderingly brilliant" and "entertaining" in Figure 7(c) Furthermore, the negative word, such
17

---

#Nodes
(a) Node pair efficiency distribution of Mutag
(b) Node pair efficiency distribution of MNIST-75sp
300
#Noes
(c) Node pair efficiency distribution of Ogbg-Molhiv
(d) Node pair efficiency distribution Graph-SST2
12000
8000
#Nodes
0000
2000
0.10
0.25
0.55
0.60
0.65
0.70
(c) Node pair efficiency distribution of PT-Motifs
Figure 5: The node pair efficiency distribution of four real-world graph datasets (Mutag, MNIST-75sp.
Ogbg-Mole Bolg-Molliv, and Graph-SST2) and the synthetic dataset PT-Motifs.
Table 8: Predictor architecture and fine-tuning details of graph classification datasets.
BA- BA-2Motifs
Spurious- Motif
Mutag
MNIST-75sp
Molhiv
Graph-SST2
Backbone
GCN
GCN
GCN
GIN
GIN
GCN
Layers
1
2
Batchsize
256
128
64
256
128
32
Learning rate
10
x 10
x
10
x 10
Epochs
30
30
30
50
40
40
as "absurd lengths" and "skip dreck" in Figure 7bb) and Figure ((d) are identified by T-GNN for transparent sentiment prediction. The visualized explanation again demonstrates the effectiveness of π r-GNN: (1) the pre-trained r-GNN interpretor can faithfully extract the most vital subgraph that contains the label-relevant information; and (2) the pre-trained T-GNN interpretor is able to cooperate with specific downstream predictor for both accuracy and interpretability. See Figure 7 for more visualized results of the explanations.
E Supplement Experiment
To further investigate the impact of the pre-training dataset on the final results, we conduct emprical studies on the size and the imbalance degree of the pre-training datasct.
18

---

#NNoes
(a) Node assortativity distribution of Mutag
(b) Node assortativity distribution of MNIST-75sp
Nodes
(c) Node assortativity distribution of Ogbg- Molhiv
Node assortativity distribution of Graph-SST2
4000
3006
#Nodes
3000
1000
-2.0 -0.00 -ass -ise wads wiod wild vore wi. vin 12.0 -006 un on
xic
ass
6.5
(c) Node assortativity distribution of PT-Motifs
Figure 6: The node assortativity distribution of four world graph datasets (Mutag, MNIST-75sp. Ogbg-Molhiy and Craph-SST2) and our synthetic dataset PT-Motifs.
Table 9: Predictor architecture and fine-tuning details of node classification datasets.
BA-Shapes
BA-Community
Tree-Cycles
Tree-Grid
Backbone
GCN
GCN
GCN
GCN
Layers
3
3
3
Learning rate
1 x 10-1
1 x 10-1
x 10
5 x 10
Epochs
20
20
20
20
First, compared with PT-Motfis (80,000 graphs) in the main experiment, we generate another two datasets with different sizes. In detail, PT-Motifs-M M with 50,000 graphs and PT-Motifs-S with 10,000 graphs represent the middle-level and the small-level pre-training datasets, respectively. The split ratio of the training, validation, and testing set is {0.7, 0.1, 0.2) in PT-Motifs-N. M and PT-Motifs-S As shown in Table I0, the results show that even the PT-Motifs-S is able to outperform the model without pre-training. Moreover, we can notice that a large pre-training dataset can indeed improve the performance more significantly than that with small size.
Furthermore, is possible that if the synthetic pre-training dataset imbalanced, the performance improvement on downstream tasks will degrade, in terms of both the interpretation and prediction. To further investigate this issue, we have added supplement experiments on imbalanced pre-training datasct. Following existing works 33 to generate the imbalanced datasets, we sample the
19

---

with
legend
lengthe
(a) Explanation subgraph: positive sentiment
(b) Explanation subgraph: negative sentiment
bowlideringly
and
dreck
antertaining
(c) Explanation subgraph: positive sentiment
(d) Explanation subgraph: negative sentiment
picture
(c) Explanation subgraph: positive sentiment
(f) Explanation subgraph: negative sentiment
very
Joke
Funny
(g) Explanation subgraph: positive sentiment
(h) Explanation subgraph: negative sentiment
to
genuinely
dime
be
satistying
auminations
(i) Explanation subgraph: positive sentiment
Explanation subgraph: negative sentiment
the
perfect
action
(k) Explanation subgraph: positive sentiment
Explanation subgraph: negative sentiment
Figure 7: Visualization G-GNN explanation on the Graph-SST2 dataset. Each graph represents comment, where the explanations highlighted by bule boxes and bolder lines.
Table 10: The impact of pre-training datasets with different sizes.
Task
Dataset
PT-Motifs
W/O Pre-train
PT-Motifs-- M
PT-Motifs-S
BA-2Motifs
99.33
93.19
98.91
97.07
Interpretation
Mutag
99.81
95.29
99.06
96.28
Ogbg-Molhiv Graph-SST2
80.86
79.71 83,48
80.77 87.59
79.82
Prediction
88.05
85.23
explanatory G. from uniform distribution, while the base Gg is determined by the following formula,
P(G+) = x ((G = G.) + 1 x T(Gb G.).
(17)
4
Therefore, we can manipulate b to control the imbalance degree and the imbalance degree is defined as 4b/(1 b). The corresponding result reported Table The results demonstrate that when pretraining on an imbalanced dataset, the performance improvement is less significant than that on the balanced one, but still better than that without pretraining. Moreover, an overly imbalanced pre-training dataset (b- = 0.7, i R 9.3) may cause negative transfer issue (on Muatg, Molhiv, and Graph-SST2). Therefore, the pre-training dataset ought to be balanced and thus can mitigate the negative transfer issue to some extents.
20

---

Table 11: The impact of pre-training datasets with different sizes.
Task
Dataset
PT-Motifs
W/O Pre-train
PT-Motifs-M
PT-Motifs-S
BA-2Motif's
99.33
93.19
98.91
97.07
Interpretation
Mutag
99.81
95.29
99.06
96.28
ggbg-Molhiv Graph-SST2
80.86
79.71
80.77
79.82
Prediction
88.05
83.48
87.59
85.23
F Hyper-parameter Analysis
we conduct analysis on the number of basic patternleearners to evaluate the tendency of T-GNN performance with the increase of the patter-learner number. Additionally, we report the performance of the variant without pre-training phase, which is marked by the subscript "DFT". One can observe that as the number of basic learners increasing, both the interpretation and prediction performance generally improves, which demonstrates the co-existence of multiple structural patterns. For the Mutag dataset, when we increase the number to the performance improvement (5.76%) is the most significant. But when the number increases to 16 and 32, the performance is inferior to that of basic pattern-learerrs We ascribe this degradation to the difficulty of integrating more and more patterns when the basic learner number increasing. For the Graph-SST2 dataset, the prediction performance consistently improves along with the number of basic learners. This may indicate that the structural patterns in Graph-SST2 are more intricate than those in Mutag dataset. Moreover, the induced variant without the pre-training phase superior to the T-GNN model consistently, which again verifies the effectiveness of the explainer pre-training phase, by 3.32% ROC-AUC score on Mutag dataset and 4.08% accuracy on Graph-STT2 dataset on average.
In Figure S
G Derivation
First, restate and prove Theorem 1 44
Theorem Let F.n be the set of all adjacency matrix A with nodes. Given graph = (V.E) ) E,, n. 2 2, let (S, A) be a most- most-expresssvve structural representation of nodes set S C V in G. VA En,, there exists most- expressxeesiv node representation AA satisfies the relationship as follows,
(S, = Ez (Z))ss))) VV
(18)
for appropriate k-variable function (-)
Proof. Given graph G (V,8) with n nodes, we construct an equivalent set of the most-expressive structural representation I' (S, A), with permutations on node indices:
IIAA) = (1,,
E 11
(19)
VVEV
Define A as the random variable with uniform measure over II(A). Assume the node subset S has no other joint isomorphic set S". Then, for any such S and any element 7r E II(A) as follow,
2x r" (u. Av T (1.2.
(20)
VVEV
there exists a bijective measurable map between the nodes and their positions in the representation vector Yr. Next, we consider the representation set Os(A) restricted to the node subset S as follows,
(=(A)== = [T" A,,,, ... ..) Vees
E n.,, {
(21)
and prove that there exists an surjection between Os (A) and [* A). The surjection exists if, A non-isomorphic node subset S₁, S2, it implies Os, (A) + Os, (A). This condition naturally holds |S1l * |S2].. When |S1 = | we prove by contradiction and assume Os, (A) = O (A). Since the node indices is unique and [ most-expressive, the representation [ (c. A, 7 (1, ................... n)) of node and permutation is unique too. As S1 is non-isomorphic to S2, there must exist at least one node u E S that has no isomorphic equivalent element S2. Hence, 3x H. that provides representation 1 (", A, x 21, and Hn' II,, v E S2 that the coresponding representation
21

---

100
n-GNN R-GNNorr
94
n-GNNor?
98
92
96
ROC-AUC
94
ROC-AUC
92
90
84
82
80
2
4
16
32
16
Number of Basic Pattem-Learners
Number Basic PatternLeaaners
(a) Interpretation performance on Mutag
(b) Interpretation performance on Spurious-Mon
76
m-NNN
GNN
n-GNNor7
I: GNNNN
74
Accuracy
88
Accuracy
70
68
84
66
64
82
16
32
2
16
32
Number of Basic Pattern-Learner
Number of Basic Pattern-Leamers
(c) Prediction performance on Graph-SST2
(d) Prediction performance on Spurious-Motif
Figure 8: The hyper-parameter analysis on the number of the basic pattern- learners.
(" A, 2',, matches. Finally, we conclude contradiction from the original assumption Os, (A) = =Os, (A) and we know the surjection between (A) and I' (S, A) does exist.
Furthermore, it has been proved that for finite multisets with real number elements, most-expressive multiset function can be defined as the expectation of function f((S)) over the multiset. Therefore, there exists some subjective function ((S)) whose expectation over Os(A) gives (S,
Next, we restate and prove Theorem 2.
Theorem The structural representation of edge 0z) can be learnt by simply approaching Al).
a function f(2) which satisfies [ee, A) (E)
Proof. According to Theorem 1, (Z.)ve A can be represented follows,
(Zo)ve = (TTr)))))a ((T(e.AA)veea))())))) A)
(22)
where the noise ((a)) is marginalized from independent noise distribution. With assumption of ((2) that is able capture the structural dependencies within the adjacent matrix A, we can compute the expectation of (Z))oe(i)) AA and eliminate the noisc.
22

---

H
Algorithm
We present the algorithm of graph-hypergraph transformation as following.
Algorithm Graph--yyeergraph Transformation
Input: Edge index Ω IR2x El of raw graph G Output: Hyperedge index Q2 of hypergraph G₁
I: initialize hyperedge set En = 0
2: initialize hyperedge index S2A, < 0
3: for edge e; (u, v) in 52 do
4: EN - U (1), 8% = EK U {i) // Record the endpoints each edge
5: end for
6: for the p-th hyperedge E in E do
7: for item q in E do
Sr < Ωr {(p, a)} // Allocate the same index for each individual hypernode 9: end for
8:
10:
end for
return 12p
We also provide an illustrative example of the transformation process in Figure 2 of Section 3.1
I Limitations
At last, we provide open discussion about the limitations the T-GNN model.
Efficiency. If the number of basic pattern-learners need to be increased to large amount for some intricate graph datasets, the computational efficiency æ GNGN will become the bottleneck. Although the efficiency can be improved by introducing muulii-theeat computation, the time complexity of T-GNN with multiple basic pattern-leareess is higher than the current interpretable GNNs.
Feature Dimension. The feature dimensions of the downstream graph datasets are usually different from that of the PTT-Motifs dataset pre-training phase. Therefore, we have additionally introduce a linear layer to align the dimension differences between PT-Motifs and the downstream datasets. Though this linear layer can be merged into the hypergraph refining module, it indeed increases the optimization difficulty when fine-tuning.
Node Feature. T-GNN focuses identifying the influential subgraphs by computing the edge contribution score, but it is unable to select the important fraction of the node features that leads to the model prediction. As future direction, consider to systematically extend 7-GNN to the interpretation problem in node classification and link prediction task, where the node features are more informative and influential than those in the graph classification task.
Unseen Structural Patterns. As shown in Appendix B.2 several structural patterns are universal and generalize to the synthetic PTT-ottfs dataset and the weald datasets (e.g., Mutag, MNIST-75sp. Molhiv, Graph-SST2). During the pre-training phase over PT-Motifs dataset, T-GNN extracts these universal structural patterns and combines them with the local structural interactions to achieve generalizable interpretation. But for some unseen structural patterns that does not exist in PT-Motifs, T-GNN is unable to capture and then employ them to identify explanations.
23


Reviews:
[{"review_id": "r7CpqBwGUP", "reviewer": "Reviewer_EENr", "summary": "The authors propose a pre-trained interpretable GNN named \\pi-GNN that can distill universal graph structural patterns. \\pi-GNN is pre-trained on a newly constructed synthetic graph datasets with ground-truth explanations and then able to generalize across different graph datasets and tasks. Technically, a structural pattern learning module is introduced to capture and fuse multiple structural patterns for generalizable graph representations. Next, a refining module based on hypergraph is proposed to incorporate the generalizable patterns with the local structural interactions. Extensive experiments demonstrate the superiority of \\pi-GNN over the SOTA baselines in terms of both interpretation and prediction performance. Additionally, an inter-task experiment which evaluates the graph classification pre-trained model on node classification task strongly verifies the excellent generalizability of \\pi-GNN.", "questions": "Q1: The main question to me is the effectiveness of the pre-training strategy on interpretable GNNs. In what cases will this strategy improve the final interpretation and prediction performance? And how to avoid the negative transfer issue of the pre-training paradigm also needs more research efforts. Moreover, to consolidate the assumption that the universal patterns behind different tasks is common, the inter-task evaluation need to be conducted on some real-world datasets.\nQ2: Will the SOTA baselines get improved by pre-training on the newly constructed datasets? There may need some modifications on the existing models to fit the pre-training paradigm, but I think it is profoundly influential to the community if the interpretable GNNs can be generally improved.", "limitations": "The authors have listed some limitations in the supplementary material. I still expect more exploration and discussion on the effectiveness of the pre-training strategy as I mentioned above.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "S1: The paper first studies the pre-training problem of interpretable GNN for generalizable graph interpretability, which is interesting and insightful to the community. \nS2: The proposed method, i.e., \\pi-GNN, is well-motivated and plausible, since the graph structure follows some universal structural patterns that is important to the graph interpretation problem. Specifically, the combination of multiple basic pattern learners and one integrated pattern learner is reasonable to provide a generalizable structural representation. The graph-to-hypergraph transformation is elegant to fuse the universal patterns and the local interactions via hypergraph message passing. In short, the architecture of \\pi-GNN is easy to understand and the figures in the manuscript are clearly illustrated.\nS3: The improvements shown in experiment are significant on almost all datasets with regard to both interpretation and prediction performance. In section 4.3, after pre-trained on graph classification task, the top-tier interpretation performance on node classification is astonishing and demonstrates the cross-task generalizability of \\pi-GNN.\nS4: Sufficient supplementaries are provided, including ablation study, hyper-parameter analysis, to probe into the effectiveness of the proposed modules and the suitable hyper-parameter of \\pi-GNN. Moreover, a great deal of visualized explanation cases are reported for an intuitive understanding towards the interpretation and prediction of \\pi-GNN.", "weaknesses": "W1: It seems like some related works about interpretable GNNs are missing in section 5, such as CAL [1] and OrphicX [2].\nW2: The subgraph selection process is a little obscure to me. \nThe authors may want to append further illustrations on how to select the explanatory edges and embrace the contribution score into gradient optimization.\n[1] Yongduo Sui, Xiang Wang, Jiancan Wu, Min Lin, Xiangnan He, Tat-Seng Chua. Causal Attention for Interpretable and Generalizable Graph Classification. KDD 2022: 1696-1705\n[2] Wanyu Lin, Hao Lan, Hao Wang, Baochun Li. OrphicX: A Causality-Inspired Latent Variable Model for Interpreting Graph Neural Networks. CVPR 2022: 13719-13728", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1: It seems like some related works about interpretable GNNs are missing in section 5, such as CAL [1] and OrphicX [2].\n\nA1: Thanks for your suggestion. We will add the suggested references in the final version.\n>Q2: The subgraph selection process is a little obscure to me. The authors may want to append further illustrations on how to select the explanatory edges and embrace the contribution score into gradient optimization.\n\nA2: Thanks for your suggestions. Following existing works [3, 4, 5], the explainer will provide a mask matrix $M\\in\\mathbb{R}^{|V|\\times|V|}$, where the element $M_{ij}$ indicates the importance of edge $(i,j)$. Afterwards, such a mask matrix can result in an attentive matrix $A^{att} = A\\odot f(M)$, where $A$ is the adjacent matrix of the raw graph and the element $A^{att}_{ij}$ indicates the probability of edge $(i,j)$ belonging to the explanatory subgraph. Finally, based on the attentive matrix $A^{att}$, the explanatory graph is sampled according to edge probability. We will add more explanation about the subgraph selection process in the Appendix.\n>Q3: The main question to me is the effectiveness of the pre-training strategy on interpretable GNNs. In what cases will this strategy improve the final interpretation and prediction performance? And how to avoid the negative transfer issue of the pre-training paradigm also needs more research efforts. Moreover, to consolidate the assumption that the universal patterns behind different tasks is common, the inter-task evaluation need to be conducted on some real-world datasets.\n\nA3: Thanks for your comments and suggestions. \n\n1). The pre-training strategy on the interpretable GNNs is significantly effective when the downstream datasets share some common structural patterns with the pre-training dataset. Therefore, a pre-training dataset that covers more structural patterns is more helpful to the downstream fine-tuning tasks.\n\n2). As to the negative transfer issue, we notice that an imbalanced pre-training dataset tends to improve less significantly than a balanced dataset, in terms of both interpretation and prediction. We further investigate the impact of an imbalanced pre-training dataset. Following existing work [3, 4], each synthetic graph consists of one base subgraph $G_b$ and one explanatory subgraph $G_e$. To generate the imbalanced datasets, we sample the explanatory $G_e$ from uniform distribution, while the base $G_b$ is determined by $P(G_b) = b\\times I(G_b=G_e) + (1-b)/4\\times I(G_b\\neq G_e)$. Therefore, we can manipulate the hyperparameter b to control the imbalance degree and the imbalance degree is defined as $i=4b/(1-b)$. The experimental results of imbalanced pre-training datasets are listed in the following tables.\n|Interpretation|BA-2Motifs|Mutag|\n|:-|:-|:-|\n|Balanced|99.33|99.81|\n|w/o pre-train|93.19|95.29|\n|b=0.3, i≈1.7|98.91|98.46|\n|b=0.5, i=4.0|96.15|97.10|\n|b=0.7, i≈9.3|93.63|94.30|\n\n|Prediction|Molhiv|Graph-SST2|\n|:-|:-|:-|\n|Balanced|80.86|88.05|\n|w/o pre-train|79.71|83.48|\n|b=0.3, i≈1.7|79.42|85.34|\n|b=0.5, i=4.0|77.18|83.14|\n|b=0.7, i≈9.3|75.04|81.90|\n\nThe results reveal that an overly imbalanced pre-training dataset (b=0.7, i≈9.3) may cause negative transfer issue (on Muatg, Molhiv, and Graph-SST2). Therefore, the pre-training dataset ought to be balanced and thus can mitigate the negative transfer issue to some extents.\n\n3). In this work, we just conduct some preliminary experiments to investigate the inter-task generalization problem and more evaluations on the real-world datasets are necessary to consolidate our assumption about universal patterns. In the future work, we will systematically extend to the interpretation problem in node classification and link prediction task.\n>Q4: Will the SOTA baselines get improved by pre-training on the newly constructed datasets? There may need some modifications on the existing models to fit the pre-training paradigm, but I think it is profoundly influential to the community if the interpretable GNNs can be generally improved.\n\nA4: Thanks for your comment. It is promising that the SOTA interpretable GNNs will get improved by incorporating the pre-training process. But on the other hand, incorporating the pre-training and fine-tuning paradigm with the interpretable GNNs is indeed challenging. For a future direction, we will investigate how to introduce the pre-training process to the SOTA baselines in a general way instead of specified modification.\n\n[1] Causal Attention for Interpretable and Generalizable Graph Classification. KDD 2022.\n\n[2] OrphicX: A Causality-Inspired Latent Variable Model for Interpreting Graph Neural Networks. CVPR 2022.\n\n[3] Parameterized Explainer for Graph Neural Network. NeurIPS 2020.\n\n[4] Towards Multi-Grained Explainability for Graph Neural Networks. NeurIPS 2021.\n\n[5] Interpretable and Generalizable Graph Learning via Stochastic Attention Mechanism. ICML 2022."}, {"review_id": "o8yAI4WaCF", "reviewer": "Reviewer_M6tM", "summary": "The paper proposes a generalizable GNN interpretation model, aiming to learn the universal structural patterns of graphs so that it can be applied any downstream applications.", "questions": "Refer the weakness section", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1)\tThe problem that the paper studies is very interesting, a model trained to identify the universal explanatory subgraph in different cases will be of great practical use.\n(2)\tOverall, the paper is easy to follow. \n(3) The authors provide theoretical analysis to support their claim.", "weaknesses": "(1)\tAlthough the paper provides empirical study to demonstrate the effectiveness of the hypergraph refining module, the motivation to incorporate such a component is still not very clear. What is the advantage of the hypergraph refining module compared with a normal graph? \n(2)\tSince the explanation model is pre-trained, the authors are encouraged to incorporate some graph SSL methods as baselines for fair comparison (like GraphCL and GraphLoG). \n(3)\tIt seems the synthetic pretraining dataset is of great importance to the final results, therefore more empirical studies should the conducted to investigate the impacts of the pretraining dataset on the final results, like the dataset size and distribution. What if the synthetic dataset is imbalanced and what if the distribution of the pretraining dataset and downstream datasets are not aligned? Also, I wonder about the cost to generate such a synthetic dataset. \n(4)\tThe authors are encouraged to provide the training details of the baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1: The motivation of the hypergraph refining module and its advantages.\n\nA1: For GNN explainers, the edges and the edge interactions are more essential compared with nodes and node interactions [1,2]. Therefore, we need a more expressive edge representation learning paradigm. To capture the edge interactions, we propose to exchange the roles of edge and node, and then perform the message passing mechanism. In this way, the edges can interact with each other via message passing and the interactions among edges can be captured by the learned representations. However, after exchanging roles, each edge may connect with several edges via a single node, where the normal graph neural networks cannot be directly applied. Hence, we propose the hypergraph refining module to implement the expressive learning on edge representations, where the edges are converted to hyper-nodes and the nodes become hyper-edges. The ablation study shows that, compared with normal GNN models, adopting the proposed hypergraph refining module improves the GNN explanation performance from 94.42% to 99.81% on Mutag dataset, which verifies the effectiveness of the hypergraph refining module.\n>Q2: Incorporate some graph SSL methods as baselines for fair comparison.\n\nA2: We subscribe the graph encoder in our model with a 2-layer GIN which is pretrained by GraphCL [3] and InfoGraph [4]. **The results are listed in Table 1 of the attached PDF.** As the results show, the graph SSL pretrained explainers are inferior to the proposed model after pretraining. As we surmise, it's because the graph SSL methods are general pre-training paradigm for GNNs, instead of specifically designing for the GNN interpretation problem.\n>Q3: More empirical studies to investigate the impacts of the pretraining dataset.\n\nA3: 1). To further investigate the impact of pre-training dataset size, we have added supplement experiments. Specifically, compared with PT-Motifs (80000 graphs) in our paper, we generate PT-Motifs-M with 50000 graphs and PT-Motifs-S with 10000 graphs. The results are listed as follows.\n|Interpretation|BA-2Motifs|Mutag|\n|-|-|-|\n|PT-Motifs (80000)|99.33|99.81|\n|w/o pretrain|93.19|95.29|\n|PT-Motifs-M (50000)|98.91|99.06|\n|PT-Motifs-S (10000)|97.07|96.28|\n\n|Prediction|Molhiv|Graph-SST2|\n|-|-|-|\n|PT-Motifs (80000)|80.86|88.05|\n|w/o pretrain|79.71|83.48|\n|PT-Motifs-M (50000)|80.77|87.59|\n|PT-Motifs-S (10000)|79.82|85.23|\n\nThe results show that even the PT-Motifs-S is able to outperform the model without pre-training. Moreover, we can notice that a large pre-training dataset can indeed improve the performance more significantly than that with small size. \n2). It is possible that if the synthetic pre-training dataset is imbalanced, the performance improvement on downstream tasks will degrade, in terms of both the interpretation and prediction. To further investigate this issue, we have added supplement experiments on imbalanced pre-training dataset. Following existing works [5,6], to generate the imbalanced datasets, we sample the explanatory $G_e$ from uniform distribution, while the base $G_b$ is determined by $P(G_b) = b\\times I(G_b=G_e) + (1-b)/4\\times I(G_b\\neq G_e)$. Therefore, we can manipulate b to control the imbalance degree and the imbalance degree is defined as $i=4b/(1-b)$. The results are listed as follows.\n|Interpretation|BA-2Motifs|Mutag|\n|-|-|-|\n|Balanced|99.33|99.81|\n|w/o pretrain|93.19|95.29|\n|b=0.3, i≈1.7|98.91|98.46|\n|b=0.5, i=4.0|96.15|97.10|\n|b=0.7, i≈9.3|93.63|94.30|\n\n|Prediction|Molhiv|Graph-SST2|\n|-|-|-|\n|Balanced|80.86|88.05|\n|w/o pretrain|79.71|83.48|\n|b=0.3, i≈1.7|79.42|85.34|\n|b=0.5, i=4.0|77.18|83.14|\n|b=0.7, i≈9.3|75.04|81.90|\n\nThe results demonstrate that when pretraining on an imbalanced dataset, the performance improvement is less significant than that on the balanced one, but still better than that without pretraining.\n3). In fact, the distribution of our pretraining dataset is not always aligned with the downstream tasks. Generally, the pretraining process is not strongly coupling with the downstream tasks as in CV [5] and NLP [6]. Hence, there is no need to deliberately align the pretraining dataset with the downstream datasets. We need the fine-tuning process to align the pre-trained model with the specific downstream tasks. Thus, the quantity and quality of downstream dataset are important to the fine-tuned performance too. Additionally, when the distribution of our pretraining dataset is severely not aligned with the downstream datasets and causes the negative transfer issue, we can adjust the synthetic algorithm to align the downstream dataset, which is the main advantage of synthetic pretraining paradigm.\n\n4). **We list the time cost to generate a synthetic graph dataset with different graph scales in Table 2 of the attached PDF.** The result shows that the time cost of graph generation is nearly a linear function of the average node numbers.\n>Q4: The training details of the baselines.\n\nA4: Here, we sketch two important baselines. For GNNExplainer, we tune the learning rate from (1,0.1,0.01,0.001) and the coefficient of the L1-norm from (0.1,0.01,0.001). The coefficient of the entropy regularization is set to the recommended value 1. For PGExplainer, we use the tuned recommended settings from [6], including the temperature, the coefficient of L1-norm regularization. The training details of the baselines will be added in the final version.\n\n[1] Discovering Invariant Rationales for Graph Neural Networks. ICLR 2022.\n\n[2] Parameterized Explainer for Graph Neural Network. NeurIPS 2020.\n\n[3] Graph Contrastive Learning with Augmentations. NeurIPS 2020.\n\n[4] InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization. ICLR 2020.\n\n[5] Momentum contrast for unsupervised visual representation learning. CVPR 2020.\n\n[6] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT 2019."}, {"review_id": "UvfReJbSfq", "reviewer": "Reviewer_XLHb", "summary": "This paper presents a novel method for GNN explainability. The key innovation of this method is that of relying on synthetic graphs with known explanations to pretrain the model. The pretraining helps to learn general explainability patterns, introducing an inductive bias. Such patterns are then aggregated and refined through specific modules. The method is experimentally evaluated on synthetic and real-world graphs, and compared to different methods recently published in the literature.", "questions": "See previous point.", "limitations": "Limitations are well addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The manuscript is well-written and relatively easy to follow.\n- The empirical performances of the reported experiments are very compelling.", "weaknesses": "- The main weakness in my opinion is that the dependency between the test tasks and the synthetic pre-training dataset is not well investigated. However, the synthetic dataset is probably one of the most important hyperparameters, which could be difficult to optimize in rael-world settings. I think that the authors should evaluate the impact of different pretraining datasets on different test tasks, also making sure that some test tasks include motifs not fully covered in the pretraining tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1: The main weakness in my opinion is that the dependency between the test tasks and the synthetic pre-training dataset is not well investigated. However, the synthetic dataset is probably one of the most important hyper-parameters, which could be difficult to optimize in real-world settings. I think that the authors should evaluate the impact of different pre-training datasets on different test tasks, also making sure that some test tasks include motifs not fully covered in the pre-training tasks.\n\nA1: Thanks for your suggestion. \n\n1). It is true that the synthetic pretraining dataset is of great importance to the downstream tasks and how to select a good pre-training dataset may be difficult in real-world settings. To further evaluate the impact of different pretraining datasets on different test tasks, we have added extended experiments on pre-training datasets with different classes of motifs. As we state, the PT-Motifs dataset contains 5 motifs (i.e., Diamond, House, Crane, Cycle, and Star). To investigate the impact of motifs in pre-training dataset, we generate two dataset that contains only 3 motifs. Specifically, PT-DCrS contains Diamond, Crane, and Star and PT-DCyH contains Diamond, Cycle and House. Note that the House and Cycle motifs in BA-2Motifs are not covered by PT-DCrS. The result is reported in the following table.\n|Interpretation|BA-2Motifs|Mutag|\n|-|-|-|\n|PT-Motifs|99.33|99.81|\n|w/o pre-train|93.19|95.29|\n|PT-DCrS|95.53|98.56|\n|PT-DCyH|99.04|98.31|\n\n|Prediction|Molhiv|Graph-SST2|\n|-|-|-|\n|PT-Motifs|80.86|88.05|\n|w/o pre-train|79.71|83.48|\n|PT-DCrS|79.95|87.26|\n|PT-DCrS|79.98|88.02|\n\nThe results demonstrate that even when the pre-training dataset does not cover all the motifs in downstream datasets, the pre-training process can still improve the interpretation and prediction performance. Moreover, when fine-tuning on BA-2Motifs, pre-training on PT-DCyH significantly outperforms that on PT-DCrS.\n\n2). Moreover, we investigate the impact of imbalanced datasets. Following existing work [3], each synthetic graph consists of one base subgraph $G_b$ and one explanatory subgraph $G_e$. To generate the imbalanced datasets, we sample the explanatory $G_e$ from uniform distribution, while the base $G_b$ is determined by $P(G_b) = b\\times I(G_b=G_e) + (1-b)/4\\times I(G_b\\neq G_e)$. Therefore, we can manipulate the hyperparameter b to control the imbalance degree and the imbalance degree is defined as $i=4b/(1-b)$. The experimental results of imbalanced pre-training datasets are listed in the following tables.\n|Interpretation|BA-2Motifs|Mutag|\n|-|-|-|\n|Balanced|99.33|99.81|\n|w/o pre-train|93.19|95.29|\n|b=0.3, i≈1.7|98.91|98.46|\n|b=0.5, i=4.0|96.15|97.10|\n|b=0.7, i≈9.3|93.63|94.30|\n\n|Prediction|Molhiv|Graph-SST2|\n|-|-|-|\n|Balanced|80.86|88.05|\n|w/o pre-train|79.71|83.48|\n|b=0.3, i≈1.7|79.42|85.34|\n|b=0.5, i=4.0|77.18|83.14|\n|b=0.7, i≈9.3|75.04|81.90|\n\nThe results demonstrate that when pretraining on an imbalanced dataset, the performance improvement is less significant than that on the balanced one, but still better than that without pretraining.\n\n3). Compared with PT-Motifs (80000 graphs) in our paper, we generate PT-Motifs-M with 50000 graphs and PT-Motifs-S with 10000 graphs. The experimental results on the pretraining datasets with different scales are listed in the following table.\n|Interpretation|BA-2Motifs|Mutag|\n|-|-|-|\n|PT-Motifs (80,000)|99.33|99.81|\n|w/o pre-train|93.19|95.29|\n|PT-Motifs-M (50,000)|98.91|99.06|\n|PT-Motifs-S (10,000)|97.07|96.28|\n\n|Prediction|Molhiv|Graph-SST2|\n|-|-|-|\n|PT-Motifs (80,000)|80.86|88.05|\n|w/o pre-train|79.71|83.48|\n|PT-Motifs-M (50,000)|80.77|87.59|\n|PT-Motifs-S (10,000)|79.82|85.23|\n\nThe results show that even the PT-Motifs-S is able to outperform the model without pre-training. Moreover, we can notice that the large pre-training dataset can indeed improve the performance more significantly than that with small size.\n\n[1] Discovering Invariant Rationales for Graph Neural Networks. ICLR 2022.\n\n[2] Parameterized Explainer for Graph Neural Network. NeurIPS 2020."}, {"review_id": "yHOA5HTKSt", "reviewer": "Reviewer_34Ne", "summary": "This paper proposed a novel pre-training method for interpretable graph neural network. Interpretable GNN is currently an important research issue and has attracted rising research attention recently. However, exiting methods are generally designed for some special types of datasets, and thus are hard to generalize well to other graphs or tasks. This paper makes the first attempt to design a pre-training framework with a carefully developed labeled synthetic graph dataset. As the synthetic graphs have ground truth explanations, the pretained model can better capture the common structure knowledge of different types of graphs. A structure pattern learning module and a hypergraph refining module are also proposed to make the pretained model achieve better performance. Generally, the paper is very well written and has clear contributions. It is a nice try to design pre-trained interpretable GNNs. The experiment is extensively studied over multiple datasets, and the performance improvement over current SOTA is significant. The evaluation on node classification and graph classification tasks further verifies the promising generalization ability of the model.", "questions": "(1)\tHow to construct a good synthetic graph datasets for pretaining? Are the ground truth GNN explanations labeled manually?\n(2)\tIn section 4.3, the authors evaluate the model pre-trained over graph classification dataset on the explanation task of node classification to evaluate the inter-task generation performance. How to do that exactly? The learned graph embeddings learned from graph classification is used for node classification? \n", "limitations": "The authors have adequately addressed the limitations of their work in the appendix.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "(1)\tThis paper for the first time proposed a pre-training interpretable GNN model, which is novel and practically important. The pre-trained model is first trained over a large synthetic graph datasets, and then fine-tuned over downstream graphs, which achieves significant performance improvement. As a general and pioneer pipeline, I believe the paper can potentially motivate a lot of following works.\n(2)\tA structure pattern learning module and a hypergraph refining module are proposed and integrated into the pre-training model. The two modules can better and more comprehensively capture the structural patterns and the edge interactions of the graphs.\n(3)\tThe experiment is extensively studied over both real and synthetic graph datasets. The results show the significant performance improvement over current SOTA. The experiment is convincing and the code is publicly available.\n(4)\tThe paper is very well written and organized in good logic. It is easy to read and follow.\n", "weaknesses": "(1)\tAs the constructed synthetic graphs with ground truth labels may significantly affect the performance of the pre-training model. How to construct a good synthetic graphs should be clearly explained. Currently, it is not clear enough on how to construct the data. More details on the synthetic datasets construction should be provided.\n(2)\tIt is not very clear to me why the authors need to give Theorem 1 and 2. Based on my understanding, the authors want to prove that the learning edge representation can fully preserve the graph pattern information. If it is the case, the authors should explain it more clearly and describe why it is important for interpretable GNN.\n(3)\tIn the evaluation, pi-GNN is also compared with pi-GNN-DFT, the authors explained that pi-GNN-DFT directly fine-tunes on downstream datasets without pre-training. How to implement that exactly is not explained. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1: As the constructed synthetic graphs with ground truth labels may significantly affect the performance of the pre-training model. How to construct good synthetic graphs should be clearly explained. Currently, it is not clear enough on how to construct the data. More details on the synthetic dataset construction should be provided. How to construct a good synthetic graph dataset for pretraining? Are the ground truth GNN explanations labeled manually?\n\nA1: Thanks for your suggestion. \n\n1). We will add the generation details of the synthetic dataset in Appendix. Following existing works [1, 2], each synthetic graph consists of a base and an influential motif and the label is determined by the motif solely. Given one certain motif, we first sample a base from a uniform distribution over all bases and the motif is attached a randomly selected node in the base.\n\n2). According to our experimental results, a good synthetic graph dataset for pre-training should first be balanced. Moreover, larger dataset size and more motif classes are also promising to guarantee the effectiveness of pre-training process.\n>Q2: It is not very clear to me why the authors need to give Theorem 1 and 2. Based on my understanding, the authors want to prove that the learning edge representation can fully preserve the graph pattern information. If it is the case, the authors should explain it more clearly and describe why it is important for interpretable GNN.\n\nA2: Thanks for your comments. Your understanding is absolutely correct and we will add more explanation about the importance of the edge representation with regard to interpretable GNNs. The edges are more essential to GNN explanation compared with nodes, as the previous works [3, 4] point out. Therefore, we need a more expressive representation of edges and thus can provide a better GNN explanation.\n>Q3: In the evaluation, pi-GNN is also compared with pi-GNN-DFT, the authors explained that pi-GNN-DFT directly fine-tunes on downstream datasets without pre-training. How to implement that exactly is not explained.\n\nA3: Thanks for your suggestion. We will append more details of the directly fine-tuning paradigm in the final version. Specifically, pi-GNN-DFT skips the pretraining process on the synthetic dataset and conduct standard training on the downstream datasets just like the other baselines.\n>Q4: In section 4.3, the authors evaluate the model pre-trained over graph classification dataset on the explanation task of node classification to evaluate the inter-task generation performance. How to do that exactly? The learned graph embeddings learned from graph classification is used for node classification?\n\nA4: Thanks for your comment. We will append more details of the inter-task evaluation in the final version. When evaluating the inter-task interpretation performance, the whole graph of the node classification task is fed into the pretrained explainer and the explainer will identify the explanatory subgraph of each target node in test set. The interpretation performance is evaluated on the target nodes.\n\n[1] GNNExplainer: Generating Explanations for Graph Neural Networks. NeurIPS 2019.\n\n[2] Graph Information Bottleneck for Subgraph Recognition. ICLR 2021.\n\n[3] Discovering Invariant Rationales for Graph Neural Networks. ICLR 2022.\n\n[4] Parameterized Explainer for Graph Neural Network. NeurIPS 2020."}]

Author rebuttal:
We thank all the reviewers for their insightful and constructive feedback. We have made point-to-point response to the comments of each reviewer. 

Moreover, we report two supplemental experiments in the attached file.

Finally, we once again thank all reviewers for their insightful comments which are very helpful for improving the quality of our paper.

Meta-review: This submission on pre-training interpretable GNNs for generalizable graph interpretability has resonated positively across the board, with reviewers cohesively emphasizing its novelty, technical depth, and potential influence on the field. 

* The paper's core theme of studying the pre-training of interpretable GNNs, especially the introduction of the \pi-GNN model, is perceived positively. 

* The significant improvements across datasets for both interpretation and prediction aspects, especially the interpretative performance on node classification after pre-training on graph classification, strongly evince the model's cross-task generalizability. The inclusion of ablation studies, hyper-parameter analyses, and an array of visual explanation cases enriches the manuscript.

Given the unanimous appreciation for the paper's novelty, technical outline, and empirical performance, this submission is suitable for publication at NeurIPS.
Decision: Accept (poster)