RegExplainer: Generating Explanations for Graph Neural Networks in Regression Tasks

Authors: jiaxing  zhang
zhuomin  chen
hao  mei
longchao  da
dongsheng  luo
hua  wei
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 03:36:31

Abstract: Graph regression is a fundamental task that has gained significant attention in
various graph learning tasks. However, the inference process is often not easily
interpretable. Current explanation techniques are limited to understanding Graph
Neural Network (GNN) behaviors in classification tasks, leaving an explanation gap
for graph regression models. In this work, we propose a novel explanation method
to interpret the graph regression models (XAIG-R). Our method addresses the
distribution shifting problem and continuously ordered decision boundary issues
that hinder existing methods away from being applied in regression tasks. We
introduce a novel objective based on the graph information bottleneck theory (GIB)
and a new mix-up framework, which can support various GNNs and explainers
in a model-agnostic manner. Additionally, we present a self-supervised learning
strategy to tackle the continuously ordered labels in regression tasks. We evaluate
our proposed method on three benchmark datasets and a real-life dataset introduced
by us, and extensive experiments demonstrate its effectiveness in interpreting GNN
models in regression tasks.

Main paper: 
 introduction: 
 Graph Neural Networks [1] (GNNs) have become a powerful tool for learning knowledge from graphstructure data and achieved remarkable performance in many areas, including social networks [2, 3] , molecular structures [4, 5] , traffic flows [6] [7] [8] [9] , and recommendation systems [10] [11] [12] . Despite the success, their popularity in sensitive fields such as fraud detection and drug discovery [13, 14] requires an understanding of their decision-making processes. To address this challenge, some efforts have been made to explain GNN's predictions in a post-hoc manner, which aims to find a sub-graph that preserves the information about the predicted label. On top of the intuitive principle, Graph Information Bottleneck (GIB) [15, 16] maximizes the mutual information I(G * ; Y ) between the target prediction label Y and the explanation G * while constraining the size of the explanation.However, existing methods focus on the explanation of the classification tasks, leaving another fundamental task, explainable regression, unexplored. Graph regression tasks exist widely in nowadays applications, such as predicting the molecular property [17] or traffic flow volume [18] . Explaining the instance-level predictions of graph regression is challenging due to two main obstacles. First, in the routinely adopted GIB framework, the mutual information between the explanation sub-graph and label, I(G * ; Y ), is estimated with the Cross-Entropy between the predictions f (G * ) from GNN model f and its prediction label Y . However, in the regression task, the regression label is the continuous value, making the approximation unsuitable. Another challenge is the distribution shifting problem in the usage of f (G * ), where the prediction of the explanation sub-graph made by the GNN model f is unsafe. Usually, explanation sub-graphs have different topology and feature information compared to the original graph. As a result, explanation sub-graphs are out-of-distribution of the original training graph dataset [19] [20] [21] . As shown in Figure 1 , a GNN model f is trained on the original graph training set and cannot be safely used to make predictions for sub-graphs. To fill the gap, in this paper, we propose RegExplainer, to generate post-hoc instancelevel explanations for graph regression tasks. Specifically, we formulate a theoretical-sound objective for explainable regression based on information theory. To further address the distribution shifting issue, RegExplainer develops a new mix-up approach with self-supervised learning. Our experiments show that RegExplainer provides consistent and concise explanations of GNN's predictions on regression tasks. We achieved up to 48.0% improvement when compared to the alternative baselines in our experiments. Our contributions can be summarized as follows.• To our best knowledge, we are the first to explain GNN predictions on graph regression tasks. We addressed two challenges in explaining the graph regression task: the mutual information estimation in the GIB objective and the distribution shifting problem with continuous decision boundaries.• We proposed a novel model with selfsupervised learning and the mix-up approach, which can address the two challenges more effectively, and better explain the graph model on the regression tasks compared to other baselines.• We designed three synthetic datasets, namely BA-Motif-Volume, BA-Motif-Counting and Triangles, as well as a real-world dataset called Crippen, which can also be used in future works, to evaluate the effectiveness of our regression task explanations. Comprehensive empirical studies on both synthetic and real-world datasets demonstrate that our method can provide consistent and concise explanations for graph regression tasks.2 Related Work and Further Discussions GNN Explainability The explanation methods for GNN models can be categorized into two types based on their granularity: instance-level [22] [23] [24] [25] and model-level [26] , where the former methods explain the prediction for each instance by identifying important sub-graphs, and the latter method aims to understand the global decision rules captured by the GNN. These methods can also be classified into two categories based on their methodology: self-explainable GNNs [27, 28] and post-hoc explanation methods [23] [24] [25] , where the former methods provide both predictions and explanations, while the latter methods use an additional model or strategy to explain the target GNN. Additionally, CGE [29] (cooperative explanation) generates the sub-graph explanation with the sub-network simultaneously, by using cooperative learning. However, it has to treat the GNN model as a white box, which is usually unavailable in the post-hoc explanation. Existing methods have only partially addressed the explanation of graph regression tasks and have not fully considered two important challenges: the distribution shifting problem and the limitations of the GIB objective, both of which are key areas our work aims to tackle.The Information Bottleneck (IB) [30, 31] provides an intuitive principle for learning dense representations that an optimal representation should contain sufficient information for the downstream prediction task with a minimal size. Based on IB, a recent work [32] unifies the most existing post-hoc explanation methods for GNN, such as GNNExplainer [23] , PGExplainer [24] , with the graph information bottleneck (GIB) principle [15, 16, 32] . Formally, the objective of explaining the prediction of f on G can be represented by arg minEQUATIONwhere G is the to-be-explained original graph, G * is the explanation sub-graph of G, Y is the original ground-truth label of G, and α is a hyper-parameter to get the trade-off between minimal and sufficient constraints. GIB uses the mutual information I(G; G * ) to select the minimal explanation that inherits only the most indicative information from G to predict the label Y by maximizing I(G * ; Y ), where I(G; G * ) avoids imposing potentially biased constraints, such as the size or the connectivity of the selected sub-graphs [15] . Through the optimization of the sub-graph, G * provides model interpretation. In graph classification task, a widely-adopted approximation to Eq. ( 1) in previous methods [23, 24] is:arg min G * I(G; G * ) + αH(Y |G * ) ≈ arg min G * I(G; G * ) + αCE(Y, Y * ),where Y and Y * , approximated by f (G) and f (G * ), is the predicted label of G and G * made by the to-be-explained model f , and the cross-entropy CE(Y, Y * ) between Y and Y * is used to approximate -I(G * ; Y ). The approximation is based on the definition of mutual informationI(G * ; Y ) = H(Y ) -H(Y |G * ):with entropy H(Y ) being static and independent of the explanation process, minimizing the mutual information between the explanation sub-graph G * and Y can be reformulated as maximizing the conditional entropy of Y given G * , which can be approximated by CE(Y, Y * ). 
preliminary: 
 Notation and Problem Formulation We use G = (V, E; X, A) to represent a graph from an alphabet G, where V equals to {v 1 , v 2 , ..., v n } represents a set of n nodes and E ∈ V × V represents the edge set. Each graph has a feature matrix X ∈ R n×d for the nodes, wherein X, X i ∈ R 1×d is the d-dimensional node feature of node v i . E is described by an adjacency matrix A ∈ {0, 1} n×n , where A ij = 1 means that there is an edge between node v i and v j ; otherwise, A ij = 0. For the graph prediction task, each graph G k has a label Y k ∈ C, where k ∈ {1, ..., N }, N represents the number of graphs in the dataset, C is the set of the classification categories or regression values in R, with a GNN model f trained to make the prediction, i.e., f : (X, A) → C. Problem 1 (Post-hoc Instance-level GNN Explanation). Given a trained GNN model f , for an arbitrary input graph G = (V, E; X, A), the goal of post-hoc instance-level GNN explanation is to find a sub-graph G * that can explain the prediction of f on G.In non-graph structured data, the informative feature selection has been well studied [33] , as well as in traditional methods, such as concrete auto-encoder [34] , which can be directly extended to explain features in GNNs. In this paper, we focus on discovering the important sub-graph typologies following the previous work [23, 24] . Specifically, the obtained explanation G * is depicted by a binary mask M * ∈ {0, 1} n×n on the adjacency matrix, e.g., G * = (V, E; X, A ⊙ M * ), ⊙ means elements-wise multiplication. The mask highlights components of G which are essential for f to make the prediction. 
methodology: 
 In this section, we first introduce a new objective based on GIB for explaining graph regression tasks.Then we showcase the distribution shifting problem in the objective for regression and propose a novel framework with the mix-up approach to solve the distribution shifting problem, by incorporating the mix-up approach with self-supervised contrastive learning.As introduced in Section 2, in the classification task, I(G * ; Y ) in Eq. ( 1) is commonly approximated by cross-entropy CE(Y * , Y ) [35] . However, it is non-trivial to extend it for regression tasks because Y is a continuous variable and it is intractable to compute the cross-entropy CE(Y * , Y ) or the mutual information I(G * ; Y ), where G * is a graph variable with a continuous variable Y * as its label. EQUATIONwhere Y ′ is the prediction label of the randomly sampled graph neighbors, Y is the set of the neighbors' prediction labels, and sim() estimates the similarity between Y * and Y . The proof is shown in the Appendix B.2. Therefore, we have the InfoNCE loss L NCE as the lower bound of the I(Y * ; Y ). We approximate Eq. (2) as:EQUATIONWe include the prediction label Y * in Eq. (4) to estimate similarity, which is approximated with f (G * ) in previous work [23, 24] . However, we argue that f (G * ) cannot be safely obtained due to the distribution shift problem [37, 19] . In classification tasks, a small shift may not cross the decision boundaries, which can still lead to a correct prediction. However, due to the continuous decision boundaries in regression, the distribution problem would cause serious prediction errors. Here in this paper, the graph distribution is indicated by its regression label in the regression task.Figure 3 shows the existence of distribution shifts between f (G * ) and f (G) in graph regression tasks.For each dataset, we sort the indices of the data samples according to the value of their labels, and visualize the label Y , prediction f (G) of the original graph from the trained GNN model f , and prediction f (G * ) of the explanation sub-graph G * from f . As we can see in Figure 3 , in all four graph regression datasets, the red points are well distributed around the ground-truth blue points, indicating that f (G) is close to Y . In comparison, the green points shift away from the red points, indicating the shifts between f (G * ) and f (G). Especially in dataset BA-Motif-Counting, the sub-graph explanation distribution was shifted extremely. Intuitively, this phenomenon indicates the GNN model f can make correct predictions only with the original graph G yet can not predict the explanation sub-graph G * correctly. This is because the GNN model f is trained with the original graph sets, whereas the explanation G * as the sub-graph is different from the original graph sets. With the shift between f (G) and f (G * ), the optimal solution in Eq. ( 4) is unlikely to work well.To address this distribution-shifting problem in graph regression, we innovatively incorporate the mix-up approach with a self-supervised contrastive learning strategy. Instead of calculating Y * with f (G * ) directly, we approximate with Y (mix) from f (G (mix) ), which contains similar information as G * but is in the same distribution with G. Specifically, our approach includes the following steps:• Step 1 (Neighbor Sampling): Learning through the triplet instances can effectively reinforce the ability of the explainer to learn the explanation self-supervised. For each target graph G with label Y to be explained, we can define two randomly sampled graphs as positive neighbor G + and negative neighbor G -, whereG + 's label Y + is closer to Y than G -'s label Y -, i.e., |Y + -Y | < |Y --Y |.Intuitively, the distance between the distributions of the positive pair ⟨G, G + ⟩ should be smaller than the distance between the distributions of the negative pair ⟨G, G -⟩.•Step 2 (Mixup for G * ): Then we generate two mixup graphs G (mix)+ and G (mix)-by mixing the sub-graph explanation G * with the label irrelevant sub-graph (G + ) ∆ = G + -(G + ) * from its positive neighbor G + and the label irrelevant sub-graph (G -) ∆ = G --(G -) * from negative neighbor G -respectively. Specifically, the label mixup approach is calculated from:G (mix)+ = G * + (G + ) ∆ = G * + (G + -(G + ) * ), G (mix)-= G * + (G -) ∆ = G * + (G --(G -) * ).G (mix)+ and G (mix)-should have the similar information to G because they have the same labelpreserving sub-graphs G * . Additionally, considering the following two pairs: (G (mix)+ , G + ) and (G (mix)-, G -). The similarity between (G (mix)+ , G + ) should be larger than the similarity between (G (mix)-, G -). Intuitively, since G (mix)+ and G (mix)-have the same label-preserving sub-graphsG * and |Y --Y | > |Y + -Y |, we can have |f (G -) -f (G (mix)-)| > |f (G + ) -f (G (mix)+ )|, where f (G) represents the prediction label of graph G.• Step 3 (InfoNCE Loss Approximation): Then we can safely estimate the similarity with sim(Y (mix) , Y ). To save more information, we use the similarity of representation embedding to approximate the similarity of the graph prediction label, where h (mix) represents the embedding for G (mix) and h represents the embedding for G. We use H to represent the neighbors set accordingly. Thus, we approximate Eq. ( 4) as:arg minEQUATION)The mix-up approach in previous work [19] generates a mixed graph by simply mixing explanation sub-graph G * with a randomly sampled label-irrelevant sub-graph G ∆ [19] , which can be formally written as G (mix) a = G * a + (G b -(G + ) ∆ = G + -(G + ) * and (G -) ∆ = G --(G -) * respectivelyto produce G (mix)+ and G (mix)-. Then the graphs are fed into the trained GNN model to retrieve the embedding vectors h + , h -, h (mix)+ and h (mix)-, where h (mix)+ ≈ h (mix)-due to the same label-preserving sub-graph G * . We use InfoNCE loss to minimize the distance between G (mix)+ and the positive sample and maximize the distance between G (mix)-and the negative sample. The explainer is trained with the GIB objective and self-supervised contrastive loss.G * b ). However, it cann't tackle the continuous decision boundaries in graph regression tasks. A detailed description of the mix-up approach can be found in Appendix C.InfoNCE Loss After generating the mix-up explanation G (mix) , we specify the InfoNCE loss to further train the parameterized explainer with a triplet of graphs ⟨G, G + , G -⟩. In practice, G + and G -are randomly sampled from the graph dataset, upon which we calculate their similarity score with the target graph G. The sample with a higher score would be the positive sample and the other one would be the negative sample. Specifically, we use sim(h, h ′ ) = h ⊤ h ′ to compute the similarity score, where G ′ can be G + or G -. h is generated by feeding G into the GNN model f and retrieving the embedding vector before the dense layers.Formally, given a target graph G, the sampled positive graph G + and negative graph G -, we formulate the InfoNCE loss in Eq. ( 5) as the following:EQUATIONwhere exp(h ⊤ h) is used to instantiate the function sim, the denominator is a sum over the similarities of both positive and negative samples.Size Constraints We optimize I(G; G * ) in Eq. ( 5) to constraint the size of the explanation subgraph G * . The upper bound of I(G; G * ) is optimized as the estimation of the KL-divergence between the probabilistic distribution between the G * and G, where the KL-divergence term can be divided into two parts as the entropy loss and size loss [16] . In practice, we follow the previous work [23, 24, 38] to implement them. Specifically,EQUATIONwhere(i,j)∈E (M * ij )means sum the weights of the existing edges in the edge weight mask M * for the explanation G * ; h * is extracted from the embedding of the graph G * before the GNN model f transforming it into prediction Y * , σ means the sigmoid function and γ is the weight for the size of the masked graph. In implementation, we set γ = (0.0003, 0.3) following previous work [19] .Overall Objective Function In practice, the denominator in Eq. ( 5) works as a regularization to avoid trivial solutions. Since the label Y is given and independent of the optimization process, we can also employ the MSE loss between Y * and Y additionally, regarding InfoNCE loss only estimates the Table 1 : Illustration of the graph regression datasets together with the explanation faithfulness in terms of AUC-ROC on edges under four datasets on RegExplainer and other baselines. The original graph row visualizes the structure of the complete graph, the explanation row highlights the explanation sub-graph of the corresponding original graph. In the Crippen dataset, different colors of the node represent different kinds of atoms and the node feature is a one-hot vector to encode the atom type. mutual information between the embeddings. Formally, the overall loss function can be implemented as:EQUATIONG (mix)+ means mix G * with the positive sample G + and α and β are hyper-parameters. The training algorithm and description of it are put in Appendix D. 
experiments: 
 In this section, we conduct experiments to demonstrate the performance of our proposed method 1 . These experiments are mainly designed to explore the following research questions:• RQ1: Can RegExplainer outperforms other baselines in explaining GNNs on regression tasks?• RQ2: How does each part of RegExplainer and hyperparameters impact the overall performance in generating explanations?• RQ3: Does the distribution shifting exist in GNN explanation? Can RegExplainer alleviate it?We formulate Three synthetic datasets and a real-world dataset, as is shown in Table 1 , in order to address the lack of graph regression datasets with ground-truth explanation. The datasets include: BA-Motif-Volume and BA-Motif-Counting, which are based on BA-shapes [23] , Triangles [39] ,and Crippen [40] . We compared the proposed RegExplainer against a comprehensive set of baselines in all datasets, including: GRAD [23] , ATT [41] , GNNExplainer [23] , PGExplainer [24] , and MixupExplainer [19] . Detailed information about experiment setting are put in the Appendix E. We elaborate on the measurement metric of methods as follows: (1) AUC-ROC: We use the AUC score to evaluate the performance of our proposed methods against baseline methods regarding the ground-truth explanation, which can be treated as a binary classification task. (2) We evaluate the similarity of distribution of the graph with Cosine Similarity and Euclidean Distance.In this section, we evaluate the performance of our approach with other baselines. For GRAD and GAT, we use the gradient-based and attention-based explanation, following the setting in the previous work [23] . We take GCN as our to-be-explained model for all post-hoc explainers. For GNNExplainer, PGExplainer, and MixupExplainer, which were previously used for the classification task, we replace the Cross-Entropy loss with the MSE loss. We run and tune all the baselines on our four datasets. We evaluate the explanation from all the methods with the AUC metric, as done in the previous work. As we can see in Table 1 , we take GNNExplainer and PGExplainer as backbones and apply our framework as RegExplainer on both of them. The experiment results demonstrate the effectiveness of our methods in explaining graph regression tasks, where our method achieves the best performance compared to the baselines in all four datasets.In Table 1 , RegExplainer based on PGExplainer improves the second best baseline with 0.175/34.3% on average and up to 0.246/48.0%. The comparison between RegExplainer and other baselines indicates the advantages of our proposed approach. This improvement indicates the effectiveness of our proposed method, showing that by incorporating the mix-up approach and contrastive learning, we can generate more faithful explanations in the graph regression tasks. In the following sections, we analyze the RegExplainer with PGExplainer as a backbone.Figure 5 : Ablation study of RegExplainer. We evaluated the AUC performance of the original RegExplainer and its variants that exclude the mixup approach, InfoNCE loss, or MSE loss, respectively. The black solid line shows the standard deviation.We conducted an ablation study to show how our proposed components, specifically, the mixup approach and self-supervised learning, contribute to the final performance of RegExplainer.To this end, we denote RegExplainer as RegE and design three types of variants as follows:(1) RegE We also investigate the hyper-parameters of our approach, which include α and β, across all four datasets. The hyper-parameter α controls the weight of the InfoNCE loss in the GIB objective while the β controls the weight of the MSE loss. We determined the optimal values of α and β with grid search. The experimental results can be found in Figure 6 . We fixed α and β at 1 and changed another parameter to visualize the change in model performance. Figure 6 illustrates that the model's performance is robust to changes in hyper parameters within the scope [0.001, 1000]. Our findings indicate that our approach, RegExplainer, is stable and robust when using different hyper-parameter settings, as evidenced by consistent performance across a range. In this section, we visualize the regression values of the graphs and calculate the prediction shifting distance for each dataset and analyze their correlations to the distance of the decision boundaries. We put our results into Figure 7 and Table 2 .(f (G), Y ), (f (G * ), Y ), (f (G), f (G * )) respec- tively. Dataset (f (G), Y ) (f (G * ), Y ) (f (G), f (G * )) BA-MotifWe observed that in Figure 3 , red points surround the blue points but green points are shifted away, which indicates that the explanation sub-graph cann't help GNNs make correct predictions. As shown in Table 2 , we calculate the RMSE score between the f (G) and Y , f (G * ) and Y , f (G) and f (G * ) respectively, where f (G) is the prediction the original graph, f (G * is the prediction of the explanation sub-graph, and Y is the regression label. We can observe that f (G * ) shows a significant prediction shifting from f (G) and Y , indicating that the mutual information calculated with (f (G * ), Y ) would be biased.We further explore the relationship of the prediction shifting against the label value with dataset BA-Motif-Volume, which represents the semantic decision boundary. This additional experiment with Figure 7 can be found in Appendix F.1.We also design experiments to illustrate how RegExplainer corrects the deviations: we calculate the graph embeddings v and predictions p of the explanation sub-graphs and the mix-up graph. Then we compare them to the ground truth and calculate the Euclidean or Cosine distance between the vectors and RMSE between prediction labels. From the results in Table 3 , we can observe that all the performances of COS(v g , v m ), EUC(v g , v m ) and prediction errors are better than those of (v g , v e ), which indicates RegExplainer can effectively fix the distribution of sub-graph explanation G * and reduce the embedding distance and prediction error. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We provided code and data in supplementary.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 We addressed the challenges in the explainability of graph regression tasks and proposed the Reg-Explainer, a novel method for explaining the predictions of GNNs with the post-hoc explanation sub-graph on graph regression task without requiring modification of the underlying GNN architecture or re-training. We showed how RegExplainer can leverage the mix-up approach to solve the distribution shifting problem and adopt the GIB objective with the InfoNCE loss to migrate it from graph classification tasks to graph regression tasks, while these existing challenges seriously affect the performances of other explainers. We formulated four new datasets: BA-Motif-Volume, BA-Motif-Counting, Triangles, and Crippen for evaluating the explainers on the graph regression task, which are aligned with the design of datasets in previous work. They can also benefit future studies on the XAIG-R. While we acknowledge the effectiveness of our method, we also recognize its limitations. Specifically, although our approach can be applied to explainers for graph regression tasks in an explainer-agnostic manner, it cannot be easily applied to explainers built for explaining the spatio-temporal graph due to the dynamic topology and node features of the STG. To overcome this challenge, a potential solution is to incorporate cached dynamic embedding memories into the framework.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: The experimental setting is presented in the paper and The full details can be found in the appendix and supplemental material. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. Justification: Our research doesn't have a concern about the ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
ethics statement: 
 This work is primarily foundational in GNN explainability, focusing on expanding the GIB objective function of the explainer framework from graph classification tasks to graph regression tasks. Its primary aim is to contribute to the academic community by enhancing the explanation in graph regression. We do not foresee any direct, immediate, or negative societal impacts stemming from the outcomes of our research.  
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Proof. From the definition of Y * , we can make a safe assumption that there is a many-to-one map (function), denoted by h, from G * to Y * as Y * is the prediction label for G * . For simplicity, we assume a finite number of explanation instances for each label y * , and each explanation instance, denoted by g * , is generated independently. Then, we have p(y * ) = g * ∈G(y * ) p(g * ), where G(y * ) = {g|h(g) = y * } is the set of explanations whose labels are y * .Based on the definition of mutual information, we have:I(G * ; Y ) = y g * p (G * ,Y ) (g * , y) log p (G * ) (g * , y) p G * (g * )p Y (y) d g * d y = y g * p (G * ,Y * ,Y ) (g * , h(g * ), y) log p (G * ,Y * ,Y ) (g * , h(g * ), y) p G * (g * )p Y (y) d g * d y = y g * p (G * ,Y * ,Y ) (g * , h(g * ), y) log p (G * ,Y * ,Y ) (g * , h(g * ), y) p (G * ,Y * ) (g * , h(g * ))p Y (y) d g * d y = y y * g * ∈G(y * ) p (G * ,Y * ,Y ) (g * , y * , y) log p (G * ,Y * ,Y ) (g * , y * , y) p (G * ,Y * ) (g * , y * )p Y (y) d y * d yBased on our many-to-one assumption, while each g * is generated independently, we know that if g / ∈ G(y * ), then we have p (G * ,Y * ,Y ) (g * , y * , y) = 0. Thus, we have:I(G * ; Y ) = I(G * ; Y ) + y y * g / ∈G(y * ) p (G * ,Y * ,Y ) (g * , y * , y) log p (G * ,Y * ,Y ) (g * , y * , y) p (G * ,Y * ) (g * , y * )p Y (y) d y * d y = y y * g * p (G * ,Y * ,Y ) (g * , y * , y) log p (G * ,Y * ,Y ) (g * , y * , y) p (G * ,Y * ) (g * , y * )p Y (y) d g * d y * d y = I(G * , Y * ; Y ).With the chain rule for mutual information, we haveI(G * , Y * ; Y ) = I(Y * ; Y ) + I(G * ; Y |Y * ).Then due to the non-negativity of the mutual information, we haveI(G * , Y * ; Y ) ≥ I(Y * ; Y ).Proof. As in the InfoNCE method, the mutual information between Y * and Y is defined as:EQUATIONHowever, the ground truth joint distribution p(Y * , Y ) is not controllable, so, we turn to maximize the similarityEQUATION)We want to put the representation function of mutual information into the NCE LossEQUATIONwhere L N denotes the NCE loss. By inserting the optimal sim (Y * , Y ) into Eq. ( 11), we can get:EQUATIONTo address the distribution shifting issue between f (G) and f (G * ) in the GIB objective, we introduce the mix-up approach to reconstruct a within-distribution graph, G (mix) , from the explanation graph G * .We follow [24] to make a widely-accepted assumption that a graph can be divided by G = G * + G ∆ , where G * presents the underlying sub-graph that makes important contributions to GNN's predictions, which is the expected explanatory graph, and G ∆ consists of the remaining label-independent edges for predictions made by the GNN. Both G * and G ∆ influence the distribution of G. Therefore, we need a graph G (mix) that contains both G * and G ∆ , upon which we use the prediction of G (mix) made by f to approximate Y * and h * .Specifically, for a target graph G a in the original graph set to be explained, we generate the explanation sub-graph G * a = G a -G ∆ a from the explainer. To generate a graph in the same distribution of original G a , we can randomly sample a graph G b from the original set, generate the explanation sub-graph of G * b with the same explainer and retrieve its label-irrelevant graphG ∆ b = G b -G * b .Then we can merge G * a together with G ∆ b and produce the mix-up explanation G (mix) a . Formally, we can haveG (mix) a = G * a + (G b -G * b). Since we are using the edge weights mask to describe the explanation, we can denote G a and G b with the adjacency matrices A a and A b , their edge weight mask matrices as M a and M b . If G a and G b are aligned graphs with the same number of nodes, we can simply mix them up byM (mix) a = M * a + (I b -M * b ), where M denotes the weight of the adjacency matrix and I b denotes the zero-ones matrix as weights of all edges in the adjacency matrix of G b , where 1 represents the existing edge and 0 represents there is no edge between the node pair.If G a and G b are not aligned with the same number of nodes, we can use a connection adjacency matrix A conn and mask matrix M conn to merge two graphs with different numbers of nodes. Specifically, the mix-up adjacency matrix can be formed as:EQUATIONAnd the mix-up mask matrix can be formed as:M (mix) a = M * a M conn M T conn M ∆ b (14) Finally, we can form G (mix) a as (X (mix) , A (mix) a ⊙ M (mix) a ), where X (mix) = [X a ; X b ].The detailed algorithm for mix-up is shown in Algorithm 1. In implantation, η = |E| * 0.03.Input: Target to-be-explained graph G a = (X a , A a ), G b sampled from a set of graphs G, the number of random connections η, explainer model E. Output: Graph G (mix) .1: Generate mask matrix M a = E(G a ) 2: Generate mask matrix M b = E(G b ) 3: Sample η random connections between G a and G b as A conn 4: Mix-up adjacency matrix A (mix) a with Eq. ( 13) 5: Mix-up edge mask M (mix) a with Eq. ( 14) 6: Mix-up node features X (mix) = [X a ; X b ] 7: return G (mix) = (X (mix) , A (mix) a ⊙ M (mix)G + , G -← Compare similarity(G b , G c ) to G 6: G (mix)+ ← Mix-up (G, G + ) 7: G (mix)-← Mix-up (G, G -) 8:Compute L NCE (G, G + , G -) with Eq. (Compute L GIB and overall loss L with Eq. (end for 11:Update E with back propagation. 12: end for 13: return Explainer E Algorithm 2 shows the training procedure for our explainer. For each epoch and each to-be-explained graph G, we first randomly sample two neighbors and decide the positive neighbor G + and negative neighbor G -according to the similarity between their embedding vectors respectively. The graph with higher similarity to G is the positive neighbor G + . We generate the explanation for graphs and mix G with G + and G -respectively. We calculate the InfoNCE for triplet ⟨G, G + , G -⟩ with Eq. ( 6) and the GIB loss, which contains the size loss and InfoNCE loss. We also calculate the MSE loss between f (G (mix)+ ) and f (G). The overall loss is the sum of size loss, InfoNCE loss, and MSE loss. We update the trainable parameters in the explainer with the overall loss.We provided implementation details for our experiments in this section. Data and code are available in the Anumuous git repo and supplementary.All experiments are conducted on a Linux machine (Ubuntu 16.04.4 LTS (GNU/Linux 4.4.0-210generic x86_64)) with 4 NVIDIA TITAN Xp (12 GB) GPUs. CUDA version is 11.8 and the Driver version is 520.56.06. All codes are written with the Python version 3.8.13 with PyTorch 1.12.1 and PyTorch Geometric (PyG) 2.1.0.post1, torch-scatter 2.0.9, and torch-sparse 0.6.15. We adopt the Adam optimizer throughout all experiments. Overall, for each dataset, a GCN regression model is well-trained first, where we take a three-layer GCN model as the backbone. Then the explainers take the to-be-explained GNN and original graph and generate explanations for its prediction on the dataset. After that, we evaluate the performance of the explanation. We split the dataset into 8:1:1, where we train the GNN base model with 8 folds, and train and test explainer models with 1 fold respectively. The hyper-parameters are illustrated in the paper correspondingly.We formulate Three synthetic datasets and a real-world dataset, as is shown in Table 1 , in order to address the lack of graph regression datasets with ground-truth explanations. (1) BA-Motif-Volume: This dataset is based on the BA-shapes [23] and makes a modification, which is adding random float values from [0.00, 100.00] as the node feature. We then sum the node values on the motif as the regression label of the whole graph, which means the GNNs should recognize the [house] motif and then sum features to make the prediction. ( 2) BA-Motif-Counting: Different from BA-Motif-Volume, where node features are summarized, in this dataset, we attach various numbers of motifs to the base BA random graph and pad all graphs to equal size. The number of motifs is counted as the regression label. Additionally, we pad base graphs to dynamic size to prevent the GNNs from making trivial predictions based on the total number of nodes. (3) Triangles: We follow the previous work [39] to construct this dataset. The dataset is a set of 5000 Erdős-Rényi random graphs denoted as ER(m, p), where m = 30 is the number of nodes in each graph and p = 0.2 is the probability for an edge to exist. The size of 5000 was chosen to match the previous work. The regression label for this dataset is the number of triangles in a graph and GNNs are trained to count the triangles. ( 4) Crippen: The Crippen dataset is a real-life dataset that was initially used to evaluate the graph regression task. The dataset has 1127 graphs reported in the Delaney solubility dataset [40] and has weights of each node assigned by the Crippen model [42] , which is an empirical chemistry model predicting the water-actual partition coefficient. We adopt this dataset, firstly shown in the previous work [43] , and construct edge weights by taking the average of the two connected nodes' weights.We compared the proposed RegExplainer against a comprehensive set of baselines in all datasets, including: (1) GRAD [23] : GRAD is a gradient-based method that learns weight vectors of edges by computing gradients of the GNN's objective function. (2) ATT [41] : ATT is a graph attention network (GAT) that learns attention weights for edges in the input graph. These weights can be utilized as a proxy measure of edge importance. (3) GNNExplainer [23] : GNNExplainer is a model-agnostic method that learns an adjacency matrix mask by maximizing the mutual information between the predictions of the GNN and the distribution of possible sub-graph structures. (4) PGExplainer [24] : PGExplainer adopts a deep neural network to parameterize the generation process of explanations, which facilitates a comprehensive understanding of the predictions made by GNNs. It also produces sub-graph explanations with edge importance masks. ( 5) MixupExplainer [19] : MixupExplainer adopts the graph mix-up approach with PGExplainer and address the Out-Of-Distribution problem in graph classification tasks. • If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Justification: There is no societal impact of the work performed, other than helping human beings better understand the black-box GNNs models.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).38th Conference on Neural Information Processing Systems (NeurIPS 2024).Our data and code are available at: https://github.com/jz48/RegExplainer 


Decision: This paper carefully crafted a new method for explanation of predictions of graph regression neural networks. The method utilizes a novel objective based on information bottle neck theory and new mix-up framework; it is an add-on (post training) model agnostic method. 

This paper is well written and easy to follow. Several reviewers praise the theoretically justified objective that was proposed. While experiments are a weak part of this paper, as mentioned by several reviewers,  synthetic datasets performance seems really convincing. The area of explainers for regression GNN is under-explored and will certainly benefit from this paper. Authors provide the code as an artifact, that will further increase the adoption of this work.
Furthermore, authors did an excellent job rebutting, resulting in several raised scores.  

The paper should be strengthened by adding new experimental results on real world datasets.

Reviews: review: paper_summary: paper_summary: The paper addresses the challenge of interpreting graph regression models, a fundamental yet less explored task in graph learning compared to classification tasks. Existing explanation techniques are predominantly designed for classification, resulting in a gap for regression tasks. Based on the recent advances on information bottleneck and mix-up framework, the author proposes a novel objective to interprete GNN models in regression tasks.
strength_weakness: strength_weakness: 1. Some of the model designs are not moviated consistently over a pronounced challenge. The paper seems to be a combination of Mixupexplainer, GNNExplainer and G-Mixup. 

2. The challgenges of graph regression explainer and distribution shift seem to be independent. The author doesn't make any justification on the graph topology and only embedding/predicted values are reported in Figure 3.
questions: questions: 1. In Figure 6, why the selection of $\alpha$ seems do not affect the overall performance? It seems the InfoNCE loss is negeligile in the optimization of the proposed method
limitations: limitations: N/A
paper_summary: paper_summary: The paper introduces XAIG-R, a novel explanation method for interpreting graph regression models. It addresses distribution shifting and decision boundary issues, leveraging the graph information bottleneck theory (GIB) and self-supervised learning.
strength_weakness: strength_weakness: - Limited Discussion on Computational Efficiency
questions: questions: - In this work, GIB is used as Explainer, have you also studied other instance-based approach like GNNExplainer etc.
-  How does the proposed method handle dynamic graph topology changes, and what are the implications for real-world applications with evolving graph structures? (Directed Graph, Hyper Graph)
- Any other general-used datasets are tested? Especially those real-world graph datasets.
limitations: limitations: The authors adequately addressed the limitations.
paper_summary: paper_summary: The authors propose an explanation method to interpret the graph regression models. The techniques are built upon the information bottleneck theory and contrastive learning. The authors show that their explanations are accurate in five graph regression datasets. 

Note: If authors address my concerns in questions and limitations, I am willing to upgrade my allocated score.
strength_weakness: strength_weakness: The paper is densely written in Section 4 and is hard to follow. Instead of explaining each part of the algorithm separately and clearly explaining why these components exist, the authors rely too much on theorizing the problem. For example, I am unsure if a reader is particularly interested in knowing about lower and upper bounds and distribution shifts before seeing the proposed method, and maybe moving this section away as motivation can solve this problem. 

The authors have mentioned that their evaluation relies on ground truth, but the measures for evaluating explanations are barely discussed. See questions below.
questions: questions: Where are the ground truth vectors for your evaluation? How are they obtained? How can ground truth for explanations be in the datasets? This is the most important part of your paper, and it is left to the imagination of readers. 

I argue that your technique is better because you add more samples, and the surrogate is learning just that. So basically, you can add this sampling to those explanations, and you don't need the rest of your method (evidence in Section 5.3 Figure 5 except for the BA-motiv dataset). Can you argue against this concern? 

Why are the ablation results different for BA-Motiv-Volume? 

What is the real limitation of your technique? You have written, "Instead, they write, " Specifically, although our approach can be applied to explainers for graph regression tasks in an explainer-agnostic manner, it cannot be easily applied to explainers built for explaining the spatio-temporal graph due to the dynamic topology and node features of the STG." What does this mean?
limitations: limitations: I personally think the main limitation is that this approach is an add-on approach. I am not in favor of add-on model agnostic approaches. What I mean is explanations that sit on top of other explanation techniques. I think this makes the design of these techniques extra complicated, and finding faults in explanations becomes harder. I think if the authors take their sampling and improve it, it can replace the GNN and PGE and become a more general approach. 

The authors have also not stated that the graph regression task is not a very popular task in graph settings. Based on this, I also would like to see how this approach can be extended for use in other graph tasks: node classification etc.
paper_summary: paper_summary: This work proposes a method to generate instance-level GNN prediction explanations specifically for graph regression tasks. This method addresses distribution shifting, a problem in regression, by using mix-up for contrastive learning. The work is evaluated on four datasets, both synthetic and real-world.
strength_weakness: strength_weakness: More real-world datasets would demonstrate this method’s more general applicability. The evaluation is limited by the inclusion of only one real-world dataset, Crippen.

See questions.
questions: questions: How does this method perform with/against classification explanation models? As the explainer can be modularly applied to existing trained GNNs, this would be interesting to see if it can improve the accuracy of explanation generations. This work [1] is a good work on instance and model-level explanations for graph classification tasks.

Is graph mix-up performed on the graphs explicitly or within latent space? If the mix-up occurs in graph space, then how is the subgraph G* combined with (G^+)^{Delta} and (G^-)^{Delta}? How are edges added between these different subgraphs? Furthermore, for more complex regression datasets, naively adding edges can drastically change the label for each graph. Is this strategy then limited to graph regression datasets which inherently rely on graph structures to derive labels? If the mix-up occurs within latent space, then how does this work differentiate itself from graph rationalization works [2, 3]? If the mix-up occurs in latent space, then additional studies to compare against graph rationalization baselines should be included.

[1] Xuanyuan, Han, et al. "Global concept-based interpretability for graph neural networks via neuron analysis." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 9. 2023.

[2] Wu, Ying-Xin, et al. "Discovering invariant rationales for graph neural networks." arXiv preprint arXiv:2201.12872 (2022).

[3] Liu, Gang, et al. "Graph rationalization with environment-based augmentations." Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2022.
limitations: limitations: The work requires an accurate trained prediction model. The model explanations are not used to retrain the GNN in any way.

Additional limitations are addressed in the conclusion.
paper_summary: paper_summary: The authors focus on the interpretability of GNN in graph regression tasks. They propose a novel explanation method called RegExplainer as a plug-in to existing explanation methods, such as GNNExplainer and PGExplainer. They also tackle mutual information estimation (graph information bottleneck), distribution shifting and continuously ordered decision boundaries that hinder current explanation techniques.
strength_weakness: strength_weakness: 1. The **contribution** is confusing:

1a. In the introduction (Lines 64-68), the authors mention *datasets* as a key contribution. However, the paper lacks detailed descriptions of these datasets in later sections. More information about the datasets, including their structure and significance, should be included to clarify this contribution.

2. Concerning the **methods** and corresponding **experimental settings**:

2a. The *subgraph connectivities* are not considered. In other words, RegExplainer could result in isolated nodes rather than "a subgraph".

2b. **Property 2 is overclaimed**. Lines 469-471 dismiss the ground truth joint distribution $p(Y^*,Y)$, and only focus on the remaining parts. The assumption weakens the paper, especially since "distribution shifting" is a key discussion of the paper.

2c. Section 4.3, especially Step 2, is hard to follow. The process for obtaining $(G^+)^{\Delta}$ and $G^*$ needs more detail. Specifically, Step 2 is used to approximate $I(G, G^*)$ and find the optimal $G^*$ as in Eq. (4) or (5). However, $G^{(mix)}$ relies on $(G^+)^{\Delta}$ and $G^*$. Given that $G^*$ is the variable to be optimized, how to pick $(G^+)^{\Delta}$ is missing.

2d. Whether $G^*$, $(G^+)^{\Delta}$ ,as well as $G^{mix}$ are binary {0,1} or continuous in [0,1] is unclear. If they are binary, the optimization for $G^*$ needs more explanation. If they are continuous, the relaxation and recovery to the final outcomes should be specified. 

2e. The method of measuring distances between graphs or graph distributions is not clearly defined. The authors should clarify and justify whether MSE/RMSE are used, or another metric is needed. 

Besides, the authors evaluate the distribution shifting by similarity/distance between embeddings (Table 3) rather than other distribution-based metrics.

2f. The last column in Table 2 shows a large distance from the original prediction, suggesting the *subgraph may not be faithful*. 

2g. The problem definition in Lines 113-115 is unclear, how to define "can" explain?
The problem definition in Lines 113-115 is unclear, particularly the criteria for an explanation to be considered valid (e.g., “can explain the prediction”).

3. **Others**:

3a. The method appears suited for graph-level regression tasks. Could it apply to the node-level regression tasks, such as traffic prediction?

3b. "N" in Line 110 is undefined.

3c. What does "regression label in regression task" mean on Line 164?
questions: questions: Please address the concerns in the "Weakness".
limitations: limitations: The authors show the limitations on Sec. 6. 

However, as mentioned above, the assumption and proof of Property 2 should be carefully considered.

Comments: title: Thank you for your review
Respectfully Requesting an Update
Second Round Reply (2/2)
Second Round Reply (1/2)
Rebuttal Part 2
comment: Dear reviewers, 

We would like to express our sincere gratitude for your thoughtful and detailed reviews of my paper. Your feedback has been invaluable in helping us refine and strengthen our work. We truly appreciate the time and effort each of you has dedicated to providing constructive criticism and suggestions.

We will carefully apply revisions to the paper based on all the discussions and findings, ensuring that it meets the rigorous expectations of both the reviewers and the broader academic community.

We are also open to further discussions. Please feel free to let us know if you have any additional questions or concerns. We are happy to continue the dialogue with you.

Thank you once again for your review and support. We hope you have a great week.
Dear reviewer KBtu,

Thank you for your valuable feedback on our paper. We truly appreciate the time you have taken to review our work and provide detailed insights. Your comments help us a lot in refining our next version of the paper.

We understand that you may have other commitments, but if you have any additional questions or require further clarification on any of our responses, we would be grateful for the opportunity to address them. **Your feedback is crucial to us, and we are eager to ensure that our revisions align with your expectations.**

Please feel free to reach out at your convenience, as we would be more than happy to discuss any remaining concerns or provide further information.

Thank you for your continued support.
Thank you for the clarification! I will raise my score.
Dear reviewer X7ek,

Thank you very much for your reply. We truly appreciate the time and effort you invested in evaluating our work and providing valuable feedback. 

Although the score did not change, your constructive comments are invaluable to us, and we are committed to addressing all your concerns thoroughly. We are grateful for your support and the opportunity to enhance our work.

If you have any further suggestions or need additional clarifications, please let us know. We are more than happy to provide any additional information or address any remaining questions.
Dear reviewer B4J7,

Thank you very much for your thoughtful review and for increasing the score of our paper. We sincerely appreciate the time and effort you invested in evaluating our work and providing valuable feedback. Your positive assessment and constructive comments have been instrumental in helping us improve the paper.
Dear Reviewer LTFm,

Thank you once again for your detailed and insightful feedback. We are committed to addressing all concerns and ensuring the highest quality of our work. Your comments have been incredibly valuable, and we have made clarifications and provided more analysis based on your suggestions.

To ensure we fully meet your expectations, could you please provide any further feedback or confirm if the revisions address your concerns? Your prompt response would be greatly appreciated as we finalize our revisions.

Thank you for your time and effort.
Dear Reviewer KBtu,

Thank you once again for your detailed and insightful feedback. We are committed to addressing all concerns and ensuring the highest quality of our work. Your comments have been incredibly valuable, and we have made clarifications based on your suggestions.

To ensure we fully meet your expectations, could you please provide any further feedback or confirm if the revisions address your concerns? Your prompt response would be greatly appreciated as we finalize our revisions.

Thank you for your time and effort.
> Re to A-3c:

A: This term is also used in previous works[5, 6, 7, 8]. We will make a clear description in the next version of the paper.

[1]. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation Learning with Contrastive Predictive Coding.

[2]. Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. Gnnexplainer: Generating explanations for graph neural networks. Advances in neural information processing systems, 32, 2019

[3]. Dongsheng Luo, Wei Cheng, Dongkuan Xu, Wenchao Yu, Bo Zong, Haifeng Chen, and Xiang Zhang. Parameterized explainer for graph neural network. Advances in neural information processing systems, 33:19620–19631, 2020.

[4]. Xu Zheng, Farhad Shirani, Tianchun Wang, Wei Cheng, Zhuomin Chen, Haifeng Chen, Hua Wei, Dongsheng Luo. Towards Robust Fidelity for Evaluating Explainability of Graph Neural Networks.

[5]. Haoliang Yuan, Junjie Zheng, Loi Lei Lai, Yuan Yan Tang. A constrained least squares regression model.

[6]. Cheng Li, Virgil Pavlu, Javed Aslam, Bingyu Wang & Kechen Qin. Learning to Calibrate and Rerank Multi-label Predictions.

[7]. Xin Ding, Yongwei Wang, Zuheng Xu, William J. Welch, Z. Jane Wang. Continuous Conditional Generative Adversarial Networks: Novel Empirical Losses and Label Input Mechanisms.

[8]. Harikrishna Narasimhan, Andrew Cotter, Maya Gupta, Serena Wang. Pairwise Fairness for Ranking and Regression.
Dear reviewer 4dqr,

We are glad to have further discussion with you. Your insightful comments are valuable to us and help improve the quality of our paper. We are grateful for the time and effort you have invested in providing such a thorough review. If you have any remaining concerns or require further clarification, please let us know. We are more than happy to address any additional questions you may have.

> Re to A-2b:

A: We see your concern regarding property 2. In this property, to develop the GIB objective into graph regression tasks, we firstly have $I(Y^*; Y) = \sum_{Y^*,Y} p(Y^*,Y) \log \frac{p(Y|Y^*)}{P(Y)}$, then, follow the previous work, we apply the “proportional to” trick [1] because $p(Y^*,Y)$ is an unbounded value. We then proportionally optimize $ \sum_{Y^*,Y} p(Y^*,Y) \log \frac{p(Y|Y^*)}{P(Y)}$ with $\text{sim}\left(Y^*, Y\right) \propto \frac{p(Y|Y^*)}{p(Y)}$. This objective is specifically designed for graph regression tasks, and we plan to include more synthetic and real-world datasets in our study to provide a more comprehensive evaluation. Currently, the performance on three synthetic datasets and one real-world dataset demonstrates its effectiveness.



> Re to A-2c:

A: Thank you for your further question. For graph sample G+, we have $(G^+)^*=E(G+)$, where $(G^+)^*$ is the label-preseving subgraph of $G^+$. Then we pick the $(G^+)^{\Delta}=G^+ - (G^+)^*$ as the label-irrelevant subgraph. This procedure is included in the equations above line189. Eg: $G^{\text{(mix)}+}=G^*+ (G^+)^{\Delta} = G^*+(G^+-(G^+)^*)$.
We will provide a more detailed description of this process in Section 4.3, Step 1, in the next version of our paper.

> Re to A-2d:

A: Thank you for pointing this out. We mentioned that we are following previous work and briefly described this from lines 119 to 122 in the paper. We will ensure these clarifications are more explicitly detailed in the next version of the paper.


> Re to A-2e:

A: Thank you for your insightful suggestion. In response, we conducted a survey of metrics commonly used in similar tasks within the literature. The most prevalent metrics include:
Graph Edit Distance (GED)
Wasserstein Distance (Earth Mover's Distance)
Maximum Mean Discrepancy (MMD)
Jensen-Shannon Divergence (JSD, a variant of KL divergence)
Graph Kernel Methods
These metrics vary in complexity and applicability depending on the nature of the graphs and the specific task at hand. Given the context of our work, we believe that MMD might be particularly suitable, while Graph Kernel Methods could also provide valuable insights into graph structure shifts.

**We are currently conducting experiments to evaluate the MMD metric, and we will include these results in the supplementary material of the next version of the paper.** This will help us better justify our choice of metrics and possibly introduce additional metrics for a more comprehensive evaluation.

We appreciate your recommendation and will incorporate these findings into the revised manuscript to enhance the clarity and rigor of our evaluation process.


> Re to A-2g:

A: We adhere to established definitions from previous work when evaluating the explainability of GNNs. For datasets with explanation ground truth, we use the *AUC-ROC* (Area Under the Receiver Operating Characteristic Curve) as the evaluation metric [2, 3] to measure explanation performance, where a higher AUC-ROC score indicates better performance.

For datasets without ground truth explanations, we utilize metrics such as the fidelity-sparsity score and the robust-fidelity score [4] to assess the quality of the explanations. These metrics are designed to balance the trade-off between explanation fidelity (how well the explanation reflects the model's predictions) and sparsity (how concise the explanation is).

In our work, the AUC-ROC metric is employed, with higher scores reflecting superior explanation performance. We rely on this metric because it has been widely accepted in the literature as a standard measure for evaluating the quality of model explanations.
Thank you to the authors for their rebuttal. After reading all of the reviews and responses, I will choose to maintain my score.
Thank you for your response. You have addressed some of my concerns, and I can raise my score to +1—best of luck.
Thanks for the responses. However, some of the answers do not fully address my concerns.

> To A-2b:

If it cannot handle, or ignore, the joint distribution, how could you justify the generality of Property 2?

> To A-2c:

Only the processes of picking $G^+$ and $G^-$ are included, while the processes of picking $(G^{+})^\Delta$ is missing.

> To A-2d: 

Such statements are missing in the manuscript.

> To A-2e:

The metrics should be further justified, since it is the key for evaluating the graph distributions, as well as the methods.
For example, is it possible to quickly survey the metrics used in the papers for the same tasks?
 
> To A-2g:

Is it possible to give a more formal definition or reference? E.g. if the task desires a lower/higher value under some specific evaluation measurements (please make it more clear and formal).

> To A-3c:

It is interesting to refer "regression *label*" to the *prediction* of a GNN.
> Why are the ablation results different for BA-Motiv-Volume? 

A4: This is a very good observation, and I am glad you pointed it out so we can explain it. We will include the relevant explanation in future versions to improve the quality of the paper. 

First, we observed that in the BA-motif-volume dataset and the Crippen dataset, the mixup module has a greater impact on method performance. In contrast, in the other two datasets, the InfoNCE loss (contrastive learning) has a more significant influence. We believe this is due to the characteristics of the datasets: in BA-motif-volume, all graphs are of the same size. Therefore, for a well-trained GNN, the explanation subgraphs (which are only parts of the original graphs) are significantly out-of-distribution (OOD), which reduces model and explainer performance. By using the mixup method to address this issue, we can effectively improve performance. In the other two datasets, the graph sizes are dynamic, and the trained GNNs are relatively more robust, so the performance loss caused by not using mixup is not as substantial.

> What is the real ... mean?

A5: This means that our method cannot be trivially applied to tasks involving dynamic graph structures, such as spatio-temporal graphs (STGs). In STGs, the graph structure changes over time, which poses challenges for selecting samples for contrastive learning and mixup. The dynamic nature of the topology and node features makes it difficult to apply our approach directly. We aim to extend our method to handle dynamic graph structures in future work.

> Limitations

A6: I understand your concern. However, the add-on approach actually offers advantages. Unlike single and fixed explainer methods, our framework can be flexibly applied to existing explainer models, making it suitable for graph regression tasks and improving their performance. In our code, we have also included implementations of Regexplainer based on GNNExplainer and PGExplainer. These implementations can directly replace the original GNNExplainer and PGExplainer and be used straightforwardly.

> The authors have also not stated ... etc.

A7: Our method is specifically designed to improve explaining the graph regression tasks. For graph classification tasks, we can adapt our approach by replacing the MSE loss in the objective function with cross-entropy loss. Additionally, our method can be extended to node-level tasks. For example, in a 3-layer GNN, a node classification task can be viewed as a graph task on 3-hop subgraphs centered around nodes, and then a variant of our method with cross-entropy loss could be easily adapted to the dataset.