Causally-guided Regularization of Graph Attention improves Generalizability
Authors: Alexander P Wu
Thomas Markovich
Bonnie Berger
Nils Yannick Hammerla
Rohit Singh
Venue: Submitted to ICLR 2023 (2023)
Keywords: graph neural network
attention
generalization
regularization
causal effect
causal interventions
interpretability

Abstract: Graph attention networks estimate the relational importance of node neighbors to aggregate relevant information over local neighborhoods for a prediction task. However, the inferred attentions are vulnerable to spurious correlations and connectivity in the training data, hampering the generalizability of the model. We introduce CAR, a general-purpose regularization framework for graph attention networks. Embodying a causal inference approach, CAR aligns the attention mechanism with the causal effects of active interventions on graph connectivity in a scalable manner. CAR is compatible with a variety of graph attention architectures, and we show that it systematically improves generalizability on various node classification tasks. Our ablation studies indicate that CAR hones in on the aspects of graph structure most pertinent to the prediction (e.g., homophily), and does so more effectively than alternative approaches. Finally, we also show that CAR enhances interpretability of attention weights by accentuating node-neighbor relations that point to causal hypotheses. For social media network-sized graphs, a CAR-guided graph rewiring approach could allow us to combine the scalability of graph convolutional methods with the higher performance of graph attention.

CAUSALLY-GUIDED REGULARIZATION OF GRAPH AT- TENTION IMPROVES GENERALIZABILITY
Anonymous authors Paper under double-blind review
ABSTRACT
Graph attention networks estimate the relational importance of node neighbors to aggregate relevant information over local neighborhoods for a prediction task. However, the inferred attentions are vulnerable to spurious correlations and con- nectivity in the training data, hampering the generalizability of models. We intro- duce CAR, reneral-purpose regularization framework for graph attention net- works. Embodying a causal inference approach based on invariance prediction, CAR aligns the attention mechanism with the causal effects of active interventions on graph connectivity in a scalable manner. CAR is compatible with a variety of graph attention architectures, and we show that it systematically improves gener- alizability on various node classification tasks. Our ablation studies indicate that CAR hones in on the aspects of graph structure most pertinent the prediction (e.g., homophily), and does more effectively than alternative approaches. Fi- nally, we also show that CAR enhances interpretability of attention coefficients by accentuating neig-neighhor relations that point to causal hypotheses.
INTRODUCTION
Graphs encode rich relational information that can be leveraged in learning tasks across a wide variety of domains. Graph neural networks (GNNs) can learn powerful node, edge graph-level representations by aggregating node's 's representations with that of its neighbors. The specifics of GNN's neighborhood aggregation scheme are critical to its effectiveness on a prediction task. For instance, graph convolutional networks (GCNs) aggregate information via simple averaging or max-pooling of neighbor features. GCNs are prone to suffer many real-world scenarios where uninformative or noisy connections exist between nodes (Kipf & Welling 2017 Hamilton et al.) 2017). Graph-based attention mechanisms combat these issues by quantifying the relevance of node-neighbor relations and softly selecting neighbors in the aggregation step accordingly (Velick- ovic et al. 2018 Brody et al. 2022 Shi et al. 202T). This process of attending to select neighbors has contributed to significant performance gains for GNNs across a variety of tasks (Zhou et al.) 2018 Veličković 2022). Similar to the use of attention in natural language processing and com- puter vision, attention in graph settings also enables the interpretability of model predictions via the examination of attention coefficients (Serrano & Smith 2019).
However, graph attention mechanisms can be prone to spurious edges and correlations that mislead them in how they attend to node neighbors, which manifests as a failure to generalize to unseen data (Knyazev et al. 2019). One approach to improve GNNs' generalizability is to regularize attention coefficients in order to make them more robust to spurious correlations/commcction in the training data. Previous work has focused on Ly regularization of attention coefficients to enforce sparsity (Ye) Ji (02T) or has co-optimized a link prediction task using attention (Kim & Oh, 2021). Since these regularization strategies are formulated independently of the primary prediction task, they align the attention mechanism with some intrinsic property of the input graph without regard for the training objective.
We take different approach and consider the question: "What is the importance of specific edge to the prediction task?" Our answer comes from the perspective regularization: we introduce CAR, causal attention regularization framework that is broadly suitable for graph attention net- work architectures (Figure []. Intuitively, an edge in the input graph is important to prediction task if removing leads to substantial degradation in the prediction performance of the GNN. The

---

key conceptual advance of this work is to scalably leverage active interventions node neighbor- hoods (i.e., deletion of specific edges) to align graph attention training with the causal impact of these interventions task performance. Theoretically,
approach is motivated by the invariant prediction framework for causal inference (Peters et al., [0166] Wu et al. 2022). While some efforts have previously been made to infuse notions of causality into GNNs, these causal approaches have been largely limited to using causal effects from pre-trained models as features for separate model (Feng et al. 2021 Knyazev et al. 2019) or decoupling causal from non-causal effects (Sui et al. 2021).
We apply CAR on three graph attention architectures across eight node classification tasks, finding that it consistently improves test loss and accuracy. CAR is able to fine-tune graph attention by improving its alignment with task-specific homophily. Correspondingly, we found that as graph het- erophily increases, the margin of CAR's outperformance widens. In contrast, non-causal approach that directly regularizes with respect to label similarity generalizes less well. On the ogbn-arxiv network, we investigate the citations into three intuitive themes. Our causal approach can thus enhance the interpretability of attention coefficients, and we provide qualitative analysis of this improved interpretability. We also present preliminary results demonstrating the applicability of CAR to graph pruning tasks. Due to the size of industrially relevant graphs, it is common use GCNs or sampling-bass based approaches on them. There, using attention coefficients learned by CAR on sampled subnetworks may guide graph rewiring of the full network to improve the results obtained with convolutional techniques.
seelown-weighted by CAR and found them to broadly group
2 METHODS
2.1 GRAPH ATTENTION NETWORKS
Attention mechanisms have been effectively used in many domains by enabling models dynam- ically attend to the specific parts of an input that are relevant to a prediction task (Chaudhari et al.) 2021). In graph settings, attention mechanisms compute the relevance of edges in the graph for prediction task. A neighbor aggregation operator then uses this information to weight the contribu-
tion of each edge (Lee et al. 2019a Li et al. 2016 Lee et al. 20196).
The approach for computing attention is similar in many graph attention mechanisms. A graph atten- tion layer takes as input a set of node features h = (f....., Jun), ). h E Rf, where N is the number of nodes. The graph attention layer uses these node features to compute attention coefficients for each edge: au a( (h, Why), where : IR X RF - (0, 1) is the attention mechanism function, and the attention coefficient aij for an edge indicates the importance of node i's input features to node j. For a node j, these attention coefficients are then used to compute a linear combination of its neighbors' features: ii = EieNo) Osj Wh., s.l. ΣIENG) Oij 1. For multi-headed atten- tion, each of the K heads first independently caleulates its attention coefficients (k) oi with its head-specific attention mechanism (kk) (,,,), after which the head-specific outputs are averaged.
In this paper, we focus three widely used graph attention architectures: the original graph atten- tion network (GAT) (Velickovic et al.) 2018), a modified version of this original network (GATv2) (Brody al. (022)) and the Graph Transformer network (Shi al. 202I). The three architectures and their equations for computing attention are presented in A..
2.2 CAUSAL ATTENTION REGULARIZATION AN INVARIANCE PREDICTION FORMULATION
CAR is motivated by the invariant prediction (IP) formulation of causal inference (Peters et al. 2016 Wu et al. 2022). The central insight of this formulation is that, given sub-models that each contain different set of predictor variables, the underlying causal model of system is comprised of the set of all sub-models for which the predicted class distributions are equivalent, up to a noise term. This approach is capable of providing statistically rigorous estimates for both the causal effect strength of predictor variables as well as confidence intervals. With CAR, our core insight is that the graph structure itself, in addition to the set of node features, comprise the set of predictor variables. This is equivalent to the intuition that relevant edges for a particular task should not only be assigned high attention coefficients but also be important to the predictive accuracy of the model (Figure II.
2

---

Shape prediction
Causel regularization
as contucder
Color prediction
Causal regularization
Node features:
Size
-Shape
Color
thickness indicates attention weight
Figure 1: Schematic of CAR: Graph attention networks learn the relative importance of each nodg-neigh for given prediction task. However, their inferred attention coefficients can be miscalibrated due to noise, spurious correlations, confounders (e.g., node size here). Our causal approach directly intervenes sampled subset of edges and supervises auxiliary task that aligns edge's 's causal importance the task with attention coefficient.
The removal of these relevant edges from the graph should cause the predictions that rely on them substantially worsen.
We leverage the residual formulation of IP to formalize this intuition. This formulation assumes that can generate sub-models for different sets of predictor variables, each corresponding to separate experiment E E. For each sub-model S we compute the predictions Y = g( is the graph structure, X' the set of features associated with G, c is the noise distribution, and S is the set of predictor variables corresponding to S We next compute the residuals R = Y-- IP requires that we perform a hypothesis test on the means of the residuals, with the generic approach being to perform an F-test for each sub-model against the null-hypothesis. The relevant assumptions (6 Fe, and e S for all e 8) are satisfied if and only the conditionals y and (st are identical for all experiments e, f E E.
where
We use an edge intervention-bant based strategy that corresponds precisely to this IP-based formulation. However, we differ from the standard IP formulation in how we estimate the final causal model. While IP provides a method to explicitly construct an estimator the true causal model (by taking the intersection of all models for which the null hypothesis guided regularization of graph attention coefficients as way to aggregate sub-models while balanc- ing model complexity and runtime considerations. In setting, each sub-model corresponds set of edge interventions and, thus, slightly different graph structures. The same GNN architecture trained on each of these sub-models. Given set of experiments & {e} with sub-models S", outputs Y and errors e", we regularize the attention coefficients to align with sub-model errors, thus learning GNN architecture primarily from causal sub-models. Incorporating this regularization as an auxiliary task, we seek to minimize the following loss:
rejected), we rely intervention-
== == + XC
(1)
The full loss function L consists the loss associated with the prediction L?, the loss associated with causal attention task L and a causal regularization strength hyperparameter that medi- ates the contribution of the regularization loss to the objective. For the prediction loss. we have _ + function for the given prediction task, g(") is the prediction for entity n, and y(a) is the ground truth value for entity n. We seek to align the attention coefficient for an edge with the causal effect of removing that edge through the use of the following loss function:
(P (z)(")) ())()))))))) where N is the size of the training set, (? corresponds to the loss
R
L
Σ
e
(1 (3)
(2)
R
Stri
(S.).))ES.))
Here, n represents a single entity for which we aim to make a prediction. For a node prediction task, the entity corresponds to node, and in graph prediction task, n corresponds graph. In this
3

---

paper, assume that all edges are directed and, if necessary, decompose undirected edge into two directed edges. In each mini-batch, we generate R separate sub-models, each of which consists of a set of edge interventions S = ...... R. Each edge intervention in S(r) is represented by set of tuples (n, () which denote a selected edge (i. for an entity n. Note that in node classification task, n is the same as j (i.c., the node with the incoming edge). More details for the
edge intervention procedure and causal effect calculations can be found in the sections below. The causal effect Cij scores the impact of deleting edge (i, through a likelihood ratio test. This causal
(12) via the loss function (t.)). A detailed effect is compared to the edge's 's attention coefficient aij algorithm for CAR is provided in Appendix A.2
Edge intervention procedure We sample a set of edges in each round r such that the prediction for each entity will strictly be affected by at most one edge intervention in that round to ensure effect independence. For example, in a node classification task for a model with one GNN layer, round of edge interventions entails removing only single incoming edge for each node being classified. In the graph property prediction case, only one edge will be removed from each graph per round. Because a model with one GNN layer only aggregates information over a I-hop neighborhood. the removal of each edge will only affect the model's prediction for that edge's 's target node, To select set of edges in the L-layer GNN case, edges are sampled from the 1 h-hop neighborhood of each node being classified, and sampled edges that lie in more than one target nodes' L-hop neighborhood are removed from consideration as intervention candidates.
This edge intervention selection procedure is crucial, as it enables the causal effect of each selected intervention Cij on an entity n to be calculated independently of other interventions on graph. Moreover, by selecting only one intervention per entity within each round, we can parallelize the computation of these causal effects across all entities per round instead of iteratively evaluating just one intervention for the entire graph per entity, significantly aiding scalability.
Calculating task-specific causal effects We quantify the causal effect of an intervention at edge (4,5) on entity n through the use of approximate iikeihhoddratte test
(n) o ((13))
(n) Pij
d(n)
-1) where Pig E (g(n), y(i)
(n) es (0 (n))
Here, 0(3) the prediction for entity n upon removal of edge (i. d(n) represents the node degree for node classification, while it represents the number of edges in a graph for graph classification. Interventions will likely have smaller effects on higher degree nodes due to the increased likelihood of there being multiple edges that are relevant and beneficial for predictions for such nodes. In graph classification, graphs with many edges are likely to be less affected by interventions well. (n) Exponentiating the relative intervention effect Pij by d(n)) is intended to adjust for these biases. We have experimented both with and without raising p to the factor of d, but have found empirically better results with. We do not have a rigorous explanation as to why, although we suspect the correlated nature of edges implies some amount of shrinkage is necessary.
The link function o R + (0, 1) maps its input to the support of the distribution of attention coefficients. The predictions (((m) and Vig (n) are generated from the graph attention network being
trained with CAR. We emphasize that g(v) and g(()) are both computed during the same training run, rather than in two separate runs.
Scalability CAR increases the computational cost of training in two ways: (i) additional evalua- tions of the loss function due to the causal effect calculations. and (ii) searches to ensure independent interventions. CAR performs (RNN) interventions, where R is the number of interventions per en- tity and N is the number of entities in the training set. Because our edge intervention procedure ensures that the sampled interventions per round are independent, the causal effects of these inter- ventions can be computed in parallel. In addition, if edge interventions were sampled uniformly across the graph, ensuring the independence each intervention would require a L-layer deep BFS that has time complexity Obb4), resulting in a worst case time complexity of O(RNb-), where L.
4

---

is the depth of the GNN, and b is the mean in-degree. We note that, in practice, GNNs usually have L Σ 2 layers since greater depth increases the computational cost and the risk of oversmoothing (Li et al. 2018, Chen et al., 2020; Topping et al., 2022). The contribution of this intervention overlap search step would, therefore, be minimal for most GNNs. We are also able to mitigate much of the computational cost by parallelizing these searches. Further speedups can achieved by identifying non-overlapping interventions as a preprocessing step. In summary, we have found CAR to have modest effect on the scalability of graph attention methods, with training runtimes that are only increased by 1.5-2 fold (Appendix A.3)
3 RELATED WORK
The performance gains associated with graph attention networks have led to number of efforts to enhance and better understand graph attention mechanisms (Lee et al. 2019a). One category of methods aims to improve the expressive power of graph attention mechanisms by supplementing given prediction objective with supervised attention mechanism (Feng et al., [ or a self-
supervised connectivity prediction approach (Kim & Oh 2021]. A related set of methods leverage signals from interventions graphs from causal perspective to aid in training GNNs. One class of techniques performs interventions on nodes (Knyazev et al. 2019; Feng et al. 2021). CAL (Sui et al. (02TT, , method designed for graph property prediction tasks, performs abstract interventions on representations of entire graphs instead of specific nodes or edges to identify the causally attended subgraph for given task. Specifically, it uses these interventions achieve robustness rather than directly leveraging information about the effect of these interventions on model predictions.
Our intervention-orienter framework can also be understood a graph structure perturbation method. Perturbation methods can be broadly split into three different categories: graph data aug- mentation (Zhao et al., 2022). structural graph rewiring (Rong et al., 2020). and geometric graph rewiring (Topping et al., (022). Inspired by the success of data augmentation approaches in com- puter vision, graph data augmentation methods seek to generate new training samples through dif- ferent augmentation techniques. One of the earliest methods is DropEdge (Rong et al. (020) duces overfitting by randomly selecting edges from a uniform distribution to delete. Other meth- ods build on DropEdge and select edges according to additional constraints including geometrical invariants (Gao et al. 2021), target probability distributions (Park et al. 202I). and information cri- approach by Wu et al. (2022) takes a graph-editing approach to learn GNNs that are robust to distribution shifts in the data. Structural graph rewiring instead seeks to enforce structural priors such as sparsity homophily during the graph alteration phase. Examples of these rewiring priors include fairness (Kose & Shen, [002 Spinelli et al., 2022), temporal structure (Wang et al. 2021), predicted homophily (Chen et al. 2020), sparsity(Jin et al. 2020; Zheng et al. 2020). or information transfer efficiency (Klicpera al. 2019). Geometric approaches, instead, choose view the graph a discrete geometry and alter the connectivity according to the balanced Forman curvature (Topping et al., (022), the stochastic discrete Ricci flows (Bober et al. 2022), commute times
teria (Suresh et al., 2021). EERM, powerful invariance-baat
al. 2022), et al. 2022). All of these approaches are designed ei- ther for use either in self-supervised learning or in a task-agnostic fashion, and consider the input graph independently of the task at hand.
or algebraic
To summarize, CAR introduces a combination of advances not previously reported: applicability to diverse attention architectures; based supervised regularization (rather than task-agnostic or supervueer regularization) that leads to improved generalization; and a causal approach that scal- ably and directly relates edge's importance its attention coefficient, enhancing interpretability.
4 RESULTS
4.1
EXPERIMENTAL. SETUP
We assessed the effectiveness of CAR by comparing the performance of a diverse range of models trained with and without CAR on 8 node classification datasets. Specifically, we aimed to assess the matching baseline models across various graph atten-
consistency of CAR's R's outperformance tion mechanism and hyperparameter choices. Accordingly, we evaluated numerous combinations of such configurations (48 settings for each dataset and graph attention mechanism), rather than testing
5

---

only limited set of optimized hyperparameter configurations. The configurable model design and hyperparameter choices that we evaluated include the graph attention mechanism (GAT, GATv2, or Graph Transformer). the number of graph attention layers L (1,2), the number of attention heads K' = 11,3,07 5). the number of hidden dimensions F {10, 25, 100, 200), and the regular- ization strength À (0.1, 0.5, 1,5). See Appendix A.4 for details on the network architecture, hyperparameters, and training configurations.
4.2 NODE CLASSIFICATION
Datasets and evaluation: We used a total of 8 real-world node classification datasets of varying sizes and degrees of homophily: Cora, CiteSeer, PubMed, ogbn-arxiv, Chameleon, Squirrel, Cornell and Wisconsin. Each model was evaluated according to its accuracy on a held-out test set. We also evaluated the test cross-entropy loss, as it accounts for the full distribution of per-class predictions rather than just the highest-valued prediction considered in accuracy calculations. See Appendix A.5 for details on dataset statistics, splits, and references.
Generalization performance: We compared both the test accuracy and the test loss of each model when trained with and without CAR. Across all model architecture and hyperparameter choices for a dataset, we applied the one-tailed paired Wilcoxon signed-rank test to quantify the overall outperformance of models trained with CAR against models trained without it. CAR resulted in higher test accuracy in 7 of the 8 node classification datasets and a lower test loss in all datasets (Figure 2 p < 0.05). We report the relative performance when averaging over all hyperparameter choices in Table | (test loss), Table 7 (test accuracy, in Appendix [A.6). and Appendix AA.7 We also observed that even small values of R were effective (Appendix AA.)). We believe that this is due to there being an adequate number of causal-effect examples for regularization, as even in the minimum R- case, we have roughly one causal-effect example per training example.
We also compared CAR with Sui et al. (202T)'s 's CAL, a method for graph property prediction that relies on an alternative formulation of causal attention with interventions on implicit representations. We adapted CAL for node classification by removing its final pooling layer. CAR-trained models substantially outperformed CAL (Appendix (A.9). suggesting that CAR's direct edge-intervention approach results in better generalization. Taken together, these results highlight the consistency of performance gains achieved with CAR and its broad applicability across across graph attention architectures and hyperparameters.
Table 1: Test loss 8 node classification datasets
Cora
CiteSeer
PubMed
ogbu-arxiv
Chameleon
Squirrel
Cornell
Wisconsin
GAT
2.33 + 0.75
3.68 + 2.64
1.05 + 0.24
1.49 + 0.02
1.29 + 0.03
1.48 + 0.02
1.19 + 0.23
0.88 0.11
GAT CAR
1.82 0.77
3,06 + 2.34
0.89 = 0.18
1.49 + 0.02
1.26 : 0.03
1.47 = 0.02
1.14 = 0.17
0.84 0.13
GATv2
2.26 1. 0.54
3.08 1 1.06
1.07 1. 0.29
1.48 1. 0.02
1.28 1. 0.03
1.48 1. 0.02
1.09 1 0.11
0.88 1 0.15
GATv2 + CAR
1.63 1.3--000 0.59
3.04 + 2.07
0.90 + 0.17
1.47 + 0.03
1.24 + 0.03
1.46 + 0.02
1.04 + 0.10
0.81
0.09
Transformer
3.43 = 1.61
9.78 + 8.27
1.79 = 0.88
1.50 - 0.03
1.38 + 0.06
1.50 + 0.02
1.22 = 0.25
1.03 = 0.28
Transf. CAR
1.71 0.50
3.92 2.25
1.60
1
1.
0.12
0.86 0.13
4.3 MODEL INVESTIGATION AND ABLATION STUDIES
Impact of regularization strength We explored 4 settings of X: (0.1, 0.5, 1,5). For 6 of the 8 node classification datasets, CAR models trained with the higher causal regularization strengths (A= (1, 5)) demonstrated significantly larger reductions in test loss (p < 0.05, one-tailed Welch's t-test) compared to those trained with weaker regularization (A f0.1. 0.5)). Notably, all four of the datasets with lower homophily (Chamelon. Squirrel, Cornell and Wisconsin) displayed signifi- cantly larger reductions in test loss with the higher regularization strengths, suggesting that stronger regularization may contribute improved generalization in such settings (Appendix 1.
Connection to spurious correlations and homophily Graph attention networks that are prone to spurious correlations mistakenly attend to parts of the graph that are irrelevant to their prediction task. To evaluate if CAR reduces the impact of such spurious correlations, trained with CAR more effectively prioritized relevant edges. For node classification tasks, the relevant neighbors given node are expected to be those that share the same label as that node.
assessed if models

---

Cora
CiteSeer
PubMed
ogbn-arxiv
10-1a
10
10
-1
10 11
10
0.05
0.05
CAR
ACC
REG LOSS
NC 10
160
LOss
81
06
MM 1965
AGC
MEG
Mel Loos
Baseline
Baseline
Baseline
Baseline
Chameleon
Squirrel
Cornell
Wisconsin
03
< 10
< 10
10
10
10-17
CAR
ACC
ADC
-3.5 -LAS
ASC
1095
Baseline
Baseline
Baseline
Baseline
Graph Attention Mechanism
GAT
GATv2
Transformer
Figure 2: Test accuracy and negative loss node classification datasets. Each point corresponds comparison between baseline model trained without CAR and an identical model trained with CAR. The point size represents the magnitude of the À value chosen for the CAR-trained model. p-values are computed from one-tailed paired Wilcoxon signed-rank tests evaluating the improvement of CAR Attriiieed models over the baseline models.
We, therefore, used the label agreement between nodes connected by edge as proxy for the edge's s ground-truth relevance. We assigned a reference attention coefficient ety to an edge based on label agreement in the target node's 's neighborhood: eij é.z/ Σ EKEN, v, εkj (Kim & Ohl 2021). Here, 6G 1 if nodes and have the same label and ēGj 0 otherwise. N, denotes the in-neighbors of node j. We then calculated the KL divergence of an edge's attention coefficient a; from its reference attention coefficient Ety and summarize model's ability identify relevant edges as the mean of these KL divergence values across the edges in the held-out test set.
We compared these mean KL divergences between baseline models trained without CAR and models trained with CAR across the same broad range of model architecture and hyperparameter choices de- scribed above. We found that CAR-trained models consistently yielded lower mean KI. divergence values than models trained without CAR for 6 of 8 node classification datasets (Figure 3 < 0.05, one-tailed paired Wilcoxon signed-rant test). Notably, this enhanced prioritization of relevant edges was achieved without explicitly optimizing label agreement during training and is an inherent man- ifestation of aligning attention with the node classification tasks' causal effects.
Low homophily graphs associated with greater proportions of introduce more spurious correlations (Zheng et al., 2022). We reasoned that CAR's relative effec- tiveness should be greater in such settings. We assessed this by evaluating CAR 33 synthetic Cora datasets with varying levels of homophily (Zhu et al. 2020). We observed that CAR-trained models outperformed baseline models most substantially low homophily settings, with perfor- mance gains generally increasing with decreasing homophily (Appendix II). Altogether, these results demonstrate that CAR not only more accurately prioritizes the edges that are most relevant to the desired task but also highlights its utility in low homophily settings most prone to spurious correlations.
edges and thus may
Comparison to homophily- based regularization We next assessed if the performance gains by our causal approach could be replicated by a non-causal approach that systematically aligns attention coefficients with generic measure of homophily. To do so, performed ablation study in which
(n) computed from the network being trained (Equation 3) with an alternative score derived from hhmopphil-bbsse classification scheme. Briefly, this scheme entails assigning a prediction for each node based on the counts of its neighbors' labels (see Appendix A2 for details). By regularizing attention with respect to this homophily-ber classification scheme, the attention mechanism for a network will be guided no longer by causal effects ussociated with the network but rather this separate measure of homophily.
we replace the causal effects di
7

---

Cora 10
CiteSeer
PubMed
ogbn-arxiv
10-1h
0.06
10-24
DV (atg
10 DW 1:30
CAR
37 2
KL LLDiiiti
KL DV (xti
DI 1sto
Baseline
Baseline
Baseline
Baseline
Chameleon
Squirrel
Cornell
Wisconsin
< 10
p 10-4
0.05
10
KL DW pamp
R. siv ix10
RL Dv 000
DI un
CAR
(x
(-10-1e
KL. DW ix
KL DWV is10-1e
Baseline
Baseline
Baseline
Baseline
Figure Coherence between attention coefficients and label agreement for AAainrd models, compared baseline. Lower KL divergence implies greater coherence. Point colors and sizes have the same meaning as in Figure 2 p-values are computed from one-tailed paired Wilcoxon signed-rank tests evaluating the improvement of trainerained models over the baseline models.
We compared CAR-trained models with those trained with homophily- basedd regularization by eval- uating the consistency of their test loss improvements relative to the baseline models without reg- ularization. We used the one-tailed paired Wilcoxon signed-rank test to evaluate the significance of the regularized models' test loss improvements across the set of model architecture and hyper- parameter choices, focusing on models trained with the higher Models trained when regularizing attention coefficients with the homophily-based scheme underper- formed those trained with CAR in 7 of the 8 datasets (Table 2b. Interestingly, the homophily-base -based regularization showed overall gain in performance relative to the baseline models (i.e., those trained without any regularization), suggesting that even non-specific regularization can be somewhat useful for training attention. Overall, these results demonstrate the effectiveness of using CAR to improve generalization performance for many node classification tasks.
{1.5) regularization strengths.
Table 2: Ablation study comparing improvements in performance using CAR regularization and Nomophily-based regularization (- (p), one-tailed paired Wilcoxon signed-rank test)
Cora
CiteScer
PubMed
ogbn-arxiv
Chameleon
Squirrel
Cornell
Wisconsin
CAR
12.36
5.92
10.51
1.81
12.77
9.72
6.72
9.34
Homaphily
10.04
5.59
7.30
0.53
12.34
10.97
4.15
8.20
Additional applications To explore the utility of CAR as general-purpose framework, we em- ployed CAR for graph pruning. CAR directly uses local pruning (i.e. edge interventions) to guide the training of graph attention in a manner that down-weight task-irrelevant edges. As such, we reasoned that attention coefficients produced by CAR- models could be used prune task- irrelevant edges (see Appendix A.13 for more details). In this approach, we used CAR's 's edge pruning procedure as pre-processing step for training and inference with GCNs, which are more scalable than graph attention networks (Rossi et al., 2020) but indiscriminate in how they aggregate information over node neighborhoods. We found that using CAR- -aidddd pruning improved the test accuracy of GCNs, outperforming vanilla GCNs trained and evaluated over the full graph well as GCNs trained and evaluated on graphs pruned with baseline graph attention mechanisms. These preliminary results open the door for further exploration of CAR's 's utility these tasks.
4.4 INTERPRETING ATTENTION CORFFICIRNTS: QUALITATIVE ANALYSIS
In addition to providing robust way increase both model quality and generalization, explored the interpretability of CAR attention coefficients in a post hoc analysis. Here, we evaluated the edge- wise difference of the attention coefficients between method and baseline GAT applied to the ogbn-arxiv dataset. In this dataset, nodes represent arXiv papers, edges are citation links, and the prediction task is classify papers into their subject areas.
8

---

We manually reviewed citations that were up/down-weighte by CAR-trained models and observed that these citations broadly fit into one of three categories: (i) weightegg self-citations, (ii) Nown-weighting popular "anchor" papers, citations. In the first case, we found that CAR down-weights edges associated with self-citations (Table 3). The causal story here is clear- machine learning is fast-moving field with authors moving into the field and changing specialties as those specialties are born. Because of this, the narrative are that set of authors constructs to present this idea can include citations to their own previous work from different sub-fields. While these citations help situate the work within the broader literature and can provide background that readers might find valuable, they are not relevant to the subject area prediction task.
(iii) upweighting topicallynarroe papers with few
Table 3: nown-weighted self-citations.
Paper Title
Cited Title (from a different subject area)
A
Viterbinet Deep Learning Based Viterbi Al- gorithm for Symbol Detection
Frequency Shift Filtering for Ofdm Signal Re- covery Narrowband Power Line Communi- cations
-89.1 %
Solving Underdetermined Systems with Error Correcting Codes
Systems of MDS Codes from Units and Idem-
-89.80 %
potents
Performance Analysis for Multichannel Re- ception of OOFSK Signaling
On-Off Frequency-Shiff Keying for Wideband -81.03 %
Fading Channels
In the second case, we found that CAR down-weights edges directed towards popular or otherwise seminal "anchor" papers (Appendix(A. 14 TT]le These papers tends to included introduc- tions to provide references for common concepts or methods, such as Adam, ResNet, and ImageNet. They are also widely cited across subject areas and hence have little bearing on the subject area pre- diction task. Notably, CAR does not simply learn to ignore edges from high-degree nodes. For the Word2 paper, we observed notable increases in attention coefficients for edges connecting it to multiple highly related papers, including a 2.5 x 10th % increase for Efficient Graph Computation for Node2Vec and a 2.0 x 106 % increase for Multi- Diimensionn Explanation of Reviews
In the final case, we observed that CAR weighttt edges directed towards deeply related but largely unnoticed papers (Appendix observed that these papers are those that are close the proposed method. These papers are the type that are often found only after a thorough literature review. Such edges should play a key role in predicting paper's topic and should be up-weighted.
Table 12). In our manual exploration of the data, we
5 CONCLUSION
We introduced CAR, an invariance principle-based causal regularization scheme that can be applied to graph attention architectures. Unlike other invariance- saaseedceebbeed approaches (Wu et al. 2022). our fo- cus is on scalably improving overall generalization rather than handling distribution shifts. Towards that, we introduce an efficient scheme directly intervene multiple edges in parallel. Applying to both homophilic and heterophilic node-classification tasks, we found accuracy improvements and loss reductions in almost all circumstances. We performed ablation studies for a deeper under- standing, and found that CAR aligns attention with task-specific homophily and does so better than homophily-based regularizer. A qualitative review also suggested that the attention-weight changes produced by CAR are intuitive and interpretable.
Understanding how, and improving what, GNNs learn remains a major open problem and an ac- tive area of research. For instance, Zheng et al. (2022) have discussed the challenges that GNNs face when handling low-homophily graphs or when different tasks could be specified on the same underlying graph (e.g., predicting citation year vs. topic in obgn-arxiv). Towards this, our method provides a principled and scalable approach to align attention coefficients with the relevant task. Our work bridges two families of techniques: attention regularization and causal interventions. The synthesis of these techniques is not only a promising direction for enhancing the performance and in- terpretability of graph attention but also opens the door for leveraging similar techniques for general GNNs without attention as well. Lastly, while our graph pruning results are preliminary, they also suggest promising direction for future work scaling CAR-based insights web-scale graphs.

---

REPRODUCIBILITY STATEMENT
To ensure the reproducibility of the results in this paper, we have included the source code for method supplementary materials. The datasets used in this paper all publicly available, and we also use the publicly available train/validation/test splits for these datasets. We provide details these datasets in the Appendix and have provided references them in both the main text and the Appendix. In addition, we have provided detailed descriptions of the experimental setup, model training schemes, model architecture design choices, and hyperparameter choices in the "Experimental Setup" section as well as in Appendix A.4
REFERENCES
Adrián Modrii-R Ahmed Begga, Francisco Escolano, and Nuria Oliver. Diffwire: Inductive graph rewiring via the lovász bound, 2022.
Jakub Bober, Anthea Monod, Emil Saucan, and Kevin N. Webster. Rewiring networks for graph neural network training using discrete geometry, 2022.
Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https::/ f72ximsx7C1
net net/ wwtpeeeete/a...eeeete.eeeeeeee.e fooummidd
Sneha Chaudhari, Varun Mithal, Gungor Polatkan, and Rohan Ramanath. An attentive survey of attention models. ACM Trans. Intell. Syst. Technol. 1265):53:1-53:32, 2021. doi: 10.1145/ 3465055. URL
Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In The Thirty- Artificial Intelligence, AAAI 2020. The Shirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 3438 3445. AAAI Press, 2020. URL https: aaai org iin/xxnddxx.///ee php/
Fourth AAAI Conference
AAAI/article/view//
Fuli Feng, Weiran Huang, Xiangnan He, Xin Xin, Qifan Wang, and Tat-Seng Chua. Should graph convolution trust neighbors? A simple causal inference method. In Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai (eds.), SIGIR '21: The 44th Inter- national ACM SIGIR Conference
Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pp. 1218. ACM, 2021, doi: 10.1145/3404835.3666997 CoRR, htthhtts::// doi org. 10. 1145/ 3404835. 3462971
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. 2019. URL http:://aaaaa org absg///b/// 1903.13.. 02428
Zhan Gao, Subhrajit Bhattacharya, Leiming Zhang, Rick S Blum, Alejandro Ribeiro, and Brian M Sadler. Training robust graph neural networks with topology adaptive edge dropping. arXiv preprint 1rXiv:2206.022 02892, 2021.
William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Infor- mation Processing Systems 2017. December 4-9, 2017, Long Beach. CA, USA, pp.
on large graphs.
2017. URL httts::/ppoceeeiingng 5dd9db5e033dd 9ww.b56a Abst httpp/scc...pppee//////aaaaa/
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual
10

---

Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. htwph//pss//pprceedooo...oeee .tml
cttppcpppaaeeee cc/paperr2220/hatt)
Wei Jin, Yao Ma, Xiaorui Liu, Xianfeng Tang, Suhang Wang. and Jiliang Tang. Graph structure learning for robust graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery && Duta Mining, KDD '20, pp. 66-74, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450379984, doi: 10.1145/3394486. 3403049. URL https:
wrw.orrg///... 11 1144134//444334 T...866.440000 3403049
Dongkwan Kim and Alice Oh. How to find your friendly neighborhood: Graph attention design with self-superviisin. In International Conference on Learning Representations, 2021. URL
https::/oppnrevv/ ew net www..nee/oorumm d 5KUN1
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In 5th International Conference on Learning Reprexentations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. BtRpttts: //oppnreview. net http:/ewwwnnfffroummmmmmm Sattt/ooommmii--AAAaass.
Johannes Klicpera, Stefan Weißenberger, and Stephan Günnemann. Diffusioon Improves Gruph Learning. Curran Associates Inc., Red Hook, NY. USA, 2019.
Boris Knyazev, Graham W. Taylor, and Mohamed R. Amer. Understanding attention and generalization in graph neural networks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp- 4204 4214, 2019. URL https: http:///cccceeeeesseecee neur ips wwtpppppe//1a.ccc/ce 1//2/sh//hh/// 4c5bc 4afod9670066660 1ab101 9ca4b Abst html
Neural Information Pro-
O. Deniz Kose and Yanning Shen. Fair node representation learning via adaptive data augmentation, 2022.
John Boaz Lee, Ryan A. Rossi, Sungchul Kim, Nesreen K. Ahmed, and Eunyee Koh. Attention models in graphs: A survey. ACM Trans. Knowl. Discov. Data, (366):22:162222 2019a, doi: 10.1145/3363574, URL httthttts:://aat wtt.///...r//11..11/333333
Junhyum Lee, Inyeop Lee, and Jaewoo Kang. Self-attention graph pooling. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 3734-3743, PMLR, 2019b. mlr. wwwpppwwweeeppppppppppppp.pp.p / html
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Proceedings of the Thirty-Secome AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 3538-3545. AAAI Press, 2018 URL https::///// bttpa://wwerr /www aaai www.aaaa...c.coa..a.raaaooooocat inneeiiiiixxxxxeco php//
Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard S. Zemel. Gated graph sequence neural networks. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico May 2-4, 2016, Conference Track Proceed- ings, 2016.
05493
Hyeonjin Park, Seunghun Lee, Sihyeon Kim, Jinyoung Park, Jisu Jeong, Kyung-Min Kim, Jung- Woo Ha, and Hyunwoo J Kim. Metropolis-hastings data augmentation for graph neural net- works. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 19010-19020. Curran As- sociates, Inc., 2021. URL St http:////...ppeeeeeeeoo...eeeeea

---

Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant pre- diction: identification and confidence intervals. Journa of the Royal Statistical Society Series B, 78(5):44710022, November 2016. URL https: http://deaas.. reper. wwtpae.pp///..aaa... httpg///blaa/// jorssb/
v #78y201615p947- 1etpp944-111.hhae h12.html
Yu Rong. Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks on node classification. In 8th International Conference Learning Rep- resentations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
Stttp:://pperevviee
net wwtt/eerrreeee Forum?id-Hkx
Emanuele Rossi, Fabrizio Frasca, Ben Chamberlain, Davide Eynard, Michael M. Bronstein, and Federico Monti. SIGN: scalable inception graph neural networks. CoRR, abs/2004.1 110004.11199,, 198, 2020.
URL httttt://aaiiii http//awwvvv.g.g/aaa///a wttp///ggga../oo/////////
Benedek Rozemberczki, Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding. J. Complex Networks, 9(2), 2021. doi: 10.1093/commee/cnab0144 URL https://ca .org/
10.1093/ htt/////mmmee//caa//000
Sofia Serrano and Noah A. Smith. Is attention interpretable? In Anna Korhonen, David R. Traum, and Lluís Märquez (eds.), Proceedings of the 57th Conference of the Association for Computa- tional Linguistics, ACL. 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pp. 2931-299521 Association for Computational Linguistics, 2019. doi: 10. 10.18653/w1/p1..11 1282. URL http:///oooo....111......./1 wttpo///.......///pppp.....
1282
Yunsheng Shi, Zhengjie Huang. Shikun Feng, Hui Zhong, Wenjing Wang, and Yu Sun. Masked label prediction: Unified message passing model for semi-supervised classification. In Zhi-Hua Zhou (ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19.27 August 2021, pp. 1548 1554. ijcai.org, 2021. doi: 0t.24463/ijaai22222121
URL htthsts::// doi. org. /ttp:///o................/...... 20221/32//1
Indro Spinelli, Simone Scardapane, Amir Hussain, and Aurelio Uncini. Fairdrop: Biased edge dropout for enhancing fairness in graph representation learning. IEEE Transactions Artificial
Intelligence, 3(3):344-358 3544, 2022. doi: 10. 10.100//1.2001111333311
Yongduo Sui, Xiang Wang, Jiancan Wu, Min Lin, Xiangnan He, and Tat-Seng Chua. Causal at- tention for interpretable and generalizable graph classification. 2021. doi: 10. 10.1145/3534678. 3539366.
Susheel Suresh, Pan Li, Cong Hao, and Jennifer Neville. Adversarial graph augmentation to im- prove graph contrastive learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https: //openzeview. http://wwweee..rrrreeeeee lommmiddiooyyaaa [NsR1K]
Jake Topping, Francesco Di Giovanni, Benjamin Paul Chamberlain, Xiaowen Dong, and Michael M. Bronstein. Understanding over-squashing and bottlenecks on graphs via curvature. In Interna- tional Conference on Learning Representations, 2022. URL htttps://ppenrreeeee http///peeeeeeeeeeeeeeeeee
forum?1d=7Um
Petar Veličković. Message passing all the way up. In ICLR 2022 Workshop on Geometrical and Topological Representation Learning, 2022. URL bttps://openzeviett wet.i/ewwwwwe..o....eeeeee nerr/frum?
10-BcgGiF2kTe55
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Truck Proceedings. OpenReview.net, 2018. URL Destps://openreve//
kCZ
Yiwei Wang, Yujun Cai, Yuxuan Liang, Henghui Ding, Changhu Wang, Siddharth Bha- tia, and Bryan Hooi. Adaptive data augmentation temporal graphs. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang. and J. Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems, volume 34, pp. 1440-1452. Curran Asso- ciates, Inc., 2021. URL
htw.:/eeeeddinn.....ii.o.ee.eeee www.e//is..cco/ppp//////11111111 pd
12

---

Qitian Wu, Hengrui Zhang, Junchi Yan, and David Wipf. Handling distribution shifts graphs: An invariance perspective. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URLhttps://oppeeeeeee net net/forurm derrummmd==QQoo d000050-1111
Zhilin Yang, William W. Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning with graph embeddings. In Maria-Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 40-48. JMLR.org. 2016. URL http::ttpptppeeeeeeeeeeese
ngs. m www.pp.s......eee. 6. htmmll
Yang Ye and Shihao Ji. Sparse graph attention networks. Data Engineering, 2021.
IEEE Transactions Knowledge and
Tong Zhao, Gang Liu, Daheng Wang, Wenhao Yu, and Meng Jiang. Learning from counterfactual
links for link prediction, 2021.
Tong Zhao, Gang Liu, Stephan Günnemann, and Meng Jiang. Graph data augmentation for graph machine learning: A survey, 2022.
Cheng Zheng, Bo Zong, Wei Cheng, Dongjin Song, Jingchao Ni, Wenchao Yu, Haifeng Chen, and Wei Wang. Robust graph representation learning via neural sparsification. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume I 119 of Proceedings of Machine Learning Research, pp. 1448-11468, PMLR, 13-18 Jul 2020. URL https: cettps:/pproceedinggg
mlr www.s/ppeessvllll https//ss......reo/o/////////n/0
Xin Zheng, Yixin Liu, Shirui Pan, Miao Zhang, Di Jin, and Philip S. Yu. Graph neural networks for graphs with heterophily: A survey, CoRR, abs/2202.07082, 2022. URL httppttss://axiit
org/abs/220l 22tp:2222..00 07082
Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Graph neural networks: A review of methods and applications. CoRR, abs/1812.08434, 2018. URL
http://arxiv
08434
Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Be- yond homophily graph neural networks: Current limitations and effective designs. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL httpo/////ttteeppeeeeee.....goo hwtp::/www...s..c.rreeeeeeeeaeae cc/paperr/222//aas/
130 html
13

---

A APPENDIX
A.1 GRAPH ATTENTION ARCHITECTURE VARIANTS
Table 4: Attention coefficient caleulation across graph attention architecture variants
GAT Velickovic et al.
2018
Eij LeakyReLU a" WhillWhh
v2 (Brody al. 2022
Gj a' LeakkRR (Wh, (W)))
(Wh,) (Wh,)
Graph Transformer (hi et al.
2021
Cij
A.2 CAR ALGORITHM
Algorithm CAR Framework
Input: Training set Dirain. validation set Duals model M, regularization strength À repeat
for each mini-batch (B [n1 (1) by 1=1 do
Prediction loss: C f B&T EneBe E (g)(n)) .y(m)
procedure EDGE INTERVENTION
Causal attention loss: L +- for round r + to It do
Set of edge interventions S(r) 0 for each entity {nz fj=1 do
Sample edge (i,j) ~ E n; if (i independent of S(r) then
> E nn
set of edges related to entity ng.")
>> See "Edge intervention procedure" Add edge to set of edge interventions
S(r) S() (n) (M) ...))
Compute causal effect ci t (p
(n) d(m) -1)
Equation
end end for L c C
Emnisess co (a (a ste
(a)
Equation
end for end procedure
Total loss: C= C ACE
Update model parameters to minimize L end for
until Convergence criterion
We use convergence of the validation prediction loss.
A.3
RUNTIME STATISTICS
Table 5: Training times for node classification datasets
# nodes
# edges
Train time
Train time
Dataset
(train set)
(train set)
w/o CAR (sec)
w/ CAR (sec)
Cora
140
638
1.1 + 0.4
2,2 2.1
CiteSeer
120
364
1.3 1 0.5
2.1 1. 1.7
PubMed
60
297
1.4 4 0.7
2.3 2.6
ogbn-arxiv
32,970
93,942
165 78
286 166
Chameleon
1,092
17,157 +: 1,013
1.6
2.6 1.I
Squirrel
2,496
105,517 + 3,062
2.7 0.6
4.0 + 1.9
Comell
87
148 +: 18
1.3 1.2
1.4 144:00 0.9
Wisconsin
120
239 10
+ 0.4
0.6
14

---

A.4 GRAPH ATTENTION NETWORK ARCHITECTURES AND TRAINING
A.4.1 NODE CLASSIRICATION MODEL ARCHITECTURE
The GNN model used for node classification tasks takes as imput the original node features 2 R and applies non-linear projection to these features to yield a set of hidden features hi LeakyReLU (i)+ where W E IR xd and b E RR.. These hidden features are then passed through graph attentional layers of the same chosen architecture, yielding new hidden features per node of the same dimensionality MC E FR.. The pre- and post-graph attention layer hidden features are then concatenated [hx Kg] after which final linear layer and softmax transformation axoftmax (.) are applied to produce the prediction output 9; sutimma ( W2LeakkReLU (lh,) [[[]] + by). Here, f: E CRC W 1R2FxC, and b2 E R.F, where C is the number of classes in the classification task. Models were implemented in PyTorch and PyTorch Geometric (Fey & Lenssen 2019). Self-loops were not included the graph attention layers; otherwise, default PyTorch Geometric parameter settings were used for the graph attention layers.
A.4.2 TRAINING DETAILS
We used cross-entropy loss for the prediction loss ((..)) and binary cross-entropy loss for the causal regularization loss (-:-). The link function (-) chosen be the sigmoid function with temperature 7 = 0.1. Unless otherwise specified, we performed = rounds of edge interventions per mini-batch when training with CAR.
All models were trained using the Adam optimizer with a learning rate of 0.01 and mini-batch size of 10,000. Each dataset was partitioned into training, validation, and test splits in line with previous work (Appendix(A.5) and early stopping was applied during training with respect to the validation loss. Training was performed single NVIDIA Tesla T4 GPU.
A.S DATASETS
We provide overviews of the various node classification datasets along with accompanying statistics in Table For all datasets, use the publically available train/validation/eest splits that accompany these datasets.
Planetoid:
The Cora, CiteSeer, and PubMed datasets are citation networks from Yang et al. (2016). Nodes represent documents and directed edges represent citation links. Nodes are featurized as bag-of-word representations of their respective documents. The prediction task for this dataset is to classify a given paper into its respective subject area.
ogbn-arxiv: The ogbn -oxiv dataset is citation network between computer science arXiv paper indexed by MAG (Hu et al.) 2020). Nodes represent papers and node's 's features are the mean embeddings of words its corresponding paper's title and abstract. Edges are directed and represent a citation by one paper of another. The prediction task for this dataset is to predict the subject area of given arXiv paper.
Wikipedia:
The Chameleon and Squirrel datasets are Wikipedia networks from Rozemberezki et al. (02T), in which nodes represent web pages and edges represent hyperlinks between them. Nodes featurized bag-of-won representations of important nouns in their respective Wikipedia pages. Average monthly traffic of web pages are converted into categories, and the prediction task assign given page its corresponding category.
The Cornell and Wisconsin datasets are networks of web pages from various computer science departments, in which nodes represent web pages and edges are hyperlinks between them. Node features are bag-of-word representations of their respective web pages, and the prediction task assign given web page to the category that describes its content.
WebKB:
15

---

Table 6: Node Classification Dataset Statisties
Dataset
# classes
# nodes
# edges
# splits
Mean degree
Homophily
Cora
7
2,708
10,556
3,9
0.81
CiteSeer
3,327
9,104
2.7
PubMed
3
19,717
88,648
1
4.5
0.80
ogbn-arxiv
40
169,343
1,166,243
6.9
0.66
Chameleon
5
2,277
36,051
10
15.8
0.23
Squirrel
5
5,201
216,933
10
41.7
0.22
Comeel
5
183
295
10
1.6
0.12
Wisconsin
5
251
499
10
2.0
0.17
A.6
TEST ACCURACY ON NODE CLASSIFCAATION DATASETS
Table 7: Test accuracy 8 node classification datasets
Cora
CiteSeer
ogbn-arxiv
Chumeleon
Squirrel
Cornell
Wisconsin
GAT
0.58 0.13
0.46 + 0.09
0.69 0.01
0.56 0.01
= 0.01
0.37 0.01
0.66 0.06
0.78 :
GAT
0.60
+ 0.06
0.71 0.01
0.56 0.01
+
0.37 0.01
0.67
GATv2
0.60 0.14
0.47 + 0.08
0.69 0.01
0.56 0.01
0.51 + 0.01
0.37 0.01
0.68 0.04
0.79 4 0.02
GATv2 CAR
0.61
0.50 + 0.06
0.71 0.01
0.56 0.01
0.51 + 0.01
0.37 0.01
0.70
0.79 + 0.02
Transformer
+
0.72 0.02
0.56
+
0.37 0.01
0.70 0.02
+
Transformer
A.7
CHANGES IN TEST ACCURACY AND LOSS BY GRAPH ATTENTION MECHANISM AND REGULARIZATION STRENGTH
Table 8: Average percent change test accuracy
Cora
CiteSeer
PubMed
Value
GAT
GATv2
Transformer
GAT
GATv2
Transformer
GAT
GATv2
Transformer
A 51.0,5.0)
+3.8% +2.3%
+7.4% +6.6%
+7.0% +7.6%
+7.4% +4.2%
+7.1% +3.0%
+12.5% +1.5%
+2.7% +0.4%
E (0.1,0.5)
+0.3%
0.0%
ogbn-arxiv
Chameleon
Squirrel
Value
GAT
GATv2
Transformer
GAT
GATv2
Transformer
GAT
GATv2
Transformer
E 5.00
-0.9%
-0.6%
-0.3% -0.7%
-0.2%
-1.1%
-0.2% 0.0%
+0.3%
+0.4%
-0.5%
0..1.0.5
-0.3%
+0.3%
+0.2%
+0.5%
-0.1%
-0.4%
Cornell
Wisconsin
A Value
GAT
GATv2
Transformer
GAT
GATv2
Transformer
(1.0,500 C (0.1,0.5)
+1.8%
+3.1%
+0.6% +0.7%
+0.6%
+0.4% 0.0%
+1.2%
+1.4%
+1.6%
+0.4%
+0.9%
16

---

Table 9: Average percent change in test loss
Cora
CiteSeer
PubMed
À Value
GAT
GATv2
Transformer
GAT
GATv2
Transformer
GAT
GATv2
Transformer
E (1.0,5.0)
-22.6%
-290%%
-4366 %
-1.7% -12.9%
+7.6%
-45.0%
-2.8% -2.8%
13.9%
-7.0%
A (0.1,0.5)
-15.3%
-19.8%
-28.8%
1.2%
-20.9%
-2.9%
-1.1%
ogbn-arxiv
Chameleon
Squirrel
Value
GAT
GATv2 Transformer
GAT
GATv2
Transformer
GAT
GATv2
Transformer
À 5.00,500
+0.6% -0.1% %
+1.9% -0.2% %
-0.2% -0,8%
-2.4%
-2.7% -0.7%
-3,8% -0.8%
-1.1% -0.2%
-1.5%
-2.0%
À (0.1.0.5)
-0.7%
0.0%
0.0%
Cornell
Wisconsin
Value
GAT
GATv2
Transformer
GAT
GATv2
Transformer
E (1.0,50)
-1.8% -1.4%
-3.1%
-0.6 %
-0.6% -0.4%
-0.4% 0.0%
-1.2% -0.9%
A E (0.1,0.5)
-1.6%
-0.7 %
A.8 NUMBER OF ROUNDS OF INTERVENTIONS AND PERCENT CHIANGE IN TEST ACCURACY
Figure 4: Percent change in test accuracy for models trained with CAR, Each boxplot represents all combina- tions of the three graph attention layers (GAT, GATv2, Transformer) and the following sets of hyperparameter choices: x E 5), L = (1,2), K {3). F" [100, 200}.
Cora
CiteSeer
PubMed
% Change
0.4
% Change
0.50
% Change
0.050
0.2
0.25
0.000
0.0
-0.025
3
10
10
10
of Interventions (R)
# of Interventions
# of Interventions (R)
(R)
A.9 COMPARISON OF CAR-TRAINES MODELS WITH CAL MODELS
CAL is an approach for identifying causally attended subgraphs for graph prediction tasks that lever- ages causal interventions on graph representations to achieve robustness of model predictions (Zhao ct al. 2021). While CAL and CAR have related goals of enhancing graph attention using concepts from causal theory, CAL uses abstract perturbations on graph representation perform causal inter- ventions while we propose an edge intervention strategy that enables causal effects to be computed scalably. addition, CAL is designed identify causally attended subgraphs for graph property prediction tasks, while our work primarily focuses on node classification tasks. Furthermore, CAL uses interventions to achieve robustness and does not directly leverage the effects of interventions on model predictions during training.
Despite these differences, we sought to determine whether the causal principles underlying CAL could be effectively applied to the various node classification tasks evaluated in paper. We modified the CAL architecture to make suitable for node prediction tasks by simply removing the final pooling layer that aggregates node representations within each graph directly upstream of classifier, thus enabling node-level prediction. We evaluated the CausalGAT model from CAL using all combinations of the following hyperparameter choices: F' {128, 256), K {1, 2, 4), L {1. 2). A = {0.2, 0.4, 0.6, 0.8, 1}. and A2 = {0.2, 0.4, 0.6, 0.8, 1}, where F' refers to the number of hidden dimensions, K is the number of attention heads, L is the number of GNN layers, and λ and A are CAL-specific hyperparameters. For each dataset, we report the maximum test accuracy observed for the CAL CausalGAT across all combinations of these hyperparameter choices. We compare these test accuracies from the CAL CausalGAT models with the test accuracies from CAR- trained models averaged over all hyperparameter choices, which also appear above in Appendix A.6
17

---

Table 10: Test accuracy on node classification datasets compared CAL
Cora
CiteSeer
PubMed
ogbn-arxiv
Chameleon
Squirrel
Cornell
Wisconsin
GAT + CAR
0.60 +: 0.12
0.49 + 0.06
0.71 0.01
0.56 = 0.01
0.50 + 0.01
0.37 0.01
0.67 0.06
0.78 + 0.03
GATv2 CAR
0.61 # 0.12
0,30 +: 0.06
0.71 + 0.01
0.56 = 0.01
0.51 = 0.01
0.37 = 0.01
0.70 0.03
0.79 4 0.02
Transformer + CAR
0.65 # 0.04
0.47 + 0.09
0.72 + 0.01
0.56 = 0.01
0.50 + 0.01
0.37 0.01
0.71 = 0.02
0.79 + 0.01
CAL CausalGAT
0.49 (best)
0.37 (best)
0.55 (best)
0.21 (best)
0.29 (best)
0.31 (best)
0.51 (best)
0.47 (best)
A.10
REGULARIZATION STRENGTH AND GENERALIZATION PERFORMANCE
Figure 5:
Test loss reduction for CAR-trained models across regularization strengths. p-values computed from one-tailed t-tests evaluating the significance of the test loss reductions for the À E (1,5) CAR-trained models being greater than those the E {0.1, 0.53 CAR-trained models.
Cora
CiteSeer
PubMed
ogbn-arxiv
p 0.05
>
< 10
p 0.05
Reduction in Loss
Reduction in Loss
1.5
Reduction in Loss
20
1.0
Reduction
2
0.5
-0.5
0.0
-0.5
(0.1.0.5) (1.5)
(0.1.0.5) (1.5)
(0.1.0.5) (1.5)
(0.1.0.5) (1.5)
A
Chameleon
Squirrel
Cornell
Wisconsin
10-7
p <
p <
0.05
Reduction in Loss
0.2
Reduction Loss
Reduction Loss
0.75
Reduction in Loss
0.75
0.50
0.05
0.25
0.25
0.00
0.0
0.00
0.00
0.25
0.51
(1.5)
(..1.0.5)
(1.5)
(0.1.055)
(1.5)
(0.1.0.5)
(1.5)
2
A.11 EXPERIMENTAL SETUP AND TEST ACCURACY FOR SYNTHETIC CORA DATASETS
To assess the relationship between the effectiveness of CAR and the homophily of a dataset, we obtained a set of synthetic Cora datasets from (Zhu al. 2020). These synthetic datasets mod- ified versions of the original Cora dataset that feature varying levels of edge homophily h, which is defined as the fraction of edges in graph which connect nodes that have the same class label. Here, is the set of edges, Ya is the class label for node u, and yo is the class label for node v.
K(u, v): (u, EE ya = yoll
(4)
[3]
We evaluated 33 synthetic Cora datasets that spanned 11 different settings for h, each of which were represented by replicate datasets. For each of these datsets, we performed similar analysis as above, in which we aimed to evaluate the consistency of improvements in test loss using CAR across a number of graph attention and hyperparameter choices. We evaluated the GAT, GATv2, and Trans- former graph attention layers along with all combinations of the following sets of hyperparameter choices: Fr (100), > (1,5).
{1, 5). L = {1, 2}. We then performed a one-tailed paired Wilcoxon rank-sum test to quantify the consistency of CAR-trained models' improvement in test loss over baseline models trained without CAR.
18

---

Figure 6: Generalization performance of CAR-trained models compared baseline various levels of edge homophily (- (p). one-tailed paired Wileoxon rank-sum test).
10010 (P)
7.5
5.0
2.5
0.0
0.0
0.10
0.20
0.30
0.50 0.60 0.70 0.80 0.90 0.99
Edge homophily ratio h
A.12 HOMOPHILY-BASEE BASEDD REGULARIZATION DETAILS
For the neighbor voting model, each node's prediction is coftmax-normalicn histogram of class labels from the node's 's neighbors: j(() (Pi...., Pc), where Pk. is the normalized count of neigh- bors with label k. We employed the above edge intervention procedure and causal effect calculations to compute causal effect scores for interventions to this neighbor voting model c These values
are used in place of the network-specific c values from the original implementation of CAR. Oth- erwise, the training procedure for a network trained with this neighbor voting scheme exactly the same as for training with CAR. We note that, for given node, calculating the intervention-affected prediction Vi simply entails updating the normalized counts of the class labels from the node's remaining neighbors after the intervention.
A.13 CAR-GUIDED GRAPH REWIRING
While graph attention networks have demonstrated notable performance gains, its inclusion of graph attention layers currently limits its use in large-scale applications compared to GCNs, for which a number of advances in scalability have been made (Rossi et al. 2020).
To leverage the advantages CAR in graph attention alongside the scalability GCNs, we explored graph rewiring approach based on CAR-guided edge pruning. For given dataset, we first use a trained graph attention network to assign an attention weight for each edge in the training and validation sets, after which edges with attention weights below threshold ar pruned. A GCN is then trained on the rewired training set with early stopping imposed with respect to the validation loss the rewired validation set. The trained GCN is then evaluated We use a similar network architecture for the GCN as the various graph attention networks described in Appendix A.4. with the graph attention layers replaced with graph convolutional layers. We set the number of hidden dimensions in the GCN models to be F" = 100.
similarly rewired test set.
For the Chameleon dataset, we identified the hyperparameter settings that contributed the highest validation accuracy for each of the lne-l GAT, GATv2, and Transformer We then trained GCN models on graphs that are pruned based on each of these models' attention mechanisms. We also pruned graphs using the counterparts of these models that were trained with- out CAR and trained another set of GCN models on these pruned graphs. We evaluated the test accuracy of the GCN models when performing this procedure across various attention thresholds (Figure 7). We observed that training and evaluating GCN models on pruned graphs contributed to enhanced test accuracy compared to the baseline GCN models that were trained and evaluated the original graph. Furthermore, we compared GCN models trained and evaluated on CAR-guides pruned graphs against similar GCN models trained and evaluated on graphs pruned without CAR by computing the area under the curve (AUC) associated with the test accuracies at various attention thresholds. Each AUC calculated as the area below its models' test accuracies line and above the baseline GCN models' test accuracy. CAR-ge -uidgddded graph pruning was associated with higher AUC values across the three graph attention mechanisms, demonstrating the potential for CAR's utility in graph pruning tasks.
models.
19

---

Figure 7: Test accuracy of models trained and evaluated graphs rewired using graph attention
causal
(AUC 0.031)
0.50
GAT
(AUC 0.019)
GATv2 causal
(AUC 0.034)
ACC
0.49
GATv2
(AUC 0.030)
0.48
Transformer causal
(AUC 0.029)
Transformer
(AUC 0.023)
0.47
GCN without rewiring
0.00 0.25 0.50 0.75 1.00 Attention Threshold
A.14 ATTENTION INTERPRETABILITY
Table 11: Dewn-weighted citations of "anchor" papers. Each of the cited works that is down-
weighted is well known ML paper.
Paper Tide
Cited Title
Reservoir Computing Hardware with Cellular
Adam: A Method for Stochastic Optimization -99.8%%
Automata
Quantization Networks
Deep Residual Learning for Image Recogni- -99.86% tion
Compressive Hyperspherical Energy Mini- mization
ImageNet Large Scale Visual Recognition -99.75% Challenge
Table 12: Upweighted highly relevant citations.
Paper Title
Cited Title
A
Generalized Random Gilbert Var- shamov Codes
Expurgated Random Coding En-
7700 %
sembles Exponents Refinements and Connections
Sign Language Recognition Gen- eration and Translation: an Inter- disciplinary Perspective
Swift a SignWriting Improved Fast Transcriber Random Beamforming over Quasi- Static and Fading Channels: De- terministic Equivalent Approach
2470 %
Optimal Selective Feedback Poli- cies for Opportunistic Beamform- ing
536
20


Reviews:
[{"review_id": "zNfTN41i77", "reviewer": "Reviewer_5nKt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a regularization framework for training graph attention networks. The proposed method is based on the idea of performing edge interventions to determine the relative importance of an edge to specific prediction tasks. Extensive experiments show that the new regularization framework helps improve generalization while at the same time makes the attention coefficients more interpretable.", "summary_of_the_review": "The paper proposes an interesting regularization framework to improve the generalizability and interpretability of graph attention networks. The idea is interesting and the paper is well written. However the overall method development is incremental and I still have some questions regarding the empirical results. I would increase the score if the authors properly address those questions.", "strength_and_weaknesses": "Pros: \n- The paper is well structured, well motivated and nicely written. The presentation is clear, the idea is interesting, and the results are easy to follow.\n\nCons/Questions:\n- The test accuracies reported in Table 11 in Appendix A.6 are much lower than what are typically achieved by GNNs. For example, the test accuracy for Cora is usually higher than 0.8 for most existing GNN methods with 1-2 layers, but in Table 11 the test accuracy for GAT is only 0.58. Could the authors comment on what differences had caused such a large gap between existing results and the results reported in this work?\n- For semi-supervised node classification task, usually only a small fraction (e.g. 5% or less) of nodes are labelled. In this case, does it mean that the regularization loss in Equation 3 is computed only on the training nodes, and consequently only the attention coefficients for the training nodes are regularized? If this was true, then the regularization would have no effect on the test nodes. I think I must have misunderstood something, could the authors help clarify the effect of regularization on attention coefficients for nodes whose labels are unknown?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "oiQyGxoyVp", "reviewer": "Reviewer_ZGgA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a framework for learning attention weights on the edges of the graph through a graph attention framework. The learned attention is causally-guided, which means that it aims at aligning the attention mechanism with the causal effects of interventions on the graph connectivity. The causal effect of an edge is measured through the impact it has in the final task, thus it is integrated in a loss function of the training phase. The authors show that the proposed framework improves the performance on node classification tasks in standard datasets. Moreover, the learned attention seems to be more interpretable.", "summary_of_the_review": "Below are a few comments that should be addressed by the authors, and could hopefully improve the quality:\n\n1) The causal attention regularization scheme is not clear. A summary of the algorithm should be provided, or a schematic overview that would help the reader understand how Eq (5) is integrated into (3). For example, $\\sigma$ in Eq 5 is not defined. Please make the algorithm as accessible as possible to readers.\n\n2) Is there a reason why the regularisation is applied only to attention networks? Could it be applied to any other architecture as well by adding a regularisation term that represents the auxiliary supervised prediction task that aim to align the weights of the networks with the causal effect of removing specific edges or nodes on the graph?\n\n3) The definition of causality is vague as it captures more correlation of a specific edge with the outcome than causality. Can you comment/clarify that part?\n\n4) How do you define R? It seems that it is an important parameter of your algorithm, and one of the reasons why it does not scale on real graphs. \n\n5) While the method is tested in different datasets, the results are not sufficiently described. Can you comment on the performance of homophily/heterophily datasets? Is there any specific pattern? Does this improved attention scheme help in one or another direction?\n\n6) The neighbor voting regularization scheme is not clear. Please make it precise and clear. \n\n7) The graph pruning approach needs also more details. Why is that method more interesting than classical pruning approaches? \n\n8) While the intuition of the paper is reasonable, there is no clear conclusion on the usefulness of the method. The authors touch upon different aspects such as 1) generalisation 2) interpretability 3) graph pruning, but the results are not strong enough in any of these aspects. The postdoc analysis of the interpretability weight on the ogbn-arxiv is also relatively vague, as some of the statements could be subjective. \n\n9) The authors should include F1 scores in the experiments.\n", "strength_and_weaknesses": "Strength: The proposed idea is relatively intuitive and valid. The paper is generally easy to follow. \n\nWeaknesses: Parts of the paper (e.g., definition of causality, discussion of experimental results) are superficially treated (please refer to the detailed comments below)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bkvM5izUx8H", "reviewer": "Reviewer_J9zG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper claims that existing attention-based GNNs are vulnerable to spurious correlations, thereby hampering the generalizability of the model. To improve the generalization ability of the attention-based GNNs, the authors propose a causal inference approach. Specifically, they use a causally-inspired regularization scheme (CAR).  It can be applied to any attention-based GNN architecture. They conduct experiments on node classification tasks, and find the accuracy improvements and loss reductions.", "summary_of_the_review": "The authors claim that they can solve the spurious correlation problem, but give no examples of their task or failures of existing methods. The description and theoretical justification of the method in this paper are insufficient. ", "strength_and_weaknesses": "Strength:\n1.\tThey conduct extensive experiments to demonstrate the effectiveness of their work.\n2.\tThe issue of spurious correlation in GNNs is significant.\n\nWeaknesses:\n1. The authors claim that they can solve the problem of spurious correlations. But where is the spurious correlation in their tasks? The authors do not give any detailed descriptions or examples, nor do they show examples of failures of mainstream approaches.\n2. Does the example in Figure 1 exist? The authors use the example of Figure 1 to introduce confounders, but it is doubtful whether there is a corresponding phenomenon in the real-world dataset.\n3. The authors claim that they can improve ANY attention-based GNNs, but only GAT-like networks are introduced in the experiments. But there exist numerous attention-based methods for GNNs [1-5]. It is questionable then whether it will work on other types of attention mechanisms.\n4. For equation (5), the authors only give the regularization term they propose, but do not explain it from a theoretical and causal point of view. For equation (4), the author does not give a detailed definition of $L^p$.\n5. How the edge intervention process is implemented, the author does not give a detailed algorithm process or model architecture diagram to explain in detail.\n6. For experiments, the author reproduces the statement that they use a public data splitting method. However, compared to the public results, the performance of node classification is questionable, for example, the performance of GAT on Cora is probably around 80-83, based on mainstream packages, such as Pyg or DGL.\n7. The authors claim that their method can improve generalization by overcoming spurious correlations, but lack many baseline methods, such as [6].\n\n[1] Gated Graph Sequence Neural Networks, ICLR 2016\n[2] How to find your friendly neighborhood: Graph attention design with self-supervision, ICLR 2020\n[3] Self-attention graph pooling, ICML 2019\n[4] Understanding Attention and Generalization in Graph Neural Networks, NeurIPS 2019\n[5] Causal Attention for Interpretable and Generalizable Graph Classification, KDD 2022\n[6] Handling Distribution Shifts on Graphs: An Invariance Perspective, ICLR 2022\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "V3eSMLtfQfN", "reviewer": "Reviewer_iUpK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The Paper argues that existing studies to improve the generalizability of attention mechanism in GNN are independent of the actual prediction without regard for the training objective. Thus the paper leverages active interventions on nodes’ neighborhoods by deleting some specific edges to align graph attention training with the causal impact of these interventions on task performance. Experiments on three graph attention architectures across an eight-node classification task show improvements.", "summary_of_the_review": "The proposed method is effective and easy to follow. However, the novelty needs to justify based on related works. And an analysis and discussion of the causal effect score is also required.", "strength_and_weaknesses": "Strengths:\n1. The paper conducts abundant experiments on a broad range of datasets, which shows consistent performance to demonstrate the effectiveness of CAR.\n2. Qualitative analyses are provided to suggest that the attention-weight changes produced by CAR are intuitive and interpretable.  \n\nWeaknesses:\n1. The paper shares a lot of common idea with the work “Causal Attention for Interpretable and Generalizable Graph Classification, KDD22”, but lack discussion and comparison with it.\n2. The paper does not provide an analysis and discussion of the causal effect score, which is a very important component of the work. Is it theoretical solid in its current form? And experiments are also required to prove its effectiveness compared to other estimations of the causal effect.\n3. Equation 5, p_ij^{n} can be greater than 1 in practice, since removing some noisy edges can also improve the performance. And d(n) >1 will make the value C greater when d(n) increases. However, the author claims that “per-edge attention weights are expected to be smaller for higher-degree nodes”.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject