Causally-guided Regularization of Graph Attention Improves Generalizability
Authors: Alexander P Wu
Thomas Markovich
Bonnie Berger
Nils Yannick Hammerla
Rohit Singh
Venue: Accepted by TMLR ()

Abstract: Graph attention networks estimate the relational importance of node neighbors to aggregate relevant information over local neighborhoods for a prediction task. However, the inferred attentions are vulnerable to spurious correlations and connectivity in the training data, hampering the generalizability of models. We introduce CAR, a general-purpose regularization framework for graph attention networks. Embodying a causal inference approach based on invariance prediction, CAR aligns the attention mechanism with the causal effects of active interventions on graph connectivity in a scalable manner. CAR is compatible with a variety of graph attention architectures, and we show that it systematically improves generalizability on various node classification tasks. Our ablation studies indicate that CAR hones in on the aspects of graph structure most pertinent to the prediction (e.g., homophily), and does so more effectively than alternative approaches. Finally, we also show that \methodname enhances interpretability of attention coefficients by accentuating node-neighbor relations that point to causal hypotheses.

Published in Transactions on Machine Learning Research (05/2023)
Causally-guided Regularization of Graph Attention Improves Generalizability
Alexander P. Wu MTT CSAIL
alerun@mmit.cmm edu
Thomas Markovich
thomasmarrknnichh@@maaa.ccm
Twitter Corter
Bonnie Berger MIT CSAIL and Deportment of Mathematics
bab edu
Nils Y. Hammerla
w.hammerto@gmaillccom
Twitter Corter
Rohit Singh" MIT CSAIL
rsingh @nith..mtt..mmm edu
Reviewed on OpenReview: https: // openreview. net/ forum? (DNM2gjJuJ
Abstract
Graph attention networks estimate the relational importance of node neighbors to aggregate relevant information over local neighborhoods for a prediction task. However, the inferred attentions are vulnerable to spurious correlations and connectivity in the training data, ham- pering the generalizability of models. We introduce CAR, a general-purposo regularization framework for graph attention networks. Embodyyng a causal inference approach based on invariance prediction, CAR aligns the attent ion mechanism with the causal effects of active interventions on graph connectivity in a scalable manner. CAR is compatible with a variety of graph attention architectures, and we show that systematically improves generalizabil- ity on various node classification lasks. Our ablation studies indicate that CAR hones on the aspects of graph structure most pertinent to the prediction (e.g., homophily), and does so more effectively than alternative approaches. Finally, we also show that CAR enhances interpretability of attention coefficients by accentuating node-neighbor relations that point to causal hy hotheess..
1 Introduction
Graphs encode rich relational information that can be leveraged in learning tasks across a wide varicty of domains. Graph neural networks (GNNs) can learn powerful node, edge or graph-level representations by aggregating a node's representations with that of its neighbors. The specifics of a GNN's neighborhood aggregation scheme are critical to its effectiveness on a prediction task. For instance, graph convolutional networks (GCNs) aggregate information via simple averaging or max-pooling of neighbor features. GCNs are prone to suffer in many real-world scenarios where uninformative noisy connections exist. between nodes (Kipf & Welling 2017: Hamilton et al. 2017). Graph-based attention mechanisms combat these issues by quantifying the relevance of node-neighbor relations and softly selecting neighbors in the aggregation step accordingly (Velickovic et al.) 2018). This process of attending to select neighbors has contributed to significant performance gains for GNNs across a variety of tasks (Zhou et al.) 2018 Veličković 2022). These successes have led to the development of a broad class of attention-based GNNN architectures (Brody et al.
*Co-corresponding authors

---

Published in Transactions on Machine Learning Research (05/2023)
Shi et al., 2021,, which include some of the most widely used GNN architectures spanning diverse 2022
range of domains (Bronstein et al. 2021 Fey & Lenssen 2019: Wang et al.) 2019). Similar to the use of attention in natural language processing and computer vision, attention in graph settings also enables the interpretability of model predictions via the examination of attention coefficients (Serrano & Smith 2019).
However, graph attention mechanisms can be prone to spurious edges and correlations that mislead them in how they attend to node neighbors, which manifests as failure to generalize to unseen data (Knyazev et al. 2019). These vulnerabilities are particularly pertinent in settings where node features are noisy [Fountoulakis et al. 2022). One approach to improve GNNs' generalizability is to regularize attention cocfficients in order to make them more robust to spurious correlations coniiinconcoonnnnionnooooooo in the training data. Previous work has focused on Lo regularization of attention coefficients to enforce sparsity (Ye & Ji 2021) or has co-optimized a link prediction task using attention (Kim & Oh 2021). Since these regularization strategies are formulated independently of the primary prediction task, they align the attention mechanism with some intrinsic property of the input graph without regard for the training objoctive.
We take a different approach and consider the question: "What is the importance of specific edge to the prediction task?" Our answer comes from the perspective of regularization: we introduce CAR, a causal attention regularization framework that is broadly suitable for graph attention network architectures (Figure D. Intuitively, an edge in the input graph is important to a prediction task if removing it leads to substantial degradation in the prediction performance of the GNN. The key conceptual advance of this work is to scalably leverage active interventions on node neighborhoods (i.e., deletion of specific edges) and regularization to align graph attention training with the causal impact of these interventions on task performance. Theoretically, our approach is motivated by the invariant prediction framework for causal inference (Peters et al.) 2016 Wu et al. 2022 While some efforts have previously been made to infuse notions of causality into GNNs, these causal approaches have been largely limited to using causal effects from pre-trained models as features for separate model (Feng et al. 2021 Knyazev et al., 2019) or decoupling causal from non-causal effects (Sui et al. 2021). Notably, these existing methods do not directly charactcrize the causal effects of individual edge interventions during graph atlention training in a manner that is specific to the training objective.
We apply CAR on three graph attention architectures across cight node classification tasks, finding that it consistently improves test loss and accuracy. CAR is able to fine-tune graph attention by improving its alignment with task-specific homophily. Correspondingly, we found that as graph heterophily increases, the margin of CAR's outperformance widens. In contrast, a non-causal approach that directly regularizes with respect to label similarity generalizes less well. On the ogbn-arxiv network, we investigate the citations ho/ddwn--eighten
by CAR and found them to broadly group into three intuitive themes. Our causal approach can thus enhance the interpretability of attention coefficients, and we provide qualitative analysis of this improved interpretability. We also present preliminary results demonstrating the applicability of CAR to graph pruning tasks. Due to the size of industrially relevant graphs, it is common to use GCNs sampling- based approaches on them. There, using attention coefficients learned by CAR on sampled subnetworks may guide graph rewiring of the full network to improve the results obtained with convolutional techniques.
2 Methods
2.1 Graph Attention Networks
Attention mechanisms have been effectively used in many domains by enabling models to dynamically attend to the specific parts of an input that are relevant to prediction task (Chaudhari et al., 2021). In graph settings, attention mechanisms compute the relevance of edges in the graph for a prediction task. A neighbor aggregation operator then uses this information to weight the contribution of each edge (Lee et al. 2019a
Li et al. 2016 Lee et al. 2019b).
The approach for computing attention is similar in many graph attention mechanisms. graph attention
layer takes as input set of node features (hi...., byy. ). h, E R², where N is the number of nodes. The graph attention layer uses these node features to compute attention coefficients for each edge: aij a(h;, hj);
2

---

Published in Transactions on Machine Learning Research (05/2023)
Causal regularization
Shape prediction
confounder
Color prediction
Causal regularization
Node features:
Size
Shape
Color
Edge thickness indicates attention weight
Figure Schematic of CAR: Graph attention networks learn the relative importance each node-neighbor for given prediction lask. However, their inferred atlention coefficients can be miscalibrated due to noise, spurious correlations, or confounders (e.g.. node size here). Our causal approach directly intervenes on a sampled subset of edges and supervises auxiliary task that aligns an edge's cansal importance to the task with its attention coefficient.
where FF xR - (0, 1) is the attention mechanism function, and the attention coefficient Cij for an edge indicates the importance of node i's input features to node j- For node j, these attention coefficients are then used to compute a linear combination of its neighbors' features: h/ = Σ aiy aiy Whi,
s.L. Σ iENG)
0
iEN()
For multi-headed attention, each of the K heads first independently calculates its own attention coefficients (k) oi with its head-specific attention mechanism (.,),, after which the head-specific outputs are averaged.
Tn this paper, we focus three widely used graph attention architectures: the original graph attention network (GAT) [Velickovic et al. 2018), modified version of this original network (GATv2) (Brody et al. 2022), and the Graph Transformer network (Shi et al. 2021). The three architectures and their equations for computing attention are presented in Appendix A.1
2.2 Causal Attention Regularization: an Invariance Prediction Formulation
CAR is motivated by the invariant prediction (IP) formulation of causal inference (Peters et. al. 2016 Wu jet al. 2022). The central insight of this formulation is that, given sub-models that each contain different set of predictor variables, the underlying causal model of a system comprised of the set sub-models for which the predicted class distributions are equivalent, up to a noise term. This approach capable of providing statistically rigorous estimates for both the causal effect strength of predictor variables as well as confidence intervals. With CAR, our core insight is that the graph structure itself, in addition the set of node features, comprise the set of predictor variables. This is equivalent the intuition that relevant edges for particular task should not only be assigned high attention coefficients but also be important to the predictive accuracy of the model (Figure D The removal of these relevant edges from the graph should cause the predictions that rely on them to substantially worsen.
We leverage the residual formulation of TP formalize this intuition. This formulation assumes that we can gencrate sub-models for different sets of predictor variables, each corresponding to scparate experiment E E. For each sub-model St, we compute the predictions Y ( X3, é where G is the graph structure, X is the set of features associated with G, e is the noise distribution, and S is the set of predictor variables corresponding to S We next compute the residuals R = y - ye, IP requires that we perform hypothesis test on the means of the residuals, with the generic approach being to perform an F-test for each sub-model against the mull-hypothesis. The relevant assumptions (e ~ F., and e Il. Se for all e E E) are satisfied if and only if the conditionals Se and f are identical for all experiments e, f E E.
We use an edge intervention- pearenttoo--beedd strategy that corresponds precisely this based formulation. However, differ from the standard TP formulation in how we estimate the final causal model. While IP provides
3

---

Published in Transactions on Machine Learning Research (05/2023)
method to explicitly construct estimator of the true causal model (by taking the intersection of all models for which the null typothesis was rejected), we rely on intervention-guidee regularization of graph attention coefficients as a way to aggregate sub-models while balancing model complexity and runtime considerations. In our setting, each sub-model corresponds to a set of edge interventions and, thus, slightly diffcrent graph structures. The same GNN architecture is trained on each of these sub-models. Given a set of experiments ε (e) with sub-models S outputs Y and errors c,, we regularize the attention coefficients to align with sub-model errors, thus learning a GNN architecture primarily from causal sub-models. Incorporating this regularization an auxiliary task, we seek to minimize the following loss:
L= + XC
(1)
The full loss function L consists of the loss associaled with the prediction LP, the loss associated with causal attention task L and causal regularization strength hyperparameter À that mediates the contribution of the regularization loss to the objective. For the prediction loss, we have L N En=r"(( (((("),)()) where N is the size of the training set, ( corresponds to the loss function for the given prediction lask, g/a) is the prediction for entity n, and y(") is the ground truth value for entity n. We seek to align the attention cocfficient for an edge with the causal effect of removing that edge through the use of the following loss function:
R
Σ
e (of))d"))
(2)
R
[Sr)]
(m,i))))S)))
Here, n represents a single entity for which we aim to make a prediction. For a node prediction task, the entity n corresponds to node, and a graph prediction task, n corresponds to a graph. In this paper, we assume that all edges are directed and, if necessary, decompose an undirected edge into two directed edges. In each mini-batch, we generate R separate sub-models, each of which consists of set of edge interventions s(r), r r= = I,..,, (see 'Edge intervention procedure' below), R also corresponds to the number of edge interventions per entity. Each edge intervention in S(r) is represented by a set of tuples i.j) which denote a selected edge (i,j) for an entity n. Note that in a node classification task, n is the same as j (i.e., the node with the incoming edge). More details for the edge intervention procedure and causal effect calculations can be found in the sections below. The causal effect city) scores the impact of deleting edge (i,j) through a likelihood ratio test.
R. Note that, because of the specific approach use to define the sets of edge interventions
(n) This causal effect is compared to the edge's attention coefficient 0if via the loss function (.,)). A detailed algorithm for CAR is provided in Algorithm
Edge intervention procedure We sample set of edges in each round r such that the prediction for each entity will strictly be affected by at most one edge intervention in that round to ensure effect independence. For example, in a node classification task for a model with one GNN layer, a round of edge interventions entails removing only a single incoming edge for each node being classified. In the graph property prediction case, only one edge will be removed from each graph per round. Because model with one GNN layer only aggregates information over 1-hop neighborhood, the removal of cach edge will only affect the modcl's prediction for that edge's target node. To select set of edges in the L-layer GNN case, edges are sampled from the 1-hop neighborhood of each node being classified, and sampled edges that lie in more than one target nodes' L-hop neighborhood are removed from considcration as intervention candidates.
This edge intervention selection procedure crucial, as it enables the causal effect of each selected inter- vention (x) on an entity n to be calculated independently of other interventions on a graph. Moreover, by selecting only one intervention per entity within each round, we can parallelize the computation of these causal effects across all entities per round instead of iteratively evaluating just one intervention for the entire graph per entity, significantly aiding scalability.
Calculating task-specific causal effects We quantify the causal effect of an intervention edge (i,j) entity through the use of an approximate likelihood-ratio test.
4

---

Published in Transactions on Machine Learning Research (05/2023)
Algorithm CAR Framework
Input: Training set Dirain, validation set Drat: model M, regularization strength À
repeat
for each mini-batch {B* {n, (n,
bb 1-=1 } do
Prediction loss: L +- [ En EP ( (()))
procedure EDGE INTERVENTION Causal attention loss: L + 0
for round r + 1 to R do
Set of edge interventions S(r) f 0 for each entity {ng
(z) by J== do
Sample edge (i,j) 00000
D E ni
set of edges related to entity nj
(k)
if (1. independent of Sr) then
See "Edge intervention procodure"
Sr) - S(r) (n,,)) ii.))
Add edge to set of edge interventions
(n) Compute causal effect cij t- ( ((p))) (1)) d(n) ( -1)
Equation 3
end if end for L + L +
E Emnanessoo ee (n) (n)
aig "6j"
Equation 2
R 5
end for end procedure Total loss: [ L + X
Update model parameters to minimize C end for until Convergence crilerion
We use convergence of the validation prediction loss.
(n) o
((()))
(n)
d(x)
-1) where
(n) 1 (1/2)) 2/()) Pij (f(n), yy(r))
Pij
(3)
Here, (n) is the prediction for entity n upon removal of edge (i,j). Pij (n)
represents the relative intervention effect, such that Pij (m) > 1 if the intervention at edge (i.j) increases the prediction loss associated with entity (m) > it decreases the prediction loss. This relative intervention effect Pij is exponentiated by an n and Pij
entity-specific value d(n) to account for biases affecting the magnitudes of the intervention effects. For node classification, d(n) represents the node degree. In node classification, intervent ions on a high degree node will likely have smaller effects due to the increased likelihood of it possessing multiple edges that are relevant for its predictions. Exponentiating by the node degree d(n) is intended to adjust for the smaller expected intervention effect for higher degree nodes. Similarly, in graph classification, graphs with many edges are likely to be less affected by individual interventions. In graph classification, therefore, d(n) represents the number of edges a graph to account for this bias. The adjusted intervention effect (el) (()))d())) is an exponentiated ratio that equals 1 if an intervention has no effect, so we normalize this term by subtracting 1. The link function IR + (0, 1) maps its input to the support of the distribution of attention coefficients. The predictions g(n) and ( are generated from the graph attention network being trained with CAR. We
[(n) emphasize that g(n) and Viy are both computed during the same training run, rather than in two separate runs.
Scalability CAR increases the computational cost of training in two ways: (i) additional evaluations of the loss function due to the causal effect calculations, and (ii) searches to ensure independent interventions. CAR performs ORNN interventions, where R is the number interventions per entity and N is the number of entities in the training set. Because our edge intervention procedure ensures that the sampled interventions per round are independent, the causal effects of these interventions can be computed in parallel. In addition, if edge interventions sampled uniformly across the graph, ensuring the independence each intervention
5

---

Published in Transactions on Machine Learning Research (05/2023)
would require L-layer deep BFS that has time complexity O(b"), resulting in worst case time complexity of O(RNb)) where I, is the depth of the GNN, and b is the mean in-degree. We note that, in practice, GNNs usually have L < 2 layers since greater depth increases the computational cost and the risk of oversmoothing (Li et al. 2018 Chen et al.) 2020: Topping et al. 2022). The contribution of this intervention overlap search step would, therefore, be minimal for most GNNs. We are also able to mitigale much of the computational cost by parallelizing these searches. Further speedups can achieved by identifying non- overlapping interventions as a preprocessing step. In summary, we have found CAR to have a modest effect on the scalability of graph attention methods, with training runtimes that are only increased by 1.5-2 fold (A (Appendix A.2
3 Related Work
The performance gains associated with graph attention networks have led to a number of efforts to enhance and better understand graph attention mechanisms (Lee ct al.) 2019a). One category of methods aims to improve the expressive power of graph attention mechanisms by supplementing a given prediction objective with supervised attention mechanism (Feng et al., 2021p, or self-sup connectivity prediction approach (Kim & Oh 2021). A related set of methods leverage signals from interventions
graphs from a causal perspective to aid in training GNNs. One class of techniques performs interventions on nodes (Knyazev et al. 2019 Feng et al. 2021). CAL (Sui et al. 2021), a method designed for graph property prediction entire graphs instead of specific nodes or edges to identify the causally attended subgraph for a given task. Specifically, it uses these interventions to achieve robustness rather than directly leveraging information about the effect of these interventions on model predictions.
tasks, performs abstract interventions on representations
Our intervention-oorenttl framework is also related to several graph structure perturbation methods. Per- turbation methods can be broadly split into three different categories: graph data augmentation (Zhao ct al. 2022), structural graph rewiring (Rong et al. 2020), and geometric graph rewiring (Topping et al., Inspired by the success of data augmentation approaches in computer vision, graph data augmnen- tation methods seek to generate new training samples through different augmentation techniques. One of the earliest methods is DropEdgee (Rong al. 2020) reduces overfitting by randomly selecting edges from a uniform distribution to delete. Other methods build DropEdge and select edges according to additional constraints including geometrical invariants (Gao et al. 2021), larget probability distributions (Park et al. 2021), and information criteria (Suresh et al.) 2021). EERM, powerful invariance-basee approach by Wu et al. ) takes a graph-editing approach to learn GNNs that are robust to distribution shifts in the data. Structural graph rewiring instead seeks to enforce structural priors such as sparsity or homophily during the
2022
graph alteration phase. Examples of these rewiring priors include fairness (Kose & Shen 2022 Spinelli et al.
2022), temporal structure (Wang et al., (002)), predicted homophily (Chen al. 2020). sparsity (Jin et al.
2020 Zheng al. 2020), information transfer efficiency (Klicpera et al.) 2019). Geometric approaches, discrete geometry and alter the connectivity according to the balanced Forman curvature Topping et al. 2022), the stochastic discrete Ricci flows (Bober et al. 2022), commute times (Arnaiz-Rodriguez et al. 2022), or algebraic connectivity Saaeiivity(AAniizeie Arnaiz-Rodrígue
instead, choose to view the graph
et al. 2022]. All of these approaches are designed either for use either in self-supervised learning or in a task-agnostic fashion, and consider the input graph independently of the task at hand.
To summarize, CAR introduces a combination of advances not previously reported: applicability to diverse attention architectures; task-based supervised regularization (rather than task-agnostic or self-supervised regularization) that leads to improved generalization; and causal approach that scalably and directly relates an edge's importance to its attention coefficient, enhancing interpretability.
4 Results
4.1 Experimental Setup
We assessed the effectiveness of CAR by comparing the performance of a diverse range of models trained with and without CAR on 8 node classification datasets. Specifically, we aimed to assess the consistency
6

---

Published in Transactions on Machine Learning Research (05/2023)
of CAR's outperformance over matching baseline models across various graph attention mechanism and hyperparameter choices. Accordingly, we evaluated numerous combinations of such configurations (48 set- tings for each dataset and graph attention mechanism), rather than testing only limited set of optimized hyperparameter configurations. The configurable model design and hyperparameter choices that we evalu- aled include the graph allention mechanism (GAT, GATv2, or Graph Translormer), the number of graph attention layers I {1,2), the number of attention heads K (1,3,5) the number of hidden dimensions F" {10, 25, 100, 200), and the regularization strength À = {0.1, 0.5, 1,5). In all experiments, we used R 5 interventions per node during training. See Appendix A.3 for details the network architecture, hyperparameters, and training configurations.
4.2 Node Classification
Datasets and evaluation: We used total of 8 real-world node classification datasets of varying sizes and degrees of homophily: Cora, CiteSeer, PubMed, ogbn-arxiv, Chameleon, Squirrel, Cornell and Wisconsin. Each model was evaluated according to its accuracy on a held-out test set. We also evaluated the test cross- entropy loss, as it accounts for the full distribution of per-class predictions rather than just the highest-valuee prediction considered in accuracy calculations. See Appendix A.4 for details on dataset statistics, splits, and references.
Generalization performance: We compared both the lest accuracy and the test. loss of each model when trained with and without CAR. Across all model architecture and hyperparameter choices for a dataset, we applicd the onc-tailed paired Wilcoxon signed-rank test to quantify the overall outperformance of models trained with CAR against models trained without it. CAR resulted in higher Lest accuracy in of the node classification datasets and a lower test loss in all datasets (Figure 2 p 0.05). We also report the average performance over all hyperparameter choices in Table as well as the relative change in performance with respect to each hyperparameter choice in Appendix A.5 both of which demonstrate consistent improvement in test. loss and test accuracy. In addition, we observed that the effectiveness of CAR was robust to the choice of the number of interventions Rt with even small values of R being effective (Appendix A.6 We believe that this is due to there being an adequate number of causal-effect examples for regularization, as even in the minimum R case, we have roughly one causal-effect example per training example.
Cora
CiteSeer
PubMed
ogbn-arxiv
p 10-18
10-21
10-5
10-11
10-11
p
10-13
0.05
0.05
CAR
ArC
27
NES LOSS
LOGS
2.0
ACC LOSG
NEC
NEG
05 0.6 2.7
CAB 5.70 8.2 6.6
ACC
MEG Loss
ACC
NEG LOSs
Baseline
Baseline
Baseline
Baseline
Chameleon
Squirrel
Cornell
Wisconsin
0.05
p
21
0.05
10
< 10
< 10
17
NEC Lops
1000
BECC -1.00
CAR
0.37
0.s
0:58
0.10
040
0.50
6.2
NEO LOSS
03500 leo
-.55
-1:20 -1,65
0.5
ACC
NEG L065
ACC
NEC LOS5
Baseline
Baseline
Baseline
Baseline
Graph Attention Mechanism
GAT
GATv2
Transformer
Figure 2: Test accuracy and negative loss on node classification datasets. Each point corresponds to a comparison between a baseline model trained without CAR and identical model trained with CAR. The point size represents the magnitude of the value chosen for the CAR-trained model. p-values are computed from one-tailed paired Wilcoxon signed-rank tests evaluating the improvement of CAR-trained models over the baseline models.
7

---

Published in Transactions on Machine Learning Research (05/2023)
We also compared CAR with Sui et al. 2022 's CAL, a method for graph property prediction that relies an alternative formulation of causal attention with interventions on implicit representations. We adapted CAL for node classification by removing its final pooling layer. CAR-trained models substantially outper- formed CAL (Appendix A.7). suggesting that CAR's direct odge-intervention approach results in better generalization, Taken logether, these results highlight the consistency of performance gains achieved with CAR and its broad applicability across across graph attention architectures and hyperparameters.
Table 1: Test loss and accuracy on 8 node classification datasets. In order to assess the consistency of CAR across hyper-paraan pperr-paraeeter settings, each result below is aggregated over 48 hyperparameter settings for each of the graph attention mechanisms, and we show the mean and standard deviation of metrics across these set settings.
(A) Thst lose
Cora
CiteSeer
PubMed
ogbn-arxiv
Chameleou
Squirrel
Cornell
Wisconsin
GAT
2:53 1 0.75
3.68 1. 2.64
1.05 1 0.24
1.49 1 0.02
1.29 A: 0.03
1.48 a. 0.02
1.19 a 0.23
0.88 4 0.11
GAT CAR
1.82 1. 0.77
3,06 2:34
0.89 1. 0.18
1.49 0.02
1.26 0.03
1.47 0.02
1.14 0,17
0.84 0.13
CATV2
2:25 + 0.54
3.08 + 1.06
1.07 + 0.29
1.48 + 0.02
1.28 = 0.03
1.48 = 0.02
1.09 = 0.11
0.88 # 0.16
GATV2 + CAR
1.63 + 0.69
8.04 2.07
0.90 0.37
1.47 0.08
1.24 0.03
1.46
0.02
1.04
0.10
0.81
0.09
Trandoemer
3.43 # 1.61
9.76 # 8:27
1.79 +: 0.88
1.50 # 0.03
1.38 = 0.06
1.00 = 0.02
1.22 = 0.25
1.033 = 0.28
Trans.. CAR
1.71 0.50
5.92 2.25
1.60 0.59
1.38 0.04
1.48 0.08
1.07 0.12
0.86 0.18
(B) Test accuracy
Cora
CiteSrer
PubMed
ogbn-arsiv
Chameleon
Squierel
Carnell
Wisconsin
GAT
0.58 + 0.13
0.46 + 0.00
0.80 + 0.01
0.56 + 0.01
0.50 + 0.01
0.37 #: 0.01
0.60 + 0.06
0.78 + 0.03
GAT CAR
0.60
0.49 0.00
0.71
0.01
0.50
0.01
0.67
0.06
0.78 0.03
GATY2
0.00 1 0.14
0.47 1 0.08
0.89 1 0.01
0.56 1. 0.01
0.51 1 0.01
0.37 0.01
0.68 1 0.04
0.79 1 0.02
GATY2 + CAN
0.50 0.06
0.01
0.51
0.01
0.70
0.03
0.02
Transfornce
0.60 1 0.04
0.43 1 0.09
0.72 1 0.02
0.56 1 0.01
0.50 1 0.01
0.37 0.01
0.70 1 0.02
0.78 1 0.02
Transformee CAR
0.65 0.04
0.47 0.09
0.72 0.01
0.01
0.50
0.01
0.71 0.02
0.79 0.01
4.3 Model Investigation and Ablation Studies
Impact of regularization strength We explored 4 settings of X: 1,55) For 6 of the 8 node classification datasets, CAR models trained with the higher causal regularization strengths (A = {1, 5)) demonstrated significantly larger reductions in test loss (p < 0.05, one-tailed Welch's t-test) compared to those trained with weaker regularization (X = {0.1, 0.5))- Notably, all four of the datasets with lower homophily (Chamelon, Squirrel, Cornell and Wisconsin) displayed significantly larger reductions in test loss with the higher regularization strengths, suggesting that stronger regularization may contribute to improved generalization in such settings (Appendix A.8
Connection to spurious correlations and homophily Graph attention networks that are prone to spurious correlations mistakenly attend to parts of the graph that are irrelevant to their prediction task. To evaluate if CAR reduces the impact of such spurious correlations, we assessed models trained with CAR more effectively prioritized relevant edges. For node classification tasks, the relevant neighbors to given node are expected to be those that share the same label as that node. The label agreement between nodes connected by an edge is also not explicitly optimized during training, making it suitable for benchmarking CAR's prioritization of relevant edges. We, therefore, used the label agreement as a proxy for each edge's 's ground-truth relevance. Specifically, we assigned a reference attention coefficient Cij to an edge based on label agreement. the target node's neighborhood: eg = / ExEN, êky (Kim & Oh 202T ). Here, éss = 1 if nodes and j have the same label and éG otherwise. N, denotes the m--eaghborr node j. We then caleulated the KL divergence of an edge's attention coefficient oig from its reference attention coelficient eij and summarize model's 's ability to identify relevant edges as the mean of these KL divergence values across the edges in the held-out test set.
We compared these mean KL divergences between baseline models trained without CAR and models trained with CAR across the same broad range of model architecture and hyperparameter choices described above. We found that CAR-trained models consistently yielded lower mcan KL divergence values than models trained without CAR for 6 of 8 node classification datasets (Figure 3p < 0.05, one-tailed paired Wilcoxon signed-rank test). Notably, this enhanced prioritization of relevant edges was achieved without explicitly
8

---

Published in Transactions on Machine Learning Research (05/2023)
optimizing label agreement during training and is an inherent manifestation of aligning attention with the node classification lasks' causal effects. These result, therefore, also point to the improved interpretability of CAR-trained models, which we explore in greater depth below.
Low homophily graphs are associated with greater proportions of task-irrelevant edges and thus may intro- duce more spurious correlations (Zheng et al. 2022). We reasoned that CAR's relative effectiveness should be greater in such settings. We assessed this by evaluating CAR on 33 synthetic Cora datasets with varying levels of homophily (Zhu et al. 2020). We observed that CAR-trained models outperformed baseline models most substantially in low homophily settings, with performance gains generally increasing with decreasing homophily (A (Appendix]a) A.9). Altogether, these results demonstrate that CAR not only more accurately pri- oritizes the edges that are most relevant to the desired task but also highlights its utility in low homophily settings most prone to spurious correlations.
Cora
CiteSeer
PubMed
ogbn-arxiv
K0, DIV ix00 1a)
< 10 18
p < 10 10-19
KL DIV 1eg0 ta
p p> 0.05
p. 10-24
0Tx Ala
DI 1230
2.5
CAR
*
2.0
2.5
2.0
0.5
1.0
19 20
1
35
DIV (x10
DIV (x10
KL (x10 10
(xio "
Baseline
Baseline
Baseline
Baseline
Chameleon
Squirrel
Cornell
Wisconsin
p 10
p < 10
p >> 0.05
p 10-4
KL, DW 1x10
abx AI Tx
KL DIV (x in
DW 1x10
CAR
3
100 15
(x10011))
KL DIV [x10 "9)
KL DIV [x10 "
Baseline
Baseline
Baseline
Baseline
Figure 3: Coherence between attention coefficients and label agreement for CAR-trained models, compared to baseline. Lower KL divergence implies greater coherence. Point colors and sizes have the same meaning as in Figure g p-values are computed from one-tailed paired Wilcoxon signed-rane -ank tests evaluating the improvement of CAR- trained models over the baseline models.
Comparison to homophily-based regularization We next assessed if the performance gains by our causal approach could be replicated by a non-causal approach that systematically aligns attention coefficients with generic mcasure of homophily. To do so, we performed an ablation study in which we replace the causal effects Gj computed from the network being trained (Equation 3) with an alternative score derived from homophilyy classification scheme. Briefly, this scheme entails assigning a prediction for each node based on the counts of its neighbors' labels (see Appendix A. for details). By regularizing attention with respect to this homophily-baset classification scheme, the attention mechanism for a network will be guided no longer by causal effects associated with the network but rather this separate measure of homophily,
We compared CAR-trained models with those trained with homophily- regularization by evaluating the consistency of their test loss improvements relative to the baseline models without regularization. We used the one-tailed paired Wilcoxon signed-rank test to evaluate the significance of the regularized modcls' test loss improvements across the set of model architecture and hyperparameter choices, focusing on models trained with the higher X {1,5) regularization strengths. Models trained when regularizing attention cocfficients with the homophily-basee scheme underperformed those trained with CAR in 7 of the 8 datascts (Table 2 Interestingly, the bomophily-basee based regularization showed an overall gain in performance relative to the baseline models (i.e., those trained without any regularization), suggesting that even non-specific regu- larization can be somewhat usefuu for training attention. Overall, these results demonstrate the effectiveness of using CAR improve generalization performance for many node classification tasks.
9

---

Published in Transactions on Machine Learning Research (05/2023)
Ablation study comparing improvements in performance using CAR regularization and homophily- basod regularization (-log10 (p), one-tailed paired Wilcoxon signed-rank test)
Table 2:
Cora
CiteSeer
PubMed
ogbn-arxiv
Chameleon
Squirrel
Cornell
Wisconsin
CAR
12.36
5.92
10.51
1.81
12.77
9.72
6.72
9.34
Homophily
10.04
5.59
7.30
0.53
12.34
10.97
4.15
8.20
Additional applications To explore the utility of CAR as a general-purpose framework, we employed CAR for graph pruning. CAR directly uses local pruning (i.e. edge interventions) to guide the training of graph attention in a manner that down-weights iask-irrelevant edges. As such, we reasoned that attention coefficients produced by CAR-trained models could be used to prune task-irrelevant edges (see Appendix A.11 for more details). In this approach, we used CAR's edge pruning procodure a pre-processing step for training and inference with GCNs, which are more scalable than graph altention networks (Rossi et al.) 2020) but indiscriminate in how they aggregate information over node neighborhoods. We found that using CAR- guided pruning improved the test accuracy of GCNs, outperforming vanilla GCNs trained and evaluated over the full graph as well as GCNs trained and evaluated on graphs pruned with baseline graph attention mechanisms. These preliminary results open the door for further exploration of CAR's utility on these tasks.
4.4 Interpreting Attention Coefficients: Qualitative Analysis
CAR's ability to improve the prioritization of relevant edges and its implications for interpretability (Section led us to explore the interpretability of CAR attention coefficients in a post hoc analysis. Here, we evaluated the edge-wise difference of the attention coefficients between our method and a baseline GAT applied to the ogbn-arxiv dataset. In this dataset, nodes represent arXiv papers, edges are citation links, and the prediction task is to classify papers into their subject areas.
4.3
We manually reviewed citations that were these citations broadly fit into one of three categories: (i) down-weighting self-citations, (ii) down-weighting popular "anchor" papers, or (ili) upweighting topically-narrow papers with few citations. In the first case, we found that CAR down-weights. edges associated with self-citations (Table 3). The causal story here is clear- machine learning is a fast-moving field with authors moving into the field and changing specialties as those specialtics are born. Because of this, the narrative are that a set of authors constructs to present this idea can include cilations to their own previous work from different sub-fields. While these citations help situate the work within the broader literature and can provide background that readers might find valuable, they are not relevant to the subject area prediction task.
by CAR-trained models and observed that
Table 3: Down-weighted self-citations.
Paper Title
Cited Title (from different subject area) Frequency Shift Filtering for Ofdm Signal Recov- ery in Norrowband Power Line Communicatiins Systems of MDS Codes from Units and Idempo- tents
Viterbinet a Deep Learning Based Viterbi Algo- rithm for Symbol Delection
-89.91 %
Solving Underdetermined Systems with Error Correcting Codes
-89.80 %
Performance Analysis for Mudtichannel Reception of OOFSK Signaling
On-Off Sheqtency-Shfft Keying for Wideband -81.03 %
Foding Channels
In the second case, we found that CAR dowg-weights edges directed towards popular or otherwise seminal "anchor" papers (A (Appendix 1.12 Table TOp. These papers tends to be included in introductions to provide references for common concepts or mothods, such as Adam, ResNet, and ImageNet. They are also widely cited across subject areas and hence have little bearing on the subject. area prediction task. Notably, CAR does not simply learn to ignore edges from high-degree nodes. For the Word2Vec paper, we observed notable incrcases in attention coefficients for edges connecting it to multiple highly related papers, including a 2.5x 10 % increase for Efficient Gruph Computation for Node2 Vec and a 2.0 x 10 % increase for Multi-Diimensional Explanation of Reviews.
In the final case, we observed that CAR up-weighted edges directed towards deeply related but largely Table I.I. In our manual exploration of the data, we observed that these papers are those that are close to the proposed method. These papers are the type that often
unnoticed papers (Appendix Table
10

---

Published in Transactions on Machine Learning Research (05/2023)
found only after thorough literature review. Such edges should play key role in predicting paper's topic and should be up-weighted.
5 Conclusion
We introduced CAR, an invariance principle-based causal regularization scheme that can be applied to graph attention architectures. Unlike other invariance- based approaches (Wu et al. 2022), our focus is on scalably improving overall generalization rather than handling distribution shifts. Towards that, we introduce an efficient scheme to directly intervene on multiple edges in parallel. Applying it to both homophilic and heterophilic mode-clasífication lasks, we found accuracy improvements and loss reductions in almost. all circumstances. We performed ablation studies for deeper understanding, and found that CAR aligns attention with task-specific homophily and does so better than a homophily-basee regularizer. A qualitative review also suggested that the altention-weight changes produced by CAR are intuilive and interpretable.
While we have shown CAR to be a robust approach that consistently increases both model quality and gencralization, our results point to several future extensions that may drive further performance gains. Firstly, causal regularization can be combined with other auxiliary lasks, like self-supervised link prediction, to potentially drive additional improvements in edge prioritization and accuracy. In addition, a node feature intervention sceheme can possibly be integrated with CAR's causal edge interventions to further enhance graph attention training and interpretability. Lastly, while our graph pruning results are preliminary, they also suggest a promising direction for future work on scaling CAR-based insights to web-scale graphs.
Understanding how, and improving what, GNNs learn remains a major open problem and is an active area of research. For instance, Zheng et al. (2022) have discussed the challenges that GNNs face when handling low- homophily graphs or when different tasks could be specified on the same underlying graph (e.g., predicting citation year vs. topic in obgn-arxiv). Towards this, method provides principled and scalable approach to align allention coelficients with the relevant lask. Our work bridges two families of techniques: attention regularization and causal interventions. The synthesis of these techniques is not only a promising direction for enhancing the performance and interpretability of graph attention but also opens the door for leveraging similar techniques for general GNNs without attention as well.
6 Reproducibility Statement
To cnsure the reproducibility of the results in this paper, we have included the souree code for our method as supplementary materials. The datasets used in this paper are all publicly available, and we also use the publicly available aeann/vlliattion/tes splits for these datasets. We provide details these datasets in the Appendix and have provided references to them in both the main text and the Appendix. In addition, we have provided detailed descriptions of the experimental setup, model training schemes, model architecture design choices, and hyperparameter choices in the "Experimental Setup" section as well as in Appendix A.3
References
Adríán Ardaiz-Rod Ahmed Begga, Francisco Escolano, and Nuria Oliver. Diffwire: Inductive graph rewiring via the lovász bound, 2022.
Jakub Bober, Anthea Monod, Emil Saucan, and Kevin N. Webster. Rewiring networks for graph neural network training using discrete geometry, 2022.
Shaked Brody, Uri Alon, and Eran Yahav. How atlentive are graph allention networks? In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25- 25-29, 2022. OpenReview.net, 2022. URL http:: http:///wnneeieeeoooooooooorrr//
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiu preprint arXio:2104.13 134778 2021.
11

---

Published in Transactions on Machine Learning Research (05/2023)
Sneha Chaudhari, Varun Mithal, Gungor Polatkan, and Rohan Ramanath. An attentive survey of attention models. ACM Trans. Intell. Syst. Technol., (2(5):53:1-53:33 2021. doi: 10.1145/346506: URL https:
org/10. 1145/3465055
Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Mcasuring and relieving the over-smoothing problem for graph neural networks from the topological view. In The Thirig-Fourlh AAAT Conference on Artificial Intelligence, AAAI 2020, The Shcooy-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 3438 3445. AAAI Press, 2020. URL https: btrjs.aaai.oom index.ddexxx//e wttpiiiwwww.......eeeeee http//AA///attccl///ees///
Fuli Feng, Weiran Huang, Xiangnan He, Xin Xin, Qifan Wang, and Tat-Seng Chua. Should graph convolution trust neighbors? A simple causal inference method. In Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai (eds.), SIGIR '21: The 44th Interna- tional ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pp. 1208 1218. ACM, 2021. doi: 00.1145/3044855.4400 7/doi. org/ 10. 0141//144/441..../ 3462971
URL https:
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. CoRR, abs/1903.02425, 2019. URL http:://axxiv.org/bbb////// 02288
Kimon Fountoulakis, Amit Levi, Shenghao Yang, Aseem Baranwal, and Aukosh Jagannath. Graph attention retrospective aixiv preprint or ari::2200. 1....1100 13060, 2022.
Zhan Gao, Subhrajit Bhattacharya, Leiming Zhang, Rick S Blum, Alejandro Ribeiro, and Brian M Sadler. Training robust graph neural networks with topology adaptive edge dropping. orXiv preprint arXiv:2106. 02892, 2021.
William T.. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Sys- tems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 1024-1034, 2017. URL https://pproceednngs htarrr/ssscpppppp///////aaa html
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Newral Information Processing Systems 33: Arnual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12. 2020, virtual, 2020. URL https::/ sttps://prroceedings neurips.cc/papeppp hwo/hhaaa/ft00e11116////00183333 wtt.c//ccoaooo-aaaa.a...... Aat.00---sttrat..tt....
Wei Jin, Yao Ma, Xiaorui Liu, Xianfeng Tang, Suhang Wang, and Jiliang Tang. Graph structure learning for robust graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery IE Data Mining, KDD '20, pp. 66-74, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450379984. doi:
3403049, URL https://aoi.org org/
10. 00.1145/3394486 3440443399448633330449 3403049
Dongkwan Kim and Alice Oh. How to find your friendly neighborhood: Graph attention design with self- suporvision. In International Conference on Learning Representations, 2021. URL htttps://pperrevier
het/forrum
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview net, 2017. URL https://openreriew..nn wwt.arrrrioonnet/oouru...dd SJU4ayYg1
12

---

Published in Transactions on Machine Learning Research (05/2023)
Johannes Kliepera, Stefan Weißenberger, and Stephan Günneeaan. Diffusion Improves Graph Learning. Curran Associates Inc., Red Hook, NY, USA, 2019.
Boris Knyazev, Graham W. Taylor, and Mohamed R. Amer. Understanding attention and generalization in graph neural networks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32:
Annual Conference Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancowwer, BC, Canada, pp. 4204-4214, 2019. URL. https::/pprceeeeiie http:/roocedngg...u............ catrrips.ccpppppr//eee
hat1//cbbceee@@kac00000111000000 Abstract. html.
O. Deniz Kose and Yanning Shen. Fair node representation learning via adaptive data augmentation, 2022.
John Boaz Lee, Ryan A. Rossi, Sungchul Kim, Nesreen K. Ahmed, and Eunyce Koh. Attention models in graphs: A survey. ACM Trans. Knowl. Discov. Data, 13(6):62:1 62:25, 2019a. doi: 10.1145/3363574. URL
https://doo org/ 10. 1145/3363574
Junhyun Lee, Inyeop Lee, and Jaewoo Kang. Self-attention graph pooling. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 3734 3743. PMLR, 2019b. URL http://ppoceeddiggg mlr. wttpppssss///eeeeee.ooo
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi- supervised learning. In Sheila A. McIlraith and Kilian Q. Weinberger (eds.), Proceedings of the Thirly- Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Arti- ficial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelli- gence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 3538-3545. 3545. AAAI Press, 2018. URL https:// hwwp//wwwwkoooo/oo aaai wwtg//cc/innng php/ TAAA//AAA11///apee//veeeee
Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard S. Zemel. Gated graph sequence neural networks. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerlo Rico, May 2-4, 2016, Conference Truck Proceedings, 2016. URL http:
//arxiv.orge hhrgg/bb//5500 wwt.a//1/..............
Hyeonjin Park, Seunghun Lee, Sihyeon Kim, Jinyoung Park, Jisu Jeong, Kyung-Min Kim, Jung- Woo Ha, and Hyunwoo J Kim. Metropolis-hastings data augmentation for graph neural networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Infor- mation Processing Systems, volume 34, pp. //proceedings.
Curran Associates, Inc., 2021. URL https: pdf
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Sociely Series B, 78(5):947 1012, November 2016. URL https: wwwpw.cpppc.o....... wwtgg///g///////////////g/ http://///./...////////////11110
html
Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks on node classification. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL http:ts://ppeeeevee...ec..
Emanuele Rossi, Fabrizio Frasca, Ben Chamberlain, Davide Eynard, Michael M. Bronstein, and Federico
Monti. SIGN: scalable inception graph neural networks. CoRR, abs/2004.11199 2020. URL https:
//arxiv. stw.ii.orgg/abb/00.1199
Benedek Rozemberezki, Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding. J. Complex Networks, 9(2), 2021. doi: 10.1093/ /o.109//comnt/cnabbbll
URL https://doi. org/10. 1093 ctt3///cmmee//cab111.
Sofia Scrrano and Noah A. Smith. Is attention interpretable? In Anna Korhonen, David R. Traum, and Lluis Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Baly, July 28- August 2, 2019, Volume 1: Long Papers, pp. 19311-955 Association
13

---

Published in Transactions on Machine Learning Research (05/2023)
for Computational Linguistics, 2019. doi: 10.18653/v1pp19-2222 URL https://bol..og//0011855533 129-1282
Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjing Wang, and Yu Sun. Masked label predic- tion: Unified message passing model for semi-supervised classification. In Zhi-Hua Zhou (ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pp. 1548-155.. ijeai.org, 2021. doi: 10.24963 /tt/:///icai2/11///// URL https://doi.oon org.i..rg111. 10. 24963/ijcai 2021/214
Indro Spinelli, Simone Scardapane, Amir Hussain, and Aurelio Uncini. Fairdrop: Biased edge dropout for enhancing fairness in graph representation learning. IEFF Transactions on Artificial Intelligence, 3(3): 344 354, 2022. doi: 00.1199/TAL.222..111
Yongduo Sui, Xiang Wang, Jiancan Wu, Min Lin, Xiangnan He, and Tat-Seng Chua. Causal attention for interpretable and generalizable graph classification. 2021. doi: 01144//5534778.335
Sushoel Suresh, Pan Li, Cong Hao, and Jennifer Nevillc. Adversarial graph augmentation to improve graph contrastive learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Ad-
vances in Neural Information Processing Systems, 2021. URL https://opearreiee...e wtwp://rrrei...ttooooouuitaa...
1oyq7NsRIKJ
Jake Topping, Francesco Di Giovanni, Benjamin Paul Chamberlain, Xiaowen Dong, and Michael M. Bron- stein. Understanding over-squashing and bottlenecks on graphs via curvature. In International Conference on Learning Represenlations, 2022. URL htttp:://openrrrriee..ettttttt www..ee//oorum/ii///n//a222
Potar Veličkovié. Mcssage passing all the way up. In ICLR 2022 Workshop on Geometrical and Topological
Representation Learning, 2022. URL http:////ppaarrreeeeeeeeeerro/rr
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liò, and Yoshua Bengio. Graph attention networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. JXMpikkk
URL httpp:://openrevierr.
Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, et al. Deep graph library: A graph-centric, berbly-perfommntt package for graph neural networks. arXiv preprint 1rx::11000 01315, 2019.
Yiwei Wang, Yujun Cai, Yuxuan Liang, Henghui Ding, Changhu Wang, Siddharth Bhatia, and Bryan Hooi Adaptive data augmentation on temporal graphs. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Advances in Newral Information Processing Systems, volume 34, pp. 1440-1452. Curran Associates, Inc., 2021. URL https://proceedings. wwtprrps..c.pppee////..... ffle//bbb9994112223333331111
Paper. pdf
Qitian Wu, Hengrui Zhang, Junchi Yan, and David Wipf. Handling distribution shifts on graphs: An invariance perspective. Tn The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL httpe:://pparrviis..n/////orrrr
FQOC5u-1egI
Zhilin Yang, William W. Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning with graph embeddings. In Maria-Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference on Machine Learning, 1CML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 40-48. JMLR.org, 2016. URL http://procesdinggs mlr. press//48. v48//ynng16..... htmll
Yang Ye and Shihao Ji. Sparse graph attention networks. Engineering, 2021.
IEEE Transactions Knowledge and Data
Tong Zhao, Gang Liu, Daheng Wang, Wenhao Yu, and Meng Jiang Learning from counterfactual links for link prediction, 2021.
14

---

Published in Transactions on Machine Learning Research (05/2023)
Tong Zhao, Gang Liu, Stephan Günnemann, and Meng Jiang. Graph data augmentation for graph machine learning: A survey, 2022.
Cheng Zheng, Bo Zong, Wei Cheng, Dongjin Song, Jingchao Ni, Wenchao Yu, Haifeng Chen, and Wei Wang. Robust graph representation learning via neural sparsification. In Hal Daumé III and Aarti Singh (ods.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 11458 11468. PMLR, 13 18 Jul 2020. URL https://proceedi:ng
mlr. wwtpp//eeee/eeeeeeeeee www.19/hhen22..... html
Xin Zheng, Yixin Liu, Shirui Pan, Miao Zhang, Di Jin, and Philip S. Yu. Graph neural networks for graphs with heterophily: A survey. CoRR, abs/2202.07082, 2022. URL htttpss//arxzvvoor////////00...0
Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Graph neural networks: A review of methods and applications. CoRR, abs/ abs/1112.08834, 2018. URL http://arxx http:///aaaaavvorraa////////// org/abs/8822.
08434
Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural In Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, 2020. URL httpps:/pproceeding.. /ttp:////ooooooopppooo/////////p hhash// (5a22288884444436658884222440000 Abstrrcctttt html
15

---

Published in Transactions on Machine Learning Research (05/2023)
A Appendix
A.1
Graph Attention Architecture Variants
Table 4: Attention coefficient calculation across graph attention architecture variants
GAT (Velickovic el al.
2018
Gij = LeakyReLU (A' [Wh, (Wh)))
GATv2 (Brody et al. 2022)
Gjj Á' LeakyReL.U Wh (h)))
Eij- = (Wh) h,,)) (Wh,)
Graph Transformer (Shi et al.
2021
F
A.2 Runtime Statistics
Table 5: Training times for node classification dalasets
# nodes
# edges
Train time
Train time
Dataset
(train set)
(train set)
w/o CAR (sec)
w/ CAR (sec)
Cora
140
638
1.1 + 0.4
2.2 2.1
CiteSeer
120
364
1.3 + 0.5
2.1 = 1.7
PubMed
60
297
1.4 1: 0.7
2.3 2.6
ogbn-arxiv
32,970
93,942
165
286 166
Chameleon
1,092
17,157
1,013
=
2.6 1.1
Squirrel
2,496
105,517
3,062
+
4.0 1.9
Coruell
1
14 0.9
Wisconsin
239 =
0.4
1.1 = 0.6
A.3 Graph Attention Network Architectures and Training
A.3.1 Node Classification Model Architecture
The GNN model used for node classification tasks takes as input the original node features H, E Rd and applies a non-linear projection to these features to yield a set of hidden features h; = Teakk Re where W, E Fxd and b E RRF These hidden features are then passed through L graph attentional layers of the same chosen architecture, yielding new hidden features per node of the same dimensionality 1 E R² The pre- and post-graph attention layer hidden features are then concatenated /h,1(h") after which a final linear layer and softmax transformation (softmax(-) are applied to produce the prediction output d4 = asoftmax WWLLeakyReL... ([h:([())))))))) b2). Here, Úi E IRC W2 22FxC and b2 E RR where C is the number of classes in the classification task. Models were implemented in PyTorch and PyTorch Geometric (Fey & Lenssen 2019). Sclf-loops not included in the graph attention layers; otherwise, default PyTorch Geometrie parameter settings were used for the graph attention layers.
A.3.2 Training Details
We used cross-entropy loss for the prediction loss (P(--) and binary cross-entropy loss for the causal regular- ization loss (...)). The link function of-) was chosen to be the sigmoid function with temperature = 0.1. Unless otherwise specified, we performed R rounds of edge interventions per mini-batch when training with CAR.
All models were trained using the Adam optimizer with a learning rate of 0.01 and mini-batch size of 10,000. Each dataset was partitioned into training, validation, and test splits in line with previous work (Appendix A.4), and early stopping was applied during training with respect the validation loss. Training was performed single NVIDIA Tesla T4 GPU.
16

---

Published in Transactions on Machine Learning Research (05/2023)
A.4 Datasets
We provide overviews the various node classification datasets along with accompanying statistics in Table For all datasets, we use the publically available seain/vvaidaaion/teee splits that accompany these datasets.
The Cora, CiteSeer, and PubMed datasets citation networks from Yang et al. (2016) Nodes represent documents and directed edges represent citation links. Nodes are featurized as bag-of-word representations of their respective documents. The prediction task for this dataset is to classify a given paper into its respective subject area.
Planctoid:
ogbn-arxiv: The ogbn-arxiv dataset is a citation network between computer science arviv paper indexed by MAG (Hu et al.) 2020). Nodes represent papers and a node's features are the mean embeddings of words in its corresponding paper's title and abstract. Edges are directed and represent a citation by one paper of another. The prediction task for this datasct is to predict the subject area of given arXiv paper.
Wikipedia: The Chameleon and Squirrel datasets are Wikipedia networks from Rozemberezki et al. 2022 in which nodes represent web pages and edges represent hyperlinks between them. Nodes are featurized as bag-of-word representations of important nouns in their respective Wikipedia pages. Average monthly traffic of web pages are converted into categories, and the prediction task is to assign a given page to its corresponding category.
WebKB: The Cornell and Wisconsin datasets networks of web pages from various computer science departments, in which nodes represent web pages and edges are hyperlinks between them. Node features are bag-of-word representations of their respective web pages, and the prodiction task is to assign a given web page to the category that describes its content.
Table 6: Node Classification Dalaset Statistics
Dataset
# classes
# nodes
# edges
# splits
Mean degree
Homophily
Cora
2,708
10,556
3.9
CiteSeer
3,327
9,104
2.7
0.74
PuhMed
19,717
88,648
4.5
ogbu-arxiv
40
169,343
1,166,243
6.9
0.66
Chamelcon
2,277
36,051
15.8
0.23
Squirrel
5,201
216,933
41.7
0.22
Cornell
183
295
10
1.6
0.12
Wiscousin
251
499
2.0
0.17
A.5 Changes in test accuracy and loss by graph attention mechanism and regularization strength
Table 7: Average percent change in test accuracy
Cora
CiteSeer
PubMed
À Value
GAT
GATv2
Transformer
GAT
GATv2
Trausformer
CAT
GATv2
Transformer
A e (1.0,5.0
+3.8%
+ 7.4%%
+7.0%
+ 7.4%
+7.1% +3.0%
+ +1.55%
+2.7% +0.4%
+0.3% +0.3%
0.0%
A e (0.1.0.5)
+2.3%
+6.0%
+7.6%
+4.2%
+1.5%
0.0%
ogbn-arxiv
Chameleon
Squirrel
A Value
GAT
GATv2
Trausfortr
GAT
GATv2
Transformer
GAT
GATv2
Tranformer
5.00
-0.9% 0%
-0.6%
-0.3% -0.7%
-0.2%
-1.1%
-0.2%
0.4% -0.1%
-0.5%
A 0.51.
-0.3%
+ +0.3
+0.2%
0.0%
+0.5%
-0.4%
Cornell
Wisconsin
Value
GAT
GATv2
Transformer
GAT
GATv2
Transfrmer
(1.0,5.0)
+1.8%
+3.1% +1.6%
+0.6%
+0.6%
++0.4% 0.0%
+1.2%
(0.1,0.5)
+1.4%
+0.7%
+0.4%
+0.9%
17

---

Published in Transactions on Machine Learning Research (05/2023)
Table 8: Average percent change test loss
Cora
CiteSeer
PubMed
A Value
GAT
GATv2
Transormer
GAT
GATv2
Transformer
GAT
GATv2
Transformer
A (100,5.0)
-22.6% -15.3%
-9.00% -19.8%
-43.6% -28.8%
11.77%
+ 7.6%
-5.0%
-1.8%
-13.9% -2.9%
-7.0%
A 0..1,0.5)
-12.9%
+ 11.2%
-20.9%
-2.8%
-1.1%
ogbn-arxiv
Chamelcon
Squirrel
Value
GAT
GATv2
Transformrr
GAT
GATv2
Transformer
GAT
GATv2
Transformer
e 51.0,500
+0.6% -0.1%
+1.9% -0.2%
-0.2%
-2.4%
-2.7% -0.7%
-3.8% -0.8%
-1.1%
-1.5% 0.0%
-2.0%
e 0.5)
-0.8%
-0.7%
-0.2%
0.0%
Cornell
Wisconsin
A Value
GAT
GATv2
Transformerr
GAT
GATv2
Trausformer
A E (.00,500
-1.8%
-3.1%
-0.6%
-0.6% -0.4%
-0.4% 0.0%
-1.2%
e (0.1.
-1.4%
-1.6%
-0.7%
-0.9%
A.6 Number of rounds of interventions and percent change in test accuracy
Figure 4: Percent change in test accuracy for models trained with CAR. Each boxplot represents all combinations of the three graph attention layers (GAT, GATv2, Transformer) and the following sets of hyperparameter choices: A (1,5), (1, K (3), F' (100, 200).
Cora
CiteSeer
PubMed
0.4
0.50
0.050
% Change
% Change
Change
0.2
0.25
0.025
0.0
0.00
0.000
-0.025
3
5
10
# of Interventions (R)
# of Interventions (R)
# of Interventions
(R)
A.7 Comparison of CAR-trained models with CAL models
CAL is an approach for identifying causally attended subgraphs for graph prediction tasks that leverages causal interventions on graph representations to achieve robustness of model predictions (Zhao et al., 2021 While CAL and CAR have related goals enhancing graph attention using concepts from causal theory, CAL uses abstract perturbations on graph representation lo perform causal interventions while we propose an edge intervention strategy that enables causal effects to be computed scalably. In addition, CAL is designed to identify causally attended subgraphs for graph property prediction tasks, while work primarily focuses node classification tasks. Furthermore, CAL uses interventions to achieve robustness and does not directly leverage the effects of interventions on model predictions during training.
Despite these differences, we sought to determine whether the causal principles underlying CAL could be effectively applied to the various node classification tasks evaluated in our paper. We modified the CAL architecture make suitable for node prediction tasks by simply removing the final pooling layer that aggregates node representations within each graph directly upstream of a classifier, thus enabling node- level prediction. We evaluated the CausalGAT model from CAL using all combinations of the following hyperparameter choices: F² {128, 256), K {1, 4}, L = {1, 2}, X {0.2, 0.4, 0.6, 0.8, 1}, and X₂ {0.2, 0.4, 0.6, 0.8, 1), where F' refers to the number of hidden dimensions, K is the number of attention heads, L is the number of GNN layers, and A₁ and X2 are CAL-specific hyperparameters. For each dataset, we report the maximum test accuracy observed for the CAL CausalGAT across all combinations of these hyperparameter choices. We compare these test accuracies from the CAL CausalGAT models with the test accuracies from CAR-trained models averaged over all hyperparameter choices, which also appear above in A Appendix ??
18

---

Published in Transactions on Machine Learning Research (05/2023)
Table
Test accuracy on node classification datasets compared to CAL
Cora
CiteSeer
PubMed
ogbn-arxiv
Chameleon
Squirrel
Cornell
Wisconsin
GAT CAR
0.60 0.12 0.49 0.06
0.71 0.01
0.56
0.01
0.50
0.01
0.37 0.01
0.67 0.06
0.78 0.03
GATv2 CAR
0.56
Transformer
0.65 0.47
0.50
0.37
0.02
CAL CauslGAT
0.49 (best) 0.37 [best)
0.55 (best)
0.21 (best)
0.29 (best)
0.31 (best)
0.51 (best)
0.47 (best)
A.8 Regularization strength and generalization performance
Figure 5: Test loss reduction for CAR-trained models across regularization strengths. p-values computed from one-tailed I-tests evaluating the significance
test loss reductions for the E (1,5) CAR-trained models being greater than those of the À {0.1, 0.5} CAR-trained models.
Cora
CiteSeer
PubMed
ogbn-arxiv
p 0.05
p >> 0.05
p < 10
p > 0.05
Reduction in Loss
Reduction Loss
Reduction Loss
Reduction in Loss
0.0
2
0.5
0.5
0.0
-0.5
-1.0
0.51
0.5) (1,5)
(0.1, (1.5)
0.51
(1.5)
(0.1, (1.5)
Chameleon
Squirrel
Cornell
Wisconsin
p 10
<
10
< 10-3
p 0.05
Reduction in Loss
0.2
Reduction in Loss
Reduction Loss
0.75
Reduction in Loss
0.75
0.50
0.50
0.1
0.25
0.25
0.0
0.00
0.00
0.00
-0.25
(0.1, (1,5)
(0.1.0.5)
(1,5)
(0.1,0.5)
(1.5)
0.51
À
À
A.9 Experimental Setup and Test Accuracy for Synthetic Cora Datasets
To assess the relationship between the effectiveness of CAR and the homophily of dataset, we obtained set of synthetic Cora datasets from (Zhu et al. 2020 These synthetic datasets modified versions of the original Cora dataset that feature varying levels of edge homophily h, which is defined the fraction of edges in graph which connect nodes that have the same class label. Here, & is the set of edges, Yu is the class label for node u, and is the class label for node e.
l((, v): (u, U) EE = yoHl
(4)
[3
We evaluated 33 synthetic Cora datasets that spanned 11 different settings for h, each of which were repre- sented by 3 replicate datasets. For each of these datsets, we performed a similar analysis as above, in which we aimed to evaluate the consistency of improvements in test. loss using CAR across a number of graph attention and hyperparameter choices. We evaluated the GAT, GATv2, and Transformer graph attention layers along with all combinations of the following sets of hyperparameter choices: F' {100), X (1,5), K 3,, 5). L (1,2). We then performed one-tailed paired Wilcoxon rank-sum test to quantify the consistency of CAR-trained models' improvement in test loss over bascline models trained without CAR.
19

---

Published in Transactions on Machine Learning Research (05/2023)
Figure 6: Generalization performance of CAR-trained models compared to baseline across various levels of edge homophily (- (p), one-tailed paired Wilcoxon rank-sum test).
10.0
log10 (p)
7.5
5.0
2.5
0.0
0.0
0.10
0.20
0.30
0.40 0.50 0.60 0.70 0.80 0.90 0.99
Edge homophily ratio h
A.10 Homophily-based Regularization Details
For the neighbor voting model, each node's prediction is softmax-normalizee histogram of class labels from the node's neighbors: g(m) (pi,..., .P),, where ps is the normalized count of neighbors with label k. We employed the above edge intervention procedure and causal effect calculations to compute causal effect scores for interventions to this neighbor voting model afg) These values used in place of the network-speeific (n) values from the original implementation of CAR. Otherwise, the training procodure for network trained S/
with this neighbor voling scheme is exactly the same as for training with CAR. We nole that, for given node, calculating the intervention-affeeted prediction (()) simply entails updating the normalized counts of the class labels from the node's remaining neighbors after the intervention.
A.11 CAR-Guided Graph Rewiring
While graph attention networks have demonstrated notable performance gains, its inclusion of graph at- tention layers currently limits its use in large-soale applications compared GCNs, for which a number of advances in scalability have been made (Rossi et al., 2020).
To leverage the advantages of CAR in graph attention alongside the scalability of GCNs, we explored graph rewiring approach based on CAR-guided edge pruning. For a given dataset, we first use a trained graph attention network to assign an attention weight for each edge in the training and validation sets, after which edges with attention weights below threshold or are pruned. A GCN is then trained on the rewired training set with early stopping imposed with respect to the validation loss on the rewired valiclation set. The trained GCN then evaluated on a similarly rewired test set. We use a similar network architecture for the GCN as the various graph attention networks described in Appendix A.3 with the graph attention layers replaced with graph convolutional layers. We set the number of hidden dimensions in the GCN models to be F" = 100.
For the Chameleon dataset, we identified the hyperparameter settings that contributed to the highest val- idation accuracy for each of the one-layer GAT, GATv2, and Transformer CAR-trained models. We then trained GCN models on graphs that are pruned based on each of these models' attention mechanisms. We also pruned graphs using the counterparts of these models that were trained without CAR and trained an- other set. of GCN models these pruned graphs. We evaluated the test accuracy of the GCN models when performing this procedure across various attention thresholds (Figure 7 We observed that training and evaluating GCN models on pruned graphs contributed to enhanced test accuracy compared to the baseline GCN models that were trained and evaluated on the original graph. Furthermore, we compared GCN models trained and evaluated on CAR-guided pruned graphs against similar GCN models trained and evaluated on graphs pruned without CAR by computing the area under the curve (AUC) associated with the test accura- cies at various allention thresholds. Each AUC was calculated as the area below its models' test accuracies line and above the baseline GCN models' test accuracy. CAR-guided graph pruning was associated with higher AUC values across the three graph attention mechanisms, demonstrating the potential for CAR's utility in graph pruning tasks.
20

---

Published in Transactions on Machine Learning Research (05/2023)
Figure 7: Test accuracy of models trained and evaluated on graphs rewired using graph attention
GAT causal
(AUC 0.031)
0.50
GAT
(AUC 0.019)
GATv2 + causal
(AUC 0.034)
ACC
0.49
GATv2
(AUC 0.030)
0.48
Transformer causal
(AUC = 0.029)
Transformer
(AUC = 0.023)
0.47
GCN without rewiring
0.00 0.25 0.50 0. 0.75 1.00 Attention Threshold
A.12
Attention Interpretability
Table 10: Dewn-weighted citations of "anchor" papers. Each of the cited works that is down-weighted is well known MI. paper.
Paper Title
Cited Title
Reservoir Computing Hardware with Cellulor Au- tomata
Adam: A Method for Stochastic Optimization
-99.87%
Quantization Networrs
Deep Residual Learning for Image Recogrition
-99.86%
Compressive Hyperspherical Eneryy Minimiiza lion
ImageNel Large Scale Visual Recognition Chal- lenge
-99.75%
Table 11: Upweighted highly relevant citations.
Paper Title
Cited Title
A
Generalized Random Gilbert Vur-
Expurgated Random Coding Ensemn- bles Exponents Refinements and Con- nections
7700 %
shamov Codes
Sign Language Recognition Generu- tion and Translation: an Interdisci- plinary Perspective
Swift a Sign Wriling Improved Fast 2470 Transcriber
Random Beamsorming over Quasi- Static and Fading Channels: A Deter- ministic Equivalent Approach
Optimal Selective Feedback Policies 536 % for Opportunistic Beamforming
21


Reviews:
[{"review_id": "cIdrfHxNXh", "reviewer": "Reviewer_n2Ur", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strength of the paper are the experiments. First, I would strongly recommend to the authors to replace test-loss Table 1 with the actual accuracy Table 7. The former is clearly misleading just so there are more bold numbers, this is not good.\n\nI really liked Section 4.4 \"Interpreting Attention Coefficients\"\n\nQuestions to the authors.\n\nSince you are doing supervised learning, why don't you add an edge-loss using the ground-truth labels of the nodes. For example, if two nodes have the same label, then the corresponding attention edge-weight should be high. If two nodes have different label, then the edge-weight should be low. This might boost performance further.\nYour reported performance on Cora in Table 7 seems to be much lower than the current state-of-the-art for this dataset. For example, the current sota for Cora seems to be around 90%. Even vanilla GCN can achieve around 81%. See here: https://paperswithcode.com/sota/node-classification-on-cora, Could you please explain why? Maybe you didn't use the right number of layers, number of parameters etc? If that's the case, I would strongly recommend to try to match the sota numbers for each vanilla architecture that you are using. I would double-check sota for all datasets that you used, not just Cora.\nYou cited Knyazev et al. for the failures of attention mechanisms, which is a good empirical paper. However, the issues with vanilla GAT and GATv2 have been formalized here: Graph Attention Retrospective, arXiv:2202.13060.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In vanilla graph attention models the parameters of the attention mechanism are trained indirectly through the loss on the ground-truth labels of the nodes. There is no edge-loss which is used to train the parameters of the attention mechanism directly. The authors study a new regularized loss to solve this problem. I find the proposed approach interesting, I wouldn't say that I am excited about it, but it's interesting and it's good to see that someone has tried this. For this reason, I would recommend acceptance after some revision.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above.", "broader_impact_concerns": "No impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Hq9Vh7BG19", "reviewer": "Reviewer_KFFR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros: \n- Relatively good coverage on related works. \n- Overall well written and easy to understand. \n- Practicality and reproducibility seems reasonable from the provided evidence. \n\nCons: \n- Minor typos: in algorithm 1, \"equation 5\" should be \"equation 3\", and \"equation 3\" should be 2. Double \"-based\" in second paragraph in page 9. \n- The framework is pretty generic approach for modeling causality, but parts are chosen somewhat arbitrary and lack of detail explanation/intuition. e.g. the number of interventions R, the power of \\rho to the node degree in regularization term, and the loss function l^c. \n- Interpretability of CAR trained weights are not very well elaborated with only hand picked examples from one dataset. There is no general guidance on how to generalize to other dataset and what could be expected. \n- The main goal for node classification tasks are to improve classification accuracy, yet all the tables/plots in the main paper are for testing loss. The improvement in accuracy is rather small considering the method add almost 2x overhead in training time and raise my concern on the real world effectiveness of this method. \n- The statistics in tables and plots are average across different hyperparameter settings, which is not the standard approach. It is unfair to take average of same set of hyperparameter choices for two different model and the results should be reported as repeated experiments of same set of optimized hyperparameter per model. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors proposed a causal regularization method named CAR that can be added to existing attention-based graph neural networks. CAR introduces perturbation into each nodes neighborhood connection, which will then be reflected on the regularization term of the loss and shifts the attention coefficients based on the causal effect of removed edge. The authors performed multiple experiments to validate the benefits of CAR regularization on 8 real-world datasets and compared with 3 existing graph attention network baselines. Results show consistent improvement in terms of testing loss. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Besides the listed weakness mentioned above, there are multiple essential changes needed for this paper, including but not limited to: \n- Why only focus on attention networks? The regularization terms doesn't look like it's constrained to attention module only. More comparison with different neighborhood aggregation scheme would better justify or even expand the impact of paper. \n- In the experiment with heterophily graph, why not compare CAR regularized method with GNNs designed for heterophily graphs, such as GPRGNN, GeomGCN, GCNII etc. Does CAR + GAT outperformed them, and if not, does CAR + heterophily GNN outperform the heterophily GNN baselines?\n- The assumption of \"ground-truth\" relevance completely ignore the impact of node features and solely rely on node labels. It should be comparable directly to attention coefficients which are node feature based. More explanation on the choice of comparing the divergence of these two are needed. ", "broader_impact_concerns": "I didn't see concerns on ethical implication of this work. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "F3ZcZxOifc", "reviewer": "Reviewer_9tw7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Utilizing causal inference to promote the effectiveness of graph neural networks is welcomed since they can provide better interpretability.\n\n2. The paper is well-written and easy to follow.\n\n\nWeaknesses:\n\nI have two major concerns.\n\n1. In the Introduction, the authors introduce several related studies. However, why they are not good enough is not clearly illustrated.\n\n2. In my opinion, the proposed model in the paper is relatively straightforward and incremental. Only some common interventions such as removing edges are proposed in the model, making the contribution rather limited.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the problem of causally-guided regularization of graph attention networks, in order to improve their generalizability. To achieve this, the authors introduce CAR, a general-purpose regularization framework for graph attention networks, which can align the attention mechanism with the causal effects of interventions based on the edges in the graphs. Experiments on several benchmark graphs demonstrate the effectiveness of the proposed model.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weaknesses.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a causality-based regularizer called CAR to improve the generalization of graph neural nets on several node classification tasks. While some reviewers felt the idea is simple and not entirely novel, they also appreciated the authors' efforts in exploring the direction of causality-inspired graph learning techniques. The authors' rebuttal and revision addressed most of the reviewers' concerns. Given that novelty is not the primary criterion for TMLR, I suggest accepting this submission.


Decision: Accept (poster)