How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision
Authors: Dongkwan Kim
Alice Oh
Venue: ICLR 2021 Poster (2021)
Keywords: Graph Neural Network
Attention Mechanism
Self-supervised Learning

Abstract: Attention mechanism in graph neural networks is designed to assign larger weights to important neighbor nodes for better representation. However, what graph attention learns is not understood well, particularly when graphs are noisy. In this paper, we propose a self-supervised graph attention network (SuperGAT), an improved graph attention model for noisy graphs. Specifically, we exploit two attention forms compatible with a self-supervised task to predict edges, whose presence and absence contain the inherent information about the importance of the relationships between nodes. By encoding edges, SuperGAT learns more expressive attention in distinguishing mislinked neighbors. We find two graph characteristics influence the effectiveness of attention forms and self-supervision: homophily and average degree. Thus, our recipe provides guidance on which attention design to use when those two graph characteristics are known. Our experiment on 17 real-world datasets demonstrates that our recipe generalizes across 15 datasets of them, and our models designed by recipe show improved performance over baselines.

Published as a conference paper at ICLR 2021
HOW TO FIND YOUR FRIENDLY NEIGHBORHOOD: GRAPH ATTENTION DESIGN WITH SELF-SUPERVISION
Dongkwan Kim & Alice Oh KAIST, Republic of Korea dongkwan. wimmkaisst.cmm ac. alice. www.eehhkkait.cccm
ABSTRACT
Attention mechanism in graph neural networks is designed to assign larger weights to important neighbor nodes for better representation. However, what graph attention learns is not understood well, particularly when graphs are noisy. In this paper, we propose a self self-supe graph attention network (SuperGAT), improved graph attention model for noisy graphs. Specifically, attention forms compatible with a self-supervised task to predict edges, whose presence and absence contain the inherent information about the importance of the relationships between nodes. By encoding edges, SuperGAT learns more expressive attention in distinguishing mislinked neighbors. We find two graph characteristics influence the effectiveness of attention forms and self-supervision: homophily and average degree. Thus, our recipe provides guidance on which at- tention design to use when those two graph characteristies are known. Our experi- ment on 17 real-world datasets demonstrates that our recipe generalizes across 15 datasets of them, and our models designed by recipe show improved performance over baselines.
exploit two
INTRODUCTION
Graphs are widely used in various domains, such as social networks, biology, and chemistry. Since their patterns are complex and irregular, learning to represent graphs is challenging (Bruna et al. 2014 Henaff al.] 2015 Defferrard et al.] 2016) Duvenaud et al.] 2015 Atwood & Towsley
2016 Recently, graph neural networks (GNNs) have shown a significant performance improve- ment by generating features of the center node by aggregating those of its neighbors (Zhou et al.) 2018 Wu et al., 2020). However, real-world graphs are often noisy with connections between un- related nodes, and this causes GNNs to learn suboptimal representations. Graph attention networks (GATs) (Veličković et al.] (018)) adopt self-attention to alleviate this issue. Similar to attention in sequential data (Luong et al. 2015 Bahdanau et al. 2015, Vaswani et al.) (007). graph attention captures the relational importance of graph, in other words, the degree of importance of each of the neighbors to represent the center node. GATs have shown performance improvements in node classification, but they are inconsistent in the degree of improvement across datasets, and there is little understanding of what graph attention actually Iearns.
Hence, there still room for graph attention to improve, and we start by assessing and learning the relational importance for each graph via self-supervised attention. We leverage edges that explicitly encode information about the importance of relations provided by graph. If node i and linked, they are more relevant to each other than others, and if node and j are not linked, they are not important to each other. Although conventional attention is trained without direct supervision, if we have prior knowledge about what to attend, we can supervise attention using them (Knyazev) al. 2019 Yu al. 2017). Specifically, we exploit a self-supervised task, using the attention value input to predict the likelihood that an edge exists between nodes.
To encode edges in graph attention, we first analyze what graph attention learns and how it relates to the presence of edges. In this analysis, we focus two commonly used attention mechanisms, GAT's original single-layer neural network (GO) and dot-product (DP), as building blocks of our proposed model, self-supervised graph attention network (SuperGAT). We observe that DP attention shows better performance than GO attention in the task to predict link with attention value. On the other hand, GO attention outperforms DP attention in capturing label-agreement between target

---

Published as a conference paper at ICLR 2021
node and its neighbors. Based analysis, propose two variants of SuperGAT, scaled dot- product (SD) and mixed GO and DP (MX), to emphasize the strength of GO and DP.
Then, which graph attention models the relational importance best and produces the best node repre- sentations? We find that it depends on the average degree and homophily of the graph. We generate synthetic graph datasets with various degrees and homophily, and analyze how the choice of atten- tion affects node classification performance. Based on this result, we propose a recipe to design graph attention with edge self-supervision that works most effectively for given graph characteris- tics. We conduct experiments on a total of 17 real-world datasets and demonstrate that our recipe can be generalized across them. In addition, we show that models developed by our method improve performance over baselines.
We present the following contributions. First, we present models with self-supervised attention using edge information. Second, we analyze the classic attention forms GO and DP using label-agreement and link prediction tasks, and this analysis reveals that GO is better label agreement and DP at link prediction. Third, we propose recipes to design graph attention concerning homophily and average degree and confirm its validity through experiments on real-world datasets. We make our code available for future research (https: git.ub..tuo..ooooggooo cww donn/ong kim/ SuperGAT).
2 RELATED WORK
Deep neural networks are actively studied in modeling graphs, for example the graph convolutional networks (Kipf & Welling 2017) which approximate spectral graph convolution (Bruna al., 2014
Defferrard et al.] [2016]. A representative work in a non-spectral way is the graph attention net-
works (GAT's) Veličković et al. 2018) which model relations in graphs using self-attention mech-
anism (Vaswani et al. 2017). Similar to attention in sequence data (Bahdanau et al. 2015 Luong jet al., 2015; Vaswani et al. (017). variants of attention in graph neural networks (Tekkumpaa-
mppll al. 2018: Zhang et al. 2018 Wang et al. 2019a Gao & Ji 2019 Zhang et al. 2020: Hou cct al. 2020) are trained without direct supervision. Our work is motivated by studies that improve attention's is expressive power by giving direct supervision (Knyazev et al., 2019; Yu et al., 2017).
Specifically, we employ a self-supervised task to predict edge presence from attention value. This is in line with two branches of recent GNN research: self-supervision and graph structure learning.
Recent studies about self-supervised learning for GNNs propose tasks leveraging the inherent in- formation in the graph structure: clustering, partitioning, context prediction after node masking, and completion after attribute masking (Hu et al.) 2020bb Hui et al., [2020] [un et al., [2200] You al., 2020). To the best of our knowledge, ours the first study analyze self-supervised learn- ing of graph attention with edge information. Our self-supervised task is similar to link predic- tion (Liben-Nowell & Kleinberg 2007), which is a well-studied problem and recently tackled by neural networks (Zhang & Chen, 2017 2018). Our DP attention to predict links is motivated by graph autoencoder (GAE) (Kipf & Welling (0016) and its extensions (Pan et al.) 2018 Park et al.) reconstructing edges by applying a dot-product decoder to node representations.
2019)
Graph structure learning an approach to learn the underlying graph structure while jointly learning downstream tasks (Jiang et al., 2019: Franceschi al. 2019 Klicpera et al. [019 SStretuu et al. 2019 Zheng et al., 2020). Since real-world graphs often have noisy edges, encoding structure information contributes to learn better representation. However, recent models with graph structure learning suffer from high memory and computational complexity. Some studies target all spaces where edges can exist, they require O(IVP) space and computational complexity (Jiang et al. 2019 Franceschi et al.) 2019). Others using iterative training (or co-training) between the GNNs and the structure learning model time-intensive in training (Franceschi et al. 2019 (Stretcu al., 2019). We moderate this problem using graph attention, which consists of parallelizable operations, and our model built on without additional parameters. Our model learns attention values that are predictive of edges, and this can be seen as a new paradigm of learning the graph structure.
MODEL
In this section, we review the original GAT (Veličković et al. 2018) and then describe our self- supervised GAT (SuperGAT) models.
2

---

Published as a conference paper at ICLR 2021
Original
C GO
4 G.
Wh
Wh
a
softmax,
Cjj
M.MX
Dot-product
DD
PiMMX.DP
Dij
P(Ú.)) E)
UVF
BDSD
Figure I: Overview of attention mechanism of SuperGATss GO. DP, MX, and SD. Blue circles (eiz) represent the unnormalized attention before softmax and red diamonds (Ø:;) indicate the probability of edge between node and j. The attention mechanism of the original GAT (Veličković et al. 2018) the dashed rectangle.
Notation For a graph g = EV, N the number of nodes and F' is the number of features at layer Graph attention layer takes set of features IH! = (., 1xxx }. h E R input and pro- duces output features IH[+1 [h|+1 ..... ......,,,,, 1) To compute h/ the model multiplies the weight matrix W E E "xf""""F" xF" to HH' linearly combines the features of its first-order neighbors (includ- ing itself) E N; U (i) by attention coefficients at and finally applies a non-linear activation p- That is h/ 1+1 P EjEN, a / w(+)h(). We can compute a
[+1 softmax, (attmmaxxxLLLLLLI) (LRcLU(e))
by normalizing c Gel Whi w++hh)) with softmax on N, {i) under leaky ReLU activa- tion (Maas al. 2013). where a. function of the form gr'+
1 1 R.
Graph Attention Forms Among two widely used attention mechanisms, the original GAT (GO) computes the coefficients by single-layer feed-forward network parameterized by a'+1 E R2P'+1 The other is the dot- dot-product (DP) attention, (Luong et al., [013)] Vaswani et al. 2017) motivated by prior work node representation learning, and adopts the same mathematical expression for link prediction score (Tang et al.) 2015 Kipf & Welling 2016).
J+I GO
[W++ (TT
h! w and
I+1 .j.DP (W)+1N))
w
(1)
From now on, we call GAT that uses GO and DP as GAT and GAT respectively.
Self-supervised Graph Attention Network We propose SuperGAT with the idea of guiding tention with the presence or absence of an edge between node pair. We exploit the link prediction
task to self-supervise attention with labels from edges: for a pair and j. if edge exists and otherwise. We introduce as with sigmoid to infer the probability dig of an edge between and j-
R x R + R and
dij = P (G E E) = (as Wh;, Wh;))
(2)
We employ four types (GO, DP, SD, and MX) of SuperGAT based on GO and DP attention. For ags. the form of which the same as a, GATgo and GATDE, we name them SuperGAT and SuperGATTp respectively. For more advanced versions, we describe SuperGATssD (Scaled Dot- product) and SuperGAT (Mixed GO and DP) by unnormalized attention eig and probability dy that edge exist between i and
SuperGATsD: Csj,SD eij.De/VF, @IJ.SD = =(egs.DD)
(3)
SuperGATwx: eij.MX eij,00 (eig,DP), dij.M ((E),DPP)
SuperGATsD divides the dot-product of nodes by a square root dimension as Trans- former /Vaswani et al. (2017). This prevents some large values to dominate the entire attention after softmax. SuperGAT Mx multiplies GO and DP attention with sigmoid. The motivation this form comes from the gating mechanism of Gated Recurrent Units (Cho et al., 2014). Since DP attention with the sigmoid represents the probability an edge, it can softly drop neighbors that are not likely linked while implicitly assigning importance to the remaining nodes.

---

Published as a conference paper at ICLR 2021
Training samples a set of edges E and the complementary set E (V V) E. However, if the number nodes is large, it is not efficient to use all possible negative cases in E So. we use negative sampling as in training word or graph embeddings (Mikolov et al. 2013 Tang al. 2015 Grover & Leskovec 2016), arbitrarily choosing a total of Pa | negative samples Æ- from h where the negative sampling ratio ps, E R+ is hyperparameter. SuperGAT is capable of modeling
graphs that are sparse with a sufficiently large number of negative samples (i.e., V x V > [B),
but this is generally not a problem because most real-world graphs are sparse (Chung 2010).
We define the optimization objective of layer as a binary cross-entropy loss L'p.
C
BUE- EGGREEEE 10.0=11 log oi + + (1.0=0 log (1 - (1-0%)
(5)
where 1. is indicator function. We use subset of E' U E sampled by probability Pe (0, 1]
(also a hyperparameter) at each training iteration for a regularization effect from randomness.
Finally, we combine cross-entropy loss node labels (Lv), self self-supervised graph attention losses for all L layers (C'g), and L2 regularization loss, with mixing coefficients Ag and 22.
Ly + Ag Ele CE + A2 |
(6)
We use the same form of multi-head attention in GAT and take the mean of each head's atten- tion value before the sigmoid to compute dij. Note that SuperGAT has equivalent time and space complexity as GAT. To compute C' for one head, we need additional operations in terms of O(F EUUE E-EE) and we do not need extra parameters.
EXPERIMENTS
Our primary research objective is to design graph attentions that are effective with edge self- supervision. To do this, pose four specific research questions. We first analyze what basic graph attentions (GO and DP) learn (RQ1 and 2) and how that can be improved with edge self-supervision (RQ3 and 4). We describe each research question and the corresponding experiment design below.
RQ1. Does graph attention learn aabeemagremmmn?? First, we evaluate what the graph attentions of GATgo and GAT learn without edge supervision. For this, present ground-trute truth of relational importance and a metric to assess graph attention with ground-truth. Wang et al. (2019a) showed that node representations in the connected component converge the same value in deep GAT's If there is an edge between nodes with different labels, then it will be hard to distinguish the two corresponding labels with GAT of sufficiently many layers; that is, ideal attention should give all weights to label-agreed neighbors. In that sense, we choose label-agreement between nodes as ground-truth of importance.
We compare -abee-etre and graph attention based on Kullback-Leibler divergence of the nor- malized attention On [akk, ag1, akJ] with label agreement distribution for the center node and its neighbors 1 to J.. The label agreement distribution, ls = [LN,, fki....,44 is defined by,
kkj = / EE.,
laj = 1 (if i and j have the same label) or 0 (otherwise).
We employ KI. divergence in Eq. g whose value becomes small when attention captures well the label-agreement between a node and its neighbors.
KLD(a Rx) = EjeN, auj log(auz/kkz)
RQ2. Is graph attention predictive of edge presence? To evaluate how well edge information is encoded in SuperGAT, we conduct link prediction experiments with SuperGATGo and SuperGATpp using dij of the last layer predictor. We measure the performance by AUC over multiple runs. Since link prediction performance depends on the mixing coefficient Ag in Eq. 6 we adopt multiple Ag {10-), 10
103). We train with an incomplete edges and the same number of negative samples. At the same time, node classification performance measured with the same settings to see how learning edge presence affects node classification.
of edges, and test with the missing
4

---

Published as a conference paper at ICLR 2021
RQ3. Which graph attention should we use for given graphs? The above two research ques- tions explore what different graph attention learns with or without supervision of edge presence. Then, which graph attention is effective among them for given graphs? We hypothesize that differ- ent graph attention will have different abilities to model graphs under various homophily and average degree. We choose these two properties among various graph statistics because they determine the quality and quantity of labels in our self-supervised task. From the perspective of supervised learn- ing of graph attention with edge labels, the learning result depends on how noisy labels are (i.e., how low the homophily is) and how many labels exist (i.e., how high the average degree is). So, we gen- erate 144 synthetic graphs (Section (4.1) controlling homophily (0.1 0.9) and 16 average degree (1 100) and perform the node classification task in the transductive setting with GCN, GATco. SuperGATsD. and SuperGATMx-
In RQ3, there are also practical reasons to use the average degree and homophily, out of many graph properties (e.g., diameter, degree sequence, degree distribution, average clustering coefficient). First, the graph property can be computed efficiently even for large graphs. Second, there should be an algorithm that can generate graphs by controlling the property interest only. Third, the property should be scalar value because if the synthetic graph space is too wide, it would be impossible to conduct an experiment with sufficient coverage. Average degree and homophily satisfy the above conditions and are suitable for our experiment, unlike some of the other graph properties.
RQ4. Does design choice based on RQ3 generalize to real-world datasets? Experiments on synthetic datasets provide an understanding of graph attention models' performance, but they are oversimplified versions of real-world graphs. Can design choice from synthetic datasets be gener- alized to real-world datasets, considering more complex structures and rich features in real-world graphs? To answer this question, we conduct experiments on 17 real-world datasets with the various average degree (1.8 35.8) and homophily (0.16 0.91), and compare them with synthetic graph experiments in RQ3.
4.1 DATASETS
Real-world datasets We use total of real-world datasets (Cora, CiteSeer, PubMed, Cora-MI,, Cora-Full, DBLP, ogbn-arxiv, CS, Physics, Photo, Computers, Wiki-CS, Four-Univ, Chameleon, Crocodile, Flickr, and PPI) in diverse domains (citation, co-authorship, co-purchase, web page, and biology) and scales (2k- 169k nodes). We try to use their original settings as much possible. To verify research questions and 2, we choose four classic benchmarks: Cora, CiteSeer, PubMed in the transductive setting, and PPI in the inductive setting. See appendix A. for detailed description, splits, statistics (including degree and homophily), and references.
Synthetic datasets We generate random partition graphs of n. nodes per class and classes (For- Iunato 2010), using Net work library (Hagberg al. 2008). A random partition graph is graph of communities controlled by two probabilities Pi. and Pout- If the nodes have the same class labels, they are connected with Pin. and otherwise, they are connected with Pout- To generate a graph with an average degree of dang features of nodes are sampled from overlapping multi-Gaussian distributions (Abu-EI-Haija et al. 2019). We set to 500, co 10, and choose dang between 1 and 100, Pin from (0.16, 0.20 0.98 ). We use 20 samples per class for training, 500 for validation and 1000 for test.
6, we choose Pin. and Pout by pin + (c- 1) Pout = 6. The input
4.2 EXPERIMENTAL SET-UP
We follow the experimental set-up of GAT with minor adjustments. All parameters are initialized by Glorot initialization (Glorot & Bengio 2010) and optimized by Adam (Kingma & Ba, 2014).. We apply L.2 regularization, dropout (Srivastava et al. 2014) to features and attention coefficients, and early stopping validation loss and accuracy. We use ELU (Clevert et al. 2016) linear activation Unless specified, we employ a two-layer SuperGAT with F = 8 features and K = 8 attention heads (total 64 features). All models are implemented in PyTorch (Paszke et al. 2019) and PyTorch Geometric (Fey & Lenssen 2019). See appendix A.5 for detailed model and hyperparameter configurations.
non-
5

---

Published as a conference paper at ICLR 2021
KLDXATT, AGPU
KLD-ATT AGN
Layor
Layer 2
Layer
Layer
Layer
Lager
Loyor 3
Layer 4
60
Untiom
Uniorm
Figure 2: Distribution of KL divergence between normalized attention and label-agreement all nodes and layers for Cora dataset (Left: two-layer GAT, Right: four-layer GAT).
Baselines For all datasets, we compare our model against representative graph neural models: graph convolutional network (GCN) (Kipf & Welling 2017). GraphSAGE (Hamilton et al. 2017), and graph attention network (GAT) (Veličković et al. 2018). Furthermore, for Cora, CiteSeer, and PubMed, choose recent graph neural architectures that learn aggregation coefficients (or dis- crete structures) over edges: constrained graph attention network (CGAT") (Wang et al. 2019a), graph Saaming-convolutionall network (GLCN) (Jiang et al., (009), learning discrete structure (LDS) (Franceschi et al.) 2019), graph agreement model (GAM) (Stretcu et al.] (019). and Neu- ralSparse (NS in short) (Zheng et 2020). For the PPI, we use CGAT an additional baseline.
5
RESULTS
This section describes the experimental results which answer the research questions in Section We include qualitative analysis of attention, quantitative comparisons on node classification and link prediction, and recipes of graph attention design. The results for sensitivity analysis of important hyper-parameteer are the appendix B.5
Does graph attention learn label-agreement? GO learns label-agreement better than DP.
We draw box plots of KL divergence between attention and label agreement distributions of two- layer and four-layer GAT with GO and DP attention for Cora dataset in Figure 2 We see similar patterns in other datasets and place their plots in the appendix B33] At the rightmost of each sub- figure, we draw the KLD distribution when uniform attention is given to all neighborhoods, Note that the maximum value of KLD each node different since the degree of nodes different. Also, the KLD distribution shows long-tail shape like a degree distribution of real-world graphs.
There are three observations regarding distributions of KLD. First, we observe that the KLD dis- tribution of GO attention shows pattern similar to the uniform attention for all citation datasets. This implies that trained GO attention similar to the uniform distribution, which is line with previously reported results in the case of entropy Waan et al. 2019b). Second, KLD values of DP attention tend to be larger than those of GO attention for the last layer, resulting in bigger long-tails. This mismatch between the learned distribution of DP attention and the label agreement distribution suggests that DP attention does not learn label-agreement in the neighborhood. Third, the deeper the model (more than two), the larger the KLD value of DP attention the last layer. This is because the variance of DP attention increases as the layer gets deeper, as explained below in Proposition[] []] independent and identically drawn from
Proposition 1. For I 1th GAT layer, W and zero-mean uniform distribution with variance o² and o² respectively, assuming that parameters are independent imput features h' and elements of h' are independent each other,
Varle 2r ( and Varle 2 pi+log ot ( (((h)) (T))) + Var((h)TT()) h2))) (9)
The proof is given in the appendix B.1 While the variance of GO depends the norm of features only, the variance of DP depends on the expectation of the square of input's dot-product and variance of input's dot-product. Stacking GAT layers, the more features of and j correlate with each other, the larger the input's dot-product will be. After DP attention normalized by softmax, which intensifies the larger values among them, normalized DP attention attends to only small portion of the neighbors and learns a biased representation.
Since CGAT uses node labels in the loss function, it difficult to use it semi-supervised learning. So, we modify its auxiliary loss for SSL. See appendix A.6 for details.
lsmodee s/ www_/ggggnnn..//// html ht //tccs.ddcccooooo dg1 http://en/aattttteeeeeeeeeee

---

Published as a conference paper at ICLR 2021
Dataset Cora
Dataset CiteSeer
Dataset PubMed
Dataset PPI
Test Pert.
GAT
GO Link
0.50
DP & Link
Node
Node
-2.5 0.0 2.5
-2.5
0.0 2.5
-2.5
0.0
2.5
-2.5
2.5
Mixing Coeff, (Log)
Mixing Coeff, (Log)
Mixing Coeff. (Log)
Mixing Coeff. (Log)
Figure 3: Test performance node classification and link prediction for GO and DP attentions against the mixing coefficient Åg. We report accuracy (Cora, CiteSeer, PubMed) and micro fl-score (PPI) for node classification, and AUC for link prediction.
Homophily. 0.2
Homophily 0.4
Hemophily 0.6
Homophily 0.8
Mindl GCV
-2.5
SupertAA1- SuperGATWiX
-5.0
Dilt
15
-7.5
0.0
0.3
1.0
1.5
2.0
0.0
0.5
1.4
1.5
2.0
0.0
0.3
1.0
1.5
2.0
2.0
0.5
3.0
2.5
2.0
Avg. Degree (Log1 D)
Aug. Degree (Log10)
Avg. Degree (Log10)
Avg. Degree (Log10)
Figure 4: Mean test accuracy gains (of 5 runs) against GATgo synthetic datasets, varying ho- mophily and average degree of the input graph.
Is graph attention predictive for edge presence? DP predicts edge presence better than GO.
In Figure 3 we report the mean AUC over multiple runs (5 for PPI and 10 for others) for link prediction (red lines) and node classification (gray lines). As the mixing coefficient λg increases, the link prediction score increases all datasets and attentions. This is a natural result considering that Äg the weight factor of self-supervised graph attention loss (LE in Equation 6). For three out of four datasets, DP attention outperforms GO for link prediction for all range of X in our experiment. Surprisingly, even for small λg. DP attention shows around 80 AUC, much higher than the performance of GO attention. PPI is an exception where GO attention shows higher performance for small Äg than DP, but the difference is slight. The results of this experiment demonstrate that DP attention is more suitable than GO attention in encoding edges.
This figure also includes node classification performance. For all datasets except PubMed, we ob- serve a trade-off between node classification and link prediction; that is, node classification perfor- mance decreases in SuperGAT and SuperGAT as Ag increases and thus link prediction perfor- mance increases. PubMed also shows decrease in performance at the largest Ag we have tested. This implies that is hard to learn the relational importance from edges by simply optimizing graph attention for link prediction.
Which graph attention should we use for given graphs? It depends on homophily and average degree of the graph.
In Figure 4 we draw the mean test accuracy gains (over runs) against GAT as the average
degree increases from to 100, for different values of homophily, 64 synthetic graphs with GCN, Sup.rGATsD. SuperGATwx- See full results from appendix B.2 We define homophily as the average ratio of neighbors with the same label as the center node (Pei et al. 2020). That
is h = - (EjeN, (()=o)/|N1)) where I(i) is the label of node i. The expectation of homophily for random partition graphs is analytically pon/ 5, and we just adopt this value to label the homophily of graphs in Figure
We make the following observations from this figure. First, if the homophily is low (< 0.2), SuperGATss performs best among models because DP attention tends to focus on a small number of neighbors. This result empirically confirms what we analytically found in Proposition[] Second, even when homophily is low, the performance gain of SuperGAT against GAT increases as the av- erage degree increases to a certain level (around 10), meaning relation modeling can benefit from self-supervision if there are sufficiently many edges providing supervision. This is in agreement with prior study of label noise for deep neural networks where they find that the absolute amount of data with correct labels affects the learning quality more than the ratio between data with noisy and correct labels (Rolnick et al. 2017). Third, if the average degree and homophily high enough,
7

---

Published as a conference paper at ICLR 2021
Table 1: Summary of classification accuracies of GCN, GraphSAGE, GAT, SuperGATso: SD. and SuperGATTxx Mx for real-world datasets (30 runs for ogbn-arxiv and Flickr, and 100 runs for others).
Model
ogba-arsiv araiv
CS
Physics
Cora ML
Cora-Foll DBLP
Cham
FourUnii Univ
Wiki-CS
Photo
Comp.
Flickr
Croco.
GCN
33.3
91 Sea5
02.5
85.0a0a
30.5gaa
778
33.1
7483ua
74.0g1.
Qlaa 84.61a
11.2404
32.6.04
GmphSAGE
54.8aa.
90.0aaa
22.2
50.7ana
592
78.7.t.
41,0caa
74.404
77.5ma
904e1a
88.0s14
50.7002
53.0e1.
GAT
64.1a9a
89.5ea.
21.2
80.240a
08.7a0.
7822ss
40.8,07
74:2407
77.6gs
91 Mann
85.7209
60.9.00
68.3
SupertiATsp
54.51aa
88.8
91.6jas
84.5104
05-8104
79.4%a 41.0%a
76.2
77.9100
86.8jas 82.2100
45.1
53.3
SuperGiATax 55.1iaa
90.2
91 9ins 84.726 59.6
80.71a 42.0 76.3
77.91ns
91 8e0a 85.7211 50.8
53.3
Table 2: Summary of classification accuracies with 100 ran- dom seeds for Cora, CiteSeer, and PubMed. We mark with daggers (+) the reprinted results from the respective papers.
Table 3: Summary of micro fl- scores with 30 random seeds for PPI.
Model
Cora
CiteSeer
PubMed
Model
PPI
GCN
81.5
70,3
79.0
GCN
61.5 + 0.4
GraphSAGE
82.1 + 0.6
71.9 + 0.9
78.0 + 0.7
GraphSAGE
59.0 + 1.2
CGAT
81.4 + 1.1
70.1 + 0.9
78.1 + 1.0
CGAT
68.3 1: 1.7
GLCN
85.5
72.0
78.3
GAT
72.2 - 0.6
LDSt
84.1
75
SuperGAT
74.420.4
GCN + GAM
86.2
73.5
86.0
SuperGATxxx 67.211.2
GCN + NS
83.7 + 1.4
74.1 + 1.4
GAT
83.0 +: 0.7
72.5 + 0.7
79.0 + 0.4
p-value < 0000
p-value < 0000
SuperGATsD
82.71..8
72.5=0.8
81.340.s
Worse than GAToo
SuperGAT>x
84.340.6
72.6
81.7-0.6
Color
Best graph attention (See Fig
there is no difference between all models, including GCNs. there are more correct edges beyond certain amount, we can learn fine representation without self-supervision. Most importantly, if the average degree is not too low or high and homophily is above 0.2, SuperGATMx performs better than or similar to SuperGATsD- This implies that can take advantage of both GO attention to learn -abeemagreementt and DP attention to learn edge presence by mixing GO and DP. Note that many of the roal-world graphs belong this range of graph characteristics.
The results on synthetic graphs imply that understanding of graph domains should be preceded to design graph attention. That is, by knowing the average degree and homophily of the graphs, can choose the optimal graph attention in our design space.
Does design choice based on RQ3 generalize to real-world datasets? It does for 15 of 17 real- world datasets.
In Figure 5 we plot the best-performed graph attention for synthetic graphs with square points in the plane of average degree and homophily. The size is the performance gain of SuperGAT against GAT, and the color indicates the best model. If the difference is not statistically significant (p-value 2 .05) between GAT and SuperGAT, and between SuperGATMx and SuperGAT we mark as GAT-Any and SuperGAT-Any, respectively. We call this plot a recipe since it introduces the optimal attention to specific region's 's graph.
Now we map the results of 17 real-world datasets in Figure 5 accordiin to their average degree and homophily. Average degree and homophily can be found in the appendix A. and experimental results of graph attention models are summarized in Tables D 2 and 3 We report the mean and standard deviation of performance over multiple seeds (30 for graphs with more than 50k nodes and 100 for others). We put unpaired t-test results of SuperGAT with GAT with asterisks.
We find that the graph attention recipe based on synthetic experiments can generalize across real- world graphs. PPI and Four-Univ (A) are surrounded by squares of SuperGAT sD (m) the bottom of the plane. Wiki-CS (4), located in the SuperGAT-Any region (.), also show no difference in performance between SuperGATs. Nine datasets (A), which SuperGAT shows the highest per- formance, are located in the MX regions or within the margin of two squares. Note that there are two MX regions: lower-middle average degree (2.5 - 7.5) and high homophily (0.8 - 0.9), and upper- middle average degree (7.5 50) and lower-middle homophily (0.3 0.5). There five datasets with no significant performance change across graph attention ()). CiteSeer, Photo, and Computers within margin of one square from the GAT-Any region
); however, Flickr and Crocodile are
8

---

Published as a conference paper at ICLR 2021
Physics
0.9
Cola
0.8
PubMedd
Cora. ML/BLPP
Comp.
CiteSeer
Models
0.7
ogbn-arxiv
Wiki-Cs
Syn-SuperGAT-MX
Homophity
0.6
Syn-SuperGAT-SD
Syn-SuperGAT-Any
0.5
Syn-GAT-Any
0.4
Real-SuperrAT-MXX
Flickr
Real-SuperGAT-SD
0.3
Real-SupperGA--Ayy
0.2
Real-GAT-Any
Four-Univ
Cham.
0.0
0.5
1.0
1.5
2.0
Avg. Degree (Log10)
Figure 5: The best-performed graph attention design for synthetic and real-world graphs with various average degree and homophily.
in the SuperGAT- region. To find out the cause of this irregularity, we examine the distribution of degree and per-node homophily (appendix[AA AA.4) We observe more complex mixture distribution of homophily and average degree in Flickr and Crocodile, and this seems to be equivalent to mixing graphs of different characteristics, resulting in inconsistent results with attention design recipe.
Comparison with baselines For a total of 17 datasets, SuperGAT outperforms GCN for 13 datasets, GAT for 12 datasets, GraphSAGE for 16 datasets. Interestingly, for CS, Physics, Cora- ML, and Flickr, in which our model performs worse than GCN, GAT also cannot surpass GCN. not yet known when the degree-normalized aggregation of GCN outperforms the attention- based aggregation, and more research is needed to figure out how to embed the degree information into graph attention. Tables 2 and show performance comparisons between SuperGAT and re- cent GNNs for Cora, CiteSeer, PubMed, and PPI. Our model performs better for CiteSeer (0.6%p) and PubMed (3.4%p) than GLCN, which gives regularization to relations in a graph. GCN NS (NeuralSparse) performs better than our model for CiteSeer (1.5%p) but not for Cora (0.6%p). CGAT modified for semi-supervised learning shows lower performance than GAT. Although LDS and GAM which use iterative training show better performance except for LDS on Cora. these mod- els require significantly more computation for the iterative training. For example, GCN + GAM compared model needs x34 more training time for Cora, 772 for CiteSeer, and x82 for PubMed. See appendix A.7 and B.4 for the experimental set-up and the result of wall-clock time analysis.
CONCLUSION
We proposed novel graph neural architecture designs to self-supervise graph attention following the input graph's 's characteristics. We first assessed what graph attention is learning and analyzed the effect of edge self-supervision link prediction and node classification performance. This analysis showed two widely used attention mechanisms (original GAT and dot-product) have difficulty en- coding label-agreement and edge presence simultaneously. To address this problem, we suggested several graph attention forms that balance these two factors and argued that graph attention should be designed depending on the input graph's average degree and homophily. Our experiments demon- strated that our graph attention recipe generalizes across various real-world datasets such that the models designed according the recipe outperform other baseline models.
ACKNOWLEDGMENTS
This research supported by the Engineering Research Center Program through the Na- tional Research Foundation of Korea (NRF) funded by the Korean Government MSIT (NRF- 201 1R1A5A 1059921)

---

Published as a conference paper at ICLR 2021
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangging Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vin- cent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Watten- berg. Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. httthhttp///eeesrrrrerrre www.fllooooggoo..o Software available from tensorflow.org.
Sami Abu-EI-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In International Conference on Machine Learn- ing. pp. 21-29, 2019.
James Atwood and Don Towsley. Diffusiin--oonvluutional neural networks. In Advances in Neural Information Processing Systems, pp. 1993 2001, 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. In International Conference Learning Representations, 2015.
Aleksandar Bojchevski and Stephan Günnemann. Deep gaussian embedding of graphs: Unsuper- vised inductive learning via ranking. In International Conference on Learning Representations, 2018.
Joan Bruna, Wojeiech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks graphs. In International Conference on Learning Representations, 2014.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder- decoder for statistical machine translation. In Proceedings of the 2014 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pp. 1724-1734, Doha, Qatar, Oc- tober 2014. Association for Computational Linguistics. doi: 10.3115/1//14-1119,, URL
https:///mat htpp:://www... acl web www. wtt.oor///ncoooooo/1111111111
Fan Chung. Graph theory in the information age. Notices of the AMS, 5766)726-732, 2010
Djork-Ame Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). In International Conference on Learning Representa- tions, 2016.
Mark Craven, Dan DiPasquo, Dayne Freitag, Andrew McCallum, Tom Mitchell, Kamal Nigam, and Seán Slattery. Learning to extract symbolic knowledge from the world wide web. In Proceedings of the Fifteenth National//enth Conference on Artificial Inteliigencelnnooaaiiee Applications of Artificial Intelligence, AAAI '98/IAAI '98, pp. 509-516, 1998.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances Neural Information Processing Systems, pp. 3844-3852, 2016.
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P Adams. Convolutional networks graphs for learning molecular fingerprints. In Advances in Neural Information Processing Systems, pp. 2224-2232, 2015.
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In International Conference on Learning Representations Workshop on Representation Learning Graphs and Manifolds, 2019.
Santo Fortunato. Community detection in graphs. Physics Reports, 486(3-5):55-174, 2010
10

---

Published as a conference paper at ICLR 2021
Luca Franceschi, Mathias Niepert, Massimiliano Pontil, and Xiao He. Learning discrete structures for graph neural networks. In International Conference on Machine Learning, pp. 1972-1982, 2019.
Hongyang Gao and Shuiwang Ji. Graph representation learning via hard and channel-wise attention networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 741-749, 2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 249-256, 2010.
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 855-864, 2016.
Aric Hagberg, Pieter Swart, and Daniel S Chult. Exploring network structure, dynamics, and func- tion using networkx. Technical report, Los Alamos National Lab.(LANL), Los Alamos, NM (United States), 2008
Will Hamilton, Zhitao Ying. and Jure Leskovec. Inductive representation learning on large graphs. In Advances Neural Information Processing Systems, pp. 1024-1034, 2017.
Mikael Henaff, Joan Bruna, and Yann LeCun. Deep convolutional networks on graph-structured data. arXiv preprint arXiv:1506.05 2015.
Yifan Hou, Jian Zhang, James Cheng, Kaili Ma, Richard TB Ma, Hongzhi Chen, and Ming-Chang Yang. Measuring and improving the of graph information in graph neural networks. In International Conference on Learning Representations, 2020.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint ar arXiv:2005.00687 2020a.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In International Conference on Learning Representations, 2020b.
Binyuan Hui, Pengfei Zhu, and Qinghua Hu. Collaborative graph convolutional networks: Unsu- pervised learning meets semi-supervised learning. AAAI, pp. 4215-4222, 2020.
Bo Jiang, Ziyan Zhang, Doudou Lin, Jin Tang, and Bin Luo. Semi-supervised learning with graph doaming-convolutionn networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 11313-11320, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Thomas N Kipf and Max Welling. Variational graph auto-encoders. Advances in Neural Information Processing Systems Workshop on Bayesian Deep Learning. 2016.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net-
works. In International Conference Learning Representations, 2017.
Johannes Klicpera, Stefan WeiBenberger, and Stephan Günnemann. Diffusion improves graph learn- ing. In Advances in Newral Information Processing Systems, pp. 13333-13345, 2019.
Boris Knyazev, Graham W Taylor, and Mohamed Amer. Understanding attention and generalization in graph neural networks. In Advances in Neural Information Processing Systems, pp. 4204-4214, 2019.
David Liben-Nowel and Jon Kleinberg. The link- predictionn problem for social networks. Journal of the American Society for Information Science and Technology, 1817)10019-103, 2007.

---

Published as a conference paper at ICLR 2021
Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention-based neural machine translation. In Empirical Methods in Natural Language Processing, pp. 1412- 1421, 2015.
Andrew L Maas, Awni Y Hannun, and Andrew Ng. Rectifier nonlinearities improve neural network acoustic models. In International conference on Machine learning Workshop on Deep Learning for Audio, Speech and Language Processing, 2013.
Julian McAuley and Jure Leskovec. Image labeling on network: using social-network metadata for image classification. European conference on computer vision. pp. 828- 88-1811.22 Springer, 2012.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-based rec- ommendations on styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pp. 43-52, 2015.
Péter Mernyei and Cătälina Cangea. Wiki-cs: A wikipedia-based benchmark for graph neural net- works. arXiv preprint arXiv:2007.02901. 2020.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv preprint arXiv: 2013.
S Pan, R Hu, G Long, J Jiang, L Yao, and C Zhang. Adversarially regularized graph autoencoder for graph embedding, In International Joint Conference on Artificial Intelligence, 2018.
Jiwoong Park, Minsik Lee, Hyung Jin Chang, Kyuewang Lee, and Jin Young Choi. Symmetric graph convolutional autoencoder for unsupervised graph representation learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1519-6528, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In Advances in neural information processing systems, pp. 8026-8037, 2019.
Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gen: Geometric graph convolutional networks. In International Conference on Learning Representations, 2020.
David Rolnick, Andreas Veit, Serge Belongie, and Nir Shavit. Deep learning is robust to massive 2017.
label noise. arXiv preprint
Benedek Rozemberezki, Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding. arXiv preprint arXiv:1909. 2019.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Ran Rad. Collective classification in network data. Al Magazine, 29(3):93-93, 2008.
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv: /8/ 1.05868,, 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-195) 1958, 2014.
Otilia Stretcu, Krishnamurthy Viswanathan, Dana Movshovitz-Atie Emmanouil Platanios, Sujith Ravi, and Andrew Tomkins. Graph agreement models for semi- -unniisseeeiieed learning. In Advances in Neural Information Processing Systems, pp. 8710-8720, 2019.
Aravind Subramanian, Pablo Tamayo, Vamsi K Mootha, Sayan Mukherjee, Benjamin L Ebert, Michael A Gillette, Amanda Paulovich, Scott L. Pomeroy, Todd Golub, Eric S Lander, et al. Gene set enrichment analysis: knowledge-basee approach for interpreting genome-wide expres- sion profiles. Proceedings of the National Academy of Sciences, 102(43):15545-1555) 2005.
Ke Sun, Zhouchen Lin, and Zhanxing Zhu. Multi-stage self-supervised learning for graph convolu- tional networks on graphs with few labeled nodes. In AAAI, pp. 5892-5899, 2020.
12

---

Published as a conference paper at ICLR 2021
Jian Tang. Meng Qu, Mingzhe Wang, Ming Zhang. Jun Yan, and Qiaozhu Mei. Line: Large- scale information network embedding. In Proceedings of the 24th International Conference on World Wide Web, pp. 1067-1077. International World Wide Web Conferences Steering Commit- tee, 2015.
Kiran K Thekumparampil, Chong Wang, Sewoong Oh, and Li-Jia Li. Attention-based graph neural network for semi-supervised learning. arXiv preprint
03033, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Infor- mation Processing Systems, pp. 5998-6008, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference Learning Representations, 2018.
Guangtao Wang, Rex Ying, Jing Huang, and Jure Leskovec. Improving graph attention networks with large margin-based constraints. arXiv preprint arXiv: 1945, 2019a
Kuansan Wang, Zhihong Shen, Chiyuan Huang, Chich-Han Wu, Yuxiao Dong, and Anshul Kanakia. Microsoft academic graph: When experts are not enough. Quantitative Science Studies, 1(1):396- 413, 2020.
Minjie Wang. Lingfan Yu, Da Zheng, Quan Gan, Yu Gai, Zihao Ye, Mufei Li, Jinjing Zhou, Qi Huang, Chao Ma. Ziyue Huang, Qipeng Guo, Hao Zhang, Haibin Lin, Junbo Zhao, Jinyang Li, Alexander Smola, and Zheng Zhang. Deep graph library: Towards efficient and scalable deep learning on graphs. International Conference Learning Representations Workshop Rep- resentation Learning on Graphs and Manifolds, 2019b. URL
1909. 013155
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Yu Philip. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2020.
Zhilin Yang, William W Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning with graph embeddings. In Proceedings of the 33rd International Conference on International Conference on Machine Learning-Volume Volume 48, pp. 40-48, 2016.
Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. When does self-supervision help graph convolutional networks? In International Conference on Machine Learning, 2020.
Youngjae Yu, Jongwook Choi, Yeonhwa Kim, Kyung Yoo, Sang-Hun Lee, and Gunhee Kim. Su- pervising neural attention models for video captioning by human gaze data. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 490-498, 2017.
Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. Graph- saint: Graph sampling based inductive learning method. In International Conference on Learning Representations, 2020. URL httpt::///// /ttp::/
Jiani Zhang, Xingjian Shi, Junyuan Xie, Hao Ma, Irwin King, and Dit-Yan Yeung. Gaan: Gated attention networks for learning large and spatiotemporal graphs. In Conference on Uncertainty in Artificial Intelligence, 2018.
Kai Zhang, Yaokang Zhu, Jun Wang, and Jie Zhang. Adaptive structural fingerprints for graph attention networks. In International Conference on Learning Representations, 2020.
Muhan Zhang and Yixin Chen. Weisfeiler-lehman neural machine for link prediction. In Proceed- ings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 575-583, 2017.
Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. In Advances in Neural Information Processing Systems, pp. 5165-5175, 2018.
13

---

Published as a conference paper at ICLR 2021
Cheng Zheng, Bo Zong, Wei Cheng, Dongjin Song, Jingchao Ni, Wenchao Yu, Haifeng Chen, and Wei Wang. Robust graph representation learning via neural sparsification. In International Conference on Machine Learning, 2020.
Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. arXiv preprint arXiv:1812.08444. 2018.
Marinka Zitnik and Jure Leskovec. Predicting multicellular function through multi-layer tissue networks. Bioinformatics, 33(14);1100ii198, 2017.
14

---

Published as a conference paper at ICLR 2021
A EXPERIMENTAL SET-UP
A. REAL-WORLD DATASET
In this section, we describe details (including nodes, edges, features, labels, and splits) of real-world datasets. We report statistics of real-world datasets in Tables 3 we extend the homophily to the average of the ratio of shared labels on neighbors over nodes, i.e., IV Lier (ZjEN, (,nn,, (110//N11-1100/ ICCD) of all labels.
and For multi-label graphs (PPI),
where C,
labels for node i and C is
A.1.1 CITATION NETWORK
We use total of citation network datasets. Nodes are documents, and edges are citations. The task for all citation network datasets is to classify each paper's topic.
Cora, CiteSeer, PubMed We use three benchmark datascts for semi- supervipe node classifica- tion tasks in the transductive setting (Sen et al. 2008 Yang et al. 2016). The features of the nodes bag-of-worde representations of documents. We follow the train/validation/ees split of previ- ous work (Kipf & Welling 2017). We use 20 samples per class for training, 500 samples for the validation, and 1000 samples for the test.
Cora-ML, Cora-Full, DBLP These are other citation network datasets from Bojchevski & Günnemann 2018). Node features are bag-of-words representations of documents. For CoraFull with features more than 5000, reduce the dimension to 500 by performing PCA. We use the split setting in Shehur et al. (2018): 20 samples per class for training, 30 samples per class for validation, the rest for the test.
ogbn-arxiv The ogbn-arxiv is recently proposed large-scale dataset of citation networks (Hu al.) 2020aa Wang al., 2020). Nodes represent arXiv papers, and edges indicate citations be- tween papers, and node features mean vectors of skip-gram word embeddings of their titles and abstracts. We use the public split by publication dates provided by original paper.
A.1.2 CO-AUTHOR NETWORK
CS, Physics The CS and Physics are co-author networks in each domain (Shchur et al. 2018). Nodes are authors, and edges mean whether two authors co-authored paper. Node features paper keywords from the author's papers, and we reduce the original dimension (6805 and 8415) to 500 using PCA. The split is the 20- of this task to classify each author's respective field of study.
plasssss/st from Shehur et al. (2018). The goal
A.1.3 AMAZON CO-PURCHASE
Photo, Computers The Photo and Computers are parts of the Amazon co-purchase graph (McAuley al. 2015 Shehur al. 2018). Nodes are goods, and edges indicate whether two goods are frequently purchased together, and node features are a bag-of-words representation of product reviews. The split the 0l-per-laas/33--pe-classst from Shehur al. (2018). The task is to classify the categories of goods.
A.1.4 WEB PAGE NETWORK
Wiki-CS The Wiki-CS dataset is computer science related page networks in Wikipedia (Mernyei Cangea 2020). Nodes represent articles about computer science, and edges represent hyperlinks between articles. The features of nodes are mean vectors of GloVe word embeddings of articles.
There are 20 standard splits, and we experiment with five random seeds for each split (total 100 runs). The task is to classify the main category of articles.
Chameleon, Crocodile These datasets are Wikipedia page networks about specific topics, Chameleon and Crocodile (Rozemberczki al. 2019). Nodes are articles, and edges mutual
15

---

Published as a conference paper at ICLR 2021
Table 4: Average degree and homophily real-world graphs.
Dataset
Degree
Homophily
Four-Univ
1,83 + 1.71
0.16
PPI
28.0 39.26
0.17
Chameleon
15.85 + 18.20
0.21
Crocodile
15.48 13.97
0.26
Flickr
10.08 31.75
0.32
Cora-Full
6.41
0.59
ogbn-arxiv
0.63
Wiki-CS
2
0.68
CiteSeer
2.78
0.72
PubMed
4.50
0.79
Cora-ML
5.45
0.81
DBLP
5.37
0.81
Computers
35.76
0.81
Cora
+
0.83
CS
8.93 + 9.11
0.83
Photo
31.13 + 47.27
0.85
Physics
14.38 15.57
0.91
links between them. Node features are bag-of-words representation with informative nouns in the article. The number of features 13183, but we use a reduced dimension of 500 by PCA. The split from [Shchur et al. (2018). The original dataset is for the regression task to predict monthly traffic, but we group values into six bins and make a classification problem.
is
Four-Univ The Four-Univ dataset is a web page networks from computer science departments of diverse universities (Craven al., 1998). Nodes are web pages, edges are hyperlinks between them, and node features are TF-IDF vectors of web page's contents. There are five graphs consists of four universities (Cornell, Texas, Washington, and Wisconsin) and a miscellancous graph from other universities. As the original authors suggested miscellaneous graph for training, another one graph for validation (Cornell), and the other one graph for the test (Texas). Classification labels are types of web pages (student, faculty, staff, department, course, and project).
three graphs of universities and
A.1.5 FLICKR
The Flickr dataset a graph of images from Flickr (McAuley & Leskovec 2012 Zeng et al. 2020).
Nodes are images, and edges indicate whether two images share common properties such geo- graphic location, gallery, and users commented. Node features are a bag-of-words representation of images. We use labels and split in in Zeng al. (2020). For labels, they construct seven classes by manually merging 81 image tags.
A.1.6 PROTEIN- PROTEIN-PROTEIN INTERACTION
The protein-protein interaction (PPI) dataset (Zitnik Leskovec 2017 Hamilton et al., 2017: Sub- ramanian et al., 2005) is well-known benchmark in the inductive setting. A graph is given for human tissue, the nodes are proteins, the node's features are biological signatures like genes, and the edges illustrate proteins' interactions. The dataset consists of 20 training graphs, two validation graphs, and two test graphs. This dataset has multi-labels of gene ontology sets.
A.2 LINK PREDICTION
For link prediction, split 5% and 10% of edges for validation and test set, respectively. We fix the negative edges for the test set and sample negative edges for the training set at each iteration.
16

---

Published as a conference paper at ICLR 2021
Table 5: Statistics of the real-world datasets.
Dataset
# Nodes
# Filges
Features
# Classes
Split
Training Nodes
# Val. Nodes
# Fest Nodes
Oniv-Univ
4518
3426
2000
6
fixed
4014 (3 Gs)
248 (1 G)
256 (1 G)
PPI
56944
818716
50
121
44906 (20 Gs)
6514 (2 Gs)
5524 Gs)
Chameleon
2277
36101
500
6
random
120
180
1977
Crocodile
11631
180020
500
6
random
120
180
11331
Flickr
89250
449878
500
7
fixed
44625
22312
22313
Cora-Full
19793
63421
500
70
random
1395
2049
16349
ogbn-arxiv
169343
1166243
128
90941
29799
48603
Wiki-CS
11701
297110
300
10
lixed
580
1769
5847
CiteSeer
3327
4732
3703
6
fixed
120
500
1000
PubMed
19717
44338
500
3
fixed
60
500
1000
Cora-MLL
2995
8158
2879
7
random
140
210
2645
DBLP
17716
$2867
1639
4
random
80
17516
Computers
13752
245861
767
10
random
200
300
13252
Cora
2708
5429
1433
7
fixed
140
500
1000
CS
18333
81894
500
15
random
300
450
17583
Photo
7650
119081
745
8
random 160
7250
Physies 34493
247962
500
5
random
100
34243
A.3 SYNTHETIC DATASET
To the best of our knowledge, our synthetic datasets are not used in recent literature. Therefore, we give some small examples of synthetic datasets see qualitatively how the average degree and homophily vary according to 5 and pin. Specifically, we draw 2D I-SNE plot of node features and edges in Figure 6 In this figure, we can observe that the average degree (davg n 8) increases as 5 increases, and homophily (h Pin/5) increases as Pin increases. Note that these are raw input features sampled from the 2D Gaussian distribution, not learned node representation. We use the code from the prior w (Abu-El-Hajja et al. 2019) and apply normalization by standard score. We choose 5 from (0.025, 0.2], Pin from {0.15, 0.5Ø, 0.96), and fix 100 and
Figure 6: t-SNE plots of node features and edges for synthetic graph examples. Hyperparameters 8 (0.025 (Top), 0.2 (Bottom)) and Pin (0.18 (Left), 0.58 (Center), 0.98 (Right)).
A.4 DISTRIBUTION OF DEGREE AND HOMOPHILY OF DATASETS
In Figure 7 we draw kernel density estimation plots of per-node homophily and degree of nodes in real-world datasets. We define per-node homophily the ratio of neighbors with the same label as the center node, that is, h; EjeN, 1(0)0())/ [N.] [N.] Note that we define homophily as
M Eiev h; in Section
https://gi thub. com sami hai http://hhhhhhihhhh///aa///aaa//a
py
17

---

Published as a conference paper at ICLR 2021
Per-node homophily
Dataset Chameleon
Dataset Crocodile
Dataset CoraFull
1.0
0.5
0.0
Per-node homophily
Dataset Cora ML
Dataset DBLP
Dataset CS
1.0
0.5
0.0
Per-node hamophily
Dataset Physics
Dataset Photo
Dataset Computers
1.0
0.5
0.0
Per-node homophily
Dataset Flickr
Dataset WebKB4Univ
Dataset WikICS
1.0
0.5
0.0
Per-node hamophily
Dataset ogbn-arxiv
Dataset PPI
Dataset Cora
1.0
0.5
0.0
Degree (log10)
Per-node homophily
Dataset CiteSeer
Dataset PubMed
1.0
0.5
0.0
2
Degren (lag10)
Degree (log10)
Figure 7: Kernel density estimate plot of distribution of degree and per-node homophily in real- world graphs.
We focus more on the part outside of where the degree is 1 (0 with log scale), and the por-node homophily is O. These are leaf nodes incorrectly connected and does not significantly affect the learning overall graph representation. some small modes around it. However, in Flickr and Crocodile, between modes is wide. More specifically, Crocodile's 's modes are in the area of (high degree, low per-node homophily) and (low degree. high per-node homophily), and Flickr's modes cover most homophily at specific degree. Note that we can regard mixture of distribution as a mixture of different sub- aab-graphs. We argue that this is the reason our recipe does fit for these two datasets.
most datasets, only the largest mode exists, or there are
observe that the interval
A.5 MODEL & HYPERPARAMETER CONFIGURATIONS
Model
Since we experiment with numbers of datasets, we maintain almost the same configurations across datasets. We do not other methods such residual connections, deeper layers, batch normalization, edge augmentation. and more hidden features, although we have confirmed from previous studies that these techniques contribute performance improvement. For example, prior
18

---

Published as a conference paper at ICLR 2021
Table 6: Hyperparameters for experiments on real-world datasets.
Dataset
Model
A2
A
Pe
Pn
Cora
SuperGAT
0.007829935945
10.88266937
0.8
0.5
SuperGATMx
0.008228864973
11.34657453
0.8
0.5
CiteSeer
SuperGAT SD
0.04823808657
0.09073992828
0.8
0.3
SuperGATMx Mx
0.04161321832
0.01308169271
0.8
0.5
PubMed
SuperGAT
0.0002030927563
18,82560333
0.6
0.7
SuperGATMx
2.19E-04
10,4520518
0.6
0.5
PPI
SuperGAT
3,39E-07
0.001034351842
0.5
SuperGAT
1.00E-07
1.79B-06
0.5
Table 7: Summary of classification accuracies with 10 random seeds for Cora, CiteSeer, and PubMed in the full-supervised setting. We mark with asterisks the reprinted results from the respective papers.
Model GAT
Cora
CiteSeer
PubMed
87.2 + 0.3
77.3 + 0.3
87.0 + 0.3
CGAT
88.2 + 0.3
78.9 0.2
87.4 + 0.3
CGAT (Our Impl.: 2-layer w/ 64-features)
88.9 0.3
7.2
86.9 = 0.2
SuperGATMxx (2-layer w/ 64-features)
88.7 + 0.2
79.1 + 0.2
87.0 + 0.1
work has shown fl fl-score close 100 for PPI. To clearly see the difference between the various graph attention designs, we intentionally keep simple model configuration.
Hyperparameter For real-world datasets, we tune two hyperparameters (mixing coefficients X2 and Ag) by Bayesian optimization for the mean performance of random seeds. We choose negative sampling ratio Pr, from (0.3, 0.5, 0.7, 0.9). and edge sampling ratio Pe from {0.6, 0.8, 1.0]. We fix dropout probability to 0.0 for PPI, 0.2 for ogbn-arxiv, 0.6 for others. We set learning rate 0.05 (ogbn-arxiv), 0.01 (PubMed, PPI, Wiki-CS, Photo, Computers, CS, Physics, Crocodile, Cora-Full, DBLP), 0.005 (Cora, CiteSeer, Cora-Mo ML, Chameleon), 0.001 (Four- Univ). For ogbn-arxiv, we set the number features per head to 16 and the number of heads in the last layer to one; otherwise, eight features per head and eight heads in the last layer.
For synthetic datasets, we choose Dg from {10-5, 5 10-4, 10-3, 10-2 10-110011111 1,, 102] and A2 from {10-7, 10-5 10- We fix learning rate 0.01, dropout probability 0.2, pn. 0.5, and, Pe to 0.8 for all synthetic graphs.
Table hyperparameters for SuperGAT on four real-world datasets. For other datasets and experiments, please see the code (. /StpeerrTTaaagsss yaml).
A.6 CGAT IMPLEMENTATION
CGAT (Wang al. 20 has two auxiliary losses: graph structure based constraint C, and class boundary constraint Lz. We borrow their notation for this section: V as a set of nodes, N, as a set of one-hop neighbors, N as set of neighbors that share labels, N, set of neighbors that do not share labels, Ç. as a margin between attention values, and ø (v;, Ux) as unnormalized attention value.
cg= = Σ Σ
Σ max (0, o (v, Uk) Sg (vr, Uy)
(10)
JENAAN; KE(V(N)
C, Σ Σ Σ max (0, (1;, vk) Go & (vi,u,)
(11)
JENT REN,
Since label information is included in these two losses, they difficult to use in semi-supervise settings that provide few labeled samples. In fact, in the CGAT paper, they conduct experiments in full-supervised settings; that is, they use all nodes in training except validation and test nodes.
19

---

Published as a conference paper at ICLR 2021
So, only use Eg modified for semi-supervised learning.
SSL ΣΣ Σ
max (0, Φ (vi, v&) Sg o (vi, vj))
(12)
JEN, KE(V)N,)
With Le, the multi-class cross-entropy on node labels, CGAT's optimization objective is
C Le + AgCy + Xehb.
(13)
and our modified CGAT's loss is
L= Le Ag
(14)
In addition to losses, CGAT proposes top-k softmax and node importance based negative sampling (NINS). Top-k softmax picks up nodes with top-k attention values among neighbors. NINS adopts importance sampling when choosing negative sample nodes.
Since the code for CGAT has not been released, implement in our paper, we use only the modified losses (Equation] 14) and top-k softmax due to the training and implementation complexity of NINS. For PPI, even if do not assume semi-supervised setting, we use the same loss because we could not accurately implement multi-label cases for Equation TO and with only CGAT's description.
version. In all experiments
To verify the functionality of our implementation, we report the results of a full-supervised setting with the original loss (Equation [3) like CGAT paper, in Table Our implementation of CGAT shows almost the same performance reported in the original paper. In addition, SuperGAT and CGAT showed almost similar performance in full-supervised setting. Note that the original paper employs two hidden layers with hidden dimensions as 32 for Cora, 64 for CiteSeer, and three hidden layers with hidden dimensions 32 for PubMed, where models in our experiments are two-layer with 64 features.
A.7 WALL-CLOCK TIME EXPERIMENTAL SET-UP
To demonstrate our model's ' efficiency, we measure the mean wall-clock time of the entire training process of three runs using a single GPU (GeForce GTX 1080Ti). We compare our model with GAT (Veličković et al. 2018) and GAM (Stretcu et al. 2019). GAT is the basic model using simpler attention mechanism than ours, and GAM the state-of-the-art model using co-training with the auxiliary model.
For GAT and Sup we use our implementation (including hyperparameter settings) in Py- Torch (Paszke et al., (2099). For GAM, we adopt the code in TensorFlow (Abadi et 2015) from the authors Saahorrsaand and choose GCN + GAM model, which showed the best performance. We retain the default settings in the code but use the hyperparameters reported in the paper, if possible. With this setting, GCN + GAM on PubMed not finished after 24 hours; therefore, we manually early-stop the training at the best accuracy.
research/gam
20

---

Published as a conference paper at ICLR 2021
B RESULTS
B.1 PROOF OF PROPOSITION
Proposition For 1 1th GAT layer, if W and a are independent and identically drawn from zero-mean uniform distribution with variance a and o², respectively, assuming that parameters are independent input features h' and elements of h are independent each other,
Varle = 2 and Varle 2 Futlo (& (((h)) 12)9) Var(( (15)
Proof. Leth' Wh', then n = DF' Wirh
Note that,
I+1 a [h[[lhh]] and 9,00
GP h² h/3
1+1
(16)
First, we compute B(a²) and B.(
E((a)) = Var(a) + E(a)² = o²
(17)
B(A = E (E
(18)
E
W (tl..,)))
(19)
E (
E F )3)
(20)
o E (11h² (
(21)
For the variance of [+1
fj.G0*
Var(e 1+1
(j,00)
Var(a T [h]l1h1]>
(22)
Var Ft+1 (artihrr + ap+pt+th(
(23)
2F Var(ah!)
(24)
2F++1 (E (a²) E (1(2) - E(a))
(25)
-F (a") E (h²²)
(26)
22 E (11h 11))
(27)
Now we compute B(K())) and E(N(2)/2).
E(H h/,a)
(IriT p' Warh! (IFLT) Warths)))
(28)
=E DF'th F'+1 W 1/ (ir)
(29)
=E(W2 TE
h k,,s)
(30)
= Th²)
(31)
21

---

Published as a conference paper at ICLR 2021
(32)
(Err) Wky Wir 2
)²
(In=t
Warhhi.r) ))
E
(33)
p1+1
pt+1
E(W,, k.. E
Σ (ni h//r)) E(W j&B
8f1
(34)
pl+s
B(W K.. E
Σ (nl hb
+ +E( J&B
(mhl,r) -
Emhl
(35)
(I
2
+ 2E (W2,) '
- Σ (h) (b kbtr))
(36)
F
F+11
9 o 4 E
Σ (h' h(rr))
+ +
||0:131113118 - -3 Σ (h k(rr) + 20 (((h)) hh)22)
5
(37)
F111
1 E
||1|12 MILIBI
Σ
(h h( h2,,) +
(38)
5
o E
((h)) h2)²²
+
1/1
1
Σ
(Ohl + h/, 8(11.1b5119)) ) ((hh)) (hl)))
)²
10
eft
(39)
Note that for the zero-mean uniform distribution U(-u, u) with variance 2
B(W?)) = Var( W,,) + (W.,,) o
(40)
12 (u (-u))² = u²
(41)
3
B(W Σ u-)"m "u""""
1
4
9 4
(42)
5
5
[+1 For the variance of Ej.DP*
Var
(())DP)
(43)
Var (h/T h²))
(44)
Var
(45)
-1 Var (KGN))
(46)
=Fit1 (B((H)A))))- - -B(
(47)
4 ((h/) hh/)=== + 1/1 Σ Rh + hhi." + 8(h)
=1tt1 1
5
10 oftt
(48)
+ -1+ 1
(E (((n))TT()))) - B((N)) Th()2)
(49)
2 F o
(((n))TT)))) + Var((h)) Th'))
(50)
22

---

Published as a conference paper at ICLR 2021
Table 8: Summary of classification accuracies (of runs) for synthetic datasets.
Avg. degree
Homophily
GCN
0.2
0.9
52.7 =
35.1 =
34.7 +
40.8 = 0.4
43.4 1 0.3
46.2 =
49.7 0.5
54.8
14.8200 0.9
56.2 0,3
1.5
28.7 1: 0.5
30.0
34,6
0.4
40.8
0.4
41.7
1: 0.4
46.9
1: 0.5
52.1 0.6
54.5
0.5
63.6
2.5
24.6 0.2
27.9
+
31.0 +-0+-33
39.7 +: 0.5
44.0
+ 0.6
51.4 +: 0.8
56.6 0.5
65.2 0.6
74.5 0.4
3,5
25.8
30.5 + 0.8
31.9
38.4
0.4
45.0
1.0
49.8 + 0.5
60.8 0.7
0.3
79.5 0.3
33.0
39.8
50.2
58.6
78.0
7.5
26.1 +
29.8
0.6
34.0 + 0.6
438
0.9
52.0 0.4
70.7 + 3.2
74.6
88.6 0.8
95.0 0.2
10
05
29.2
37.1
0.5
47.6 1.0
58.1
73.1
= 2.8
85.3
92.7 1.1
97.9
12.5
27.6 0.
39.6
48,9
044
65.8
77.8
2.4
95.0
99.2 0.0
15
0.5
28.2
44.0 1 0.7
51.0
0.5
69.9 1 4.7
79.1
2.0
92.2 1.8
9..0
99.6
20
0.6
30.9 +
41.9
58.1
1.2
75.0 + 2.9
86.7 +
0.3
9.19+0.1
99.8
25
26.6 1.0
31.1
+ 0.3
44.9 + 0.2
62.4 0.7
80.1
+ 2.6
91.0
97.5 0.5
9..5
1.0
33.8
83.5
95.7
40
0.5
34.2
0.5
53.4 + 0.4
728
87.6
0.9
96.1 0.6
99.7 0.1
99.9 0.0
100.0
50
25.0 1.1
36.4
1.0
55.1
0.5
82.7
= 3.0
91.2 = 0.7
98.6
99.8
0.0
99.9 0.0
100.0
75
25.9 1.
40.6
1
68.0
1: L.
87.9
2.6
97.1 1 0.6
99.6 0.1
100.0 0.0
100,0 1:
100,0 1 0.0
100
25.1 + 0.5
42.0
0.9
72.4 +: 1.4
92.6
0.4
98.6
+ 0.2
100.0 + 0.0
0.0 100.0
0.0 100.0
100.0 + 0.0
GAT
0.2
0.3
33.7 0.9
36.7
= 0.8
35.2 +
40.1 =
43.0 = 0.8
46.3 = 0.6
49.4
53.8 1,4
56.1 0.5
1.5
28.8
33,0
34.8
+ 0.9
40.7 = 0.4
42.2 = 0.9
46.7 = 1.0
$2.2 0.8
53.6
1.4
62.8
2.5
28.8
30.6
1 0.6
32.3
1 0.5
40.1 1 0.8
44.0
1 1.3
$2.2 1 0.6
55.5
1.5110
64.6 1.3
73.1
3.5
28.0 +
33.4
1.6
33.5 + 0.6
40.1 + 1.0
46.2 + 0.6
50.8 + 0.6
62.5 0.6
71.7 0.6
78.0 0.5
28.0
30.0
36.9
53.6
61.0
0.5
78.7
87,5
7.5
30.0 = 0.9
31.6
2.0
40.4 1 0.5
43.0 1.0
39,5 = 0.6
72.3 = 1.0
79.4
9.44
95.0 0.5
10
34.0
+
546
+ 0.6
77.4 1.5
90.6 +
0.7
98.4
12.5
29.8 1.4
35.5
47,3
586
72.1 = 1.4
86.3 = 0.5
90,7 0.7
9,22
98.9 0
15
31,9 1: 1.6
34.6
1.6
49.5 1 0.9
62.0
0.7
77.2 1 1.5
85.9
0.3
94.4 0.9
0.5
99.4 0.1
20
34.4
38.3
1.6
54.1 1: 19
70.0
1.1
83.9
1.0
93.7 1: 0.1
0.
0.3
99.7
25
35.8 2.0
42.0
24
$7.7 + 0.6
71.7 + 0.9
87.9 + 1.2
9,
98.7 0.6
9..6
99.9 0.1
32.5
1.2
914
9.
0.3
0.1
1.0
40
2..
45.1
0.9
66.5 1: 11
85.7 =: 1.5
93.5 1 1.0
97.6
9.1
9,
99.9 0.1
50
38.7 1.8
49.5
68.9 +
89.5
96.0 = 0.8
99.0
99.7
99.8 0,1
100.0
75
39.6 3.0
53.5
77.6 2.6
92.8
2.3
98.0 + 1.2
99.5 0.3
99.8 0.2
100.0 =
100,0 1 0.0
41.3 1.2
56.6
1.4
81.1
3.5
1.2 95.9
99.0 1 0.4
99.8 0.1
99,9 1..9.0..11
100.0 L 0.1
100.0 1. 0.0
Su
0.2
0.3
0,9
38.5 1.0
40.5 +
39.5 + 1.6
42. = 0.9
44.0 = 0.5
47.1 = 0.5
50.1 + 1.0
$3.0 0.3
56.1 = 1.0
1.5
36.7 =: 1.2
37.8 + 1.8
42.0 +
42.9 = 0.8
45.0 0.7
47,8
= 0.5
$2.9
$4.7 1.2
63.0
2.5
35.7
37.3
1.6
39.2 1:
47.8
49.0 0.3
56.1 1 0.5
1.44
65.9 0.8
70.6
3.5
35.7 1: 1.0
39.2
1.9
41.4
0.5
440 0.7
$1.0 1 0.3
$5.0 1 1.2
65.1
73.5
78.1 0.6
37.0
39.0
1.5
45.0
48.1 1.0
57.5 + 0.6
66.1 0.9
0.2
8.00
88.3
7.5
36.8 = 1.5
38.7
= 0.7
47.3 0.5
$11 0.8
61.2
= 0.6
75.5 - 0.5
82.3 0.9
9.2
95.2 0.3
70,0
80.1
0.6
12.5
38.5 1 1.0
42.0
0.7
$0.5 L 0.6
62.4 1 0.5
73.6 1 0.4
86.8 1 0.4
92.9 0.44
98.0
99.3 1 0.3
15
40.4 =
42.8
+ 0.7
53.6
67.4 =
79.7
1 0,3
89.0
96.3
99.1
99.6
20
37.8 1
42.1
1
56.9 1
70.0 1 0.5
86.1
1 0.8
0..
1..1
99.6 0,2
99.7 1
25
40.0 1.0
48.8
1.2
0.5 59.5
78.7
+ 0.2
90.7
+ 0.2
97.5 0.1
9.1
100.0 + 0.0
99.9 0.1
32.5
48.5
694
827
0.5
94.3
0.2
0.1
0.1
9.1
99.9
40
44.2 1.1
48.7
1.3
69.2 +: 0.7
88.3
= 0.4
97.1
= 0.1
99.7 0.1
9..
99.8 0.2
1.000:0.000
91.2
97.7
75
44,8 H 0.7
56.1 + 0.9
82.7
4 0.6
95.7
4 0.2
99.8 4 0.2
99.9 + 0.1
100.0 0.0
100.0 H: 0.0
100.0 I 0.0
43.1
60.7
1: 0.7
0.3 87.4
98.3 1 0.1
99.9 1 0.0
100.0 1 0.0
100.0 0.0
100.0 0.0
100.0 1 0.0
SuperGAT
0.1
0.2
D.3
0.4
0.3
0.8
0.9
33.4 1: 0.7
36.3 1 0.8
35.4 +: 0.7
40.5 = 0.3
43.4 + 0.6
47.0 = 0.8
50.0 1 0.5
54,2 1.1
$6.4 1: 0.4
47.5 0.8
$2.3 0.7
2.5
26.6
30.5
1.3
33.6
42.7 = 1.5
44.7 1 0.5
$2.2 + 0.5
56.5 0.8
66.7 0.8
74.5 0.7
3.5
27.7 0.9
33.9
1.7
37.9 0.7
42.4 1. 0.6
49.7 1 0.8
$4.2
65.3
1.1
7.72
79.4
28.4
33.4 +
40.9 +
46.2 + 0.4
5644
0.2
64.1
1.8
74.6
1.4
83.1 1.0
90.6 0
7.5
31.1 + 0.7
35.0
0.9
46.5 + 1.1
$1.9 = 0.7
64.3 + 1.5
75.2 + 1.6
82.1 + 1.3
92.6 1.8
95.8 0.2
10
41.5
81,3
1.1
0.7
0.
9..2
12.5
34,2 1: 3.1
41.1 1 0.8
53.3 1 1.0
65.9 1 1.1
77.2 1 1.4
88.1 1 0.7
92.6 1.6
97.5
99.3 0.2
15
68.5
81.0
90.2
96.1
99.2
20
35.2 + 3.1
41.0
62.3 + 1.2
73.9
0.9
86,3 + 0.6
95.2
1.3
9.7
99.6
99.9 0.2
25
36.8 1. 1.9
46.6
1.7
62.7 L 1.1
81.
90.7
1 0.8
96.8 1.5
0.3
99.8 1 0.2
100,0 1 0.0
32,5
36.7
46.8
68.9 0.6
838 0.7
93.7 + 1.0
9.74
0.1
10000
1.000
40
39.8
48.8
1.7
72.1
0.8
81.517
96.6
0.8
0.3
9..
100.0
10000
42.1
92.7
97.1 0.9
100.0
75
41.9 1 111
55.9 1 2.4
80.8 L 1.2
95.0
1.1
99.5 1 0.2
99,9
0.1
100.0
1.0 0.0
100.0 1.001000 0.0
1000 0.0
59.2
96.9
0.7
99.7
100.0
100.0
0.0
100.0
100.0
B.2 FULL RESULT SYNTHETIC GRAPH EXPERIMENTS
In Table 8, report all results of synthetie graph experiments. We experiment on total of 144 synthetic graphs controlling 9 homophily (0.1, 0.2, 0.9) and 16 average degree (1, 1.5, 2.5, 3.5, 7.5, 10, 12.5, 15, 20, 25, 32.5, 40, 50, 75, 100).
23

---

Published as a conference paper at ICLR 2021
KL.OATT, AGR)
ELD.ATT ACH)
C0 10-ayen1 GD-aye:2 3r Drlayan2
Type Uniom
GO lapar-
SDlayerd COlayar- GO lapoe44 DP-layr Diaaye-22
DP-ayen-3
DP DP-laye-4
Unitom
KLDIATI ABR)
KLDIATT (.GR)
GC-layen] G0-ayer? DRlayen11 DP-ayee?
Unibm
GOlayen1 60-byen? 50-aye--3
colayard DRlayar] DP-ayen? OPlayen3 Attention Type (ChtSoor)
DPlayer- Unfom
Attontion Type (CiceSoc1
g
S.A.AT: AGIU
KLOYATT,
16
bayen1
GD-leyer3 0P-aaw--1 DPlayen2
Unitor
COdaean-1
GB laye-
coleyend DPaaynn BP ayen2 DMiayer-3 DPAayend Attention Type (PuDMed)
Un Unitam
Altertion Type
KLDIATT. .20)
60
40
20
GD-tyyr-3
G2-ayer22
Attontion Type (PPI)
Attention (PP)
Figure 8: Distribution of KL divergence between normalized attention and label-agreement on all nodes and layers for Cora, CiteSeer, PubMed, and PPI (Left: two-layer GAT, Right: four-layer GAT).
Table Mean wall-clock time (seconds) of three runs of the training process on real-world datasets.
Model
Cora
CiteSeer 20.4 + 6.7
PubMed
GAT GCN + GAM
11.3 2.7
21.1 + 2.0
709.3 + 235.9
1099.3 + 812.5
6923.3 + 7042.0
SuperGATMx: MX
30.8 0.5
19.3 = 1.1
151.4 11.4
SuperGAT MX+ MPNS
20.8 + 0.2
15.2 + 0.1
84.6 + 0.9
B.3 LABEL-AGREEMENT STUDY FOR OTHER DATASETS AND DEEPER MODELS
In Figure 8 we draw box plots of KL divergence between attention distribution and label agreement distribution for all nodes and layers of two-layer GATs and four-layer GATs. As shown in the paper, we can see that DP attention does not capture label-agreement rather than GO attention. Also, the degree of this phenomenon becomes stronger as the layer goes down.
B.4 WALL-CLOCK TIME RESULT
In Table we report the mean wall-clock time (over three runs) of the training of GAT, GAM, and SuperGATMxx. SuperGAT, we find that negative sampling of edges is the bottleneck of train- ing. So, we additionally implement SuperGAT + MPNS, which employs multi-processing when sampling negative edges. There are three observations this experiment. GCN + GAM is highly time-intensive in the training stage (x53.9 328. versus GAT) for all datasets. Compared to GAT, our model needs x2.7 more training time for Cora and x7.2 for PubMed, and we reduce the time by applying multi-processing to negative sampling (x1.8 1.8 for Cora and x4.0 for PubMed). For CiteSeer, we can see that SuperGAT ends faster than GAT because of faster convergence and fewer epochs.
24

---

Published as a conference paper at ICLR 2021
0.82
577
0.6
0.6
0.5
0.5
54
-2
Maxing (Log) (Cora)
Mixing Comfiicint (Log) (ChaSeer)
Micing Conficert [Log! (Pubited)
Miking Conficient (Log) (19)
Figure 9: Test performance on node classification against the mixing coefficient Ag for
SuperGATMxx (Cora, CiteSeer, PubMed) and SuperGAT (PPI).
0.05
0.02
Test Paff
0.72
0,00
0.746
0.83
0.73
6.744
3
Negative Sampling
Negatave Samping
Negative Sompling (PubMed)
Negative Sampling
Figure 10: Test performance on node classification against the negative sampling ratio Pn. for SuperGATMx (Cora, CiteSeer, PubMed) and SuperGATsp (PPI).
0.73
3.748
20.0130
Text
0.835
0.71
0.8109
8.74
Bdge Sampling Retio (Cora)
Eddg Samplng Rabio (Cheeseee')
Ralio (Pubiedd
Edge Sampling Ratio (PP1)
Figure 11: Test performance node classification against the edge sampling ratio Pe for SuperGATMx (Cora, CiteSeer, PubMed) and SuperGAT'sD (PPI).
B.5 SENSITIVITY ANALYSIS OF HYPER PARAMETERS
We analyze sensitivity of mixing coefficient of losses Åg. negative sampling ratio Pr. and edge sampling ratio P.. We plot mean node classification performance (over 5 runs) against each hyper- parameter in Figure(9 rreppectiivyll We use the best model for each dataset: SuperGAT^x for citation networks and SuperGATTsD for PPI.
For Äg. there is specific range that maximizes test performance in all datasets. Performance PPI is the largest when Ag 10-3, but the difference is relatively small comparing to others. We observe that there an optimal level of the edge supervision for each dataset, and using too large Åg degrades node classification performance.
For Pa. using many negative samples has been shown decrease performance. The optimal number of negative samples is different for each dataset, and all are less than the number of positive samples (Pn. < 1.0). Note that as Pn, increases, the required GPU memory also increases. When Pa 5.0, the model and data for PPI could not be accommodated by one single GPU (GeForce GTX 1080Ti).
When changes, the performance also changes, but the pattern is different by datasets. For Cora and PubMed, the performance against Pe shows the convex curve. Performance for CiteSeer gen- erally decreases as P. increases, but there are intervals the performance change of which is nearly zero. In the case of PPI, there are no noticeable changes against Pe-
25


Reviews:
[{"review_id": "oGOSyw_tZGG", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose an enhanced GAT model named SuperGAT by adding link prediction as an auxiliary task when conducting node classification and compare different methods of attention forms. The authors have also conducted evaluation based on both synthetic datasets and real-world datasets to analyze how different attention methods perform on various data and task types. In general, the paper is written well and easy to follow. However, there are still several issues the authors need to address:\n\nFirst, the authors need to better justify the novelty of the proposed method. The claimed self-supervision task can be considered as general link prediction task where the attention weights is used as features. Though they authors propose two new types of attention forms, i.e., 1) scaled dot-product and 2) mixed GO and DP, they are normalized and combined version of existing attention mechanisms. I suggest the authors to better justify the novelty of the proposed technique and how they differ from existing work.\n\nAnother major concern is the experiment settings. It is good to see that the authors raise several research questions to guide experiment design. However, some assumptions in these research questions are questionable. For example, in RQ1, the authors claim that “ideal node representation can be generated by aggregating only neighbors with the same label”. As the neighborhood information of a node can be also informative when predicting node labels in certain cases, a good node representation does not necessarily need to only aggregates neighbors of the same labels, which makes the proposed method that uses KL divergence to compare label-agreement and graph attention questionable. I suggest the authors to provide more justification on this assumption. RQ2’s primary goal is to understand how different graph attention methods perform for the link prediction task, it would be better if the authors can justify why they didn’t conduct experiments where only link prediction (self-supervised) loss is used and discard the node classification task. In RQ3, the authors hypothesize that “different graph attention will have different abilities to model graphs under various homophily and average degree”. This is probably true. However, given the so many graph properties (e.g., degree distribution, graph diameter, and average clustering coefficient) and model configuration (e.g., # of layers and task type), it is unclear why the authors choose these two controlled variables and why they believe they are the most important ones. I suggest the authors provide more rationale on how they choose the controlled variables and how other factors may impact model performance.\n\nAnother minor question is that why do the authors add an activation function for $e_{ij, DP}$ in Eqn. 4, given that $e_{ij, DP}$ is already a dot product that indicates the weight of a link. It would be better if the authors can elaborate more on the design rationale.\n\nIn summary, I think the authors focuses on an interesting problem but need to further address the issues listed above.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "bxd5kTltVEU", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new attention mechanism SuperGAT (with various flavours) for graph neural networks that is self-supervised. They exploit the presence/absence of an edge between a pair of nodes to guide the attention. The authors then make the observation that the homophily and average degree of a graph influence the design of the attention mechanism.  Extensive experiments are shown, where the various versions of SuperGAT are tested on 17 real-world dataset and many synthetic ones, and these results are compared against other state-of-the-art models (including the original GAT work). \n\nThe paper is well-written and it is clear that the authors have done extensive experimentation to test their hypothesis. \nHowever, the paper has some weaknesses that I try to summarize below.\n- To obtain SuperGAT, the authors have made some tweaks made to the original GAT formulation. These tweaks are minor and are not surprising or inspired from a deep/novel insight.  \n\n- The choice of studying two graph properties homophily and average degree seem arbitrary. What is the reasoning behind these properties? Were there other properties (e.g. diameter, degree sequence, ...etc) that the authors have studied that did not yield good results? This is particularly of interest due to the fact that experiments do not entirely support/explain the importance of these two properties, such as in the case of Flickr and Crocodile datasets.\n\n- The above reasons would still not be a major disadvantage had the experimental results shown strong superiority of SuperGAT over previous models. In all the experiments that the authors perform, SuperGAT’s performance is only slightly better than older models (e.g. in Table 2, difference between an accuracy of 72.5 vs 72.6 can hardly be considered superior). And even then, SuperGAT does not have the best performance across all datasets.\n\nProposition 1 and its proof are interesting contributions of this work, but they may not be enough. Perhaps instead of concentrating on performance superiority, the authors can look at the explainability aspect of their proposed architecture and look deeper into other graph properties that may guide the design/use of self-supervision.\n\nAnother minor comment: on Page 3, the authors say “if the number of nodes is large, it is not efficient to use all possible negative cases”. The number of negative cases is a function of the number of edges and not the number of nodes. If a graph contains all possible edges, then the number of all possible negative cases is zero, no matter how many nodes there are. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TX4JV7-EeLC", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "********Summary\nIn this paper, they introduced self-supervised graph attention network (SuperGAT), which is claimed to perform well in noisy graphs. They used information in the edges as an indicator of importance of relations in the graph, then they learn the relational importance using self-supervised attention. After learning the attentions values using their self-supervised method, they can predict the likelihood of an edge between nodes. They worked on two popular attention mechanisms: GO and DP, and showed in their experiments DP has better performance than GO for link prediction task. And Go has better performance in label-agreements between nodes. The other question they answered in their experiments was what the best attention model is to choose. They introduce a recipe based on two graph characteristics: homophily and average degree.\n\n********Positives\n- One thing I liked about this paper was thorough and neat experiments. I enjoyed the way they designed their experiments by mentioning several important questions followed by their answered backed up with their experiments. They used two attention mechanism as the base, then applied their method on these two methods for link prediction and label-agreements tasks and compare their results.\n\n- I also liked that they examined their recommendation for the choice of attention model on real world datasets, and their answer for real-word data was almost similar to synthetics data.\n\n\n- The paper was well-organized and well-written. They clearly explained their method.\n\n********Notes\n- I would recommend adding a picture showing their architecture and compare it with other two attention models\n\n- I sort-of understand the reading as to why \"GO learns label-agreement better than DP.\" Based on the argument on page 6. A strong argument would be helpful to explain why \"DP predicts edge presence better than GO.\"\n\n- (minor note:)No need the parenthesis in this sentence line 8: \"Interestingly, for datasets (CS, Physics, Cora-ML, and Flickr) in which\"\n\n********Reason to accept\nI am in general positive about this paper. The innovation is not significant; however, their experiments were interesting, and they prove how well their method works well empirically. I think this research will be useful for people in this area.\n\n******* After Rebuttal\nI have read the author's response\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "GUGjDIJ7MXm", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n===========\nThe paper provides an interesting direction for improving the Graph Attention Networks. More specifically, the authors propose a self-supervised graph attention network (SuperGAT) designed for noisy graphs. They encode positive and negative edges so that SuperGAT learns more expressive attention. They focus on two characteristics that influence the effectiveness of attention and self-supervision: homophily and average degree. They show the superiority of their method (4 variations) by comparing it with many state-of-the-art methods, in 144 synthetic datasets (with varying homophily and average node degree) and 17 real-world datasets (again with various ).\n\n\nReasons for score:\n===========\nOverall, I vote for accepting. I find very interesting the idea of using self-supervision to improve graph attention networks and the experiments are nicely done and convincing. The authors do an impressive work to include as much information and results as they can in the given space.\n\n\nStrengths:\n===========\n- The paper is about an interesting problem in the ICLR community. Graph Attention Networks have gained a lot of attention in the recent years from researchers in the field of graph and node representations with applications in node classification and link prediction. The idea of adding the advancements in recent direction of self-supervised learning to improve the learnt representations seems very promising.\n- The authors have done a great job in the structure and the presentation of the paper. The paper is well-written and especially the sections Experiments and Results are well-structured and contain a lot of packed details in the design and the outcome of the experiments. More specifically, Figure 4 stands out as in a very limited space contains the information for the best performed model in all 144+17 graphs!\n- The contributions of this work include the proposed method (GANs with self-supervision), but also an analysis for the selection of the best model depending on two important features of the graph (homophily and average degree).\n- The authors compare their method with all state-of-the-art methods and also four variations of their own model and exhaust their evaluation by testing the performance in 144 synthetic graphs and 17 real-world datasets (including the benchmark datasets that are usually being used in this domain).\n\n\nWeaknesses:\n===========\n- The proposed method uses two known graph attention mechanism as building blocks, they use negative sampling and they add cross-entropy loss for all node labels and self-supervised graph attention losses for all layers. These building blocks and mechanisms are known in the literature, and as a result the proposed method adds incremental novelty compared to the related works.\n- In Appendix, A.3, the description and discussion for t-SNE plots is limited or absent. It would be better to add more details to it, as for example why this is a good representation and how the representations improve or not based on the hyperparameters. Also, how the results in the subfigures differ in terms of representations. It is difficult to get any insights from these plots.\n\n\nQuestions during rebuttal:\n===========\n- Overall, my recommendations for more analysis and insights of the results are all responded from the Appendices. I would like a comment and clarification from the authors regarding Appendix A.3 Figure 5 (t-SNE plots), even though it is not in the main paper submission.\n- My understanding is that the authors are going to release the code upon acceptance, is this correct? In the repository that the code will be released, it will be useful to also add links to all 17 public datasets to ease research in the field. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two reviewers are very positive about this paper and recommend acceptance, one indicates rejection and one is on the fence. Although all referees appreciate the extensive experiments and analysis presented in the paper, their main concerns are related to the limited superiority of the method wrt state of the art [R1], seemingly arbitrary choices and questionable assumptions [R4]. The rebuttal adequately addresses R1's concerns by highlighting statistical significance of the results, and partially covers R4's concerns. Although the proposed approach may be perceived as incremental [R1, R2, R3, R4], the authors argue that introducing self-supervision to graph attention is not trivial, and emphasize their findings on how/when this is beneficial. Moreover, R2 and R3 acknowledge that the contribution of the paper holds promise, is worth exploring, and may be useful to the research community. Most reviewers are satisfied with the answers in the rebuttal. After discussion, three referees lean towards acceptance and the fourth reviewer does not oppose the decision. I agree with their assessment and therefore recommend acceptance. Please do include your comments regarding the choice of average degree and homophily in the final version of paper.
Decision: Accept (Poster)