Learnable Graph Convolutional Attention Networks
Authors: Adrián Javaloy
Pablo Sanchez Martin
Amit Levi
Isabel Valera
Venue: NeurIPS 2022 Submitted (2022)
Keywords: GNN
GCN
GAT

Abstract: Existing Graph Neural Networks (GNNs) compute the message exchange between nodes by either aggregating uniformly (convolving) the features of all the neighboring nodes, or by applying a non-uniform score (attending) to the features. Recent works have shown the strengths and weaknesses of the resulting GNN architectures, respectively, GCNs and GATs. In this work, we aim at exploiting the strengths of both approaches to their full extent. To that end, we first introduce a graph convolutional attention layer (CAT), which relies on convolutions to compute the attention scores. Unfortunately, as in the case of GCNs and GATs, we then show that there exists no clear winner between the three—neither theoretically nor in practice—since their performance directly depends on the nature of the data (i.e., of the graph and features). This result brings us to the main contribution of this work, the learnable graph convolutional attention network (L-CAT): a GNN architecture that allows us to automatically interpolate between GCN, GAT and CAT in each layer, by only introducing two additional (scalar) parameters. Our results demonstrate that L-CAT is able to efficiently combine different GNN layers across the network, outperforming competing methods in a wide range of datasets, and resulting in a more robust model that needs less cross-validation.

Learnable Graph Convolutional Attention Networks
Anonymous Author(s) Affiliation Address email
Abstract
Existing Graph Neural Networks (GNNs) compute the message exchange between nodes by either aggregating uniformly (convolving) the features all the neighbor- ing nodes, by applying non non-unifon uniform score (attending) to the features. Recent works have shown the strengths and weaknesses of the resulting GNN architectures, respectively, GCNs and GAT's In this work, we aim at exploiting the strengths of both approaches to their full extent. To that end, we first introduce graph convolutional attention layer (CAT), which relies convolutions to compute the attention scores. Unfortunately, as the case of GCNs and GATs, we then show that there exists clear winner between the three-neithhr, theoretically nor in practice-since their performance directly depends on the nature of the data (i.e., of the graph and features). This result brings to the main contribution of this work, the learnable graph convolutional attention network (L-CAT): a GNN archi- tecture that allows to automatically interpolate between GCN, GAT and CAT in each layer, by only introducing two additional (scalar) parameters. Our results demonstrate that L-CAT is able to efficiently combine different GNN layers across the network, outperforming competing methods in a wide range of datasets, and resulting in a more robust model that needs less cross-validation.
2
6
8
10
12
14
16
17
Introduction
19 In recent years, Graph Neural Networks (GNNs) [25] have become ubiquitous in machine learning, emerging as the standard approach in many settings. For example, they have been successfully applied for tasks such as topic prediction in citation networks [26]; molecule prediction [11]; and link prediction in recommender systems [33]. These applications typically make use of message- passing GNNs [11], whose idea is fairly simple: in each layer, nodes are updated by aggregating the information (messages) coming from their neighboring nodes.
20
21
22
23
24
25
Depending on how this aggregation is implemented, we can define different types of GNN layers. Two important and widely adopted layers graph convolutional networks (GCNs) [18], which uniformly average the neighboring information; and graph attention networks (GATs) [30]. which instead perform a weighted average, based on an attention score between receiver and sender nodes. More recently, number of works have shown the strengths and limitations of both approaches from
26
27
28
29
30 a theoretical [2, 3, 10], and empirical [19] point of view. These results show that their performance depends on the nature of the data at hand (i.e., the graph and the features), thus the standard approach select between GCNs and GATs via computationally demanding vooss-validation.
31
32
33 In this work, we aim to exploit effectively and efficiently the benefits of both convolution and
34 attention operations in the design of GNN architectures. To that end, we first introduce novel graph
35 convolutional attention layer (CAT), which extends existing attention layers by taking the convolved
36 features as inputs of the score function, thus taking advantage of both operations. Following [10].
37 we rely on a contextual stochastic block model to theoretically compare GCN, GAT, and CAT
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

38
architectures. Our analysis shows that, unfortunately. no free lunch exists among these three GNN
39
architectures since their performance, as expected, is fully data-dependent.
40 This result motivates the main contribution of the paper, the learnable graph convolutional attention network (L-CAT)): novel GNN which, in each layer, is capable of automatically interpolating between the three operations during training by introducing only two additional (scalar) parameters.
42
43 As result, L-CAT is able learn the proper operation to apply each layer, thus combining different
44 layer types in the same GNN architecture while overcoming the need to cross-validate- process
45 that was prohibitively expensive prior to this work. Our extensive empirical analysis demonstrates the
48 capabilities of L-CAT on wide range of datasets, outperforming existing baseline GNNs in terms of both performance, and robustness to input noise and network initialization.
47
2
Preliminaries
48
Assume we are given as an input an undirected graph = (V, E), where V [x]] denotes the set of vertices of the graph, and E C V x V the set of edges. Each node i E [n] is represented by d-dimensional feature vector X; E R", and the goal is to produce a set of predictions {ūr)
49
51
To this end, message-passing GNN layer yields, for each node i, a representation h; E IRd", by collecting the information from each of its neighbors; aggregating them into a single message; and 54 using the aggregated message to update its representation from the previous layer, h; E Rd. For the purposes of this work, we can define this operation as the following:
52
53
55
h, = f(h)
where
h
def Σ
76 W.hj
(1)
JEN;
56 where N = N; (i), and N; denotes the set of neighbors of node i, W, Rd'xd learnable weight matrix, f an elementwise function, and Vig E [0, [] are coefficients such that Σjen; Vii =
57
58 for each node i.
Let the input features be h
X,, and the predictions be h D.. we can readily define message- passing GNN [11] as a sequence of L layers as defined above. Depending on the way the coefficients 7ij are computed, we can identify different GNN flavors.
60
61
62
Graph convolutional networks (GCNs) [18] are a simple (yet effective) type of layers. In short,
63
GCNs simply compute the average of the messages, i.e., they assign the same coefficient Ky = 1/| to every neighbor:
64
h; = f((( where h del
W.hj, JEN;
(2)
N
65 Graph attention networks take different approach. Instead of assigning fixed value to each
66 coefficient Tky, they dynamically compute it as a function of the sender and receiver nodes. A general formulation for these models can be written as follows:
87
h; = f(4) where
1
def Σ Ki W.h,, and Vi def
exp( T (he, h2)) EIEN; exp V (h, he)
JEN;
68 Here, (1t,, h2)) def
= a eWW,,, Wah,) is known as the score function (or attention architecture), and 89 measures the similarity between the messages h; and h, (or more generally, between a learnable
70 mapping of the messages). From these scores, the (attention) coefficients are obtained by normalizing
71
them, such that E, Mly We can find in the literature different attention layers. Throughout this
72 work, we focus on two types, the original GAT [30], and its extension GATv2 [5]:
GAT: V (h;, h2) = LeakyRelu [WahallWshs))
(4)
(5) 73 where the learnable parameters are now the attention vector a; and the matrices W., We. and W.. Following previous work [5, 30], we assume that these matrices coupled, i.e., W W. W
GATv2: V/ (h;, h2 = a LeakyRelu (W,h: + Wyh,)
75 Note that the difference between the two layers lies the position the vector a: by taking it out of the nonlinearity, Brody et al. [5] increased the expressiveness of GATv2. Now, the product of a and 77 weight matrix does not collapse into another vector. More importantly, the addition of two different attention layers will help show the versatility of the proposed models later in $6.

---

79
3
Previous work
80 In recent years, there has been a surge of research in GNNs. Here, we discuss other GNN models, attention mechanisms, and the recent findings on the limitations of GCNs and GATs.
81
The literature on GNNs is extensive [4, 14, 21, 34], and more abstract definitions of a message- passing GNN are possible, leading to other lines of work trying different ways to compute messages, aggregate them, or update the final message [7, 13, 35]. Alternatively, another line of work fully abandons message-passingg working instead with higher-order interactions [22]. While some of this work is orthogonal-or directly applicable-to the proposed model [7, 13, 35], here we focus on convolutional and attention graph layers, as they are the most widely used (and cited) as of today.
83
84
85
86
87
While we consider the original GAT [30] and GATv2 [5], our work can be directly applied to any attention model that sticks to the formulation in Eq. 3. For example, some works propose different metrics for the score function, like the dot- dot-product [5], cosine similarity [28]. or a combination of various functions [17]. Other works introduce transformer- based mechanisms [29] based on positional encoding [9, 20] or on the set transformer for specific type of graphs, such as relational [6, 37] or heterogeneous graphs [16, 32].
89
90
91
[31]. Finally, there also exist attention approaches designed
92
93
94
3.1
On the limitations of GCN and GAT networks
In [2]. the authors study classification on Gaussian mixture, where the data correspond to the node features of a stochastic block model. They showed that when the graph is neither too sparse nor noisy, applying one layer of graph convolution increases the regime in which the data is linearly separable. Namely, if the distance between the means of the classes is not too small, the convolved features are linearly separable, whilst the original features are not. However, the above result is highly sensitive 99
96
97
98
100
to the graph structure. Indeed, even if the distance between the means is large, the convolution cannot make the data linearly separable when the graph is noisy, since the convolution operation essentially collapses the means of the two classes to the same value.
101
102
103
More recently, Fountoulakis et al. [10] showed that GAT is able to remedy the above issue, and provide perfect node separability regardless of the noise level in the graph. Specifically, they showed that if the distance between the means is large compared to the standard deviation, then GAT achieves perfect node separability with high probability. However, a classical argument (see [1] states that in this setting graph-basee models are unnecessary, since simple linear classifier already achieves perfect separability (see Proposition 4 in [10]). In addition, when the distance between the means small compared to o, no score function v can drop inter-class edges (the noisy edges), and thus might not achieve perfect node separability (see Conjecture 7 in [10]).
104
106
107
108
109
110
The above discussion implies that for some datasets, GAT might not work well as expected. However, it leaves open the question of which architecture (GCN or GAT) is preferable in terms of performance.
112
113
4
Convolved attention: benefits and hurdles
114
115
In this section, we propose to combine attention with convolution operations. To motivate it, we complement the results of [10], providing synthetic dataset for which any 1-layer GCN fails, but 1-layer GAT does not. Thus, proving clear distinction between GAT and GCN layers. Besides, we show that convolution helps GAT as long as the graph noise is reasonable. The proofs for the two statements in this section appear in Appendix A and follow similar arguments as in [10].
116
117
118
119
120
This synthetic dataset is based on the contextual stochastic block model (CSBM) [8]. Let ....... En be i.id uniform samples from {-1, 0, 1}. Let C₁ = {j E [n] Ej = k} for l E {-1,0, 1}. We set the feature vector X; ~ Nss; p,, I o²) where μ E Rd, o E R, and I E {0, 1), dxd is the identity matrix. For a given pair p, q E [0, 1] we consider the stochastic adjacency matrix
121
122
123
124 A E {0, 1)2xn defined as follows: for i,j E [n] in the same class (i.e., intra-edge). we set
125 ai ~ Ber(p):" for i.j in different classes (i.e., inter-edge). we set aij Ber(g). We denote by (X. A) CSBM(n. p, q, n., d²) sample obtained according to the above random process. Our task
126
127
is then to distinguish (or separate) nodes from Co vs. C_1 U U.-
Ber(-) denote the Bernoulli distribution.
3

---

128
Note that it is impossible to separate Co from C-1UC1 with a linear classifier (with high probability). In addition, by applying similar arguments as in [2], using one convolutional layer detrimental for node classification on the CSBM This follows from the fact that although the convolution brings the means closer and shrinks the variance, the geometric structure of the problem does not change. On the other hand, we prove that GAT' is able to achieve perfect node separability when the graph is not too sparse:
129
131
132
133
Theorem 1. Suppose that p,q 22oog n/n) and ||al|2 V(ooloo Then, there exists choice of attention architecture v such that, with probability at least on(1) over the data X A) CSBM(n, q. nr, a²), GAT separates nodes Co from C₁ UC--1
135
136
Moreover, we show using methods from [2], that the above classification threshold || can be improved when the graph noise is reasonable. Specifically, by applying convolution prior to the attention score, the variance of the data is greatly reduced, and if the graph is not too noisy, the operation dramatically lowers the bound on ||||| in Theorem 1. Motivated by this, we introduce the graph comvolutional attention layer (CAT), which formalizes this idea:
138
140
(h; h2) (Wh, Wh;) where
h
1
Σ he: BEN;
(6)
INT
and where h, are the convolved features of the neighborhood of node i. As we show now, CAT improves over GAT by combining convolutions with attention, when the graph noise is low.
142
143
Corollary 2. Suppose q = 22(lg"" n/n) and |pI|| 2 w (ov (p+2g) logn Then, there is a choice n(p-q)²
144
145
of attention architecture V such that, with probability at least o(1) over the data (X, A) CSBM p, p, q, μl, o²). CAT separates nodes Co from C1 UC-1-
The above proposition shows that under the CSBM data model, convolving prior attention changes the regime for perfect node separability by factor of | 9 /n/nn(pp + 2q). This is desirable when
Ip- /l (p- + 2q) > 1, since the regime for perfect classification is increased. Nonetheless, when 149
P - al ///n/pp 2q) < 1, applying convolution prior to attention reduces the regime for perfect separability. Therefore, it is not always clear whether convolving prior to attention is beneficial.
150
151
5 L-CAT: Learning to interpolate
152
153
From the previous analysis, we can conclude that it is hard to know a priori whether attention, convolution, or convolved attention, will perform the best. In this section, we argue that this issue can be easily overcome by learning to interpolate between the three.
155
156 First, notice that the formulations of GCN and GAT only differ in that GCN weighs all neighbors cqually (Eq. 2) and, the more similar the attention scores are (Eq. 3), the more uniform the coefficients
157
158
Tij will be. Thus, we can interpolate between GCN and GAT by introducing a learnable parameter
159
A E [o. 1]. Similarly, the formulation of GAT (Eq. 3) and CAT (Eq. 6) differ in the convolution
160
within the score, which can be interpolated by another learnable parameter X2 E [0, 1].
161
Following this observation, we propose the learnable convolutional attention layer (L-CAT), which can be formulated as an attention layer with the following score:
162
h; A2 ELEN, h
V (h,, h₂) = A1 - - (Wh,, Why) where h = 1 >22NNa
(7)
and where \1, A2 [0, 1). As mentioned before, this formulation lets L-CAT learn to interpolate between GCN (A1 = 0), GAT (A1 = 1 and X₂ = 0), and CAT (A1 = 1 and X22 = 1).
163
164
165
Despite its simplicity, L-CAT enables a number of non-trivial benefits. Not only can switch between existing layers, but it can also learn to use the amount of attention necessary for each use-case. Moreover, by comprising the three layers
186
167
single learnable formulation, it removes the necessity of cross validatii the type of layer, since their performance is detendeeenddnt (see $$3.1 1 and 4). More importantly, eases the task of combining different layer types within the same architecture.
168
189
We note that this problem can be easily solved by two layers of GCN [3].

---

GON
GAT
CAT
||
||,||=4.3
-=0.1
g=0.3
100
100
100
Accuracy (%)
80
75
75
60
50
50
01.2
0.4
0.2 0.4
[ill 10
9
1Pl
Figure I: Synthetic data results. In the two left-most figures, we show how the accuracy varies with the noise level y for || 0.1 and | 4.3. In the right-most figures, show how the accuracy varies with the norm of the means | for q = 0.1 and = 0.3. We use two vertical lines present the classification threshold stated in Theorem (solid line) and Corollary 2 (dashed line).
Experiments
170
In this section, assess the performance of the proposed models, CAT and L-CAT. First, validate our theoretical findings on synthetic data ($6.1), 1). Second, we compare all methods in various scall-scale node classification tasks ($6.2). And finally, we evaluate the proposed models on more realistic scenarios from the Open Benchmark Graph [15] framework, assessing their performance and robustness feature noise and network initialization ($6.3).
172
173
174
175
176
6.1
Synthetic data
177
In this section, we empirically validate our theoretical results (Theorem I and Corollary 2). We aim to better understand the behavior of each layer as the properties of the data change, i.c., the noise level (proportion of inter-edges) and the distance hetween the means consecutive classes /| We provide in Appendix B extra results and additional experiments.
179
181
Experimental setup. As data model, we use the proposed CSBM (see $4) with n = 10000, p = 0.5, 0.1, and d n/ (5 log (n)). All results are averaged over 50 runs, and parameters are set as described in Appendix A. To assess the sensitivity to structural noise, we perform two experiments. First, we vary the noise level q between and 0.5, leaving the mean vector μa fixed. We test two values of |A]: the first corresponds to the easy regime (i)pal 10 /2 log ni) where classes are far apart; and the second correspond to the hard regime (Il)4]) = o) where the distance between the clusters small. In the second experiment
182
183
184
186
modify instead the distance between the means, sweeping || [I|| in the range a /20 20 / log ra). and thus covering the transition from the hard setting (small |||]]) to the easy one (large || Here, we fix g to 0.1 (low noise) and 0.3 (high noise). In both cases, we compare the behavior of 1-layer GAT and CAT, and include GCN the baseline.
188
189
190
191
The two left-most plots of Fig. show the node classification performance for the hard and casy regimes, respectively, we vary the noise level g. In the hard regime, observe that GAT unable to achieve separation for any value of q, whereas CAT achieves perfect classification when q is small enough. This exemplifies the advantage of CAT over GAT stated in Corollary 2. W hen the distance between the means is large enough, we see that GAT achieves perfect results independently of g, stated in Theorem In contrast, as increase g, CAT fails to satisfy the condition in Corollary 2, and therefore achieves inferior performance.
Results.
199
195
197
The results for the second set of experiments, where we fix g and sweep || are shown in the right-most part of Fig. In these two plots, we can appreciate the transition in the accuracy of both GAT and CAT a function of ||д/] We observe that GAT achieves perfect accuracy when the distance between the means satisfies the condition in Theorem (solid vertical line in Fig. I). Moreover, we see the improvement CAT obtains over GAT. Indeed, when |||/|| satisfies the conditions of Corollary 2 (dashed vertical line in Fig. 1), the classification threshold is improved. As that the gap between the two vertical lines decreases, which means that the improvement decreases as q increments, exactly as stated in Corollary
199
201
203
increase
205
These coulls--alone with empirical evidence in the next sections sections-reinforee the idea that there is priori no way to tell which layer to use, as their performance highly depend on the properties of the data. Prior to this work, this has been solved by cross-validating the layer type. In the next sections,
207
5

---

Table 1: Test accuracy (%) of the considered convolution and attention models for different datasets (sorted by their average node degree), and averaged over ten runs. Bold numbers are statistically different their baseline model (a 0.05). Best average performance is underlined.
Dataset
Amazon Computers
Amazon Photo
GitHub
Facebook PagePage
Coauthor Physics
TwitchEN
Avg. Deg.
35.76
31.13
15.33
15.22
14.38
10.91
GCN
90.59 + 0.36
95.13 + 0.57
84.13 + 0.44
94.76 + 0.19
96.36 + 0.10
57.83 + 1.13
GAT
89.59 0.61
94.02 + 0.66
83.31
0.18
94.16 + 0.48
96.36
0.10
57.59 + 1.20
CAT
90.58 + 0.40
94.77 + 0.47
84.11 0.66 84.05 0.70
94.71 = 0.30
96.40 1:400:. 0.10
58.09 + 1.61
L-CAT
90.34 + 0.47
94.93 + 0.37
94.81 + 0.25
96.35
+ 0.10
57.88 + 2.07
GATv2
89.49 0.53
93.47 + 0.62
82.92
0.45
93.44 + 0.30
96.24 0.19
57.70 + 1.17
CATv2
90.44 + 0.46
94.81 + 0.55
84.10 0.88
94,27 + 0.31
0.12
57.99 + 2.02
L-CATv2
90.33 + 0.44
94.79 + 0.61
84.31 0.59
94.44 + 0.39
96.29 - 0.13
57.89 + 1.53
209
we empirically demonstrate that L-CAT can automatically perform layer selection during training, completely removing the need of cross-validating and, thus, saving computational resources.
211
6.2
Real data
212
In this section, we study the performance of the proposed models in comprehensive set of real-world experiments, in order to gain further insights into the settings in which they excel. Specifically, we found CAT and L-CAT to outperform their baseline models as the average node degree increases. For a detailed description of the datasets and additional results, refer to Appendices C and D.
213
214
215
216
We consider as baselines a simple GCN layer [18] where all neighbors are uniformly weighted, as well as the original GAT layer [30] and its recent extension, GATv2 [5]. See $2 for an introduction. Based the two attention models, consider their CAT-extensions, CAT and CATv2, as well as their interpolatable counterparts, L-CAT and L-CATVv.. To ensure fair comparisons, all layers use the same number of parameters and share the same implementation, appropriately setting
Models.
217
218
219
220
221
Å and A (see Eq. 7) for each model.
Datasets. We take six node classification datasets. The
datasets relate to nettokk graphs [24], whose nodes represent verified pagesdeevllperxssttemmrrr and where the task is predict the topic/expertise/expiiti language usage of the node. The Coauthor Physics dataset [27] represents a co-authorship network whose nodes represent authors, and the task infer their main research field. Finally, the Amazon Computers/Amazzon Photo datasets represent two product-similarity graphs [27]. where each node is a product, and the task is to infer its category,
223
225
226
227
Experimental setup. To ensure the best results, cross- vals-val all optimization-relin rameters for each model using GraphGym [36]. All models use four GNN layers with hidden size of 32, and thus have an equal number of parameters. For evaluation, we take the best-validation configuration during training, and report test-set performance. For further details, refer to Appendix D.
hyperpa-
229
231
232
Results are presented in Table 1, averaged over 10 runs. In contrast with $6.1, we here find GCN to be a strong contender, reinforcing its viability in real-world data despite its simplicity. Moreover, we observe both CAT and L-CAT not only holding up the performance with respect to their baselines models for all datasets, but in most cases they also improve the test accuracy significant manner. These results validate the effectiveness of CAT as GNN layer, and show the viability of L-CAT as a drop-in replacement, achieving good results on all datasets.
233
235
statistically
237
238
As explained in $4, CAT differs from a usual GAT in that the score computed with respect to the convolved features. Intuitively, this means that CAT should excel in those settings where nodes are better connected, allowing CAT to extract more information from their neighborhoods. Indeed, there exists a positive correlation between performance improvement and average degree of the graph.
1.5
239
Acc. Improvement (%))
240
1.0
241
0.5
242
243
0.0
In the inset figure, we can observe the improvement in accuracy of
-0.5
245
CAT with respect to its baseline model, as a function of the average
node degree of the datasets, and the linear regression of these results
10
20
30
Average node degree

---

Table 2: Test performance of the considered convolutional and attention layers on four OGB datasets, averaged over five runs. Bold numbers are statistically different to their baseline model (a = 0.05). Best average performance is underlined. Left table: accuracy (%); right table: AUC-ROC (%)
Dataset
arxiv
products
mag
proteins
GCN
71.58 0.20
74.12 + 1.20
32.77 = 0.36
80.10 0.55
GAT
71.58 = 0.16
78.53 + 0.91
32.15 = 0.31
79.08 + 1.47
CAT
72.14 0.21
77.38 + 0.36
31.98 = 0.46
73.26 + 1.65
L-CAT
71.99 = 0.08
77.19 + 1.11
32.47 = 0.38
79.63 + 0.71
GATv2
71.73 = 0.24
76.40 + 0.71
32.76 = 0.18
78.65 + 1,44
CATv2
72.03 = 0.09
74.81 + 1.12
32.43 = 0.22
74.33 0.94
L-CATv2
71.97 = 0.22
76.37 + 0.92
32.68 = 0.50
79.07 + 0.98
(dashed line). This plot includes all datasets (from the manuscript and Appendix D), and shows positive trend between node connectivity and improved performance by CAT.
248
249
6.3
Open Graph Benchmark
250
In this section, we assess the robustness of the proposed models, in order to fully understand their benefits. For further details and additional results, refer Appendix E.
251
252
Datasets. We consider four different datasets from the Open Graph Benchmark (OGB) suite [15]: proteins, producís, arxiv, and mag. Note that these datasets are significantly larger than those from
254
$6.2 and correspond to more difficult tasks, c.g., arxiy is 40-class classification problem (see Table for details). This makes them more suitable for the proposed analysis.
256
Experimental setup. We adopt the same experimental setup as Brody al. [5] for the proteins, products, and mag datasets. For the arxiv dataset, we use instead the example code from OGB [15], as yields better performance than that of Brody al. [5]. Just as in $6.2, we compare with GCN [18], GAT [30], GATv2 [5], and their CAT and L-CAT counterparts. We cross-validate the number of heads (1 and 8), repeat each experiment five times, and select the best-validation models during training. All models share the network architecture, number of parameters, and network initialization.
257
258
259
260
262
Results are summarized in Table 2, averaged over 5 runs. Here we do not observe a clear preferred baseline: GCN performs really well in proteins and mag; GAT excels in products; and GATv2 does well in arxiv and mag. Let us now focus on the proposed models. While CAT obtains the best results on arxiv, its performance on proteins and products is significantly worse than the baseline model. Presumably, an excessive amount of inter-edges could explain why convolving the features prior to computing the score is harmful, as seen in 1. As we explore in $6.3.2, however, CAT improves over its baseline for most proteins scenarios, specially with a single head. In stark contrast, L-CAT performs remarkably well, improving the baseline models in all datasets but products- even those which CAT fails demonttrtiing the adaptability L-CAT to different scenarios.
263
264
285
266
267
268
270
271 In order to better understand the training dynamics of the different models, we plot in Fig. 2a the test accuracy of GCN and the GATv2 models during training on the arxiv dataset. Interestingly, this plot shows that while all models obtained similar final results, CATv2 and L-CATv2 drastically improved their convergence speed and stability with respect GATv2, matching that of GCN. To understand the behavior of L-CATv2, we show in Fig. 2b the evolution of the parameters X1 Az. We observe that achieve these results, L-CATv2 converged to a GNN network that combines three types of
272
273
274
275
276
277
layers: the first layer is CATv2 layer, taking advantage of the neighboring information; the second layer is quasi-GCN layer, in which scores are almost uniform and some neighboring information is still used in the score; and the third layer is pure GCN layer, in which all scores are uniformly distributed. It is important to remark that these dynamics are fairly consistent, as L-CATv2 reached the same \ A2 values over all five runs.
278
280
7

---

ArXiv
1500 ArXiv L-CATV2
70
1.0
L1
Test accuracy
60
11, values
0.8
A2
50
GCN
40
GATV2
0.4
30
CATV2
0.2
L-CATv2
20
100
200
300
400
500
250
500
750 Epoch
1000
1250
Epoch
(a) Test accuracy.
(b) Evolution of X1, A2-
Figure 2: Behavior of GCN and GATv2 models during training on the arxiv dataset. (a) CAT and L-CAT converge quicker and are more stable than their baseline model. (b) L-CAT consistently converges to the same type of layers during training: CAT -quasi-GCN-GGGT network.
6.3.1
Robustness to noise
One intrinsic aspect of real world data is the existence of noise. In this section, explore the robustness of the proposed models to different levels of homoscedastic noise in the features. That is, attempt simulate scenarios where there exist measurement inaccuracies in the input features.
284
285
286
Experimental setup. For these experiments we consider the arxiv dataset, and the same experimental setup in $6.3. To simulate homoscedastic noise, introduce to the node features additive noise of the form x' = x + E, where & ~ (0, 1o), and where we consider different levels of noise, {0, 0.25, 0.5, 0.75 1).
288
specifically, take
290
Results can be seen the inset figure, which shows test accuracy as function of the feature noise level This plot summarizes performance of all considered models, over five runs and two numbers of heads (1 and 8). We can observe that baseline attention models exhibit high variance and are quite sensitive
70
GAT
292
CAT
68
L-CAT
66
294
to small perturbations. GCNs, instead, exhibit better robustness to noise and small variance. In concordance with the synthetic experiments (see $54 and 6.1), we observe that CAT is able to leverage convolutions as a variance-reet -rduucce-eeuitiin technique, boosting the performance of the attention mechanisms, and reducing their variance. Moreover, L-CAT proves to be strictly more robust than all other models: it boosts the performance and reduces the uncertainty-iiin CAT-and it is effective than other approaches as it can adapt the amount of attention used in each layer, outperforming competing methods.
64
296
62
60
298
0.00
0.25
0.50
0.75
1.00
(o)
300
302
303
6.3.2
Robustness to network initialization
304
Another important aspect of real-world applications that robustness to network initialization, ie., the ability to obtain satisfying performance independently of the initial parameters. Otherwise, a practitioner could waste lots of resources trying different initilizations or, even worse, give up on model just because they did not try the initial parameters that would yield great results. In this section, we test such scenario using the proteins dataset as an example setting.
305
308
307
308
Experimental setup. We follow once again the same setup for proteins in $6.3. We consider two different network initializations. The first one, uniform, uses uniform Glorot initilization [12] with a gain of 1, which is the standard initialization used throughout this work. The second one, normal, uses instead normal Glorot initialization [12] with gain of /2. This is the initialization employed on the original GATv2 paper [5] exclusively for the proteins dataset.
310
313
314
Results segregated by number of heads are shown in Table 3, while results for GCN appear the inset table. These results show that the baseline models perform poorly on the uniform initialization. However,
GCN
315
uniform 61.08 2.56
normal 80.10 0.55
317
this is somewhat alleviated when using heads the attention models. Moreover, all baselines significantly improve with normal initialization,
average
70.59 + 10.21

---

Table 3: Test AUC-ROC (%) the proteins dataset for attention models with two different network initializations (see $6.3.2), using head (top) and 8 heads (bottom).
GAT
CAT
L-CAT
GATv2
CATv2
L-CATv2
uniform
39.73 + 3.61
64.32
2.33
77.77 + 1.28
59.85 + 2.73
64,32 + 2.33
79.08 + 0.95
normal
66.38 + 6.94
73.26 + 1.65
78.06 + 1.25
69.13 + 8.48
74.33 + 0.94
79.07 + 0.98
uniform
72.23 2.86
73.60
1.14
78.85
1.57
75.21 = 1.61
74.16 +: 1.30
78.77
0.97
normal
79.08 1.47
74.67
1.15
79.63 +: 0.71
78.65 = 1.44
73.40 + 0.56
79.30 0.49
average
69.36 8.52
73.93 + 1.35
78.58 + 1.48
70.71 = 8.70
71.55 + 4.54
79.05 + 0.91
uniform- - A
uniform A2
normal A
normal A2 heads
0.75
Value
0.50
0.25
0.00
1
3
5
2
4 Layer
5
6
1
2
3
5
1
2
3 Layer
5
Layer
Layer
Figure 3: Distribution of λ1, A2 on proteins dataset for L-CAT across initializations.
319
being GCN the best model, and attention models obtaining 79 % accuracy on average with 8 heads. Compared to the baselines, CAT does a good job and improves the performance in all cases except for normal with heads. Remarkably, L-CAT consistently obtains high accuracy in all scenarios and runs. This can be further appreciated by looking at the average accuracy (bottom row), showing that L-CAT is clearly more robust to parameter initialization than competing models.
320
321
323
324
To understand this performance, we inspect the distribution of \1, A2 for L-CAT in Fig. 3. Here, we can spot few interesting patterns. First, the first and last layers are always GCNs, while the inner layers progressively admit less attention. Second, the number of heads affects the amount of attention allowed in the network; the more heads, the more expressive the layer tends to be, and the
325
326
327
328
more attention that is permitted. Third, L-CAT adapts to the initialization used: in uniform, it stays competitive by allowing more attention in the second layer; normal, allows more attention in the score inputs. Table 3 and Fig. 3 thus consolidate the effectiveness and flexibility of L-CAT.
329
331
7
Conclusions and future work
In this work, studied how to combine the strengths of convolution and attention layers in GNNs. We proposed CAT, which computes attention with respect to the convolved features, and analyzed its benefits and limitations
333
new synthetic dataset. This analysis revealed different regimes where one model is preferred over the others, reinforcing the idea that selecting between GCNs, GATs, and now CATs, is difficult task, their performance directly depend on the data. For this reason, proposed L-CAT, a model which is able to interpolate between the three via two learnable parameters. Extensive experimental results demonstrated the effectiveness of L-CAT, yielding great results while heing more robust than other methods due to its adaptability. As result, L-CAT proved to be a viable drop-in replacement that removes the need cross-validate the layer type.
335
337
338
339
340
341 We do not consider this work adds any societal concerns. On the contrary, L-CAT eases the applica- bility of GNNs to the practitioner, and removes the need of cross-validating the layer type, which can potentially benefit other areas and applications, as GNNs have already proven.
343
344
We strongly believe learnable interpolation can get us long way, and we hope L-CAT to motivate new and exciting work. For example, would be interesting to see L-CAT applied to other GCN and GAT variants, such as those in [17, 28, 35]. Specially, we are eager to see L-CAT in real applications, and thus finding out what combining different GNN layers across a model (without the annoyance of cross- cross-validan validating all combinations) can lead to in the real-world.
345
347

---

References
349
350
[1] T.W. Anderson, An introduction to multivariate statistical analysis. John Wiley & Sons, 2003.
[2] Aseem Baranwal, Kimon Fountoulakis, and Aukosh Jagannath. Graph convolution for semi- supervised classification: Improved linear separability and out-o--distriuution generalization. In International Conference on Machine Learning (ICML). PMLR, 2021.
352
354
[3] Aseem Baranwal, Kimon Fountoulakis, and Aukosh Jagannath. Effects of graph convolutions in deep networks. arXiv preprini wwXiv:224.009997 2022.
355
356 [4]
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanzhez-Gonzalez, Vinicius Zambaldi. Mateusz Malinowski, Andrea Tacchetti, David Raposo. Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806. 02006, 2018.
357
358
359
[5]
Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? In International Conference on Learning Representations (ICLR), 2022.
381
362
[6] Dan Busbridge, Dane Sherburn, Pietro Cavallo, and Nils Y Hammerla. Relational graph attention networks. arXiv preprini arXiv:
363
2019.
364
[7] Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličković. Principal neighbourhood aggregation for graph nets. Advances in Neural Information Processing Systems (NeurIPS). 33, 2020.
385
386
[8]
Yash Deshpande, Subhabrata Sen, Andrea Montanari, and Elchanan Mossel. Contextual stochastic block models. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018.
368
370
99
Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. arXiv preprint 20XX::2012200699, 2020.
371
372
[10]
Kimon Fountoulakis, Amit Levi, Shenghao Yang, Aseem Baranwal, and Aukosh Jagannath. Graph attention retrospective. arXiv preprint arXiv:2202.1 2022.
373
374
[1]] Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning (ICML), 2017.
375
377
[1] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2010.
378
379
380
[13]
Will Hamilton, Zhitao Ying. and Jure Leskovee. Inductive representation learn- ing on large graphs. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017. URL https:
381
382
383
file/
Paper.
384
[14]
William L. Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: Methods and applications. IEEE Data Eng. Bull., 40, 2017.
386
15
Weihua Hu, Matthias Fey, Marinka Zitikk Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in Neural Information Processing Systems (NeurIPS), 33:22118 10022218-221333 2221133, 2020.
387
388
389
[16]
Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer. In The Web Conference (WWW), 2020.
380
391
17]
Dongkwan Kim and Alice Oh. How to find your friendly neighborhood: Graph attention design In International Conference on Learning Representations (ICIR), 2021. diw.miidwiiiiwwwttt..
382
with self-supervision
393
URL httpp:://ooto http:://///eeeeeeeeeeeeeeeeee
394
[18] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.
385
396
[19] Boris Knyazev, Graham W Taylor, and Mohamed Amer. Understanding attention and gen- cralization in graph neural networks. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
387
398
[20] Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létorrnaa, and Prudencio Tossou. Rethinking graph transformers with spectral attention. Advances Neural Information Pro- cessing Systems (NeurIPS), 34, 2021.
400
10

---

402
[21]
John Boaz Lee, Ryan A Rossi, Sungchul Kim, Nesreen K Ahmed, and Eunyee Koh. Attention models in graphs: survey, ACM Transactions on Knowledge Discovery from Data (TKDD), 13, 2019.
403
405
[22] Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen,
406
Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In AAAI conference on artificial intelligence, volume 33, 2019.
407
408
[23]
Rigollet and J.-C. Hütter. High dimensional statistics. Lecture notes for course 18S997, 813:
8 814, 2015.
[24]
Benedek Rozemberezki, Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding. Journal of Complex Networks, 2021.
411
[25]
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions neural networks, 20, 2008.
413
414
[26]
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-
Rad. Collective classification in network data. Al magazine, 29(3), 2008.
[27]
Oleksandr Shehur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv: /1110//88000000 1.05868, 2018.
417
418
[28]
Kiran K Thekumparampil. Chong Wang. Sewoong Oh, and Li-Jia Li. Attention- graph neural network for semi-supervised learning. arXiv preprint arXiv: 03735, 2018.
419
420
[29]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
422
425
[30] Petar Veličković, Guillem Cucurull, Aranixa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations (ICLR), 2018.
424
425
426
[3]] Guangtao Wang. Rex Ying, Jing Huang, and Jure Leskovee. Multi-hop attention graph neural networks. In International Joint Conference on Artificial Intelligence (IJCAI), 2021.
427
428
[32] Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. Heteroge- neous graph attention network. In The World Wide Web Conference (WWW), 2019.
[33] Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. Graph neural networks in recom- mender systems: a survey, ACM Computing Surveys (CSUR), 2020.
431
432
[34] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32, 2020.
433
434
[35] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations (ICLR), 2019.
436
437
[36]
Jiaxuan You, Zhitao Ying, and Jure Leskovec, Design space for graph neural networks. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020.
[37] Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J Kim. Graph transformer networks. Advances Neural Information Processing Systems (NeurIPS), 32, 2019.
440

---

Checklist
442
443
1. For all authors...
444
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's s contributions and scope? [Yes] In both, the abstract and introduction, we faithfully reflect the contributions of this work: i) the introduction of new GNN layer, CAT; ii) theoretical analysis on a new synthetic data model; and iii) the introduction of L-CAT, a model that is capable of learning to interpolate between GCN, GAT and CAT.
445
447
(b) Did you describe the limitations of your work? [Yes] Throuhgout all the manuscript we discuss on the strenghts and limitations of the considered models. As of L-CAT, we discuss possible extensions (and thus current limitations) in the future work.
450
452
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Socictal impact of our work is mentioned in the conclusions.
453
454
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
495
456
2. If you are including theoretical results...
457
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We describe the data model used for all our theoretical results in $4 and Appendix A
458
459
(b) Did you include complete proofs of all theoretical results? [Yes] All proofs can be found in Appendix A
461
3. If you ran experiments...
462
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] We include in the supplementary material the necessary anonymized code and scripts required to reproduce our experiments. All required datasets are freely available.
463
464
485
486
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Complete details about the experimental setup can be found in Appendices B, D and E
467
468
469
(c) Did you report error bars (c.g., with respect to the random seed after running exper- iments multiple times)? [Yes] In all our results we report the mean and standard deviation computed using five trials or more. In addition, we highlight in bold the results that statistically significant.
471
472
473
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We include complete details of the experimental setup as well as computational resources used for the three sets of experiments in Appendix B, Appendix D and Appendix E respectively.
474
475
478
477
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
478
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite the baseline models [5, 18, 30J, the creators of the datasets we use [15, 27], and the programming framework we use to run our experiments [15, 36]
479
480
481
(b) Did you mention the license of the assets?[N/A]
482
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] We provide the code needed for running the experiments in the supplementary material.
484
(d) Did you discuss whether and how consent was obtained from people whose data you' using/curating? [N/A] Datasets are publicly available at the Torch Geometric or the Open Graph Benchmark frameworks.
485
486
487
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] The datasets used do not contain personally identifiable information nor offensive content.
488
489
490
5. If you used crowdsourcing conducted research with human subjects...
481
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
492
http:://wccaggeeercccccccceee/ee "acrps://ogb.staaford.de
12

---

493
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
494
495
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "hHTJVXZFbZW", "reviewer": "Reviewer_8bmS", "summary": "The work developed by the authors in the paper consists in a design of a novel convolutional operation to achieve learning on graph data. This convolutional operation lies at the crossroad of \"isotropic\" GCN and \"anisotropic\"/ attentional GNN to take the advantages of both operations without increasing the complexity of the operation.\nIn this study, appropriate datasets and baselines are introduced to support the theoretical motivation and the robustness of the proposed operation. ", "questions": "1/ Is the idea of combining GAT and GCN could be generalized to any pairs of convolutional operations on graphs ? or the proposed framework is restricted to these two convolutional operations\n\n2/ Can you shed light on the fact that the proposed combination doesn't interfere with any recent convolutional operations design ?\n\n3/ Could you compare (table 1, table 2) with recent convolutional operations ? it's a good starting that L-CAT outperforms GAT and GCN, however, one may wonder if it the case w.r.t to other convolutional operations.", "limitations": "The limitations are fairly discussed and the work does't have any societal concern.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This work is a novel combination of two well known convolutional operations on unstructured data.  The combination is well motivated / studied methodologically and empirically. It can bring a value for a better understanding of learning on graphs and might give insight for the design of more expressive convolutional operations on graphs. \nThe authors studied theoretically the expressive powers of GCN and GAT while identifying their failures. Based on the latter, their combination is proposed in a straightforward manner while showing a gain in a performance without substantial increase of the complexity of GNN architectures. Moreover, their architecture is less sensitive to noisy data compared to GCN and GAT while being quasi-agnostic to the weights initialization which add significant value to the work. This direction could be interesting to explore further towards invariant architectures w.r.t weight initializations to make the networks more robust and get rid-off careful choice of initialization which is a tedious engineering task. \n\nThe related work is well studied and the experiments are compared with appropriate GNN models. Moreover, the paper is well structured and easy to follow, including the consistency of math variables. \n\nI argue that the work is technically correct where the claims are well motivated theoretically and validated empirically with an appropriate experimental protocol. However, l am not sure to find some  mathematical parts developed  in the supplementary necessary to understand the contribution of the paper. Namely line 477 and 519.\n\nOne may wonder if the proposed framework can be generalized to any convolutional operations on graphs and to what extent it is possible or it is restricted to combining only GAT and GCN.  As a consequence, it's not clear if the proposed combination doesn't interfere with recent convolutional operations, including the seminal works on the design of convolutional operations from dynamical systems, partial differential equations and differential geometry standpoints. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VufhW6ZFcQW", "reviewer": "Reviewer_QHuE", "summary": "This submissions has two parts of contribution. \n\nFirst, it proposes a novel attention mechanism for graph neural networks. The proposed technique first aggregates neighborhood features (as done in GCN) before computing the attention weights. This new method is named CAT. By a theoretical, analysis, the authors show that CAT can be more effective than GAT but their comparison depends on the graph topology. \n\nSecond, for a given layer, the submission proposes to learn two additional scalar weights that interpolates between GCN , GAT, the newly proposed CAT. The intuition here is that the network can discover the correct network type to use depends on the input data. \n\nOverall, I think the proposed method is novel and the theoretical analysis provides an interesting perspective. The empirical analysis shows that the proposed methods are more robust to data noise and weight initialization scheme. However, in the standard setting, the proposed method is mostly on-par with GCN (the comparison is not consistent across datasets). To take this empirical weakness into account, I am giving a weak accept.", "questions": "Questions:\n\n1. As the authors acknowledged, many new variants of graph neural networks have been proposed. Can the authors include more baselines into comparison to contextualize the results better? I am especially interested in knowing how other baselines would react to the added feature noises. \n2. Can the theretical analysis  also be applied to L-cat? Similarly, how would L-CAT behave on the synthetic data in Figure 1?\n3. Another form of data noise is corruption to the graph adjacency matrix. Does L-CAT also handle this form of corruption better?", "limitations": "I do not see any negative societal impact. ", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n1. The proposed method is novel\n2. The proposed method is motivated by a theoretical analysis. I appreciate the effort to understand the proposed method.\n3. On the empirical side, the proposed method is more robust to data noise and different initialize schemes\n4. The writing is very clear\n\nWeakness\n\n1. For the standard benchmarks, the proposed method is only on par with GCN, which is a simpler method. \n2. I provide more requests and suggestions for improving the current draft in the questions section below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "K_o_32Cfw-f", "reviewer": "Reviewer_edc7", "summary": "The authors propose a new graph neural network by combining existing graph convolution networks: GCNs and GAT. The authors proposed a new learnable convolutional attention layer (L-CAT), which subsumes the graph convolution in GCNs, the graph attention layer in GAT, and a normalized attention layer (CAT).", "questions": "1. The experiments are somewhat obvious. The proposed method subsumes GCN, GAT, CAT. Then L-CAT must be on par with the models or outperform them.\n2. In Table 1, L-CAT and CAT are comparable. The performance gain is marginal compared to GAT. Unless I missed something, both GCN and GAT are baselines of the proposed method.\n3. In the literature, more powerful GNNs have been recently studied. Why do the authors compare their method with GCN?", "limitations": "1.  The proposed method is a simple generalized graph neural network that subsumes GCN and GAT layers. The simplicity of the proposed method is great as long as it has practical merits. However, the performance gain by the proposed method is marginal. In addition, the authors did not compare the proposed method with more recent works that exhibit stronger performance.\n2. The proposed method is incremental, and the lack of generality of the proposed method makes the impact of this paper questionable. ", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. A new layer that subsumes a couple of popular graph convolution layers has been proposed. Without changing layers in GNNs, by optimizing parameters in the proposed layer, the proposed graph neural network can change its behavior. \n2. This paper theoretically discussed the expressive power of GAT and the proposed layer CAT with a contextual stochastic block model (CSBM). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1NtVW5l67_t", "reviewer": "Reviewer_CqLh", "summary": "In this paper, the authors first investigate the expressiveness of convolutional-based and attention-based GNN architectures under the CSBM setting, and propose graph convolutional attention (CAT) which hinges the attention with convolution. Knowing there is no clear winner between attention and convolution, the authors propose a technique to interpolate in the middle of all the three mechanisms, called L-CAT.", "questions": "- In “Graph Attention Retrospective”, the authors considered two-class CSBM. Is there any special reason that this paper considers three classes?\n\n- CAT computes the attention score based on the neighborhood information of the two nodes. Does this mean CAT actually cheats using second-hop information? And naturally improve the expressiveness? Would it be more fair to compare CAT with two-layer GAT or GCN?\n", "limitations": "The analysis in this paper is fundamental and provides good insights. However, the major limitation lies in the performance on the benchmark datasets. According to the theoretical results in this paper, L-CAT generalizes GCN and GAT and learns to trade off between these two architectures. However, the final performance on OGB shows even GCN outperforms L-CAT on two datasets. Moreover, current SOTA models faraway outperform GCN and GAT. Trading off between GCN and GAT seems not a key to improve the performance.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ The paper is clearly written, well formulated, and easy to follow.\n\n+ Both theoretical and empirical analysis on the separate power of attention using CSBM seems interesting, though largely followed from the “Graph Attention Retrospective” paper.\n\n+ The ideas of combining and interpolating between convolution and attention are novel proposals and well-motivated.\n\n+ The empirical evaluation justifies the theory on CSBM, demonstrates performance on both real data and benchmark, and presents many insightful visualization and ablation studies e.g., the learning curve and distributions of tunable parameters, robustness to initialization and noises.\n\nWeaknesses:\n\n- The theoretical analysis is based on toy settings. And the theoretical significance is not claimed very clearly given the “Graph Attention Retrospective” paper. Also the separation power of convolution-based architecture still remains obscure without a formal argument.\n\n- The performance on the benchmark is not as good as expected. GCN and GAT seem to be a special case of L-CAT. However, L-CAT fails to outperform GCN on proteins and mag data. Also, lots of strong baselines are missing.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject