Self-Attention in Colors: Another Take on Encoding Graph Structure in Transformers
Authors: Romain Menegaux
Emmanuel Jehanno
Margot Selosse
Julien Mairal
Venue: Accepted by TMLR ()

Abstract: We introduce a novel self-attention mechanism, which we call CSA (Chromatic Self-Attention), which extends the notion of attention scores to attention _filters_, independently modulating the feature channels. We showcase CSA in a fully-attentional graph Transformer CGT (Chromatic Graph Transformer) which integrates both graph structural information and edge features, completely bypassing the need for local message-passing components. Our method flexibly encodes graph structure through node-node interactions, by enriching the original edge features with a relative positional encoding scheme. We propose a new scheme based on random walks that encodes both structural and positional information, and show how to incorporate higher-order topological information, such as rings in molecular graphs. Our approach achieves state-of-the-art results on the ZINC benchmark dataset, while providing a flexible framework for encoding graph structure and incorporating higher-order topology.

Published in Transactions on Machine Learning Research (10/2023)
Self-Attention in Colors: Another Take on Encoding Graph Structure in Transformers
Romain Menegaux
romain. umaiii.meneouuuhhhrrrr..f.
Unic. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK 38000 Grenoble, France
Emmanuel Jehanno
Unio. Crenoble Alpes, Inria, CNRS, Crenoble INP, LJK 38000 Grenoble, France
Margot Selosse
margol..ecosse@@@mii..com
work done while at Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK 38000 Grenoble, France
Julien Mairal
Sulieen.maral@innri....
Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK 38000 Grenoble, France
Reviewed on OpenReview: https: // openreview. net/ form? id= 3dDCNggu2d
Abstract
We introduce novel self-attention mechanism, which call CSA (Chromatic Self-Attention) which extends the notion of attention scores to attention filters, independently modulating the feature channels. We showcase CSA in fully-attentional graph Transformer CGT (Chromatic Graph Transformer) which integrates both graph structural information and edge features, completely bypassing the need for local message- paasggg-pssinggggg components. Our method flexibly encodes graph structure through node-node interactions, by enriching the original edge features with a relative positional encoding scheme. We propose new scheme based on random walks that encodes both structural and positional information, and show how to incorporate higher-order topological information, such as rings in molecular graphs. Our approach achieves state-of- of-the-art results on the ZINC benchmark dataset, while providing flexible framework for encoding graph structure and incorporating higher-order topology.
1 Introduction
The field of graph representation learning and graph-structured data has seen significant growth and maturity in recent years, with successful applications in a wide range of domains such as pharmaceutics, drug discovery (Gaudelet al., 2021), recommender systems (Ying al. 2018), or navigation systems (Derrow-Pinonn et al. 2021).. Key to these advances has boen the success graph neural networks (GNNs), which up until 2021 took mostly the form of local pussage-passingg neural networks (MPNNs). In addition, the field has benefited
from the development of fiurry of high-quality benchmark datasets (Dwivedi et al. 2020 Hu et al. 2020
2021
Dwivedi et al. 2022b
Local MPNNs have been known to suffer from limitations such as over-smoothing (Oono & Suzuki 2020
bottlenecks information propagation (Alon & Yahav 2021 and limited expressiveness (Xu et al. 2019
Morris et al.] (022)). This issue is alleviated by (i) adding positional or structural encodings to node features
Bouriisas al. 2020 Abboud al. 22021 Dwivedi et 2022a Eliasof al.) (022)) and (fi) softly decoupling
the message passing flow from the graph structure, by enhancing the graph with virtual nodes or higher order

---

Published in Transactions on Machine Learning Research (10/2023)
structures meaningful ones such rings for molecules (eey et al random subgraphs (Zhao et al. 2022), leading to forms of hierarchical message- paaggge-ppaggg......
2020
Bodnar et al. 2021a simply
Another way to overcome these limitations is a whole avenue of work on graph transformers (GT) (Ying ct al.
2021 Mialon et al., 2021 Kreuzer et al. 2021 Park et al. 2022 Hussain et al.) 2022 Rampášek et al,
2022
Global attention GTss perform dense peessage-passingg where all nodes communicate with each other.
Recently these have surpassed MPNNs on large-scale datasets (Ying al.) 2021 Park et al.) 2022] Hussain
et al. 2022 Rampášek et al., 2022). However they not yet competitive smaller datasets and tasks that are highly dependent on local substructures (Kreuzer et al. 2021 Rampášek et al., 2022). In this work, we develop novel self-attention mechanism which - when coupled with expressive node interaction features are competitive on benchmark datasets of all scales.
The output of the standard Transformer self-attention mechanism (Vaswani et al. 2017 is invariant to permutations. This issue mostly solved for sequences by concatenating positional encodings the element features. The natural equivalent for graphs are positional encodings based on the eigenvectors of the graph Laplacian (Dwivedi et al. 2020 Kreuzer et al. 2021). Albeit theoretically justified, in practice they are outperformed by other structural encodings specially tailored for graphs @Dwivedi et al. 2022a). Furthermore, even toate-of-theearr node-wise positional encodings alone are not enough to bridge the performance gap between GTTs and MPNNs (Rampášek et al. 2022 Furthermore edge features which describe semanlic relationships between nodes, such as the type of chemical bond in molecules can be essential for valid graph representations, yet there is no standard or clear-cut way to incorporate them in GTBs In this work we give an answer the question:
How to erpressivcly incorporate both graph structural information and edge features in graph Transformers?
Some recent GTss delegate this task to a local MPNN module (Wu et al. 2021 Chen et al 2022 Rampášek et al. 2022), but otherwise all standalone GTss directly bias the attention matrix with relative positional encodings (Shaw et al.) 2018 of different natures, enginecring in both edge features and the adjacency matrix. We propose a unilying framework for the self-attention layer that flexibly subsumes most of the GT-flavors aforementioned.
Contributions
We eurich this framework by introducing Chromatic (or Channel-wise) Self-AAttetiion (CSA): selectively modulating message channels, damping or highlighting them based on the relative positions of two nodes.
novel relative positional encoding scheme for graphs, based on random walks. We inject them in CSA and reach state-of-the-art results on several graph benchmarks.
We showcase the generality of our framework by scamlessly integrating higher-order graph topological features such as rings. This augmented version CSA-rings greatly improves the state-of-the-arl on the ZINC molecular dataset.
We show in rigorous ablation study on set of graph benchmarks the added benefits of our proposed methods.
2 Setting and Related Work
2.1 Graph neural networks
Regular graph neural networks (GNNs) consider graph as set of node features X E R Nuoder xd. an adjacency matrix A E {0; Vooden x NNoder and (optionally) a set of edge features E E Natgmad xd The information contained in the adjacency matrix A describes the connections between nodes and is henceforth referred to as the graph structure.
GNNs node representations (initialized by (0)) X) are iteratively updated in layers, by receiving and aggregating messages M from other nodes. The edge features can also be updated, which do not write
2

---

Published in Transactions on Machine Learning Research (10/2023)
here.
h (B+1) < AGG {M (n.7. 26) m0)}
On the one hand, in local message passing networks (local-MPNNs) the aggregation spans only the neighbors of node The types of local-MPNNs (Kipf & Welling. 2017 Bresson & Laurent 2017 Corso et al. 2020 to cite just a few of the most popular variants mostly differ on how the aggregation is performed. On the other hand, Graph Transformers (GTs), which can be interpreted as dense MPNNs, aggregate messages from all nodes together.
2.2 Transformer Architecture
In vanilla transformers, messages are linear transformations of node representations the node values and the aggregation is a simple weighted sum:
Nuodes
h4 4- 14 +
Σ j=1
V.
The coofficients in this weighted sum, the attention scores ā(i.j) E [0, 1], are obtained by softmax normalization such that ā(i.j) a(t,))/ Σx a(i, k) with
a(i,j) = exp (2)0:-1)
(3)
= Wkh; V = Wyh are commonly named the query, key and value matrices, respectively. In the multi-head sctting, Nheads separated attention scores are computed, modulating the values V by blocks of size d/Nnoads.
From now on, omit the normalization term Va Q Woh; K=
In this basic formulation, the update rule is invariant to permutations in the input nodes, and in fact oblivious to the graph structure and edge features Egj. It is therefore crucial to include these, either by encoding and injecting them in the node fratures, or by tweaking the update rule.
2.3 Encoding graph structure
In this section, we give a brief overview of existing methods to encode graph structure and edge features in GTs.
Local-MPNN hybrids One natural way to include the graph structure is simply to mix in local-MPNN layers, by either interleaving (Rampášek et al. 2022) stacking (Wu et al. 2021 Chen et al.) 2022 them with dense Transformer layers. The graph structural information, well as the edge features, are handled by the local components, conveniently leveraging the rich lilerature on local MPNNs. In this work however we restrict ourselves to purely attentional networks, for which appropriate positional encodings We briefly go over the different flavors of graph PEs the literature, see (Rampášck et al. 2022 for a more systematic overview and a thorough categorization. We adopt their distinction of structural and positional encodings.
(PE) are needed.
Node structural encodings (SE) Most successful node encodings for graph rely on local structural information, such as centrality measures (Ying et al. 2021] or local substructure counts (Bouritsas et al. 2020 One such encoding, related to the one we will introduce in section 3 node-RWSB (Dwivedi et al. 2022a
is based on random walks. Instrumental to the success of GTss they have also shown to improve performance and expressivity local MPNNs. However they cannot fully convey the relative spatial information in a Transformer.
For the sake of intuition, take the key and query matrices to be identity, and suppose have concatenated positional encoding p to the features h t h || p. The attention score can then be written as the product of semantic part and a positional one:
a(i,j) exp (h; hg) exp (pr pz)
3

---

Published in Transactions on Machine Learning Research (10/2023)
The original cosine positional embeddings p designed for sequences have the desirable property that p; Pj is function of the relative distance between i and j: l j]- For most of the successful graph node encodings presented, these inner products can at most represent structural similarity iee. how the local neighborhoods of nodes and j compare to each other. They are independent of the distance between and j within the graph, whereas the model could benefit. from this information, for instance by attenuating signals coming from far away nodes.
Node positional encodings (PE) This distance-encoding role could be fulfilled by spectral positional encodings based on the eigenvectors of the graph Laplacian. These have been adapted into node-PE Dwivedi et al. 2020 Kreuzer et al.) 2021). but suffer from implementation difficulties such as their invariance to sign flip of the eigenvectors, in practice hindering their performance (Dwivedi et al. 2022a). SitnNet [Lim
et al. 2022 propose to learn these encodings via a sign-invariant GNN.
Regardless of their particular type, current node-wise positional encodings are helpful addition but do not suffice their own make global attention even remotely competitive with sparse cresage-paasiing Ramppáek et al. 2022).
Relative positional encodings (RPE) Each flavor of graph transformer comes with its own way of further incorporating the graph structure. One the first graph transformers, SAN (Kreuzer et al. 2021 learns two different sets of attention parameters: one for adjacent pairs, the other for non-connected nodes. Graphormer (Ying al. 2021) adds a learnable bias bo(t,) to the attention score a j) = exp (Q; K2 + bo(k))), indexed the shortest path length o(i,j) between and j. GraphiT (Mialon al. 2021 bias the attention matrix with graph kernel K, encoding the structural similarity: a(i.)) exp (Q₂ K,) K,(i,j). The choice of the kernel K, correspond to imposing different inductive biases. For instance K; (i,j) exp (pr Pÿ) with one of the structural encodings mentioned above would encode structural similarity. On the other hand choosing the heat diffusion or the PageRank kernel as K, would encode a notion of walking distance between nodes. If Kr is chosen as the adjacency matrix or the 1-step random walk then GraphiT becomes a local-MPNN, very similar to GAT (Veličković et al.] 2018).
What propose is an extension of GraphiT, and a generalization of Graphormer, where the allention mask K, is learned based on structural edge encodings.
3 Method
3.1
Unifying framework
Most of the self-attention modules of the aforementioned GTs can be rewritten as in equation with ey a scalar encoding the relationship between nodes and j-
a(i.j) = exp (Q. K2 + eig)
E R,
(4)
By choosing eij bo(t,) we recover Graphormer (the expression for eig is slightly more complex in the presence of bond features, we do not write it here), and with eyj log (og(K,(i,,)) we replicate GraphiT. In another recent work GRPE (Park et al. 2022 reeuse the query and key vector to compute the RPE bias eiy Q E Kg- E
Note that if eij can be factorized as an inner product ps Py (or more exactly as WOP: WKP;), we recover the standard Transformer with positional encodings p:.
To the best of our knowledge, this framework encompasses all previously proposed GTs, with the exception of EGT (Hussain et al.] (022), who scale the attention score after the fy-normalization ā(i.j) with a log- normalized gating term on edge features af ((E log(1 + Σx ((E Another recent GT (Kim al. 2022 falls out of the scope of this framework entirely, by treating both nodes and edges separate tokens.
4

---

Published in Transactions on Machine Learning Research (10/2023)
Updated Node Encodings ht
Node Encodings /
Linear
Messages
Node features
Positional Encoding
h
10++1)
Attention filters
al K+ B
Σ EV)
Attention matrix per channel
Node prior
Linear gat
Linear F
Embedding g
Edge prior information
Example w/ rings
Graph G
Edge features
Structural Encodings
Topology Encoding
Figure 1: Chromatic Graph Transformer. (a) Graph features are preprocessed into node features h (top-left) and two edge feature matrices Eatt and E E TR Nuedes x Nuodes xd (bottom). (b) These are input to the CSA layers, which iteratively update the node representations h. (c) These representations are to a classification or regression head (not shown herc).
Monochrome attention In the coarsest form of our proposed model, CSA-mono, ei is simply a scalar projection of the initial edge features Giy = WE E This is the "black and white", or mono-chromatic version of
3.2 Chromatic Aaff-Attention
Attention filters In the self-attention formulation given above by 4 and 2), the entirety of the node-node interaction both their structural and semantic relationship is contracted into single scalar a(i,j) (resp. Nheads scalars in multi-headed attention) which then modulates the node message Vj.
We refine this scheme by making the attention scores multi-dimensional,, allowing separate attention score (i per output channel d1, generalizing the scalar attention score into an allention filter j(i,j) E Rd
a al,)) exp (Q: K, Eij)
e Rdd
(5)
L d-noommalzaaion is then performed across each channel
a (i, = j) > a(i,k)
These filters are applied to the messages V via an element-wise product.
Nuodes
h4 h
a(i,j)oV
(6)
This allows selectively attend to, or dampen certain channels in the incoming messages. Note that the two formulations (4) and (5) are equivalent if we impose Eg be a constant vector, piccewiso-constant vector in the meati-head setting constant the channels of each head.

---

Published in Transactions on Machine Learning Research (10/2023)
We emphasize that (6) comes enhancement over standard multi-headee attention, and is notably different from using multi-head version of with Theads Mcchanncss-
This is explained in detail in Appendix B
Enhancing messages Most MPNNS that handle edge features combine them with the node features to form the message Mh;, E,g) with M being a simple function such as concatenation, addition or elementwise product (Hu* al., 2020 Corso al. 2020p. Similarly, we also enrich the individual node messages with relational information, by adding an edge value vector Eij to the node value vector Vj, the odge values being
lincar transformation of the edge features E
Wev E
Nuodes
h4 + h4 +
a(i,j) O (V, + big))
Note that the update rule opens up other possibilities for the attention filters, such as additive attention [aaddanau et al. 2015 a(i,j) exp (Qi + Kj + Bsy) instead, completely foregoing the need for separate attention heads.
3.3 Design of the edge features
Fundamental to the success and expressive power of this scheme is the choice of these initial edge features.
Semantic edge features After transforming the original or semantic edge features into suitable vector form Ebond g. xd we extend them to node-pairs that were not originally connected in A, by introducing two learned embeddings Rd one for self-connections Eself (if not already given), the other for non-connected nodes Eu-c.
Eboud sj
i and j are connected
bbond
Fself
if if
E
otherwise
To encode both semantic and st structural information in the final edge features E we concatenate them with relative positional encoding ERPE,
E
Elond
ERPE.
E.
Here we choose concatenation but any simple function such as addition or multiplication should yield similar results.
This system is general and flexible enough to leverage all possible pairwise relations in graphs that can be encoded onto vectors.
Choice the relative positional encoding For the RPE ERPE, we draw upon the family of functions of the successive powers of the random walk matrix RW = D-1 A (where D is the diagonal degree matrix):
E & (RW)).
E RR,
(10)
With an appropriate choice of T these constitute powerful descriptors of the graph relations Li et al. 2020
Indeed, one can recover positional information the index of the first non-zero component is the shortest path distance (SPD) but also structural information. For instance RW is nonzero iff and j are neighbors and, in the absence of self-connections, RW3 i is nonzero iff is part of a triangle.
We give two possible RPEs constructed from these features. One of them SPDE present in some form in other GTs (Ying et al. 2021 Park et al. 2022 Hussain al. 2022). The other RWSE commonly used a nodo-wise SE but has our knowledge never been used before as an RPE.
Shortest Path Distance Encoding (SPDE) The shortest path distance encoding is obtained by choosing T as a d-dimensional embedding of the index of the first nonzero component:
ESPDE = WSPDE
(11)

---

Published in Transactions on Machine Learning Research (10/2023)
with WSPDE Raxp the embedding vectors and (o(.)) R2 the one-hot encoding of the shortest path length j), Similarly to the semantic features above, we complete the feature matrix with a special embedding for nodes unreachable in less than p hops, and another one for self connections.
(a)
(b)
Random Walk Structural Encoding (RWSE) In our proposed RPE, & is simply matrix multipli- cation with learnable weights WEWSE:
Figure 2: Shortest Path Distance Encoding can- not distinguish the relationship above graphlcts, as they are both neighbors with (i.j) = 1. On the other hand the three first coordi- nates of the RWSE; vectors are markedly different: [0.33, 0.33, 0.26] and [0.5, 0.0, 0.63] for the graphlets on the left and right, respectively.
in the two
RWSE WRWSS. [RW] sy....,, RW3] Dig
(12)
If the weights matrix WE WRWSE is constrained to be non-negative, these features can be considered as re-weighted diffusion kernels between nodes, hence constituting a learnable and multi-dimensional ex- tension of GraphiT. We argue that RWSE is richer RPE than SPDE as it also preserves some local structural information. A simple illustrative example is shown in Figure 2
3.4 Adding substructures
Incorporating higher-order structure information can be beneficial to performance. In molecular graphs for example, atoms in ring (a chordless cycle) share covalent electrons, which influence their distribution in space and their chemical properties. Hierarchical message-passing schemes (Fey et al. 2020 Bodnar et al.) 2021b a were designed both to propagate information effectively along meaningful substructures and to suitably modulate the messages. We emulate this form hierarchical message- message-passing in CSA by injecting specific topological information directly into the edge features (and hence in both the attention matrix and the messages).
For chosen set of structures (graphs) 8 we define binary relation 3 nodes of graph in definition 3.1 3 is symmetric but not necessarily reflexive or transitive.
Definition 3.1. Given a set of structures S, two nodes i and j of a graph G are S-bound written ig if and only if there exists an element S E S and a subgraph of G containing both and such that G is isomorphic to S.
ig = ES E e
Sub(G)
(13)
s.t.i.j E & and S~G
We then design boolean features based on this relation c = 1ii j). In the more generic form of our topology encoding, which can be applicd any type of data, these fcatures are embodded and then added the edge features:
Fij Faj Embed(e
In the cases where the original edge features Eboud are calegorical, i.e. belong to discrete set C, one give more degrees of freedom to the substructure encoding by forming the Cartesian product of the semantic and the topology features, effectively doubling the number of edge types.
D Embed ((blond e()))
Note this could also be done on the SPDE positional encoding for instance. Although this choice of encoding is less easily applicable (due to the restrictive condition on the edge features) it is also more expressive, and yields slightly improved results empirically.
In practice we have implemented this topology encoding for rings molecular graphs, by choosing as the set of rings up to certain length, but the same could be done for any type of substructure deemed relevant. to the task at hand.
7

---

Published in Transactions on Machine Learning Research (10/2023)
3.5 Discussion and implementation
Model architecture The main components of our architecture are illustrated in Figure 1 As is standard in Transformer layers, our CSA layer is composed of the self-attention block followed by 2 linear layers with batch normalization. We stack L of these layers which yield final representations h(2), These are fed to a prediction head that depends on the task ggraph/node/eett
(lasificatiou/reeressiion
h(++1) CSA(h E
( = EdgeEncoder (L boodd .A)
h (0) NodeEncoder(X,
Regularization Aside from standard regularization techniques, we perform dropout directly on the attention tensor, silencing either whole nodes (node-ablation), connections (edge-ablation) or channels (feature dropout). All of these are options in model and our code.
Generality Semantic edge feature information are preserved (this guaranteed by adding it to the messages directly), and structural information is also taken into account, modulating both the attention matrix and the messages.
Note that this framework is general enough to be applied to other types of data such as sequences, images or 3D point clouds. The graph structure is not essential to the CSA formulation, which can be transposed to other fields by designing an appropriate RPE for instance embedding 3D distance and angles in 3D) points clouds, or pixel distances images.
Limitations There are several limitations with our proposed model, the main one being the scalability to large and sparse graphs. Indeed the computational complexity of our method scales as O(INnodess d), compared to the [d) complexxy of local MPNNs. For small graphs however this is partially offset. by the efficiency of GPU dense matrix computation. To scale method to larger graphs one could explore the numerous attempts at a linear or sub-quadratic Transformers (Zaheer et al. 2020 Jacgle et al. 2022 transposing them to our framework. The pre-processing steps also incur a one-time cost of 0() modes p) for
the computation the RWSE or the SPDE base values.
Another limitation could be the reliance on hand-crafted features in the RPEs although the same could be said of all GTss but in fact there are very few srper-parameters to tune, and fixed step sizes of RWSE-16, SPDE-8 work across varicty of datascts.
Implementation We integrate our proposed CSA layer in the general GraphGPS framework, which enables testing on several benchmark datasets, choice of wide range of positional encodings well adding local MPNN layers in parallel (we do not use this feature in this paper though). We also add our inplementations of the relative positional encodings Edge-RWSE and SPDE. The code of the model, and scripts to reproduce the experiments are freely available at lhttps://githin conmiinraatthothcsss
4 Experiments
4.1 Experimental setup
We evaluate the performance of our model on 3 medium-scale benchmark datasets from (Dwivedi et al. 2020 ZINC, PATTERN and CLUSTER and on the large molecular dataset PCQM4Mv2 Hu et al.) 2021). ). We arrange the compared methods in 3 broad categories: MPNNs, extended (or hierarchical) MPNNs, and graph transformers. We chose to include the best performing ones (to our knowledge) in each category, with an emphasis on the more comparable graph transformers. We show SOTA results in all 4 datasets.
We conduct ablation studies to showcase the individual contributions of (i) chromatic attention, (ii) attending to distant nodes (iii) the choice of the RPE and (iv) incorporating topological sub-structures.
8

---

Published in Transactions on Machine Learning Research (10/2023)
ZINC
Model
MAE 4
MPNN MPNN
GCN (Kipf & Welling
2017
0.367 = 0.011
GatedGCN (Dwivedi al.
2022a
0.090 + 0.001
GPS
Rampašek et al.
2022
0.070 = 0.004
CIN Bodnar et al. 2021a
0.079 = 0.006
b
CRaWL
(Tonshoof et al.
2021
0.085 = 0.004
GIN-AK+ Zhao et al.
2022
0.080 = 0.001
Transformers
SAN
Krenzer et al. 2021
0.139 + 0.006
Graphormer
Ying et al.
2021
0.122 = 0.006
SAT
Chen et al. 2022
0.094 + 0.008
EGT
Hussain et al. 2022
0.108 = 0.009
GRPE Park et al. 2022
0.094 = 0.002
CSA (ours)
0.070 + 0.003
CSA-rings (ours)
0.056 1 0.002
Table 1: Results on the ZINC dataset, ordered by the broad model class and performance. (GPS is placod among local MPNNs here, their classification performance can be fully imputed to their local module).
Following RRampsškk et al. 2022 the ablation studies are performed on ZINC and subset of PCQM4Mv2 containing 10% of the whole training set (but the same validation set as the full version).
Little hyperparameter search was done, defaulting to the setup in GraphGPS or other comparable models whenever possible. We chose maximum ring-size as (Bodnar al. 2021a) 18 for ZINC, and for OGB datasets. For ZINC we use the categorical ring encoding scheme described section 3.4 and for OGB datasets use the boolean encoding.
All models were trained on single NVidia V100 GPU system, with 16GB or 32GB memory depending on the dataset. See Appendix for the complete experimental details and hyperparameters.
4.2 Benchmarking CSA
ZINC and Benchmarking Graph Neural Networks ZINC is a popular molecular graph regression benchmark, for which Graph Transformers have historically lagged behind their sparse counterparts. The dataset splits fixed as in (Dwivedi et al. 2020), with 10K graphs in the training set, 1K in the both the validation and lest sels. The graphs contain on average 20-30 nodes.
Our results are presented in Table in which the ring-augmented version of our model CSA-ring im- proves the state of the art by a significant margin. Adding ring information greatly improves the per- formance in ZINC, which is be expected as the artificial objective partially depends the counts of these patterns.
PATTERN
CLUSTER Accuracy 1
Model
Accuracy
SAN Graphormer
86.581 + 0.037
76.691 + 0.65
SAT
86.848 0.037
77.856 0.104
However we note that CSA achieves state-of-the-art results even without this substructure information, far surpassing other standalone GTs.
EGT
86.821 + 0.020
79.232 = 0.348
GPS
86.685 0.059
78.016 + 0.180
CSA (ours)
87.011 0.036
79.175 0.126
PATTERN and CLUSTER PATTERN and CLUSTER are synthetic datasets of inductive node- level classification. In both cases, graphs were gen- erated using Stochastic Block Model (SBM). In
Table 2: Results on the node classification datasets PATTERN and CLUSTER.
9

---

Published in Transactions on Machine Learning Research (10/2023)
PATTERN, must detect which nodes were generated with different SBM parameters than the rest of the graph, and in CLUSTER one must assign each node to one of the 6 clusters (or blocks) it belongs to. These datasets contain 14K and 12K graphs, each with 100 nodes and 2K edges on average (much denser than molecule graphs). As shown in Table 2 we reach state-of-the-art performances on both datasets.
Larger scale datascts Finally, we assess the performance of our method on the large-scale benchmark PCQM4Mv2. The dataset is composed of 3.7M small undirected graphs (14.6 nodes on average).
PCQM4M
Model
Validation MAE + # Param.
GCN
0.1379
2.0M
MPNN
GCN-virtual
0.1153
4.9M
GIN
0.1195
3.8M
GIN-virtual
0.1083
6.7M
Transformers
Graphormer
0.0864
48.3M
EGT
0.0869
89.3M
GRPE
0.0890
46.2M
GPS-small
0.0938
6.2M
Pes-medium
0.0858
19.4M
CSA-small (ours)
0.0898
2.8M
CSA-doep (ours)
0.0853
8.3M
Table 3: Results the PCQM4M-v2 dataset, ordered by the broad model class and performance. As the test set was kept private by challenge organizers, evaluation is done using the validation set.
We lest both a small and a deep version of our model in Table 3 which outperforms SOTA with less parameters. The difference performance is however not as significant as for the ZINC dataset, we presume this is because the comparative advantages of our method diminishes with larger models and datascts.
4.3 Ablation study
Usefulness of the chromatic attention In Figure 3 we visualize the different learned attention score matrices, for 3 output channels of a 16-channel model trained on ZINC. Each these channels clearly focuses on structurally different nodes: the first channel on neighboring nodes, the second one on atoms in the same ring, and the third one focuses exclusively on faraway nodes. The figure shows both the complete node-node attention maps for each channel, and also some selected rows plotted the graphs, for ease of visualization. These show the attention scores of the highlighted node, across 4 structurally diverse nodes.
The attention maps for all 16 channels are available in the appendix.
To ensure that this is not learned artifact and that the model actually benefits from these extra degrees of freedom, we empirically validate the increased expressiveness of CSA in Table 5 by successively ab-
ZINC
PCQM4Mv2
RPE
MAE 4
MAE 1
lating the chromatie nature of the attention scores, and the edge value vectors. On the two datascts we have tested, both additions make an impact.
None
0.112 +: 0.002
0.1192 + 0.0001
SPDE
0.068 + 0.002
0.1123 + 0.0006
RWSE
0.066 + 0.001
0.1109 + 0.0002
Structural encoding We confront our proposed positional and structural Random Walks Encodings with the Shortest Path Distance Encoding in Table and confirm the intuition that RWSE outperforms
Table 4: CSA performance with different relative posi- lional and structural encodings (RPE).
SPDE, regardless of the topological encoding.
10

---

Published in Transactions on Machine Learning Research (10/2023)
Channel 1: Focus on neighbours
Channel Focus on rings
Channel 12: Focus on faraway nodes
Full Attention Matrix
Node
Node
Node
Node 13
Figure 3: Attention scores for multiple nodes and channels at the last layer of 16-channel model trained ZINC. Scores are shown for the molecule ZINC44549294. Each column corresponds different channel, which we hand-picked based on their diversity and interpretability. The top row shows the attention channel scores for all node pairs, and each subsequent row corresponds to a different receiving node, highlighted red.
PCQM4Mv2
ZINC
subset
Color
Edge Value
MAE
1
MAE 1
0.214 +: 0.067
0.129 + 0.0010
0.176 + 0.053
0.121 0.0018
V
0.068 0.004
0.112 0.0008
V 0.066 + 0.001
0.111 0.0002
Table 5: Added value of including multidimensionall attention filters ('Color" column) and injecting edge information in the value messages ("Edge Value" column)
5
Conclusion
We have proposed an extension of the Transformer sell-attention, generalizing atlention scores to atlention filters. We adapt the resulting model to graph representation learning, further extending the self-attention mechanism by incorporating odge features in the message values. We combine our model with a novel relative positional encoding scheme and topology encoding one and we show its added value on a diverse of graph benchmarks.
11

---

Published in Transactions on Machine Learning Research (10/2023)
Acknowledgments
This project was supported by ANR 31A MIAl@Grenoble Alpes (ANR-19-P31A-0003) and by ERC grant number 101087696 (APHELEIA project). The bulk of the computations were performed using HPC resources from GENCI IDRIS (Projett/A0LL1102200)).
References
Ralph Abboud, Ismail Ilkan Ceylan, Martin Grohe, and Thomas Lukasiewicz. The Surprising Power of Graph Neural Networks with Random Node Initialization. International Joint Conference Artificial Inlelligence (IJCAI), 2021.
Uri Alon and Eran Yahav. On the bottleneck of graph neural networks and its practical implications. In International Conference on Learning Represenlalions, 2021.
Damitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. 3rd International Conference on Learning Representations (ICLR), 2015.
Cristian Bodnar, Fabrizio Frasca, Nina Otter, Yuguang Wang, Pictro Liò, Guido Montufar, and Michael Bronstein. Weisfeiler and Lehman go cellular: CW networks. In Advances in Neurul Information, Processing Systems (NewrIPS), 2021a.
Cristian Bodnar, Fabrizio Frasca, Yuguang Wang, Nina Otter, Guido F Montufar, Pietro Lió, and Michael Bronstein. Weisfeiler and Lehman go topological: Message passing simplicial networks. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research (PMLR), pp. 1026-1037, 18 24 Jul 2021b.
Giorgos Bouritsas, Fabrizio Frasca, Stefanos Zafeiriou, and Michael Bronstein. Improving graph neural network expressivity via subgraph isomorphism counting. arXiv preprint arXiv:2006. 09252, 2020.
Xavier Bresson and Thomas Laurent. Residual Gated Graph ConvNets. arXiv preprint or Xiv: Xic:1711.00553, 2017.
Dexiong Chen, Leslie O' O'Bray, and Karsten Borgwardt. Structure-aware transformer for graph representation learning. In Proceedings of the 39th International Conference on Machine Learning (ICML), Proceedings Machine Learning Research, 2022.
Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Velickovic. Principal neighbourhood aggregation for graph nets. Advances in Neural Information Processing Systems (NewrIPS), 2020.
Austin Derrow-Pinion, Jennifer She, David Wong, Oliver Lange, Todd Hester, Luis Perez, Mare Nunkesser, Scongjae Lee, Xueying Guo, Brett Wiltshire, Peter W. Battaglia, Vishal Gupta, Ang Li, Zhongwen Xu, Alvaro Sanchez-Gonvaler,, Yujia Li, and Petar Velickovic. ETA Prediction with Graph Neural Networks in
Google Maps. International Conference on Information and Knowledge Management (CIKM), 2021.
Vijay Prakash Dwivedi, Chaitanya K Joshi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. arXiv preprint ar arXiv:2003.009: 2020.
Vijay Prakash Dwivedi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Graph neural networks with learnable structural and positional representations. In International Conference on Learning Representations (ICLR), 2022a.
Vijay Prakash Dwivedi, Ladislav Rampášek, Mikhail Galkin, Ali Parviz, Guy Wolf, Anh Tuan Luu, and Dominique Beaini. Long Range Graph Benchmark. arXiv preprint ar 08164, 2022b.
Moshe Elinsof, Fabrizio Frasca, Beatrice Bevilacqua, Eran Treister, Gal Chechik, and Haggai Maron. Graph positional encoding via random feature propagation, 2023.
M. Fey, J. G. Yuen, and F. Weichert. Hierarchical inter-message passing for learning on molocular graphs. ICML Graph Representation Learning and Beyond (GRL+) Workhop, 2020.
12

---

Published in Transactions on Machine Learning Research (10/2023)
Thomas Gaudelet, Ben Day, Arian R Jamasb, Jyothish Soman, Cristian Regep, Gertrude Liu, Jeremy B R. Hayter, Richard Vickers, Charles Roberts, Jian Tang, David Roblin, Tom I, Blundell, Michael M Bronstein, and Jake P Taylor- King. Utilizing graph machine learning within drug discovery and development. Briefings in Bioinformatics, 22(6), 05 2021.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In Advances in Neural Information Processing Systems (NewrIPS), 2020.
Weihua Hu", Bowen Liu", Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategics for pre-training graph neural networks. In International Conference on Learning Representations (ICLR), 2020.
Weihua Hu, Malthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. OGB-LSC: A large-scale challenge for machine learning on graphs. In Advances in Newral Information Processing Systems Dalasets and Benchmarks Track (Round 2), 2021.
Md Shamim Hussain, Mohammed J. Zaki, and Dharmashankar Subramanian. Global Sell-Attention as a Replacement for Graph Convolution. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2022.
Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Tonescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier Henaff, Matthew Botvinick, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver IO: A general architecture for structured inputs & outputs. In International Conference on Learning Representations (ICLR), 2022.
Jinwoo Kim, Dat Tien Nguyen, Seonwoo Min, Sungjun Cho, Moontae Lee, Honglak Lee, and Seunghoon Hong. Pure transformers are powcrful graph learners. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https:
//openreview.oo wtt.:///w.....ttoooooooro/////aa
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.
Devin Kreuzer, Dominique Beaini, William L. Hamilton, Vincent Létourncau, and Prudencio Tossou. Rethinking graph transformers with spectral attention. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems (NeurIPS), 2021.
Pan Li, Yanbang Wang, Hongwei Wang, and Jure Leskovec. Distance encoding: Design provably more powerful neural networks for graph representation learning. Advances in Newral Information Processing Systems (NewrIPS), 2020. ISSN 10495258.
Derek Lim, Joshua Robinson, Lingxiao Zhao, Tess Smidt, Suvrit Sra, Haggai Maron, and Stefanie Jegelka. Sign and Basis Invariant Networks for Spectral Graph Representation Learning. arXiv preprint 130133 2022.
Tyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
Grégoire Mialon, Dexiong Chen, Margot Selosse, and Julien Mairal. GraphiT: Encoding Graph Structure in Transformers. arXiv preprint 25:::2200000607 2021.
Christopher Morris, Yaron Lipman, Haggai Maron, Bastian Rieck, Nils M. Kriege, Martin Grohe, Matthias Fey, and Karsten M. Borgwardt. Weisfeiler and leman go machine learning: The story so far. CoRR, abs/2111.099992, 2021.
Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification. In International Conference on Learning Representations (TCI.R, 2020.
13

---

Published in Transactions on Machine Learning Research (10/2023)
Wonpyo Park, Woong-Gi Gi Chang, Donggeon Lee, Juntae Kim, and Seungwon Hwang. Grpe: Relative positional encoding for graph transformer. In ICLR 2022 Machine Learning for Drug Discovery, 2022.
Ladislav Rampášek, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for General, Powerful, Scalable Graph Transformer. Advances in Neural Information Processing Systems (NeurIPS), 35, 2022.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 2 (Short Papers), pp. 464-468. Association for Computational Linguisties, 2018.
Jan Toenshoff, Martin Ritzert, Hinrikus Wolf, and Martin Grohe. Graph Learning with 1D Convolutions on Random Walks, arXiv preprint arXiv:2102. 08786, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Petar Veličkovié, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks, International Conference on Learning Representations (ICLR), 2018.
Zhanghao Wu, Paras Jain, Matthew A. Wright, Azalia Mirhoseini, Joseph E. Gonzalez, and Ton Stoica. Representing Long- Range Context for Graph Neural Networks with Global Attention. Advances in Neural Information Processing Systems (NewrIPS), 2021.
Keyulu Xu, Stefanie Jegelka, Weihua Hu, and Jure Leskovec. How powerful are graph neural networks? 7th International Conference on Learning Representations (ICLR), 2019.
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tic- Yan Tiu. Do transformers really perform badly for graph representation? Tn Advances in Neural Information Processing Systems (NeurIPS), 2021.
Rex Ying, Ruining He, Kaifeng Chen, Pong Elssombatcha,, William L. Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2018.
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems (NeurIPS). 2020.
Lingxiao Zhao, Wei Jin, Leman Akoglu, and Neil Shah. From stars subgraphs: Uplifting any GNN with local structure awareness. In International Conference on Learning Representations (TCI.R), 2022.
A Experimental Details
Implementation As previously mentioned, our code is integrated in the GraphGPS framework introduced by (Rampášek et al., (022). In particular, the data-loaders and splitting strategies are kept the same ensure a fair comparison with other methods. Results in the benchmark section were all obtained with 10 seeds cach, except for PCQM4Mv2-full which was run single sced. Results in the ablation study are aggregated over 4 seeds. We report the mean loss along with its standard deviation over runs.
The table p recaps the chosen hyperparameters for each dataset, as well as training execution time. In all our experiments we use the AdamW [Loshchllov & Hutter 2019) optimizer with default. values for 31, 32 and e, and a cosine learning rate scheduler with warmup.
Hyper-parameters
For the ZINC ablation studies, the number of epochs was reduced from 2000 to 1000, and rings were not taken into account.
14

---

Published in Transactions on Machine Learning Research (10/2023)
PCQM4Mv2
Hyperparameter
ZINC
PATTERN
CLUSTER
CSA- sma-small
CSA-deep
# CSA Layers
10
6
16
5
16
Hidden Dim
64
64
48
256
256
# Heads
4
8
16
16
Dropout
0
0.1
0
0
Attention Dropout
0.5
0.5
0.5
0.5
0.5
Node Positional Encoding
RWSF-20
RWSE-16
RWSE-16
None
None
PE dim
28
16
16
Relative Positional Encoding
RWSF-20
RWSE-16
RWSE-16
RWSF-16
SPDE-8
Edge parameter sharing
V
X
X
Rings
V
X
X
Max rings size
18
Encoding
categorical
additive
Batch Size
32
32
16
256
256
Learning rate
0.001
0.0005
0.0005
0.0005
0.0001
# Epochs
2000
100
100
300
150
# Warmup epochs Weight decay
50
5
10
10
le-5
le-5
le-5
0
0
# Parameters
350k
259k
382k
2.8M
8.2M
Time (per epoch)
21s
40s
85s
822s
1900s
Table 6: Hyperparameters used for all datascts. Ablation studies on the PCQM4Mr2-suboot are all done based on the CSA- smalsmall configuration.
PCQM4Mv2
Hyperparameter
ZINC PATTERN
CLUSTER
small model
medium model
GraphGPS (Time/epoch)
21s
32s
86s
619s
1124s
CSA (Time/epoch)
21s
40s
85s
822s
1520s
Table 7: Training times of CSA on single A100 GPU, compared to GraphGPS (Rampášek et al. 2022
Edge parameter sharing To reduce computation time and in some cases improve performance, we provide the option to share the edge representations gutt and geal among all layers.
To get frame of reference for the exccution times, we report the training times of GraphGPS RRampšcck
et al. 2022 in table 7
figure we show all 16 channels of the first layer single attention head CSAmodel trained on ZINC. of these were chosen figure 3 of the main text. One can clearly see that channels are learning different attention patterns, with some being identifiable immediate neighbors, nodes in the same ring, and also complementary channels concentrating faraway nodes.
15

---

Published in Transactions on Machine Learning Research (10/2023)
Channel
Channel
Channel
Channel
Channel
Channel
Channel
Channel
Channel 10
Channel
Channel
Channel 13
Channel 14
Channel 15
Channel 16
Figure 4: Attention scores for all 16 channels at the first layer single-head model trained on ZINC. Scores are shown for the molecule ZINC44549294.
16

---

Published in Transactions on Machine Learning Research (10/2023)
Color
Heads
ZINC
PCQM4Mv2
0.074 0.004
0.147 = 0.003
0.0059 + 0.001
0.121 = 0.001
best
0.060 0.002
0.1145 = 0.0002
best
0.056 + 0.001
0.112 + 0.001
h 1= d
0.062 0.001
0.118 = 0.005
Table 8: Performance CSA vs CSA-mono with different number of heads. best for 64 dimensions for
ZINC, 16 for 256 dimensions in PCQM4Mv2. (when the color and monochrome versions are the same)
B Difference between CSA and multi-head attention
In short, our method enhances the standard multi- head attention (which we call CSA-mono in the paper).
Indeed, note d the mumber channels and h the number of heads. For a given channel c E [1, d], we denote da = d/h the implicit dimension of the attention heads, and C₁ = [c/dd_] the index of the attention head which channel e belongs The c-th channel of the unnormalized attention score between two nodes q and k of dimension d, linked by the edge is then given by:
Standard multi-head attention (called CSA-mono in the paper) with d = h:
C c,d, tdk
0g ddn
Σ
qiki + Coy.
The edge bias Cc,, here is constant across all channels of the same attention head. (In fact in most. other GTss this bias is constant across all attention heads: 6ca, = E1, Vc).
Colored multi-head attention (CSA-color with 1 1< h < d), e Rd:
C,d +d
0g
Σ
qiki + ec
ddh
Leeydn
The difference here that the edge bias e, varies per channel instead of per head.
In multi-headed dot-product attention, choosing the hyperparameter h Theads amounts to striking balance between the selectivity of the per-head cosine similarities Qn Kn, (of implicit dimension d/h), and the potential diversity of those similaritics across attention heads (h different scores).
In CSA we maintain this balancing option intact, but also keep separate edge bias ea per channel, without loss of expressivity.
We compare CSA with its monochrome version in table varying the number of attention heads.
17


Reviews:
[{"review_id": "F4qWf4jAQf", "reviewer": "Reviewer_L9Qc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\nThe paper seems novel to me, and the idea is interesting. As stated by the authors, the proposed method can also be applied to other domains and this would be interesting to see how will it work.\n\nThe paper is well-written and easy to follow.\n\nThe results are promising and show a significant improvement compared to other methods.\n\nCons:\n\nThe citation format seems to be broken / not fitting the citation style of the paper. That is, when the authors cite a paper, in the middle of a sentence and then the name of the authors of the referenced papers show. I think this should be moved into parentheses, or change into numbered citations.\n\nIn page 3, \"Node positional encoding\", what is $r(\\Lambda$) ?\n\nThere is a rather cryptic sentence in page 4, saying \"On the other hand the heat diffusion or the PageRank kernels encode a notion of distance between nodes\" that comes slightly out of context. Perhaps, the authors can expand on that.\n\nThe authors are missing a citation about shortest path encoding in GNNs \"Shortest Path Networks for Graph Property Prediction\".\n\nBesides graph attention modules, another well performing approach on datasets like ZINC, is the use of subgraphs. I think that in order to provide a more broad comparison of existing methods, the authors should also add subgraph methods to their comparisons.\n\nRegarding positional encoding methods with powers of the adjacency matrix or the random walk matrix, the authors should also discuss \"SIGN: Scalable Inception Graph Neural Networks\", \"From Local to Global: Spectral-Inspired Graph Neural Networks\", \"Graph Positional Encoding via Random Feature Propagation\".\n\nIn Section 3.5, the authors use $E^{(0)}$. Can the authors comment on the possibility of also updating the edge features and using them in deeper layers?\n\nHave the authors tried to experiment with 3D point clouds as claimed? For instance, on point cloud classification?\n\n \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes CSA - a new \"Chromatic\" Self Attention mechanism. Here, the Chromatic sense stands for the ability to learn per channel weights that allow to dampen or highlight messages. The paper builds on recent advances in graph transformers such as GraphGPS. The authors show that the proposed CSA achieves state-of-the-art results on several benchmarks such as ZINC.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see my notes in my main review.", "broader_impact_concerns": "I find no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gXZoxfzFMv", "reviewer": "Reviewer_F8E3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper studies the problem of the positional encoding in graph transformer, which is a novel and relevant topic of interest.\n* The paper’s idea of extending the self-attention to the feature is interesting. The proposed method seems simple yet effective.\n* The SOTA performance on two benchmark datasets validate the effectiveness of the proposed CSA method.\n\nWeaknesses:\n* I found several claims and notations that are difficult to follow. For example, the notations for the edge feature matrix are not consistent. It seems like $E_{ij}$ is a vector, but is $\\mathbf{E}_{ij}$ from Equation (5) a matrix or vector as well? Meanwhile, the two different edge feature matrices $E^{att}$ and $E^{val}$ become matrices. I cannot also find the proper definition for both $E^{att}$ and $E^{val}$. Should $E^{val}$ be $E^{value}$? The authors are suggested to further proofread.\n* Some claims seem vague in my opinion. For example, on page 6, the authors claim that, as I quote, “RWSE is commonly used as a node-wise SE but has to our knowledge never been used before as an RPE.” However, in the RWSE section, the authors show that Equation (12) is an extension of GraphiT. Does this mean that RWSE is already used in GraphiT as RPE but in a different form? \n* The connection between edge features and node feature channels needs further discussion. Given the design of CSA are relatively intuitive, why the edge feature information does a great job in modulating the attention across node feature channels seems disconnected. Could the authors provide more insights into this design?\n* Figure 3 is interesting. How about the attention scores of other channels aside from the shown three? Can the authors reach a consistent insight on what kind of structural pattern a certain channel prefers? \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the self-attention mechanism in graph transformers, which is said to be able to adapt to broader types of structured data such as 3D point cloud. The key idea is to extend the self-attention scores to the domain of feature channels. By integrating the graph structural information and original edge features, the proposed Chromatic Self-Attention CSA could independently modulate the feature channels and go beyond the local message-passing mechanism. Higher-order topological information such as a ring - type motif is incorporated to further enhance the expressiveness of CSA. Extensive experiments on real-world datasets demonstrate the effectiveness of the proposed method and achieve SOTA performance on the ZINC benchmark dataset.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please kindly refer to the Strengths and Weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ATdjaNh0nK", "reviewer": "Reviewer_y6nv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths :\n1. A new CSA Mechanism extends the traditional attention scores to attention filters. This novel approach independently modulates feature channels, allowing for flexible encoding of graph structure through node-node interactions. \n2. The proposed method, CGT incorporates both graph structural information and edge features without the need for local message-passing components. By incorporating edge features in the message values, the model captures rich information from the graph, enhancing its expressive power. It can be considered as the generalized version of GraphiT and Graphormer,\n3.\tState-of-the-Art Performance on the real-world and synthetic datasets of all scales, showing the effectiveness of the method on small and large datasets.\n\nWeaknesses:\n1.\tLack the analysis of complexity. Will the method cost more time or space compared with other graph transformer?\n2.\tLack the introduction of some baselines and metrics. Why you select these methods as baselines? Are they representative?\n3.\tCode is not published.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents CSA (Chromatic Self-Attention), a novel self-attention mechanism for encoding graph structure in transformers. The approach is showcased in CGT (Chromatic Graph Transformer), which integrates graph structural information and edge features without local message-passing. By incorporating edge features and relative positional encoding, the method achieves state-of-the-art results on the benchmark datasets with various scales. It provides a flexible framework for graph structure encoding and incorporates higher-order topology.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.\tAdd some statistics of the used datasets to show they have different scales from small to large.\n2.\tSome equations are not indexed.\n3.\tIntroduce some baselines and metrics. For example, why you use MAE for some datasets and use accuracy for others?\n4.\tAlso refer to the weakness.", "broader_impact_concerns": "I do not have further concerns on the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tGoAtluS0L", "reviewer": "Reviewer_3Q4W", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The paper is well-written and clear. The model is well-motivated with several examples to provide intuition, and there is a comprehensive comparison of the architecture with other types of graph transformers. The experimental setup is well-documented.\n- The proposed chromatic self-attention generalizes existing graph transformer architectures and is more flexible, allowing the network to learn positional encodings and making fewer arbitrary choices.\n- CSA appears be competitive with state-of-the-art graph transformers on several datasets.\n## Weaknesses\n\n- The proposed architecture has issues with scalability, since computing attention has quadratic complexity in the number of nodes in the graph. It is not clear how much of the benefit of this model is due to the encoding structure versus the type of attention used. Would this advantage carry over even using linear or sparse attention like other graph transformers?\n- The new relative positional encoding introduced (RWSE) also seems to be expensive to compute - it requires computing $k$-th powers of the normalized adjacency matrix of the graph.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new framework for computing attention in graph transformers, by taking multidimensional attention filters instead of scalar attention coefficients. This allows for greater flexibility in choosing which part of a node's features the network passes to another node. The paper also introduces a new type of positional encoding based on the random walk adjacency matrix, which is used to assign learnable encodings to the edges.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- It would be nice to have a comparison of the runtime of CSA with other architectures.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers find the proposed attention approach for Graph Transformers interesting and the experimental results strong. Main concerns were around clarity of some of the claims and computational cost. Authors have updated the draft addressing the clarity concerns, and additional results. One reviewer felt the paper needs more experiments to be convincing, but I think the paper has enough experiments to establish the usefulness of the proposed method. Overall I think the current draft has nice contributions and recommend acceptance.
Decision: Accept (poster)