Do Transformers Really Perform Badly for Graph Representation?

1 Introduction . The Transformer [ 46 ] is well acknowledged as the most powerful neural network in modelling sequential data , such as natural language [ 11 , 35 , 6 ] and speech [ 17 ] . Model variants built upon Transformer have also been shown great performance in computer vision [ 12 , 36 ] and programming language [ 19 , 57 , 41 ] . However , to the best of our knowledge , Transformer has still not been the de-facto standard on public graph representation leaderboards [ 22 , 14 , 21 ] . There are many attempts of leveraging Transformer into the graph domain , but the only effective way is replacing some key modules ( e.g. , feature aggregation ) in classic GNN variants by the softmax attention [ 47 , 7 , 23 , 48 , 56 , 43 , 13 ] . Therefore , it is still an open question whether Transformer architecture is suitable to model graphs and how to make it work in graph representation learning . In this paper , we give an affirmative answer by developing Graphormer , which is directly built upon the standard Transformer , and achieves state-of-the-art performance on a wide range of graphlevel prediction tasks , including the very recent Open Graph Benchmark Large-Scale Challenge ( OGB-LSC ) [ 21 ] , and several popular leaderboards ( e.g. , OGB [ 22 ] , Benchmarking-GNN [ 14 ] ) . The Transformer is originally designed for sequence modeling . To utilize its power in graphs , we believe ∗Interns at MSRA . †Corresponding authors . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the key is to properly incorporate structural information of graphs into the model . Note that for each node i , the self-attention only calculates the semantic similarity between i and other nodes , without considering the structural information of a graph reflected on the nodes and the relation between node pairs . Graphormer incorporates several effective structural encoding methods to leverage such information , which are described below . First , we propose a Centrality Encoding in Graphormer to capture the node importance in the graph . In a graph , different nodes may have different importance , e.g. , celebrities are considered to be more influential than the majority of web users in a social network . However , such information isn ’ t reflected in the self-attention module as it calculates the similarities mainly using the node semantic features . To address the problem , we propose to encode the node centrality in Graphormer . In particular , we leverage the degree centrality for the centrality encoding , where a learnable vector is assigned to each node according to its degree and added to the node features in the input layer . Empirical studies show that simple centrality encoding is effective for Transformer in modeling the graph data . Second , we propose a novel Spatial Encoding in Graphormer to capture the structural relation between nodes . One notable geometrical property that distinguishes graph-structured data from other structured data , e.g. , language , images , is that there does not exist a canonical grid to embed the graph . In fact , nodes can only lie in a non-Euclidean space and are linked by edges . To model such structural information , for each node pair , we assign a learnable embedding based on their spatial relation . Multiple measurements in the literature could be leveraged for modeling spatial relations . For a general purpose , we use the distance of the shortest path between any two nodes as a demonstration , which will be encoded as a bias term in the softmax attention and help the model accurately capture the spatial dependency in a graph . In addition , sometimes there is additional spatial information contained in edge features , such as the type of bond between two atoms in a molecular graph . We design a new edge encoding method to further take such signal into the Transformer layers . To be concrete , for each node pair , we compute an average of dot-products of the edge features and learnable embeddings along the shortest path , then use it in the attention module . Equipped with these encodings , Graphormer could better model the relationship for node pairs and represent the graph . By using the proposed encodings above , we further mathematically show that Graphormer has strong expressiveness as many popular GNN variants are just its special cases . The great capacity of the model leads to state-of-the-art performance on a wide range of tasks in practice . On the large-scale quantum chemistry regression dataset3 in the very recent Open Graph Benchmark Large-Scale Challenge ( OGB-LSC ) [ 21 ] , Graphormer outperforms most mainstream GNN variants by more than 10 % points in terms of the relative error . On other popular leaderboards of graph representation learning ( e.g. , MolHIV , MolPCBA , ZINC ) [ 22 , 14 ] , Graphormer also surpasses the previous best results , demonstrating the potential and adaptability of the Transformer architecture . 2 Preliminary . In this section , we recap the preliminaries in Graph Neural Networks and Transformer . Graph Neural Network ( GNN ) . Let G = ( V , E ) denote a graph where V = { v1 , v2 , · · · , vn } , n = |V | is the number of nodes . Let the feature vector of node vi be xi . GNNs aim to learn representation of nodes and graphs . Typically , modern GNNs follow a learning schema that iteratively updates the representation of a node by aggregating representations of its first or higher-order neighbors . We denote h ( l ) i as the representation of vi at the l-th layer and define h ( 0 ) i = xi . The l-th iteration of aggregation could be characterized by AGGREGATE-COMBINE step as a ( l ) i = AGGREGATE ( l ) ( { h ( l−1 ) j : j ∈ N ( vi ) } ) , h ( l ) i = COMBINE ( l ) ( h ( l−1 ) i , a ( l ) i ) , ( 1 ) where N ( vi ) is the set of first or higher-order neighbors of vi . The AGGREGATE function is used to gather the information from neighbors . Common aggregation functions include MEAN , MAX , SUM , which are used in different architectures of GNNs [ 26 , 18 , 47 , 50 ] . The goal of COMBINE function is to fuse the information from neighbors into the node representation . 3https : //ogb.stanford.edu/kddcup2021/pcqm4m/ In addition , for graph representation tasks , a READOUT function is designed to aggregate node features h ( L ) i of the final iteration into the representation hG of the entire graph G : hG = READOUT ( { h ( L ) i | vi ∈ G } ) . ( 2 ) READOUT can be implemented by a simple permutation invariant function such as summation [ 50 ] or a more sophisticated graph-level pooling function [ 1 ] . Transformer . The Transformer architecture consists of a composition of Transformer layers [ 46 ] . Each Transformer layer has two parts : a self-attention module and a position-wise feed-forward network ( FFN ) . Let H = [ h⊤1 , · · · , h⊤n ] ⊤ ∈ Rn×d denote the input of self-attention module where d is the hidden dimension and hi ∈ R1×d is the hidden representation at position i . The input H is projected by three matrices WQ ∈ Rd×dK , WK ∈ Rd×dK and WV ∈ Rd×dV to the corresponding representations Q , K , V . The self-attention is then calculated as : Q = HWQ , K = HWK , V = HWV , ( 3 ) A = QK⊤√ dK , Attn ( H ) = softmax ( A ) V , ( 4 ) where A is a matrix capturing the similarity between queries and keys . For simplicity of illustration , we consider the single-head self-attention and assume dK = dV = d. The extension to the multi-head attention is standard and straightforward , and we omit bias terms for simplicity . 3 Graphormer . In this section , we present our Graphormer for graph tasks . First , we elaborate on several key designs in the Graphormer , which serve as an inductive bias in the neural network to learn the graph representation . We further provide the detailed implementations of Graphormer . Finally , we show that our proposed Graphormer is more powerful since popular GNN models [ 26 , 50 , 18 ] are its special cases . 3.1 Structural Encodings in Graphormer . As discussed in the introduction , it is important to develop ways to leverage the structural information of graphs into the Transformer model . To this end , we present three simple but effective designs of encoding in Graphormer . See Figure 1 for an illustration . 3.1.1 Centrality Encoding . In Eq.4 , the attention distribution is calculated based on the semantic correlation between nodes . However , node centrality , which measures how important a node is in the graph , is usually a strong signal for graph understanding . For example , celebrities who have a huge number of followers are important factors in predicting the trend of a social network [ 38 , 37 ] . Such information is neglected in the current attention calculation , and we believe it should be a valuable signal for Transformer models . In Graphormer , we use the degree centrality , which is one of the standard centrality measures in literature , as an additional signal to the neural network . To be specific , we develop a Centrality Encoding which assigns each node two real-valued embedding vectors according to its indegree and outdegree . As the centrality encoding is applied to each node , we simply add it to the node features as the input . h ( 0 ) i = xi + z − deg− ( vi ) + z+deg+ ( vi ) , ( 5 ) where z− , z+ ∈ Rd are learnable embedding vectors specified by the indegree deg− ( vi ) and outdegree deg+ ( vi ) respectively . For undirected graphs , deg− ( vi ) and deg+ ( vi ) could be unified to deg ( vi ) . By using the centrality encoding in the input , the softmax attention can catch the node importance signal in the queries and the keys . Therefore the model can capture both the semantic correlation and the node importance in the attention mechanism . 3.1.2 Spatial Encoding . An advantage of Transformer is its global receptive field . In each Transformer layer , each token can attend to the information at any position and then process its representation . But this operation has a byproduct problem that the model has to explicitly specify different positions or encode the positional dependency ( such as locality ) in the layers . For sequential data , one can either give each position an embedding ( i.e. , absolute positional encoding [ 46 ] ) as the input or encode the relative distance of any two positions ( i.e. , relative positional encoding [ 42 , 44 ] ) in the Transformer layer . However , for graphs , nodes are not arranged as a sequence . They can lie in a multi-dimensional spatial space and are linked by edges . To encode the structural information of a graph in the model , we propose a novel Spatial Encoding . Concretely , for any graph G , we consider a function ϕ ( vi , vj ) : V × V → R which measures the spatial relation between vi and vj in graph G. The function ϕ can be defined by the connectivity between the nodes in the graph . In this paper , we choose ϕ ( vi , vj ) to be the distance of the shortest path ( SPD ) between vi and vj if the two nodes are connected . If not , we set the output of ϕ to be a special value , i.e. , -1 . We assign each ( feasible ) output value a learnable scalar which will serve as a bias term in the self-attention module . Denote Aij as the ( i , j ) -element of the Query-Key product matrix A , we have : Aij = ( hiWQ ) ( hjWK ) T √ d + bϕ ( vi , vj ) , ( 6 ) where bϕ ( vi , vj ) is a learnable scalar indexed by ϕ ( vi , vj ) , and shared across all layers . Here we discuss several benefits of our proposed method . First , compared to conventional GNNs described in Section 2 , where the receptive field is restricted to the neighbors , we can see that in Eq . ( 6 ) , the Transformer layer provides a global information that each node can attend to all other nodes in the graph . Second , by using bϕ ( vi , vj ) , each node in a single Transformer layer can adaptively attend to all other nodes according to the graph structural information . For example , if bϕ ( vi , vj ) is learned to be a decreasing function with respect to ϕ ( vi , vj ) , for each node , the model will likely pay more attention to the nodes near it and pay less attention to the nodes far away from it .

Summary: This paper considers adapting a standard Transformer to work on general graphs, and applying it to a few recent graph representation learning benchmarks. They propose three ways of augmenting a standard Transformer architecture by adding new node features and additive attention biases, and show that their approach (GraphFormer) yields state-of-the-art performance on the considered benchmarks. They also prove some theoretical results stating that GraphFormer is more expressive than many standard graph neural network architectures, and conduct an ablation of the modifications they make to a standard Transformer, showing that each of them leads to better performance.  As the authors note, there has been a lot of work in adapting attention and other parts of transformers to a graph domain. In particular, there has been a lot of work on using attention mechanisms in graphs, and also more specifically in using edges to bias attention between nodes or using features based on shortest paths between nodes. Taken on their own, the modifications proposed in GraphFormer seem fairly incremental.  Nevertheless, most of the prior work I'm aware of focuses on individual modifications for specific tasks instead of a unified set of changes for standardized benchmark datasets, and often deviate from a standard Transformer backbone in a large number of ways. It does seem useful to demonstrate that the Transformer architecture can be applied to graph understanding tasks in a straightforward and general way, and that doing so leads to strong performance both theoretically and empirically. Transformers have shown strong results in many domains, and I think this paper could be a good starting point for future developments on using transformers for graph representation learning.