Rethinking Graph Transformers with Spectral Attention

1 Introduction . The prevailing strategy for graph neural networks ( GNNs ) has been to directly encode graph structure structure through a sparse message-passing process [ 17 , 19 ] . In this approach , vector messages are iteratively passed between nodes that are connected in the graph . Multiple instantiations of this message-passing paradigm have been proposed , differing in the architectural details of the message-passing apparatus ( see [ 19 ] for a review ) . However , there is a growing recognition that the message-passing paradigm has inherent limitations . The expressive power of message passing appears inexorably bounded by the Weisfeiler-Lehman isomorphism hierarchy [ 29 , 30 , 39 ] . Message-passing GNNs are known to suffer from pathologies , such ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . as oversmoothing , due to their repeated aggregation of local information [ 19 ] , and over-squashing , due to the exponential blow-up in computation paths as the model depth increases [ 1 ] . As a result , there is a growing interest in deep learning techniques that encode graph structure as a soft inductive bias , rather than as a hard-coded aspect of message passing [ 14 , 24 ] . A central issue with message-passing paradigm is that input graph structure is encoded by restricting the structure of the model ’ s computation graph , inherently limiting its flexibility . This reminds us of how early recurrent neural networks ( RNNs ) encoded sequential structure via their computation graph—a strategy that leads to well-known pathologies such as the inability to model long-range dependencies [ 20 ] . There is a growing trend across deep learning towards more flexible architectures , which avoid strict and structural inductive biases . Most notably , the exceptionally successful Transformer architecture removes any structural inductive bias by encoding the structure via soft inductive biases , such as positional encodings [ 36 ] . In the context of GNNs , the self-attention mechanism of a Transformer can be viewed as passing messages between all nodes , regardless of the input graph connectivity . Prior work has proposed to use attention in GNNs in different ways . First , the GAT model [ 37 ] proposed local attention on pairs of nodes that allows a learnable convolutional kernel . The GTN work [ 42 ] has improved on the GAT for node and link predictions while keeping a similar architecture , while other message-passing approaches have used enhancing spectral features [ 8 , 13 ] . More recently , the GT model [ 14 ] was proposed as a generalization of Transformers to graphs , where they experimented with sparse and full graph attention while providing low-frequency eigenvectors of the Laplacian as positional encodings . In this work , we offer a principled investigation of how Transformer architectures can be applied in graph representation learning . Our primary contribution is the development of novel and powerful learnable positional encoding methods , which are rooted in spectral graph theory . Our positional encoding technique — and the resulting spectral attention network ( SAN ) architecture — addresses key theoretical limitations in prior graph Transformer work [ 14 ] and provably exceeds the expressive power of standard message-passing GNNs . We show that full Transformer-style attention provides consistent empirical gains compared to an equivalent sparse message-passing model , and we demonstrate that our SAN architecture is competitive with or exceeding the state-of-the-art on several well-known graph benchmarks . An overview of the entire method is presented in Figure 1 , with a link to the code here : https : //github.com/DevinKreuzer/SAN . 2 Theoretical Motivations . There can be a significant loss in structural information if naively generalizing Transformers to graphs . To preserve this information as well as local connectivity , previous studies [ 37 , 14 ] have proposed to use the eigenfunctions of their Laplacian as positional encodings . Taking this idea further by using the full expressivity of eigenfunctions as positional encodings , we can propose a principled way of understanding graph structures using their spectra . The advantages of our methods compared to previous studies [ 37 , 14 ] are shown in Table 1 . 2.1 Absolute and relative positional encoding with eigenfunctions . The notion of positional encodings ( PEs ) in graphs is not a trivial concept , as there exists no canonical way of ordering nodes or defining axes . In this section , we investigate how eigenfunctions of the Laplacian can be used to define absolute and relative PEs in graphs , to measure physical interactions between nodes , and to enable ” hearing ” of specific sub-structures - similar to how the sound of a drum can reveal its structure . 2.1.1 Eigenvectors equate to sine functions over graphs . In the Transformer architecture , a fundamental aspect is the use of sine and cosine functions as PEs for sequences [ 36 ] . However , sinusoids can not be clearly defined for arbitrary graphs , since there is no clear notion of position along an axis . Instead , their equivalent is given by the eigenvectors φ of the graph Laplacian L. Indeed , in a Euclidean space , the Laplacian ( or Laplace ) operator corresponds to the divergence of the gradient and its eigenfunctions are sine/cosine functions , with the squared frequencies corresponding to the eigenvalues ( we sometimes interchange the two notions from here on ) . Hence , in the graph domain , the eigenvectors of the graph Laplacian are the natural equivalent of sine functions , and this intuition was employed in multiple recent works which use the eigenvectors as PEs for GNNs [ 15 ] , for directional flows [ 4 ] and for Transformers [ 14 ] . Being equivalent to sine functions , we naturally find that the Fourier Transform of a function F [ f ] applied to a graph gives F [ f ] ( λi ) = 〈f , φi〉 , where the eigenvalue is considered as a position in the Fourier domain of that graph [ 6 ] . Thus , the eigenvectors are best viewed as vectors positioned on the axis of eigenvalues rather than components of a matrix as illustrated in Figure 2 . 2.1.2 What do eigenfunctions tell us about relative positions ? . In addition to being the analog of sine functions , the eigenvectors of the Laplacian also hold important information about the physics of a system and can reveal distance metrics . This is not surprising as the Laplacian is a fundamental operator in physics and is notably used in Maxwell ’ s equations [ 16 ] and the heat diffusion [ 6 ] . In electromagnetic theory , the ( pseudo ) inverse of the Laplacian , known in mathematics as the Green ’ s function of the Laplacian [ 9 ] , represents the electrostatic potential of a given charge . In a graph , the same concept uses the pseudo-inverse of the LaplacianG and can be computed by its eigenfunctions . See equation 1 , where G ( j1 , j2 ) is the electric potential between nodes j1 and j2 , φ̂i and λ̂i are the i-th eigenvectors and eigenvalues of the symmetric LaplacianD −1 2 LD −1 2 , andD is the degree matrix , and φ̂i , j the j-th row of the vector . G ( j1 , j2 ) = d 1 2 j1 d −1 2 j2 ∑ i > 0 ( φ̂i , j1φ̂i , j2 ) 2 λ̂i ( 1 ) Further , the original solution of the heat equation given by Fourier relied on a sum of sines/cosines known as a Fourier series [ 7 ] . As eigenvectors of the Laplacian are the analogue of these functions in graphs , we find similar solutions . Knowing that heat kernels are correlated to random walks [ 6 , 4 ] , we use the interaction between two heat kernels to define in equation 2 the diffusion distance dD between nodes j1 , j2 [ 6 , 10 ] . Similarly , the biharmonic distance dB was proposed as a better measure of distances [ 28 ] . Here we use the eigenfunctions of the regular Laplacian L. d2D ( j1 , j2 ) = ∑ k > 0 e−2tλi ( φi , j1 − φi , j2 ) 2 , d2B ( j1 , j2 ) = ∑ i > 0 ( φi , j1 − φi , j2 ) 2 λ2i ( 2 ) There are a few things to note from these equations . Firstly , they highlight the importance of pairing eigenvectors and their corresponding eigenvalues when supplying information about relative positions in a graph . Secondly , we notice that the product of eigenvectors is proportional to the electrostatic interaction , while the subtraction is proportional to the diffusion and biharmonic distances . Lastly , there is a consistent pattern across all 3 equations : smaller frequencies/eigenvalues are more heavily weighted when determining distances between nodes . 2.1.3 Hearing the shape of a graph and its sub-structures . Another well-known property of eigenvalues is how they can be used to discriminate between different graph structures and sub-structures , as they can be interpreted as the frequencies of resonance of the graph . This led to the famous question about whether we can hear the shape of a drum from its eigenvalues [ 23 ] , with the same questions also applying to geometric objects [ 12 ] and 3D molecules [ 33 ] . Various success was found with the eigenfunctions being used for partial functional correspondence [ 32 ] , algorithmic understanding geometries [ 26 ] , and style correspondence [ 12 ] . Examples of eigenvectors for molecular graphs are presented in Figure 3 . 2.2 Laplace Eigenfunctions etiquette In Euclidean space and sequences , using sinusoids as PEs is trivial : we can simply select a set of frequencies , compute the sinusoids , and add or concatenate them to the input embeddings , as is done in the original Transformer [ 36 ] . However , in arbitrary graphs , reproducing these steps is not as simple since each graph has a unique set of eigenfunctions . In the following section , we present key principles from spectral graph theory to consider when constructing PEs for graphs , most of which have been overlooked by prior methods . They include normalization , the importance of the eigenvalues and their multiplicities , the number of eigenvectors being variable , and sign ambiguities . Our LPE architectures , presented in section 3 , aim to address them . Normalization . Given an eigenvalue of the Laplacian , there is an associated eigenspace of dimension greater than 1 . To make use of this information in our model , a single eigenvector has to be chosen . In our work , we use the L2 normalization since it is compatible with the definition of the Green ’ s function ( 1 ) . Thus , we will always chose eigenvectors φ such that 〈φ , φ〉 = 1 . Eigenvalues . Another fundamental aspect is that the eigenvalue associated with each eigenvector supplies valuable information . An ordering of the eigenvectors based on their eigenvalue works in sequences since the frequencies are pre-determined . However , this assumption does not work in graphs since the eigenvalues in their spectrum can vary . For example , in Figure 3 , we observe how an ordering would miss the fact that both molecules resonate at λ = 1 in different ways . Multiplicities . Another important problem with choosing eigenfunctions is the possibility of a high multiplicity of the eigenvalues , i.e . when an eigenvalue appears as a root of the characteristic polynomial more than once . In this case , the associated eigenspace may have dimension 2 or more as we can generate a valid eigenvector from any linear combination of eigenvectors with the same eigenvalue . This further complicates the problem of choosing eigenvectors for algorithmic computations and highlights the importance of having a model that can handle this ambiguity . Variable number of eigenvectors . A graphGi can have at mostNi linearly independent eigenvectors with Ni being its number of nodes . Most importantly , Ni can vary across all Gi in the dataset . Prior work [ 14 ] elected to select a fixed number k eigenvectors for each graph , where k ≤ Ni , ∀i . This produces a major bottleneck when the smallest graphs have significantly fewer nodes than the largest graphs in the dataset since a very small proportion of eigenvectors will be used for large graphs . This inevitably causes loss of information and motivates the need for a model which constructs fixed PEs of dimension k , where k does not depend on the number of eigenvectors in the graph . Sign invariance . As noted earlier , there is a sign ambiguity with the eigenvectors . With the sign of φ being independent of its normalization , we are left with a total of 2k possible combination of signs when choosing k eigenvectors of a graph . Previous work has proposed to do data augmentation by randomly flipping the sign of the eigenvectors [ 4 , 15 , 14 ] , and although it can work when k is small , it becomes intractable for large k .

Summary: This work proposes a transformer designed for graph structured data. As opposed to GNNs, the transformer allows non-neighboring nodes to communicate. To avoid loosing information on the structure of the graph, the transformer needs some form of structure encoding of the nodes, a non trivial problem for graphs. A common PE scheme consists in using the spectrum of the graphs Laplacian but, as detailed by the authors, this raises some issues. The authors propose an improved positional encoding circumventing these problems and demonstrate that a fully connected transformer is competitive with GNNs on four datasets.