Rethinking Graph Transformers with Spectral Attention

1 Introduction . The prevailing strategy for graph neural networks ( GNNs ) has been to directly encode graph structure structure through a sparse message-passing process [ 17 , 19 ] . In this approach , vector messages are iteratively passed between nodes that are connected in the graph . Multiple instantiations of this message-passing paradigm have been proposed , differing in the architectural details of the message-passing apparatus ( see [ 19 ] for a review ) . However , there is a growing recognition that the message-passing paradigm has inherent limitations . The expressive power of message passing appears inexorably bounded by the Weisfeiler-Lehman isomorphism hierarchy [ 29 , 30 , 39 ] . Message-passing GNNs are known to suffer from pathologies , such ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . as oversmoothing , due to their repeated aggregation of local information [ 19 ] , and over-squashing , due to the exponential blow-up in computation paths as the model depth increases [ 1 ] . As a result , there is a growing interest in deep learning techniques that encode graph structure as a soft inductive bias , rather than as a hard-coded aspect of message passing [ 14 , 24 ] . A central issue with message-passing paradigm is that input graph structure is encoded by restricting the structure of the model ’ s computation graph , inherently limiting its flexibility . This reminds us of how early recurrent neural networks ( RNNs ) encoded sequential structure via their computation graph—a strategy that leads to well-known pathologies such as the inability to model long-range dependencies [ 20 ] . There is a growing trend across deep learning towards more flexible architectures , which avoid strict and structural inductive biases . Most notably , the exceptionally successful Transformer architecture removes any structural inductive bias by encoding the structure via soft inductive biases , such as positional encodings [ 36 ] . In the context of GNNs , the self-attention mechanism of a Transformer can be viewed as passing messages between all nodes , regardless of the input graph connectivity . Prior work has proposed to use attention in GNNs in different ways . First , the GAT model [ 37 ] proposed local attention on pairs of nodes that allows a learnable convolutional kernel . The GTN work [ 42 ] has improved on the GAT for node and link predictions while keeping a similar architecture , while other message-passing approaches have used enhancing spectral features [ 8 , 13 ] . More recently , the GT model [ 14 ] was proposed as a generalization of Transformers to graphs , where they experimented with sparse and full graph attention while providing low-frequency eigenvectors of the Laplacian as positional encodings . In this work , we offer a principled investigation of how Transformer architectures can be applied in graph representation learning . Our primary contribution is the development of novel and powerful learnable positional encoding methods , which are rooted in spectral graph theory . Our positional encoding technique — and the resulting spectral attention network ( SAN ) architecture — addresses key theoretical limitations in prior graph Transformer work [ 14 ] and provably exceeds the expressive power of standard message-passing GNNs . We show that full Transformer-style attention provides consistent empirical gains compared to an equivalent sparse message-passing model , and we demonstrate that our SAN architecture is competitive with or exceeding the state-of-the-art on several well-known graph benchmarks . An overview of the entire method is presented in Figure 1 , with a link to the code here : https : //github.com/DevinKreuzer/SAN . 2 Theoretical Motivations . There can be a significant loss in structural information if naively generalizing Transformers to graphs . To preserve this information as well as local connectivity , previous studies [ 37 , 14 ] have proposed to use the eigenfunctions of their Laplacian as positional encodings . Taking this idea further by using the full expressivity of eigenfunctions as positional encodings , we can propose a principled way of understanding graph structures using their spectra . The advantages of our methods compared to previous studies [ 37 , 14 ] are shown in Table 1 . 2.1 Absolute and relative positional encoding with eigenfunctions . The notion of positional encodings ( PEs ) in graphs is not a trivial concept , as there exists no canonical way of ordering nodes or defining axes . In this section , we investigate how eigenfunctions of the Laplacian can be used to define absolute and relative PEs in graphs , to measure physical interactions between nodes , and to enable ” hearing ” of specific sub-structures - similar to how the sound of a drum can reveal its structure . 2.1.1 Eigenvectors equate to sine functions over graphs . In the Transformer architecture , a fundamental aspect is the use of sine and cosine functions as PEs for sequences [ 36 ] . However , sinusoids can not be clearly defined for arbitrary graphs , since there is no clear notion of position along an axis . Instead , their equivalent is given by the eigenvectors φ of the graph Laplacian L. Indeed , in a Euclidean space , the Laplacian ( or Laplace ) operator corresponds to the divergence of the gradient and its eigenfunctions are sine/cosine functions , with the squared frequencies corresponding to the eigenvalues ( we sometimes interchange the two notions from here on ) . Hence , in the graph domain , the eigenvectors of the graph Laplacian are the natural equivalent of sine functions , and this intuition was employed in multiple recent works which use the eigenvectors as PEs for GNNs [ 15 ] , for directional flows [ 4 ] and for Transformers [ 14 ] . Being equivalent to sine functions , we naturally find that the Fourier Transform of a function F [ f ] applied to a graph gives F [ f ] ( λi ) = 〈f , φi〉 , where the eigenvalue is considered as a position in the Fourier domain of that graph [ 6 ] . Thus , the eigenvectors are best viewed as vectors positioned on the axis of eigenvalues rather than components of a matrix as illustrated in Figure 2 . 2.1.2 What do eigenfunctions tell us about relative positions ? . In addition to being the analog of sine functions , the eigenvectors of the Laplacian also hold important information about the physics of a system and can reveal distance metrics . This is not surprising as the Laplacian is a fundamental operator in physics and is notably used in Maxwell ’ s equations [ 16 ] and the heat diffusion [ 6 ] . In electromagnetic theory , the ( pseudo ) inverse of the Laplacian , known in mathematics as the Green ’ s function of the Laplacian [ 9 ] , represents the electrostatic potential of a given charge . In a graph , the same concept uses the pseudo-inverse of the LaplacianG and can be computed by its eigenfunctions . See equation 1 , where G ( j1 , j2 ) is the electric potential between nodes j1 and j2 , φ̂i and λ̂i are the i-th eigenvectors and eigenvalues of the symmetric LaplacianD −1 2 LD −1 2 , andD is the degree matrix , and φ̂i , j the j-th row of the vector . G ( j1 , j2 ) = d 1 2 j1 d −1 2 j2 ∑ i > 0 ( φ̂i , j1φ̂i , j2 ) 2 λ̂i ( 1 ) Further , the original solution of the heat equation given by Fourier relied on a sum of sines/cosines known as a Fourier series [ 7 ] . As eigenvectors of the Laplacian are the analogue of these functions in graphs , we find similar solutions . Knowing that heat kernels are correlated to random walks [ 6 , 4 ] , we use the interaction between two heat kernels to define in equation 2 the diffusion distance dD between nodes j1 , j2 [ 6 , 10 ] . Similarly , the biharmonic distance dB was proposed as a better measure of distances [ 28 ] . Here we use the eigenfunctions of the regular Laplacian L. d2D ( j1 , j2 ) = ∑ k > 0 e−2tλi ( φi , j1 − φi , j2 ) 2 , d2B ( j1 , j2 ) = ∑ i > 0 ( φi , j1 − φi , j2 ) 2 λ2i ( 2 ) There are a few things to note from these equations . Firstly , they highlight the importance of pairing eigenvectors and their corresponding eigenvalues when supplying information about relative positions in a graph . Secondly , we notice that the product of eigenvectors is proportional to the electrostatic interaction , while the subtraction is proportional to the diffusion and biharmonic distances . Lastly , there is a consistent pattern across all 3 equations : smaller frequencies/eigenvalues are more heavily weighted when determining distances between nodes . 2.1.3 Hearing the shape of a graph and its sub-structures . Another well-known property of eigenvalues is how they can be used to discriminate between different graph structures and sub-structures , as they can be interpreted as the frequencies of resonance of the graph . This led to the famous question about whether we can hear the shape of a drum from its eigenvalues [ 23 ] , with the same questions also applying to geometric objects [ 12 ] and 3D molecules [ 33 ] . Various success was found with the eigenfunctions being used for partial functional correspondence [ 32 ] , algorithmic understanding geometries [ 26 ] , and style correspondence [ 12 ] . Examples of eigenvectors for molecular graphs are presented in Figure 3 . 2.2 Laplace Eigenfunctions etiquette In Euclidean space and sequences , using sinusoids as PEs is trivial : we can simply select a set of frequencies , compute the sinusoids , and add or concatenate them to the input embeddings , as is done in the original Transformer [ 36 ] . However , in arbitrary graphs , reproducing these steps is not as simple since each graph has a unique set of eigenfunctions . In the following section , we present key principles from spectral graph theory to consider when constructing PEs for graphs , most of which have been overlooked by prior methods . They include normalization , the importance of the eigenvalues and their multiplicities , the number of eigenvectors being variable , and sign ambiguities . Our LPE architectures , presented in section 3 , aim to address them . Normalization . Given an eigenvalue of the Laplacian , there is an associated eigenspace of dimension greater than 1 . To make use of this information in our model , a single eigenvector has to be chosen . In our work , we use the L2 normalization since it is compatible with the definition of the Green ’ s function ( 1 ) . Thus , we will always chose eigenvectors φ such that 〈φ , φ〉 = 1 . Eigenvalues . Another fundamental aspect is that the eigenvalue associated with each eigenvector supplies valuable information . An ordering of the eigenvectors based on their eigenvalue works in sequences since the frequencies are pre-determined . However , this assumption does not work in graphs since the eigenvalues in their spectrum can vary . For example , in Figure 3 , we observe how an ordering would miss the fact that both molecules resonate at λ = 1 in different ways . Multiplicities . Another important problem with choosing eigenfunctions is the possibility of a high multiplicity of the eigenvalues , i.e . when an eigenvalue appears as a root of the characteristic polynomial more than once . In this case , the associated eigenspace may have dimension 2 or more as we can generate a valid eigenvector from any linear combination of eigenvectors with the same eigenvalue . This further complicates the problem of choosing eigenvectors for algorithmic computations and highlights the importance of having a model that can handle this ambiguity . Variable number of eigenvectors . A graphGi can have at mostNi linearly independent eigenvectors with Ni being its number of nodes . Most importantly , Ni can vary across all Gi in the dataset . Prior work [ 14 ] elected to select a fixed number k eigenvectors for each graph , where k ≤ Ni , ∀i . This produces a major bottleneck when the smallest graphs have significantly fewer nodes than the largest graphs in the dataset since a very small proportion of eigenvectors will be used for large graphs . This inevitably causes loss of information and motivates the need for a model which constructs fixed PEs of dimension k , where k does not depend on the number of eigenvectors in the graph . Sign invariance . As noted earlier , there is a sign ambiguity with the eigenvectors . With the sign of φ being independent of its normalization , we are left with a total of 2k possible combination of signs when choosing k eigenvectors of a graph . Previous work has proposed to do data augmentation by randomly flipping the sign of the eigenvectors [ 4 , 15 , 14 ] , and although it can work when k is small , it becomes intractable for large k .

Summary: This paper presents an interesting idea on designing a novel graph transformer with learning positional encoding by a spectral attention network (SAN) using the full Laplacian spectrum. Three theoretical limitations from previous efforts are overcome by this spectral attention, namely WL test and universality, over-squashing and physical interactions. Experimental results on four standard datasets demonstrate the effectiveness of the proposed SAN.