Global Node Attentions via Adaptive Spectral Filters

1 INTRODUCTION . Graph neural networks ( GNNs ) have recently demonstrated great power in graph-related learning tasks , such as node classification ( Kipf & Welling , 2017 ) , link prediction ( Zhang & Chen , 2018 ) and graph classification ( Lee et al. , 2018 ) . Most GNNs follow a message-passing architecture where , in each GNN layer , a node aggregates information from its direct neighbors indifferently . In this architecture , information from long-distance nodes is propagated and aggregated by stacking multiple GNN layers together ( Kipf & Welling , 2017 ; Velickovic et al. , 2018 ; Defferrard et al. , 2016 ) . However , this architecture underlies the assumption of local homophily , i.e . proximity of similar nodes . While this assumption seems reasonable and helps to achieve good prediction results on graphs with strong local homophily , such as citation networks and community networks ( Pei et al. , 2020 ) , it limits GNNs ’ generalizability . Particularly , determining whether a graph has strong local homophily or not is a challenge by itself . Furthermore , strong and weak local homophily can both exhibit in different parts of a graph , which makes a learning task more challenging . Pei et al . ( 2020 ) proposed a metric to measure local node homophily based on how many neighbors of a node are from the same class . Using this metric , they categorized graphs as assortative ( strong local homophily ) or disassortative ( weak local homophily ) , and showed that classical GNNs such as GCN ( Kipf & Welling , 2017 ) and GAT ( Velickovic et al. , 2018 ) perform poorly on disassortative graphs . Liu et al . ( 2020 ) further showed that GCN and GAT are outperformed by a simple multilayer perceptron ( MLP ) in node classification tasks on disassortative graphs . This is because the naive local aggregation of homophilic models brings in more noise than useful information for such graphs . These findings indicate that these GNN models perform sub-optimally when the fundamental assumption of local homophily does not hold . Based on the above observation , we argue that a well-generalized GNN should perform well on graphs , regardless of their local homophily . Furthermore , since a real-world graph can exhibit both strong and weak homophily in different node neighborhoods , a powerful GNN model should be able to aggregate node features using different strategies accordingly . For instance , in disassortative graphs where a node shares no similarity with any of its direct neighbors , such a GNN model should be able to ignore direct neighbors and reach farther to find similar nodes , or at least , resort to the node ’ s attributes to make a prediction . Since the validity of the assumption about local homophily is often unknown , such aggregation strategies should be learned from data rather than decided upfront . To circumvent this issue , in this paper , we propose a novel GNN model with global self-attention mechanism , called GNAN . Most existing attention-based aggregation architectures perform selfattention to the local neighborhood of a node ( Velickovic et al. , 2018 ) , which may add local noises in aggregation . Unlike these works , we aim to design an aggregation method that can gather informative features from both close and far-distant nodes . To achieve this , we employ graph wavelets under a relaxed condition of localization , which enables us to learn attention weights for nodes in the spectral domain . In doing so , the model can effectively capture not only local information but also global structure into node representations . To further improve the generalizability of our model , instead of using predefined spectral kernels , we propose to use multi-layer perceptrons ( MLP ) to learn the desired spectral filters without limiting their shapes . Existing works on graph wavelet transform choose wavelet filters heuristically , such as heat kernel , wave kernel and personalized page rank kernel ( Klicpera et al. , 2019b ; Xu et al. , 2019 ; Klicpera et al. , 2019a ) . They are mostly low-pass filters , which means that these models implicitly treat high-frequency components as “ noises ” and have them discarded ( Shuman et al. , 2013 ; Hammond et al. , 2011 ; Chang et al. , 2020 ) . However , this may hinder the generalizability of models since high-frequency components can carry meaningful information about local discontinuities , as analyzed in ( Shuman et al. , 2013 ) . Our model overcomes these limitations by incorporating fully learnable spectral filters into the proposed global self-attention mechanism . From a computational perspective , learning global self-attention may impose high computational overhead , particularly when graphs are large . We alleviate this problem from two aspects . First , we sparsify nodes according to their wavelet coefficients , which enables attention weights to be distributed across the graph sparsely . Second , we observed that spectral filters learned by different MLPs tend to converge to be of similar shapes . Thus , we use a single MLP to reduce redundancy among filters , where each dimension in the output corresponds to one learnable spectral filter . In addition to these , following ( Xu et al. , 2019 ; Klicpera et al. , 2019b ) , we use a fast algorithm to efficiently approximate graph wavelet transform , which has computational complexity O ( p× |E| ) , where p is the order of Chebyshev polynomials and |E| is the number of edges in a graph . To summarize , the main contributions of this work are as follows : 1 . We propose a generalized GNN model which performs well on both assortative and disassortative graphs , regardless of local node homophily . 2 . We exhibit that GNN ’ s aggregation strategy can be trained via a fully learnable spectral filter , thereby enabling feature aggregation from both close and far nodes . 3 . We show that , unlike commonly understood , higher-frequency on disassortative graphs provides meaningful information that helps improving prediction performance . We conduct extensive experiments to compare GNAN with well-known baselines on node classification tasks . The experimental results show that GNAN significantly outperforms the state-of-the-art methods on disassortative graphs where local node homophily is weak , and performs comparably with the state-of-the-art methods on assortative graphs where local node homophily is strong . This empirically verifies that GNAN is a general model for learning on different types of graphs . 2 PRELIMINARIES . Let G = ( V , E , A , x ) be an undirected graph with N nodes , where V , E , and A are the node set , edge set , and adjacency matrix of G , respectively , and x : V 7→ Rm is a graph signal function that associates each node with a feature vector . The normalized Laplacian matrix of G is defined as L = I −D−1/2AD−1/2 , whereD ∈ RN×N is the diagonal degree matrix of G. In spectral graph theory , the eigenvalues Λ = diag ( λ1 , ... , λN ) and eigenvectors U of L = UΛUH are known as the graph ’ s spectrum and spectral basis , respectively , where UH is the Hermitian transpose of U . The graph Fourier transform of x is x̂ = UHx and its inverse is x = Ux̂ . The spectrum and spectral basis carry important information on the connectivity of a graph ( Shuman et al. , 2013 ) . Intuitively , lower frequencies correspond to global and smooth information on the graph , while higher frequencies correspond to local information , discontinuities and possible noise ( Shuman et al. , 2013 ) . One can apply a spectral filter g as in Equation 1 and use graph Fourier transform to manipulate signals on a graph in various ways , such as smoothing and denoising ( Schaub & Segarra , 2018 ) , abnormally detection ( Miller et al. , 2011 ) and clustering ( Wai et al. , 2018 ) . Spectral convolutions on graphs is defined as the multiplication of a signal x with a filter g ( Λ ) in the Fourier domain , i.e . g ( L ) x = g ( UΛUH ) x = Ug ( Λ ) UHx = Ug ( Λ ) x̂ . ( 1 ) When a spectral filter is parameterized by a scale factor , which controls the radius of neighbourhood aggregation , Equation 1 is also known as the Spectral Graph Wavelet Transform ( SGWT ) ( Hammond et al. , 2011 ; Shuman et al. , 2013 ) . For example , Xu et al . ( 2019 ) uses a small scale parameter s < 2 for a heat kernel , g ( sλ ) = e−λs , to localize the wavelet at a node . 3 PROPOSED APPROACH . Graph neural networks ( GNNs ) learn lower-dimensional embeddings of nodes from graph structured data . In general , given a node , GNNs iteratively aggregate information from its neighbor nodes , and then combine the aggregated information with its own information . An embedding of node v at the kth layer of GNN is typically formulated as mv = aggregate ( { h ( k−1 ) u |u ∈ Nv } ) h ( k ) v = combine ( h ( k−1 ) v , mv ) , whereNv is the set of neighbor nodes of node v , mv is the aggregated information from the neighbors , and h ( k ) v is the embedding of the node v at the kth layer ( h ( 0 ) v = xv ) . The embedding hnv of the node v at the final layer is then used for some prediction tasks . In most GNNs , Nv is restricted to a set of one-hop neighbors of node v. Therefore , one needs to stack multiple aggregation layers in order to collect the information from more than one-hop neighborhood within this architecture . Adaptive spectral filters . Instead of stacking multiple aggregation layers , we introduce a spectral attention layer that rewires a graph based on spectral graph wavelets . A spectral graph wavelet ψv at node v is a modulation in the spectral domain of signals centered around the node v , given by an N -dimensional vector ψv = Ug ( Λ ) U Hδv , ( 2 ) where g ( · ) is a spectral filter and δv is a one-hot vector for node v. The common choice of a spectral filter is heat kernel . A wavelet coefficient ψvu computed from a heat kernel can be interpreted as the amount of energy that node v has received from node u in its local neighborhood . In this work , instead of using pre-defined localized kernels , we use multilayer perceptrons ( MLP ) to learn spectral filters . With learnable spectral kernels , we obtain wavelet coefficients ψv = Udiag ( MLP ( Λ ) ) UHδv . ( 3 ) Similar to that of a heat kernel , the wavelet coefficient with a learnable spectral filter ψvu can be understood as the amount of energy that is distributed from node v to node u , under the conditions regulated by the spectral filter . Note that we use the terminology wavelet and spectral filter interchangeably as we have relaxed the wavelet definition from ( Hammond et al. , 2011 ) so that learnable spectral filters in our work are not necessarily localized in the spectral and spatial domains . Equation 3 requires the eigen-decomposition of a Laplacian matrix , which is expensive and infeasible for large graphs . We follow Xu et al . ( 2019 ) ; Klicpera et al . ( 2019b ) to approximate graph wavelet transform using Chebyshev polynomials ( Shuman et al. , 2013 ) ( see Appendix A for details ) . Global self-attention . Unlike the previous work ( Xu et al. , 2019 ) where wavelet coefficients are directly used to compute node embeddings , we normalize wavelet coefficients through a softmax layer av = softmax ( ψv ) , where av ∈ RN is an attention weight vector . With attention weights , an update layer is then formalized as h ( k ) v = σ ( N∑ u=1 avuh ( k−1 ) u W ( k ) ) , ( 4 ) where W ( k ) is a weight matrix shared across all nodes in the kth layer and σ is ELU nonlinear activation . Unlike heat kernel , the wavelet coefficient with a learnable spectral kernel is not localized . Hence , our work can actively aggregate information from far-distant nodes . Note that the update layer is not divided into aggregation and combine steps in our work . Instead , we compute the attention avv directly from a spectral filter . Sparsified node attentions . With predefined localized spectral filters such as a heat kernel , most of wavelet coefficients are zero due to their locality . In our work , spectral filters are fully learned from data , and consequently attention weights obtained from learnable spectral filters do not impose any sparsity . This means that to perform an aggregation operation we need to retrieve all possible nodes in a graph , which is time consuming with large graphs . From our experiments , we observe that most of attention weights are negligible after softmax . Thus , we consider two sparsification techniques : 1 . Discard the entries of wavelet bases that are below a threshold t , i.e . ψ̄vu = { ψvu if ψvu > t −∞ otherwise . ( 5 ) The threshold t can be easily applied on all entries of wavelet bases . However , it offers little guarantee on attention sparsity since attention weights may vary , depending on the learning process of spectral filters and the characteristics of different datasets , as will be further discussed in Section 4.2 . 2 . Keep only the largest k entries of wavelet bases for each node , i.e . ψ̄vu = { ψvu if ψvu ∈ topK ( { ψv0 , ... , ψvN } , k ) −∞ otherwise , ( 6 ) where topK is a partial sorting function that returns the largest k entries from a set of wavelet bases { ψv0 , ... , ψvN } . This technique guarantees attention sparsity such that the embedding of each node can be aggregated from at most k other nodes . However , it takes more computational overhead to sort entries since topK has a time complexity of O ( N +k logN ) . The resulting ψ̄ from either of the above techniques is then fed into the softmax layer to compute attention weights . The experiments for comparing these techniques will be discussed in Section 4.2 . We adopt multi-head attention to model multiple spectral filters . Each attention head aggregates node information with a different spectral filter , and the aggregated embedding is concatenated before being sent to the next layer . We can allocate an independent MLP for each of attention heads ; however , we found independent MLPs tend to learn spectral filters of similar shapes . Hence , we adopt a single MLP : RN → RN×M , where M is the number of attention heads , and each column of the output corresponds to one adaptive spectral filter . We name the multi-head spectral attention architecture as a global node attention network ( GNAN ) . The design of GNAN is easily generalizable , and many existing GNNs can be expressed as special cases of GNAN ( see Appendix D ) . Figure 1 illustrates how GNAN works with two attention heads learned from the CITESEER dataset . As shown in the illustration , the MLP learns adaptive filters such as low-band pass and high-band pass filters . A low-band pass filter assigns high attention weights in local neighborhoods , while a high-band pass filter assigns high attention weights on far-distant nodes , which can not be captured by a one-hop aggregation scheme .

Summary: In this paper, the authors study the problem of GCN for disassortative graphs. The authors proposed the GNAN method to allow attention on distant nodes indeed of limiting to local neighbors. The authors generalized the idea of graph wavelet with MLP to generate the attention score and utilized it to generate multiple attention heads. The authors carried out experiments on several real-world networks (4 assortative and 3 disassortative) with comparison to several state-of-art GCN methods.