Effective Polynomial Filter Adaptation for Graph Neural Networks

1 INTRODUCTION . In this work , we are interested in the problem of classifying nodes in a graph . In this problem , a graph and features for all the nodes in the graph are made available . We are also given labels for a subset of the nodes in the graph . The learning model utilizes this information to predict labels for the remaining nodes . Graph Neural Networks ( GNNs ) perform well on such problems ( Kipf & Welling , 2017 ) . GNNs predict a node ’ s label by aggregating information from its neighbours . Thus , the performance of GNN is dependent on how well the given graph correlates with the node labels . Characterizing the correlation between the graph and node labels is an active area of research . Several metrics have been proposed including edge homophily ( Abu-El-Haija et al. , 2019a ; Zhu et al. , 2020 ) , node homophily ( Pei et al. , 2020 ) , class homophily ( Lim et al. , 2021 ) . All these metrics show that GNNs perform well when the graphs and node labels are positively correlated . For example , in the simplest case , GNNs work well when the node and its neighbours share similar labels . However , the performance can be poor if this criterion is not satisfied . Several proposed GNN models attempt to be robust to the underlying correlation between graphs and labels . Some approaches modify the aggregation mechanism ( Pei et al. , 2020 ; Zhu et al. , 2020 ; Kim & Oh , 2021 ) , while other approaches propose to estimate and leverage the label-label compatibility matrix as a prior ( Zhu et al. , 2021 ) . More recent approaches have tackled this problem from a filter learning perspective ( Bo et al. , 2021 ; Chien et al. , 2021 ) . These methods learn a filter function that operates on the eigenvalues of the graph . With eigenvalues having frequency interpretations ( Shuman et al. , 2013 ) , the filter function selectively accentuates and suppresses various frequencies as required by the task . This process enables the model to learn better embeddings , which translates to improved performance . Our interest lies in the area of modelling GNNs that can learn an effective filter . Chien et al . ( 2021 ) proposed to learn a polynomial filter . This model achieves good performance gains on several datasets with varying correlations between graphs and labels . However , we observe that learning a polynomial of a degree lower than the number of nodes can be seen as solving an over-determined system of equations . Our observation suggests needing a higher-order polynomial to model richer or more complex frequency responses for certain datasets . The design of the model proposed in Chien et al . ( 2021 ) makes it ineffective in learning a higher-order polynomial . It happens because the model attempts to overcome over-smoothing by giving smaller coefficient values to larger powers . In this paper , we address this issue and make the following contributions : • Inspired by Chien et al . ( 2021 ) , we propose to learn a polynomial filter ; however , we do so by learning a sum of polynomials over different subsets of eigenvalues . Such modelling enables learning higher-order polynomials with several low order polynomials acting over the different subsets . • Such modelling , however , requires an eigendecomposition of the graph , which can be expensive . We leverage the model in Chien et al . ( 2021 ) and the fact that there exist efficient algorithms for computing top and bottom eigen components to propose a practically efficient model . • We present theoretical analysis that suggests that our sum of polynomials approach can approximate a latent optimal filter better than a single polynomial . We also show that the space of learnable filters and graphs using our approach is larger than what is possible with Chien et al . ( 2021 ) . • Our experimental results show that our model performs better than state-of-the-art methods on several datasets , giving up to 10 % gains . These gains indicate that modelling the filter as a sum of polynomials is needed to approximate the latent filter better . We also answer other research questions through ablative studies , including the importance of the top and bottom eigenvalues , the number of top and bottom eigenvalues needed , the number of polynomials needed , and the trade-offs involved . The rest of the paper is organized as follows : We present related work in Section 2 , problem setup and motivation for the approach in Section 3 , details of the proposed approach with complexity and theoretical analysis in Section 4 and finally experimental results and empirical analysis in Section 5 . 2 RELATED WORKS . In the recent times , Graph Neural Networks ( GNNs ) have become an increasingly popular method for semi-supervised classification with graphs . Bruna et al . ( 2014 ) set the stage for early GNN models , which was then followed by various modifications ( Defferrard et al. , 2016 ; Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Veličković et al. , 2018 ) . Incorporating random walk information ( AbuEl-Haija et al. , 2019b ; a ; Li et al. , 2018 ) gave further improvements in these models , but they still suffer from over smoothing . Several proposed models attempt to overcome this problem ( Klicpera et al. , 2019 ; Lukovnikov & Fischer , 2021 ; Chamberlain et al. , 2021 ; Yang et al. , 2021 ) . Another line of research explored the question of which graphs worked well with GNNs . The critical understanding was that the performance of GNN was dependent on the correlation of the graphs with the node labels . Several approaches ( Abu-El-Haija et al. , 2019a ; Zhu et al. , 2020 ; Wang & Derr , 2021 ) considered edge homophily and proposed a robust GNN model by aggregating information from several higher-order hops . Kim & Oh ( 2021 ) also considered edge homophily and mitigated the issue by learning robust attention models . Pei et al . ( 2020 ) talks about node homophily and proposes to aggregate information from neighbours in the graph and neighbours inferred from the latent space . Zhu et al . ( 2021 ) proposes to estimate label-label compatibility matrix and uses it as a prior to update posterior belief on the labels . However , these approaches still rely on the given graphs for information like the hop neighbours etc. , which might be poor approximations of the desired neighbours with similar labels . Some of the recent approaches focused on learning filter functions that operate on the eigenvalues of the graph . Learning these filter functions can be viewed as directly adapting the graph for the desired task . Bo et al . ( 2021 ) models the filter function as an attention mechanism on the edges , which learns the difference in the proportion of low-pass and high-pass frequency signals . Chien et al . ( 2021 ) proposes a polynomial filter on the eigenvalues that directly adapts the graph for the desired task . Zheng et al . ( 2021 ) decompose the graph into low-pass and high-pass frequencies , and define a framelet based convolutional model . Our work is closely related to these lines of exploration . In this work , we propose to learn a filter function as a sum of polynomials over different subsets of the eigenvalues , enabling design of effective filters to model task-specific complex frequency responses with compute trade-offs . 3 PROBLEM SETUP AND MOTIVATION . We focus on the problem of semi-supervised node classification on a simple graph G = ( V , E ) , where V is the set of vertices and E is the set of edges . Let A ∈ { 0 , 1 } n×n be the adjacency matrix associated with G , where n = |V| is the number of nodes . Let Y be the set of all possible class labels . Let X ∈ Rn×d be the d-dimensional feature matrix for all the nodes in the graph . Given a training set of nodes D ⊂ V whose labels are known , along with A and X , our goal is to predict the labels of the remaining nodes . Let AI = A + I where I is the identity matrix . Let DAI be the degree matrix of AI and Ã = D −1/2 AI AID −1/2 AI . Let Ã = UΛUT be the eigendecomposition of Ã . Then , the spectral convolution of X on the graph A can be defined via the reference operator Ã and a filter function h operating on the eigenvalues , in the Fourier domain ( Tremblay et al. , 2017 ; Chien et al. , 2021 ) as , Z = UH ( Λ ) UTX = Udiag ( Vα ) UTX = k∑ j=1 αjÃ jX ( 1 ) where H ( Λ ) = diag ( h ( Λ ) ) . In this process , the filter function is essentially adapting the graph for the desired task at hand . The second equality follows from using a polynomial filter , H ( λ ) = ∑k i=1 αiλ i where αi ’ s are coefficients of the polynomial , k is the order of the polynomial and λ is any eigenvalue from Λ . Note that H ( Λ ) = diag ( Vα ) where V ∈ Rn×k is a Vandermonde matrix constructed using eigenvalues from Λ and α ∈ Rk×1 is the vector consisting of αj ’ s . The last equality follows from the eigendecomposition properties , enabling avoidance of eigendecomposition . It is well-known that polynomial filters can approximate any graph filter ( Shuman et al. , 2013 ; Tremblay et al. , 2017 ) . However , several practical challenges arise in learning a good polynomial filter . We briefly discuss them below . Let h∗ be the optimal filter for some given task . Learning a polynomial filter involves the approximation : h∗ ≈ Vα . Notice that this system is over-determined since the number of unknowns ( k , the size of α ) is less than the number of equations ( n , the number of nodes ) . To obtain a consistent solution , we may have to work with higher-order polynomials . Working with higher powers poses several practical challenges , and the following observations are in order . First , node features computed via ÃjX becomes indistinguishable ( see Figure 1a and A.2.1 ) . Since the node features are indistinguishable , the importance of the coefficient associated with ÃjX for the task at hand reduces significantly . These observations are in line with Theorem 4.2 in Chien et al . ( 2021 ) , where it states that as over-smoothing happens , the corresponding coefficients of the polynomial go to zero . We observe that with insignificant contributions from higher-order terms , the performance ( aka test accuracy ) does not improve with the increase in the order of the polynomial ( See Figure 1b , 1c and A.2.2 ) . Next , one can think of addressing this issue with ÃjX by directly working with full eigendecomposition . However , this approach is prohibitively expensive for large graphs . Even if we were to do this , λi ’ s are in the range of [ −1 , 1 ] and thus λji will diminish with higher powers . Therefore , it will continue to be less effective in learning coefficients of a higher power . Motivated by the above observations , we propose a novel piece-wise polynomial filter learning approach .

Summary: This paper considers the problem of designing polynomial graph filters for use in graph neural networks. The motivation for doing so stems from the notions of homophily and heterophily in labeling the nodes of a graph: the former requiring the use of low-pass filters, and the latter requiring the use of high-pass filters. Here, the authors consider the design of spline polynomials for spectral-domain filtering, as opposed to the typical ``global'' polynomials used in graph filter design. That is, they first partition the spectrum of the graph matrix into low-pass, band-pass, and high-pass components, learn polynomials over each partition, and then combine these polynomials, enforcing continuity at the boundary of each interval via a penalty function.  Of course, doing this requires a full eigendecomposition of the graph matrix, which has high complexity. To ameliorate this issue, the authors propose an efficient variant of their spline spectral filter. This approach only uses a coarse partition of the spectrum, rather than using the full eigendecomposition.