Effective Polynomial Filter Adaptation for Graph Neural Networks

1 INTRODUCTION . In this work , we are interested in the problem of classifying nodes in a graph . In this problem , a graph and features for all the nodes in the graph are made available . We are also given labels for a subset of the nodes in the graph . The learning model utilizes this information to predict labels for the remaining nodes . Graph Neural Networks ( GNNs ) perform well on such problems ( Kipf & Welling , 2017 ) . GNNs predict a node ’ s label by aggregating information from its neighbours . Thus , the performance of GNN is dependent on how well the given graph correlates with the node labels . Characterizing the correlation between the graph and node labels is an active area of research . Several metrics have been proposed including edge homophily ( Abu-El-Haija et al. , 2019a ; Zhu et al. , 2020 ) , node homophily ( Pei et al. , 2020 ) , class homophily ( Lim et al. , 2021 ) . All these metrics show that GNNs perform well when the graphs and node labels are positively correlated . For example , in the simplest case , GNNs work well when the node and its neighbours share similar labels . However , the performance can be poor if this criterion is not satisfied . Several proposed GNN models attempt to be robust to the underlying correlation between graphs and labels . Some approaches modify the aggregation mechanism ( Pei et al. , 2020 ; Zhu et al. , 2020 ; Kim & Oh , 2021 ) , while other approaches propose to estimate and leverage the label-label compatibility matrix as a prior ( Zhu et al. , 2021 ) . More recent approaches have tackled this problem from a filter learning perspective ( Bo et al. , 2021 ; Chien et al. , 2021 ) . These methods learn a filter function that operates on the eigenvalues of the graph . With eigenvalues having frequency interpretations ( Shuman et al. , 2013 ) , the filter function selectively accentuates and suppresses various frequencies as required by the task . This process enables the model to learn better embeddings , which translates to improved performance . Our interest lies in the area of modelling GNNs that can learn an effective filter . Chien et al . ( 2021 ) proposed to learn a polynomial filter . This model achieves good performance gains on several datasets with varying correlations between graphs and labels . However , we observe that learning a polynomial of a degree lower than the number of nodes can be seen as solving an over-determined system of equations . Our observation suggests needing a higher-order polynomial to model richer or more complex frequency responses for certain datasets . The design of the model proposed in Chien et al . ( 2021 ) makes it ineffective in learning a higher-order polynomial . It happens because the model attempts to overcome over-smoothing by giving smaller coefficient values to larger powers . In this paper , we address this issue and make the following contributions : • Inspired by Chien et al . ( 2021 ) , we propose to learn a polynomial filter ; however , we do so by learning a sum of polynomials over different subsets of eigenvalues . Such modelling enables learning higher-order polynomials with several low order polynomials acting over the different subsets . • Such modelling , however , requires an eigendecomposition of the graph , which can be expensive . We leverage the model in Chien et al . ( 2021 ) and the fact that there exist efficient algorithms for computing top and bottom eigen components to propose a practically efficient model . • We present theoretical analysis that suggests that our sum of polynomials approach can approximate a latent optimal filter better than a single polynomial . We also show that the space of learnable filters and graphs using our approach is larger than what is possible with Chien et al . ( 2021 ) . • Our experimental results show that our model performs better than state-of-the-art methods on several datasets , giving up to 10 % gains . These gains indicate that modelling the filter as a sum of polynomials is needed to approximate the latent filter better . We also answer other research questions through ablative studies , including the importance of the top and bottom eigenvalues , the number of top and bottom eigenvalues needed , the number of polynomials needed , and the trade-offs involved . The rest of the paper is organized as follows : We present related work in Section 2 , problem setup and motivation for the approach in Section 3 , details of the proposed approach with complexity and theoretical analysis in Section 4 and finally experimental results and empirical analysis in Section 5 . 2 RELATED WORKS . In the recent times , Graph Neural Networks ( GNNs ) have become an increasingly popular method for semi-supervised classification with graphs . Bruna et al . ( 2014 ) set the stage for early GNN models , which was then followed by various modifications ( Defferrard et al. , 2016 ; Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Veličković et al. , 2018 ) . Incorporating random walk information ( AbuEl-Haija et al. , 2019b ; a ; Li et al. , 2018 ) gave further improvements in these models , but they still suffer from over smoothing . Several proposed models attempt to overcome this problem ( Klicpera et al. , 2019 ; Lukovnikov & Fischer , 2021 ; Chamberlain et al. , 2021 ; Yang et al. , 2021 ) . Another line of research explored the question of which graphs worked well with GNNs . The critical understanding was that the performance of GNN was dependent on the correlation of the graphs with the node labels . Several approaches ( Abu-El-Haija et al. , 2019a ; Zhu et al. , 2020 ; Wang & Derr , 2021 ) considered edge homophily and proposed a robust GNN model by aggregating information from several higher-order hops . Kim & Oh ( 2021 ) also considered edge homophily and mitigated the issue by learning robust attention models . Pei et al . ( 2020 ) talks about node homophily and proposes to aggregate information from neighbours in the graph and neighbours inferred from the latent space . Zhu et al . ( 2021 ) proposes to estimate label-label compatibility matrix and uses it as a prior to update posterior belief on the labels . However , these approaches still rely on the given graphs for information like the hop neighbours etc. , which might be poor approximations of the desired neighbours with similar labels . Some of the recent approaches focused on learning filter functions that operate on the eigenvalues of the graph . Learning these filter functions can be viewed as directly adapting the graph for the desired task . Bo et al . ( 2021 ) models the filter function as an attention mechanism on the edges , which learns the difference in the proportion of low-pass and high-pass frequency signals . Chien et al . ( 2021 ) proposes a polynomial filter on the eigenvalues that directly adapts the graph for the desired task . Zheng et al . ( 2021 ) decompose the graph into low-pass and high-pass frequencies , and define a framelet based convolutional model . Our work is closely related to these lines of exploration . In this work , we propose to learn a filter function as a sum of polynomials over different subsets of the eigenvalues , enabling design of effective filters to model task-specific complex frequency responses with compute trade-offs . 3 PROBLEM SETUP AND MOTIVATION . We focus on the problem of semi-supervised node classification on a simple graph G = ( V , E ) , where V is the set of vertices and E is the set of edges . Let A ∈ { 0 , 1 } n×n be the adjacency matrix associated with G , where n = |V| is the number of nodes . Let Y be the set of all possible class labels . Let X ∈ Rn×d be the d-dimensional feature matrix for all the nodes in the graph . Given a training set of nodes D ⊂ V whose labels are known , along with A and X , our goal is to predict the labels of the remaining nodes . Let AI = A + I where I is the identity matrix . Let DAI be the degree matrix of AI and Ã = D −1/2 AI AID −1/2 AI . Let Ã = UΛUT be the eigendecomposition of Ã . Then , the spectral convolution of X on the graph A can be defined via the reference operator Ã and a filter function h operating on the eigenvalues , in the Fourier domain ( Tremblay et al. , 2017 ; Chien et al. , 2021 ) as , Z = UH ( Λ ) UTX = Udiag ( Vα ) UTX = k∑ j=1 αjÃ jX ( 1 ) where H ( Λ ) = diag ( h ( Λ ) ) . In this process , the filter function is essentially adapting the graph for the desired task at hand . The second equality follows from using a polynomial filter , H ( λ ) = ∑k i=1 αiλ i where αi ’ s are coefficients of the polynomial , k is the order of the polynomial and λ is any eigenvalue from Λ . Note that H ( Λ ) = diag ( Vα ) where V ∈ Rn×k is a Vandermonde matrix constructed using eigenvalues from Λ and α ∈ Rk×1 is the vector consisting of αj ’ s . The last equality follows from the eigendecomposition properties , enabling avoidance of eigendecomposition . It is well-known that polynomial filters can approximate any graph filter ( Shuman et al. , 2013 ; Tremblay et al. , 2017 ) . However , several practical challenges arise in learning a good polynomial filter . We briefly discuss them below . Let h∗ be the optimal filter for some given task . Learning a polynomial filter involves the approximation : h∗ ≈ Vα . Notice that this system is over-determined since the number of unknowns ( k , the size of α ) is less than the number of equations ( n , the number of nodes ) . To obtain a consistent solution , we may have to work with higher-order polynomials . Working with higher powers poses several practical challenges , and the following observations are in order . First , node features computed via ÃjX becomes indistinguishable ( see Figure 1a and A.2.1 ) . Since the node features are indistinguishable , the importance of the coefficient associated with ÃjX for the task at hand reduces significantly . These observations are in line with Theorem 4.2 in Chien et al . ( 2021 ) , where it states that as over-smoothing happens , the corresponding coefficients of the polynomial go to zero . We observe that with insignificant contributions from higher-order terms , the performance ( aka test accuracy ) does not improve with the increase in the order of the polynomial ( See Figure 1b , 1c and A.2.2 ) . Next , one can think of addressing this issue with ÃjX by directly working with full eigendecomposition . However , this approach is prohibitively expensive for large graphs . Even if we were to do this , λi ’ s are in the range of [ −1 , 1 ] and thus λji will diminish with higher powers . Therefore , it will continue to be less effective in learning coefficients of a higher power . Motivated by the above observations , we propose a novel piece-wise polynomial filter learning approach .

Summary:  This paper proposes PP-GNN, a novel graph neural network that learns multiple adaptive polynomial filters acting on different subsets of the eigenvalues. The authors combine GPR-GNN with existing efficient algorithms for generating top and bottom eigen components to reduce the expensive complexity of eigendecomposition. They show that the piece-wise polynomial method can approximate a latent optimal filter better than a single polynomial in theory.