Adaptive Stacked Graph Filter

We study Graph Convolutional Networks ( GCN ) from the graph signal processing viewpoint by addressing a difference between learning graph filters with fullyconnected weights versus trainable polynomial coefficients . We find that by stacking graph filters with learnable polynomial parameters , we can build a highly adaptive and robust vertex classification model . Our treatment here relaxes the low-frequency ( or equivalently , high homophily ) assumptions in existing vertex classification models , resulting a more ubiquitous solution in terms of spectral properties . Empirically , by using only one hyper-parameter setting , our model achieves strong results on most benchmark datasets across the frequency spectrum . 1 INTRODUCTION . The semi-supervised vertex classification problem ( Weston et al. , 2012 ; Yang et al. , 2016 ) in attributed graphs has become one of the most fundamental machine learning problems in recent years . This problem is often associated with its most popular recent solution , namely Graph Convolutional Networks ( Kipf & Welling , 2017 ) . Since the GCN proposal , there has been a vast amount of research to improve its scalability ( Hamilton et al. , 2017 ; Chen et al. , 2018 ; Wu et al. , 2019 ) as well as performance ( Liao et al. , 2019 ; Li et al. , 2019 ; Pei et al. , 2020 ) . Existing vertex classification models often ( implicitly ) assume that the graph has large vertex homophily ( Pei et al. , 2020 ) , or equivalently , low-frequency property ( Li et al. , 2019 ; Wu et al. , 2019 ) ; see Section 2.1 for graph frequency . However , this assumption is not true in general . For instance , let us take the Wisconsin dataset ( Table 1 ) , which captures a network of students , faculty , staff , courses , and projects . These categories naturally exhibit different frequency patterns1 . Connections between people are often low-frequency , while connections between topics and projects are often midrange . This problem becomes apparent as GCN-like models show low accuracies on this dataset ; for example , see ( Pei et al. , 2020 ; Chen et al. , 2020b ; Liu et al. , 2020 ) . This paper aims at establishing a GCN model for the vertex classification problem ( Definition 1 ) that does not rely on any frequency assumption . Such a model can be applied to ubiquitous datasets without any hyper-parameter tuning for the graph structure . Contributions . By observing the relation between label frequency and performance of existing GCN-like models , we propose to learn the graph filters coefficients directly rather than learning the MLP part of a GCN-like layer . We use filter stacking to implement a trainable graph filter , which is capable of learning any filter function . Our stacked filter construction with novel learnable filter parameters is easy to implement , sufficiently expressive , and less sensitive to the filters ’ degree . By using only one hyper-parameter setting , we show that our model is more adaptive than existing work on a wide range of benchmark datasets . The rest of our paper is organized as follows . Section 2 introduces notations and analytical tools . Section 3 provides insights into the vertex classification problem and motivations to our model ’ s design . Section 4 presents an implementation of our model . Section 5 summarizes related literature with a focus on graph filters and state-of-the-art models . Section 6 compares our model and other existing methods empirically . We also provide additional experimental results in Appendix A . 1 “ Frequency ” is an equivalent concept to “ homophily ” and will be explained in Section 2 . 2 PRELIMINARIES . We consider a simple undirected graph G = ( V , E ) , where V = { 1 , . . . , n } is a set of n vertices and E ⊆ V × V is a set of edges . A graph G is called an attributed graph , denoted by G ( X ) , when it is associated with a vertex feature mapping X : V 7→ Rd , where d is the dimension of the features . We define the following vertex classification problem , also known in the literature as the semi-supervised vertex classification problem ( Yang et al. , 2016 ) . Definition 1 ( Vertex Classification Problem ) . We are given an attributed graph G ( X ) , a set of training vertices Vtr ⊂ V , training labels Ytr : Vtr → C , and label set C. The task is to find a model h : V → C using the training data ( Vtr , Ytr ) that approximates the true labeling function Y : V → C. Let A be the adjacency matrix of the graph G , i.e. , Ai , j = 1 if ( i , j ) ∈ E and 0 otherwise . Let di = ∑ j Aij be the degree of vertex i ∈ V , and let D = diag ( d1 , . . . , dn ) be the n × n diagonal matrix of degrees . Let L = D −A be the combinatorial graph Laplacian . Let L = D−1/2LD−1/2 be the symmetric normalized graph Laplacian . We mainly focus on the symmetric normalized graph Laplacian due to its interesting spectral properties : ( 1 ) its eigenvalues range from 0 to 2 ; and ( 2 ) the spectral properties can be compared between different graphs ( Chung & Graham , 1997 ) . In recent literature , the normalized adjacency matrix with added self-loops , Ã = I −L+ c , is often used as the propagation matrix , where c is some diagonal matrix . 2.1 GRAPH FREQUENCY . Graph signal processing ( Shuman et al. , 2012 ) extends “ frequency ” concepts in the classical signal processing to graphs using the graph Laplacian . Let L = UΛU > be the eigendecomposition of the Laplacian , where U ∈ Rn×n is the orthogonal matrix consists of the orthonormal eigenvectors of L and Λ is the diagonal matrix of eigenvalues . Then , we can regard each eigenvector uk as a “ oscillation pattern ” and its eigenvalue λk as the “ frequency ” of the oscillation . This intuition is supported by the Rayleigh quotient as follows . r ( L , x ) , x > Lx x > x = ∑ u∼v Lu , v ( x ( u ) − x ( v ) ) 2∑ u∈V x ( u ) 2 . ( 1 ) where ∑ u∼v sums over all unordered pairs for which u and v are adjacent , x ( u ) denotes the entry of vector x corresponding to vertex u , and Lu , v is the ( u , v ) -entry of L. From the definition we see that r ( x ) is non-negative and L is positive semi-definite . r ( x ) is also known as a variational characterization of eigenvalues of L ( Horn & Johnson , 2012 , Chapter 4 ) , hence 0 ≤ r ( x ) ≤ 2 for any non-zero real vector x . We use the notation r ( x ) to denote the Rayleigh quotient when the normalized graph Laplacian is clear from context . The Rayleigh quotient r ( x ) measures how the data x is oscillating . Hence , in this study , we use the term “ frequency ” and the “ Rayleigh quotient ” interchangeably . By the definition , the eigenvector ui has the frequency of λi . The labeling y of the vertices is low-frequency if the adjacent vertices are more likely to have the same label . This is a common assumption made by the spectral clustering algorithms ( Shi & Malik , 2000 ; Ng et al. , 2002 ; Shaham et al. , 2018 ) . Commonly used terms , homophily and heterophily , used in network science , correspond to low-frequency and high-frequency , respectively . 2.2 GRAPH FILTERING . In classical signal processing , a given signal is processed by filters in order to remove unwanted interference . Here , we first design a frequency response f ( λ ) of the filter , and then apply the filter to the signal in the sense that each frequency component x̂ ( λ ) of the data is modulated as f ( λ ) x̂ ( λ ) . Graph signal processing extends this concept as follows . Same as in classical signal processing , we design a filter f ( λ ) . Then , we represent a given graph signal x ∈ R|V | as a linear combination of the eigenvectors as x = ∑ i xiui . Then , we modulate each frequency component by f ( λ ) as x = ∑ i f ( λi ) xiui . An important fact is that this can be done without performing the eigendecomposition explicitly . Let f ( L ) be the matrix function induced from f ( λ ) . Then , the filter is represented by f ( L ) x . As an extension of signal processing , graph signal processing deals with signals defined on graphs . In definition 1 , each column of the feature matrix X ∈ Rn×d is a “ graph signal ” . Let L = UΛU > be the eigendecomposition where U ∈ Rn×n consists of orthonormal eigenvectors . Signal X is filtered by function f of the eigenvalues as follow . X̄ = Uf ( Λ ) U > X = f ( L ) X ( 2 ) In general , different implementations of f ( L ) lead to different graph convolution models . For instance , GCN and SGC ( Wu et al. , 2019 ) are implemented by f ( L ) = ( I−L+ ( D+ I ) −1/2L ( D+ I ) −1/2 ) k , where the constant term stems from the fact that self-loops are added to vertices and k is the filter order . Generally , the underlying principle is to learn or construct the appropriate filter function f such that it transforms X into a more expressive representation . The filter in GCN is called a low-pass filter because it amplifies low-frequency components ( Li et al. , 2018 ; NT & Maehara , 2019 ) . 3 SPECTRAL PROPERTIES OF FILTERS . Towards building a ubiquitous solution , we take an intermediate step to study the vertex classification problem . Similar to the unsupervised clustering problem , an ( implicit ) low-frequency assumption is commonly made . However , the semi-supervised vertex classification problem is more involved because vertex labels can have complicated non-local patterns . Table 1 shows three groups of datasets , each with different label frequency ranges . Notably , WebKB datasets ( Wisconsin , Cornell , Texas ) have mixed label frequencies ; some labels have low frequencies while others have midrange frequencies . Therefore , in order to relax the frequency assumptions , we need to learn the filtering function f ( λ ) in a similar way as proposed by Defferrard et al . ( 2016 ) . The filtering function f ( λ ) is often approximated using a polynomial of the graph Laplacian as f ( L ) ≈ poly ( L ) = K∑ i=0 θiLi . ( 3 ) Because polynomials can uniformly approximate any real continuous function on a compact interval ( see , e.g. , ( Brosowski & Deutsch , 1981 ) ) , such approximation scheme is well-justified . Kipf & Welling ( 2017 ) derived their GCN formulation as follows . In their equation 5 , they approximated a graph filter gθ by Chebyshev polynomials Tk as gθ ∗ x ≈ K∑ k=0 θkTk ( D −1/2AD−1/2 ) x . ( 4 ) Then , they took the first two terms and shared the parameters as θ0 = −θ1 to obtain their equation 7 : gθ ∗ x ≈ θ ( IN +D −1/2AD−1/2 ) x ≈ θ ( 2IN − L ) ( 5 ) Finally , they extended a scalar θ to a matrix Θ to accommodate multiple feature dimensions as Z = D̃−1/2ÃD̃−1/2XΘ ( 6 ) Kipf & Welling ( 2017 ) claimed that the weight matrix Θ can learn different filters , and subsequent works ( e.g. , ( Veličković et al. , 2018 ; Spinelli et al. , 2020 ; Chen et al. , 2020b ) ) also learned filters by Θ . However , neither in theory nor practice it is the case ( Oono & Suzuki , 2020 ) . As the construction suggest , a GCN layer only represents a filter of the form f ( λ ) ≈ 2− λ . To properly learn different graph filters , we should learn the multiplying parameters θ0 , θ1 , . . . , θK in equation 3 . In the next section , we propose a learning model which directly learns these multiplying parameters .

Summary: This paper addresses the problem of vertex classification using a new Graph Convolutional Neural Network (NN) architecture. The linear operator within each of the layers of the GNNN is formed by a polynomial graph filter (i.e., a matrix polynomial of either the adjacency or the Laplacian novelty). Rather than working on the frequency domain, the paper focuses on learning the polynomial coefficients of the filter on the vertex domain. The key novelty is the consideration of a stack architecture for which the polynomial filter is formed by the successive application (i.e., matrix multiplication) of filters of order one. Numerical experiments with real datasets showcase the merits, including superior classification performance, of the proposed architecture.