BernNet: Learning Arbitrary Graph Spectral Filters via Bernstein Approximation

1 Introduction . Graph neural networks ( GNNs ) have received extensive attention from researchers due to their excellent performance on various graph learning tasks such as social analysis [ 24 , 17 , 29 ] , drug discovery [ 12 , 25 ] , traffic forecasting [ 18 , 3 , 6 ] , recommendation system [ 38 , 32 ] and computer vision [ 39 , 4 ] . Recent studies suggest that many popular GNNs operate as polynomial graph spectral filters [ 7 , 13 , 5 , 16 , 2 , 35 ] . Specifically , we denote an undirected graph with node set V and edge set E as G = ( V , E ) , whose adjacency matrix is A . Given a signal x = [ x ] ∈ Rn on the graph , where n = |V | is the number of nodes , we can formulate its graph spectral filtering operation as∑K k=0 wkL kx , wk ’ s are the filter weights , L = I −D−1/2AD−1/2 is the symmetric normalized Laplacian matrix of G , and D is the diagonal degree matrix of A . Another equivalent polynomial filtering operation is ∑K k=0 ckP kx , where P = D−1/2AD−1/2 is the normalized adjacency matrix and ck ’ s are the filter weights . * Zhewei Wei and Hongteng Xu are the corresponding authors . Work partially done at Gaoling School of Artificial Intelligence , Beijing Key Laboratory of Big Data Management and Analysis Methods , MOE Key Lab of Data Engineering and Knowledge Engineering , Renmin University of China , and Pazhou Lab , Guangzhou , 510330 , China . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We can broadly categorize the GNNs applying the above filtering operation into two classes , depending on whether they design the filter weights or learn them based on observed graphs . Some representative models in these two classes are shown below . • The GNNs driven by designing filters : GCN [ 13 ] uses a simplified first-order Chebyshev polynomial , which is proven to be a low-pass filter [ 1 , 31 , 34 , 41 ] . APPNP [ 14 ] utilizes Personalized PageRank ( PPR ) to set the filter weights and achieves a low-pass filter as well [ 15 , 41 ] . GNN-LF/HF [ 41 ] designs filter weights from the perspective of graph optimization functions , which can simulate high- and low-pass filters . • The GNNs driven by learning filters : ChebNet [ 7 ] approximates the filtering operation with Chebyshev polynomials , and learns a filter via trainable weights of the Chebyshev basis . GPR-GNN [ 5 ] learns a polynomial filter by directly performing gradient descent on the filter weights , which can derive high- or low-pass filters . ARMA [ 2 ] learns a rational filter via the family of Auto-Regressive Moving Average filters [ 21 ] . Although the above GNNs achieve some encouraging results in various graph modeling tasks , they still suffer from two major drawbacks . Firstly , most existing methods focus on designing or learning simple filters ( e.g. , low- and/or high-pass filters ) , while real-world applications often require much more complex filters such as band-rejection and comb filters . To the best of our knowledge , none of the existing work supports designing arbitrary interpretable spectral filters . The GNNs driven by learning filters can learn arbitrary filters in theory , but they can not intuitively show what filters they have learned . In other words , their interpretability is poor . For example , GPR-GNN [ 5 ] learns the filter weights wk ’ s but only proves a small subset of the learnt weight sequences corresponds to low- or high-pass filters . Secondly , the GNNs often design their filters empirically or learn the filter weights without any necessary constraints . As a result , their filter weights often have poor controllability . For example , GNN-LF/HF [ 41 ] designs its filters with a complex and non-intuitive polynomial with difficult-to-tune hyperparameters . The multi-layer GCN/SGC [ 13 , 31 ] leads to “ ill-posed ” filters ( i.e. , those deriving negative spectral responses ) . Additionally , the filters learned by GPR-GNN [ 5 ] or ChebNet [ 7 ] have a chance to be ill-posed as well . To overcome the above issues , we propose a novel graph neural network called BernNet , which provides an effective algorithmic framework for designing and learning arbitrary graph spectral filters . As illustrated in Figure 1 , for an arbitrary spectral filter h : [ 0 , 2 ] 7→ [ 0 , 1 ] over the spectrum of the symmetric normalized Laplacian L , our BernNet approximates h by a K-order Bernstein polynomial approximation , i.e. , h ( λ ) = ∑K k=0 θkb K k ( λ ) . The non-negative coefficients { θk } Kk=0 of the Bernstein basis { bKk ( λ ) } Kk=0 work as the model parameter , which can be interpreted as h ( 2k/K ) , k = 0 , . . . , K ( i.e. , the filter values uniformly sampled from [ 0 , 2 ] ) . By designing or learning the θk ’ s , we can obtain various spectral filters , whose filtering operation can be formulated as ∑K k=0 θk 1 2K ( K k ) ( 2I − L ) K−kLkx , where x is the graph signal . We further demonstrate the rationality of our BernNet from the perspective of graph optimization — any valid polynomial filers , i.e. , those polynomial functions mapping [ 0 , 2 ] to [ 0 , 1 ] , can always be expressed by our BernNet , and accordingly , the filters learned by our BernNet are always valid . Finally , we conduct experiments to demonstrate that 1 ) BernNet can learn arbitrary graph spectral filters ( e.g. , band-rejection , comb , low-band-pass , etc . ) , and 2 ) BernNet achieves superior performance on real-world datasets . 2 BernNet . 2.1 Bernstein approximation of spectral filters . Given an arbitrary filter function h : [ 0 , 2 ] 7→ [ 0 , 1 ] , let L = UΛUT denote the eigendecomposition of the symmetric normalized Laplacian matrix L , where U is the matrix of eigenvectors and Λ = diag [ λ1 , ... , λn ] is the diagonal matrix of eigenvalues . We use h ( L ) x = Uh ( Λ ) UTx = Udiag [ h ( λ1 ) , ... , h ( λn ) ] UTx ( 1 ) to denote a spectral filter on graph signal x . The key of our work is approximate h ( L ) ( or , equivalently , h ( λ ) ) . For this purpose , we leverage the Bernstein basis and Bernstein polynomial approximation defined below . Definition 2.1 ( [ 10 ] ) . ( Bernstein polynomial approximation ) Given an arbitrary continuous function f ( t ) on t ∈ [ 0 , 1 ] , the Bernstein polynomial approximation ( of order K ) for f is defined as pK ( t ) : = K∑ k=0 θk · bKk ( t ) = K∑ k=0 f ( k K ) · ( K k ) ( 1− t ) K−ktk . ( 2 ) Here , for k = 0 , ... , K , bKk ( t ) = ( K k ) ( 1− t ) K−ktk is the k-th Bernstein base , and θk = f ( kK ) is the function value at k/K , which works as the coefficient of bKk ( t ) . Lemma 2.1 ( [ 10 ] ) . Given an arbitrary continuous function f ( t ) on t ∈ [ 0 , 1 ] , let pK ( t ) denote the Bernstein approximation of f ( t ) as defined in Equation ( 2 ) . We have pK ( t ) → f ( t ) as K →∞ . For the filter function h : [ 0 , 2 ] 7→ [ 0 , 1 ] , we let t = λ2 and f ( t ) = h ( 2t ) , so that the Bernstein polynomial approximation becomes applicable , where θk = f ( k/K ) = h ( 2k/K ) and bKk ( t ) = bKk ( λ 2 ) = ( K k ) ( 1 − λ2 ) K−k ( λ2 ) k for k = 1 , ... , K. Consequently , we can approximate h ( λ ) by pK ( λ/2 ) = ∑K k=0 θk ( K k ) ( 1 − λ2 ) K−k ( λ 2 ) k = ∑K k=0 θk 1 2K ( K k ) ( 2 − λ ) K−kλk , and Lemma 2.1 ensures that pK ( λ/2 ) → h ( λ ) as K →∞ . Replacing { h ( λi ) } ni=1 with { pK ( λi/2 ) } ni=1 , we approximate the spectral filter h ( L ) in Equation ( 1 ) as Udiag [ pK ( λ1/2 ) , ... , pK ( λn/2 ) ] UT and derive the proposed BernNet . In particular , given a graph signal x , the convolutional operator of our BernNet is defined as follows : z = Udiag [ pK ( λ1/2 ) , ... , pK ( λn/2 ) ] UT︸ ︷︷ ︸ BernNet x = K∑ k=0 θk 1 2K ( K k ) ( 2I− L ) K−kLkx ( 3 ) where each coefficient θk can be either set to h ( 2k/K ) to approximate a predetermined filter h , or learnt from the graph structure and signal in an end-to-end fashion . As a natural extension of Lemma 2.1 , our BernNet owns the following proposition . Proposition 2.1 . For an arbitrary continuous filter function h : [ 0 , 2 ] → [ 0 , 1 ] , by setting θk = h ( 2k/K ) , k = 0 , . . . , K , the z in Equation ( 3 ) satisfies z→ h ( L ) x as K →∞ . Proof . According to the above derivation , we have pK ( λ/2 ) = ∑K k=0 θk ( K k ) ( 1− λ2 ) K−k ( λ 2 ) k =∑K k=0 θk 1 2K ( K k ) ( 2 − λ ) K−kλk , and Lemma 2.1 ensures that pK ( λ/2 ) → h ( λ ) as θk = h ( 2k/K ) and K →∞ . Consequently , we have z = Udiag [ pK ( λ1/2 ) , ... , pK ( λn/2 ) ] UTx→ Udiag [ h ( λ1 ) , ... , h ( λn ) ] UTx = h ( L ) as θk = h ( 2k/K ) and K →∞ . 2.2 Realizing existing filters with BernNet .. As shown in Proposition 2.1 , our BernNet can approximate arbitrary continuous spectral filters with sufficient precision . Below we give some representative examples of how our BernNet exactly realizes existing filters that are commonly used in GNNs . Note that even for those non-continuous spectral filters , e.g. , the impulse low/high/band-pass filters , our BernNet can also provide good approximations ( with sufficient large K ) . • Impulse low-pass filter h ( λ ) = δ0 ( λ ) .† We set θ0 = 1 and θk = 0 for k 6= 0 , and pK ( λ 2 ) = ( 1− λ2 ) K . Accordingly , the BernNet becomes 12K ( 2I−L ) K , deriving anK-layer linear low-pass filter . • Impulse high-pass filter h ( λ ) = δ2 ( λ ) . We set θK = 1 and θk = 0 for k 6= K , and pK ( λ 2 ) = ( λ 2 ) K . The BernNet becomes 1 2K LK , i.e. , an K-layer linear high-pass filter . • Impulse band-pass filter h ( λ ) = δ1 ( λ ) . Similarly , we set θK/2 = 1 and θk = 0 for k 6= K/2 , and pK ( λ2 ) = ( K K/2 ) ( 1− λ/2 ) K/2 ( λ/2 ) K/2 . The BernNet becomes 1 2K ( K K/2 ) ( 2I− L ) K/2LK/2 , which can be explained as stacking a K/2-layer linear low-pass filter and a K/2-layer linear high-pass filter . Obviously , K should be an even number in this case . Table 1 summarizes the design of the BernNet for the filters above . We can find that an appealing advantage of our BernNet is that its coefficients are highly correlated with the spectral property of the target filter . In particular , we can determine to pass or reject the spectral signal with λ ≈ 2kK by using a large or small θk because each Bernstein base bKk ( λ ) corresponds to a “ bump ” located at 2k K . This property provides useful guidance when designing filters , which enhances the interpretability of our BernNet .

Summary: This paper proposes a new graph neural network (GNN) architecture whose convolutional layer is capable of learning spectral graph filters of arbitrary characteristics via Bernstein approximation. Under the constraint of positive filter response, the authors show that the polynomial which approximates the solution to a generalised graph-based optimisation necessarily has the form of a non-negative linear combination of Bernstein basis, which justifies the proposed GNN convolutional layer. Experimental results demonstrate that the proposed scheme can learn complex filters and achieve competitive results on a number of benchmark datasets.