BernNet: Learning Arbitrary Graph Spectral Filters via Bernstein Approximation

1 Introduction . Graph neural networks ( GNNs ) have received extensive attention from researchers due to their excellent performance on various graph learning tasks such as social analysis [ 24 , 17 , 29 ] , drug discovery [ 12 , 25 ] , traffic forecasting [ 18 , 3 , 6 ] , recommendation system [ 38 , 32 ] and computer vision [ 39 , 4 ] . Recent studies suggest that many popular GNNs operate as polynomial graph spectral filters [ 7 , 13 , 5 , 16 , 2 , 35 ] . Specifically , we denote an undirected graph with node set V and edge set E as G = ( V , E ) , whose adjacency matrix is A . Given a signal x = [ x ] ∈ Rn on the graph , where n = |V | is the number of nodes , we can formulate its graph spectral filtering operation as∑K k=0 wkL kx , wk ’ s are the filter weights , L = I −D−1/2AD−1/2 is the symmetric normalized Laplacian matrix of G , and D is the diagonal degree matrix of A . Another equivalent polynomial filtering operation is ∑K k=0 ckP kx , where P = D−1/2AD−1/2 is the normalized adjacency matrix and ck ’ s are the filter weights . * Zhewei Wei and Hongteng Xu are the corresponding authors . Work partially done at Gaoling School of Artificial Intelligence , Beijing Key Laboratory of Big Data Management and Analysis Methods , MOE Key Lab of Data Engineering and Knowledge Engineering , Renmin University of China , and Pazhou Lab , Guangzhou , 510330 , China . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We can broadly categorize the GNNs applying the above filtering operation into two classes , depending on whether they design the filter weights or learn them based on observed graphs . Some representative models in these two classes are shown below . • The GNNs driven by designing filters : GCN [ 13 ] uses a simplified first-order Chebyshev polynomial , which is proven to be a low-pass filter [ 1 , 31 , 34 , 41 ] . APPNP [ 14 ] utilizes Personalized PageRank ( PPR ) to set the filter weights and achieves a low-pass filter as well [ 15 , 41 ] . GNN-LF/HF [ 41 ] designs filter weights from the perspective of graph optimization functions , which can simulate high- and low-pass filters . • The GNNs driven by learning filters : ChebNet [ 7 ] approximates the filtering operation with Chebyshev polynomials , and learns a filter via trainable weights of the Chebyshev basis . GPR-GNN [ 5 ] learns a polynomial filter by directly performing gradient descent on the filter weights , which can derive high- or low-pass filters . ARMA [ 2 ] learns a rational filter via the family of Auto-Regressive Moving Average filters [ 21 ] . Although the above GNNs achieve some encouraging results in various graph modeling tasks , they still suffer from two major drawbacks . Firstly , most existing methods focus on designing or learning simple filters ( e.g. , low- and/or high-pass filters ) , while real-world applications often require much more complex filters such as band-rejection and comb filters . To the best of our knowledge , none of the existing work supports designing arbitrary interpretable spectral filters . The GNNs driven by learning filters can learn arbitrary filters in theory , but they can not intuitively show what filters they have learned . In other words , their interpretability is poor . For example , GPR-GNN [ 5 ] learns the filter weights wk ’ s but only proves a small subset of the learnt weight sequences corresponds to low- or high-pass filters . Secondly , the GNNs often design their filters empirically or learn the filter weights without any necessary constraints . As a result , their filter weights often have poor controllability . For example , GNN-LF/HF [ 41 ] designs its filters with a complex and non-intuitive polynomial with difficult-to-tune hyperparameters . The multi-layer GCN/SGC [ 13 , 31 ] leads to “ ill-posed ” filters ( i.e. , those deriving negative spectral responses ) . Additionally , the filters learned by GPR-GNN [ 5 ] or ChebNet [ 7 ] have a chance to be ill-posed as well . To overcome the above issues , we propose a novel graph neural network called BernNet , which provides an effective algorithmic framework for designing and learning arbitrary graph spectral filters . As illustrated in Figure 1 , for an arbitrary spectral filter h : [ 0 , 2 ] 7→ [ 0 , 1 ] over the spectrum of the symmetric normalized Laplacian L , our BernNet approximates h by a K-order Bernstein polynomial approximation , i.e. , h ( λ ) = ∑K k=0 θkb K k ( λ ) . The non-negative coefficients { θk } Kk=0 of the Bernstein basis { bKk ( λ ) } Kk=0 work as the model parameter , which can be interpreted as h ( 2k/K ) , k = 0 , . . . , K ( i.e. , the filter values uniformly sampled from [ 0 , 2 ] ) . By designing or learning the θk ’ s , we can obtain various spectral filters , whose filtering operation can be formulated as ∑K k=0 θk 1 2K ( K k ) ( 2I − L ) K−kLkx , where x is the graph signal . We further demonstrate the rationality of our BernNet from the perspective of graph optimization — any valid polynomial filers , i.e. , those polynomial functions mapping [ 0 , 2 ] to [ 0 , 1 ] , can always be expressed by our BernNet , and accordingly , the filters learned by our BernNet are always valid . Finally , we conduct experiments to demonstrate that 1 ) BernNet can learn arbitrary graph spectral filters ( e.g. , band-rejection , comb , low-band-pass , etc . ) , and 2 ) BernNet achieves superior performance on real-world datasets . 2 BernNet . 2.1 Bernstein approximation of spectral filters . Given an arbitrary filter function h : [ 0 , 2 ] 7→ [ 0 , 1 ] , let L = UΛUT denote the eigendecomposition of the symmetric normalized Laplacian matrix L , where U is the matrix of eigenvectors and Λ = diag [ λ1 , ... , λn ] is the diagonal matrix of eigenvalues . We use h ( L ) x = Uh ( Λ ) UTx = Udiag [ h ( λ1 ) , ... , h ( λn ) ] UTx ( 1 ) to denote a spectral filter on graph signal x . The key of our work is approximate h ( L ) ( or , equivalently , h ( λ ) ) . For this purpose , we leverage the Bernstein basis and Bernstein polynomial approximation defined below . Definition 2.1 ( [ 10 ] ) . ( Bernstein polynomial approximation ) Given an arbitrary continuous function f ( t ) on t ∈ [ 0 , 1 ] , the Bernstein polynomial approximation ( of order K ) for f is defined as pK ( t ) : = K∑ k=0 θk · bKk ( t ) = K∑ k=0 f ( k K ) · ( K k ) ( 1− t ) K−ktk . ( 2 ) Here , for k = 0 , ... , K , bKk ( t ) = ( K k ) ( 1− t ) K−ktk is the k-th Bernstein base , and θk = f ( kK ) is the function value at k/K , which works as the coefficient of bKk ( t ) . Lemma 2.1 ( [ 10 ] ) . Given an arbitrary continuous function f ( t ) on t ∈ [ 0 , 1 ] , let pK ( t ) denote the Bernstein approximation of f ( t ) as defined in Equation ( 2 ) . We have pK ( t ) → f ( t ) as K →∞ . For the filter function h : [ 0 , 2 ] 7→ [ 0 , 1 ] , we let t = λ2 and f ( t ) = h ( 2t ) , so that the Bernstein polynomial approximation becomes applicable , where θk = f ( k/K ) = h ( 2k/K ) and bKk ( t ) = bKk ( λ 2 ) = ( K k ) ( 1 − λ2 ) K−k ( λ2 ) k for k = 1 , ... , K. Consequently , we can approximate h ( λ ) by pK ( λ/2 ) = ∑K k=0 θk ( K k ) ( 1 − λ2 ) K−k ( λ 2 ) k = ∑K k=0 θk 1 2K ( K k ) ( 2 − λ ) K−kλk , and Lemma 2.1 ensures that pK ( λ/2 ) → h ( λ ) as K →∞ . Replacing { h ( λi ) } ni=1 with { pK ( λi/2 ) } ni=1 , we approximate the spectral filter h ( L ) in Equation ( 1 ) as Udiag [ pK ( λ1/2 ) , ... , pK ( λn/2 ) ] UT and derive the proposed BernNet . In particular , given a graph signal x , the convolutional operator of our BernNet is defined as follows : z = Udiag [ pK ( λ1/2 ) , ... , pK ( λn/2 ) ] UT︸ ︷︷ ︸ BernNet x = K∑ k=0 θk 1 2K ( K k ) ( 2I− L ) K−kLkx ( 3 ) where each coefficient θk can be either set to h ( 2k/K ) to approximate a predetermined filter h , or learnt from the graph structure and signal in an end-to-end fashion . As a natural extension of Lemma 2.1 , our BernNet owns the following proposition . Proposition 2.1 . For an arbitrary continuous filter function h : [ 0 , 2 ] → [ 0 , 1 ] , by setting θk = h ( 2k/K ) , k = 0 , . . . , K , the z in Equation ( 3 ) satisfies z→ h ( L ) x as K →∞ . Proof . According to the above derivation , we have pK ( λ/2 ) = ∑K k=0 θk ( K k ) ( 1− λ2 ) K−k ( λ 2 ) k =∑K k=0 θk 1 2K ( K k ) ( 2 − λ ) K−kλk , and Lemma 2.1 ensures that pK ( λ/2 ) → h ( λ ) as θk = h ( 2k/K ) and K →∞ . Consequently , we have z = Udiag [ pK ( λ1/2 ) , ... , pK ( λn/2 ) ] UTx→ Udiag [ h ( λ1 ) , ... , h ( λn ) ] UTx = h ( L ) as θk = h ( 2k/K ) and K →∞ . 2.2 Realizing existing filters with BernNet .. As shown in Proposition 2.1 , our BernNet can approximate arbitrary continuous spectral filters with sufficient precision . Below we give some representative examples of how our BernNet exactly realizes existing filters that are commonly used in GNNs . Note that even for those non-continuous spectral filters , e.g. , the impulse low/high/band-pass filters , our BernNet can also provide good approximations ( with sufficient large K ) . • Impulse low-pass filter h ( λ ) = δ0 ( λ ) .† We set θ0 = 1 and θk = 0 for k 6= 0 , and pK ( λ 2 ) = ( 1− λ2 ) K . Accordingly , the BernNet becomes 12K ( 2I−L ) K , deriving anK-layer linear low-pass filter . • Impulse high-pass filter h ( λ ) = δ2 ( λ ) . We set θK = 1 and θk = 0 for k 6= K , and pK ( λ 2 ) = ( λ 2 ) K . The BernNet becomes 1 2K LK , i.e. , an K-layer linear high-pass filter . • Impulse band-pass filter h ( λ ) = δ1 ( λ ) . Similarly , we set θK/2 = 1 and θk = 0 for k 6= K/2 , and pK ( λ2 ) = ( K K/2 ) ( 1− λ/2 ) K/2 ( λ/2 ) K/2 . The BernNet becomes 1 2K ( K K/2 ) ( 2I− L ) K/2LK/2 , which can be explained as stacking a K/2-layer linear low-pass filter and a K/2-layer linear high-pass filter . Obviously , K should be an even number in this case . Table 1 summarizes the design of the BernNet for the filters above . We can find that an appealing advantage of our BernNet is that its coefficients are highly correlated with the spectral property of the target filter . In particular , we can determine to pass or reject the spectral signal with λ ≈ 2kK by using a large or small θk because each Bernstein base bKk ( λ ) corresponds to a “ bump ” located at 2k K . This property provides useful guidance when designing filters , which enhances the interpretability of our BernNet .

Summary: This paper proposes BernNet, a novel graph neural network with theoretical support that provides a simple but effective scheme for designing and learning arbitrary graph spectral filters. The key property of BernNet is that 1) any valid polynomial filers that maps [0,2] to [0,1] can always be expressed by a BernNet, and 2) the filters learned by our BernNet are always non-negative. The experiments demonstrate that BernNet can learn complex spectral filters, such as band-rejection and comb filters, and it achieves superior performance in real-world graph modeling tasks. Overall, the motivation is clear and the idea is interesting. Technical contributions are clearly shown with both theoretical and empirical evidence.