Graph Information Matters: Understanding Graph Filters from Interaction Probability

1 INTRODUCTION . Graph Neural Networks ( GNNs ) have continuously attracted interest as their promising performance in various graph learning problems . It is known that most of GNNs are intrinsically graph filters ( Kipf & Welling , 2017 ; Defferrard et al. , 2016 ; Ortega et al. , 2018 ; Nt & Maehara , 2019 ) . With the theoretical foundation of filters , there is an increasing attempt at model explanation , e.g . explaining the behavior of various GNNs in node classification . Nt & Maehara ( 2019 ) investigated the superiority of low-pass filters backed up with theoretical arguments while recent research ( Balcilar et al. , 2020 ; Chang et al. , 2020 ; Bo et al. , 2021 ) empirically revealed the weakness of GNNs with only low-pass filters in certain datasets . These contradictory views on low-pass filters pose a significant problem : Why does a filter work on one dataset but not on another ? More precisely , for a given filter , what kinds of structure and features are useful for prediction ? This makes it clear to us that in order to solve this problem , it is necessary to take into account graph information , including the graph structure , features , and labels . Existing theoretical research is mostly restricted to the investigation of filters themselves such as exploring their expressive power ( Oono & Suzuki , 2020 ; Balcilar et al. , 2020 ) , without considering their inconsistency of performance on different graphs . It is clear that structural and feature information lead to the possible inconsistency . However , there has been little explicit analysis of how graph information influences the performance of graph filters . For instance , GNNs have formulated a variety of graph filters in a heuristic manner under a suppressed homophily assumption , i.e. , nodes with similar attributes/labels tend to have connections . There remains a paucity of quantitative description of homophily until Pei et al . ( 2020 ) designed a rough index to measure it . In this paper , we establish a comprehensive theoretical analysis of the effect of structure and feature information on node label prediction to fill the gap and provide deep insights into the explanation of graph filters . We first establish a systematic investigation on graphs with an indicator in terms of homophily - the interaction probability and a distributional representation of input information - the frequency distribution . The interaction probability derived from random walk theory relates node labels with its local topology and quantifies the degree of clustering of nodes in the same/different class . We argue that interaction probability reflects the difficulty in identifying one class from others . In terms of feature information , we draw on spectral analysis representing features as frequency distributions . Furthermore , we consider the moment of frequency and build an explicit relation with graph structure . Interestingly , we find that the moment of label frequency ( noting that a one-hot label vector can be regarded as a special node feature ) is determined by interaction probability . The aforementioned preparations underpin our deep understanding of graph filters . We validate the prediction error of a graph filter under two settings : a. fixed graph structure , unravel the influence of input ( original or transformed node features ) ; b. given input , show how structure matters , and provide analysis utilizing frequency distribution and interaction probability . The main conclusions are : 1. given structure , the frequency response of an ideal graph filter should be consistent with the main frequency band of label frequency , that is , a matched frequency response is the premise of success ; 2. given input , a graph filter essentially tunes the weight of edges - failing to make a homophily degree large enough may cause an unsatisfactory prediction accuracy . These interpretations of graph filters imply a data-driven filter design principle . In addition , we apply these theoretical results to three types of filters - low-pass , high-pass , and band-pass filters with specified form . It shows that a single graph filter is hard to comply with the principle of ideal filters , especially when the homophily degree and label frequency distribution of different classes are very different . For example , when frequency distributions of labels are far from each other , it is hard to find a single filter whose frequency response can cover all the main frequency bands well . In this paper , we leverage a combination of band-pass graph filters to overcome this problem and develop a simple yet effective framework to show how to learn multiple filters depending on datasets . We empirically validate our theoretical analysis and investigate structure and feature information of benchmarks . We verify our model on a variety of datasets and explain the behavior of baselines and our model . Experimental results show that our model achieves a consistent and significant performance improvement across all benchmarks . Our main contributions are : 1.We develop a theoretical analysis of graph information based on the introduction of interaction probability and frequency distribution ; 2.We provide a deep understanding of the performance of graph filters illustrating how graph structure and input information matter ; 3.We indicate the weakness of GNNs with a single graph filter and propose a general framework to learn a data-specified filter bank which contributes to significant improvement . 2 RELATED WORK . In this paper , we focus on the analysis of graph filters in the context of graph neural networks . Since Bruna et al . ( 2014 ) defined spectral graph filters and extended convolutional operations to graphs , various spectral graph neural networks have been developed . For example , ChebNet ( Defferrard et al. , 2016 ) defines the Chebyshev polynomial filter which can be exactly localized in the k-hop neighborhood . Kipf & Welling ( 2017 ) simplified the Chebyshev filters using a first-order approximation and derived the well-known graph convolutional networks ( GCNs ) . Bianchi et al . ( 2021 ) proposed the rational auto-regressive moving average graph filters ( ARMA ) which are more powerful in modeling the localization and provide more flexible graph frequency response , however more computationally expensive and also more unstable . Very recently , Min et al . ( 2020 ) augmented conventional GCNs with geometric scattering transforms which enabled band-pass filtering of graph signals and alleviated the oversmoothing issue . In addition , most graph neural networks originally defined in the spatial domain are also found essentially connected to the spectral filtering ( Balcilar et al. , 2020 ) . By bridging the gap between spatial and spectral graph neural networks , Balcilar et al . ( 2020 ) further investigated the expressiveness of all graph neural networks from their spectral analysis . However , their analysis is limited to the spectrum coverage of a graph filter itself and lacks deeper insights into the graph-dependent performance of these filters . Another related topic is the measurement of graph homophily . Beyond the interaction probability that we define in this paper , there are some other heuristic metrics for homophily . Pei et al . ( 2020 ) defined a node homophily index to characterize their datasets and help explain their experimental results for Geom_GCN : β = 1 # nodes ∑ v # neighbors of v that have the same label as v # neighbors of v . Zhu et al . ( 2020 ) defined edge homophily ratio instead and identified a set of key designs that can boost learning from the graph structure in heterophily : h = # edges whose end nodes have same labels # edge . This edge homophily definition is sensitive to the number of classes and size of each class , and Lim et al . ( 2021 ) made a modification to alleviate this problem . Our work differentiates from these works in that we not only use our definition to characterize the graph but also directly relate it to the performance of graph filters ( or GNNs ) . 3 THEORETICAL ANALYSIS OF GRAPH INFORMATION . 3.1 NOTATION . Let Gn = ( Vn , En ) be an undirected graph with additional self-connection , where Vn = { v0 , . . . , vn−1 } is the set of nodes and En ⊂ Vn × Vn is the set of edges . Let A ∈ Rn×n be the adjacency matrix and L = D − A be the Laplacian matrix , where D is a diagonal degree matrix with Dii = ∑ j Aij . we denote Ã = D − 12AD− 1 2 , then L̃ = D− 1 2LD− 1 2 = I − Ã is the symmetric normalized Laplacian . Let ( λi , ui ) be a pair of eigenvalue and unit eigenvector of L̃ , where 0 = λ0 ≤ · · · ≤ λn−1 ≤ 2 . 3.2 PROBLEM SETTING . In this paper , we are mainly interested in node classification problems on undirected graphs . Given Gn = ( Vn , En ) , we consider T = { 0 , . . . , K−1 } as the set of all node labels . For ∀k ∈ T , we denote Ck as the set of nodes with label k and R ∈ RK×K as a size matrix which is a diagonal matrix with Rk = |Ck| . Considering single-label problems in which classes are mutually exclusive , we use onehot encoding to indicate the class label and introduce a label matrix Y ∈ Rn×K = ( y0 , . . . , yK−1 ) to represent the labels of Vn , where yk is the indicator vector of Ck . Obviously , R = Y ⊤Y , Y ⊤1 = diag ( R ) and Y 1 = 1 . A signal x on Gn can be arranged the signal values in a vector form x = ( x0 , . . . , xn−1 ) ⊤ . Particularly , labels { yk|k ∈ T } are also graph signals . 3.3 A STRUCTURE INDICATOR - INTERACTION PROBABILITY . Homophily of graphs is an implicit assumption widely leveraged in graph learning methods including GNNs . It is considered an indisputable common property of most graphs , despite its descriptive and unquantifiable definition , which introduces a variety of uncertainties . In this section , starting with the random walk , we introduce interaction probability to overcome this challenge . For a random walk on Gn , we denote P = D−1A as its transition matrix which is also a row Markov matrix . From the random walk theory , P k is the k-step transition matrix , and P kij is the probability that a random walker starting from node vi arrives at vj after k steps . For a node v and a class Cl , we denote πki ( Cl ) as the probability that a random walker starting from vi stays in Cl at the k-th step . It is trivial that πki ( Cl ) = ∑ j∈Cl P k ij with ∑ l∈T π k i ( Cl ) = 1. πki ( Cl ) demonstrates the relative preference/closeness of node vi for Cl with k-scale . To meet the homophily assumption , for vi in Cl , πki ( Cl ) is expected to gap away from others . Since πki ( Cl ) − ∑ m ̸=l π k i ( Cm ) = 2πki ( Cl ) − 1 , πki ( Cl ) can be regarded as a measure of the k-scale homophily degree of node vi . Particularly , for ∀k ∈ N and vi ∈ Cl , πki ( Cl ) = 1 means that Cl is a community and will never communicate with other classes . However , this case is rare in real graphs . Below , we investigate the homophily of a class and propose a method to measure the communication strength between two classes . Definition 3.1 ( k-step interaction probability ) . For l , m ∈ T , we define Πk as the k-step interaction probability matrix formulated as follows : Πklm = 1 Rl ∑ vi∈Cl πki ( Cm ) = 1 Rl ∑ vi∈Cl , vj∈Cm P kij = y⊤l P kym y⊤l yl ( 1 ) Πk = ( Y ⊤Y ) −1Y ⊤P kY = R−1Y ⊤P kY. ( 2 ) Πklm is the probability that a random walker from Cl arrives at Cm after k steps . Remark 1 . Obviously , Πk1 = 1 . Πklm is the mean proportion of Cm in the k-hop neighbors of nodes from Cl . Noting that rank ( Y ) = K , when K ̸= n , Y R−1Y ⊤ ̸= I , thus ( R−1Y ⊤PY ) k ̸= R−1Y ⊤P kY , i.e . ( Π ) m ̸= Πm . More generally , for an arbitrary polynomial function g , R−1Y ⊤g ( P ) Y is likely not equal to g ( R−1Y ⊤P kY ) . In the rest of paper , we write g̃ ( Π ) = R−1Y ⊤g ( P ) Y and g ( Π ) = g ( R−1Y ⊤P kY ) . For instance , if g ( · ) = ( · ) m , then g̃ ( Π ) = Πm and g ( Π ) = ( Π ) m. Also , we denote Πkll , the self-interaction probability , as π k l for short . 1-step interaction probability intuitively reflects the degree of clustering of two classes and ∑k i=1 Π i measures the strength of interaction between classes in the scale of k steps . Since P is not symmetric , Πklm ̸= Πkml . To facilitate analysis , here we propose a symmetric variant of interaction probability to identify the interactions between two classes . We denote this symmetric k-step interaction probability matrix as Π̃k , by replacing P with Ã = D− 1 2AD− 1 2 , we obtain Π̃k = R− 1 2Y ⊤ÃkY R− 1 2 . Below , we investigate the important properties of Πk and Π̃k . Proposition 3.1 . For l , m ∈ T and an arbitrary polynomial function g ( · ) , we have : a. RlΠklm +RmΠ k ml ≥ 2 √ RlRmΠ̃ k lm , where Rl is the l-th diagonal element of R ; b . ( g̃2 ( Π̃ ) ) ll ≥ ( g̃ ( Π̃ ) ll ) 2 , where g̃k ( Π̃ ) = R− 1 2Y ⊤gk ( Ã ) Y R− 1 2 . The proof can be found in Appendix B . Since Π̃k1 ̸= 1 , that is , the measure is no longer a probability measure . However , according to Prop.3.1.a ( let m = l ) , π̃kl is the lower bound of π k l , and π̃ k l = π k l when Gn is a regular graph . In the rest of theoretical analysis , we use π̃kl to measure the degree of Cl ’ s clustering . Let g ( · ) = ( · ) k , from Prop.3.1.b , we have π̃2kl ≥ ( π̃kl ) 2 . In Section.4.2 , we leverage this inequality to derive a lower bound of our prediction error and further illustrate how structure influences the performance of a given filter .

Summary: The paper presents an insightful analysis of the induced graph filters in GNNs. To accommodate the heterogeneity of graphs, the authors provide a family of novel GNNs for learning data-specific filter banks. Overall, the introduction part is well written, and the studied problem is novel and interesting. However, the technical part of this paper has some issues: (1) the contributions of this paper are over-claimed, (2) the theoretical analysis and algorithm are disconnected, (3) mistakes in the theoretical results.