Graph Information Matters: Understanding Graph Filters from Interaction Probability

1 INTRODUCTION . Graph Neural Networks ( GNNs ) have continuously attracted interest as their promising performance in various graph learning problems . It is known that most of GNNs are intrinsically graph filters ( Kipf & Welling , 2017 ; Defferrard et al. , 2016 ; Ortega et al. , 2018 ; Nt & Maehara , 2019 ) . With the theoretical foundation of filters , there is an increasing attempt at model explanation , e.g . explaining the behavior of various GNNs in node classification . Nt & Maehara ( 2019 ) investigated the superiority of low-pass filters backed up with theoretical arguments while recent research ( Balcilar et al. , 2020 ; Chang et al. , 2020 ; Bo et al. , 2021 ) empirically revealed the weakness of GNNs with only low-pass filters in certain datasets . These contradictory views on low-pass filters pose a significant problem : Why does a filter work on one dataset but not on another ? More precisely , for a given filter , what kinds of structure and features are useful for prediction ? This makes it clear to us that in order to solve this problem , it is necessary to take into account graph information , including the graph structure , features , and labels . Existing theoretical research is mostly restricted to the investigation of filters themselves such as exploring their expressive power ( Oono & Suzuki , 2020 ; Balcilar et al. , 2020 ) , without considering their inconsistency of performance on different graphs . It is clear that structural and feature information lead to the possible inconsistency . However , there has been little explicit analysis of how graph information influences the performance of graph filters . For instance , GNNs have formulated a variety of graph filters in a heuristic manner under a suppressed homophily assumption , i.e. , nodes with similar attributes/labels tend to have connections . There remains a paucity of quantitative description of homophily until Pei et al . ( 2020 ) designed a rough index to measure it . In this paper , we establish a comprehensive theoretical analysis of the effect of structure and feature information on node label prediction to fill the gap and provide deep insights into the explanation of graph filters . We first establish a systematic investigation on graphs with an indicator in terms of homophily - the interaction probability and a distributional representation of input information - the frequency distribution . The interaction probability derived from random walk theory relates node labels with its local topology and quantifies the degree of clustering of nodes in the same/different class . We argue that interaction probability reflects the difficulty in identifying one class from others . In terms of feature information , we draw on spectral analysis representing features as frequency distributions . Furthermore , we consider the moment of frequency and build an explicit relation with graph structure . Interestingly , we find that the moment of label frequency ( noting that a one-hot label vector can be regarded as a special node feature ) is determined by interaction probability . The aforementioned preparations underpin our deep understanding of graph filters . We validate the prediction error of a graph filter under two settings : a. fixed graph structure , unravel the influence of input ( original or transformed node features ) ; b. given input , show how structure matters , and provide analysis utilizing frequency distribution and interaction probability . The main conclusions are : 1. given structure , the frequency response of an ideal graph filter should be consistent with the main frequency band of label frequency , that is , a matched frequency response is the premise of success ; 2. given input , a graph filter essentially tunes the weight of edges - failing to make a homophily degree large enough may cause an unsatisfactory prediction accuracy . These interpretations of graph filters imply a data-driven filter design principle . In addition , we apply these theoretical results to three types of filters - low-pass , high-pass , and band-pass filters with specified form . It shows that a single graph filter is hard to comply with the principle of ideal filters , especially when the homophily degree and label frequency distribution of different classes are very different . For example , when frequency distributions of labels are far from each other , it is hard to find a single filter whose frequency response can cover all the main frequency bands well . In this paper , we leverage a combination of band-pass graph filters to overcome this problem and develop a simple yet effective framework to show how to learn multiple filters depending on datasets . We empirically validate our theoretical analysis and investigate structure and feature information of benchmarks . We verify our model on a variety of datasets and explain the behavior of baselines and our model . Experimental results show that our model achieves a consistent and significant performance improvement across all benchmarks . Our main contributions are : 1.We develop a theoretical analysis of graph information based on the introduction of interaction probability and frequency distribution ; 2.We provide a deep understanding of the performance of graph filters illustrating how graph structure and input information matter ; 3.We indicate the weakness of GNNs with a single graph filter and propose a general framework to learn a data-specified filter bank which contributes to significant improvement . 2 RELATED WORK . In this paper , we focus on the analysis of graph filters in the context of graph neural networks . Since Bruna et al . ( 2014 ) defined spectral graph filters and extended convolutional operations to graphs , various spectral graph neural networks have been developed . For example , ChebNet ( Defferrard et al. , 2016 ) defines the Chebyshev polynomial filter which can be exactly localized in the k-hop neighborhood . Kipf & Welling ( 2017 ) simplified the Chebyshev filters using a first-order approximation and derived the well-known graph convolutional networks ( GCNs ) . Bianchi et al . ( 2021 ) proposed the rational auto-regressive moving average graph filters ( ARMA ) which are more powerful in modeling the localization and provide more flexible graph frequency response , however more computationally expensive and also more unstable . Very recently , Min et al . ( 2020 ) augmented conventional GCNs with geometric scattering transforms which enabled band-pass filtering of graph signals and alleviated the oversmoothing issue . In addition , most graph neural networks originally defined in the spatial domain are also found essentially connected to the spectral filtering ( Balcilar et al. , 2020 ) . By bridging the gap between spatial and spectral graph neural networks , Balcilar et al . ( 2020 ) further investigated the expressiveness of all graph neural networks from their spectral analysis . However , their analysis is limited to the spectrum coverage of a graph filter itself and lacks deeper insights into the graph-dependent performance of these filters . Another related topic is the measurement of graph homophily . Beyond the interaction probability that we define in this paper , there are some other heuristic metrics for homophily . Pei et al . ( 2020 ) defined a node homophily index to characterize their datasets and help explain their experimental results for Geom_GCN : β = 1 # nodes ∑ v # neighbors of v that have the same label as v # neighbors of v . Zhu et al . ( 2020 ) defined edge homophily ratio instead and identified a set of key designs that can boost learning from the graph structure in heterophily : h = # edges whose end nodes have same labels # edge . This edge homophily definition is sensitive to the number of classes and size of each class , and Lim et al . ( 2021 ) made a modification to alleviate this problem . Our work differentiates from these works in that we not only use our definition to characterize the graph but also directly relate it to the performance of graph filters ( or GNNs ) . 3 THEORETICAL ANALYSIS OF GRAPH INFORMATION . 3.1 NOTATION . Let Gn = ( Vn , En ) be an undirected graph with additional self-connection , where Vn = { v0 , . . . , vn−1 } is the set of nodes and En ⊂ Vn × Vn is the set of edges . Let A ∈ Rn×n be the adjacency matrix and L = D − A be the Laplacian matrix , where D is a diagonal degree matrix with Dii = ∑ j Aij . we denote Ã = D − 12AD− 1 2 , then L̃ = D− 1 2LD− 1 2 = I − Ã is the symmetric normalized Laplacian . Let ( λi , ui ) be a pair of eigenvalue and unit eigenvector of L̃ , where 0 = λ0 ≤ · · · ≤ λn−1 ≤ 2 . 3.2 PROBLEM SETTING . In this paper , we are mainly interested in node classification problems on undirected graphs . Given Gn = ( Vn , En ) , we consider T = { 0 , . . . , K−1 } as the set of all node labels . For ∀k ∈ T , we denote Ck as the set of nodes with label k and R ∈ RK×K as a size matrix which is a diagonal matrix with Rk = |Ck| . Considering single-label problems in which classes are mutually exclusive , we use onehot encoding to indicate the class label and introduce a label matrix Y ∈ Rn×K = ( y0 , . . . , yK−1 ) to represent the labels of Vn , where yk is the indicator vector of Ck . Obviously , R = Y ⊤Y , Y ⊤1 = diag ( R ) and Y 1 = 1 . A signal x on Gn can be arranged the signal values in a vector form x = ( x0 , . . . , xn−1 ) ⊤ . Particularly , labels { yk|k ∈ T } are also graph signals . 3.3 A STRUCTURE INDICATOR - INTERACTION PROBABILITY . Homophily of graphs is an implicit assumption widely leveraged in graph learning methods including GNNs . It is considered an indisputable common property of most graphs , despite its descriptive and unquantifiable definition , which introduces a variety of uncertainties . In this section , starting with the random walk , we introduce interaction probability to overcome this challenge . For a random walk on Gn , we denote P = D−1A as its transition matrix which is also a row Markov matrix . From the random walk theory , P k is the k-step transition matrix , and P kij is the probability that a random walker starting from node vi arrives at vj after k steps . For a node v and a class Cl , we denote πki ( Cl ) as the probability that a random walker starting from vi stays in Cl at the k-th step . It is trivial that πki ( Cl ) = ∑ j∈Cl P k ij with ∑ l∈T π k i ( Cl ) = 1. πki ( Cl ) demonstrates the relative preference/closeness of node vi for Cl with k-scale . To meet the homophily assumption , for vi in Cl , πki ( Cl ) is expected to gap away from others . Since πki ( Cl ) − ∑ m ̸=l π k i ( Cm ) = 2πki ( Cl ) − 1 , πki ( Cl ) can be regarded as a measure of the k-scale homophily degree of node vi . Particularly , for ∀k ∈ N and vi ∈ Cl , πki ( Cl ) = 1 means that Cl is a community and will never communicate with other classes . However , this case is rare in real graphs . Below , we investigate the homophily of a class and propose a method to measure the communication strength between two classes . Definition 3.1 ( k-step interaction probability ) . For l , m ∈ T , we define Πk as the k-step interaction probability matrix formulated as follows : Πklm = 1 Rl ∑ vi∈Cl πki ( Cm ) = 1 Rl ∑ vi∈Cl , vj∈Cm P kij = y⊤l P kym y⊤l yl ( 1 ) Πk = ( Y ⊤Y ) −1Y ⊤P kY = R−1Y ⊤P kY. ( 2 ) Πklm is the probability that a random walker from Cl arrives at Cm after k steps . Remark 1 . Obviously , Πk1 = 1 . Πklm is the mean proportion of Cm in the k-hop neighbors of nodes from Cl . Noting that rank ( Y ) = K , when K ̸= n , Y R−1Y ⊤ ̸= I , thus ( R−1Y ⊤PY ) k ̸= R−1Y ⊤P kY , i.e . ( Π ) m ̸= Πm . More generally , for an arbitrary polynomial function g , R−1Y ⊤g ( P ) Y is likely not equal to g ( R−1Y ⊤P kY ) . In the rest of paper , we write g̃ ( Π ) = R−1Y ⊤g ( P ) Y and g ( Π ) = g ( R−1Y ⊤P kY ) . For instance , if g ( · ) = ( · ) m , then g̃ ( Π ) = Πm and g ( Π ) = ( Π ) m. Also , we denote Πkll , the self-interaction probability , as π k l for short . 1-step interaction probability intuitively reflects the degree of clustering of two classes and ∑k i=1 Π i measures the strength of interaction between classes in the scale of k steps . Since P is not symmetric , Πklm ̸= Πkml . To facilitate analysis , here we propose a symmetric variant of interaction probability to identify the interactions between two classes . We denote this symmetric k-step interaction probability matrix as Π̃k , by replacing P with Ã = D− 1 2AD− 1 2 , we obtain Π̃k = R− 1 2Y ⊤ÃkY R− 1 2 . Below , we investigate the important properties of Πk and Π̃k . Proposition 3.1 . For l , m ∈ T and an arbitrary polynomial function g ( · ) , we have : a. RlΠklm +RmΠ k ml ≥ 2 √ RlRmΠ̃ k lm , where Rl is the l-th diagonal element of R ; b . ( g̃2 ( Π̃ ) ) ll ≥ ( g̃ ( Π̃ ) ll ) 2 , where g̃k ( Π̃ ) = R− 1 2Y ⊤gk ( Ã ) Y R− 1 2 . The proof can be found in Appendix B . Since Π̃k1 ̸= 1 , that is , the measure is no longer a probability measure . However , according to Prop.3.1.a ( let m = l ) , π̃kl is the lower bound of π k l , and π̃ k l = π k l when Gn is a regular graph . In the rest of theoretical analysis , we use π̃kl to measure the degree of Cl ’ s clustering . Let g ( · ) = ( · ) k , from Prop.3.1.b , we have π̃2kl ≥ ( π̃kl ) 2 . In Section.4.2 , we leverage this inequality to derive a lower bound of our prediction error and further illustrate how structure influences the performance of a given filter .

Summary: It is observed that for Graph Neural Networks (GNNs), the correlation between the labels and the graph structure matters. The paper gives a theoretical analysis of this behavior via interaction probability and frequency distribution. The analysis shows why homophily is favorable for GNNs. Additionally, they are also able to identify some conditions when GNNs can do well. But the key point made is that it is not possible to cater to all possible scenarios with a single filter. The paper further goes to propose a model that builds multiple band-pass filters over the range of [-1, 1] and then aggregates information over these filters to give improved performance.