Structure-Aware Random Fourier Kernel for Graphs

1 Introduction . Gaussian Process ( GP ) is a typical Bayesian non-parametric method for function learning . Choosing appropriate kernels profoundly improves the generalization performance of GPs on various learning tasks [ 1 , 2 ] . Stationary kernels , such as Radial Basis Function ( RBF ) kernel , are common choices of GP kernels and have been successfully applied in many regression and classification problems [ 3 , 4 , 5 ] . However , due to their restricted smoothness assumption and the limited expressiveness , stationary kernels may not be capable for complex data structures , such as graph-structured data [ 6 ] , where both object features and geometric structures are essential . To improve GP ’ s performance on data with complex structures , some expressive kernels have been proposed . Two typical expressive kernels are : compositional kernels [ 7 , 8 , 9 ] , which are obtained based on the kernel composition rules [ 10 ] , and deep kernels [ 11 , 12 ] , which are defined as the stack of a deep neural network and a base kernel . Alternatively , the spectral kernels are defined in the spectral domain [ 2 , 13 , 14 ] . Nonetheless , when modeling graph-structured data , prior kernels face several challenges . First of all , the flexibility and expressiveness of these kernels are limited by the simple base kernels upon which they are built , and the optimization of base kernels is computationally expensive [ 15 ] . Second , these kernels are usually feature-based , which indicates that they capture the similarities between graph objects based solely on object features . It may be inefficient for them to ∗Corresponding Author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . capture the structural smoothness in graph data , i.e. , neighboring objects tend to share similar function values [ 6 ] . However , capturing such structural smoothness is critical to the success of many graph learning methods [ 16 , 17 , 18 ] . In another line of work , structure-based Laplacian kernels [ 19 , 20 , 21 ] have been proposed to model functions over graphs . They are obtained by applying transformation functions to the graph Laplacian matrices . However , they rarely utilize the object features and may be suboptimal to model complicated functions over large-scale graphs . To tackle the aforementioned challenges , we propose a Structure-aware Random Fourier ( SRF ) kernel , which is both feature-based and structure-aware . To improve the kernel expressiveness , inspired by the Bochner ’ s theorem [ 22 ] , we define SRF kernel with a spectral distribution based on the Fourier duality between the kernel and the spectral distribution . This definition brings some benefits : ( 1 ) the SRF kernel converts the kernel learning problem into a distribution inference problem , and the resulting SRF kernel will be fairly expressive if the dual spectral distribution is adequately complex ; ( 2 ) the SRF kernel admits a random Fourier feature formulation for scalable learning , which leads to a linear computational complexity with respect to the number of data points . Compared with feature-based kernels and Laplacian kernels , SRF kernel enables to leverage both object features and local geometric structures to measure similarities by taking subgraphs as inputs . Moreover , to effectively optimize the GPs with proposed SRF , abbreviated as GPSRF , we develop a variational Expectation-Maximization ( EM ) method [ 23 ] , which alternates between an inference procedure ( Estep ) for posterior inference and a learning procedure ( M-step ) for parameter learning . It ’ s empirically verified that GP with SRF kernel can achieve superior performance than GPs with feature-based kernels or Laplacian kernels when modeling graph-structured data . Our contributions can be summarized as follows : ( 1 ) We propose a novel SRF kernel for modeling functions over graphs , which enables to leverage both geometric structures and object features . ( 2 ) Based on the SRF kernel , we propose a GP model , GPSRF , for graph learning tasks . ( 3 ) To effectively optimize GPSRF , we develop a variational Expectation-Maximization algorithm . ( 4 ) Experimental results on five real-world graph datasets show that GPSRF can outperform strong baselines in two typical graph learning tasks , i.e. , object classification and link prediction . 2 Related Work . Gaussian Processes and Kernel Learning for Graphs . GP models for graph-structured data have been widely studied . They have been proposed for different learning tasks , such as object classification [ 24 , 25 , 18 ] , link prediction [ 26 , 27 ] , user profiling [ 28 , 29 , 29 , 30 ] and graph classification [ 5 , 31 ] . Nonetheless , the kernels applied in these methods are feature-based , which may be suboptimal in capturing the local structural smoothness in graph data . In contrast , the structure-based Laplacian kernels [ 19 , 20 , 21 ] capture object similarities based on the geometric structures . However , they rarely utilize the rich object features and may not be capable enough to model the complicated functions over large-scale graphs . Our GPSRF differs from these works in that : ( 1 ) the SRF kernel in GPSRF is both feature-based and structure-aware ; ( 2 ) the random Fourier feature formulation of our GPSRF enables efficient optimization with reduced computational complexity . Similar to this work , the graph convolutional kernel [ 32 ] proposes to incorporate graph convolution network in kernels to leverage local structure information . However , the variational inducing point method [ 33 ] used in [ 32 ] for posterior inference has a higher computational cost than our GPSRF ( see Appendix E.6 for empirical results ) . In a different manner , the deep spectral kernel network ( DSKN ) [ 34 ] proposes to form an expressive kernel by staking multiple layers of base kernels , while we propose to define an expressive SRF kernel with a complex spectral distribution . The kernel graph convolutional network ( KGCN ) [ 35 ] proposes to learn graph representations with graph kernels and then uses the convolution network to deal with the graph classification task . By contrast , our GPSRF uses graph neural network to obtain SRF kernel and then leverages the Gaussian Process to handle the graph learning tasks . Another line of research that is closely related to ours is the graph kernels [ 36 ] , which are kernel functions between pairs of graphs . The graph kernels measure similarities between two graphs for graph classification tasks , whereas our SRF kernel measures similarity between two objects in a single graph for the node-oriented tasks such as object classification and link prediction . Of special interest to us is the neighborhood aggregation based graph kernels such as Weisfeiler-Lehman kernels [ 37 , 38 , 39 ] , and the subgraph pattern based kernels [ 40 , 41 ] . These two kinds of graph kernels are both defined based on the subgraph structures . Our SRF kernel differs from them in a number of ways . First , the neighborhood aggregation based graph kernels aggregates the neighborhood ’ s label information while our SGK kernel aggregates the neighborhood information based on the continuous feature vectors . In our semi-supervised setting , where the labeled data is scarce , it is infeasible to straightforwardly apply these graph kernels to our problem . Second , the subgraph pattern based graph kernels proposes to measure the similarities between two graphs by counting the occurrences of subgraph patterns ( graphlets ) of fixed size . These graph kernels , which are obtained based only on graph structures , do not consider any labels or feature vectors . It is also computationally expensive to find all the subgraph patterns ( graphlets ) in graphs . By contrast , our SGK kernel measures similarities between nodes by aggregating the L-hop neighborhood ’ s feature information using an aggregation function , and it is also more computationally efficient to obtain these subgraph representations . Random Fourier Feature . The random Fourier feature ( RFF ) [ 43 ] is a powerful technique for kernel approximation [ 44 , 45 , 46 ] . It has been successfully applied to address the high computational cost of GPs [ 43 , 47 , 48 , 45 , 49 ] . Typically , the sparse spectrum GP ( SSGP ) [ 47 ] proposes to directly optimize spectral points along with the hyperparameters . The variational method used in [ 48 ] infers the posteriors over the frequencies of the kernel to avoid potential overfitting of SSGP . In comparison , we learn the SRF kernel by inferring the spectral distribution in a data-driven way , which makes the learnt kernel more flexible . We would like to clarify the differences between RFF and our SRF kernel . The RFF is a technique to approximate a pre-defined kernel function , which requires calculating the spectral distribution of the pre-defined kernel function through the Fourier transform ( we refer to [ 43 ] or Appendix A for more details about RFF ) . We emphasize that for most complex kernel functions , it is not feasible to obtain the corresponding spectral distributions due to the intractability of Fourier transform . This is also the reason why RFF is only applicable to a small range of simple kernel functions so far , such as RBF kernel and arc-cosine kernel . By contrast , our SRF kernel is a kernel function directly defined with a spectral distribution ( we define SRF kernel with a hierarchy Bayesian distribution in our paper ) based on Bochner ’ s theorem . Similar ideas have also been explored to obtain time embedding [ 50 ] . The definition of SRF kernel sidesteps calculating the potentially intractable spectral distribution . Also , we can obtain a complex and expressive kernel by defining a complicated spectral distribution . Hence , we generalize RFF to a family of complex and expressive kernel functions , i.e. , SRF kernels . Graph Neural Networks . Graph neural networks ( GNNs ) have been proposed for graph learning . They leverage both graph structures and features to learn useful representations , and have achieved state-of-the-art performance in some graph learning tasks [ 16 , 17 , 51 , 52 , 53 , 54 , 55 ] . Compared with GNNs , which are parametric deterministic models , our GPSRF is a Bayesian non-parametric GP model . Moreover , GPSRF enables to incorporate background knowledge by selecting a proper kernel function , and obtain uncertainty estimations of predictions , which are infeasible for GNNs . 3 Preliminaries . 3.1 Gaussian Process with Random Fourier Feature . A Gaussian Process defines a distribution in a function space , which assumes that the marginal distribution over function values of any finite inputs is Gaussian . Specifically , a GP places prior distribution over functions , i.e. , f ( x ) ∼ GP ( 0 , kθ ( x , x′ ) ) , where kθ ( x , x′ ) ∈ R denotes the kernel ( covariance ) function with parameters θ between x , x′ ∈ RF . One critical challenge of GPs is the cubic computational cost incurred by the inverse of kernel matrix in posterior inference [ 10 ] . To address the challenge , the random Fourier feature ( RFF ) method has been proposed for scalable learning [ 43 , 45 ] . Based on the Bochner ’ s theorem [ 22 ] , the RFF method proposes to approximate shift-invariant kernels with explicit feature mappings [ 43 ] . Specifically , for a shift-invariant kernel kθ ( x , x′ ) = kθ ( r ) , where r = x − x′ , the Fourier duality between the kernel and spectral distribution [ 10 ] enables to define an explicit feature mapping function : φω ( x ) = √ 2 M [ cos ( 2πω > mx+ bm ) ] M m=1 , ( 1 ) such that the inner product between the feature maps is an unbiased estimate of the kernel function , i.e. , kθ ( r ) ≈ φω ( x ) > φω ( x′ ) . In Eq . ( 1 ) , ωm ∈ RF and bm are samples drawn from the spectral distribution p ( ω ) and the uniform distribution Unif [ 0 , 2π ] respectively , and M represents the total number of samples . The features obtained with φω is referred to as the random Fourier features [ 43 ] . More details about RFF can be found in Appendix A . When modeling graph-structured data with GPs , in addition to the inference problem , it is nontrivial to choose a suitable kernel function as : ( 1 ) the RFF method is limited to a small range of kernels since it requires to calculate the spectral distribution , which is not always feasible due to the intractability of Fourier transform ; ( 2 ) a majority of GP kernels are feature-based , which capture similarities based solely on features , making them inefficient in capturing the local geometric structures in graphs .

Summary: This paper proposed a novel SRF kernel defined with spectral distribution based on the Fourier duality given by Bochner's theorem on the vector embedding of subgraphs using GCNN.  Based on the SRF kernel, the paper  proposes a GP model, GPSRF,  for graph learning tasks. Paper developed a variational EM algorithm to solve kernel parameters. It also studies the run time complexity of the proposed method. Model has been evaluated for semi-supervised object classification and link prediction. Experimental results show that GPSRF yields best performance compared with state-of-the-art baselines.