Structure-Aware Random Fourier Kernel for Graphs

1 Introduction . Gaussian Process ( GP ) is a typical Bayesian non-parametric method for function learning . Choosing appropriate kernels profoundly improves the generalization performance of GPs on various learning tasks [ 1 , 2 ] . Stationary kernels , such as Radial Basis Function ( RBF ) kernel , are common choices of GP kernels and have been successfully applied in many regression and classification problems [ 3 , 4 , 5 ] . However , due to their restricted smoothness assumption and the limited expressiveness , stationary kernels may not be capable for complex data structures , such as graph-structured data [ 6 ] , where both object features and geometric structures are essential . To improve GP ’ s performance on data with complex structures , some expressive kernels have been proposed . Two typical expressive kernels are : compositional kernels [ 7 , 8 , 9 ] , which are obtained based on the kernel composition rules [ 10 ] , and deep kernels [ 11 , 12 ] , which are defined as the stack of a deep neural network and a base kernel . Alternatively , the spectral kernels are defined in the spectral domain [ 2 , 13 , 14 ] . Nonetheless , when modeling graph-structured data , prior kernels face several challenges . First of all , the flexibility and expressiveness of these kernels are limited by the simple base kernels upon which they are built , and the optimization of base kernels is computationally expensive [ 15 ] . Second , these kernels are usually feature-based , which indicates that they capture the similarities between graph objects based solely on object features . It may be inefficient for them to ∗Corresponding Author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . capture the structural smoothness in graph data , i.e. , neighboring objects tend to share similar function values [ 6 ] . However , capturing such structural smoothness is critical to the success of many graph learning methods [ 16 , 17 , 18 ] . In another line of work , structure-based Laplacian kernels [ 19 , 20 , 21 ] have been proposed to model functions over graphs . They are obtained by applying transformation functions to the graph Laplacian matrices . However , they rarely utilize the object features and may be suboptimal to model complicated functions over large-scale graphs . To tackle the aforementioned challenges , we propose a Structure-aware Random Fourier ( SRF ) kernel , which is both feature-based and structure-aware . To improve the kernel expressiveness , inspired by the Bochner ’ s theorem [ 22 ] , we define SRF kernel with a spectral distribution based on the Fourier duality between the kernel and the spectral distribution . This definition brings some benefits : ( 1 ) the SRF kernel converts the kernel learning problem into a distribution inference problem , and the resulting SRF kernel will be fairly expressive if the dual spectral distribution is adequately complex ; ( 2 ) the SRF kernel admits a random Fourier feature formulation for scalable learning , which leads to a linear computational complexity with respect to the number of data points . Compared with feature-based kernels and Laplacian kernels , SRF kernel enables to leverage both object features and local geometric structures to measure similarities by taking subgraphs as inputs . Moreover , to effectively optimize the GPs with proposed SRF , abbreviated as GPSRF , we develop a variational Expectation-Maximization ( EM ) method [ 23 ] , which alternates between an inference procedure ( Estep ) for posterior inference and a learning procedure ( M-step ) for parameter learning . It ’ s empirically verified that GP with SRF kernel can achieve superior performance than GPs with feature-based kernels or Laplacian kernels when modeling graph-structured data . Our contributions can be summarized as follows : ( 1 ) We propose a novel SRF kernel for modeling functions over graphs , which enables to leverage both geometric structures and object features . ( 2 ) Based on the SRF kernel , we propose a GP model , GPSRF , for graph learning tasks . ( 3 ) To effectively optimize GPSRF , we develop a variational Expectation-Maximization algorithm . ( 4 ) Experimental results on five real-world graph datasets show that GPSRF can outperform strong baselines in two typical graph learning tasks , i.e. , object classification and link prediction . 2 Related Work . Gaussian Processes and Kernel Learning for Graphs . GP models for graph-structured data have been widely studied . They have been proposed for different learning tasks , such as object classification [ 24 , 25 , 18 ] , link prediction [ 26 , 27 ] , user profiling [ 28 , 29 , 29 , 30 ] and graph classification [ 5 , 31 ] . Nonetheless , the kernels applied in these methods are feature-based , which may be suboptimal in capturing the local structural smoothness in graph data . In contrast , the structure-based Laplacian kernels [ 19 , 20 , 21 ] capture object similarities based on the geometric structures . However , they rarely utilize the rich object features and may not be capable enough to model the complicated functions over large-scale graphs . Our GPSRF differs from these works in that : ( 1 ) the SRF kernel in GPSRF is both feature-based and structure-aware ; ( 2 ) the random Fourier feature formulation of our GPSRF enables efficient optimization with reduced computational complexity . Similar to this work , the graph convolutional kernel [ 32 ] proposes to incorporate graph convolution network in kernels to leverage local structure information . However , the variational inducing point method [ 33 ] used in [ 32 ] for posterior inference has a higher computational cost than our GPSRF ( see Appendix E.6 for empirical results ) . In a different manner , the deep spectral kernel network ( DSKN ) [ 34 ] proposes to form an expressive kernel by staking multiple layers of base kernels , while we propose to define an expressive SRF kernel with a complex spectral distribution . The kernel graph convolutional network ( KGCN ) [ 35 ] proposes to learn graph representations with graph kernels and then uses the convolution network to deal with the graph classification task . By contrast , our GPSRF uses graph neural network to obtain SRF kernel and then leverages the Gaussian Process to handle the graph learning tasks . Another line of research that is closely related to ours is the graph kernels [ 36 ] , which are kernel functions between pairs of graphs . The graph kernels measure similarities between two graphs for graph classification tasks , whereas our SRF kernel measures similarity between two objects in a single graph for the node-oriented tasks such as object classification and link prediction . Of special interest to us is the neighborhood aggregation based graph kernels such as Weisfeiler-Lehman kernels [ 37 , 38 , 39 ] , and the subgraph pattern based kernels [ 40 , 41 ] . These two kinds of graph kernels are both defined based on the subgraph structures . Our SRF kernel differs from them in a number of ways . First , the neighborhood aggregation based graph kernels aggregates the neighborhood ’ s label information while our SGK kernel aggregates the neighborhood information based on the continuous feature vectors . In our semi-supervised setting , where the labeled data is scarce , it is infeasible to straightforwardly apply these graph kernels to our problem . Second , the subgraph pattern based graph kernels proposes to measure the similarities between two graphs by counting the occurrences of subgraph patterns ( graphlets ) of fixed size . These graph kernels , which are obtained based only on graph structures , do not consider any labels or feature vectors . It is also computationally expensive to find all the subgraph patterns ( graphlets ) in graphs . By contrast , our SGK kernel measures similarities between nodes by aggregating the L-hop neighborhood ’ s feature information using an aggregation function , and it is also more computationally efficient to obtain these subgraph representations . Random Fourier Feature . The random Fourier feature ( RFF ) [ 43 ] is a powerful technique for kernel approximation [ 44 , 45 , 46 ] . It has been successfully applied to address the high computational cost of GPs [ 43 , 47 , 48 , 45 , 49 ] . Typically , the sparse spectrum GP ( SSGP ) [ 47 ] proposes to directly optimize spectral points along with the hyperparameters . The variational method used in [ 48 ] infers the posteriors over the frequencies of the kernel to avoid potential overfitting of SSGP . In comparison , we learn the SRF kernel by inferring the spectral distribution in a data-driven way , which makes the learnt kernel more flexible . We would like to clarify the differences between RFF and our SRF kernel . The RFF is a technique to approximate a pre-defined kernel function , which requires calculating the spectral distribution of the pre-defined kernel function through the Fourier transform ( we refer to [ 43 ] or Appendix A for more details about RFF ) . We emphasize that for most complex kernel functions , it is not feasible to obtain the corresponding spectral distributions due to the intractability of Fourier transform . This is also the reason why RFF is only applicable to a small range of simple kernel functions so far , such as RBF kernel and arc-cosine kernel . By contrast , our SRF kernel is a kernel function directly defined with a spectral distribution ( we define SRF kernel with a hierarchy Bayesian distribution in our paper ) based on Bochner ’ s theorem . Similar ideas have also been explored to obtain time embedding [ 50 ] . The definition of SRF kernel sidesteps calculating the potentially intractable spectral distribution . Also , we can obtain a complex and expressive kernel by defining a complicated spectral distribution . Hence , we generalize RFF to a family of complex and expressive kernel functions , i.e. , SRF kernels . Graph Neural Networks . Graph neural networks ( GNNs ) have been proposed for graph learning . They leverage both graph structures and features to learn useful representations , and have achieved state-of-the-art performance in some graph learning tasks [ 16 , 17 , 51 , 52 , 53 , 54 , 55 ] . Compared with GNNs , which are parametric deterministic models , our GPSRF is a Bayesian non-parametric GP model . Moreover , GPSRF enables to incorporate background knowledge by selecting a proper kernel function , and obtain uncertainty estimations of predictions , which are infeasible for GNNs . 3 Preliminaries . 3.1 Gaussian Process with Random Fourier Feature . A Gaussian Process defines a distribution in a function space , which assumes that the marginal distribution over function values of any finite inputs is Gaussian . Specifically , a GP places prior distribution over functions , i.e. , f ( x ) ∼ GP ( 0 , kθ ( x , x′ ) ) , where kθ ( x , x′ ) ∈ R denotes the kernel ( covariance ) function with parameters θ between x , x′ ∈ RF . One critical challenge of GPs is the cubic computational cost incurred by the inverse of kernel matrix in posterior inference [ 10 ] . To address the challenge , the random Fourier feature ( RFF ) method has been proposed for scalable learning [ 43 , 45 ] . Based on the Bochner ’ s theorem [ 22 ] , the RFF method proposes to approximate shift-invariant kernels with explicit feature mappings [ 43 ] . Specifically , for a shift-invariant kernel kθ ( x , x′ ) = kθ ( r ) , where r = x − x′ , the Fourier duality between the kernel and spectral distribution [ 10 ] enables to define an explicit feature mapping function : φω ( x ) = √ 2 M [ cos ( 2πω > mx+ bm ) ] M m=1 , ( 1 ) such that the inner product between the feature maps is an unbiased estimate of the kernel function , i.e. , kθ ( r ) ≈ φω ( x ) > φω ( x′ ) . In Eq . ( 1 ) , ωm ∈ RF and bm are samples drawn from the spectral distribution p ( ω ) and the uniform distribution Unif [ 0 , 2π ] respectively , and M represents the total number of samples . The features obtained with φω is referred to as the random Fourier features [ 43 ] . More details about RFF can be found in Appendix A . When modeling graph-structured data with GPs , in addition to the inference problem , it is nontrivial to choose a suitable kernel function as : ( 1 ) the RFF method is limited to a small range of kernels since it requires to calculate the spectral distribution , which is not always feasible due to the intractability of Fourier transform ; ( 2 ) a majority of GP kernels are feature-based , which capture similarities based solely on features , making them inefficient in capturing the local geometric structures in graphs .

Summary: The idea of this paper is to build a kernel on the features of the graph that takes into account the underlying structure of the graph. Once found, this kernel is used to define a GP on the graph. This is useful for learning tasks on graphs such as node classification/link prediction. In order to build this kernel, the authors propose to use a base translation invariant kernel that admits, by Bochner, a feature expansion, i.e. an embedding $\phi(x,w)$ (like RFF). More precisely, they propose to learn the law on the drawing of the frequencies $w$ of these features under the constraint that the features must incorporate the structure of the graph. These features are built in two steps: 1) the law on the drawing of frequencies $w$ is defined by a "hierarchical" law parameterized by a neural network that can be "arbitrarily complex" (in the manner of a GAN generator) 2) the features take the embedding of a subgraph centered at this node as input $x$. This embedding is defined by a GNN.