Graph Classification with Geometric Scattering

1 INTRODUCTION . Over the past decade , numerous examples have established that deep neural networks ( i.e. , cascades of linear operations and simple nonlinearities ) typically outperform traditional “ shallow ” models in various modern machine learning applications , especially given the increasing Big Data availability nowadays . Perhaps the most well known example of the advantages of deep networks is in computer vision , where the utilization of 2D convolutions enable network designs that learn cascades of convolutional filters , which have several advantages over fully connected network architectures , both computationally and conceptually . Indeed , in terms of supervised learning , convolutional neural networks ( ConvNets ) hold the current state of the art in image classification , and have become the standard machine learning approach towards processing big structured-signal data , including audio and video processing . See , e.g. , Goodfellow et al . ( 2016 , Chapter 9 ) for a detailed discussion . Beyond their performances when applied to specific tasks , pretrained ConvNet layers have been explored as image feature extractors by freezing the first few pretrained convolutional layers and then retraining only the last few layers for specific datasets or applications ( e.g. , Yosinski et al. , 2014 ; Oquab et al. , 2014 ) . Such transfer learning approaches provide evidence that suitably constructed deep filter banks should be able to extract task-agnostic semantic information from structured data , and in some sense mimic the operation of human visual and auditory cortices , thus supporting the neural terminology in deep learning . An alternative approach towards such universal feature extraction was presented in Mallat ( 2012 ) , where a deep filter bank , known as the scattering transform , is designed , rather than trained , based on predetermined families of distruptive patterns that should be eliminated to extract informative representations . The scattering transform is constructed as a cascade of linear wavelet transforms and nonlinear complex modulus operations that provides features with guaranteed invariance to a predetermined Lie group of operations such as rotations , translations , or scaling . Further , it also provides Lipschitz stability to small diffeomorphisms of the inputted signal . Scattering features have been shown to be effective in several audio ( e.g. , Bruna & Mallat , 2013a ; Andén & Mallat , 2014 ; Lostanlen & Mallat , 2015 ) and image ( e.g. , Bruna & Mallat , 2013b ; Sifre & Mallat , 2014 ; Oyallon & Mallat , 2015 ) processing applications , and their advantages over learned features are especially relevant in applications with relatively low data availability , such as quantum chemistry ( e.g. , Hirn et al. , 2017 ; Eickenberg et al. , 2017 ; 2018 ) . Following the recent interest in geometric deep learning approaches for processing graph-structured data ( see , for example , Bronstein et al . ( 2017 ) and references therein ) , we present here a generalization of the scattering transform from Euclidean domains to graphs . Similar to the Euclidean case , our construction is based on a cascade of bandpass filters , defined in this case using graph signal processing ( Shuman et al. , 2013 ) notions , and complex moduli , which in this case take the form of absolute values ( see Sec . 3 ) . While several choices of filter banks could generally be used with the proposed cascade , we focus here on graph wavelet filters defined by lazy random walks ( see Sec . 2 ) . These wavelet filters are also closely related to diffusion geometry and related notions of geometric harmonic analysis , e.g . the diffusion maps algorithm of Coifman & Lafon ( 2006 ) and the associated diffusion wavelets of Coifman & Maggioni ( 2006 ) . Therefore , we call the constructed cascade geometric scattering , which also follows the same terminology from geometric deep learning . We note that similar attempts at generalizing the scattering transform to graphs have been presented in Chen et al . ( 2014 ) as well as Zou & Lerman ( 2018 ) and Gama et al . ( 2018 ) . The latter two works are most closely related to the present paper . In them , the authors focus on theoretical properties of the proposed graph scattering transforms , and show that such transforms are invariant to graph isomorphism . The geometric scattering transform that we define here also possesses the same invariance property , and we expect similar stability properties to hold for the proposed construction as well . However , in this paper we focus mainly on the practical applicability of geometric scattering transforms for graph-structured data analysis , with particular emphasis on the task of graph classification , which has received much attention recently in geometric deep learning ( see Sec . 4 ) In supervised graph classification problems one is given a training database of graph/label pairs { ( Gi , yi ) } Ni=1 ⊂ G × Y sampled from a set of potential graphs G and potential labels Y . The goal is to use the training data to learn a model f : G → Y that associates to any graph G ∈ G a label y = f ( G ) ∈ Y . These types of databases arise in biochemistry , in which the graphs may be molecules and the labels some property of the molecule ( e.g. , its toxicity ) , as well as in various types of social network databases . Until recently , most approaches were kernel based methods , in which the model f was selected from the reproducing kernel Hilbert space generated by a kernel that measures the similarity between two graphs ; one of the most successful examples of this approach is the Weisfeiler-Lehman graph kernel of Shervashidze et al . ( 2011 ) . Numerous feed forward deep learning algorithms , though , have appeared over the last few years . In many of these algorithms , task based ( i.e. , dependent upon the labels Y ) graph filters are learned from the training data as part of the larger network architecture . These filters act on a characteristic signal xG that is defined on the vertices of any graph G , e.g. , xG may be a vector of degrees of each vertex ( we remark there are also edge based algorithms , such as Gilmer et al . ( 2017 ) and references within , but these have largely been developed for and tested on databases not considered in Sec . 4 ) . Here , we propose an alternative to these methods in the form of a geometric scattering classifier ( GSC ) that leverages graph-dependent ( but not label dependent ) scattering transforms to map each graph G to the scattering features extracted from xG . Furthermore , inspired by transfer learning approaches such as Oquab et al . ( 2014 ) , we consider treatment of our scattering cascade as frozen layers on xG , either followed by fully connected classification layers ( see Fig . 2 ) , or fed into other classifiers such as SVM or logistic regression . We note that while the formulation in Sec . 3 is phrased for a single signal xG , it naturally extends to multiple signals by concatenating their scattering features . In Sec . 4.1 we evaluate the quality of the scattering features and resulting classification by comparing it to numerous graph kernel and deep learning methods over 13 datasets ( 7 biochemistry ones and 6 social network ones ) commonly studied in related literature . In terms of classification accuracy on individual datasets , we show that the proposed approach obtains state of the art results on two datasets and performs competitively on the rest , despite only learning a classifier that come after the geometric scattering transform . Furthermore , while other methods may excel on specific datasets , when considering average accuracy : within social network data , our proposed GSC outperforms all other methods ; in biochemistry or over all datasets , it outperforms nearly all feed forward neural network approaches , and is competitive with state of the art results of graph kernels ( Kriege et al. , 2016 ) and graph recurrent neural networks ( Taheri et al. , 2018 ) . We regard this result as crucial in establishing the universality of graph features extracted by geometric scattering , as they provide an effective task-independent representation of analyzed graphs . Finally , to establish their unsupervised qualities , in Sec . 4.2 we use geometric scattering features extracted from enzyme data ( Borgwardt et al. , 2005a ) to infer emergent patterns of enzyme commission ( EC ) exchange preferences in enzyme evolution , validated with established knowledge from Cuesta et al . ( 2015 ) . 2 GRAPH RANDOM WALKS AND GRAPH WAVELETS . We define graph wavelets as the difference between lazy random walks that have propagated at different time scales , which mimics classical wavelet constructions found in Meyer ( 1993 ) as well as more recent constructions found in Coifman & Maggioni ( 2006 ) . The underpinnings for this construction arise out of graph signal processing , and in particular the properties of the graph Laplacian . Let G = ( V , E , W ) be a weighted graph , consisting of n vertices V = { v1 , . . . , vn } , edges E ⊆ { ( v ` , vm ) : 1 ≤ ` , m ≤ n } , and weights W = { w ( v ` , vm ) > 0 : ( v ` , vm ) ∈ E } . Note that unweighted graphs are considered as a special case , by setting w ( v ` , vm ) = 1 for each ( v ` , vm ) ∈ E. Define the n × n ( weighted ) adjacency matrix AG = A of G by A ( v ` , vm ) = w ( v ` , vm ) if ( v ` , vm ) ∈ E and zero otherwise , where we use the notation A ( v ` , vm ) to denote the ( ` , m ) entry of the matrix A so as to emphasize the correspondence with the vertices in the graph and to reserve sub-indices for enumerating objects . Define the ( weighted ) degree of vertex v ` as deg ( v ` ) = ∑ m A ( v ` , vm ) and the corresponding diagonal n × n degree matrix D given by D ( v ` , v ` ) = deg ( v ` ) , D ( v ` , vm ) = 0 , ` 6= m. Finally , the n × n graph Laplacian matrix LG = L on G is defined as L = D−A . The graph Laplacian is a symmetric , real valued positive semi-definite matrix , and thus has n nonnegative eigenvalues . Furthermore , if we set 0 = ( 0 , . . . , 0 ) T to to be the n× 1 vector of all zeroes , and 1 = ( 1 , . . . , 1 ) T to be the analogous vector of all ones , then it is easy to see that L1 = 0 . Therefore 0 is an eigenvalue of L and we write the n eigenvalues of L as 0 = λ0 ≤ λ1 ≤ · · · ≤ λn−1 with corresponding n × 1 orthonormal eigenvectors 1/ √ n = ϕ0 , ϕ1 , . . . , ϕn−1 . If the graph G is connected , then λ1 > 0 . In order to simplify the following discussion we assume that this is the case , although the discussion below can be amended to include disconnected graphs as well . Since ϕ0 is constant and every other eigenvector is orthogonal to ϕ0 , it is natural to view the eigenvectors ϕk as the Fourier modes of the graph G , with a frequency magnitude √ λk . Let x : V → R be a signal defined on the vertices of the graph G , which we will consider as an n × 1 vector with entries x ( v ` ) . It follows that the Fourier transform of x can be defined as x̂ ( k ) = x · ϕk , where x · y is the standard dot product . This analogy is one of the foundations of graph signal processing and indeed we could use this correspondence to define wavelet operators on the graph G , as in Hammond et al . ( 2011 ) . Rather than follow this path , though , we instead take a related path similar to Coifman & Maggioni ( 2006 ) ; Gama et al . ( 2018 ) by defining the graph wavelet operators in terms of random walks defined on G , which will avoid diagonalizing L and will allow us to control the “ spatial ” graph support of the filters directly . Define the n×n transition matrix of a lazy random random walk as P = 12 ( D−1A + I ) . Note that the row sums of P are all one and thus the entry P ( v ` , vm ) corresponds to the transition probability of walking from vertex v ` to vm in one step . Powers of P run the random walk forward , so that in particular Pt ( v ` , vm ) is the transition probability of walking from v ` to vm in exactly t steps . We will use P as a left multiplier , in which case P acts a diffusion operator . To understand this idea more precisely , first note that a simple calculation shows that P1 = 1 and furthermore if the graph G is connected , every other eigenvalue of P is contained in [ 0 , 1 ) . Note in particular that L and P share the eigenvector 1 . It follows that Ptx responds most significantly to the zero frequency x̂ ( 0 ) of x while depressing the non-zero frequencies of x ( where the frequency modes are defined in terms of the graph Laplacian L , as described above ) . On the spatial side , the value Ptx ( v ` ) is the weighted average of x ( v ` ) with all values x ( vm ) such that vm is within t steps of v ` in the graph G. High frequency responses of x can be recovered in multiple different fashions , but we utilize multiscale wavelet transforms that group the non-zero frequencies of G into approximately dyadic bands . As shown in Mallat ( 2012 , Lemma 2.12 ) , wavelet transforms are provably stable operators in the Euclidean domain , and the proof of Zou & Lerman ( 2018 , Theorem 5.1 ) indicates that similar results on graphs may be possible . Furthermore , the multiscale nature of wavelet transforms will allow the resulting geometric scattering transform ( Sec . 3 ) to traverse the entire graph G in one layer , which is valuable for obtaining global descriptions of G. Following Coifman & Maggioni ( 2006 ) , define the n× n diffusion wavelet matrix at the scale 2j as Ψj = P 2j−1 −P2 j = P2 j−1 ( I−P2 j−1 ) ( 1 ) Since Pt1 = 1 for every t , we see that Ψj1 = 0 for each j ≥ 1 . Thus each Ψjx partially recovers x̂ ( k ) for k ≥ 1 . The value Ψjx ( v ` ) aggregates the signal information x ( vm ) from the vertices vm that are within 2j steps of v ` , but does not average the information like the operator P2 j . Instead , it responds to sharp transitions or oscillations of the signal x within the neighborhood of v ` with radius 2j ( in terms of the graph path distance ) . Generally , the smaller j the higher the frequencies Ψjx recovers in x . These high frequency wavelet coefficients up to the scale 2J are denoted by : Ψ ( J ) x ( v ` ) = [ Ψjx ( v ` ) : 1 ≤ j ≤ J ] , ` = 1 , . . . , n . ( 2 ) Since 2J controls the maximum scale of the wavelet , in the experiments of Sec . 4 we select J such that 2J ∼ diam ( G ) . Figure 1 plots the diffusion wavelets at different scales on two different graphs .

Summary: This paper generalizes scattering transform to graphs. It defines wavelets, scattering coefficients on graph signals. The experimental section describes their use in classification tasks with comparisons with recent methods. It seems scattering performs less well than SOTA methods, but has the advantages of not requiring any training so potentially good candidates for low data regimes application. Interesting and original paper and ideas being developed, but might be a tiny bit weak in term of results, both theoretical and experimental ?