Data-driven Learning of Geometric Scattering Networks

1 INTRODUCTION . Geometric deep learning has recently emerged as an increasingly prominent branch of machine learning in general , and deep learning in particular ( Bronstein et al. , 2017 ) . It is based on the observation that many of the impressive achievements of neural networks come in applications where the data has an intrinsic geometric structure which can be used to inform network design and training procedures . For example , in computer vision , convolutional neural networks use the spatial organization of pixels to define convolutional filters that hierarchically aggregate local information at multiple scales that in turn encode shape and texture information in data and task-driven representations . Similarly , in time-series analysis , recurrent neural networks leverage memory mechanisms based on the temporal organization of input data to collect multiresolution information from local subsequences , which can be interpreted geometrically via tools from dynamical systems and spectral analysis . While these examples only leverage Euclidean spatiotemporal structure in data , they exemplify the potential benefits of incorporating information about intrinsic data geometry in neural network design and processing . Indeed , recent advances have further generalized the utilization of geometric information in neural networks design to consider non-Euclidean structures , with particular interest in graphs that represent data geometry , either directly given as input or constructed as an approximation of a data manifold . At the core of geometric deep learning is the use of graph neural networks ( GNNs ) in general , and graph convolutional networks ( GCNs ) in particular , which ensure neuron activations follow the geometric organization of input data by propagating information across graph neighborhoods ( Bruna et al. , 2014 ; Defferrard et al. , 2016 ; Kipf & Welling , 2016 ; Hamilton et al. , 2017 ; Xu et al. , 2019 ; Abu-El-Haija et al. , 2019 ) . However , recent work has shown the difficulty in generalizing these methods to more complex structures , identifying common problems and phrasing them in terms of oversmoothing ( Li et al. , 2018 ) , oversquashing ( Alon & Yahav , 2020 ) or under-reaching ( Barceló et al. , 2020 ) . Using graph signal processing terminology from Kipf & Welling ( 2016 ) , these issues can be partly attributed to the limited construction of convolutional filters in many commonly used GCN architectures . Inspired by the filters learned in convolutional neural networks , GCNs consider node features as graph signals and aim to aggregate information from neighboring nodes . For example , Kipf & Welling ( 2016 ) presented a typical implementation of a GCN with a cascade of averaging ( essentially low pass ) filters . We note that more general variations of GCN architectures exist ( Defferrard et al. , 2016 ; Hamilton et al. , 2017 ; Xu et al. , 2019 ) , which are capable of representing other filters , but as investigated in Alon & Yahav ( 2020 ) , they too often have difficulty in learning long range connections . Recently , an alternative approach was presented to provide deep geometric representation learning by generalizing Mallat ’ s scattering transform ( Mallat , 2012 ) , originally proposed to provide a mathematical framework for understanding convolutional neural networks , to graphs ( Gao et al. , 2019 ; Gama et al. , 2019a ; Zou & Lerman , 2019 ) and manifolds ( Perlmutter et al. , 2018 ) . Similar to traditional scattering , which can be seen as a convolutional network with nonlearned wavelet filters , geometric scattering is defined as a GNN with handcrafted graph filters , typically constructed as diffusion wavelets over the input graph ( Coifman & Maggioni , 2006 ) , which are then cascaded with pointwise absolute-value nonlinearities . This wavelet cascade results in permutation equivariant node features that are typically aggregated via statistical moments over the graph nodes , as explained in detail in Sec . 2 , to provide a permutation invariant graph-level representation . The efficacy of geometric scattering features in graph processing tasks was demonstrated in Gao et al . ( 2019 ) , with both supervised learning and data exploration applications . Moreover , their handcrafted design enables rigorous study of their properties , such as stability to deformations and perturbations , and provides a clear understanding of the information extracted by them , which by design ( e.g. , the cascaded band-pass filters ) goes beyond low frequencies to consider richer notions of regularity ( Gama et al. , 2019b ; Perlmutter et al. , 2019 ) . However , while graph scattering transforms provide effective universal feature extractors , their rigid handcrafted design does not allow for the automatic task-driven representation learning that naturally arises in traditional GNNs . To address this deficiency , recent work has proposed a hybrid scattering-GCN ( Min et al. , 2020 ) model for obtaining node-level representations , which ensembles a GCN model with a fixed scattering feature extractor . In Min et al . ( 2020 ) , integrating channels from both architectures alleviates the well-known oversmoothing problem and outperforms popular GNNs on node classification tasks . Here , we focus on improving the geometric scattering transform by learning , in particular its scales . We focus on whole-graph representations with an emphasis on biochemical molecular graphs , where relatively large diameters and non-planar structures usually limit the effectiveness of traditional GNNs . Instead of the ensemble approach of Min et al . ( 2020 ) , we propose a native neural network architecture for learned geometric scattering ( LEGS ) , which directly modifies the scattering architecture from Gao et al . ( 2019 ) ; Perlmutter et al . ( 2019 ) , via relaxations described in Sec . 3 , to allow a task-driven adaptation of its wavelet configuration via backpropagation implemented in Sec . 4 . We note that other recent graph spectrum-based methods approach the learning of long range connections by approximating the spectrum of the graph with the Lancoz algorithm Liao et al . ( 2019 ) , or learning in block Krylov subspaces Luan et al . ( 2019 ) . Such methods are complementary to the work presented here , in that their spectral approximation can also be applied in the computation of geometric scattering when considering very long range scales ( e.g. , via spectral formulation of graph wavelet filters ) . However , we find that such approximations are not necessary in the datasets considered here and in other recent work focusing on whole-graph tasks , where direct computation of polynomials of the Laplacian is sufficient . The resulting learnable geometric scattering network balances the mathematical properties inherited from the scattering transform ( as shown in Sec . 3 ) with the flexibility enabled by adaptive representation learning . The benefits of our construction over standard GNNs , as well as pure geometric scattering , are discussed and demonstrated on graph classification and regression tasks in Sec . 5 . In particular , we find that our network maintains the robustness to small training sets present in graph scattering while improving classification on biological graph classification and regression tasks , and we show that in tasks where the graphs have a large diameter relative to their size , learnable scattering features improve performance over competing methods . 2 PRELIMINARIES : GEOMETRIC SCATTERING FEATURES . Let G = ( V , E , w ) be a weighted graph with V : = { v1 , . . . , vn } the set of nodes , E ⊂ { { vi , vj } ∈ V × V , i 6= j } the set of ( undirected ) edges and w : E → ( 0 , ∞ ) assigning ( positive ) edge weights to the graph edges . Note that w can equivalently be considered as a function of V ×V , where we set the weights of non-adjacent node pairs to zero . We define a graph signal as a function x : V → R on the nodes of G and aggregate them in a signal vector x ∈ Rn with the ith entry being x [ vi ] . We define the weighted adjacency matrix W ∈ Rn×n of the graph G as W [ vi , vj ] : = { w ( vi , vj ) if { vi , vj } ∈ E 0 otherwise , and the degree matrix D ∈ Rn×n of G as D : = diag ( d1 , . . . , dn ) with di : = deg ( vi ) : =∑n j=1W [ vi , vj ] being the degree of the node vi . The geometric scattering transform ( Gao et al. , 2019 ) relies on a cascade of graph filters constructed from a row stochastic diffusion matrix P : = 12 ( In + WD −1 ) , which corresponds to transition probabilities of a lazy random walk Markov process . The laziness of the process signifies that at each step it has equal probability of either staying at the current node or transitioning to a neighbor , where transition probabilities in the latter case are determined by ( normalized ) edge weights . Scattering filters are then defined via the graph-wavelet matrices Ψj ∈ Rn×n of scale j ∈ N0 , as Ψ0 : = In − P , Ψj : = P 2j−1 − P 2 j = P 2 j−1 ( In − P 2 j−1 ) , j ≥ 1 . ( 1 ) These diffusion wavelet operators partition the frequency spectrum into dyadic frequency bands , which are then organized into a full wavelet filter bankWJ : = { Ψj , ΦJ } 0≤j≤J , where ΦJ : = P 2 J is a pure low-pass filter , similar to the one used in GCNs . It is easy to verify that the resulting wavelet transform is invertible , since a simple sum of filter matrices inWJ yields the identity . Moreover , as discussed in Perlmutter et al . ( 2019 ) , this filter bank forms a nonexpansive frame , which provides energy preservation guarantees as well as stability to perturbations , and can be generalized to a wider family of constructions that encompasses the variations of scattering transforms on graphs from Gama et al . ( 2019a ; b ) and Zou & Lerman ( 2019 ) . Given the wavelet filter bankWJ , node-level scattering features are computed by stacking cascades of bandpass filters and element-wise absolute value nonlinearities to form Upx : = Ψjm |Ψjm−1 . . . |Ψj2 |Ψj1x|| . . . | , ( 2 ) indexed ( or parametrized ) by the scattering path p : = ( j1 , . . . , jm ) ∈ ∪m∈NNm0 that determines the filter scales captured by each scattering coefficient . Then , a whole-graph scattering representation is obtained by aggregating together node-level features via statistical moments over the nodes of the graph ( Gao et al. , 2019 ) . This construction yields the geometric scattering features Sp , qx : = n∑ i=1 |Upx [ vi ] |q . ( 3 ) indexed by the scattering path p and moment order q . Finally , we note that it can be shown that the graph-level scattering transform Sp , q guarantees node-permutation invariance , while Up is permutation equivariant ( Perlmutter et al. , 2019 ; Gao et al. , 2019 ) . 3 RELAXED GEOMETRIC SCATTERING CONSTRUCTION TO ALLOW TRAINING . The geometric scattering construction , described in Sec . 2 , can be seen as a particular GNN with handcrafted layers , rather than learned ones . This provides a solid mathematical framework for understanding the encoding of geometric information in GNNs , as shown in Perlmutter et al . ( 2019 ) , while also providing effective unsupervised graph representation learning for data exploration , which also has some advantages even in supervised learning task , as shown in Gao et al . ( 2019 ) . While the handcrafted design in Perlmutter et al . ( 2019 ) ; Gao et al . ( 2019 ) is not a priori amenable to task-driven tuning provided by end-to-end GNN training , we note that the cascade in Eq . 3 does conform to a neural network architecture suitable for backpropagation . Therefore , in this section , we show how and under what conditions a relaxation of the laziness of the random walk and the selection of the scales preserves some of the useful mathematical properties established in Perlmutter et al . ( 2019 ) . We then establish in section 5 the empirical benefits of learning the diffusion scales over a purely handcrafted design . We first note that the construction of the diffusion matrix P that forms the lowpass filter used in the fixed scattering construction can be relaxed to encode adaptive laziness by setting Pα : = αIn + ( 1−α ) WD−1 . Where α ∈ [ 1/2 , 1 ) controls the reluctance of the random walk to transition from one node to another . α = 1/2 gives an equal probability to stay in the same node as to transition to one of its neighbors . At this point , we note that one difference between the diffusion lowpass filter here and the one typically used in GCN and its variation is the symmetrization applied in Kipf & Welling ( 2016 ) . However , Perlmutter et al . ( 2019 ) established that for the original construction , this is only a technical difference since P can be regarded as self-adjoint under an appropriate measure which encodes degree variations in the graph . This is then used to generate a Hilbert space L2 ( G , D−1/2 ) of graph signals with inner product 〈x , y〉D−1/2 : = 〈D−1/2x , D−1/2y〉 . The following lemma shows that a similar property is retained for our adaptive lowpass filter Pα . Lemma 1 . The matrix Pα is self-adjoint on the Hilbert space L2 ( G , D−1/2 ) from Perlmutter et al . ( 2019 ) . We note that the self-adjointness shown here is interesting , as it links models that use symmetric and asymmetric versions of the Laplacian or adjacency matrix . Namely , Lemma 1 shows that the diffusion matrix P ( which is column normalized but not row normalized ) is self-adjoint , as an operator , and can thus be considered as “ symmetric ” in a suitable inner product space , thus establishing a theoretical link between these design choices . As a second relaxation , we propose to replace the handcrafted dyadic scales in Eq . 1 with an adaptive monotonic sequence of integer diffusion time scales 0 < t1 < · · · < tJ , which can be selected or tuned via training . Then , an adaptive filter bank is constructed asW ′J : = { Ψ′j , Φ′J } J−1 j=0 , with Φ′J : = P tJ α , Ψ′0 : = In − P t1α , ( 4 ) Ψ′j : = P tj α − P tj+1α , 1 ≤ j ≤ J − 1 . The following theorem shows that for any selection of scales , the relaxed construction ofW ′J constructs a nonexpansive frame , similar to the result from Perlmutter et al . ( 2019 ) shown for the original handcrafted construction . Theorem 1 . There exist a constant C > 0 that only depends on t1 and tJ such that for all x ∈ L2 ( G , D−1/2 ) , C‖x‖2D−1/2 6 ‖Φ ′ Jx‖2D−1/2 + J∑ j=0 ‖Ψ′jx‖2D−1/2 6 ‖x‖ 2 D−1/2 , where the norm considered here is the one induced by the space L2 ( G , D−1/2 ) . Intuitively , the upper ( i.e. , nonexpansive ) frame bound implies stability in the sense that small perturbations in the input graph signal will only result in small perturbations in the representation extracted by the constructed filter bank . Further , the lower frame bound ensures certain energy preservation by the constructed filter bank , thus indicating the nonexpansiveness is not implemented in a trivial fashion ( e.g. , by constant features independent of input signal ) . In the next section we leverage the two relaxations described here to design a neural network architecture for learning the configuration α , t1 , . . . , tJ of this relaxed construction via backpropagation through the resulting scattering filter cascade . The following theorem establishes that for any such configuration , extracted fromW ′J via Eqs . 2-3 , is permutation equivariant at the node-level and permutation invariant at the graph level . This guarantees that the extracted ( in this case learned ) features indeed encode intrinsic graph geometry rather than a priori indexation . Theorem 2 . Let U ′p and S′p , q be defined as in Eq . 2 and 3 ( correspondingly ) , with the filters from W ′J with an arbitrary configuration 0 < α < 1 , 0 < t1 < · · · < tJ . Then , for any permutation Π over the nodes of G , and any graph signal x ∈ L2 ( G , D−1/2 ) U ′pΠx = ΠU ′ px and S ′ p , qΠx = S ′ p , qx p ∈ ∪m∈NNm0 , q ∈ N where geometric scattering implicitly considers here the node ordering supporting its input signal . We note that the results in Lemma 1 and Theorems 1-2 , as well as their proofs , closely follow the theoretical framework proposed by Perlmutter et al . ( 2019 ) . We carefully account here for the relaxed learned configuration , which replaces the originally handcrafted configuration there . For completeness , the adjusted proofs appear in Sec . A of the Appendix .

Summary: This paper proposes a novel graph neural network-based architecture. Building upon the theoretical success of graph scattering transforms, the authors propose to learn some aspects of it providing them with more flexibility to adapt to data (recall that graph scattering transforms are built on pre-designed graph wavelet filter banks and do not learn from data). By dropping the dyadic distribution of frequencies within the wavelet bank, the proposed architecture actually learns a more suitable frequency separation among the different wavelets.