Pruned Graph Scattering Transforms

1 INTRODUCTION . The abundance of graph-structured data calls for advanced learning techniques , and complements nicely standard machine learning tools that can not be directly applied to irregular data domains . Permeating the benefits of deep learning to the graph domain , graph convolutional networks ( GCNs ) provide a versatile and powerful framework to learn from complex graph data ( Bronstein et al. , 2017 ) . GCNs and variants thereof have attained remarkable success in social network analysis , 3D point cloud processing , recommender systems and action recognition . However , researchers have recently reported inconsistent perspectives on the appropriate designs for GCN architectures . For example , experiments in social network analysis have argued that deeper GCNs marginally increase the learning performance ( Wu et al. , 2019 ) , whereas a method for 3D point cloud segmentation achieves state-ofthe-art performance with a 56-layer GCN network ( Li et al. , 2019 ) . These ‘ controversial ’ empirical findings motivate theoretical analysis to understand the fundamental performance factors and the architecture design choices for GCNs . Aiming to bestow GCNs with theoretical guarantees , one promising research direction is to study graph scattering transforms ( GSTs ) . GSTs are non-trainable GCNs comprising a cascade of graph filter banks followed by nonlinear activation functions . The graph filter banks are mathematically designed and are adopted to scatter an input graph signal into multiple channels . GSTs extract scattering features that can be utilized towards graph learning tasks ( Gao et al. , 2019 ) , with competitive performance especially when the number of training examples is small . Under certain conditions on the graph filter banks , GSTs are endowed with energy conservation properties ( Zou & Lerman , ∗This work was mainly done while V. N. Ioanndis was working at Mitsubishi Electric Research Laboratories . 2019 ) , as well as stability meaning robustness to graph topology deformations ( Gama et al. , 2019a ) . However , GSTs are associated with exponential complexity in space and time that increases with the number of layers . This discourages deployment of GSTs when a deep architecture is needed . Furthermore , stability should not come at odds with sensitivity . A filter ’ s output should be sensitive to and “ detect ” perturbations of large magnitude . Lastly , graph data in different domains ( social networks , 3D point clouds ) have distinct properties , which encourages GSTs with domain-adaptive architectures . The present paper develops a data-adaptive pruning framework for the GST to systematically retain important features . Specifically , the contribution of this work is threefold . C1 . We put forth a pruning approach to select informative GST features that we naturally term pruned graph scattering transform ( pGST ) . The pruning decisions are guided by a criterion promoting alignment ( matching ) of the input graph spectrum with that of the graph filters . The optimal pruning decisions are provided on-the-fly , and alleviate the exponential complexity of GSTs . C2 . We prove that the pGST is stable to perturbations of the input graph signals . Under certain conditions on the energy of the perturbations , the resulting pruning patterns before and after the perturbations are identical and the overall pGST is stable . C3 . We showcase with extensive experiments that : i ) the proposed pGSTs perform similarly and in certain cases better than the baseline GSTs that use all scattering features , while achieving significant computational savings ; ii ) The extracted features from pGSTs can be utilized towards graph classification and 3D point cloud recognition . Even without any training on the feature extraction step , the performance is comparable to state-of-the-art deep supervised learning approaches , particularly when training data are scarce ; and iii ) By analyzing the pruning patterns of the pGST , we deduce that graph signals in different domains call for different network architectures ; see Fig . 1 . 2 RELATED WORK . GCNs rely on a layered processing architecture comprising trainable graph convolution operations to linearly combine features per graph neighborhood , followed by pointwise nonlinear functions applied to the linearly transformed features ( Bronstein et al. , 2017 ) . Complex GCNs and their variants have shown remarkable success in graph semi-supervised learning ( Kipf & Welling , 2017 ; Veličković et al. , 2018 ) and graph classification ( Ying et al. , 2018 ) . To simplify GCNs , ( Wu et al. , 2019 ) has shown that by employing a single-layer linear GCN the performance in certain social network learning tasks degrades only slightly . On the other hand , ( Li et al. , 2019 ) has developed a 56-layer GCN that achieves state-of-the-art performance in 3D point cloud segmentation . Hence , designing GCN architectures guided by properties of the graph data is a highly motivated research question . Towards theoretically explaining the success of GCNs , recent works study the stability properties of GSTs with respect to metric deformations of the domain ( Gama et al. , 2019b ; a ; Zou & Lerman , 2019 ) . GSTs generalize scattering transforms ( Bruna & Mallat , 2013 ; Mallat , 2012 ) to non-Euclidean domains . GSTs are a cascade of graph filter banks and nonlinear operations that is organized in a tree-structured architecture . The number of extracted scattering features of a GST grows exponentially with the number of layers . Theoretical guarantees for GSTs are obtained after fixing the graph filter banks to implement a set of graph wavelets . The work in ( Zou & Lerman , 2019 ) establishes energy conservation properties for GSTs given that certain energy-preserving graph wavelets are employed , and also prove that GSTs are stable to graph structure perturbations ; see also ( Gama et al. , 2019b ) that focuses on diffusion wavelets . On the other hand , ( Gama et al. , 2019a ) proves stability to relative metric deformations for a wide class of graph wavelet families . These contemporary works shed light into the stability and generalization capabilities of GCNs . However , stable transforms are not necessarily informative , and albeit highly desirable , a principled approach to selecting informative GST features remains still an uncharted venue . 3 BACKGROUND . Consider a graph G : = { V , E } with node set V : = { vi } Ni=1 , and edge set E : = { ei } Ei=1 . Its connectivity is described by the graph shift matrix S ∈ RN×N , whose ( n , n′ ) th entry Snn′ is nonzero if ( n , n′ ) ∈ E or if n = n′ . A typical choice for S is the adjacency or the Laplacian matrix . Further , each node can be also associated with a few attributes . Collect attributes across all nodes in the matrix X : = [ x1 , . . . , xF ] ∈ RN×F , where each column xf ∈ RN is a ‘ graph signal. ’ Graph Fourier transform . A Fourier transform corresponds to the expansion of a signal over bases that are invariant to filtering ; here , this graph frequency basis is the eigenbasis of the graph shift matrix S. Henceforth , S is assumed normal with S = VΛV > , where V ∈ RN×N forms the graph Fourier basis , and Λ ∈ RN×N is the diagonal matrix of corresponding eigenvalues λ0 , . . . , λN−1 . These eigenvalues represent graph frequencies . The graph Fourier transform ( GFT ) of x ∈ RN is x̂ = V > x ∈ RN , while the inverse transform is x = Vx̂ . The vector x̂ represents the signal ’ s expansion in the eigenvector basis and describes the graph spectrum of x . The inverse GFT reconstructs the graph signal from its graph spectrum by combining graph frequency components weighted by the coefficients of the signal ’ s graph Fourier transform . GFT is a tool that has been popular for analyzing graph signals in the graph spectral domain . Graph convolution neural networks . GCNs permeate the benefits of CNNs from processing Euclidean data to modeling graph structured data . GCNs model graph data through a succession of layers , each of which consists of a graph convolution operation ( graph filter ) , a pointwise nonlinear function σ ( · ) , and oftentimes also a pooling operation . Given a graph signal x ∈ RN , the graph convolution operation diffuses each node ’ s information to its neighbors according to the graph shift matrix S , as Sx . The nth entry [ Sx ] n = ∑ n′∈Nn Snn′xn′ is a weighted average of the one-hop neighboring features . Successive application of S will increase the reception field , spreading the information across the network . Hence , a Kth order graph convolution operation ( graph filtering ) is h ( S ) x : = K∑ k=0 wkS kx = Vĥ ( Λ ) x̂ ( 1 ) where the graph filter h ( · ) is parameterized by the learnable weights { wk } Kk=0 , and the graph filter in the graph spectral domain is ĥ ( Λ ) = ∑K k=0 wkΛ k. In the graph vertex domain , the learnable weights reflect the influences from various orders of neighbors ; and in the graph spectral domain , those weights adaptively adjust the focus and emphasize certain graph frequency bands . GCNs employ various graph filter banks per layer , and learn the parameters that minimize a predefined learning objective , such as classification , or regression . Graph scattering transforms . GSTs are the nontrainable counterparts of GCNs , where the parameters of the graph convolutions are selected based on mathematical designs . GSTs process the input at each layer by a sequential application of graph filter banks { hj ( S ) } Jj=1 , an elementwise nonlinear function σ ( · ) , and a pooling operator U . At the first layer , the input graph signal x ∈ RN constitutes the first scattering feature vector z ( 0 ) : = x . Next , z ( 0 ) is processed by the graph filter banks and σ ( · ) to generate { z ( j ) } Jj=1 with z ( j ) : = σ ( hj ( S ) z ( 0 ) ) . At the second layer , the same operation is repeated per j . The resulting computation structure is a tree with J branches at each non-leaf node ; see also Fig . 2 . The ` th layer of the tree includes J ` nodes . Each tree node at layer ` in the scattering transform is indexed by the path p ( ` ) of the sequence of ` graph convolutions applied to the input graph signal x , i.e . p ( ` ) : = ( j ( 1 ) , j ( 2 ) , . . . , j ( ` ) ) .1 The scattering feature vector at the tree node indexed by ( p ( ` ) , j ) at layer ` + 1 is z ( p ( ` ) , j ) = σ ( hj ( S ) z ( p ( ` ) ) ) ( 2 ) where the variable p ( ` ) holds the list of indices of the parent nodes ordered by ancestry , and all path p ( ` ) in the tree with length ` are included in the path set P ( ` ) with |P ( ` ) | = 2 ` . The nonlinear transformation function σ ( · ) disperses the graph frequency representation through the spectrum , and endows the GST with increased discriminating power ( Gama et al. , 2019a ) . By exploiting the sparsity of the graph , the computational complexity of ( 2 ) is O ( KE ) , where E = |E| is the number of edges in G.2 Each scattering feature vector z ( p ( ` ) ) is summarized by an aggregation operator U ( · ) to obtain a scalar scattering coefficient as φ ( p ( ` ) ) : = U ( z ( p ( ` ) ) ) , where U ( · ) is typically an average or sum operator that effects dimensionality reduction of the extracted features . The scattering coefficient at each tree node reflects the activation level at a certain graph frequency band . These scattering coefficients are collected across all tree nodes to form a scattering feature map Φ ( x ) : = { { φ ( p ( ` ) ) } p ( ` ) ∈P ( ` ) } L ` =0 ( 3 ) where |Φ ( x ) | = ∑L ` =0 J ` . The GST operation resembles a forward pass of a trained GCN . This is why several works study GST stability under perturbations of S in order to understand the working mechanism of GCNs ( Zou & Lerman , 2019 ; Gama et al. , 2019a ; b ) .

Summary: In this paper, the authors developed graph scattering transforms (GST) with a pruning algorithm, with the aim to reduce the running time and space cost, improve robustness to perturbations on input graph signal, and encourage flexibility for domain adaption. To this end, pruned graph scattering transform (pGST) was proposed based on the alignment between graph spectrum of the graph filters and the scattering feature. The intuition is to consider tree nodes as subbands in the graph spectrum and prune tree nodes that do not have sufficient overlap with the graph spectrum of a graph signal. The pruning problem was formulated as an optimization problem, and a solution was developed with theoretical analysis. Moreover, the analysis on the stability and sensitivity to perturbations were provided. Overall, the algorithm development is solid. The experimental results demonstrate the proposed pGST can outperform GST on graph classification task with less running time. Comparing with some supervised GNN methods, the proposed pGST can still achieve comparable results on several datasets.