Learning from multiscale wavelet superpixels using GNN with spatially heterogeneous pooling

Neural networks have become the standard for image classification tasks . On one hand , convolutional neural networks ( CNNs ) achieve state-of-the-art performance by learning from a regular grid representation of images . On the other hand , graph neural networks ( GNNs ) have shown promise in learning image classification from an embedded superpixel graph . However , in the latter , studies have been restricted to SLIC superpixels , where 1 ) a single target number of superpixels is arbitrarily defined for an entire dataset irrespective of differences across images and 2 ) the superpixels in a given image are of similar size despite intrinsic multiscale structure . In this study , we investigate learning from a new principled representation in which individual images are represented by an image-specific number of multiscale superpixels . We propose WaveMesh , a wavelet-based superpixeling algorithm , where the number and sizes of superpixels in an image are systematically computed based on the image content . We also present WavePool , a spatially heterogeneous pooling scheme tailored to WaveMesh superpixels . We study the feasibility of learning from the WaveMesh superpixel representation using SplineCNN , a state-of-the-art network for image graph classification . We show that under the same network architecture and training settings , SplineCNN with original Graclus-based pooling learns from WaveMesh superpixels on-par with SLIC superpixels . Additionally , we observe that the best performance is achieved when replacing Graclus-based pooling with WavePool while using WaveMesh superpixels . 1 INTRODUCTION . Convolutional neural networks ( CNNs ) achieve state-of-the-art performance on a variety of image classification tasks from different domains ( Tan & Le , 2019 ; Gulshan et al. , 2016 ) . CNNs learn from a regular pixel-grid representation of the images . Although not all pixels provide equal amount of new information , by design the filters in the first layer of a CNN operate on each pixel from top-left to bottom-right in the same way . Additionally , images are typically resized to a prescribed size before feeding into a CNN . In applications that use standard CNN architectures or pre-trained models on a new image classification dataset , the images are typically uniformly downsampled to meet the input size requirements of the architecture being used . Uniform downsampling may be suboptimal as real data naturally exhibits spatial and multiscale heterogeneity . Few studies have explored the impact of input image resolution on model performance ( Sabottke & Spieler , 2020 ) , despite its recognized importance ( Lakhani , 2020 ) . Graph neural network ( GNN ) is a type of neural network that learns from graph structured data . Recent studies have shown the performance of GNNs on image graph classification tasks ( Monti et al. , 2017 ; Fey et al. , 2018 ; Knyazev et al. , 2019 ; Dwivedi et al. , 2020 ) . In this task , a GNN learns to classify images from embedded graphs that represent superpixels in the images . However , prior studies have been restricted to SLIC superpixels ( Achanta et al. , 2012 ) . In this framework , a single target number of superpixels is arbitrarily defined for an entire dataset irrespective of differences across images , and the superpixels in a given image are of similar size despite intrinsic multiscale structure . Our proposed approach circumvents these limitations , as shown in Figure 1 . The objectives of our work are twofold . First , we aim to rethink the process of downsampling and/or superpixeling images by introducing a multiscale superpixel representation that can be considered as in between the regular grid and similar-sized superpixel representations . Secondly , we systematically study the feasibility of learning to classify images from embedded graphs that represent the multiscale superpixels . In this context , the contributions of our study are as follows . • We present WaveMesh , an algorithm to superpixel ( compress ) images in the pixel domain . WaveMesh is based on the quadtree representation of the wavelet transform . Our sample-specific method leads to non-uniformly distributed and multiscale superpixels . The number and size of superpixels in an image are systematically computed by the algorithm based on the image content . WaveMesh requires at most one tunable parameter . • We propose WavePool , a spatially heterogeneous pooling method tailored to WaveMesh superpixels . WavePool preserves spatial structure leading to interpretable intermediate outputs . WavePool generalizes the classical pooling employed in CNNs , and easily integrates with existing GNNs . • To evaluate the WaveMesh representation and the WavePool method for image graph classification , we compare them with SLIC superpixels and graclus-based pooling by conducting several experiments using SplineCNN , a network proposed by Fey et al . ( 2018 ) . 2 RELATED WORK . Superpixeling . Grouping pixels to form superpixels was proposed by Ren & Malik ( 2003 ) as a preprocessing mechanism that preserves most of the structure necessary for image segmentation . Since then many superpixeling algorithms have been proposed including deep learning based methods ( Liu et al. , 2011 ; Li & Chen , 2015 ; Tu et al. , 2018 ; Giraud et al. , 2018 ; Yang et al. , 2020 ; Zhang et al. , 2020 ) . The SLIC algorithm proposed by Achanta et al . ( 2012 ) is based on k-means clustering . GNN for image graph classification . Prior studies have demonstrated the representational power and generalization ability of GNNs on image graph classification tasks using SLIC superpixels . Dwivedi et al . ( 2020 ) show that message passing graph convolution networks ( GCNs ) outperform Weisfeiler-Lehman GNNs on MNIST and CIFAR-10 datasets . Recognizing the importance of spatial and hierarchical structure inherent in images , Knyazev et al . ( 2019 ) model images as multigraphs that represent SLIC superpixels computed at different user-defined scales , and then successfully train GNNs on the multigraphs . SplineCNN proposed by Fey et al . ( 2018 ) is another network for learning from irregularly structured data . It builds on the work of Monti et al . ( 2017 ) , but uses a spline convolution kernel instead of Gaussian mixture model kernels . Graclus-based pooling . Pooling is used in GNNs to coarsen the graph by aggregating nodes within specified clusters . Graclus is a kernel-based multilevel graph clustering algorithm that efficiently clusters nodes in large graphs without any eigenvector computation . Graclus is used in many GNNs to obtain a clustering on the nodes , which is then used by the pooling operator to coarsen the graph ( Defferrard et al. , 2016 ; Monti et al. , 2017 ; Fey et al. , 2018 ) . Hereafter , we refer to pooling based on graclus clustering as graclus-based pooling . 3 WAVEMESH : MULTISCALE WAVELET SUPERPIXELS . Superpixel meshes . The WaveMesh algorithm is broken down into its elementary steps below : 1 ) images are wavelet transformed , 2 ) images are filtered in wavelet space by thresholding the wavelet coefficients , and 3 ) the superpixel mesh is generated from the wavelet-filter mask . The algorithm is rooted in wavelet theory ’ s seminal work ( Mallat , 1989 ; Donoho & Johnstone , 1994b ) . The particular way in which wavelets are used in this work is inspired by their related application in the physical sciences ( Schneider & Vasilyev , 2010 ; Bassenne et al. , 2017 ; 2018 ) . 3.1 STEP 1 : WAVELET TRANSFORMATION OF THE INPUT IMAGE . Consider a two-dimensional ( 2D ) image I discretely described by its pixel values I [ x0 ] centered at locations x0 = 2−1 ( i∆ , j∆ ) of a N×N regular grid , where ∆ is the inter-pixel spacing and ( i , j ) = 1 , 3 , . . . , 2N − 1 . A continuous wavelet representation of I is I ( x ) = ∑ x0 I ∧ ( 0 ) [ x0 ] φ ( 0 ) ( x− x0 ) , where x is the continuous pixel-space coordinate , and φ0 ( x− x0 ) are scaling functions that form a orthonormal basis of low-pass filters centered at x0 , with filter width ∆ . The scaling functions have unit energy 〈 φ0 ( x− x0 ) φ0 ( x− x0 ) 〉 = 1 , where the bracket operator 〈y〉 = 1/ ( N∆ ) 2 ∫ y ( x ) dx denotes the global average for a general 2D continuous field y ( x ) . In practice , when dealing with discrete signals , I ∧ ( 0 ) [ x0 ] can not be computed exactly , since I is only known at discrete points x0 . Instead , it is numerically discretized and the approximation coefficients I ∧ ( 0 ) [ x0 ] are estimated as an algebraic function of I [ x0 ] . Assuming that φ0 ( x− x0 ) decays fast away from x = x0 , we get I ∧ ( 0 ) [ x0 ] = I [ x0 ] /N ( Addison , 2017 ) . This estimate for I ∧ ( 0 ) [ x0 ] is the initialization stage of the recursive wavelet multiresolution algorithm ( MRA ) of Mallat ( 1989 ) , which enables the computation of wavelet coefficients at coarser scales . The decomposition of the finest-scale low-pass filter φ0 ( x− x0 ) in terms of narrow-band wavelet filters ψ ( s , d ) ( x − xs ) with increasingly large filter width and a coarsest-scale scaling function φ ( S ) ( x− xS ) yields the full wavelet-series expansion of I , I ( x ) = S∑ s=1 ∑ xs 3∑ d=1 I ∧ ( s , d ) [ xs ] ψ ( s , d ) ( x− xs ) + I ∧ ( S ) [ xS ] φ ( S ) ( x− xS ) . ( 1 ) Here , I ∧ ( s , d ) [ xs ] = 〈 I ( x ) ψ ( s , d ) ( x− xs ) 〉 and I ∧ ( S ) [ xS ] = 〈 I ( x ) φ ( S ) ( x− xS ) 〉 are wavelet and approximation coefficients at scale s and S , respectively , obtained from the orthonormality properties of the wavelet and scaling functions . In this formulation , d = ( 1 , 2 , 3 ) is a wavelet directionality index , and s = ( 1 , 2 . . . , S ) is a scale exponent , with S = log2N the number of resolution levels allowed by the grid ( 5 for 32×32 images ) . Similarly , xs = 2s−1 ( i∆ , j∆ ) is a scale-dependent wavelet grid of ( N/2s ) × ( N/2s ) elements where the basis functions are centered , with i , j = 1 , 3 , . . . , N/2s−1 − 1 . The wavelet coefficients represent the local fluctuations of I centered at xs at scale s , while the approximation coefficient is proportional to the global mean of I . At each scale , the filter width of the wavelets is 2s∆ . In this study , the 2D orthonormal basis functions ψ ( s , d ) ( x− xs ) are products of one-dimensional ( 1D ) Haar wavelets ( Meneveau , 1991 ) . The definition of 2D wavelets as multiplicative products of 1D wavelets is a particular choice that follows the MRA formulation ( Mallat , 1989 ) . Haar wavelets have a narrow spatial support that provides a high degree of spatial localization . However , they display large spectral leakage at high wavenumbers since infinite spectral and spatial resolutions can not be simultaneously attained due to limitations imposed by the uncertainty principle ( Addison , 2017 ) . Different boundary conditions can be assumed for the field I . We do not require such a choice in this study as we restrict ourselves to square images . However , the wavelet MRA framework is not limited to square inputs and can be generalized to rectangular inputs ( Addison , 2017 ; Kim et al. , 2018 ) . The definition of 2D wavelets as multiplicative products of 1D wavelets is a particular choice that follows the MRA formulation described by Mallat ( 1989 ) , in which , the multivariate wavelets are characterized by an isotropic scale and therefore render limited information about anisotropy in the image . A large number of alternative basis functions have been recently proposed for replacing traditional wavelets when analyzing multi-dimensional data that exhibit complex anisotropic structures such as filaments and sheets . These include , but are not limited to , curvelets , contourlets , and shearlets ( see Kutyniok & Labate ( 2012 ) for an extensive review on this topic ) .

Summary: The paper introduces a new approach to leveraging graph neural networks for image tasks. While prior work has been based on constructing graphs using a super-pixel map using methods like SLIC, that generate super-pixels of all roughly the same size, the proposed method generates a scale-adaptive partition. It also introduces a new pooling operator on this graph structure, and demonstrates that, together, these lead to improved performance when used with GNNs.