Towards a Deep Network Architecture for Structured Smoothness

We propose the Fixed Grouping Layer ( FGL ) ; a novel feedforward layer designed to incorporate the inductive bias of structured smoothness into a deep learning model . FGL achieves this goal by connecting nodes across layers based on spatial similarity . The use of structured smoothness , as implemented by FGL , is motivated by applications to structured spatial data , which is , in turn , motivated by domain knowledge . The proposed model architecture outperforms conventional neural network architectures across a variety of simulated and real datasets with structured smoothness . 1 INTRODUCTION . The effectiveness of predictive models often depends on the choice of inductive bias , and the extent to which this inductive bias captures real-world structure . For instance , one example of such bias encoding leading to improved performance is convolution . In principle , convolutional weights could be learned directly from data . However , in practice , imposing this structure leads to improved performance when compared to fully connected models , and as a result , Convolutional neural networks ( CNNs ) have enjoyed wide use for computer vision tasks ( Krizhevsky et al. , 2012 ) . Similarly , recurrent neural networks such as LSTMs are effective for text ( Sundermeyer et al. , 2012 ) , and certain graphical models are ideal for sentence segmentation and labeling ( Lafferty et al. , 2001 ) . Our work follows this philosophy . Specifically , we propose a feedforward layer for deep neural networks that is suitable for neuroimaging and potentially useful for other data where variables can be grouped due to underlying structure . Data with multiple input variables often exhibit some structure . For example , the El Nino dataset ( Bay et al. , 2000 ) consists of measurements by weather buoys in the ocean , and one expects that nearby buoys can be grouped together . Similarly , socio-economic data can often be grouped together by geographic proximity . Financial market data of individual stocks can be grouped together based on the industrial sector to which a company belongs . Along similar lines , brain parcellations are a well studied paradigm for capturing the structure of brain activity ( Thirion et al. , 2014 ) , often via statistical parcellation based on ward clustering ( Ward Jr , 1963 ) . The result of ward clustering is a tree where leaf nodes represent voxels of the brain and interior nodes represent grouping of voxels into spatial clusters . Figure 1 visualizes the output of ward clustering at various granularities when applied to the human connectome project resting state brain data ( Van Essen et al. , 2012 ) . Contribitions : Our primary technical contribution is the Fixed Grouping Layer ( FGL ) . FGL is designed to extract features within each group , and additionally guarantees that each output vector is only affected by the input vectors related to it by the specified grouping . We demonstrate the benefit of using FGL on simulated experiments and real neuroimaging data . We compare FGL against fully connected networks , convolutional neural networks , CoordConv ( Liu et al. , 2018 ) , and a closely related method proposed by Aydore et al . ( 2018 ) . We extensively evaluate the performance of FGL on simulated and real brain imaging data showing improved performance . 1.1 DECODING BRAIN IMAGES : BACKGROUND AND RELATED WORK . Functional Magnetic Resonance Imaging ( fMRI ) is a popular brain imaging technique which measures a physiological correlate of neuron activity ( Huettel et al. , 2004 ) . The brain imaging scans are generally of two kinds : resting state and task data . Resting state data ( rfMRI ) is collected while the subject is at rest , i.e. , while the subject is not actively engaged in a task . Task data ( tfMRI ) is collected while the subject is engaged in a predefined task , for example , a motor task such as moving their fingers . fMRI data can be represented as 3-dimensional images , and have rich structure that has been studied extensively , including in the ML literature ( Koyejo et al. , 2014 ; Park et al. , 2013 ) . Importantly , coarse correspondences have been discovered between brain regions and specific functions or behavior ( Frost and Goebel , 2012 ; Sporns , 2013 ) We particularly focus on brain decoding – a standard task in fMRI brain data analysis where the brain image is used to predict the associated task or stimulus . Broadly , there are two types of brain decoding models : end to end models and models which perform dimensionality reduction followed by a low-dimensional prediction . On one hand , dimension reduction directly captures the notion of grouping variables together . On the other hand , end to end models rarely employ brain spatial structure . This observation motivates our work . In recent years , decoding from fMRI studies has been attempted using a variety of methods : Bzdok et al . ( 2015 ) use factored logistic regression while Sarraf and Tofighi ( 2016 ) use convolutional neural networks . Mensch et al . ( 2017 ) use a factored model after performing dimensionality reduction . Inspired by similar motivations , Aydore et al . ( 2018 ) construct a regularizer using feature groupings obtained from a fast clustering method . They demonstrate that such a regularizer outperforms dropout and L2 regularization . However , they do not consider employing this structure in a deep model , opting instead for a wide shallow approach . Compared to Aydore et al . ( 2018 ) , our results illustrate the benefits of depth combined with spatial sparsity for brain image decoding . 2 FIXED GROUPING LAYER . Next , we formally define our idea of groups . Given a set of input variables X = { xi : 0 ≤ i < nin , i ∈ Z } , a grouping of variables , denoted by G is a subset of the power-set of X such that each xi is in at least one set in G. That is , G ⊂ 2X , such that , ∀xi ∈ X : xi ∈ ⋃ G. Each set in G is a group of variables . For example , in the case of a colored image , each pixel can be considered a variable with an associated feature vector of length 3 , i.e. , each xi represents a pixel and xi ∈ [ 0 , 1 ] 3 . A spatially smooth grouping of these variables corresponds to a segmentation of the image . Optionally , the groups can be mutually exclusive . An FGL layer takes as input nin vectors of cin length each , where the nin vectors are grouped into nout groups . Further , let cout be the length of the vector associated with each group . Note that these groups do not have to be mutually exclusive , but mutually exclusive groups offer benefits that we describe in the supplementary . The model architecture is allowed to use multiple channels ( analogous to the # of filters of standard convolutional networks ) . cin and cout are the number of input and output channels . Mathematically , the Fixed Grouping Layer is given by : z = A ( ( xv ) u ) + b , where : z ∈ Rnout , cout is the matrix representing the output with each row represents one group . x ∈ Rnin , cin is a matrix representing the input - one row for each input vector . A is a binary matrix that represents the grouping - Aj , i = 1 if and only if xi is in group j. represents the Hadamard product ( elementwise multiplication ) ( Horn , 1990 ) . u , v , b are parameters of the model . v is used for a linear transformation from Rcin to Rcout , i.e. , v ∈ Rcin , cout . u is a matrix of size nin × cout . b the bias , is a matrix of size nout × cout We denote the ith input vector , i.e. , the ith row of x , by xi . Observe that FGL is a fully connected layer when there is only one group which contains all variables . 3 EXPERIMENTS . We construct a deep neural network for classification using repeated layers of FGL ( and activation functions ) , followed by either a fully connected network or an FGL that groups all inputs into a single group . This is inspired from traditional CNN based classification models . We provide a visualization of a simplified model in Figure S2 . Regularization : We use weight normalization ( Salimans and Kingma , 2016 ) , which is a reparameterization of the weights of a neural network that decouples the norm and the direction of the weights . That is , for a single dimension of a fully connected layer , weight normalization reparameterizes w as w = g ( θ/||θ|| ) , where θ is a vector of the same length as w and g is a scalar . The network now learns g , θ instead of w. For FGL we apply weight norm on both u and v. Weight norm is applied by treating u as cout different vectors and v as the weights of a fully connected layer . We use Pytorch ( Paszke et al. , 2017 ) to implement models , and nilearn ( Abraham et al. , 2014 ) to preprocess and visualize fMRI images . Training was done using Adam ( Kingma and Ba , 2014 ) on 4 K80 GPUs . Code is available at https : //www.github.com/anon/repo and a minimal version is provided in the supplementary . 3.1 BASELINES . We consider brain decoding as a classification task , and use two common types of models as baselines : fully connected networks and convolution based models such as standard Convolutional Neural Networks ( CNNs ) and their CoordConv variant . To the best of our knowledge , our baselines include state of the art approaches on the considered datasets . Multinomial Logistic Regression ( LR ) : Multinomial logistic regression is a standard model ( Bishop , 2006a ) , given by : ŷ = softmax ( Wx+b ) , for an input x ∈ Rd , parameterized by weightsW ∈ Rk , d and bias b ∈ Rk where k is the number of possible labels . Here , ŷ is the vector of predicted probabilities for each class or label . Clearly multinomial logistic regression by itself uses no spatial information . Feedforward Neural Networks ( FNN ) : FNNs are ideal for tasks where the input has neither a grid-like structure nor a sequential structure . The architecture is an alternating sequence of linear transformations and activation functions ( Goodfellow et al. , 2016 ) . Convolutional Neural Networks ( CNNs ) : CNNs LeCun et al . ( 1995 ) , are a popular tool in deep learning . Many problems , where the inputs have a spatial representation , lend themselves to convolution . CNNs are popular not only for their flexibility but also because of the assumptions they make about the nature of the input - one of them being that dependencies between pixels are local and spatially invariant . These assumptions are usually appropriate for natural images . However , in the case of brain fMRI , since features are also dependent on position , i.e. , features are not position invariant , CNNs might not work as well . CoordConv ( CC ) : Liu et al . ( 2018 ) demonstrate that CNNs are sometimes unable to capture locationdependent spatial representations . e.g . transforming a pair of coordinates to a one-hot representation . One reason for this failure could be that the coordinate transformation problem directly conflicts with the underlying assumptions of CNNs . They propose the CoordConv layer as a solution . CoordConv is essentially a convolutional layer except that it includes coordinates as additional input channels . Thus , CoordConv enables CNNs where local dependencies change across the image .

Summary: This work introduces fixed grouping layers to deep learning models. Unlike convolutional layers, the fixed grouping layer only allows the output to be impacted by the specific inputs associated to it by its group. The paper stays firmly in the area of brain scans with the authors hoping to generalize to other applications later. Their experiments compare to what they consider to be the state-of-the-art approaches to the problem.