Towards a Deep Network Architecture for Structured Smoothness
Authors: Haroun Habeeb
Oluwasanmi Koyejo
Venue:  (2020)

Abstract: We propose the Fixed Grouping Layer (FGL); a novel feedforward layer designed to incorporate the inductive bias of structured smoothness into a deep learning model. FGL achieves this goal by connecting nodes across layers based on spatial similarity. The use of structured smoothness, as implemented by FGL, is motivated by applications to structured spatial data, which is, in turn, motivated by domain knowledge. The proposed model architecture outperforms conventional neural network architectures across a variety of simulated and real datasets with structured smoothness. 

Published as a conference paper at ICLR 2020
TOWARDS A DEEP NETWORK ARCHITECTURE FOR STRUCTURED SMOOTHNESS
Haroun Habeeb
Sanmi Koyejo
Department of Computer Science
Department of Computer Science
University of Illinois at Urbana Champaign Champaign, IL 61820 ha Maroun7@gmal com
University of Illinois at Urbana Champaign
Champaign, IL 61820 sanmi @ iiliinoooi edu
ABSTRACT
We propose the Fixed Grouping Layer (FGL); novel feedforward layer designed incorporate the inductive bias of structured smoothness into a deep learning model. FGL achieves this goal by connecting nodes across layers based on spatial similarity. The use of structured smoothness, as implemented by FGL, motivated by applications to structured spatial data, which is, in turn, motivated by domain knowledge. The proposed model architecture outperforms conventional neural network architectures across variety of simulated and real datasets with structured smoothness.
INTRODUCTION
The effectiveness of predictive models often depends on the choice of inductive bias, and the extent to which this inductive bias captures real-world structure. For instance, one example of such bias encoding leading to improved performance is convolution. In principle, convolutional weights could be learned directly from data. However, in practice, imposing this structure leads to improved performance when compared to fully connected models, and as result, Convolutional neural networks (CNNs) have enjoyed wide use for computer vision tasks (Krizhevsky et 2012).
Similarly, recurrent neural networks such as LSTMs are effective for text (Sundermeyer et al. 2012
and certain graphical models are ideal for sentence segmentation and labeling (Lafferty et al. 2001 Our work follows this philosophy. Specifically, we propose feedforward layer for deep neural networks that is suitable for neuroimaging and potentially useful for other data where variables can be grouped due underlying structure.
Data with multiple input variables often exhibit some structure. For example, the El Nino dataset (Bay al., (0000) consists of measurements by weather buoys in the ocean, and expects that nearby buoys can be grouped together. Similarly, socio-economic data can often be grouped together by geographic proximity. Financial market data of individual stocks can be grouped together based on the industrial sector to which a company belongs. Along similar lines, brain parcellations are well studied paradigm for capturing the structure of brain activity (Thirion al. 2014), often via statistical parcellation based on ward clustering (aard Jr 1963). The result of ward clustering is tree where leaf nodes represent voxels of the brain and interior nodes represent grouping of voxels into spatial clusters. Figure visualizes the output of ward clustering at various granularities when applied to the human connectome project resting state brain data (Van Essen al. 2012).
Our primary technical contribution is the Fixed Grouping Layer (FGL). FGL is designed to extract features within each group, and additionally guarantees that each output vector is only affected by the input vectors related to it by the specified grouping. We demonstrate the benefit of using FGL on simulated experiments and real neuroimaging data. We compare FGL against fully connected networks, convolutional neural networks, CoordConv (Liu al.) 2018)) and closely related method proposed by Aydore et al. (2018). We extensively evaluate the performance of FGL on simulated and real brain imaging data showing improved performance.
Contribitions:

---

Published as a conference paper at ICLR 2020
regions
32 regions
256 regions
1024 regions
Figure I: Computed brain parcellation at various granularities (details in the text). The text in the top left indicates the number of regions. Each color in each figure corresponds region/group. Notice the consistency between parcellations while increasing granularity.
1.1 DECODING BRAIN IMAGES: BACKGROUND AND RELATED WORK
Functional Magnetic Resonance Imaging (fMRI) is a popular brain imaging technique which measures physiological correlate of neuron activity (Huettel al.) 2004). The brain imaging scans generally of two kinds: resting state and task data. Resting state data (rfMRI) is collected while the subject is at rest, i.e., while the subject is not actively engaged in a task. Task data (uMRI) is collected while the subject is engaged in predefined task, for example, a motor task such as moving their fingers. fMRI data can be represented as 3-dimensional images, and have rich structure that has been studied extensively, including in the ML literature (Koyejo et al.) [2014] Park et al.) 2013). Importantly, coarse correspondences have been discovered between brain regions and specific functions or behavior (Frost and Goebel 2012 Sporns 2013)
We particularly focus on brain decoding a standard task in fMRI brain data analysis where the brain image is used to predict the associated task or stimulus. Broadly, there are two types of brain decoding models: end to end models and models which perform dimensionality reduction followed by low-dimensional prediction. On one hand, dimension reduction directly captures the notion of grouping variables together. On the other hand, end to end models rarely employ brain spatial structure. This observation motivates work. In recent years, decoding from fMRI studies has been attempted using a variety of methods: Bzdok et al.) (2015) use factored logistic regression while Sarraf and Tofighi (2066) use convolutional neural networks. Mensch al. (201)) use factored model after performing dimensionality reduction. Inspired by similar motivations, Aydore et al. (2018) construct a regularizer using feature groupings obtained from a fast clustering method. They demonstrate that such regularizer outperforms dropout and L2 regularization. However, they do not consider employing this structure in a deep model, opting instead for wide shallow approach. Compared to Aydore et al. (2018) our results illustrate the benefits of depth combined with spatial sparsity for brain image decoding.
2 FIXED GROUPING LAYER
Next, we formally define our idea of groups. Given a set of input variables X {z; 0 < < Mins& i Z}. grouping variables, denoted by G is a subset of the power-set of X such that each x;; is in at least one set in G. That is.
G x such that, Va, X: x, Ug.
Each set in g is a group of variables. For example, in the case of a colored image, each pixel can be considered variable with associated feature vector of length 3, i... each æ; represents pixel and I; E [0, 1).. A spatially smooth grouping of these variables corresponds to a segmentation of the image. Optionally, the groups can be mutually exclusive.
2

---

Published as a conference paper at ICLR 2020
Random
Regions
Activations per Region (2)
Labels
y=5
07
*
y=2
02
Nuriee of datapoint
(a) Generating Process
(b) Voronoi diagrams
(c) Probabiliit of label
Random images are aggregated over regions/groupings to obtain activations (z) that are used to assign labels. Groupings are comprised of multiple smaller regions, all spatially connected. (b) Voronoi diagrams and induced groups. The upper left image is a Voronoi diagram of 8 random sites (blue points) and lines that partition space into sets of points closest the same site. The other images are possible groupings using partitions. of assigned label. The y-axis is probability and x-axis is number of points. Most datapoints have Pr(y) > 0.5 (labels are low-noise).
Figure 2: (a) Simulated Dataset:
(c) Histogram of probability
An FGL layer takes as input 7in vectors of G. length each, where the Tain vectors are grouped into Nout groups. Further, let Cout be the length of the vector associated with each group. Note that these groups do not have to be mutually exclusive, but mutually exclusive groups offer benefits that we describe in the supplementary. The model architecture is allowed to multiple channels (analogous to the # of filters of standard convolutional networks). Cin and Cout are the number of input and output channels. Mathematically, the Fixed Grouping Layer given by:
2 = A ((re) @ u) b,
where: fout is the matrix representing the output with each row represents one group. E fi is matrix representing the input one row for each input vector. A is binary matrix that represents the grouping AJ, if and only if x is in group j. represents the Hadamard product (elementwise multiplication) (Horn 1990).
U. are parameters of the model. is used R Fost i.e., v E Rornt is a matrix of size 7in X Cout- b the bias, is matrix of size Rout X Cout We denote the ith input vector, i.e., the 7th row of 2r by ry Observe that FGL. a fully connected layer when there only one group which contains all variables.
linear transformation from R
EXPERIMENTS
We construct deep neural network for classification using repeated layers of FGL (and activation functions), followed by either a fully connected network or an FGL that groups all inputs into single group. This is inspired from traditional CNN based classification models. We provide visualization of a simplified model in Figure [22
We use weight normalization (Salimans and Kingma 2016). which is reparame- terization of the weights of a neural network that decouples the norm and the direction of the weights. That is, for single dimension of fully connected layer, , weight normalization reparameterizes as w g0/|||0)) [ where 0 is a vector of the same length as and g is scalar. The network now learns g, 0 instead w. For FGL we apply weight norm both and Weight norm is applied by treating u as Cout different vectors and as the weights of a fully connected layer.
Regularization:
We Pytorch (Paszke et al., 2017) to implement models, and ni aiileane rn (Abraham al. 2014) to preprocess and visualize fMRI images. Training was done using Adam (Kingma and Ba 2014 on 4 K80 GPUs. Code is available at fttttps:://e wwwww//www..//////////////w../// qithub. com/ anon repo minimal version is provided in the supplementary.
3.1 BASELINES
We consider brain decoding as a classification task, and use two common types of models as baselines: fully connected networks and convolution based models such as standard Convolutional

---

Published as a conference paper at ICLR 2020
Neural Networks (CNNs) and their CoordConv variant. To the best of our knowledge, our baselines include state of the art approaches on the considered datasets.
Multinomial Logistic Regression (LR): Multinomial logistic regression is standard model (Bishop 2006a), given by: 9 = softmarr(W W +6). for an input æ E R. parameterized by weights W R and bias b E IRth where k is the number of possible labels. Here, y is the vector of predicted probabilities for each class or label. Clearly multinomial logistic regression by itself uses no spatial information.
Feedforward Neural Networks (FNN): FNNs are ideal for tasks where the input has neither a grid-like structure nor a sequential structure. The architecture is an alternating sequence of linear transformations and activation functions (Goodfellow et al. 2016).
Convolutional Neural Networks (CNNs): CNNs LeCun et al. (1995), are a popular tool in deep learning. Many problems, where the inputs have a spatial representation, lend themselves to con- volution. CNNs are popular not only for their flexibility but also because of the assumptions they make about the nature of the input one of them being that dependencies between pixels are local and spatially invariant. These assumptions are usually appropriate for natural images. However, in the case of brain IMRI, since features are also dependent on position, i.e., features are not position invariant, CNNs might not work as well.
ConrdConv (CC): Liu et al. (2018) demonstrate that CNNs are sometimes unable to capture location- dependent spatial representations. e.g. transforming a pair coordinates to a one-hot representation. One reason for this failure could be that the coordinate transformation problem directly conflicts with the underlying assumptions of CNNs. They propose the CoordConv layer as a solution. CoordConv is essentially convolutional layer except that it includes coordinates as additional input channels. Thus, CoordConv enables CNNs where local dependencies change across the image.
3.2 SIMULATED DATA
First, discuss grouping via Voronoi Diagrams. Given set of points, called sites, a Voronoi diagram (Aurenhammer) [9ae et al.] 2009) is the division of a plane into regions based on distance to the sites. Usually, positions whose closest site is the same are grouped together. Consider the grouping induced by a set of m sites P = (p: Ps E [0, 22, 0 s i < m) for some indicating the size of the plane: gi [ min [zj
pal = i) OA S
We use Voronoi diagrams because they create regions which are spatially connected. To increase the complexity the task, we use groupings which are unions of arbitrarily chosen Voronoi regions resulting in groups comprised of multiple spatially connected regions which may not be connected to each other. We provide an example in Figure 26
Consider input data sampled from Gaussian prior: ~ (0 S), where 0 is a zero-vector and S is an arbitrary covariance matrix of appropriate size. We let x E R=² for an integer the idea being that z is flattened version of grayscale image of size s x s. Next, suppose that datapoints are labelled based on linear function. That is, 2|x N(Fx, E), for fixed covariance matrix Σ. and matrix F of size k x 82 where i is the number of labels. The label, y, is assigned based on y = arg max, softmaxz (WonnaaaW))) (Wz);- for a full rank matrix W.
We briefly analyze this simple generative model in the context of FGL. Using conjugate priors (Bishop 2006b), it straightforward to show that,
2/2 NNFTFF -- (S- (S-1+FF=-1))))
(1)
To explore the implications of equation (I). consider an F that is sparse such that the non-zero positions in each row of F correspond
segmentation of the input image (a grouping of pixels). For example, circular patches on the image, or in case, Voronoi diagrams. Additionally, if Σ is an identity matrix, then each dimension of z corresponds to the sum of values of group of pixels.
Dataset: We create a dataset which samples x from the the Gaussian prior with S being an identity matrix. We use s = 128 so that each x can be interpreted as a square image. We create F by first creating Voronoi diagram of 512 randomly selected points, and then merging these regions into = 32 groups. We sample 2 from the the corresponding conditional distribution. We fix a random W and then assign the label y with highest likelihood to the datapoint We sample 50000 points to
4

---

Published as a conference paper at ICLR 2020
Accur acy w/s Traction dala training
F1
fraction data for training
Probebility label (disagreement)
0.0
Number o' deta points
Tex: arauwey
FeL
FSL Cotte
Comy
0.4
120
0.4
Teat
0.2
130
0.0
Fract of total data Lood for train
0.2
0.4
0.6
0.5
Fraction of toral data used for traiking
0.2
04
25
00 07
Priy]
0.
(a)
(b)
(c)
Figure 3: (a) Test accuracy (with error bars) on held out 20% of simulated dataset vs. fraction of data used for training. The graph indicates that FGL has better empirical sample complexity. The small magnitude of error in estimation of performance indicates that models are well trained and the difference is due to the models themselves. (b) Minimum (across classes) F1 Score held out test set vs. fraction of data used for training. The difference in performance is not due to performance on (c) Hístogram of ground truth probability of labels for points where FGL is correct but CNN misclassifies. This demonstrates that the misclassification by CNN is not only on noisy datapoints but also for datapoints where the label should be clear.
single class/region, but rather across all labels.
create the simulated dataset. A visualization of the process is provided in Figure 2a To ensure that the dataset wasn't 't too noisy, we plot a histogram of probability assigned label in Figure 22 The histogram shows that only small number of datapoints are noisy in most cases, the assigned label has a probability of at least 0.5.
3.2.1 MODELS
To demonstrate the benefit of using the Voronoi Diagram during classification, we train 4 models Logistic Regression (I.R), Convolutional Neural Network (Conv), CoordConv variant (CC) of the same CNN, and a model using our proposed layer FGL followed by fully connected network. Our FGL model is provided the voronoi regions. The number of parameters in each model is roughly the same. Since the dataset uses labels that are linear in terms of æ, we use no non-linear activations in any of our models. We found that using maxpooling the CNN and CoordConv hurt performance. We don't It report results with an FNN because it performs similar to LR.
3.2.2 PROCEDURE AND ANALYSIS
We create test set using 20% of the simulated dataset. The remaining points are used for training. For each model, we train using various quantities of available data, and test on the held out set. The results are aggregated over 10 runs with randomly sampled test set for each run. A plot of the test accuracy vs. fraction of data used for training given in Figure We find that the standard deviation of accuracies of these models is small indicating that the failures are not due poor initialization or poor training but rather a difference models.
This experiment was designed to demonstrate failure of convolution based models and also fully connected methods. Although this satisfies our intuition that using spatial structure should help drastically improve performance, not. The first thing to check was the probability of assigned labels for these points a histogram of the same for a random subset of the testing set is provided in Figure 3c The next sanity check is to ensure that the drop in performance isn' 't just for one set of regions or one class. To that end, we plot the lowest Fl score (lowest across classes) in Figure 3bb We see the same trend FGL performs better than CNNs, CoordConv and Logistic Regression. These plots indicate the validity of the gain in performance it is due neither noisy labels, nor failure on any one label. Hence, using grouping of variables seems to provide significant benefit.
investigate the datapoints at which the CNN failed but FGL did
3.3 FMRI CONSTRAST PREDICTION
We evaluate our models on 5 datasets which were used by Mensch et al. (2017): Archi (Pinel et al. 2007), Brainomics (Orfanos et al. 2017), Cam-CAN (Shafto et al. 2014). LA5e (Poldrack et al.
5

---

Published as a conference paper at ICLR 2020
Archi
Brainomics
Cam-CAN
out semple arcuray
0.90
of sample test ancuracy
0.85
0.6
0.80
0.75
0.8
0.625
0.70
FGL Conv
FGL
RGL
0.2
0.65
Conv
0.575
Conv
0.60
out
0.6
0.550
0.1
0.2
0.3
34
0.5
06
0.6 subjects training
0.4
0.1
0.2
0.3
0.6
0.7
Praction subjects used for tortraniing
Traction of subjects used for training
Figure Test accuracy: Out of sample accuracy measured on 30% of dataset v/s fraction of subjects used for training. FGL performs well even when small amount of data is used for training.
Table 1: Test Accuracy per dataset, model. P-values dataset-specific- in this Wilcoxon test to compare our method to each of the others on the HCP dataset. We reported p-values for HCP only, the dataset was large, and the performance improvement gaps were not extremely obvious. p-value from one sided Wilcoxon rank sum test shows that FGL (3 layer) is better than other models on HCP dataset.
the
Archi
Cam
HCP
LASc
p-value (HCP)
LR FNN
81.00%
74.42%
63.29%
91.70%
61.12%
5.413c-06
82.72%
81.47%
61.52%
92.16%
60.86%
3,789c-05
Conv
84.23%
90.85 %
63.77%
91.38%
61.99%
5.413e-06
CC
83.96%
90.64%
63.07%
91.52%
62.04%
9.083e-05
FGL (1 Layer)
85.78%
88.65%
67.23%
92.70%
64.49%
0.001097
FGL (2 Layers)
85.78%
89.87%
67.46 %
92.67%
64.57%
0.000525
FGL (3 Layers)
87.07 %
90.38%
67.27%
93.36%
64.24%
Feature Grouping (b=10)
75.80%
59.00%
53.09%
Feature Grouping (b=50)
76.81%
59.77%
57.17%
Feature Grouping (b=100)
78.55%
59.66%
60.54%
Feature Grouping (b=200)
73.48%
58.56%
53.79%
2016) and HCP (Van Essen al. Archi, Brainomics, Cam-CAN and LA5c are small datasets with 2340, 1786, 3025 and 5756 images respectively. On the other hand, HCP is larger with 18070 images. These datasets have different sets of labels corresponding to different cognitive processes. Each dataset is publicly available on NeuroVaul
2012). They have 78, 94, 605, 191 and 787 subjects respectively.
: (Gorgolewski et al., 2015), an aggregation of fMRI datasets. Our only required preprocessing for the task contrast data was upsample Cam-CAN and Brainomics to the same resolution as HCP.
The assessment of fMRI decoding is important topic in its right due unique spatial characteristics and typical sample sizes. Varoquaux et al. (0117) show that leave-one-out strategies for cross validation can be unstable, and suggest that using reasonable defaults is good strategy. Additionally, it is well known that having common subjects between train and test datasets can lead to misleading results. This is because such from one subject to another. Hence, we evaluate models via out-of oft-of-sample accuracy, i.e., we hold out some subjects (30%) for the testing dataset in each run. Further, we train all models with reasonable defaults that we found were not critical for model performance.
does measure how well a model can generalize
3.3.1 PARCELLATION
Since Thirion et al. (2014) showed that ward clustering provides good parcellations of the brain, we perform ward clustering on a fraction of HCP resting state data (total size of 4TB). We downsample the resting state time series data to about 1% of the original frequency, and use the brain activations at each position as the feature vectors for clustering. The downsampling is needed due to hardware constraints. We did not use task datasets for clustering since we would have to hold out more data for clustering- exacerbating data scarcity. Additionally, resting state data is more easily acquired
https::/neurrovaatt.oorgg

---

Published as a conference paper at ICLR 2020
(Biswal et al., 2010), and there are strong correlations between tfMI and rMRI (Smith et al. 2009).
Thus, using rfMII should provide a good if not better parcellation of the brain.
To make deep network using FGL we require hicrarchical clustering and not just parcellation. Hence, instead of using the segmentation produced by the parcellation algorithm provided by nilearn, we use the computed ward tree. We then slice into the ward clustering parcellations with 32, 256 and 1024 regions. These have been visualized in Figure [ Clearly, these groups are spatially connected. We need groupings of voxels into 1024 groups, then grouping of these 1024 groups into 256 groups and finally a grouping of 256 groups into 32 groups. Since ward clustering is hierarchical clustering scheme and outputs a tree structure, we extract these groupings by making appropriate cuts into the tree.
produce
3.3.2 MODELS
Fully Connected Models: We experimented with Fully Connected Neural Networks (FNN) and Multinomial Logistic Regression (LR) with and without Dimension Reduction using our parcellation. We found that using Dimension Reduction reduced performance and hence do not report it. For FNNs, we tried 2 and 3 layer versions with intermediate sizes chosen from 64, 128, 256 and 512. The model with intermediate layer sizes of 512 and 128 worked best. The aforementioned models take masked fMRI image as input and we used the MNI152 mask provided by nilearn. Emperically, we find that, for brain decoding, using linear activation performs better than using non-linear activations. For this reason, we use a linear activation for our models. Unfortunately, we are unclear about why this occurs for this domain. We also evaluate Feature Grouping suggested by [Aydore et al. (2018) which is also designed to exploit structured sparsity but does this using a wide model (implemented using multiple randomized clusterings), unlike our deep FGL approach. We used code provided by the authors.
Convolutional Neural Networks (Conv, CC): We experimented with a variety of architectures and found no improvement by using residual connections or Batch-Norm. We also report results using CoordConv. We found that using non-linext activations hurt the model's performance, similar to our finding with FNNs. Further, maxpooling also reduced performance. The architecture is 5 3-D convolution layers of stride 2 and kernel size The input volumes have size 91 x 109 x 91, and convolution reduces the volume to 2 3 x 2 with 128 channels. We flatten this volume and pass it through a fully connected network to get the score for each label. The architecture for the CoordConv is identical to the CNN since CoordConv only concatenates few input channels to the input image. We use Conv to refer to the Convolutional Neural Network and CC to refer to the CoordConv variant.
FGL: We use 3 layers of FGL, each of which use the Parcellation described earlier. The input images have 212455 voxels after masking. We treat every voxel as a variable with a single feature. voxels are then reduced to 1024 groups with feature vectors of length 8 each. Next, these groups are reduced to 256 variables with 64 features and finally to 32 variables with 128 features. The final prediction is made by flattening the output of the last FGL layer and passing it through a fully connected layer. The resulting number of parameters is roughly 2 million, which is also roughly the same number of parameters used for CNN and CC. While this is lot of parameters, we found that reducing the number of parameters by changing the number of features for each intermediate variable decreases performance for both convolution and FGL.
3.3.3 PROCEDURE AND RESULTS
We split each dataset multiple (10) times into a train and test set. The split is done such that no subject in the test set appears in the training set. In each split, 30% of subjects are used for testing, and all or a part of the remaining subjects are used for training. Convolution based models were trained for 50 epochs, feedforward neural networks for 30 and FGL for 20. These hyperparameters were selected by monitoring for overfitting on the training set (using a further validation split). We perform experiments to study: (1) the benefit of FGL given a lot of data, (2) the benefits of FGL at small sample sizes, (3) the effect of depth, and (4) the effect of intermediate vector length.
Large sample setting: The first experiment uses all of the training data (70% of total data) to train. We report oft-ole accuracy in Table We also report the p-values of one sided Wilcoxon rank sum tests between the performance of each model compared to 3 Layer FGL the HCP dataset.
7

---

Published as a conference paper at ICLR 2020
Ablation: Across dacasets
Ablation: HCP
out of sample best accuracy
out of sample bast accuracy
0.930
0.9
0.325
0.8
Archi -+-
Brainomics
0 0.920
CamCAN
0.315
0.310
0.303
Conv
RGL
0.6
10
15
10 Intermeciats vector length (c (c_out)
Intermediate vector langth (c_our)
(a)
(b)
Figure 5: Ablation of intermediate vector length: (a) On HCP, along with baselines. (b) On all datasets, Increasing the intermediate vector length improves performance, except on Cam-CAN.
Small sample setting: The second set of experiments varies the fraction of data used for training on the smaller datasets namely, Archi, Brainomics and Cam-CAN. We explore small sample performance because limited sample sizes are typical for fMRI studies. Figure plots plotss test accuracy against fraction of data used for training. It demonstrates that FGL outperforms the baselines even when only a small amount of training data is used.
Effect of depth, FGL depth vs. width: We train two additional models to study the effect of depth: one which uses only the first layer of FGL, and another that uses the first two layers. In Table [ they're 're referred to "FGL (1 Layer)" and "FGL (2 Layers)" respectively. The results show that increasing FGL. depth provides statistically significant improvement in test accuracy on the HCP dataset. When compared to Feature grouping (Aydore al. 2018), the results in Table Sablee[shon show that even after extensive tuning (we used the reported best performing settings and attempted additional tuning), this approach not competitive suggesting significant representation benefit by exploiting FGL depth vs. width for this task. Unfortunately, the provided code did not scale to the largest datasets hence the missing results on HCP.
To study the effect of intermediate vector length, we train five single layer FGL models with intermediate vector lengths (Cout) of 1, 2, 4, and 16. We plot the test accuracy against Cout in Figures 5a and 5b On 4 out of 5 datasets, increasing Cout improves test accuracy. This is expected because the classification task is not binary and each region of the brain could contribute multiple cognitive processes in different ways. However, the effect is not as pronounced on the Cam-CAN dataset. We suspect that this is because Cam-CAN has fewer labels (5) than the other datasets. We note that Cout could be interpreted as width of our model, similar to channel size CNNs, which is known to be important.
Effect of intermediate vector length:
These experiments demonstrate the clear benefit of using FGL compared other models with roughly the same number of parameters. When using 70% data for training, FGL provides 2-6% of improvement in test accuracy 4 of datasets. A similar trend exists data is used. As for the 5th dataset, Brainomics, FGL is on par with CNN based methods but better than Fully connected networks. While the effect of depth is not clear on smaller datasets, we note that on the HCP dataset, deeper models have statistically significant improvement in performance. Further, an increase in Cout also improves performance.
when smaller amounts of
Our main argument is that current methods discard important information about spatial smoothness as encoded by hierarchical spatial clustering. As pointed out, the main cost of our method in constructing these hierarchies. For brain imaging, most datasets include readily available resting state data. From the larger view, we plan to encourage application communities to develop application- specific FGL architectures, which can be shared across several related tasks.
CONCLUSION
In this work we propose new layer architecture, the Fixed Grouping Layer (FGL), parameterized by grouping of input variables. FGL expliciity extracts features within each input group. This is in
8

---

Published as a conference paper at ICLR 2020
contrast to convolution which extracts local features across the input, and fully connected networks which extract both global and local features. We demonstrate the benefit of using FGL on real fMRI datasets of different sizes. Future work will involve the application of FGL to other tasks and application domains.
REFERENCES
Abraham, A., Pedregosa, F., Eickenberg, M., Gervais, P., Mueller, A., Kossaifi, J., Gramfort, A., Thirion, B., and Varoquaux, G. (2014). Machine learning for neuroimaging with scikit-leam. Frontiers in neuroinformatics, 8:14.
Aurenhammer, F. (1991). Voronoi diagrams a survey of fundamental geometric data structure. ACM Computing Surveys (CSUR), 23(3)345-405.
Aydore, S., Thirion, B., Grisel, O., and Varoquaux, G. (2018). Using feature grouping stochastic regularizer for high-dimensional noisy data. arXiv preprint arXiv:
Bay, S. D., Kibler, D. F,, Pazzani, M. J., and Smyth, P. (2000). The uci kdd archive of large data sets for data mining research and experimentation. SIGKDD explorations, 202):81-85.
Bishop, C. M. (2006a). Pattern recognition and machine learning, chapter Probabilistic Discrimina- tive Models, pages 203-213. Springer.
Bishop, C. M. (2006b). Patten recognition and machine learning, chapter Probability distributions, page 117. Springer.
Biswal, B. B., Mennes, M., Zuo, X.-N., Gohel, S., Kelly, C., Smith, S. M., Beckmann, C. F., Adelstein, J. S., Buckner, R. L., Colcombe, S., et al. (2010). Toward discovery science of human brain function. Proceedings of the National Academy of Sciences, 107(10):4734-4733
Bzdok, D., Eickenberg, M., Grisel, O., Thirion, B., and Varoquaux, G. (2015). Semi-supervised factored logistic regression for high- -ibeesinnniinal neuroimaging data. In Advances in neural information processing systems, pages 3348-334 -356.
Frost, M. A. and Goebel, R. (2012). Measuring diructural-functional correspondence: spatial variability of specialised brain regions after macro-anatomical alignment. Neuroimage, 59(2): 1369- 1381.
Glorot, X. and Bengio, (2010). Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pages 249-256.
Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep learning. MIT press.
Gorgolewski, K. J., Varoquaux, G., Rivera, G., Schwarz, Y., Ghosh, S. S., Maumet, C., Sochat, V. V., Nichols, T. E., Poldrack, R. A., Poline, J.-B., et al. (2015). Neurovault. org: web-based repository for collecting and sharing unthresholded statistical maps of the human brain. Frontiers in neuroinformatics, 9:8.
He, K., Zhang, X., Ren, S., and Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034.
Horn, R. A. (1990). The hadamard product. In Proc. Symp. Appl. Math, volume 40, pages 87-169.
Huettel, S. A., Song, A. W., McCarthy, G., et al. (2004). Functional magnetic resonance imaging, volume 1. Sinauer Associates Sunderland, MA.
Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980.
Koyejo, O. O., Khanna, R., Ghosh, J., and Poldrack, R. (2014). On prior distributions and approximate inference for structured variables. In Advances in Neural Information Processing Systems, pages 676-684.

---

Published as a conference paper at ICLR 2020
Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pages 1097-1105.
Lafferty. J., McCallum, A., and Pereira, F. C. (2001). Conditional random fields: Probabilistic models for segmenting and labeling sequence data. Pruceedings of the 18th International Conference on Machine Learning.
LeCun, Y., Bengio, Y., al. (1995). Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 33)1(10):199.
Liu, R., Lehman, J., Molino, P., Such, F. P., Frank, E.,, Sergeev, A., and Yosinski, J. (2018). An intriguing failing of convolutional neural networks and the coordconv solution. In Advances in Neural Information Processing Systems, pages 9628-9639.
Mensch, A., Mairal, J., Bzdok, D., Thirion, B., and Varoquaux, G. (2017). Learning neural rep- resentations of human cognition across many fmri studies. In Advances in Neural Information Processing Systems. pages 1883-5893
Okabe, A., Boots, B., Sugihara, K., and Chiu, S. N. (2009). Spatial tessellations: concepts and applications of Voronoi diagrams, volume 501. John Wiley & Sons.
Orfanos, D. P., Michel, V., Schwartz, Y., Pinel, P., Moreno, A., Le Bihan, D., and Frouin, V. (2017). The brainomies/localizer database. Neuroimage, 144:309-314.
Park, M., Koyejo, O., Ghosh, J., Poldrack. R., and Pillow, J. (2013). Bayesian structure learning for functional neuroimaging. In Artificial Intelligence and Statistics, pages 489-497.
Paszke, A., Gross, S.. Chintala, S., Chanan, G., Yang. E., VeVoto, Z., Lin, Z., Desmaison, A., Antiga,
L., and Lerer, A. (2017). Automatic differentiation in pytorch. In NIPSWW.
Pinel, P., Thirion, B., Meriaux, S., Jobert, A., Serres, J., Le Bihan, D., Poline, J.-B., and Dehaene, S. (2007). Fast reproducible identification and large-scale databasing of individual functional cognitive networks. BMC neuroscience, 8(1):91.
Poldrack, R. A., Congdon, E., Triplett, W., Gorgolewski, K., Karlsgodt, K., Mumford, J., Sabb, F., Freimer, N., London, E., Cannon, T., et al. (2016). A phenome examination of neural and cognitive function. Scientific data, 3:160110.
Salimans, T. and Kingma, D. P. (2016). Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances Neural Information Processing Systems, pages 901-909.
Sarraf. and Tofighi. G. (2016). Deep learning- based pipeline to recognize alzheimer's disease using finri data. In 2016 Future Technologies Conference (FTC), pages 816-820, IEEE.
Shafto, M. A., Tyler, L. K., Dixon, M., Taylor, J. R., Rowe, J. B., Cusack, R., Calder, A. J., Marslen- Wilson, W. D., Duncan, J., Dalgleish, T., et al. (2014). The cambridge centre for ageing and neuroscience (cam-can) study protocol: a cross-sectional, lifespan, multidisciplinary examination of healthy cognitive ageing. BMC neurology, 14(1):204.
Smith, S. M.. Fox, P. T., Miller, K. L., Glahn, D. C., Fox, P. M., Mackay, C. E., Filippini, N., Watkins, K. E., Toro, R., Laird, A. R., et al. (2009). Correspondence the brain's 's functional architecture during activation and rest. Proceedings of the National Academy Sciences, 106(31):1304) 13045.
Sporns, O. (2013). Structure and function of complex brain networks. Dialogues in clinical neuroscience, 15(3):247.
Sundermeyer, M., Schlüter, R., and Ney, H. (2012), Lstm neural networks for language modeling, In Thirteenth annual conference of the international speech communication association.
Sutskever, I., Martens, J., Dahl, G. E., and Hinton, G. E. (2013). On the importance of initialization and momentum in deep learning. ICML (3), 28(1 139-1147):5.
10

---

Published as a conference paper at ICLR 2020
Thirion, B., Varoquaux, G., Dohmatob, E., and Poline, J.-B. (2014). Which fmni clustering gives good brain parcellations? Frontiers in neuroscience, 8:167.
Van Essen, D. C., Ugurbil, K., Auerbach, E., Barch, D., Behrens, T., Bucholz, R., Chang, A., Chen,
L., Corbetta, M., Curtiss, S. W., al. (2012). The human connectome project: data acquisition perspective. Neuroimage, 62(4):2222-2231.
Varoquaux, G., Raamana, P. R., Engemann, D. A., Hoyos-Idrobo, A., Schwartz, Y., and Thirion, B. (2017). Assessing and tuning brain decoders: vross-validation, caveats, and guidelines. Neurolmage. 145:166-179,
Ward Jr, J. H. (1963). Hierarchical grouping optimize an objective function. Journal of the American statistical association, 58(301):236-2)

---

Published as a conference paper at ICLR 2020
3D Convolation Platten Linear
91x100xd1
45x54x45
2zx2722
11x1bxtt
channel
channel
32 channel
channel
128 channel
128 channel
The architecture used for Convolutional Neural Networks and CoordConv. In CoordConv variant, each volume is concantenated with channels coordinate along each dimension being passed in as input. The volumes are written above each volume while the number of channels for each volume is written underneath it.
Figure S1: Baseline:
Table S1: Test Accuracy to study imperfect A: Reported numbers for each model are averaged over multiple (3) runs.
Model
Test accuracy
T.R.
43.0%
Conv
56.4%
CC
56.6%
FGL. (16 clusters)
61.3%
FGL (32 clusters)
66.1
FGL (48 clusters)
66.3%
FGL (perfect A)
70.2%
Supplementary Materials
SI
FGL DETAILS
Input Specification: In this work deal with image-like data, either in 2D 3D. Consider input with s pixels or voxels in c channels for example, 64 x 64 image with RGB colors will have 4096 and 3. Such an input treated variables with feature vectors of length for each variable.
Groupings: Since the output of the first FGL layer is feature vectors for each group, the grouping of the second FGL layer must group together the outputs of the first layer. Hence, we need a hierarchical structure with input variables the leaf nodes. this work, we use a Ward clustering the brain, however, other clusterings may be more appropriate in other settings.
S2 ABLATION: IMPERFECT A
We ran an experiment using the simulated dataset to estimate how robust FGL was imperfect A. Apart from providing FGL the true voronoi diagrams, we also ran FGL using the clusters from K-Means clustering That is, Ay = pixel was cluster j according to K-Means. We did this using 16, 32 and 48 clusters obtained by clustering pixels in the training dataset. The results reported in []] We see that although there a drop when using imperfect A, FGL still outperforms logistic regression and convolution. This emphasises the benefit of capturing structure.
We do see a larger drop when we use clustering that is not representative enough (16 clusters), but don't see much gain from using representative one (48 clusters).
S3 FGL VARIANTS
While the FGL model is straightforward, multiple variants are possible. First, notice that FGL essentially the following three operations (ignoring the bias):

---

Published as a conference paper at ICLR 2020
Linear transformation:
The multiplication, zt, transforms the data from one basis to another using a linear transform v.
Rescaling: The hadamard product with u rescales each vector along each dimension inde- pendently
Aggregation:
The multiplication by A aggregates the vectors (av) within each group using summation.
Performing these operations in a different order creates some basic variants: for example, we could aggregate within groups, then rescale, and finally perform a linear transformation. These changes to operation order will require the parameters to be defined differently. For example, if the hadamard product with is done after aggregation, then u will need have Nout rows.
S3.1 REDUCTIONS
Another interesting variant is to replace the aggregation with max operation within each group along each dimension. We think this is similar to doing maxpool operation in convolution neural networks while the summation by A is similar to a weighted-sum-pont depending on the values of Ayi- Early results showed worse results when using the max reduction variant and hence did not investigate it further. However, it might prove effective in cases where a signal being present in one variable within group is equivalent the group showing that variable.
S3.2 MULTIPLE GROUPINGS
Another possible benefit that we do investigate is the use of multiple variable groupings we can concatenate the A matrices that represent each grouping to make FGL extract features within each group from the union of both groups. That is, if A A are the matrices that represent two groupings, we could use A = [A A This would allow one to make use of multiple types of groupings. For example, reproducibility tradeoff studied by Thirion et al. (2014), and make use of both. Similarly, one could create parcellations from different datasets and use them at once. However, using single parcellation was sufficient to create a significant gain in performance, hence we don't go deeper in this direction. We mention few other variants that did perform as well in supplement S3
could create parcellations at different points of the accuracy-
S4 IMPLEMENTATION
Our implementation of FGL using PyTorch is available at https:://ittub. com/ anon /Iink /nink In this section, discuss some challenges in implementation - the number of input variables is large, performing A((vv) O u) as matrix multiplication is expensive. There are some ways work around this:
Since A is binary matrix,
treat ((xv) O u) as matrix of embeddings, and lookup the indices at which A is non-zero, then performing necessary aggregation.
If the variable groups are mutually exclusive that is, each input variable only belongs to one group, then A ((zv) @ u) can be performed by scattering (æv) according to the indices at which A is non zero.
$4.1 FGL INITIALIZATION
Prior literature (He et al. 2015 Glorot and Bengio 2010 Sutskever et al.) 2013) has shown that initialization deep networks matters. Generally. a layer's s weights are randomly initialized by sampling from uniform distribution, denoted by U[-m, m] for some m based the number of inputs, outputs and the activation function. Hence, after minor modifications for different activation functions, we the following initialization:
(1+E, Ag)
(1+ Aji)
Uik U - E, - ARa)
Vij U -V
EE
1 5cin
V 1 5c
2

---

Published as a conference paper at ICLR 2020
FGL
FGL
FC
Labels
Figure S2: Illustration of FGL. Given the numbered hierarchical segmentation, FGL extracts features for each segment. The inputs are variables corresponding to segments square, which are first grouped as {(1,2), (3, 4, ), {6, 7), {8, ). The resulting 4 groups are then grouped into 2 groups using the grouping {{1, 2). (3,4]} The output is passed to a fully connected network to predict labels. Note that intermediate layers can use feature vectors of length greater than 1.
SS PARAMETER SHARING
One of the major benefits of using convolution is that it performs parameter sharing which comes with its benefits. Adapting FGL perform parameter sharing is much harder. Typically, fully connected from Yiin X Gin numbers to Nout X Cout numbers would require 7ki x nout X Cin X Cout parameters. But this number astronomical. To avoid using as many parameters, we decompose the operation into a multiplication by followed by a Hadamard product with u. Doing so reduces the number of parameters to Gn X Cout + Thin X Cout. This much more tractable, but more reduction might be possible: sharing parameters between groups seems lucrative, unfortunately. different groups can have different sizes and an arbitrary ordering prevent us from parameter sharing further. If group sizes were constant and an ordering of variables was fixed, it would be possible to further reduce the number of parameters from ( to Ogrroussize).
S6 CNN ARCHITECTURE
We use a straightforward architecture for convolution repeated convolutional layers of stride 2 and kernel size of 4 with appropriate padding, followed by a fully connected network. We found that using maxpool for downsampling reduced performance, and so did using non- non-linear activation functions. A visualization is provided in ST


Reviews:
[{"review_id": "H1emraEJcB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work introduces fixed grouping layers to deep learning models. Unlike convolutional layers, the fixed grouping layer only allows the output to be impacted by the specific inputs associated to it by its group. The paper stays firmly in the area of brain scans with the authors hoping to generalize to other applications later. Their experiments compare to what they consider to be the state-of-the-art approaches to the problem. \n\nI did find Figure S2A helpful to quickly understand some of the simulated data approach, maybe it should be in the paper and not supplementary. \n\nI argue for accepting this paper because the experiments, while focused solely in one discipline, are thorough. I would have liked clearer description of other areas that might benefit in the last sentence. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BygdP4u6FH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\"Towards a Deep Network Architecture for Structured Smoothness\" proposes a new layer, dubbed FGL, specifically focused on structured smoothness.\nThe paper clearly discusses the proposed layer, introduces the necessary formulation, and demonstrates the effectiveness of this layer in a suite of practical applications in fMRI analysis. The writing is clear, the method is well formulated, and performance compared to relevant and recent benchmarks is strong. Thus most of the questions remaining are about the finer details of the proposed layer and method.\n\nWhat is the sensitivity of FGL to non-optimal group proposals for A? Given the authors reference Aydore et. al., do randomized (or perhaps meta-optimized) groupings perform poorly? Are there any potential ways to jointly-or-iteratively learn the groupings? Is Ward clustering the primary practical method for getting the groupings?\n\nCan the authors discuss a bit the relationship and potential tradeoffs between \"early\" segmentation architectures (such as FGL), and those often favored in semantic segmentation (\"late\" segmentation)? Is there a particular reason why \"late\" segmentation is particularly a poor choice in brain imaging?\n\nMy primary concerns come in relation to other methods for structured smoothness such as CRF/MRF, and the trade-offs and drawbacks of imperfect or downright wrong A, and alternative methods for practically finding good A matrices (potentially with an eye to applications outside fMRI). Addressing some or all of these in a section or two of the text body would potentially raise my score.\n\nSome small errors:\nOptimg -> Opting Introduction\nGuassian -> Gaussian section 3.2\n\nPotential useful references:\nCRF as RNN https://arxiv.org/abs/1502.03240\nEfficient Piecewise Training https://www.cv-foundation.org/openaccess/content_cvpr_2016/html/Lin_Efficient_Piecewise_Training_CVPR_2016_paper.html\nDeep Learning Markov Random Field for Semantic Segmentation https://arxiv.org/abs/1606.07230\nPixel Adaptive Convolutional Neural Networks http://openaccess.thecvf.com/content_CVPR_2019/html/Su_Pixel-Adaptive_Convolutional_Neural_Networks_CVPR_2019_paper.html", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The AC has carefully looked at the paper/comments/discussion in order to arrive at this meta-review.

Looking over the paper, the FGL layer is an interesting idea, but its utility is only evaluated in a limited setting (fMRI data), rather that other types of images/data. Also, the approach seems to work on some of the fMRI datasets, on others the performance is on par with the baselines. 

Overall, the paper is borderline but the AC believes the paper would be a good contribution to the conference.
Decision: Accept (Poster)