Group-Connected Multilayer Perceptron Networks
Authors: Mohammad Kachuee
Sajad Darabi
Shayan Fazeli
Majid Sarrafzadeh
Venue:  (2021)

Abstract: Despite the success of deep learning in domains such as image, voice, and graphs, there has been little progress in deep representation learning for domains without a known structure between features. For instance, a tabular dataset of different demographic and clinical factors where the feature interactions are not given as a prior. In this paper, we propose Group-Connected Multilayer Perceptron (GMLP) networks to enable deep representation learning in these domains. GMLP is based on the idea of learning expressive feature combinations (groups) and exploiting them to reduce the network complexity by defining local group-wise operations. During the training phase, GMLP learns a sparse feature grouping matrix using temperature annealing softmax with an added entropy loss term to encourage the sparsity. Furthermore, an architecture is suggested which resembles binary trees, where group-wise operations are followed by pooling operations to combine information; reducing the number of groups as the network grows in depth. To evaluate the proposed method, we conducted experiments on different real-world datasets covering various application areas. Additionally, we provide visualizations on MNIST and synthesized data. According to the results, GMLP is able to successfully learn and exploit expressive feature combinations and achieve state-of-the-art classification performance on different datasets.

Under review as a conference paper at ICLR 2021
GROUP-CONNECTED MULTILYYER PERCEPTRON NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Despite the success of deep learning domains such as image, voice, and graphs, there has been little progress in deep representation learning for domains without a known structure between features. For instance, a tabular dataset of different demographic and clinical factors where the feature interactions are not given as a prior. In this paper, we propose Group-CConected Multilayer Perceptron (GMLP) networks to enable deep representation learning in these domains. GMLP is based on the idea of learning expressive feature combinations (groups) and exploiting them to reduce the network complexity by defining local group-wise operations. During the training phase, GMLP learns a sparse feature grouping matrix using temperature annealing softmax with an added entropy loss term encourage the sparsity. Furthermore, an architecture is suggested which resembles binary trees, where group-wise operations are followed by pooling operations combine information; reducing the number of groups as the network grows in depth. To evaluate the proposed method, we conducted experiments on different real-world datasets covering various application areas. Additionally, we provide visualizations on MNIST and synthesized data. According to the results, G.MPP is able to successfully learn and exploit expressive feature combinations and achieve state-of- of-the-art art classification performance on different datasets.
INTRODUCTION
Deep neural networks have been quite successful across various machine learning tasks. However, this advancement has been mostly limited to certain domains. For example in image and voice data, one can leverage domain properties such as location invariance, scale invariance, coherence, etc. via using convolutional layers (Goodfellow al., 2016). Alternatively. for graph data, graph convolutional networks were suggested to leverage adjacency patterns present in datasets structured as graph (Kipf & Welling 2016 Xu et al.) 2019).
However, there has been little progress in learning deep representations for datasets that do not follow a particular known structure in the feature domain. Take for instance the case of a simple tabular dataset for disease diagnosis. Such a dataset may consist of features from different categories such as demographics (e.g., age, gender, income, etc.), examinations (e.g.. blood pressure, lab results, etc.), and other clinical conditions. In this scenario, the lack of any known structure between features to be used as a prior would lead to the use of cully-connected multilayer perceptron network (MLP). Nonetheless, it has been known in the literature that MLP architectures, due their huge complexity, do not usually admit efficient training and generalization for networks of more than a few layers.
In this paper, propose Group-Connecee behind GMLP is to learn and leverage expressive feature subsets, henceforth referred to as feature groups. A feature group is defined as a subset of features that provides a meaningful representation or high-level concept that would help the downstream task For instance, in the disease diagnosis example, the combination of certain blood factor and age might be the indicator of higher level clinical condition which would help the final classification task. Furthermore, GMLP leverages feature groups limiting network connections to local group-wise connections and builds feature hierarchy via merging groups as the network grows in depth. GMLP can be seen as an architecture that learns expressive feature combinations and leverages them via group-wine operations.
Multilayer Perceptron (GMLP) networks. The main idea
In this paper, the expression "group" is not related to the group in mathematical sense, and it only represents subset of features.

---

Under review as a conference paper at ICLR 2021
The main contributions of this paper are follows: (t) proposing method for end-to-end learning of expressive feature combinations, (it) suggesting a network architecture to utilize feature groups and local connections to build deep representations, (iλi) conducting extensive experiments demonstrating the effectiveness of GMLP as well as visualizations and ablation studies for better understanding of the suggested architecture.
We evaluated the proposed method on five different real-world datasets in various application domains and demonstrated the effectiveness of GMLP compared state-of-the-art methods in the literature. Furthermore, we conducted ablation studies and comparisons to study different architectural and training factors as well as visualizations on MNIST and synthesized data. To help to reproduce the results and encouraging future studies on group-connected architectures, we made the source code related to this paper available online Additional details and experimental results are provided as appendices to this paper.
2 RELATED WORK
Fully-conmected MLPs are the most widely-used neural models for datasets in which no prior assumption is made the relationship between features. However, due to the huge complexity of fully-connected layers, ML.ss are prone to overfitting resulting in shallow architectures limited to few layers in depth (Goodfellow et al., 2016). Various techniques have been suggested to improve training these models which include regularization techniques such as L-1/L-2 regularization, dropout, etc. and normalization techniques such as layer normalization, weigh normalization, batch normalization. etc. (Sovvastvaavv et al. 2014 Ba et al.) 2016 Salimans & Kingma 2016; [offe & Szegedy 2013). For instance, self neff-normalizing neural networks (SNNs) have been recently suggested as state of the art normalization methods that prevent vanishing or exploding gradients which help training forward networks with higher depths (Klambauer et al., 2017).
From the architectural perspective, there has been great attention toward networks consisting of sparse connections between layers rather than having dense fully-connected layers (Dey et al. 2018). Sparse connected neural networks are usually trained based on either a sparse prior structure over the network architecture (Richter Wattenhofer 2018) or based on pruning a fully-connected network to a sparse network Yun al. [2119] Lartaglione al. 2018 Mocanu al. 2018). However, it should be noted that the main objective of most sparse neural network literature has been focused on improving the memory and compute requirements while maintaining competitive accuracies compared to MIPP.
As parallel line of research, the idea of using expressive feature combinations or groups has been suggested prior over the feature domain. Perhaps, the most successful and widespread use of this idea is in creating random forest models in which different trees are trained based on different feature subsets in order to deal with nigh-dimmessonal and high-variance data (Breiman 200T). More recently, feature grouping is suggested by Aydore et al. 20199) as a statistical regularization technique to learn from datasets of large feature size and a small number of training samples. They do the forward network computation by projecting input features using samples taken from a bank of feature grouping matrices, reducing the input layer complexity and regularizing the model. In another recent study, Ke et al. (2018) used expressive feature combinations to learn from tabular datasets using recursive encoder with a shared embedding network. They suggest a recursive architecture in which more important feature groups have a more direct impact on the final prediction.
While promising results have been reported using these methods, feature grouping has been mostly considered as preprocessing step. For instance, Aydore al. (2019) uses the recursive nearest agglomeration (ReNA) Eloyos-Idrobe et al., (2018) clustering to determine feature groups prior to the analysis. A Alternatively, Ke et al. (2018) defined feature groups based a pre-trained gradient boosting decision tree (GBDT) (Friedman 200T). Feature grouping as a preprocessing step not only increases the complexity and raises practical considerations, but also limits the optimality of the selected features in subsequent analysis. In this study, we propose an end-to-end solution to learn expressive feature groups. Moreover, we introduce network architecture to exploit interrelations within the feature groups to reduce the network complexity and to train deeper representations.
We plan to include link to the source code and GitHub page related to this paper in the camera-ready version.
2

---

Under review as a conference paper at ICLR 2021
PROPOSED METHOD 3
p000
Group FC
Bacde Baad--Nom
Group Pool
0000
Group
Pood
GropuE
Bach-Nan Num
0
Suftex
Groupp Select
Braa Non
0000
Pool
10000
ITPM Ti-40
Figure 1: The GMLP network architecture.
3.1 ARCHITECTURE OVERVIEW
In this paper, we propose GMLP which intuitively can be broken down three stages: (i) selecting expressive feature groups, (ii) learning dynamics within each group individually, and (fii) merging information between groups the network grows depth (see Figure [. In this architecture, expressive groups are jointly selected during the training phase. Furthermore, GMLP is leveraging feature groups and using local group-wise weight layers to significantly reduce the number of parameters. While the suggested idea can be materialized as different architectures, in the current study, we suggest organization of the network as architectures resembling a binary tree spanning from leaves (i.e., features) to a certain abstraction depth closer the roo As the network grows deeper, after each local group-wise weight layer, half of the groups are merged using pooling operations, effectively reducing the width of the network while increasing the receptive field. At the last layer, all features within all groups are concatenated into a dense feature vector fed to the output layer.
3.2 NOTATION
We consider the generic problem of supervised classification based on dataset of feature and target pairs, D: (æI:N, V1:N), where æ; E Rd. y; E (1... C), and N is the number of dataset samples. Furthermore, we define group size, m, as the number of neurons or elements within each group, and group count, kt as the number of selected groups which are essentially subsets of input features. Also, used to refer to the total depth network. We use z! E R to refer to activation values of group in layer I, In this paper, we define all vectors column vectors.
3.3 NETWORK LAYERS
In this section, we present the formal definition of different GMLP network layers. The very first layer of the network, Group-Seledd is responsible for organizing features into k groups of size m each. A routing matrix, V, is used for connecting each neuron within each group to exactly one feature in the feature set:
(1) where v {0, 1. xd a sparse matrix determining features that are present in each group. Note that this formulation allows each feature to contribute to multiple groups as it allows features to be
1
V x
Please note that, in this paper, tree structures are considered to grow from leaves to the root In other words. in this context, limiting the depth is synonymous with considering the tree portion spanning from certain depth leave nodes.
3

---

Under review as a conference paper at ICLR 2021
selected multiple times in different groups. As are interested in jointly learning Y during the training phase, we use the following continuous relaxation:
exp(thu,/7))
Vis E exp(xxxp0
In this equation, d is real-valued matrix reparameterizing the routing matrix through softmax operation with temperature, т. The lower the temperature, the more ( converges to the desired discrete and sparse binary routing matrix. Note that, in the continuous relaxation, the matrix vi can be optimized via the backpropagation of classification loss gradients. In the next section, we provide further detail on temperature annealing schedules as well as other techniques enhance the V approximation.
Based on selected groups, we suggest local dully-connected weight layers for each group: Group-FC. The goal of Group-FC is to extract higher-level representations using the selected expressive feature subsets. This operation usually followed by non-linearity functions (e.g.. ReI.U), normalization operations (e.g, Batch Norm), and dropout. Formally, G Group-FC can be defined as:
S(W b(),
where W m and by E R the weight matrix and bias vector, applied group at layer i. Here, f represents other subsequent operations such as non-lineame oon-lineartty normalization, dropout.
Lastly, Group-Pool is defined operation which merges representations of two groups into single group, reducing network width by half while increasing the effective receptive field:
I+1 pool 2/+x//2+1);
(4)
where 2/ and 2/ 2/+2/22 are the ith group from the first and second halves, respectively, and pool is pooling function from B2am to R In this study, we explore different variants of pooling functions such as max pooling, average pooling, or using linear weight layers as transformations from 822m to Rmm Please note that while we use similar terminology pooling in convolutional networks, the pooling operation explained here is not applied bocation-wise, but instead it is applied feature-wise, between different groups pairs.
The values of m and & are closely related to the number and order of feature interactions for a certain task. Using proper m and A values enables to reduce the parameter space while maintaining the model complexity required to solve the task. However, finding the ideal m and & directly from a given dataset is very challenging problem. In this work, we treat and l as hyperparameters be found by hyperparameter search.
3.4 TRAINING
We define the objective function to used for end-to-end training of weights as well as the routing matrix as:
L -= ΣE N LI yic log(Fe(z:)) + (H +
(5)
WED
In this objective function, the first term is the standard cross-entropy classification loss where F denotes the GMLP network function with parameters 0, and N is the numher of training samples used. The second term is an entropy loss over the distribution of the routing matrix that is weighted by the hyperparameter A:
t=km
exp(1his) f'zd f= exp((br))
exp(1/i.))
H(b)
log(
y=d exp(1i25))
(())) is minimizing the entropy corresponding to the distribution of regardless of the temperature used for V approximation. Accordingly, À can be viewed as hyperparameter and an additional method for encouraging sparse V matrices. The last term in (5) is an L-2 regularization term with the hyperparameter control the magnitude of parameters in layer weights and in th. Note that
4

---

Under review as a conference paper at ICLR 2021
without the L-2 regularization term, s elements may keep increasing during the optimization loop. since d only appears in normalized form in the objective function of (5)
We use Adam (Kingma & Bal 2014) optimization algorithm starting from the default 0.001 learning rate and reducing the learning rate by a factor of as the validation accuracy stops improving. Regarding the temperature annealing. during the training, the temperature is exponentially decayed from 1.0 to 0.01. In order to initialize the Group-FC weights, we used Xavier initialization (Glorot & Bengio 2010) with for both fan-in and fan-out values. Similarly, the i matrix is initialized by setting the fan-in equal to d and fan-out to kvn.
Further detail on architectures and hyperparameters used for each specific experiment as well as details on the software implementation are provided as appendices to this paper.
3.5 ANALYSIS
The computational complexity of GMLP at the prediction time can be written as (for simplicity, ignoring bias and pooling terms):
km kan? km2
km
km 21-1 + C km
21-1
In this series, the first term, km, is the work required to organize features to groups. The subsequent terms, except the last term, are representing the computational cost of local fully-connected operations at each layer. The last term is the complexity of the output layer transformation from the concatenated features to the number of classes. Therefore, the computational complexity of GMLP at the prediction time can be written as Okm?² 2 + Clem In comparison, the computational complexity of an MLP
with similar network width would be:
22m²
k2m² k2m²² + + 21.-1 + C
km
kmd + k22m +
2
21-11
where the first term is the work required for the first network layer from d to km neurons, the second term is corresponding to hidden layer of size km, and so forth. The last term is the complexity of the output layer similar to the case of GMLP. The overall work required from this equation is of O + mm2
Ckm This is substantially higher than GMLP, for typical k d, and C' values.
Additionally, the density of the Group FC layer connections can be calculated as: Pmm L. which very small for reasonably large number of l values used our experiments. Also, assuming pooling operations in every other layer, the receptive field size or the maximum number of features impacting a neuron at layer I can be written as 2 m. For instance, a neuron in the first layer of the network is only connected to m features, and a neuron in the second layer is connected to two groups or 2m features and so forth.
kmm
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
The proposed method is evaluated on five different real- world datasets, covering various domains and applications: permutation invariant CIFAR-I0 (Krizhevsky et al. 2009). human activity recognition (HAPT) (Anguita al. 2013), toxicity prediction (Tox21) (Huang et al. 2016), and UCI Landsat (Dual & Graff 2017), and MIT-BIH arrhythmia classification (Moody & Mark (001). Additionally, we use three world tabular datasets in health domain: diabetes, hypertension. cholesterol classification tasks (Kachuee et al. 2019). We use MNIST (LeCun et al. 2010) and synthesized dataset to provide further insight into the operation of GMLP (see Appendix). Table presents summary of datasets used in this study. Regarding the CIFAR-10 10 dataset, we permute the image pixels to discard pixel coordinates in our experiments. Note that the permutation is not changing across samples, it is merely a fixed random ordering used to remove pixel coordinates for each experiment. For all datasets, basic statistical normalization with = and o = is used to normalize features as a preprocessing step. The only exception is CIFAR-10 10 for which we used the standard channel-wise normalization and standard data augmentation (i.e., random crops and random horizontal flips). The standard test and train data splits were used dictated by dataset publishers. In cases that the separated sets not.
5

---

Under review as a conference paper at ICLR 2021
Table 1: Summary of datasets used in our experiments.
# Train Samples
# Test Samples
Features
Classes
Domain
CIEAR-I
Krizhevsky
50,000
10,000
3,072
10
Image Clussification
HAPT Anguila all
2013
6,002
2,451
561
5
Activity Recognition
Tox21
Huang al.
2016
8,441
610
1,644
2
Drug Discovery
Diabetes This Work)
47,125
11,782
116
2
Discase Diagnosis
Hypertension" (This Work)
49,819
12,455
121
2
Discase Diagnosis
Cholesterol
54,360
13,591
120
Discase Diagnosis
Landsat Dua Graff 2017
4,435
2,000
36
6
Satellite Imaging
MIT-BII Moody & Mark
2001
87,554
21,892
187
5
ECG Classification
MNIST
LeCun et al. 2010
60,000
10,000
784
10
Digit Classification
Synthesized (Appendix D)
5,120
1,280
2
See Appendix I)
PPermuted version, i.c. pixel coordinates are ignored.
A Aryl hydrocarhon Receptor (AhR) activity prediction task adapted from Mayr ct al.
2016
"Data processing pipeline adopted from Kachuee et al. (2019)
"We use preprocessed data from http://kaggle.com// wtw.ee/aaoaaaaaaaaaleeeeeeee
provided, test and train subsets are created by randomly splitting samples to 20% for test and the rest for training/validation.
We compare the performance of the proposed method with recent related work including Self- Normalizing Neural Networks
(SNN) (Klambauer et al., (017), Sparse Evolutionary Training Feature Grouping as a Stochastic Regularizer (in this paper, de-
(SET) Mocanu et al. 2018
noted as FGR) (Aydore et al. 2019 as well as the basic dropout regularized and batch normalized MLPs. Additionally, as a non non-neural baseline, we make comparisons with random forest classifiers (RFC) consisting of 1000 trees trained using the gini criterion (Liaw et al. 2002). In order to ensure a fair comparison, we adapted source codes provided by other work to be compatible with our data loader and preprocessing modules.
Furthermore, for each method, we conducted an extensive hyperparameter search using Mi- crosoft Neural Network Intelligence (NNT) voolki(Jand the Tree-structured Parzen Estimator (TPE) tuner (Bergstra et al.) 2011) covering different architectural and learning hyperparameters for each case. More detail on hyperparameter search spaces and specific architectures used in this paper is We run each case using the best hyperparameter configuration eight times and report mean and standard deviation values.
provided in the appendices.
4.2
RESULTS
Table presents a comparison between the proposed method (GMLP) and 5 other baselines: MLP, SNN Klambauer et al. 2017). SET (Mocanu et al. 2018), FGR (Aydore et al. 2019). and RFC. From this comparison, GMLP outperforms other work, achieving of-te-o-thhe-ee art classification accuracies. Concerning the CIFAR-10 10 results, to the best of our knowledge, GMLP achieves a new state-of-the-art performance on permutation invariant CIFAR-10 augmented using the standard data augmentation.
Note that Lin et al.) (015) reported 78% accuracy on the permuted CIFAR- 10 using additional non-standard augmentations and about 70% otherwise. We believe that leveraging expressive feature groups enables GMLP to consistently perform better across different datasets.
To compare model complexity and performance we conduct an experiment by changing the number of model parameters and reporting the resulting test accuracies. Here, we reduce the number of parameters by reducing the width of each network; i.e. reducing the number of groups and hidden neurons for GMLP and MLP, respectively. Figure shows accuracy versus the number of parameters for the GMLPP and MLP baseline on CIFAR- 10 and MIT-BIH datasets. Based on this figure, GMLP is able to achieve higher accuracies using significantly less number of parameters. It is consistent
com wwwp///rgaaaallooee/aaaoooauauuu Regularizer
hhtps://gthub..cmm/// http://tthtt.ccom///coootttttnti

---

Under review as a conference paper at ICLR 2021
Table 2: Comparison of top-1 test accuracies for GMLP and other work.
Accuracy (%)
Dataset
GMLP
MLP
SNN
SET
FGE
RFC
CIFAR-10
Krizhevsky
2009
73.76 (40.14)
68.15 (20.56)
66.88
72.71 (m0.20)
45,90
HAPT
Anguita et al. 2013
96.34 (#0.119
95.73 (20.38)
95.47 (=0.00)
71.35 (=0.74)
91.57 (=0.31)
91.51 (=0.10)
Tox2
Huang et al 2016
88.57 (#0.35)
86.90 (=0.35)
86.68 (=0.88)
88.29 (=0.20)
87.17 (=0.38)
88.42 (=0.11)
Diabetes"
Work)
88.37 (50004)
87.80 (=0.00)
87.83 (=0.02)
72.39
87.31 (=0.00)
87.51 (=0.13)
Hypertension" (This Woek)
87.26 (29.04)
86.83 (20.00)
86,93 (=0.01)
82,92 (=11
86,29 (=00
86,24 (=0.00)
Cholesterol (This Work)
83.18 (2:0.00)
82.66 (20.00)
82.62 (20.30)
76.59 (=2:13))
82,10 (20 om
82.48 (20.00
Landsat Dua GratT 2017
91.54 (#0.16)
91.21 (=0.44)
91.37 (#0.21)
91.03 (=0.56)
90.70 (=0.
91.15 (=0.00)
MIT-BIH Moody Mark 2001
98.74 ($0.00)
98.65 (=0.01)
98.56 (=0.02)
98.10 (=0.01)
98.13 (=0
98 .46 (40.0))
Sell--norma Neural Networks (Klambauer et al.
2017
"Sparse Evolutionary Training (Mocanu et al.
2018)
"Feature Grouping as Stochastic Regularizer (Aydore et al. 2019)
PPeccentage of the area under the ROC curve is reported for this dataset.
with the complexity analysis provided Section 3.5 Note that in this comparison, we consider the number of parameters involved at the prediction time.
CIFA8-10
MI1-511
10.2
ACCUPIC 72
$0.5-
00.5
AcCUSSC/
04
GMLP MLP
GHLP
16
MLP
Number Parameters
Parameters
Figure 2: Accuracy versus number of parameters for this work (GMLP) and the MLP baseline: (a)
CIFAR-10 dataset, (b) MIT-BIH dataset. The x-axis is a logarithmic scale.
4.3 ABLATION STUDY
Figure presents an ablation study comparing the performance GMLP on CIFAR-10 dataset for networks trained: (i) using both the temperature annealing and the entropy loss objective, (ii) using only temperature annealing without the entropy loss objective, (iti) using no temperature annealing but using the entropy loss objective, (iv) not using any of the temperature annealing the entropy loss objective. From this figure, excluding both techniques leads to significantly lower performance. However, using any of the two techniques leads relatively similar high accuracies. It consistent with the intuition that the functionality of these techniques is to encourage learning sparse routing matrices, either using softmax temperatures or entropy regularization to achieve this. In this paper, in order to ensure sparse routing matrices, we use both techniques simultaneously in case (t).
Figure 4 shows a comparison between GMLP models trained on CIFAR-10 using different pooling types: (i) linear transformation, (ii) max pooling, and (iit) average pooling. As it can be seen from this comparison, while there are slight differences in the convergence speed of using different pooling types, all of them achieve relatively similar accuracies. In our experiments, decided to use max pooling and average pooling as they provide reasonable results without the need to introduce additional parameters required for the linear pooling method.
7

---

Under review as a conference paper at ICLR 2021
70-
70-
Accurssy (16)
Accursiy (76)
Temp, Annealing, Entropy
WiO Temp. Annealing, W/ Entropy Loss Temp. Anneating, Entropy
Linea Pooling
Pooling
Temp. Annealing, Entropy Loss
Average Pooling
22
2006
1250
1000
2753
46
250
3000
1902
2758
Training Epochs
Training Epochs
Figure 3: Ablation study on the impact
Figure 4: Ablation study demonstrating the impact of different pooling functions.
temperature annealing and entropy loss terms.
Accuracy
m=22
m-16
k-1536
m-12
767
3:3
250
206
159
1000
1256
1249
3759
224
589
224
1006
1259
3500
1759
Training Epochs
Training Epochs
Figure 5: Ablation study the impact of Figure 6: Ablation study the impact of using different group sizes (m). For this experiment, used k=1536.
using different number of groups (k). For this experiment, used m=16.
Figure shows learning curves for training CIFAR-10 models using different group sizes. From this figure, using very small group sizes would cause reduction in the final accuracy. At the other extreme, the improvement achieved using larger values is negligible for m values more than 16. Finally, s/swss comparison between learning curves for using different number of groups. Using very small & values result in significant reduction in performance. However, the rate of performance gains for using more groups is very small for i of more than 1536. Note that the number of model parameters and compute scales linearly with and quadratically with m (see Section 3.5
5 DISCUSSION
Intuitively, training a GMLP model with certain groups can be viewed as prior assumption over the number and order of interactions between the features. is reasonable prior assumption as many natural datasets, a conceptual hierarchy exists where only a limited number of features interact with each other. Additionally, GMLP can be considered as more general neural counterpart of random forests. Both models use subsets of features (i.e., groups) and learn interactions within each group. A major difference between the two methods is that GMLP combines information between different groups using pooling operations, while random forest trains ensemble of independent trees each group. From another perspective, the idea of studying feature groups is related to causal models such Bayesian networks and factor graphs (Darwiche, 2009; Neapolitan et al.] 2004;; Clifford 1990). These methods are often impractical for large-scale problems, because without a prior over the causal graph, they require architecture search of the NP-complete complexity more.
8

---

Under review as a conference paper at ICLR 2021
6 CONCLUSION
In this paper, we proposed GMLP as a solution for deep learning domains where the feature interactions are not known as prior and do not admit the use of convolutional other techniques leveraging domain priors. GMLP jointly learns expressive feature combinations and employs group- wise operations reduce the network complexity. We conducted extensive experiments demonstrating the effectiveness of the proposed idea compared to the state-of-the ff-the-art art methods in the literature.
REFERENCES
Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, and Jorge Luis Reyes-Ortiz A public domain dataset for human activity recognition using smartphones. In Esunn, 2013.
Sergul Aydore, Bertrand Thirion, and Gael Varoquaux. Feature grouping as stochastic regularizer for high-dimensional structured data. In International Conference Machine Learning, pp. 385-394, 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint 160vv:6607006450 2016.
James S Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter optimization. In Advances in neural information processing systems, pp. 2556-2554, 2011.
Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
Peter Clifford. Markov random fields in statistics. Disorder in physical systems: A volume in honour of John M. Hammersley, 19, 1990.
Adnan Darwiche. Modeling and reasoning with Bayesian networks. Cambridge university press, 2009.
Sourya Dey, Kuan-Wen Huang, Peter Beerel, and Keith M Chugg. Characterizing sparse connec- tivity patterns in neural networks. In 2018 Information Theory and Applications Workshop (ITA), pp. 1-9. IEEE, 2018.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL httt::ttt///hhhti ics www.ccccco edd.mddumm..
Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of statistics, pp. 1189-1232, 2001.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http: //ww.ddeelearrnnggbo.. orgg
Andrés Hoyos-Idrobo, Gaël Varoquaux, Jonas Kahn, and Bertrand Thirion. Recursive nearest
agglomeration (rena): fast clustering for approximation of structured signals. IEEE transactions on pattern analysis and machine intelligence, 41(3):669-681, 2018.
Ruili Huang, Menghang Xia, Dac-Trung Nguyen, Tongan Zhao, Srilatha Sakamuru, Jinghua Zhao, Sampada Shahane, Anna Rossoshek, and Anton Simeonov. Tox2 lchallenge build predictive models of nuclear receptor and stress response pathways as mediated by exposure to environmental chemicals and drugs. Frontiers Environmental Science, 3:85, 2016.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03166 2015.
Mohammad Kachuee, Kimmo Karkkainen, Orpaz Goldstein, Davina Zamanzadeh, and Majid Sar- rafzadeh. Nutrition and health data for sensitive learning. arXiv preprint arXiv: 07 102, 2019.

---

Under review as a conference paper at ICLR 2021
Guolin Ke, Jia Zhang, Zhenhui Xu, Jiang Bian, and Tie- Yan Liu. Tabnn: A universal neural network solution for tabular data. 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXi 6980, 2014.
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint
2016.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. Advances in neural information processing systems, pp. 971-980, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. AT&T Labs [Online]. Available: http:://yann. lecun. com/exdb/unist, 2:18, 2010.
Andy Liaw, Matthew Wiener, et al. Classification and regression by randomforest. R news, 2(3): 18-22, 2002.
Zhouhan Lin, Roland Memisevic, and Kishore Konda. How far can we go without convolution: Improving fully-connected networks. arXiv preprint arXiv: 1.02580, 2015.
Andreas Mayr, Günter Klambauer, Thomas Unterthiner. and Sepp Hochreiter. Deeptox: toxicity prediction using deep learning, Frontiers in Environmental Science, 3:80, 2016.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 91):23883, 2018.
George B Moody and Roger G Mark. The impact of the mit-bih arrhythmia database. 1EEE Engineering in Medicine and Biology Magazine, 20(3):45-50, 2001.
Richard E Neapolitan et al. Learning bayesian networks, volume 38. Pearson Prentice Hall Upper Saddle River, NJ, 2004.
Oliver Richter and Roger Wattenhofer: Treeconnect: A sparse alternative to fully connected layers. In 2018 IEEE 30th International Conference on Tools with Artificial Intelligence (ICTAI), pp. 924-931. IEEE, 2018.
Tim Salimans and Durk Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, pp. 901-909, 2016.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, ((11:1999-1193) 2014.
Enzo Tartaglione, Skjalg Lepsøy, Attilio Fiandrotti, and Gianluca Francini. Learning sparse neural networks via sensitivity-drive regularization. In Advances in Neural Information Processing Systems, pp. 3878-3888, 2018.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful graph neural networks? In International Conference on Learning Representations, 2019, URL https
/ooeenrvviw net. pott:/oorrumlld-ryysss.........
Jihun Yun, Peng Zheng, Eunho Yang, Aurelie Lozano, and Aleksandr Aravkin. Trimming the I regularizer: Statistical analysis, optimization, and applications to deep learning. In International Conference on Machine Learning, pp. 7242-7251, 2019.
10

---

Under review as a conference paper at ICLR 2021
A HYPERPARAMETER SEARCH SPACE
Tables present the hyperparameter search space considered for experiments on GMLP, MLP, SNN, and FGR, respectively. For the GMLP search space, the number of groups is adjusted based on the number of features and samples in each specific task. Also, the number of layers is adjusted be compatible with the number of groups being used. Regarding the FGR experiments, due to scalability issues of the published source provided by the original authors, we were only able to train networks with at most two hidden layers. For SET, as their architecture is evolutionary i.e., prunes certain weights and adds new ones, we only explored using a different number of hidden neurons in the range of 500 to 4000. Regarding the RFC baseline, we found that the Gini criterion tends to work very similar slightly better than the entropy measure. Also, for experiments, using ensemble of 1000 trees was more than sufficient to train stable and powerful RFC models. However, we found that the maximum depth of trees is important hyperparameter to be adjusted for each experiment.
Regarding the number epochs, we used 2000 epochs for CIFAR- 10, 1000 epochs for HAPT, and 300 epochs for the rest of the datasets. The only exception is the SNN experiments where we had to reduce the learning rate to increase the stability of the training resulting in more epochs required to converge.
Table 3: Hyperparameter search space used for GMLP experiments.
Hyperparameter
Considered Values
Number of hidden layers
{1, 2, 8)
Number of groups
16 - 4096
Size of groups
4 - 32
Lambda
[10-2 104]
Alpha
12 110-12,10-1 10 [0
Dropout rate
Table 4: Hyperparameter search space used for MLP and SNN experiments.
Hyperparameter
Considered Values
Number of hidden layers
(1,2,..., 6}
Size of hidden layers
[0.05 [eeauuees] [20 X (/feature)
Alpha
[10 10
Dropout rate
[0, 1]
Table 5: Hyperparameter search space used for FGR experiments.
Hyperparameter
Considered Values
Number of hidden layers
[1,2]
Size of hidden layers
fefeuuures [50 X featuress
Number of groups
2 - Rlfeatrres
Alpha
[10 12 10
11

---

Under review as a conference paper at ICLR 2021
B ARCHITECTURES
Table 7 18 10 show the selected architectures for GMLP, MLP, SNN, SET, and FGR, respectively. We used the following notation to indicate different layer types and parameters: GSel-k-m repre- sents Gr Group-Select layer selecting k groups of m features each. GFC indicates Group-FC lay- ers, and FC x represents fully-connected layer with x hidden neurons. GPool layer of type (max, mean, linear, etc.). Concat is concatenation of groups used prior to the output layer in GMLP architectures. SC-x refers to SET sparse evolutionary layer of size x.
Table MLP architectures used our experiments.
Dataset
Architecture
FC-3072, ReLU, BNorm, F0-2764, ReLU, BNorn,
CIFAR-10
EC-2488, ReLU, BNorm, FC-10, Softmax
HAРT
FC-106, ReLU, BNorm, FC-21, ReLU, BNorm, FC-5, SofLmax
Tox21
FC-4899, ReLU, BNorm, FC-4899, ReLU, BNorm, FC-2, Softmax
Diabetes
EC-820, ReLU, BNorm, FC-820, ReLU, BNorm, FC-2, Softmax
FC-470, ReLU, BNorm, FC-470, ReLU, BNorm,
Hypertension
EC-470, ReLU, BNorm, FC-2, Softmax
FC-480, ReLU, BNorm, FC-480, ReLU, BNorm, FC-480, ReLU, BNorm,
Cholesterol
FC-480, ReLU, BNorm, FC-2, Softmax
FC-68, ReIUU BNorm, FC-68, 68, ReI.U, BNorm, FC-68, ReLU, BNorm,
Landsat
FC-68, ReLU, BNorm, FC-6, 8oftmax
FC-1737, ReLU, BNorm, F0-1737, ReLU, BNorm, EC-1737, ReLU, BNorm, F0-5, Softmax
MIT-BIH
Table 8: SNN architectures used our experiments.
Dataset
Architecture
FC-3901, SetU, BNorm, FC-3901, SetV, BNorm, FC-3901, SeLU, BNorm, FC-10, Softmax
CIFAR-10
FC-510, ReLU, BNorm, FC-510, SeLU, ENorm,
НАРT
FC-510, SeLU, FC-5, Softmax
FC-3666, ReLU, BNorm, FC-3666, SeLU, ENorm,
Tox21
FC-3666, SeLU, FC-2, Softmax
Diabetes
FC-160, SeLU, BNorm, FC-160, SeLU, BNorm, FC-2, Softmax
Hypertension
FC-213, SeLU, BNorm, FC-213, SeLU, BNorm, FC-2, Softmax
FC-122, ReLU, BNorm, FC-122, SeLU, BNorm,
Cholesterol
FC-122, SeLU, FC-2, SofLmax
Landsat
FC-816, SeLU, BNorm, FC-811 816, SeLU, BNorm, FC-6, Softmax
EC-1140, SeLU, BNorm, FC-1140, SeLU, BNorm,
MIT-BIH
FC-1140, SeLU, BNorm, 5, Softmax
12

---

Under review as a conference paper at ICLR 2021
Table 9: SET architectures used in our experiments.
Dataset
Architecture
CIFAR-10
SC-4000, SReLU, SC-1000, SReLU, SC-4000, SReLU, FC-10, Softmax
HAРT
SC-500, SReLU, SC-500, SReLU, SC-500, SReLU, FC-5, Sof
Tox21
SC-1000, SReLU, SC-1000, sReLU, 8C-1003, SReLU, FC-2, Softmax
Diabetes
SC-1000, SReLU, SC-1000, SReLU, sC-1000, SReLU, FO-2, Softmax
Hypertension
SC-1000, SReLU, SC-1000, SReLU, SC-1000, SReLU, FC-2, So[Lmax
Cholesterol
SC-1000, SReLU, SC-1000, SReLU, SC-1000, SRetu, FC-2, Softmax
Landsat
SC-1000, SReLU, SC-1000, SReLU, SC-1000, SReLU, FC-6, Softmax
MIT-BIH
SC 1000, SReLU, SC-1000, SReLU, 1000, SReLU, FC 5, Softmax
Table 10: FGR architectures used in our experiments.
Dataset
Architecture
CIFAR-10
Group-256, FC-3072, RelU, FC-10, Softmax
HAPT
Group-104, FC-12173, ReLU, FC-5, Softmax
Tox21
Group-3468, FC-3468, ReLU, FC-2, Softmax
Diabetes
Group-10C, FC-230, ReLU, FC-2, Softmax
Hypertension
Group-100, FC-210, ReLU, FC-2, Softmax
Cholesterol
Group-100, FC-250, ReLU, FC-2, Softmax
Landsat
Group-330 32, FC 1577, ReLU, FC 6, Softmax
MIT-BIH
Group- 160, FC 3444, ReLU, FC 5, Softmax
Table 6: GMLP architecuurs used in our experiments.
Dataset
Architecture
osel-1526-16, GFC, ReLU, BNorm, GPool-max, GFC, ReLU, BNorn, GPocl-max, GFC, ReLU, BNorm, uPool-max, UFC, Relú, BNorn,
CIFAR-10
Concat, FC-10, Softmax
c3e1-288-12, GFC, ReIU, SNorm, cPool-mean, GFC, ReLU, BNorn,
НAPT
Consat, PC-5,
GSe1-320-28, GFC, Relli, 3Norm, @Focl-meen, GEC, ReLU, BNorn,
Tox21
GPool GFC, Retif, RNore, GPoal-mean,
Reti) RNorm,
@Pocl-meen, G2C, RelU, BNorm, Cencat, 20-2, Scftmax
69e1-352-4, GEC, RelU, BNorn, @Pool-mean, GFC, ReLU, BNcrn,
Diabetes
GPool-mean, GFC, ReMlr, RNorm, (Poal-mean, GPC, ReTO) RNorn,
GPacl-mean, Concat, BC-2, Softmax
GBe1-448-8, GEC, RelU, BNorm, GPool-mean,
ReLU, DNcra,
Hypertension
CPool-mean, Concat, FC-2, Softmax
CS61-384-24, CFC, ReId, sMorm, CFacl-mean, GFC, ReLü,
Cholesterol
@Pocl-meant GDC, RelU, DNorm, GPoul-meen, mean, GPC, ReLU, DMera,
GPool-nesn, Concel, FC-2, SofLnax
cSe1-88-16, GFC, Ralt, BNorm, (Fool-mean, CFC, ReLU, BNorn,
Landsat
GPacl-mean, GFC, RalU, BNorm, (Fool-mean, Cancat, FC-6, Softmox
Gse1-240-24, GFC, RelU, 3Norm, GFoel-mean, SEC, ReLU, Blorn,
MIT-BIH
GPuol-means 6PC, ReIU, BNorm, GPoul mean, Concal, PC-5, Suflmax
G8e1-64-16, GFC, RelU, BNorm, GPool-ment mean,
RetU, BNozm, GPocl-mean, GEC, RelU, Blorm, @Fool-mean, GFC, ReLU, BNera,
MNNST
Concat, PC-10, Softnax
Synthesized
08+1-4-2, GPC, Retü, BMorm, Concat, FC-2, Softmax
13

---

Under review as a conference paper at ICLR 2021
C SOFTWARE IMPLEMENTATION
Table 11 presents the list of software dependencies and versions used our implementation. To produce results related to this paper, we used workstation with NVIDIA GeForce RTX-2080Ti GPUs, a 12 core Intel Core i9-7920X processor, and 128 GB memory. Each experiment took between about 30 minutes 72 hours, based on the task and method being tested.
Table 11: Software dependencies.
Dependency
Version
python
3.7.1
pytorch
1.1.0
torchvision
0.2.1
cuda100
1.0
ipython
6.5.0
jupyter
1.0.0
numpy
1.15.4
nni
0.9.1.1
pandas
0.23.4
scikit-learn
0.19.2
scipy
1.1.0
pomegranate
0.11.1
tqdm
4.32.1
matplotlib
3.0.1
14

---

Under review as a conference paper at ICLR 2021
D EXPERIMENTS ON MNIST AND SYNTHESIZED DATA
MNIST dataset is used to visually inspect the performance of the Group-Select layer. Figure shows heat-map of how frequently each pixel is selected across all feature groups for: (a) original MNIST samples, (b) MNIST samples where the lower-half is replaced by Gaussian noise. From Figure 7al it can be that most groups selecting pixels within the center of the frame, effectively discarding margin pixels. This is consistent with other work which show the importance of different locations for MNIST i Apart from this, in Figure 7b version of the MNIST dataset is used in which half of the frame does not provide any useful information for the downstream classification task. From this figure, GMLP selecting any features be used from the lower region.
(a)
(b)
Figure 7: MNIST visualization of pixels selected by the Group-Select layer: (a) using complete images as input, (b) using images that the lower half is replaced by Gaussian noise. this figure, warmer colors represent pixels being being present in more groups.
Group #1
Group
Group
NG -
C 099
PU -001
PÚ - -u== 099
Pl EG -)
Group #4
= MVv.G D.v 0
Features
Figure 8: The Bayesian network and conditionals used to generate the synthesized dataset of binary features A-F and target J.
Figure 9: Visualization of the selected features within each group. Every two consecutive rows show features selected for a certain group.
In order show the effectiveness GMLP, synthesized dataset which has intrinsic and known expressive feature groups. Specifically, we used simple Bayesian network as depicted in Figure g This network consists of six binary features, A interacting with each other as specified by the graph edges, which determine the distribution of the target node, J. The graph and conditionals are designed such that each 99% probability. The target node, J, is essentially one with high probability if at least two of the second level nodes are one. We synthesized dataset by sampling 6,400 samples from the network (1,280 samples for test and the rest of (raining/cvaluation)) On this dataset, trained very simple GMLP consisting of four groups of size two, one group-wise fully-connected layer, and an output layer. Figure shows shows the features selected for each group after the training phase (i.e., the V matrix). From this figure, the Group Se lect layer successfully learns to detect the feature pairs that are interacting, enabling the Group FC layers decode the non-linear XOR relations.
nodes in the second level take the XOR value of their parents with a
Kachuee, M., Darabi, S., Moatamed, B., & Sarrafzadeh, M. (2018). Dynamic feature acquisition using denoising autoencoders. IEEE transactions neural networks and learning systems, 30(8), 2252-2262.
15

---

Under review as a conference paper at ICLR 2021
To investigate the impact of GMLP architectures that does match the data generating distribution, we conducted experiments by changing the group size and number groups. Based on the results presented in Figure 10 any network with and & values more than is able to fit the distribution. However, this example, using smaller m and values results in a significant degradation due to the incapability of the networks capture the dataset dynamics. This result demonstrates the importance of the m and k hyperparameters.
1
2
3
4
1
900
2
-80
70
4.
60
8
Figure 10: Visualization of prediction accuracies for the synthesized dataset using different group size (m) and number of groups (k).
16

---

Under review as a conference paper at ICLR 2021
E ADDITIONAL ABLATION STUDIES
Figure presents an ablation study comparing the performance GMLP on the Diabetes dataset for networks trained: (i) using both the temperature annealing and the entropy loss objective, (ti) using only temperature annealing without the entropy loss objective, (iii) using no temperature annealing but using the entropy loss objective, (iv) not using any of the temperature annealing or the entropy loss objective. From this figure, excluding both techniques leads to significantly lower performance. Also note that, compared to the CIFAR-10 ablation experiments (see Figure only using the entropy loss term is not sufficient for achieving best results. We found that using both techniques consistently achieves better or similar results.
Figure 12 shows comparison between GMLP models trained on the Diabetes dataset using different pooling types: (i) linear transformation, (ii) max pooling, and (íii) average pooling. As it can be seen from this comparison, while there are slight differences in the convergence speed of using different pooling types, all of them achieve relatively similar accuracies.
131
8/3-
878
67-
633
W Annealing Entropy
618
Wi Temp. Arncaling, W/O Entropy Less WIO Temp Anrealigg WD Entropy Less
111
Max Pooling
Average Pooling
Training Epochs
Training Epochs
Figure 11: Ablation study on the impact of Figure 12: Ablation study demonstrating the temperature annealing and entropy loss terms. impact of different pooling functions.
Figure shows learning curves for training the Diabetes models using different group sizes. From this figure, using very small group sizes decreases the final accuracy. At the other extreme, the improvement achieved using larger values negligible for m values more than 3. the other hand, using very large values degrades the results due to overfitting. Figure [4 shows comparison between learning curves for using a different number of groups. Using very small values result in significant reduction in performance. However, the rate of performance gains for using more groups very small for & of more than 176.
k-17
2-1
32
Training Epochs
Traiting Epochs
Figure 13: Ablation study on the impact of using different group sizes (m). For this experiment, we used k:252.
Figure 14: Ablation study on the impact of using different number of groups (k). For this experiment, we used m=4.
17

---

Under review as a conference paper at ICLR 2021
F EXPERIMENTS USING ALTERNATIVE TREE STRUCTURES
We conducted experiments comparing the suggested binary tree (B-tree) GMLP architecture with other alternatives using tree structures with different branching factors (Iree ways). As the GMLP baseline for the Diabetes dataset has 352 groups and layers, increasing the branching factor necessitates either increasing the number groups or reducing the number of layers build the tree. To investigate this, we conducted to experiments. First, in Figure [5] we fixed the number of groups and experimented
b-tree, 4-way tree, and 8-way tree architectures. Second, Figure 16 we fixed the number of layers and experimented on b-tree, 4-way tree, and 8-way tree architectures. From these results, the B-tree architecture appears consistently show better or similar results, supporting the of B-tree architectures for GMLP experiments in this paper. Also, from Figure 15 and I6 using aggressively large branching factors results significant performance degradation as pooling many groups together results in information loss.
We hypothesize that the B-tree architecture outperforms other alternatives due to two factors: (i) In B B-tree, each pooling operation only merges information from two groups whereas for larger branching factor where multiple intermediate representations are being combined this pooling operation may result in larger information loss. (fi) Using larger branching factors, results in exponentially faster merging of the groups; therefore, limiting us to a much shallower network assuming the same number of groups
GMLP (B troe. L-6)
GNLP (B tree. k-352)
GHILI Eree, Lm3))
LML (4-way (o4)04)
CMLP (Boan kc56622)
Training Epocts
Training Epochs
Figure 15: The impact of using different tree types. For this experiment, we used k=352 and adjusted the number layers.
Figure 16: The impact of using different tree types. For this experiment, we used L=6 and adjusted the number of groups.
18

---

Under review as a conference paper at ICLR 2021
G ANALYSIS OF THE SELECTED FEATURE GROUPS
We used undirected graph visualizations to illustrate the feature groups and their relationship in GMLP networks. Specifically, consider each feature graph node and groups dense connection patterns between the nodes. Here, reappearance of certain edge, i.c. same sets of features appearing multiple groups, considered by an increase in the edge weight. See Figure 17 for toy example demonstrating the representation of feature groups as an undirected weighted graph. To visualize the resulting graphs, we used the spectral graph visualization method from the NetworkX library which clusters nodes based on the eigenvectors of the graph Laplacian 11
A
Graph
B
Group #1
B
Features
A
2
D
B
Group #2
D
B
Figure 17: A toy example demonstrating the representation of feature groups undirected weighted graph.
Figure presents the resulting graph visualizations for the HAPT, Tox21, and Diabetes datasets. The nodes with the highest weighted connections clustered and placed in close proximity of each other the visualization. From this figure, we can observe different grouping patterns for each dataset. For the HAPT dataset, the groups appear to be clustered but have strong overlaps with adjacent clusters. This indicates existence of feature groups that often share features among them. However, for the Tox21 dataset, observe very strong group clusters with very limited connection to other clusters. Hence, we can conclude that most feature groups are unique and less frequently share features with other groups. The pattern for the Diabetes dataset is very different certain features are appearing in many groups resulting in large cluster in the center of the graph, while at the same time there are features that are contributing very limited number of groups appearing far from the center of the graph. This analysis shows that the feature groups are highly dataset dependent, and using proper hyperparameters, GMLP is able learn the feature groups.
"https//nctworkxx.org
Von Luxburg, Ulrike. "A tutorial on spectral clustering." Statistics and computing 17.4 (2007): 395-416.
19

---

Under review as a conference paper at ICLR 2021
HAPT
(a)
Tox21
Diabates
Figure 18: Undirected graphs showing features as nodes and groups as the strength of connection
between the nodes for: (a) HAPT, (b) Tox21, and (c) Diabetes datasets.
20

---

Under review as a conference paper at ICLR 2021
H VISUAL ANALYSIS
In Figure 19 we present a visualization of the selected feature for 25 randomly selected groups in our final CIFAR-IO 10 architecture. Red, green, and blue colors indicate which channel is selected for each location. Compared to visualizations that are frequently used for convolutional networks, as GMLP has the flexibility to select pixels at different locations and different color channels, find explicit patterns in this visualization. However, one noticeable pattern is that features selected from a certain color channel usually appear in clusters resembling irregularly shaped patches.
easy to
Figure Visualization of pixels selected by each group for the CIFAR-10 GMLP architecture. Red, green, and blue colors indicate which channel is selected for each location. Due to space limitations, 25 random groups out of 1536 total groups visualized here.
21

---

Under review as a conference paper at ICLR 2021
Figure 20 shows the frequency in which each CIFAR-10 location is selected by the GMLP network. From this visualization, GMLP is mostly ignoring the border areas which can be a result of the data augmentation process used to train the network i.c., randomly cropping the center area and padding the margins.
Figure 20: Visualization of pixels selected by the group- select layer for the CIFAR-IO GMLP model. Warmer colors represent features that are being selected more frequently.
22

---

Under review as a conference paper at ICLR 2021
I ANALYSIS OF THE GMLP TRAINING COMPLEXITY
Section 3.5 provided a complexity analysis for the GMLP at the prediction time. To extend that analysis to the training time complexity, we need to consider the fact that, at the early stages of the training the routing matrix V is not necessarily sparse. Therefore, the training time memory and compute complexity for GMLP would (for simplicity, ignoring bias and pooling terms):
km2
km
km? km
kmd + km? +
+... 2 + +C 21-11
Therefore, the training complexity is of order Olkmdd + km + ckm)
In terms of the model size, during the training (before V converges to a sparse matrix), training the GMLP model requires storage for the km X d routing matrix as well as group-wise fully-connected layers. Based on our experiments, our implementation, the size of the routing matrix usually plays would like to note that
the dominant role in determining the memory requirements. Nonetheless, we were able to run all experiments in this paper on mid-range GPU with 1 11GB memory.
At prediction time, utilizing the sparsity of the routing matrix, the first term reduces to km resulting compute and memory complexity Olmm² Ckm as suggested Section 3.5 22-1
Group FC
2000
Backh Faah-Nirm
Group Pool
0000
Group
Poot
2000
Dato Now
Grep-RC
Bac--uen
Rell
Scftes
Group Select
Biah Drat-Nom Noms
Hel.
Grapp Pool
Gerapee
fitt
Figure 21: The GMLP network architecture.
23

---

Under review as a conference paper at ICLR 2021
J ANALYSIS OF INTRA-GROUT AND INTER-GROUP CORRELATIONS
We conducted experiments using the Tox21 and Diabetes datasets to measure inter-group and intra- group feature correlations. In Figure 22 and 23 subplot (a) shows the correlation matrix for the first 256 output features of the Group Select layer. Note that the group size is 28 for Tox21 I and is 4 for Diabetes. Figure 22 and 23 subplot (b) and (c) show the histogram of correlation values computed for inter-group and intra-group feature pairs.
From these figures. we do not find any significant difference in the correlation values for features within each group and features between different groups. We believe that this is an expected result as our objective function based on classification loss and does enforce any property among the learned feature groups. Note that often a level of inter-group and intra-group redundancy improves robustness of the trained models.
Tox21 Group Correlations (showing 9 groups)
50
100
150
200
250
200
250
(a)
Group Correlations (avg-0.15)
1015)
0.4
0,89
0.5
Prog 2010
23
0.65
0.35
64
1.0
9.4
18
Figure 22: Analysis of group correlations for the Tox21 dataset: (a) the correlation matrix for the first 256 output features of the Group Select layer, (b) histogram of correlation values computed for inter-group feature pairs, and (c) the histogram of correlation values computed for intra-group feature pairs.
24

---

Under review as a conference paper at ICLR 2021
Diabetes Group Correlations (showing groups)
50
100
150
200
250
100
150
200
250
(a)
Diabetes Inter-Group Correlations lavg-0.115
Diabetes Intra-Group Correlations lavg-0.14)
14:
6.0
600
2.6
(b)
Figure 23: Analysis of group correlations for the Diabetes dataset: (a) the correlation matrix for the first 256 output features of the Group-Select layer, (b) the histogram of correlation values computed for onter-grouy feature pairs, and (c) the histogram of correlation values computed for intra-group feature pairs..
25

---

Under review as a conference paper at ICLR 2021
K COMPARISON OF THE SOFTMAX AND CONCRETE RELAXATIONS
In this paper, we used a simple softmax with temperature to learn the discrete matrix and implement the Group- Select layer. Alternatively, one can use other ideas such as the concrete distribution suggested by Maddison et al. 12 For our implementation, we use Re l axeddnee Hot Categr cal class from the PyTorch library that is based on an implementation of the concrete distribution as suggested by Maddison et al. Here, we used similar temperature annealing schedule and took new samples from the distribution at every forward path computation.
Figure 24 provides a comparison of training GMLP networks using the suggested softmax relaxation and the alternative concrete distribution for the Tox21 and Diabetes datasets. Based on this result, we can see that the simpler softmax method achieves similar results for the Tox21 dataset and better results for the Diabetes dataset. We hypothesize that as the major use case of the concrete distribution is in variational methods and it involves random sampling, might be injecting level of noise and variation that is not necessarily helpful for learning the feature groups
KUROCC (50
AURoc
05
GMLP (Softmax Relaxation) CMLP (Cancratee Relaxation
[Softmax Relaxation)
GMLP [Concrete Relaxation)
20
306
226
100
150
200
24
Training Epochs
Training Epochs
(a)
(b)
Figure 24: Comparison of using the softmax relaxation and concrete relaxation implement the Group-Select layer: (a) the Tox21 dataset and (b) the Diabetes dataset.
12C. J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In ICI.R, 2017.
26


Reviews:
[{"review_id": "wqPuh0S5cMV", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "################################\nPro:\n$\\bullet$ It is an exciting idea to learn the relations in among feature dimensions, group them, and apply the operations within groups to train an MLP.  The experiments on various datasets, and important ablations such as the effect of number and size of groups, types of pooling were done.\n\n################################\nCons:\n$\\bullet$ $\\mathtt{Group-Pool}$ operation applies pooling inside feature group. Looking into Table 6 of the Appendix, mean and max-pooling were used in different datasets.  The ablation study is done on CIFAR-10 (Fig. 4). The assumption is always to have pairs of groups and apply pooling to consecutive group members. It contradicts the initial idea. Why didn't you apply a routing layer to learn which layers to pool together?\n\n$\\bullet$ *Ablation studies on image dataset:* Among many datasets experimented on; it is difficult to understand why all ablation studies are conducted on CIFAR-10. The paper's main motivation is to propose a method that works on domains beyond image, voice, and graphs. However, all ablations are done on an image dataset naturally that has spatial relations. \n\n$\\bullet$ *Incremental performance gain:* Looking into the performance comparison with vanilla MLP and other methods (SNN, SET, and FGR), the only considerable improvement from the MLP baseline is on the CIFAR-10 dataset. The results on all other datasets, accuracy, and area under curve scores are incremental. How many trials did you conduct the same experiments (the stdev values inside brackets of Table 2)? Did you run any statistical significance test between the trials of GMLP and MLP? \n\nMLP baselines can be reported from an architecture matching with the number of parameters in GMLP. In Table 2, the number of parameters should be written or standardized in each method.\n\n$\\bullet$ *What are \"MIT-BIH datasets\"?* They are not mentioned, referred to in the text, but given in Fig. 2 (b)\n\n$\\bullet$ *Finding optimal numbers of m and k:*  Finding the optimal number of groups and group sizes can introduce a large computational overhead. Fig. 5 and 6 show the performance margin could be 10-15%. Did you compare the optimal numbers of $m$ and $k$ between the datasets with a similar feature dimension? In other words, are the optimal values calculated in one dataset generalizable to other datasets to eliminate the necessity to train many models to find a good value.\n\n$\\bullet$ How does the group routing evolve during training? In a learned model, what is the relation between inter-group and intra-group feature correlations? (Are the members of the same group more relevant to each other in vice versa?)\n\n################################\nMinor:\n$\\bullet$ *As a term, the use of \"group\":* Use of group as a term causes confusion in the reader. In the introduction, it may be useful to include that the expression is not related to group ina mathematical sense, and only represents a subset of feature dimensions.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "i91vC_wKTNa", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors proposed a neural network architecture, Group-Connected Multilayer Perceptron (GMLP), which automatically groups the input features and extracts high level representations according to the groups. This paper focuses on classification problems.\n\nThe architecture can be decomposed to three stages. The first stage is to automatically group the input features by multiplying soft-max of a routing matrix. At the second stage, a locally fully connected layer with the corresponding activation functions is used for each group, and a pooling layer merges two groups to a new group. At the final stage, all the groups would be concatenated and input to a fully connect layer to get the final output. \n\nThe experiments showed that GMLP outperforms vanilla MLP, SNN, SET, FGR on seven real-world classification datasets in different domains. GMLP ensures higher accuracy with lower complexity compared to vanilla MLPs.\n\nThe authors claimed that if we consider the groups as leaves, this method then becomes growing a binary tree from the leaf to the root. The extensive experiment results showed the effect of GMLP hyper-parameter choices, e.g. number of groups (number of nodes), width of each group (size of nodes) and type of pooling layer (way to built parent nodes). But in terms of a tree, it would be interesting to have some experiments to show the effect of the way combining feature groups. \n\nThe experiments on the simulated Bayesian network dataset supported the claim that this architecture can utilize the fact that some of the features are not related and do not need to interact with each other. However, the architecture the authors used corresponds to the model that generated data, which is almost impossible in many real life problems. It can be helpful to have some results on simulated data with mismatched architectures from the model to help better understand the performance. \n\nOne of the most important ideas in this paper is limiting the group-wise interactions. The size and number of groups the experiments chose would lead to many overlapped groups and many features chosen multiple times. It would be nice to have some analyses on the chosen groups and selected features, e.g. the existence of a set of features that always come into one group, and/or comparison of derived groups by GMLP and randomly chosen groups.\n\nA more detailed explanation of the dataset would be needed. For example, the authors used MIT-BIH dataset to compare the accuracy of GMLP and MLP with different sizes without introducing the dataset.\n\nIf the complexity analysis of GMLP, equation (7), is only for inference, please also include the training complexity. Another concern is that the results suggest that the number of feature groups may need to be quite large (compared to the number of original features). In addition to figure 2, please provide the analysis of model size in addition to complexity analysis. \n\nGMLP selects features by using soft-max of a $km \\times d$ matrix. The authors may want to investigate reparametrization tricks to solve similar problems, including concrete relaxation in the following references: \n\nC. J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In ICLR, 2017.\n\nMuhammed Fatih Balin, Abubakar Abid, and James Y. Zou. Concrete autoencoders: Differentiable feature selection and reconstruction. In ICML, 2019.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-1j5k0S_1Q8", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes an MLP architectures for problems in which the features do not have a known structure (eg, tabular data). A \"differentiable routing matrix\" partitions the data into K blocks. Then, standard MLPs are applied to each block and the results are recursively aggregated by moving forward in the model.\n\nOn the plus side, the paper is well written, the topic is significant (as evidenced by any Kaggle competition on tabular data), and the model is simple enough to be understandable even on a quick reading.\n\nHowever, I do have a few concerns that put the paper on a borderline situation.\n[Addendum after review: most of the concerns have been addressed by the authors with a large set of additional experiments.]\n\n1. The paper is based on the assumption that a \"split-then-combine\" prior is sufficiently flexible to handle most real-world cases. However, this is kind of strongly limiting, eg, each feature can only contribute to a single \"high-level concept\" in the network. I don't feel that the paper does a strong job in justifying such an inductive bias.\n[Addendum after review: As the authors point out, this is only partially true, as each feature can be selected for multiple groups.]\n\n2. In the experimental part, I don't understand the motivation for using images. I can't imagine a single case in which one is given raw sensory data with absolutely no structure on it.\n\n3. Concerning the other datasets (the tabular ones), the authors report a decent improvement compared to an MLP, with a significant decrease in complexity. However, I do not understand the rationale for excluding random forests (or any non-neural baselines) from this comparison. A random forest would be the default algorithm in these situations (both in terms of accuracy and speed), and making MLPs competitive is one of the motivating factors in the paper itself.\n[Addendum after review: the authors have added a random forest to the experiments. However, this is the only method whose parameters are not fine-tuned (which might be a smaller problem for random forests). No time comparison is given, and some accuracy deviations appear well within the standard deviations. Still, this is a very good addition to the paper.]\n\n4. Is interpretability a concern here? Inspecting the routing matrix should reveal information on the feature grouping. I feel this can provide a strong boost to the paper.\n\nSummarizing, point (3) is the most important. Showing that the proposed Group MLP is superior to a classical alternative should be the critical aim of the comparisons.\n[Addendum after review: This point has been partially addressed.]", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an MLP based approach for data without known structure (such as tabular data). At first, the data are partitioned into K blocks in a differentiable way, then the standard MLP is applied to each block. The results are then aggregated recursively to produce the final output. 

Pros:
1. Handling less structured data is surely an important problem in machine learning and is much less explored. 
2. The paper is well written, easily understandable even with a fast browsing. 
3. The experimental results show some improvement. 

Cons:
1. The approach is somewhat trivial, and the framework could be improved, see, e.g. Reviewers #3&#4. 
2. By the structure of the approach and the type of target data, a more reasonable comparison is with random forest (echoing Reviewer #1), which the authors added during rebuttal, rather than MLP etc. Maybe should even compare with deep random forest. Although the comparison with MLP etc. is quite favorable, the advantage over random forest is somewhat marginal (except on HAPT, which is a imagery data set and random forest may not be good at; also echoing Reviewers #1&#4's comment on why using imagery data, which do not fit the theme of the paper). Reviewers #3&#4 also had some concerns with the experiments. Reviewer #4 confirmed in the confidential comment that the performance improvement is incremental.

Although the rebuttal seemed to be successful, thus both Reviewers #1 and #4 raised their scores, the average score is still at the borderline. Due to the limited acceptance rate, the area chair has to reject the paper.
Decision: Reject