Group-Connected Multilayer Perceptron Networks
Authors: Mohammad Kachuee
Sajad Darabi
Shayan Fazeli
Majid Sarrafzadeh
Venue:  (2020)

Abstract: Despite the success of deep learning in domains such as image, voice, and graphs, there has been little progress in deep representation learning for domains without a known structure between features. For instance, a tabular dataset of different demographic and clinical factors where the feature interactions are not given as a prior. In this paper, we propose Group-Connected Multilayer Perceptron (GMLP) networks to enable deep representation learning in these domains. GMLP is based on the idea of learning expressive feature combinations (groups) and exploiting them to reduce the network complexity by defining local group-wise operations. During the training phase, GMLP learns a sparse feature grouping matrix using temperature annealing softmax with an added entropy loss term to encourage the sparsity. Furthermore, an architecture is suggested which resembles binary trees, where group-wise operations are followed by pooling operations to combine information; reducing the number of groups as the network grows in depth. To evaluate the proposed method, we conducted experiments on five different real-world datasets covering various application areas. Additionally, we provide visualizations on MNIST and synthesized data. According to the results, GMLP is able to successfully learn and exploit expressive feature combinations and achieve state-of-the-art classification performance on different datasets.

Under review as a conference paper at ICLR 2020
GROUP-CONNECTED MULTILYEE PERCEPTRON NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Despite the success of deep learning in domains such as image, voice, and graphs, there has been little progress in deep representation learning for domains without a known structure between features. For instance, a tabular dataset of different demographic and clinical factors where the feature interactions are given as a prior. In this paper, we propose Group-Connected Multilayer Perceptron (GMLP) networks to enable deep representation learning in these domains. GMLP is based on the idea of learning expressive feature combinations (groups) and exploiting them to reduce the network complexity by defining local group-wise operations. During the training phase, GMLP learns a sparse feature grouping matrix using temperature annealing softmax with an added entropy loss term encourage the sparsity. Furthermore, an architecture is suggested which resembles binary trees, where group-wise operations are followed by pooling operations to combine information; reducing the number of groups the network grows in depth. To evaluate the proposed method, we conducted experiments on five different real-worke datasets covering various application areas. Additionally, we provide visualizations on MNIST and synthesized data. According to the results, GMLP is able successfully learn and exploit expressive feature combinations and achieve state-of-the classification performance on different datasets.
INTRODUCTION
Deep neural networks have been quite successful across various machine learning tasks. However, this advancement has been mostly limited to certain domains. For example in image and voice data, one can leverage domain properties such as location invariance, scale invariance, coherence, etc. via using convolutional layers (Goodfellow et al., (016). Alternatively, for graph data, graph convolutional networks were suggested to leverage adjacency patterns present in datasets structured graph (Kipf & Welling 2016 Xu et al. 2019).
However, there has been little progress in learning deep representations for datasets that do not follow particular known structure in the feature domain. Take for instance the case of simple tabular dataset for disease diagnosis. Such a dataset may consist of features from different categories such as demographics (e.g., age, gender, income, etc.), examinations (e.g., blood pressure, lab results, etc.), and other clinical conditions. In this scenario, the lack of any known structure between features to be used as prior would lead the use of fully-connected multilayer perceptron network (MLP). Nonetheless, it has been known in the literature that MLP architectures, due to their huge complexity, do not usually admit efficient training and generalization for networks of more than few layers.
In this paper, we propose Group- Compp-Ceneettdd Multiplayer Perceptron (GMLP) networks. The main idea behind GMLP is learn and leverage expressive feature subsets, henceforth referred as feature groups. A feature group is defined as a subset of features that provides a meaningful representation or lige- concept that would help the downstream task. For instance, the disease diagnosis example, the combination of certain blood factor and age might be the indicator of higher level clinical condition which would help the final classification task. Furthermore, GMLP leverages feature groups limiting network connections to local group-wiss connections and builds feature hierarchy via merging groups as the network grows in depth. GMLP can be seen as an architecture that learns expressive feature combinations and leverages them via group-wist wise operations.
The main contributions of this paper are as follows: (t) proposing a method for end-to-end learning of expressive feature combinations, (fi) suggesting network architecture to utilize feature groups and

---

Under review as a conference paper at ICLR 2020
local connections to build deep representations, (iii) conducting extensive experiments demonstrating the effectiveness of GMLP as well as visualizations and ablation studies for better understanding of the suggested architecture.
We evaluated the proposed method on five different world datasets in various application domains and demonstrated the effectiveness of GMLP compared ofte-of-thheart methods in the literature. Furthermore, we conducted ablation studies and comparisons to study different architectural and training factors as well as visualizations on MNIST and synthesized data. To help reproduce the results and encouraging future studies on group-connected architectures, we made the source code related to this paper available online
2 RELATED WORK
Fully-connected MLPs are the most widely-used neural models for datasets in which no prior assumption made on the relationship between features. However, due to the huge complexity of fully-connected layers, MLPs are prone to overfitting resulting in shallow architectures limited to a few layers in depth (Goodfellow et al., 2016). Various techniques have been suggested to improve training these models which include regularization techniques such as L--1/I-2 regularization, dropout, etc. and normalization techniques such as layer normalization, weigh normalization, batch normalization, etc. (Srivastava et al., 2014 Ba et al.) 2016 Salimans & Kingmal 2016; [offe & Szegedy 2015). For instance, self-normalizing neural networks (SNNs) have been recently suggested as state of the art normalization methods that prevent vanishing or exploding gradients which help training feed-forward networks with higher depths (Klambauer et al., 2017).
From the architectural perspective, there has been great attention toward networks consisting of sparse connections between layers rather than having dense fully-connected layers (Dey et al. 2018). Sparse connected neural networks are usually trained based on either a sparse prior structure over the network architecture (Richter & Wattenhofer 2018) or based on pruning a cully-connected network to a sparse network (Yun et al. 2019) Tartaglione et al. 2018 Mocanu et al. 2018). However, it should be noted that the main objective of most sparse neural network literature has been focused on improving the memory and compute requirements while maintaining competitive accuracies compared to MLPs.
As parallel line of research, the idea of using expressive feature combinations groups has been suggested as a prior over the feature domain. Perhaps, the most successful and widespread use of this idea is in creating random forest models in which different trees are trained based on different feature subsets in order to deal with high-dimensional and high-variance data (Breiman (000). More recently, feature grouping is suggested by AAyore et al. (2119) as a statistical regularization technique to learn from datasets of large feature size and a small number of training samples. They do the forward network computation by projecting input features using samples taken from a bank of feature grouping matrices, reducing the input layer complexity and regularizing the model. In another recent study, Ke et al. (2018) used expressive feature combinations to learn from tabular datasets using a recursive encoder with a shared embedding network. They suggest recursive architecture in which more important feature groups have a more direct impact on the final prediction.
While promising results have been reported using these methods, feature grouping has been mostly considered as a preprocessing step. For instance, AAyore et al. (2019) uses the recursive nearest agglomeration (ReNA) (Hoyos-Idrobo et al., 2018) clustering to determine feature groups prior to the analysis. Alternatively. Ke et al. (2018) defined feature groups based on a pre-trained gradient boosting decision tree (GBDT) (Friedman 2001). Feature grouping as a preprocessing step not only increases the complexity and raises practical considerations, but also limits the optimality of the selected features in subsequent analysis. In this study, we propose end-to-end solution to learn expressive feature groups. Moreover, we introduce a network architecture to exploit interrelations within the feature groups reduce the network complexity and to train deeper representations.
We plan to include link to the source code and GitHub page related to this paper in the camera-ready version.
2

---

Under review as a conference paper at ICLR 2020
Group
Groug-1 PC
Rel.1
Group Pool
Group Select
Group PC 00001
Rach-Norm
1000
GRougFF
00000
Concot
Softout
Group
0
Onarp PC
ReLU
1000
Group Pool
Orau-PC
Nomm
Figure 1: The GMLP network architecture.
PROPOSED METHOD
3.1 ARCHITECTURE OVERVIEW
In this paper, we propose GMLP which intuitively can be broken down to three stages: (i) selecting expressive feature groups, (ii) learning dynamics within each group individually, and (zii merging information between groups as the network grows in depth (see Figure D In this architecture, expressive groups are jointly selected during the training phase. Furthermore, GMLP is leveraging feature groups and using local group-wise weight layers to significantly reduce the number of parameters. While the suggested idea can materialized as different architectures, in the current study, we suggest organization of the network as architectures resembling binary tree spanning from leaves (i.c., features) to a certain abstraction depth closer the roo As the network grows deeper, after each local group-wise weight layer, half of the groups are merged using pooling operations, effectively reducing the width of the network while increasing the receptive field. At the last layer, all features within all groups concatenated into dense feature vector fed to the output layer.
3.2 NOTATION
We consider the generic problem of supervised classification based on dataset of feature and target pairs, D: (æI.N, V1:N), where x; E Rd y: E (1...C) .), and N is the number of dataset samples. Furthermore, we define group size, group count, k; as the number of selected groups which are essentially subsets of input features. Also, L is used to refer to the total depth of network. We use 2 E R to refer to activation values of group in layer I. In this paper, we define all vectors as column vectors.
the number of neurons or elements within each group, and
3.3 NETWORK LAYERS
In this section, we present the formal definition of different GMLP network layers. The very first layer of the network, Sroup-Selects is responsible for organizing features into k groups of size m each. A routing matrix, V. used for connecting each neuron within each group exactly one feature in the feature set:
(I)
Pleesse note that, in this paper, tree structures are considered to grow from leaves to the root In other words. in this context, limiting the depth is synonymous with considering the tree portion spanning from certain depth leave nodes.

---

Under review as a conference paper at ICLR 2020
where V/ {0,1)
_ is sparse matrix determining features that present in each group. As we are interested in jointly learning V during the training phase, we use the following continuous relaxation:
exp(bb./T)
Vi Ej'= exp((xa/T)
In this equation, 1o is a real-valued matrix reparameterizing the routing matrix through a softmax operation with temperature, т. The lower the temperature, the more (2) converges the desired discrete and sparse binary routing matrix. Note that, in the continuous relaxation, the matrix &i can be optimized via the backpropagation of gradients from classification loss terms. In the next section, we provide further detail on temperature annealing schedules as well as other techniques to enhance the approximation.
Based on selected groups, suggest local fully-connected weight layers for each group: Group-FC. The goal of Group-FC is to extract higher-level representations using the selected expressive feature subsets. This operation is usually followed by non-linearity functions (e.g.. ReLU), normalization operations (e.g. Batch Norm), and dropout. Formally, Group FC can be defined as:
1+1 f(W] b().
where W/[ Rm and by < R are the weight matrix and bias vector, applied on group at layer I. Here, f represents other subsequent operations such as non- con-linearity, normalization, and dropout.
Lastly, Group-Pool is defined an operation which merges representations of two groups into a single group, reducing network width by half while increasing the effective receptive field:
z!+1 pool (z), z #(+4//(+1)
where 2/ and x/+2//2 are the ith group from the first and second halves, respectively, and pool is pooling function from p2am to R In this study, we explore different variants of pooling functions such as max pooling, average pooling, or using linear weight layers as transformations from 822m to Rm. Please note that while we use similar terminology as pooling in convolutional networks, the pooling operation explained here is not applied location-wise, but instead it is applied feature-wise, between different groups pairs.
The values of m and k are closely related to the number and order of feature interactions for a certain task. Using proper and k values enables us to reduce the parameter space while maintaining the model complexity required to solve the task. However, finding the ideal m and & directly from a given dataset is very challenging problem. In this work, we treat and is as hyperparameters be found by hyperparameter search.
3.4 TRAINING
We define the objective function to be used for end-to-end training of weights as well as the routing matrix
1 L L= -= N -- Us. log(Fo(z;)) + (H + a Σ |wll
(5)
In this objective function, the first term is the standard cross-entropy classification loss where Fø denotes the GMLP network function with parameters 0, and N is the number of training samples used. The second term an entropy loss over the distribution of the routing matrix that is weighted by the hyperparameter A:
j=d diimm
H = - Σ
exp(1xx5) - exp (1i..))
log( o'== (xx(()x.)))
exp(1Ii.5)
(6)
(e)) is minimizing the entropy corresponding to the distribution of regardless of the temperature used for V approximation. Accordingly, À can be viewed as hyperparameter and an additional method for encouraging sparse V matrices. The last term in (5) is an L-2 regularization term with the hyperparameter control the magnitude of parameters in layer weights and in th. Note that
4

---

Under review as a conference paper at ICLR 2020
without the L-2 regularization term, s elements may keep increasing during the optimization loop. since d only appears in normalized form in the objective function of (5)
We use Adam (Kingma & Bal 2014) optimization algorithm starting from the default 0.001 learning rate and reducing the learning rate by a factor of as the validation accuracy stops improving. Regarding the temperature annealing. during the training, the temperature is exponentially decayed from 1.0 to 0.01. In order to initialize the Group-FC weights, we used Xavier initialization (Glorot & Bengio 2010) with for both fan-in and fan-out values. Similarly, the i matrix is initialized by setting the fan-in equal to d and fan-out to kvn.
Further detail on architectures and hyperparameters used for each specific experiment as well as details on the software implementation are provided as appendices to this paper.
3.5 ANALYSIS
The computational complexity of GMLP at the prediction time can be written as (for simplicity, ignoring bias and pooling terms):
km
kmm
km 2L-1 + C
km
km + km2
+
+
2
4
24-1
In this series, the first term, km, is the work required to organize features to groups The subsequent terms, except the last term, are representing the computational cost of local culle-connected operations at each layer. The last term is the complexity of the output layer transformation from the concatenated features to the number of classes. Therefore, the computational complexity of GMLP at the prediction Ckm In comparison, the computational complexity of an MLP
time can be written as Okm² 2 + 2
with a similar network width would bc:
m22
m²2m²
k2m²
km
kmd + k2 k²² +
C
2
21.-1
21-11'
where the first term is the work required for the first network layer from d to km neurons, the second term is corresponding to hidden layer of size km, and so forth. The last term is the complexity of the output layer similar to the case of GMLP. The overall work required from this equation is of Ckm
Olkmd + k2m² 24-1) ) complexity. This is substantially higher than GMLP, for typical k, d, and C' values.
Additionally, the density of the Group FC layer connections can be calculated as: kmm t. which very small for reasonably large number of l values used in our experiments. Also, assuming pooling operations in every other layer, the receptive field size or the maximum number of features impacting a neuron at layer can be written as 2 mm For instance, a neuron in the first layer of the network only connected to m features, and a neuron in the second layer is connected to two groups or 2m features and so forth.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
The proposed method is evaluated on five different real- world datasets, covering various domains and applications: permutation invariant CIFAR-IO (Krizhevsky et al. 22009) human activity recognition (HAPT) (Anguita et al., 2013), diabetes classification (Kachuee el al. 2019), UCI Landsat (Dua & Graff] 2017). and MIT-BIH arrhythmia classification (Moody & Mark 2001) datasets. Additionally, wc use MNIST (LeCun et al. 2010) and a synthesized datasct to provide further insight into the operation of GMLP (see Section 4.4). Table I presents a summary of datasets used in this study. Regarding the CIFAR-10 dataset, we permute the image pixels to discard pixel coordinates in our experiments. Note that the permutation is not changing across samples, is merely a fixed random ordering used to remove pixel coordinates for each experiment. For all datasets, basic statistical normalization with μ = and o = 1 is used to normalize features as a preprocessing step. The only exception is CIFAR- 10 for which data augmentation (i.e., random crops and random horizontal flips). The standard test and train data splits were used dictated by dataset publishers. In cases that the separated sets not provided,
used the standard channel-wist normalization and standard
5

---

Under review as a conference paper at ICLR 2020
Table 1: Summary of datasets used in our experiments.
# Train Samples
# Test Samples
Features
Classes
Domain
CIEAR-I
Krizhevsky
50,000
10,000
3072
10
Image Clussificution
HAPT
Anguila
2013
6,002
2,451
561
5
Activity Recognition
Diabetes
Kachuce ct al.
2019
73,649
18,413
45
3
Disease Diagnosis
Landsat
Dua & Graff
2017
4,435
2,000
36
6
Satellite Imaging
MIT-BIH
Mondy & Mark
2001
87,554
21,892
Time Series
MNIST
[LeCun
60,000
10,000
784
10
Digit Classification
Synthesized (Section 4.4
5,120
1,280
See Figure
'Permuted version, i.c pixel coordinates are ignored.
test and train subsets are created by randomly splitting samples to 20% for test and the rest for training/validation..
We compare the performance of the proposed method with recent related work including Self- Normalizing Neural Networks (SNN) (Klambauer et al., 2017), Sparse Evolutionary Training
(SET) [Mocanu et al. 2018 8 3 Feature Grouping as a Stochastic Regularizer (in this paper, de- noted as FGR) (Aydore al. 2019 as well as the basic dropout regularized and batch normalized MLPs. In order to ensure fair comparison, we adapted source codes provided by other work to be compatible with our data loader and preprocessing modules.
Furthermore, for each method, we conducted an extensive hyperparameter search using Mi- crosoft Neural Network Intelligence (NNI) toolkif and the Tree-structured Parzen Estimator (TPE) tuner (Bergstra et al. 2011) covering different architectural and learning hyperparameters for each case. More detail on hyperparameter search spaces and specific architectures used in this paper is pro- vided in Appendix A and Appendix B We run each case using the best hyperparameter configuration eight times and report mean and standard deviation values.
4.2 RESULTS
Table 2 presents a comparison between the proposed method (GMLP) and 4 other baselines: MIP, SNN Klambauer et al. 2017). SET (Mocanu et al. 2018), and FGR (Aydore et al.) 2019).. As it can be seen from this comparison, GMLP outperforms other work, achieving state-of-the-arr classification accuracies. Concerning the CIFAR-10 results, to the best of our knowledge, GMLP achieves new state-of-the-art performance on permutation invariant CIFAR- 10 augmented using the standard data augmentation. We believe that leveraging expressive feature groups enables GMLP to consistently perform better across different datasets.
To compare model complexity and performance we conduct an experiment by changing the number of model parameters and reporting the resulting test accuracies. Here, we reduce the number of parameters by reducing the width of each network; i.e. reducing the number of groups and hidden neurons for GMLP and MLP, respectively. Figure 2 shows accuracy versus the number parameters for the GMLP and MLP baseline CIFAR-10 and MIT-BIH datasets. Based this figure, GMLP able to achieve higher accuracies using significantly less number of parameters. is consistent with the complexity analysis provided in Section 3.5 Note that in this comparison, we consider the number of parameters involved at the prediction time.
4.3 ABLATION STUDY
Figure presents an ablation study comparing the performance of GMLP on CIFAR-10 dataset for networks trained: (i) using both the temperature annealing and the entropy loss objective, (fi) using only temperature annealing without the entropy loss objective, (iii) using no temperature annealing but using the entropy loss objective, (ir) not using any of the temperature annealing or the entropy loss
"https//gittuu..om//// http://twww...mmmmmmmmmm/o//tnnn hwwpraaaaaaaaiiiiaaeeeeeenttttoo
hhttps//gihuu..co///ggugaaaaa http://..ccooooooeeeeeeeee https://gihuh..oo/miccroo
htwp://eeaaaaaeee/aaaaaaaarauuaa Regularizer

---

objective. From this figure, it can be seen that excluding both techniques leads to significantly lower performance. However, using any of the two techniques leads to relatively similar high accuracies. It is consistent with the intuition that the functionality of these techniques is to encourage learning sparse routing matrices, either using softmax temperatures or entropy regularization to achieve this. both techniques Under review as a conference paper at ICLR 2020
Table 2: Comparison of top-1 test accuracies for GMLP and other work.
Accuracy (%)
Dataset
GMLP
MLP
SNN
SET
FGR
CIFAR-10
Krizhevsky et
2009
73.76 (t0.14)
68.15 (#0.54)
66.88 (+0.30)
72.71 (#0.29)
45,90 (t0.40)
HAPT
Anguita et al.
2013
96.34 (40.19)
95.73 (40038)
95.47 (40.09)
71.35 (40.74)
91.57 (40.31)
Diabetes
Kachuee et al
2019
90.92 (10.07)
90.68 (10.066)
90.31 (10.10)
85.07 (10.61)
89.24 (10.08)
Landsat
Dua Grall
2017
91.54 (20.16)
91.21 (#0.44)
91.37 (#0.21)
91.03 (#0.56)
90.70 (20.17)
MIT-BIH
Moody & Mark 2001
98.74 (10.04)
98.65 (10.01)
98.56 (10.02)
98.10 (10.01)
98.13 (10.00)
"Self-Normalizing Neural Networks (Klambauer et "Sparse Evolutionary Training (Mocanu ct al. 2018)
2017)
'Feturre Grouping as Stochastre Regularizer (Aydore et al.
2019
*Permuted version, i.e pixel coordinates are ignored.
CIFAR-10
0
6
Accuracy (6)
Accuratyy 1%5)
50
38.0
GMLP
GMLP
MLP
Number Parameters
Number of Parameters
(a)
(b)
Figure 2: Accuracy versus number of parameters for this work (GMLP) and the MLP baseline: (a)
CIFAR-10 10 dataset, (b) MIT-BIH dataset. The x-axis is in a logarithmic scale
In this paper, in order to ensure sparse and low complexity routing matrices, simultancously as in case (i).
Figure shows comparison between GMLP models trained on CIFAR-10 using different pooling types: (i) linear transformation, (ii) max pooling, and (iit) average pooling. As it can be seen from this comparison, while there are slight differences in the convergence speed of using different pooling types, all of them achieve relatively similar accuracies. In our experiments, we decided to use max pooling and average pooling they provide reasonable results without the need introduce additional parameters required for the linear pooling method.
Figure shows learning curves for training CIFAR-10 10 GMLP models using different group sizes. As it can be seen from this figure, using very small group sizes would cause a reduction in the final accuracy. At the other extreme, the improvement achieved using larger values is negligible for m values more than 16. Finally, Figure 6 shows comparison between learning curves for using different number of groups. Using very small & values result in a significant reduction in performance. However, the rate of performance gains for using more groups very small for & of more than 1536. Note that the number model parameters and compute scales lincarly with k and quadratically with m (see Section 3.5
7

---

Under review as a conference paper at ICLR 2020
Accuracy (80
Accuracy (36)
W/ Temp, Annealing, W/ Entropy Loss Temp. Arnealing, Entropy Loss W/O Temp. Annealing, W/o Entropy Loss W/ Temp. Annealing, WIO Entropy Loss
Pooling
Max Pooling
Average Pooling
250
392
1000
1252
2502
1750
250
300
/50
2003
2050
1520
1/50
Training Epochs
Training Epochs
Figure 3: Ablation study demonstrating the impact Figure 4: Ablation study demonstrating the impact of different pooling functions.
of temperature annealing and entropy loss terms.
Accuraiy (16
Accuracy 1963
k-3072
m-16 16
k-1536
m=12 12
k-768
384
mm6
192
750
1502
1750
Training
Training Epochs
Figure 5: Ablation study the impact of using Figure 6: Ablation study the impact of using dif- different group sizes (m)) For this experiment, we used k= 1536.
ferent number of groups (k).. For this experiment, used m=16.
4.4 EXPERIMENTS ON MNIST AND SYNTHESIZED DATA
MNIST dataset (LeCun et al. 2010) used to visually inspect the performance of the Group-Select layer. Figure shows heat-map of how frequently each pixel is selected across all feature groups for: (a) original MNIST samples, (b) MNIST samples where the lower-half is replaced by Gaussian noise. From Figure 7a it can be within the center of the frame, effectively discarding margin pixels. This is consistent with other work which show the importance of different locations for MNIST images (Kachuee al., 2018). Apart from this, in Figure 7b version of the MNIST dataset is used in which half of the frame does provide any useful information for the downstream classification task. From this figure, GMLP is not selecting any features to be used from the lower region.
that most groups selecting pixels
In order show the effectiveness GMLP, synthesized dataset which has intrinsic and known expressive feature groups. Specifically, we used simple Bayesian network as depicted in Figure g This network consists of six binary features, A to F, interacting with each other as specified by graph edges, which determine the distribution of the target node, J. The graph and conditionals are designed such that each the nodes in the second level take the XOR value of their parents with 99% probability. The target node, J, is essentially one with high probability if at least two of the second level nodes are one. We synthesized dataset by sampling 6,400 samples from the network (1,280 samples for test and the rest of training/evaluationn.. On this dataset, trained very simple GMLP consisting of four groups of size two, one group-wise fully- cunly-connected layer, and an output layer. Figure 9 shows the features selected for each group after the training phase (i.e., the V matrix).
8

---

Under review as a conference paper at ICLR 2020
(a)
(b)
Figure 7: MNIST visualization of pixels selected by the Group-Seless layer: (a) using complete images as imput, (b) using images that the lower half is replaced by Gaussian noise. In this figure, warmer colors represent pixels being being present in more groups.
From this figure, the Group Group-Seledd Select layer successfully learns to detect the feature pairs that interacting, enabling the Group-Fo layers to decode the non-linear XOR relations.
Group #1
Group #2
Group #3
P(G = 4 0.99
PPG # A =)
P( = C e 0.99
P(H C 0.01
P( E. F) 0.99
PI EQ F)
Group #4
P[ = (G* (HT A D)] = 0.99
PU # VI D)] = 0.01
B
D Features
C
F
Figure 8: The Bayesian network and conditionals used to generate the synthesized dataset of binary features A-F and target J.
Figure Visualization of the selected features within each group. Every two consecutive rows show features selected for certain group.
DISCUSSION
Intuitively, training GMLP model with certain groups can be viewed as a prior assumption over the number and order of interactions between the features. It is reasonable prior assumption in many natural datasets, a conceptual hierarchy exists where only limited number of features interact with each other. Furthermore, it is consistent with the discoverics made in understanding the human cecision-maaing process; finding that we are only able to consider at most nine factors at the same time during decision making process (Cowan 2001 Baddeley 1994).
Additionally, GMLP can be considered as a more general neural counterpart of random forests. Both models use different subsets of features (i.e., groups) and learn interactions within each group. One major difference between the two methods is the fact that GMLP combines information between different groups using pooling operations, while random forest uses the selected features train

---

Under review as a conference paper at ICLR 2020
ensemble of independent trees each group. From another perspective, the idea of studying feature groups is closely related to causal models such as Bayesian networks and factor graphs (Darwiche) 2009 Neapolitan et al., 2004] Clifford 1990). These methods are often impractical for large-scale problems, because without a prior over the causal graph, they require an architecture search of the NP-complete complexity more.
CONCLUSION
In this paper, we proposed GMLP solution for deep learning in domains where the feature interactions are not known as prior and do not admit the use of convolutional or other techniques leveraging domain priors. GMLP jointly learns expressive feature combinations and employs group- wise operations to reduce the network complexity. We conducted extensive experiments demonstrating the effectiveness of the proposed idea and compared the achieved performances with state-of-the-ee art methods in the literature.
REFERENCES
Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, and Jorge Luis Reyes-Ortiz. public domain dataset for human activity recognition using smartphones. In Esann, 2013.
Sergul Aydore, Bertrand Thirion, and Gael Varoquaux. Feature grouping stochastic regularizer for high-dimensional structured data. In International Conference on Machine Learning, pp. 385-394, 2019.
Jimmy L.i Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 1607.06450, 2016.
Alan Baddeley. The magical number seven: Still magic after all these years? 1994.
James S Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter optimization. In Advances in neural information processing systems, pp. 2546-2554, 2011.
Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
Peter Clifford. Markov random fields in statistics. Disorder in physical systems: A volume in honour of John M. Hammersley, 19, 1990.
Nelson Cowan. The magical number 4 in short-term memory: A reconsideration of mental storage capacity, Behavioral and brain sciences, 24(1):87-114, 2001.
Adnan Darwiche. Modeling and reasoning with Bayesian networks. Cambridge university press, 2009.
Sourya Dey, Kuan-Wer Huang, Peter A Beerel, and Keith M Chugg, Characterizing sparse connec- tivity patterns in neural networks. 2018 Information Theory and Applications Workshop (ITA), pp. 1-9. IEEE, 2018.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http:://tcete /archive ics. moci.edu/me
Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of statistics, pp. 1189-1232, 2001.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http: 7/www. www.:/deepleernnnngeee. www.rnnnnnoooo..oo .rg
Andrés Hoyoo- Gaël Varoquaux, Jonas Kahn, and Bertrand Thirion. Recursive nearest agglomeration (rena): fast clustering for approximation of structured signals. IEEE transactions on pattern analysis and machine intelligence, 41 (1))6669-6681, 2018.
10

---

Under review as a conference paper at ICLR 2020
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
Mohammad Kachuee, Sajad Darabi, Babak Moatamed, and Majid Sarrafzadeh. Dynamic feature acquisition using denoising autoencoders. IEEE transactions on neural networks and learning systems, 2018.
Mohammad Kachuee, Kimmo Karkkainen, Orpaz Goldstein, Davina Zamanzadeh, and Majid Sar- rafzadeh. Nutrition and health data for cost-sensitive learning. arXiv preprint arXiv: 0et.. 002 2019.
Guolin Ke, Jia Zhang, Zhenhui Xu, Jiang Bian, and Tie-Yan Liu. Tabnn: A universal neural network solution for tabular data. 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.60 6980, 2014.
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv: 1609.02907, 2016.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. In Advances in neural information processing systems, pp. 971-980, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Yann LeCun, Corinna Cortes, and Burges. Mnist handwritten digit database. AT&T Labs [Online]. Available: http://yann. lecun. com/exdh/mnist, 2:18, 2010.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9(1):2383, 2018.
George Moody and Roger G Mark. The impact of the mit-bih arrhythmia database. IEEE Engineering in Medicine and Biology Magazine, 20(3))45-50, 2001.
Richard E Neapolitan et al. Learning bayesian networks, volume 38. Pearson Prentice Hall Upper Saddle River, NJ, 2004.
Oliver Richter and Roger Wattenhofer. Treeconnect: A sparse alternative to fully connected layers. In 2018 IEEE 30th International Conference on Tools with Artificial Intelligence (ICTAI), pp. 924 931. IEEE, 2018.
Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, pp. 901-909, 2016.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov: Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15())1199-1158, 2014.
Enzo Tartaglione, Skjalg Lepsøy, Attilio Fiandrotti,, and Gianluca Francini. Learning sparse neural networks via sensitivity-driiee regularization. In Advances in Neural Information Processing Systems, pp. 3878-3888. 2018.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019. URL https
net. for
Jihun Yun, Peng Zheng, Eunho Yang, Aurelie Lozano, and Aleksandr Aravkin. Trimming the I regularizer: Statistical analysis, optimization, and applications to deep learning. In International Conference on Machine Learning, pp. 7242-7251, 2019.
11

---

Under review as a conference paper at ICLR 2020
A HYPERPARAMETER SEARCH SPACE
Tables present the hyperparameter search space considered for experiments on GMLP, MLP, SNN, and FGR, respectively. For the GMLP search space, the number groups is adjusted based on the number of features and samples in each specific task. Also, the number of layers is adjusted be compatible with the number of groups being used. Regarding the FGR experiments, due to scalability issues of the published source provided by the original authors, we were only able to train networks with at most two hidden layers. For SET, as their architecture is evolutionary i.e., prunes certain weights and adds new ones, we only explored using a different number of hidden neurons in the range of 500 4000.
Regarding the number of epochs, we used 2000 epochs for CIFAR-10, 1000 epochs for HAPT, 100 epochs for Diabetes, 300 epochs for Landsat, and 300 epochs for MIT-BIH experiments. The only exception is the SNN experiments where we had to reduce the learning rate to increase the stability of the training resulting in more epochs required to converge.
Table 3: Hyperparameter search space used for GMLP experiments.
Hyperparameter
Considered Values
Number hidden layers
{1, 2, 8
Number of groups
16 - 4096
Size of groups
- 32
Lambda
[10-2 104]
Alpha
[10-12 12 10-1]
Dropout rate
[0. 1]
"The range is adjusted based on the number of groups. 'The range is adjusted based on the number of features and sample size.
Table 4: Hyperparameter search space used for MLP and SNN experiments.
Hyperparameter
Considered Values
Number of hidden layers
(1,2,..., 6}
Size of hidden layers
[0.05 x [ifaatures - [20 X (efeatures)
Alpha
[10-12,10-]] 10-1]
Dropout rate
[]
Table 5: Hyperparameter search space used for FGR experiments.
Hyperparameter
Considered Values
Number of hidden layers
[1,2}
Size of hidden layers
fefeatures - [50 x nl neeauures)
Number of groups
2 n nfeaauree
Alpha
[10 12 10-1]
12

---

Under review as a conference paper at ICLR 2020
B ARCHITECTURES
Table 9 10 show the selected architectures for GMLP, MLP, SNN, SET, and FGR, respectively. We used the following notation to indicate different layer types and parameters: GSel-k repre- sents a Group Select layer selecting k groups of m features each. GF C indicates Group FC lay- ers, and FC-x represents fully-connected layer with hidden neurons. GPool-x is Group-Pool layer of type x (max, mean, linear, etc.), Concat is concatenation of groups used prior to the output layer in GMLP architectures. SC-x refers to SET sparse evolutionary layer of size x.
Group-Select
Table GMI.P architectures used our experiments.
Dataset
Architecture
GSe1-1536-16, GFC, ReLU, BNorm, GPool-max, GFC, RelJ, BNorm,
CIFAR-10
GPool-max, CFC, ReLU, BNorm, CPool-max, CFC, ReLU, BNorm, Concat, FC-10, Softmax
CSe1-288-12, GPC, ReLU, 3Norm, GPool-mean CFC, ReLJ, BNorm,
НAРT
Concat, FC-5, Softmax
GSe1-16-16, GFC, ReIU, BNorm, GPool-mean, SFC, ReLU, BNorm,
Diabetes
CPool-mean, CFC, ReLU, BNorm, CPool-mean, SFC, ReLU, BNorm,
Gooel-mean, Concat, FC-3, Softmax
GSe1-88-16, GFC, ReLU, BNorm, GPool-mean, SFC, ReLU, BNorm,
Landsat
CPool-mean, CFC, ReLU, BNorm, G2ool-mean, Concat, FC-6, Softmax
GSe1-240-24, GPC, ReLU, 3Norm, GPool-mean, GFC, ReIJ, BNorm,
MIT-BIH
GPool-mean, CFC, ReLU, BNorm, CPool-mean, Concat, FC-5, Softmax
GFC, ReMu BNorm, GPool - GFC, ReLU, BNorm,
MNIST
GPool-mean, GFC, ReIU, BNorm, Goool-mean, SFC, ReLU, BNorm,
Concat, FC-10, Softmax
Synthesized
GSe1-4-2, GFC, ReLU, BNorm, Concat, FC-2, Softmax
Table MLP architectures used our experiments.
Dataset
Architecture
FC-3072, ReLU, BNorm, FC-2764, ReLU, BNorm,
CIFAR-10
FC-2488, ReLU, BNorm, FC-10, Softmax
НAРT
FC-106, ReLU, BNorm, EC-21, ReLU, BNerm, FC-5, Softmax
FC-45, ReLU, BNorm, FC-45, ReLU, ENorm, FC-45, ReLU, BNorm,
Diabetes
FC-45, ReLU, BNorm, FC-3, Softmax
FC-68, ReLU, BNorm, FC-68, ReLU, BNorm, FC-68, ReLU, BNorm,
Landsat
FC-68, ReLU, BNorm, FC-6, Softmax
FC-1737, ReIU, BNorm, FC-1737, ReTU, BNorm,
MIT-BIH
FC-1737, ReLU, BNorm, FC-5, Sofzmax
13

---

Under review as a conference paper at ICLR 2020
Table 8: SNN architectures used in our experiments.
Dataset
Architecture
FC-3901, SeLU, BNorm, FC- FC-3901, SeLU, BNorm, FC-3901, SeLU, BNorm, FC-10, Softmax
CIFAR-10
FC-510, ReLU, BNorm, FC-510, SeLU, BNorm,
НAРT
FC 510, SeLU, FC-5, Softmax
FC 10-1 SeLU, BÑorm, FC 105, SeLU, BÑorm,
Diabetes
FC-105, SeLU, ENorm, FC-3, Softmax
Landsat
FC 816,
SeLU, BÑorm, FC 816, SeLU, BÑorm, FC 6, Softmax
SeLU, BÑorm, FC-1140, SeLU, BNorm, FC-1140, SeLU, BNorm, FC-5, Softmax
FC 1140,
MIT-BIH
Table 9: SET architectures used in our experiments.
Dataset
Architecture
CIFAR-10
SC-4000, SReLU, SC-1000, SReLU, SC-4000, SReLU, FC-10, Softmax
HAPT
SC-500, SReLU, SC-500, SReLU, 8C-500, SReLU, FC-5, Softmax
Diabetes
SC-1000, SRe1l,, sC-1000, SRetU, sc-1000, SRelu, FC-3, Softmax
Landsat
sc-1000, SReLU, SC-1000, SReLU, sc-1000, SReLU, FC-6, Softmax
MIT-BIH
SC-1000, SReLU, SC-1000, SRcLU, SC-1000, SReLU, FC-5, Softmax
Table 10: FGR architectures used in experiments.
Dataset
Architecture
CIFAR-10
Group-256, FC-3072, ReLU, FC-10, Softmax
HAPT
Group-104, FC-12173, ReLU, FC-5, Softmax
Diabetes
Croup-44, FC-337, ReLU, FC-3, Softmax
Landsat
Croup-32, FC-1577, ReLU, FC-6, Softmax
MIT-BIH
Croup-160, FC-3444, ReLU, FC-5, Softmax
14

---

Under review as a conference paper at ICLR 2020
C SOFTWARE IMPLEMENTATION
Table 11 presents the list of software dependencies and versions used our implementation. To produce results related to this paper, we used workstation with NVIDIA GeForce RTX-2080Ti GPUs, a 12 core Intel Core i9-7920X processor, and 128 GB memory. Each experiment took between about 30 minutes 72 hours, based on the task and method being tested.
Table 11: Software dependencies.
Dependency
Version
python
3.7.1
pytorch
1.1.0
torchvision
0.2.1
cuda100
1.0
ipython
6.5.0
jupyter
1.0.0
numpy
1.15.4
nni
0.9.1.1
pandas
0.23.4
scikit-learn
0.19.2
scipy
1.1.0
pomegranate
0.11.1
tqdm
4.32.1
matplotlib
3.0.1
15

---

Under review as a conference paper at ICLR 2020
D VISUAL ANALYSIS
In Figure TO we present a visualization of the selected feature for 25 randomly selected groups in our final CIFAR-IO 10 architecture. Red, green, and blue colors indicate which channel is selected for each location. Compared to visualizations that are frequently used for convolutional networks, as GMLP has the flexibility to select pixels at different locations and different color channels, find explicit patterns in this visualization. However, one noticeable pattern is that features selected from a certain color channel usually appear in clusters resembling irregularly shaped patches.
easy to
Figure 10: Visualization of pixels selected by each group for the CIFAR-10 GMLP architecture. Red, green, and blue colors indicate which channel is selected for each location. Due space limitations, 25 random groups out of 1536 total groups visualized here.
16

---

Under review as a conference paper at ICLR 2020
Figure 1 shows the frequency in which each CIFAR-10 location is selected by the GMLP network. From this visualization, GMLP is mostly ignoring the border areas which can be a result of the data augmentation process used to train the network i.c., randomly cropping the center area and padding the margins.
Figure 11: Visualization of pixels selected by the group-select layer for the CIFAR-10 10 GMLP model. Warmer colors represent features that are being selected more frequently.
17


Reviews:
[{"review_id": "ryeMyViJ5H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies supervised classification problems where features are unstructured. For these problems, the authors propose a new neural network architecture that first reorganize the features into groups, then builds feed-forward networks on top each group, and finally aggregate the hidden nodes of each group to produce the final output. Empirical and ablation studies are conducted to show the performance of this approach. \n\nMy detailed comments are as follows. \n\n1. The intuition of this approach should be better explained. In equation (1) the features are group together using a binary matrix. Then the authors suggest using a relaxed version in equation (2) involving softmax function. What is the intuition here? If the feature is very high-dimensional, it seems that the normalization factor in equation (2) might be hard to compute. Moreover, what is the relationship between $k$, $m$, and $d$?\n\n2. It seems that the neural network architecture is a simple variation of the standard MLP, except that the bottom layer is changed to a linear layer $ z = \\Psi x$, where $\\Psi$ is defined using $\\{ \\psi_{ij}, i \\in [km], j\\in [d] \\} $ and a softmax operation. It seems that the contribution of the network structure is rather incremental.\n\n3. In terms of the experiments, it seems that the results are very similar to that of the MLP, although slightly better. Moreover, the datasets used seem small, with no more than 10^6 data points in all datasets. The largest dataset is the Permutation invariant CIFAR-10, which has 50000 data with 3072 features. It would be interesting to see how this method works for high-dimensional datasets where the number of features is large. \n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJeu3xwVoS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper gives a new MLP formulation and architecture that is ostensibly suited for data where the label's dependence on the input feature has form of sparsity. The paper reports the performance of this architecture on a few standard datasets and outperforms other baselines.\n\nReview:\n\nThe main contribution arguably in the \"Group select\" matrix which selects features that each Group-FC focusses on. While it is an interesting idea, it is not hugely novel and requires a lot more demonstration. Without the Group-select the architecture in the paper can simply viewed as an MLP with a block diagonal sparsity structure enforced on the weights and max-out pooling as non-linearity.\n\nBased on the above reasoning, I would rate the architectural/theoretical contribution as not significant. The empirical results also do not look particularly convincing, as there is just a few percentage improvement over MLP (except CIFAR10). \n\nQuestions for authors:\n\n1. In the modified CIFAR10 dtaset, is the same permutation of pixels applied to all images or is it a different random permutation for every? If it is different, the entire rationale of Group-select breaks down, so I am assuming it is the same, but this must be clarified.\n\n2. What were the groups selected in CIFAR10? Did the groups selected correspond to nearby pixels or some other meaningful way? A better understanding of why this improvement happened might help a lot.\n\n3. In figure 5,  learning curves are plotted for different m. But what about k? Is it kept constant? If so what constant? Similarly for figure 6.\n\n4. The visualisation for MNIST Group-select seems very limited compared to what is there in the architecture. I there any way a selected group can be shown and visualised in a meaningful way?\n\n5. The synthetic example is interesting, but this can easily be extended to much larger scales. Do the results hold up in such cases too?\n\nTypos/errors:\n\n1. In equation 4, the i+k/2 part makes sense only for the l=0 layer.\n\n\n\n\n\n\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJgm18Gaqr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the problem of learning expressive feature combinations in order to improve learning for domains where there is no known structure between features. These settings would normally lead to the use of fully-connected MLP networks, which unfortunately have problems with efficient training and generalization after a few layers of depth. The main idea is to use grouping at first, in combination with smaller fully-connected layers for each group, as well as pooling pairs of groups together as the layers go on. Results are shown as comparisons on 5 real-world datasets, and intuitive visualizations on two other datasets. Related work covered MLPs, regularization techniques, sparse networks, random forest models, and other feature grouping. The paper is well written and easy to read. This work did a good job with giving implementation details as well as performing hyperparameter searches and giving the baselines a good effort. \n\nMy current decision is a weak reject, for a well-written paper, but some concerns as follows:\n-The results do not show much improvement (i.e., < 0.3% improvement for 3 of the datasets, and < 1% for another one), aside from CIFAR-10. Considering that the premise of the paper is that MLP’s are not good enough when dealing with data in which the relationships between features are unknown, it seems like these are definitely not good datasets on which to demonstrate this notion of “there has been little progress in deep reinforcement learning for domains without a known structure between features.”\n-The MNIST visualization of group-select felt informative, but the XOR example for grouping visualizations seemed too easy. It would’ve been good to see visualizations or intuitions regarding grouping for harder datasets, in order to be convinced of the need for more expressive feature representations than standard MLP’s.\n-I’m not an expert on causality, but it seems like citations from that area are required for this problem statement of dealing with features where the connections between them are unknown but potentially very important. \n\nLess major:\n-It would have been nice to include related work on other ways to encourage inter-feature interactions, such as perhaps taking the outer product of the input with itself. \n-It seems like different sizes per group would be a more realistic expectation, and that perhaps this should be worked into the algorithm. Similarly, pooling only 2 groups together (from pre-specified positions) seems like it would be limiting as well. It also seems like the algorithm should account for being able to use a high-level feature from one layer as part of multiple groups in the future (i.e. reuse). Even if any of these options don’t make a difference, it would be good to check/evaluate.\n\nMinor:\n-Equation 8 did not fully make sense to me.\n-Why were “random horizontal flips” used as preprocessing for the permutation-invariant CIFAR-10 dataset? This shouldn’t make a difference at all if the pixels become randomly shuffled anyway.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose Group Connected Multilayer Perceptron Networks which allow expressive feature combinations to learn meaningful deep representations. They experiment with different datasets and show that the proposed method gives improved performance. 

The authors have done a commendable job of replying to the queries of the reviewers and addresses many of their concerns. However, the main concern still remains: The improvements are not very significant on most datasets except the MNIST dataset. I understand the author's argument that other papers have also reported small improvements on these datasets and hence it is ok to report small improvements. However, the reviewers and the AC did not find this argument very convincing. Given that this is not a theoretical paper and that the novelty is not very high (as pointed out by R1) strong empirical results are accepted.  Hence, at this point, I recommend that the paper cannot be accepted.


Decision: Reject