Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary:

This paper gives a new MLP formulation and architecture that is ostensibly suited for data where the label's dependence on the input feature has form of sparsity. The paper reports the performance of this architecture on a few standard datasets and outperforms other baselines.

Review:

The main contribution arguably in the "Group select" matrix which selects features that each Group-FC focusses on. While it is an interesting idea, it is not hugely novel and requires a lot more demonstration. Without the Group-select the architecture in the paper can simply viewed as an MLP with a block diagonal sparsity structure enforced on the weights and max-out pooling as non-linearity.

Based on the above reasoning, I would rate the architectural/theoretical contribution as not significant. The empirical results also do not look particularly convincing, as there is just a few percentage improvement over MLP (except CIFAR10). 

Questions for authors:

1. In the modified CIFAR10 dtaset, is the same permutation of pixels applied to all images or is it a different random permutation for every? If it is different, the entire rationale of Group-select breaks down, so I am assuming it is the same, but this must be clarified.

2. What were the groups selected in CIFAR10? Did the groups selected correspond to nearby pixels or some other meaningful way? A better understanding of why this improvement happened might help a lot.

3. In figure 5,  learning curves are plotted for different m. But what about k? Is it kept constant? If so what constant? Similarly for figure 6.

4. The visualisation for MNIST Group-select seems very limited compared to what is there in the architecture. I there any way a selected group can be shown and visualised in a meaningful way?

5. The synthetic example is interesting, but this can easily be extended to much larger scales. Do the results hold up in such cases too?

Typos/errors:

1. In equation 4, the i+k/2 part makes sense only for the l=0 layer.









review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper addresses the problem of learning expressive feature combinations in order to improve learning for domains where there is no known structure between features. These settings would normally lead to the use of fully-connected MLP networks, which unfortunately have problems with efficient training and generalization after a few layers of depth. The main idea is to use grouping at first, in combination with smaller fully-connected layers for each group, as well as pooling pairs of groups together as the layers go on. Results are shown as comparisons on 5 real-world datasets, and intuitive visualizations on two other datasets. Related work covered MLPs, regularization techniques, sparse networks, random forest models, and other feature grouping. The paper is well written and easy to read. This work did a good job with giving implementation details as well as performing hyperparameter searches and giving the baselines a good effort. 

My current decision is a weak reject, for a well-written paper, but some concerns as follows:
-The results do not show much improvement (i.e., < 0.3% improvement for 3 of the datasets, and < 1% for another one), aside from CIFAR-10. Considering that the premise of the paper is that MLP’s are not good enough when dealing with data in which the relationships between features are unknown, it seems like these are definitely not good datasets on which to demonstrate this notion of “there has been little progress in deep reinforcement learning for domains without a known structure between features.”
-The MNIST visualization of group-select felt informative, but the XOR example for grouping visualizations seemed too easy. It would’ve been good to see visualizations or intuitions regarding grouping for harder datasets, in order to be convinced of the need for more expressive feature representations than standard MLP’s.
-I’m not an expert on causality, but it seems like citations from that area are required for this problem statement of dealing with features where the connections between them are unknown but potentially very important. 

Less major:
-It would have been nice to include related work on other ways to encourage inter-feature interactions, such as perhaps taking the outer product of the input with itself. 
-It seems like different sizes per group would be a more realistic expectation, and that perhaps this should be worked into the algorithm. Similarly, pooling only 2 groups together (from pre-specified positions) seems like it would be limiting as well. It also seems like the algorithm should account for being able to use a high-level feature from one layer as part of multiple groups in the future (i.e. reuse). Even if any of these options don’t make a difference, it would be good to check/evaluate.

Minor:
-Equation 8 did not fully make sense to me.
-Why were “random horizontal flips” used as preprocessing for the permutation-invariant CIFAR-10 dataset? This shouldn’t make a difference at all if the pixels become randomly shuffled anyway.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper studies supervised classification problems where features are unstructured. For these problems, the authors propose a new neural network architecture that first reorganize the features into groups, then builds feed-forward networks on top each group, and finally aggregate the hidden nodes of each group to produce the final output. Empirical and ablation studies are conducted to show the performance of this approach. 

My detailed comments are as follows. 

1. The intuition of this approach should be better explained. In equation (1) the features are group together using a binary matrix. Then the authors suggest using a relaxed version in equation (2) involving softmax function. What is the intuition here? If the feature is very high-dimensional, it seems that the normalization factor in equation (2) might be hard to compute. Moreover, what is the relationship between $k$, $m$, and $d$?

2. It seems that the neural network architecture is a simple variation of the standard MLP, except that the bottom layer is changed to a linear layer $ z = \Psi x$, where $\Psi$ is defined using $\{ \psi_{ij}, i \in [km], j\in [d] \} $ and a softmax operation. It seems that the contribution of the network structure is rather incremental.

3. In terms of the experiments, it seems that the results are very similar to that of the MLP, although slightly better. Moreover, the datasets used seem small, with no more than 10^6 data points in all datasets. The largest dataset is the Permutation invariant CIFAR-10, which has 50000 data with 3072 features. It would be interesting to see how this method works for high-dimensional datasets where the number of features is large. 


review_title: Official Blind Review #1

Meta-review: The authors propose Group Connected Multilayer Perceptron Networks which allow expressive feature combinations to learn meaningful deep representations. They experiment with different datasets and show that the proposed method gives improved performance. 

The authors have done a commendable job of replying to the queries of the reviewers and addresses many of their concerns. However, the main concern still remains: The improvements are not very significant on most datasets except the MNIST dataset. I understand the author's argument that other papers have also reported small improvements on these datasets and hence it is ok to report small improvements. However, the reviewers and the AC did not find this argument very convincing. Given that this is not a theoretical paper and that the novelty is not very high (as pointed out by R1) strong empirical results are accepted.  Hence, at this point, I recommend that the paper cannot be accepted.


Decision: Reject