Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper introduces a skip-connection based design of fully connected networks, which is loosely based on learning latent variable tree structure learning via mutual information criteria. The goal is to learn sparse structures across layers of fully connected networks.  Compared to prior work (hierarchical latent tree model), this work introduces skip-paths. 
Authors refer to prior work for methods to learn this backbone model. Liu et.al (http://www.cse.ust.hk/~lzhang/ltm/index.htm) and Chen et.al. (https://arxiv.org/abs/1508.00973) and (https://arxiv.org/pdf/1605.06650.pdf). 

As far as I understand, the methods for learning backbone structure and the skip-path are performed independently, i.e. there is no end-to-end training of the structure and parameters of the layers. This will limit the applicability of the approach in most applications where fully connected networks are currently used. 

Originality - The paper heavily builds upon prior work on hierarchical latent tree analysis and adds 'skip path' formulation to the architecture, however the structure learning is not performed end-to-end and in conjunction with the parameters. 

Clarity - The paper is not self-contained in terms of methodology.

Quality and Significance - There is a disconnect between premise of the paper (improving efficiency of fully connected layers by learning sparser structures) and applicability of the approach (slow EM based method to learn structure first, then learn the parameters).  As is, the applicability of the method is limited. 
Also in terms of experiments, there is not enough exploration of simpler sparse learning methods such as heavy regularization of the weights. 
review_title: Learning Parsimonious Deep Feed-forward Networks

final_score: rating: 5: Marginally below acceptance threshold
review_content: There is a vast literature on structure learning for constructing neural networks (topologies, layers, learning rates, etc.) in an automatic fashion. Your work falls under a similar category. I am a bit surprised that you have not discussed it in the paper not to mention provided a baseline to compare your method to. Also, without knowing intricate details about each of 17 tasks you mentioned it is really hard to make any judgement as to how significant is improvement coming from your approach. There has been some work done on constructing interpretable neural networks, such as stimulated training in speech recognition, unfortunately these are not discussed in the paper despite interpretability being considered important in this paper. 
review_title: Learning Parsimonious Deep Feed-forward Networks

final_score: rating: 5: Marginally below acceptance threshold
review_content: The main strengths of the paper are the supporting experimental results in comparison to plain feed-forward networks (FNNs).  The proposed method is focused on discovering sparse neural networks.  The experiments show that sparsity is achieved and still the discovered sparse networks have comparable or better performance compared to dense networks.

The main weakness of the paper is lack of cohesion in contributions and difficulty in delineating the scope of their proposed approach.

Below are some suggestions for improving the paper:

Can you enumerate the paper’s contributions and specify the scope of this work?  Where is this method most applicable and where is it not applicable?

Why is the paper focused on these specific contributions?  What problem does this particular set of contributions solve that is not solvable by the baselines?  There needs to be a cohesive story that puts the elements together.  For example, you explain how the algorithm for creating the backbone can use unsupervised data.  On the other hand, to distinguish this work from the baselines you mention that this work is the first to apply the method to supervised learning problems.

The motivation section in the beginning of the paper motivates using the backbone structure to get a sparse network.  However, it does not adequately motivate the skip-path connections or applications of the method to supervised tasks.

Is this work extending the applicability of baselines to new types of problems?  Or is this work focused on improving the performance of existing methods?  Answers to these questions can automatically determine suitable experiments to run as well.  It's not clear if Pruned FNNs are the most suitable baseline for evaluating the results.  Can your work be compared experimentally with any of the constructive methods from the related work section?  If not, why?

When contrasting this work with existing approaches, can you explain how existing work builds toward the same solution that you are focusing on?  It would be more informative to explain how the baselines contribute to the solution instead of just citing them and highlighting their differences.

Regarding the experimental results, is there any insight on why the dense networks are falling short?  For example, if it is due to overfitting, is there a correlation between performance and size of FNNs?  Do you observe a similar performance vs FNNs in existing methods?  Whether this good performance is due to your contributions or due to effectiveness of the baseline algorithm, proper analysis and discussion is required and counts as useful research contribution.

review_title: Needs improvement

Meta-review: I am inclined to agree with R1 that there is an extensive literature on learning architectures now, and I have seen two others as part of my area chairing. This paper does not offer comparisons to existing methods for architecture learning other than very basic ones and that reduces the strength of the paper significantly. Further the broad exploration over 17 tasks is more overwhelming, than adding to an insight into the methods.
Decision: Reject