Venue: ICLR 2017 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Strengths

- interesting to explore the connection between ReLU DNN and simplified SFNN
- small task (MNIST)  is used to demonstrate the usefulness of the proposed training methods experimentally
- the proposed, multi-stage training methods are simple to implement (despite lacking theoretical rigor)


Weaknesses

-no results are reported on real tasks with large training set

-not clear exploration on the scalability of the learning methods when training data becomes larger

-when the hidden layers become stochastic, the model shares uncertainty representation with deep Bayes networks or deep generative models (Deep Discriminative and Generative Models for Pattern Recognition , book chapter in “Pattern Recognition and Computer Vision”, November 2015, Download PDF). Such connections should be discussed, especially wrt the use of uncertainty representation to benefit pattern recognition (i.e. supervised learning via Bayes rule) and to benefit the use of domain knowledge such as “explaining away”.

-would like to see connections with variational autoencoder models and training, which is also stochastic with hidden layers

review_title: The connection between different models is interesting, except for Bayesian net which is superficial and need to discuss more; MNIST results are interesting but more tasks need to be explored.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper builds connections between DNN, simplified stochastic neural network (SFNN) and SFNN and proposes to use DNN as the initialization model for simplified SFNN. The authors evaluated their model on several small tasks with positive results.

The connection between different models is interesting. I think the connection between sigmoid DNN and Simplified SFNN is the same as mean-field approximation that has been known for decades. However, the connection between ReLU DNN and simplified SFNN is novel.

My main concern is whether the proposed approach is useful when attacking real tasks with large training set. For tasks with small training set I can see that stochastic units would help generalize well.
review_title: interesting connection between DNN and simplified SFNN but its practical significance is unknown

final_score: rating: null
review_content: Can you compare the generative nature of Stochastic neural nets discussed in this paper with its counterpart in Generative Deep Bayesian Network(e.g., A multimodal variational approach to learning and inference in switching state space models, IEEE-ICASSP, 2004)? What are the commonality and differences between these two classes of deep generative models from both theoretical and practical standpoints? 
review_title: Generative nature of Stochastic DNN vs. Generative Deep Bayesian Network

Meta-review: No reviewer was willing to champion the paper and the authors did not adequately address reviewer comments in a revision. Recommend rejection.
Decision: Reject