Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Considering that the GELU introduces extra hyper-parameters (\mu and \sigma), 
do the authors think it is a fair to compare against ReLU which is merely a 
special case of GELU where \mu and \sigma are both zero? 
review_title: hyper-parameters

final_score: rating: 5: Marginally below acceptance threshold
review_content: The method proposed essential trains neural networks without a traditional nonlinearity, using multiplicative gating by the CDF of a Gaussian evaluated at the preactivation; this is motivated as a relaxation of a probit-Bernoulli stochastic gate. Experiments are performed with both.

The work is somewhat novel and interesting. Little is said about why this is preferable to other similar parameterizations of the same (sigmoidal? softsign? etc.) It would be stronger with more empirical interrogation of why this works and exploration of the nearby conceptual space. The CIFAR results look okay by today's standards but the MNIST results are quite bad, neural nets were doing better than 1.5% a decade ago and the SOI map results (and the ReLU baseline) are above 2%. (TIMIT results on frame classification also aren't that interesting without evaluating word error rate within a speech pipeline, but this is a minor point.)

The idea put forth that SOI map networks without additional nonlinearities are comparable to linear functions is rather misleading as they are, in expectation, nonlinear functions of their input. Varying an input example by multiplying or adding a constant will not be linearly reflected in the expected output of the network. In this sense they are more nonlinear than ReLU networks which are at least locally linear.

The plots are very difficult to read in grayscale,
review_title: Official review.

final_score: rating: null
review_content: - Your manuscript relies exclusively on baselines you've run yourself. Can you please elaborate on how your method compares to results already presented in the literature on your tasks of interest? 
- Can you explain what it is you're trying to illustrate with the loss curves? These account for 4 of 7 figures but it's not clear why they're being displayed, or why they convey more information than a table of final results at convergence.
- Can you contrast your method with other work on adaptive dropout masks, e.g. Ba et al's "standout"?
review_title: Architectural choices and comparisons to the literature

final_score: rating: null
review_content: Can the authors please elaborate how the presented approach is different from works like Adaptive dropout?

While ReLU has a fixed threshold at 0 the presented approach places the threshold randomly around zero. This might result in effect similar to decreasing noise levels over training epochs due to saturation of activations, a concept previously proposed in the literature ("Annealed dropout training of deep networks"). Can you please comment on this?  

I also have similar questions as that of AnonReviewer1.
review_title: comparison to literature 

final_score: rating: 4: Ok but not good enough - rejection
review_content: The proposed regularizer seems to be a particular combination of existing methods. Though the implied connection between nonlinearities and stochastic regularizers is intriguing, in my opinion the empirical performance does not exceed the performance achieved by similar methods by a large enough margin to arrive at a meaningful conclusion. 
review_title: Minor variant on existing regularization methods

final_score: rating: 5: Marginally below acceptance threshold
review_content: Approaches like adaptive dropout also have the binary mask as a function of input to a neuron very similar to the proposed approach. It is not clear, even from the new draft, how the proposed approach differs to Adaptive dropout in terms of functionality. The experimental validation is also not extensive since comparison to SOTA is not included. 
review_title: The proposed approach seems similar to other existing approaches in literature (eg. adaptive dropout). Experimental validation not adequate for evaluation.

Meta-review: The reviewers unanimously recommend rejecting the paper.
Decision: Reject