Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ==Main comments

The authors connect dropout parameters to a bound of the Rademacher complexity (Rad) of the network. While it is great to see deep learning techniques inspired by learning theory, I think the paper makes too many leaps and the Rad story is ultimately unconvincing.  Perhaps it is better to start with the resulting regularizer, and the interesting direct optimization of dropout parameters. In its current form, the following leaps problematic and were not addressed in the paper:

1) Why is is adding Rad as a regularizer reasonable? Rad is usually hard to compute, and most useful for bounding the generalization error. It would be interesting if it also turns out to be a good regularizer, but the authors do not say why nor cite anything. Like the VC dimension, Rad itself depends on the model class, and cannot be directly optimized. Even if you can somehow optimize over the model class, these quantities give very loose bounds, and do not equal to generalization error. For example, I feel even just adding the actual generalization error bound is more natural. Would it make sense to just add Rad to the objective in this way for a linear model?

2) Why is it reasonable to go from a regularizer based on RC to a loose bound of Rad? The actual resulting regularizer turns out to be a weight penalty but this seems to be a rather loose bound that might not have too much to do with Rad anymore. There should be some analysis on how loose this bound is, and if this looseness matter at all. 

The empirical results themselves seem reasonable, but the results are not actually better than simpler methods in the corresponding tasks, the interpretation is less confident. Afterall, it seems that the proposed method had several parameters that were turned, where the analogous parameters are not present in the competing methods. And the per unit dropout rates are themselves additional parameters, but are they actually good use of parameters?

==Minor comments

The optimization is perhaps also not quite right, since this requires taking the gradient of the dropout parameter in the original objective. While the authors point out that one can use the mean, but that is more problematic for the gradient than for normal forward predictions. The gradient used for regular learning is not based on the mean prediction, but rather the samples.

tiny columns surrounding figures are ugly and hard to read

review_title: does not actual say why using Rademacher as a regularizer is theoretically justified, and why the loose bound is reasonable

final_score: rating: 7: Good paper, accept
review_content: An important contribution. The paper is well written. Some questions that needs to be better answered are listed here.
1. The theorem is difficult to decipher. Some remarks needs to be included explaining the terms on the right and what they mean with respect to learnability or complexity. 
2. How does the regularization term in eq (2) relate to the existing (currently used) norm based regularizers in deep network learning? It may be straight forward but some small simulation/plots explaining this is important. 
3. Apart from the accuracy results, the change in computational time for working with eq (2), rather than using existing state-of-the-art deep network optimization needs to be reported? How does this change vary with respect to dataset and network size (beyond the description of scaled regularization in section 4)?
4. Confidence intervals needs to be computed for the retain-rates (reported as a function of epoch). This is critical both to evaluate the stability of regularizers as well as whether the bound from theorem is strong. 
5. Did the evaluations show some patterns on the retain rates across different layers? It seems from Figure 3,4 that retain rates in lower layers are more closer to 1 and they decrease to 0.5 as depth increases. Is this a general pattern? 
6. It has been long known that dropout relates to non-negative weighted averaging of partially learned neural networks and dropout rate of 0.5 provides best dymanics. The evaluations say that clearly 0.5 for all units/layers us not correct. What does this mean in terms of network architecture? Is it that some layers are easy to average (nothing is learned there, so dropped networks have small variance), while some other layers are sensitive? 
7. What are some simple guidelines for choosing the values of p and q? Again it appears p=q=2 is the best, but need confidence intervals here to say anything substantial. 
review_title: This is an important piece of work that relates complexity of networks' learnability to dropout rates in backpropagation. This paper answers some critical questions about dropout learning.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies the adjustment of dropout rates which is a useful tool to prevent the overfitting of deep neural networks. The authors derive a generalization error bound in terms of dropout rates. Based on this, the authors propose a regularization framework to adaptively select dropout rates. Experimental results are also given to verify the theory.

Major comments:
(1) The Empirical Rademacher complexity is not defined. For completeness, it would be better to define it at least in the appendix.
(2) I can not follow the inequality (5). Especially, according to the main text, f^L is a vector-valued function . Therefore, it is not clear to me the meaning of \sum\sigma_if^L(x_i,w) in (5).
(3) I can also not see clearly the third equality in (9). Note that f^l is a vector-valued function. It is not clear to me how it is related to a summation over j there.
(4) There is a linear dependency on the number of classes in Theorem 3.1. Is it possible to further improve this dependency?

Minor comments:
(1) Section 4: 1e-3,1e-4,1e-5 is not consistent with 1e^{-3}, 1e^{-4},1e^{-5}
(2) Abstract: there should be a space before "Experiments".
(3) It would be better to give more details (e.g., page, section) in citing a book in the proof of Theorem 3.1

Summary:
The mathematical analysis in the present version is not rigorous. The authors should improve the mathematical analysis.

----------------------------
After Rebuttal:
Thank you for revising the paper. I think there are still some possible problems. 
Let us consider eq (12) in the appendix on the contraction property of Rademacher complexity (RC).
(1) Since you consider a variant of RC with absolute value inside the supermum, to my best knowledge, the contraction property (12) should involve an additional factor of 2, see, e.g., Theorem 12 of "Rademacher and Gaussian Complexities: Risk Bounds and Structural Results" by Bartlett and Mendelson. Since you need to apply this contraction property L times, there should be a factor of 2^L in the error bound. This make the bound not appealing for neural networks with a moderate L.
(2) Second, the function g involves an expectation w.r.t. r before the activation function. I am not sure whether this existence of expectation w.r.t. r would make the contraction property applicable in this case.
review_title: mathematical analysis seems not sound

Decision: Accept (Poster)