Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper tries to study the effect of dropout by studying the trajectory of optimization via dropout. It is claimed that dropout can help find flatter minimizer.
main_review: I am not quite familiar with this area. But I think the topic is quite interesting. 

The main concerns are:

1. It seems most of the claims are not theoretically supported. I tried to find something in the supplementary material but only code is found.
2. It is observed that regularization also helps and find flatter minimizer and there is work to bridge the dropout and reguarliation: Dropout Training as Adaptive Regularization. More discussion is encouraged.

summary_of_the_review: See main review.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper uses the tools from different prior works to study the flatness brought by dropout. The paper shows that the noise induced by the dropout has a similar structured introduced by SGD that leads the training to find flatter minima.
main_review: In general, the paper is interesting; however, the novelty and contribution are limited. Most of the tools are from the existing work, and neither insights nor empirical results are strong enough to make this paper stands out.

The paper may miss some important discussions regarding the relationship between dropout, regularization, and generalization bounds, as discussed in [1, 2, 3]. Adding the discussion and extra experiments would increase the value of the current draft.

The paper divides the training process into two phases, i.e., fast convergence and exploration phase. However, this phase division is relatively rough and has no support (authors are encouraged to cite some prior works to support their justification).

The computation method for H on neural networks has never been introduced in the main text, making the current empirical results questionable. The computation cost of $H_i$ remains unclear; this comment also applies to the quality of hessian estimation, which may significantly impact the final numerical observation.

It is a bit hard to identify the (inverse) relation between variance and the flatness in Figure 2 and Figure 3a, unlike the clear pattern in Figure 3b. It would be great if the authors could polish the figures and/or explain the current observation.

## Reference
1. Dropout Training, Data-dependent Regularization, and Generalization Bounds
2. On Convergence and Generalization of Dropout Training
3. A PAC-Bayesian Tutorial with A Dropout Bound
summary_of_the_review: In general the paper is interesting; however, the novelty and contribution is limited. Most of tools are from the existing work, and neither insights nor empirical results are strong enough to make this paper stands out.
Adding the discussion related to the generalization bound (as pointed in the provided literature) and extra experiments would increase the value of the current draft.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper provides an empirical study on how the dropout can lead to minima within a flatter landscape (hence a better generalization performance). Based on the definition of minimizer flatness (i.e., eq. 5) and random trajectory data (parameters from a optimization path and gradients when loss is stably small), the authors first show that dropout can help to find flatter minima, and also find a inverse relation between the flatness (the length $F_v$) and the algorithmic variance (i.e., in area with sharper direction, algorithm with dropout has a larger variance), and further find that for areas with larger eigenvalues (i.e., with sharper landscape), the variances of the algorithm along the directions of their eigenvectors are also large. Such empirical results together demonstrate that the dropout can play a good role in improving the generalization of existing algorithms by introducing certain level of noise. 
main_review: My detailed comments are given as below. 

Strength: 
1. This paper is well written and the motivation is clear to me. Dropout is a widely used technique for improving the generalization performance of various learning algorithms, but there is limited understanding how this scheme helps to achieve this better generalization. This paper provides some empirical justification showing that dropout can lead to flatter minima. 

Weakness:
1. Some parts need to be explained more clearly. For example, I have a hard time understanding Section 3.3., where interval flatness is defined as the width of the region where $L_v(\delta \theta)<2L_v(0)$. I am wondering why $2L_v(0)$ is chosen here, and whether the factor $2$ is sufficiently general to characterize the landscapes for all settings. I am not familiar with this part, but I hope the authors should explain it a little bit.  

2. My second concern is the novelty of the results. So far I see that some important tools, e.g., flatness definition, PCA, follow from existing studies, although there is some new treatment in obtain the variance. In addition, the empirical results are not that surprising to me (but I agree that it is not clear why the noise induced by dropout plays some similar role as SGD), and a more interesting part may be to provide some theoretical justification even in some simplified settings. 

3. Experiments can be strengthened. I see that all experiments are conducted over small datasets (e.g., CIFAR-10) and small models. To make a more convincing conclusion that dropout can lead to flatter minima, I believe experiments on modern datasets and models (e.g., ResNet on ImageNet) need to be run. 






summary_of_the_review: I think this paper provides some empirical explanation on why dropout is useful to achieve a flatter minima. However,  I feel it is more interesting to rigorously explain this phenomenon given that many works already prove that SGD can reach flatter minima. In addition, the novelty of this paper is not that high based on my understanding. Therefore, I am slightly negative about this paper, but I am open to adjust my score based on the authors' feedback and other reviewers' comments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper empirically found that Dropout noise could help the neural network search for the flat minima, by showing that the covariance of parameter and gradient align well with the loss curvature and have inverse relationship with interval flatness. 
main_review: 
To my best knowledge, this is a first work that shows the relationship between Dropout and the flatness of minima. It could provide another perspective for understanding the effectiveness of Dropout. 

However, my main concern about this manuscript is the complete missing of theoretical analysis, which makes the current version of the paper not strong enough for publication. I strongly suggest the authors conduct theoretical analysis over the covariance introduced by Dropout noise and loss curvature, at least on linear cases. 

summary_of_the_review: Complete missing of theoretical analysis.
review_title: null

Meta-review: The authors make an experimental case that dropout aids generalization by
promoting "flatter minima".

The reviewers felt that the work reported in this paper makes a useful step
forward on a question of central interest.  The consensus view was that the
total weight of evidence presented was not sufficient for publication in
ICLR.  The paper could be strengthened was more extensive and varied experiments
and/or theoretical analysis.
Decision: Reject