Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: For linear regression, the paper shows that several popular heuristics used in deep learning correspond to regularized empirical risk minimization with sub-quadratic penalties, thereby, arguing that these methods indeed induce sparsity. The analysis leverages the so-called \eta-trick, where the penalty is replaced with its variational (minimization) form. Empirical results are also provided to support the theory.

main_review: One of the strengths of the paper is that it unifies several popular heuristics, including variational dropout, standout, magnitude pruning, etc. The claims are rigorously proved and the empirical evidence validates some of the claims.

The main limitation of the paper is that \eta-trick itself is limited: it can be applied only when a weighted-sum-of-squared of parameters is present in the objective. Dropout in linear regression with l2-loss is known to induce such a penalty, which is leveraged here to make connections to several adaptive dropout scenarios. However, I'm not sure if the techniques can be generalized to analyze even simple 2-layer linear networks. The equivalent form of equation (4) has been studied for deeper linear models (2-layer and beyond). Given the more complicated form of the penalty in such models, do the authors hope this approach to generalize? Another limitation is that the paper merely focuses on the duality in the optimization domain. There is no statistical or computational analysis comparing either of the two forms (adaptive dropout or the dual IRLS). Is there a benefit in doing one over the other? 

Although I’ve read several papers on the topic, following the paper was difficult at times. I think the current presentation makes it challenging for the average reader to understand the main results and constructions, and distinguish them from less important, tangential results. I suggest that the authors list the main contributions of the paper in the introduction. The duality between different penalties and their \eta-trick-representations in table 1 is interesting, but the description of their connection in sections 2.2 and 3 can be improved, in my opinion. The motivation behind studying adaptive dropout updates in section 3 is not clear at all. What is the merit of updates in equations (8) and (9), except for “facilitating your analysis” (as stated in line 123)?  Also, it is not clear how the “standardization” assumption affects the results and techniques. The authors suggest that “it is inexpensive to satisfy in practice, but our analysis can be extended to the general case”. However, they do not say if/how the main takeaways from the paper will be affected if the data is non-standard.

typo: line 20: trad-eoff
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper examines stochastic perturbation methods, such as dropout, through the lens of the "eta-trick".  This trick allows the optimization objective to be re-written as a joint objective in terms of the original parameters and auxiliary parameters eta.  This new objective consists of the original loss in addition to a quadratic penalty in the parameters and a formulation-specific function of eta.  The paper shows that any adaptive dropout method that updates the dropout parameters monotonically as a function of the weight magnitudes is equivalent to regularized risk minimization with a subquadratic penalty, which is known to induce sparsity.  The paper analyses standout, variational dropout, L0 regularization, and magnitude pruning from this perspective, deriving the corresponding penalty term (Table 2) and allowing for comparison.  The paper then experimentally compares variants of variational dropout, finding that performance is well-described by the eta-trick reformulation since stochastic variants suffer from the introduced noise.  
main_review: I found this paper well-written, interesting, and novel.  I had not seen the eta-trick before, and I found its use to compare seemingly disparate regularization methods to be fascinating.  Moreover, I like how the experiment isolates the different variants of variational dropout and compares them to the eta-trick-derived formulation.  I have no criticisms at this time; I find the paper ready for publication.
limitations_and_societal_impact: Yes, I found limitations and societal impact to have been addressed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper draws connections between adaptive dropout and regularized optimization using the $\eta$-trick. The authors show that various adaptive dropout methods can be reinterpreted as sparsity-induced regularization for linear regression models.

main_review: 
I think this work is solid and sound. The main contribution of this paper is that the authors provide a unified framework for analyzing the existing adaptive dropout methods and thus provide theoretical justifications for them. The analysis explains the underlying mechanisms of some adaptive dropout methods, which can potentially contribute to the research in developing strategies for sparse training of neural networks.


1. One concern is that the adaptive dropout methods are not so commonly used in practice (e.g., for training neural networks) and are also more complicated than the Bernoulli dropout. However, it violates the monotonic assumption and thus doesn't fit in your framework. 


2. Also, the current analysis is limited to linear regression problems. It may not hold for linear classification problems, for which the closed-form solution may not exist and hence add extra difficulty for the analysis. However, classification problems are more common. Do you think similar conclusions can still be made for linear classification problems with adaptive dropout?


3. In Figure 2, you showed similar behaviors of variational dropout with its dual formulation in terms of the fraction of nonzero parameters and validation error. I wonder if they induce similar sparsity patterns (e.g., the fraction of nonzero parameters at each layer)?


Overall, I am positive about this paper as it provides a unified framework for analyzing adaptive dropout methods with mild assumptions. It provides theoretical justifications for these methods, as some of them are proposed based on heuristics. Therefore, I vote for acceptance.

====Post rebuttal=======

Thanks for the response. My questions & concerns were addressed by the response. I would suggest the authors include the additional empirical results in the revised paper. Though the theory is for linear cases, if the authors can empirically demonstrate that the theory aligns well with non-linear systems, such as neural networks, I think this will make the paper stronger. 
limitations_and_societal_impact: The limitation was discussed, and the negative societal impact was not. I did not see any negative society impact neither.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes to use a dual formulation that majorizes the original dropout regularization, and rewrite the original dropout objective as a joint optimization on both the parameter and the auxiliary variable. The authors further show in the linear case any dropout algorithm, if it adapts to the magnitude of the weights monotonically, is equivalent to a regularized ERM with a subquadratic regularizer, which induces sparsity. 

main_review: It is nice to build a connection between the adaptive dropout strategy and sparse solution from the perspective of joint optimization and dual formulation. In this way, the dropout strategies are better understood in terms of the regularizers in the objective. It is a step forward towards understanding the effect of dropout. 

On the other hand, the analysis of the draft is mostly limited to the linear loss. For deep models, due to its non-linearity, the impact of dropout may not be captured fully as a regularizer in the objective. For example, it is still unclear to me how dropout may interact with the optimization algorithms during the training. 

A follow up question: is there any particular reason the authors used the non-linear LeNet model in the experiment? Feels like the analysis is mostly about linear models. 

limitations_and_societal_impact: the authors adequately addressed the limitations and potential negative societal impact of their work
review_title: null

Meta-review: The paper shows that popular heuristics in deep learning, e.g. dropout, in the context of linear regression amount to regularizing the empirical objective with sub-quadratic penalties which promote sparsity. The paper introduces interesting tools that may be interesting outside the scope of the paper. Overall, a good paper. 
Decision: Accept (Poster)