The Flip Side of the Reweighted Coin: Duality of Adaptive Dropout and Regularization
Authors: Daniel LeJeune
Hamid Javadi
Richard Baraniuk
Venue: NeurIPS 2021 Poster (2021)
Keywords: sparsity
deep learning
dropout
regularization
adaptive

Abstract: Among the most successful methods for sparsifying deep (neural) networks are those that adaptively mask the network weights throughout training. By examining this masking, or dropout, in the linear case, we uncover a duality between such adaptive methods and regularization through the so-called “η-trick” that casts both as iteratively reweighted optimizations. We show that any dropout strategy that adapts to the weights in a monotonic way corresponds to an effective subquadratic regularization penalty, and therefore leads to sparse solutions. We obtain the effective penalties for several popular sparsification strategies, which are remarkably similar to classical penalties commonly used in sparse optimization. Considering variational dropout as a case study, we demonstrate similar empirical behavior between the adaptive dropout method and classical methods on the task of deep network sparsification, validating our theory.

The Flip Side of the Reweighted Coin: Duality of Adaptive Dropout and Regularization
Daniel LeJeune Department of Electrical and Computer Engineering Rice University Houston, TX 77005 dle ilejeune@rice. edu
Hamid Javadi Department of Electrical and Computer Engineering Rice University Houston, TX 77005 hh35@rice. edu
Richard G. Baraniuk
Department of Electrical and Computer Engineering
Rice University Houston, TX 77005 richberice.edt edu
Abstract
Among the most successful methods for sparsifying deep (neural) networks are those that adaptively mask the network weights throughout training. By examining this masking, or dropout, in the linear case, we uncover duality between such adaptive methods and regularization through the so-called "ny-trcck" that casts both iteratively reweighted optimizations. We show that any dropout strategy that adapts to the weights in monotonic way corresponds to an effective subquadratic regularization penalty, and therefore leads to sparse solutions. We obtain the effec- tive penalties for several popular sparsification strategies, which are remarkably similar to classical penalties commonly used in sparse optimization. Considering variational dropout as a case study, we demonstrate similar empirical behavior between the adaptive dropout method and classical methods the task of deep network sparsification, validating our theory.
Introduction
In machine learning, often valuable for models to be parsimonious or sparse for a variety of reasons, from memory savings and computational speedups to model interpretability and general- izability. Classically, this is achieved by solving a regularized empirical risk minimization (ERM) problem of the form
minimize
L(w) + XQ(w),
(1)
where the data-dependent empirical risk and Q is a sparsity-inducing regularization penalty. Ideally, 9((w) ww) is equal to ||wla := # (i: w; * 0} or function of ||w] [6]. in which case an appropriately chosen X balances the trade-off between the suboptimality of L and the sparsity of
the solution. However, the use of |w directly as penalty is difficult, as it makes local search impossible, so alternative approaches are necessary for high-dimensional problems. The classical solution to this is to relax the problem using smooth surrogate Ω that approximates |wll 18)
When we know the regularization penalty Ω2 we can understand the types of solutions they encour- age; however, many popular methods for sparsifying deep (neural) networks do not appear to fit
the form (T). For example. variational dropout [27 32,, "Lo regularization" 22 and magnitude pruning [5 are the only methods compared in recent large-scale study by Gale et al. [15], and all achieve sparsity by adaptively masking or sealing the parameters of the network while optimizing the loss function in w. Although these methods are well-motivated heuristically, it is unclear how to make a principled choice between them aside from treating them as black boxes and comparing their empirical performance. However, if these methods could be cast as solutions to regularized ERM problem, then we could compare them on the basis of their regularization penalties.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Scalar Penalty A(2(w)
Vector Penalty XS2( w)
2.5
Dropout Methods
6
VARDROP
2.0
STANDOUT
HARDCONCRETE
1.5
HARD HARDTHRRSSI(1))
2
Classical Penaltics
0.5
LoGSUM(2). À
MCP(5.v2)
0.0
0
|w]
# Nonzero Coefficients of w
Figure I: Sparsity-inducing behavior in regularized ERM can be understond by considering the properties of the regularization penalty. Our analysis enables the computation and comparison of the effective regularization penalties of the adaptive dropout sparsity methods in Tahle 2 (We arbitrarily set = unless otherwise specified.) Both 'ARIATIONALDROPO strong resemblance to classical penalties. Left: Separable penalties (excluding HARD as function of weight magnitude. Right: Penalty of a unit-norm k-sparse _2d defined by wy J & k).
[32] and HARDCONCRETE (called "Ėo regularization") [2 bear
We show that, in fact, these sparsity methods do correspond to regularization penalties Ω which we can obtain, compute, and compare. As we show in Figure [ these penalties bear striking resemblance to classical pearsityiinducing penalties such as the LOGSUM ( penalty and the minimax concave penalty (MCP) [4]. More broadly. we analyze adaptive dropout methods, which apply the dropout technique of Srivastava al. (35] with adaptive parameters. By considering adaptive dropout in the linear setting, we uncover a duality between adaptive dropout methods and regularized ERM. We make this connection via the "ny-trick" [4], tool with a long history of application in sparse optimization via iteratively reweighted least squares (IRLS) (see the history presented by Daubechies al.
We prove that all adaptive dropout methods whose amount of dropout varies monotonically with the magnitudes of the parameters induce effective subquadratic (and hence sparsity-inuccing (5) penalty Q. This further supports the experimental evidence that such methods excel at inducing sparsity. We also demonstrate how to use our result determine the effective penalty for adaptive dropout methods, using as examples the sparsity methods listed above as well as the standout method (3). which has an effective penalty on the layer activations, rather than the parameters, explaining why the standout method sparsifies activations. We then numerically compute the effective penalties and plot them together in Figure
We validate our new theory by applying variational dropout on the task of deep network sparsification, and we show that the performance is similar to non-dropout methods based on the same effective penalty. This suggests that not only is considering the effective penalty a sound means for comparing sparsity methods, but also that classical regularized ERM itself an effective means for inducing sparsity in deep networks and serves as valuable tool for future work on deep network sparsity.
For theoreticians, this work provides general framework for obtaining the effective regularization penalty for adaptive dropout algorithms, enabling regularized ERM analysis tools to be applied to these methods. For practitioners, this work enables the application of classical regularization intuition when choosing between sparsifying approaches. For methods developers, this work provides a strong baseline against which new adaptive dropout methods should be compared: the effective penalty Z.
2 Background
Notation. We denote the extended reals and the non-negative reals = Ru (0) and R. = [0, 20), respectively. The operator diag (-) takes a vector and returns a matrix with that vector along the diagonal entries or takes matrix and returns the same matrix with all non-diagonal entries set zero. The matrix J denotes a matrix of all ones. We denote element- wise multiplication and exponentiation by O: i.e., [u O v), = Uyt and [US'] bi U. Division of vectors and scalar functions of vectors denote element wise operations. Order statisties of the magnitudes of
'Our code is available at atttps::/githubb http::////httttboooa////////d/tt

---

elements in a vector u E I' are denoted by uGi such that |u()) |122) > (onll/...//u///
...///))--
The function
ar: IR [0, denotes the sigmoid (t)) = (e(++e
2.1 Dropout Tikhonov Regularization
We can understand adaptive dropout methods by breaking them down into two components: the regularizing effect of dropout itself and the effeet of adaptively updating the dropout-indueed reg- ularization. For the former, it is well-known [40, 39 35 32] that dropout induces Tikhonov-like regularization for standard binary and Gaussian dropout. The same result holds more broadly given second moment condition, which we need in order to be able to consider methods like HARDCON- CRETE "En regularization" [29]. In general,
the term dropout refer to methods that mask or scale the weights w by independent "mask" variables s ~ (ASK(a) with sampling parameters a [0, 1] We emphasize that in this formulation, it the parameters that are masked, rather than the nodes of network in the original formulation of dropout [35], although in linear regression there is no difference. We assume that the masks are unbiased (i.c., that [sy] = 1), and that E [ aj
Note that with the independence assumption, this implies that Cov(s) diag (a) I. Both standard unbiased binary dropout, where sj take value aj with probability aj and 0 otherwise, as well as Gaussian dropout with ~ (1, - 1), satisfy these assumptions.
Dropout solves the optimization problem
minimize
Es (C(SOw)
via stochastic optimization such as gradient descent where each iteration performs a partial optimiza- tion of L(s O w) with random It is most insightful consider when L is the loss for linear regression of target variables y E RR given data X E Rxdd
1 2n lyy Xw
[(w)
(3)
In this case, under our assumptions on s, we recover known Tikhonov result [e.g..32
Eg [[C(s@))]] ((w) w - (a) (a)- -
- x 71 1 x
x
w.
For simplicity, we assume that the data standardized, or that diag ( x inexpensive to satisfy in practice, but our analysis can be extended to the general case. Under this assumption, see that dropout elicits diagonal Tikhonov regularization with scale diag (a) -1 -I.
I, which is
2.2 The "vp-trick"
The second component of adaptive dropout is the adaptive update of the dropout-induced regular- ization. To understand this, introduce the so-called "ny-trcck" 44] applied to the regularization penalty in ① By introducing an auxiliary variable n H R_, we can replace ?(w) with a dual formulation function of v that majorizes ((w)) and is quadratic w for fixed v). In other words,
we can find function f H s R=
E such that
1 m 2
Ω ((w)= min
diag w f(m))
(5)
If such a function f exists, then the regularized ERM problem can be rewritten as joint optimization in w and n of the dual regularized ERM formulation
A Tw diag (1) (n) w + f(m)
minimize W.F)
L(w) +
(6)
This joint optimization can be performed, for example, by alternating minimization over w and For linear regression problems, this gives rise to iteratively reweighted least squares (IRLS) algorithm, a popular method for sparse recovery [10 [1]. We note that iteratively reweighted & schemes have
also held significant place in sparse optimization [7 42). but because dropout regularization quadratic, we limit our consideration to iteratively reweighted &g (Tikhonov) regularization like (6).
3

---

Table 1: Common regularization penalties have dual formulations that often straightforward obtain. See Table in Appendix A for more complete table common penalties along with the derivations.
Penalty
S?(w)
f(n)
(w))
&
|uy|
|
Ve
(v (Ve4+ans--)
LoGSUM(E)
9
log (1w3g)
+ +e)
2 log
|ey ([wy) + c)
2
MCP(a, A)
|uyl
" |uyl & ad
nAg by +a
aA ax-1ws1
adwu,
|wys"
|kty]
< ad
|uyl > aA
|ury| 3 ad 00 15 E ToP-k(w))
20,
HARDTHRESH(K) 7
00 { [wll
k}
H (n Imlll Re)
The simplest example of a penalty Ω that fits the form 1 is [2(w) ) /w), for which f(n) = and it is used in the IRLS formulation of E regularization. More broadly, we can consider the class of functions for which f2 is concave in w These functions are said to be subquadratic, and are known to have sparsity-inducinn
effect [5]. In fact, these are the only functions 5 that can fit this form. This is because w - -202( w) is the Legendre-Fencher (LF) transform of - -7-11- - [(n), and thus 52 is concave in w By the Menchel-Moreau theorem, there is conversely a unique function satisfying (5) that is convex in - which can be obtained by the LF transform. That is, Ω and are dual functions. Additionally, for differentiable penalties, the minimizing n given fixed w is
(ww) (2V (eW=)=(((w))0-
which means that ((w)) has a closed-form expression, then so does the minimizer (w) The sim- plicity of this form means that iteratively reweighted Tikhonov regularization is simple to implement for any differentiable penalty, leading to its popularity as a sparse optimization technique.
We enumerate a few penalties along with their dual formulations in Table L For separable penalties, we list the scalar function applied to a single element. Where H. is restricted, we include it with f(m).
Hard Thresholding. Of particular note is HARDTHRESH. the indicator penalty of level sets of wwloo which gives rise to the iterative hard thresholding (IHT) algorithm 77]. We remark that to the best of our knowledge, what we show in Table T is the first characterization of IHT as an iteratively reweighted l solution to regularized ERM problem, rather than as heuristic algorithm. This connection is important for understanding the MAGNITUDEPRUNING 45 algorithm in Section 4.4
3 Adaptive Dropout as Iteratively Reweighted Stochastic Optimization
which both optimizes the weights w using dropout and updates the dropout parameters a according to some update rule depending on w. To facilitate our analysis, we can equivalently consider the algorithm to be updating parameters
We can now consider an adaptive dropout algorithm
3
27 29
n = (a 1 1)=,, which a monotonic and invertible function of aj, with
TJ aj 7 +
An adaptive dropout method could resemble, for example, the following gradient descent strategy with step size parameter p given an update function f Rdd H:
(w(')) n (w*) +x
w 1+1
w'
p w
O w s ~'~ **MK(a)))), aj
The stochastic update of w corresponds to the expected loss Eg [C(s C w)]. but we must also characterize the 17 update. We make the assumption, which will justify later, that fi(w) is the minimizer given w of a joint objective function J(w, 77 with a function f such that
A J(w,) = E. + 1())
(10)
This brings us to our main result, in which we uncover a duality between the adaptive dropout objective function J(w, 1)) and an effective regularized ERM objective function as in

---

Theorem 1. If C has the form 3 for data X such that 1 ddiagg (xx) I, then for every À >>> 0 there exists a function f such that oj w) arg min, I (w,n) n) if and only if there exists a function Q that is concave in w such that
min I (w. = [(w) + AQ(w).
(11)
NEH
Furthermore, if exists, then the corresponding Qt is unique, and if St exists, then there is a unique such J that is convex in -
Proof. For linear regression under the standardized data assumption, combining the definition of v as a function of a with 4 makes J(w, 7) of the form (6). Then by the properties of the LF transform as discussed in Section 2.2. we obtain the existence and uniqueness of and 12.
That is, every subquadratic penalty Q has a dual formulation via the 7-trick and therefore has a corresponding adaptive dropout strategy. The converse holds as well, provided the dropout update function n can be expressed as a minimizer of the joint objective. While this may seem restrictive, 9) is separable, have the following general result.
Corollary 2. If ( =
i (w;]++ is a monotonically increasing differentiable function of |wyb.
(w,))
then there exist separable f and ft satisfying Theorem 7
Proof. Because the relationship between a and n) is monotonic and invertible, fty (w;) ) is also mono- tonically increasing. Thus, using 77. (Ww)) = Sa 28(e)/2) dt Ca, for which 2² (owz)a ((w)) &
[5). which agrees with Therefore, any adaptive dropout method that updates the dropout parameters a monotonically as function of the magnitudes of the elements of equivalent to regularized ERM with a subquadratic penalty Ω. It is well-known that subquadratic penalties are sparsity-inducet the "rich get richer" intuition that larger wy, which receive larger aj, are penalized less and therefore are more likely to grow even larger, while conversely smaller w, receive smaller aj and are even more penalized. Corollary Zholds generally for the 17-trick and is not limited to adaptive dropout.
We emphasize that linear regression is the setting in which Theorem I gives an exact duality, but that we can still expect a similar correspondence more broadly. For instance, Wager et al. [39] showed that for generalized linear models, dropout elicits similar effect to 4 albeit with a somewhat more complex dependence on w and the data. Section 5 we empirically consider deep networks and demonstrate that the behavior is very similar between adaptive dropout and algorithms that solve the corresponding effective regularized ERM problem.
4 Several Sparsity Methods and Their Effective Penalties
Armed with the fact that adaptive dropout strategies correspond to regularization penalties, we now have way mathematically compare adaptive dropout methods. Table ( lists the methods that we compare and a corresponding characterization of the dual formulation. These methods vary in their parameterizations, and most do not admit closed-form expressions for their effective penalties. However, given these characterizations, we can numerically compute the effective penalty and make a principled comparison, which we plot in Figure
Because some of the methods we consider use slightly different variants the dropout algorithm, we first make clear how these fit into analysis framework.
Reparameterization Tricks. The problem of noise in stochastic gradients, especially when ay is very small, can limit the applicability of dropout methods for sparsity. One way researchers have found to reduce variance is through "reparameterization tricks" such as the local reparameterization trick 40 27 and the additive noise reparameterization trick [32), both used in VARIATIONAL.DROPOTT which consider alternative noise models without changing the optimization problem.
To consider the MAGNITUDEPRUNING algorithm, we develop another reparameterization trick. Inspired by the additive reparameterization trick from Molchanov et 32 we introduce another
Results on the LF transform do not apply to functions with discontinuities at (-xc, c0), such -17 - ((n) for the to and HARDTHRESH(() penalties, theorem does not prove the existence of such functions.
5

---

Table 2: Select spansity methods and their effective penalty characterizations.
Method
Effective Penalty Characterization
STANDOUT
3g ((2))) +) c((z))+)
f(1) 2Dxx (99.w (W) Ip(w)) S33 P(V(22)
VARIATIONALDRROOUUT 32
dt
v/22
[(i) = =2Σ, (log(())) - log(-7/S)) -/())
HARDCONGRETTLLONORM 29
AZ -331" 8j HARDCONCRETES,.C())
1
MAGNITUDEPRUNINGGG)
(w()) co1 {i Top-k(w))
variable and add the constraint v = w. under which L(sO = L(w (s 1) @ v). Then for least squares with standardized data, becomes
Eg [[(w + 1) C v)] = [(w) -5
(diag + (a))
(12)
Under this parameterization, we can now use low-variance stochastic optimization for w and deter- ministic optimization for v, along with a strategy for enforcing the equality constraint w such as ADMM [8]. thus reducing the overall variance. A simple proximal variable update strategy under this reparameterization (see Appendix (B) gives rise an adaptive proximal gradient descent algorithm, which we use to describe the MAGNITUDEPRUNINS
algorithm.
Biased Masks. It is not uncommon for dropout strategies to use biased masks in the sense that [s,] 5 e.g., for 8j ~ BERNOULLL((G)). Under few mild assumptions on the parameterization of this distribution, we can consider adaptive dropout in this setting as well. Suppose that μa :- u(a)= = E [s] is an invertible function of a. Define 5j == sy/ Mdj āj := pr/ [>]]. and w = 4 w. Then is an unbiased mask, and since s O w = $ O W, the expected loss has the same form as in
but with & and W in place of a and w. Let î = (a, 1) 1) and suppose there exists function de such that μú (M;)), which is true if the mapping between ā and μ is one-to-one. Now consider the expected dual regularized ERM formulation
minimize ((HOw) + (IH w) diag (7) (7) (n
(13)
subject to M4 = (m))) Vj.
Thus, an adaptive dropout strategy that jointly optimizes n and w given s ~ MASK(a) solves the regularized ERM problem for w, where is set according to îj.
4.1 Standout
One of the earliest methods to use adaptive probabilities with dropout was the STANDOUT method of Ba and Frey 3 who proposed to augment a neural network with an additional parallel "standout" network of the same architecture that controlled the dropout probabilities for each activation, which they showed results in sparse activations. Interestingly, the authors noted that the method worked just as well if the standout net was not trained but instead simply copied the weights of the primary network at each iteration, which fits our adaptive dropout framework.
Consider a single hidden layer neural network with ReLU activations (u)+ max {u, 0). Denote the pre-activations from the first layer as z WIx. The output of the network is given by @ = (s w.
(z)+). where the mask consider the network function of the first layer activations (z)+. Using (4) and the fact that (t) = e et, we know that the expected squared loss in this setting is
MASK (a(z)) is from the standout network. Now let us
1 -1ll w. (z)
2 Eg | w (s (z)
-
(z) (e Www,) (z)+.
(14)
However, terms where: 2j 0 make contribution to the effective penalty, (z,)+ for 2j in e This enables us to extract 77(()) +.) as in Section 2.1
substitute
(=(++
i ((2))))
(15)
[2213

---

We note that (15) represents the implicit update of 1 each iteration as the standout network weights are copied from the primary network. By Corollary 2 adaptive dropout strategy with a subquadratic effective penalty Q applied to the activations (2)+:
we know that STANDOUT therefore an
(=,)2
(((()+))
1 2A
Σ
[us]fe
di
X
[wal (1 [waa
((2g)+ + ())
(16)
This penalty is smooth approximation the lo penalty, shown in Figure and thus naturally sparsifies the activations as W, is trained. Additionally, (5(((z+) has no dependence on X, just as the adaptive dropout parameter update has dependence A.
4.2 Variational Dropout
Variational Bayes interpretations of dropout arose with the works of Kingma et al. 27 and Gal and Ghahramani [14), providing an automated means of choosing the dropout parameter through variational inference. Molchanov et al. [32] later showed that each weight is allowed to have its results in sparse solutions.
own dropout parameter, VARIATIONALDROPON
The authors of VARIATIONALDROPOUT consider maximum a posteriori inference of using the improper log-uniform prior, for which p(log (wyl) x or p([wsl) x 1/|wy/- This is equivalent to solving the LocSSuM() ERM problem
minimize w
((w)
Σ log([w;))
(17)
Obviously, this is minimized by taking
0 for common loss functions, which is uninteresting, and Hron et al. [20] discuss other issues with the framework. However, VARIATIONAL variable w, using an
does not solve the problem in (17). and instead performs inference over
approximate posterior 9nn. w (w) such that w = s O w. where s ~ MASK(a) for Gaussian dropout,
having a and related by (8). Then maximizing the variational lower bound is equivalent to solving
EEW [C(sO [[][sO))]+ 1
minimize W.1)
DEL (fnn, w |p( W)),
(18)
where (-I-- is the Kullback-Leibler (KL) divergence. With the choice of the log-uniform prior, the KL. divergence has no dependence on w and is purely a function of (27 Hron et al. 220
show
that this quantity can be expressed in terms of Dawson's 'ss integral F(u) "² S e dt:
F( 1/2)
DKL (Mon, (w)
dt,
(19)
V2L
which we can compute via numerical integration using standard software implementations of F(u). Clearly then, the Monte Carlo variational Bayes optimization w and r in VARIATIONALDROPOUU is an adaptive dropout strategy with X 1 and f(n) 2Dx1. (f (W) (lp(W)) and we can numerically compute the effective penalty Ω. VARIATIONALDROPOUT also uses additive and local reparameterization tricks to reduce variance.
As shown in Figure I the effective penalty for À = is quite similar to the LOGSUM(2) penalty for 2. We explore this connection further experimentally in Section S
4.3 Hard Concrete "Lo Regularization"
While the desired regularization is typically an lo penalty, its non-differentiabiiity makes difficult to perform gradient-based optimization, which motivates finding smooth approximations to the to penalty. Louizos et al. [29] propose to first consider L(w) + wlo using the decomposition
w = C w for binary s, where [wll- = |s|o-
We take this opportunity to note that we can consider this problem using adaptive dropout with
f(n) 1x,1 [|s|[] Σ, aj if we use biased masks 8j BERNOUI.LI(G)).. Interestingly, when = 1, the resulting penalty turns out to be equal to the minimax concave penalty 44 MCP(1, 1), giving the MCP an additional interpretation as stochastic binary smoothing of the lu penalty.
'The parameter from Molchanov et al. 32 is distinct from ours but is related to by at mn.-
7

---

However, Louizos et al. [29] do not solve this problem, but instead choose distribution for s such that the random variables themselves are differentiable with respect to a, allowing them extract gradient information for a from L (s w). They propose the following biased distribution, called the HARDCONCRETE distribution:
2j (llog u -oog(1 uy) + loga,)/8). u UNIFORM 1]
(20)
8j = min {1, max (0, (S - 7)) + +
(21)
They then use as their penalty ( 2 E Pr(s; > 0) - (o(log(a)) - 3 log(-r/()) under the biased dropout reparameterization. The relationships between EE [sy], aj. āj. and î6 are all invertible,
so we can numerically determine the corresponding penalty f2.
We plot the resulting penalty in Figure using the default values f 3. y= = 0.11 c= 1.1 29
4.4 Magnitude Pruning
is straightforward way to induce sparsity by zeroing out small values. Here we specifically refer to the strategy of Zhu and Gupta (45), who eliminate all but the k largest parameters each iteration. Initially k = d, and it is decayed number until it reaches the desired sparsity level according to a general-purpose pruning schedule. This is equivalent to IHT 7 (except with decreasing k) and therefore almost surely corresponds to proximal gradient variant of adaptive dropout where āj(v) = 1 { E ToP-k(w) w)}. or (v()) (v)], implemented in Appendix
MAGNITUDEPRUNING
43 45
function of iteration
col (j ToP-k(v))
According to Table | the effective penalty is the HARDTHRESH(()) penalty, which we plot in Figure On philosophical note, this is by definition the "correct" penalty to when searching for solutions with a constraint on the number of nonzeros, since is the characteristic function of that set of solutions. The n-trick along with alternating optimization turns this combinatorial search problem into a sequence of tractable gradient- based (for w) and closed-form (for 17) updates. is therefore no surprise that Gale et al. [15] observed that this "simple" magnitude pruning algorithm matched or outmatched the other more sophisticated methods.
5 Empirical Comparison for VARIATIONAL.
With complex methods such as the adaptive dropout methods we discuss above, it is often difficult to isolate what drives their success. Under our theory, such methods should be able be decomposed into dropout noise-related effects and n-trick effects. Diving further into the VARIATIONALDRONE algorithm, we compare it against other 17-trick strategies, examining the effects of dropout noise, -ttrick optimization strategy, and regularization with the underlying penalty.
We apply the methods the task of sparsifying LeNet-300- 100 network the MNIST dataset, a common benchmark for sparsification methods 11
16 12]. Experimental details along with layer-wise sparsification results are given in Appendix C We compare the following methods. Except for VARIATIONAL RRIIATIONAL.DRROUUTT DROPOUT methods, we use the LOGSUM penalty.
VARDROP+LR+AN VARLATIONALDROPOUT as proposed by Molchanov et al. [32] using both the local reparameterizztion trick and the additive reparameterivatioo trick.
VARDROP+LR. VARIATIONAL using the local reparameterization trick but not the additive reparameterization trick.
with reparameterization tricks. --TRICK. Joint gradient descent in and log(1)) of (6). This would be equivalent to VARIATIONAA.DROPOUT with no noise in linear regression.
VARDROP. VARIATIONAL
ADAPROX. Proximal gradient descent in w of (6) using = f(() at each iteration.
ADATTKHONOV. Gradient descent in w of (6) using r (7( at each iteration.
LOGSUM. Gradient descent in w of directly.
We plot the results in Figure 2
With all of the methods compared, we see similar behavior: the networks become more sparse over time as well as more accurate. In general, the methods appear to exhibit sparsity-accuras accurray trade-off, even when they optimize identical objective functions (consider

---

Fraction of Nonzero Parameters
Validation Error
1.0
0.14
VARDROF+LR-AR
VARDROP-LR
0.8
VARDRRO
T.TRICK
0.6
ARAPROX
0.08
ADATIKHONOW
0.4
0.06
LOGSUM
0.2
0.04
0.0
25
50
75
100
125
150
175
200
0
25
50
76
100
125
150
175
200
Training Epoch
Training Epnch
Figure 2: VARIATIONAL compared with other 17-trick strategies using LOGSUM penalty.
VARDROP+LR+AR and VARDROP+LR, or ADAPROX and 17- TRICK). This trade-off seems to be determined by the early iterations, depending whether the method sparsifies the network quickly not. We also observe that the stochastic noise due to dropout in VARDROP leads to very slow conver- gence, while the variance-reduced VARDROP+LR converges exceedingly similarly to its non-random counterpart 7/ TRICK In addition, VARDROP+LR+AN behaves remarkably similarly to ADAPROX. which was inspired by the additive reparameterization trick (see discussion on reparameterization tricks in Section 4. These results suggests that the success of PARIATIONALDROPO explained by interpreting
is well optimization of a dual formulation of a regularized ERM problem. Further, they suggest that the variance due to dropout noise at best yields similar performance to the non-dropout counterpart and at worst can drastically slow down convergence. Interestingly, the ERM approach performs as well as any of the other methods considered, and is nearly identical in behavior ADATIKHONOV, suggesting that the sparsification performance of VARIATIOOAT.DROPP is entirely due to the effective LOGSUM penalty.
simple POGSSu--rggularin
6 Discussion
Given the duality between adaptive dropout and regularization that we have presented, it is no surprise that such methods excel at sparsifying deep networks. However, many questions still remain.
For one, is there any benefit to dropout noise itself in adaptive dropout optimization, beyond the n-trick connection? There are known benefits of standard dropout for deep network optimization [ 19 31 41), but do these benefits transfer to adaptive dropout strategies? A crucial compo- nent VARIATIONAL
as demonstrated in Section 5 appears to be that its successful implementations remove most of the noise due to dropout. On the other hand, methods like HARD- CONCRETEL LOMOMM from Louizos et al. (29] exploit the fact that sparse masks yield more efficient computation, and they do not seem suffer from variance-relatee optimization issues even without taking efforts to mitigate such effects. This could be because they use a biased dropout distribution, which results in significantly lower variance of stochastic gradients. On the other hand, it is not clear that dropout noise is necessary to obtain these computational speedups; other methods where the network is sparse during training, such MAGNITUDEPRUNING should have the similar properties.
is evident from Figure that, even when using the same objective function, the optimization strategy can have significant impact on the result. It is beyond the scope of this work to examine the convergence rates, even for convex problems, of the adaptive algorithms we consider in Section 5 (MM) algorithms, for which some theory exists [30]. However, for non-convex penalties and losses, such as in sparse deep network optimization, there no guarantecs about the quality of the solution. It is also not clear what benefit 7 optimization provides in this setting, if any. For linear problems it has the advantage of drastically simplifying implementations of penalized methods by reducing them to successive least squares problems, but with modern automatic differentiation packages it is straightforward to implement any differentiable parsity-inducing penalty. To the best of our knowledge, the only large-scale comparison of sparsity methods for deep networks is that of Gale et al. [13], and they do not consider classical penalty-basee methods.
although many of them can be cast as stochastic anjorization-miniiizaaii
We have fully analyzed the we know that the exact correspondence between dropout and Tikhonov regularization does not hold
duality in the linear regression case, and

---

in other settings, such as deep networks. However, from the neural tangent kernel (NTK) 23 perspective, very wide networks become linearized, and parameter-level dropout becomes dropout for linear regression, like we study. Furthermore, recent work has shown that for very high dimensional settings, the choice between classification and regression loss functions often irrelevant [3 21].
28
For finite-width settings, good quadratic approximations to the general effective regularization of dropout have been developed [39 41]. which relate to the Fisher information. These approximations could be used to develop adaptive sparsification strategies that enjoy the data-sensitive properties of dropout, such as logistic regression or deep networks.
The duality presented in this paper can be the basis of developing more efficient principled sparsifica- tion schemes as future work. In particular. using penalties that have been proposed in the compressed sensing literature such as scaled lasso [37] which is adaptive to noise level, or SLOPE [36], which is shown to minimax optimal in feature selection, can lead adaptive dropout schemes with better performance in sparsifying deep networks. In addition, analytical tools such as debiasing [25], which has been used for near-optimal feature selection could be employed to design improved sparsity methods.
Acknowledgments and Disclosure of Funding
This work supported by NSF grants CCF-191 199144,, I1S-1838177, and IIS-1730574; ONR grants N00014-18- 18-11-11- N00014-20-11 11144-00--2555 1-544,,,,, and MURI N00014-20 11-1--0---77777 AFOSR grant FA9550-18-1- 0478; and Vannevar Bush Faculty Fellowship, ONR grant N00014-18-1-2047,
References
[1] R. Arora, Bartlett, P. Mianjy, and N. Srebro. Dropout: Explicit forms and capacity control. arXiv e-prints, arXiv:2003.03397, 2020.
[2] A. Ashukha. Sparse Variational Dropout a Minimal Working Example, 2018. URL https
wwtp:///wyye--sshhhh//rrreeeeeee pyytorch Accessed: May 2021.
[3] J. Ba and B. Frey. Adaptive dropout for training deep neural networks. In Advances in Neural Information Processing Systems, volume 26, 2013.
[4] F. Bach. The "n-trick" or the effectiveness of reweighted least-squares, Jul 2019. Ittps com/ -CChBe tree7--rrickee or http://rrr.....ooooeeeeeeeeeeeee the-effectiveness of of-rrewiighted-eeete squassss Accessed: May 2021.
[5] Bach, R. Jenatton, J. Mairal, and G. Obozinski. Optimization with sparsity-inducing penalties. Foundations and Trends in Machine Learning, 2011.
[6] L. Birgé and P. Massart. Gaussian model selection. Journal of the European Mathematical Society, 3:203-268, Aug 2001.
[7] T. Blumensath and M. E. Davies. Iterative hard thresholding for compressed sensing. Applied and Computational Harmonic Analysis, 27(3):265 274, 2009.
[8] S. Boyd, N. Parikh, and E. Chu. Distributed optimization and statistical learning via the alternating direction method of multipliers. Now Publishers Inc, 2011.
[9] E. J. Candès, M. B. Wakin, and S. P. Boyd. Enhancing sparsity by reweighted & minimization. Journal of Fourier Analysis and Applications, 14(5-6):877-966 2008.
[10] R. Chartrand and W. Yin. Iteratively reweighted algorithms for compressive sensing. In 2008 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 1869-3872 2008.
[11] I. Daubechies, R. Vorer, M. Fornasier, and C. S. Güntürk. Iteratively reweighted least squares minimization for sparse recovery. Communications on Pure and Applied Mathematics, 63(1): 1-38, 2010.
10

---

[12] X. Dong, S. Chen, and S. Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. In Advances in Neural Information Processing Systems, volume 30, 2017.
[13] Fan and R. Li. Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association, 134 2001.
[14] Y. Gal and Z. Ghahramani. Dropout as Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of The 33rd International Conference on Machine Learning, volume 48, pages 1050-1059, Jun 2016.
[15] I. Gale, E. Elsen, and S. Hooker. The state of sparsity in deep neural networks. arXiv e-prints, arXiv:1902.09574, 2019.
[16] Y. Guo, A. Yao, and Y. Chen. Dynamic network surgery for efficient DNNs. In Advances in Neural Information Processing Systems, volume 29, 2016.
[17] S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, volume 28, 2015.
[18] T. Hastie, R. Tibshirani, and M. Wainwright. Statistical learning with sparsity: the lasso and generalizations. Chapman and Hall/CRC Press, 2019.
[19] D. P. Helmbold and P. M. Long. Surprising properties of dropout in deep networks. In Proceedings of the 2017 Conference on Learning Theory, volume 65, pages 1123-1146, Jul 2017.
[20] Hron, Matthews, and Z. Ghahramani. Variational Bayesian dropout: pitfalls and fixes. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages Jul 2018.
[21] D. Hsu, V. Muthukumar, and J. Xu. On the proliferation of support vectors in high dimensions. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130, pages 91-99, Apr 2021.
[22] J. Huber. Robust estimation of location parameter. The Annals of Mathematical Statistics, 35(1):73-101, 1964.
[23] A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, volume 31, 2018.
[24] A. Javammard and H. Javadi. False discovery rate control via debiased lasso. Electronic Journal of Statistics, 13(1):1212 100:11222112551 1213333, 2019.
[25] A. Javanmard and A. Montanari. Debiasing the lasso: Optimal sample size for Gaussian designs. The Annals of Statistics, 46(6A))2593 222222222,2,,, 2018.
[26] R. Jenatton, G. Obozinski, and F. Bach. Structured sparse principal component analysis. In Proceedings of the Thirteenth International Conference Artificial Intelligence and Statistics, volume 9, pages 366-373, May 2010.
[27] D. P. Kingma, T. Salimans, and M. Welling. Variational dropout and the local reparameterizatioo trick. Advances in Neural Information Processing Systems, volume 28, 2015.
[28] Lee, L. Xiao, S. Schoenholz, Y. Bahri, R. Novak, Sohl-Dickstein, and Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Advances in Neural Information Processing Systems, volume 32, 2019.
[29] C. Louizos, M. Welling, and D. P. Kingma. Learning sparse neural networks through Lo regularization. In International Conference on Learning Representations, 2018.
[30] J. Mairal. Stochastic
minimization algorithms for large-scale optimization. In Advances Neural Information Processing Systems, volume 26, 2013.
[31] Mianjy and R. Arora. convergence and generalization of dropout training. In Advances in Neural Information Processing Systems, volume 33, pages 21151-21161, 2020.

---

[32] D. Molchanov, A. Ashukha, and D. Vetrov. Variational dropout sparsifies deep neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 2498 2507, Aug 2017.
[33] V. Muthukumar, A. Narang, V. Subramanian, M. Belkin, D. Hsu, and A. Sahai. Classification vs regression in overparameterized regimes: Does the loss function matter? Journal of Machine Learning Research, 22(222):1 69, 2021.
[34] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. Vito,, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035. 2019.
[35] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15 156) 2014.
[36] W. Su and E. CandAšs. SLOPE adaptive to unknown sparsity and asymptotically minimax. The Annals of Statistics, 443))10338-068, 2016.
[37] T. Sun and C.-H. Zhang. Scaled sparse linear regression. Biometrika, 99(4):879-88) 898, Sep 2012.
[38] M. Tietz, T. J. Fan, D. Nouri, B. Bossan, and skorch Developers. skorch: A scikit-learn compatible neural network library that wraps PyTorch, Jul 2017. URL. httthtttt///korrt readthedocs. craeeddoossi//eneeee in/ stable/
[39] S. Wager, Wang, and P. S. Liang. Dropout training as adaptive regularization. In Advances in Neural Information Processing Systems, volume 26, 2013.
[40] S. Wang and C. Manning. Fast dropout training. In Proceedings of the 30th International Conference on Machine Learning, volume 28, pages 118-126, Jun 2013.
[41] C. Wei, S. Kakade, and T. Ma. The implicit and explicit regularization effects of dropout. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 10181-10192, Jul 2020.
[42] D. Wipf and S. Nagarajan, Iterative reweighted l and l methods for finding sparse solutions. IEEE Journal of Selected Topics in Signal Processing, 4(2):317-329, 2010.
[43] X. Xiao, Y. Yang. Ahmad, L. Jin, and T. Chang. Design of a very compact CNN classifier for online handwritten Chinese character recognition using Wrigheight and global pooling. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), volume 01, pages 891-895, 2017.
[44] C.-H. Zhang. Nearly unbiased variable selection under minimax concave penalty. The Annals of Statistics, 38(2):894-99) 942,,922272 2010.
[45] M. Zhu and S. Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv e-prints, 1rXiv:1710.0188 2018.
[46] H. Zou and T. Hastie. Regularization and variable selection via the clastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67(2):301-322 2005.
[47] H. Zou and R. Li. One-step sparse estimates in nonconcave penalized likelihood models. The Annals of Statisties, 36(4):
2008
12


Reviews:
[{"review_id": "g5xQsNBHTi", "reviewer": "Reviewer_t3cP", "summary": "For linear regression, the paper shows that several popular heuristics used in deep learning correspond to regularized empirical risk minimization with sub-quadratic penalties, thereby, arguing that these methods indeed induce sparsity. The analysis leverages the so-called \\eta-trick, where the penalty is replaced with its variational (minimization) form. Empirical results are also provided to support the theory.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "One of the strengths of the paper is that it unifies several popular heuristics, including variational dropout, standout, magnitude pruning, etc. The claims are rigorously proved and the empirical evidence validates some of the claims.\n\nThe main limitation of the paper is that \\eta-trick itself is limited: it can be applied only when a weighted-sum-of-squared of parameters is present in the objective. Dropout in linear regression with l2-loss is known to induce such a penalty, which is leveraged here to make connections to several adaptive dropout scenarios. However, I'm not sure if the techniques can be generalized to analyze even simple 2-layer linear networks. The equivalent form of equation (4) has been studied for deeper linear models (2-layer and beyond). Given the more complicated form of the penalty in such models, do the authors hope this approach to generalize? Another limitation is that the paper merely focuses on the duality in the optimization domain. There is no statistical or computational analysis comparing either of the two forms (adaptive dropout or the dual IRLS). Is there a benefit in doing one over the other? \n\nAlthough I’ve read several papers on the topic, following the paper was difficult at times. I think the current presentation makes it challenging for the average reader to understand the main results and constructions, and distinguish them from less important, tangential results. I suggest that the authors list the main contributions of the paper in the introduction. The duality between different penalties and their \\eta-trick-representations in table 1 is interesting, but the description of their connection in sections 2.2 and 3 can be improved, in my opinion. The motivation behind studying adaptive dropout updates in section 3 is not clear at all. What is the merit of updates in equations (8) and (9), except for “facilitating your analysis” (as stated in line 123)?  Also, it is not clear how the “standardization” assumption affects the results and techniques. The authors suggest that “it is inexpensive to satisfy in practice, but our analysis can be extended to the general case”. However, they do not say if/how the main takeaways from the paper will be affected if the data is non-standard.\n\ntypo: line 20: trad-eoff", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "bLj4IbryuWZ", "reviewer": "Reviewer_QmiQ", "summary": "This paper examines stochastic perturbation methods, such as dropout, through the lens of the \"eta-trick\".  This trick allows the optimization objective to be re-written as a joint objective in terms of the original parameters and auxiliary parameters eta.  This new objective consists of the original loss in addition to a quadratic penalty in the parameters and a formulation-specific function of eta.  The paper shows that any adaptive dropout method that updates the dropout parameters monotonically as a function of the weight magnitudes is equivalent to regularized risk minimization with a subquadratic penalty, which is known to induce sparsity.  The paper analyses standout, variational dropout, L0 regularization, and magnitude pruning from this perspective, deriving the corresponding penalty term (Table 2) and allowing for comparison.  The paper then experimentally compares variants of variational dropout, finding that performance is well-described by the eta-trick reformulation since stochastic variants suffer from the introduced noise.  ", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found this paper well-written, interesting, and novel.  I had not seen the eta-trick before, and I found its use to compare seemingly disparate regularization methods to be fascinating.  Moreover, I like how the experiment isolates the different variants of variational dropout and compares them to the eta-trick-derived formulation.  I have no criticisms at this time; I find the paper ready for publication.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, I found limitations and societal impact to have been addressed.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "LXzuP_XDfqr", "reviewer": "Reviewer_8X6v", "summary": "This paper draws connections between adaptive dropout and regularized optimization using the $\\eta$-trick. The authors show that various adaptive dropout methods can be reinterpreted as sparsity-induced regularization for linear regression models.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nI think this work is solid and sound. The main contribution of this paper is that the authors provide a unified framework for analyzing the existing adaptive dropout methods and thus provide theoretical justifications for them. The analysis explains the underlying mechanisms of some adaptive dropout methods, which can potentially contribute to the research in developing strategies for sparse training of neural networks.\n\n\n1. One concern is that the adaptive dropout methods are not so commonly used in practice (e.g., for training neural networks) and are also more complicated than the Bernoulli dropout. However, it violates the monotonic assumption and thus doesn't fit in your framework. \n\n\n2. Also, the current analysis is limited to linear regression problems. It may not hold for linear classification problems, for which the closed-form solution may not exist and hence add extra difficulty for the analysis. However, classification problems are more common. Do you think similar conclusions can still be made for linear classification problems with adaptive dropout?\n\n\n3. In Figure 2, you showed similar behaviors of variational dropout with its dual formulation in terms of the fraction of nonzero parameters and validation error. I wonder if they induce similar sparsity patterns (e.g., the fraction of nonzero parameters at each layer)?\n\n\nOverall, I am positive about this paper as it provides a unified framework for analyzing adaptive dropout methods with mild assumptions. It provides theoretical justifications for these methods, as some of them are proposed based on heuristics. Therefore, I vote for acceptance.\n\n====Post rebuttal=======\n\nThanks for the response. My questions & concerns were addressed by the response. I would suggest the authors include the additional empirical results in the revised paper. Though the theory is for linear cases, if the authors can empirically demonstrate that the theory aligns well with non-linear systems, such as neural networks, I think this will make the paper stronger. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitation was discussed, and the negative societal impact was not. I did not see any negative society impact neither.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "6dyoNX9xmT", "reviewer": "Reviewer_U2rc", "summary": "The paper proposes to use a dual formulation that majorizes the original dropout regularization, and rewrite the original dropout objective as a joint optimization on both the parameter and the auxiliary variable. The authors further show in the linear case any dropout algorithm, if it adapts to the magnitude of the weights monotonically, is equivalent to a regularized ERM with a subquadratic regularizer, which induces sparsity. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It is nice to build a connection between the adaptive dropout strategy and sparse solution from the perspective of joint optimization and dual formulation. In this way, the dropout strategies are better understood in terms of the regularizers in the objective. It is a step forward towards understanding the effect of dropout. \n\nOn the other hand, the analysis of the draft is mostly limited to the linear loss. For deep models, due to its non-linearity, the impact of dropout may not be captured fully as a regularizer in the objective. For example, it is still unclear to me how dropout may interact with the optimization algorithms during the training. \n\nA follow up question: is there any particular reason the authors used the non-linear LeNet model in the experiment? Feels like the analysis is mostly about linear models. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "the authors adequately addressed the limitations and potential negative societal impact of their work", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper shows that popular heuristics in deep learning, e.g. dropout, in the context of linear regression amount to regularizing the empirical objective with sub-quadratic penalties which promote sparsity. The paper introduces interesting tools that may be interesting outside the scope of the paper. Overall, a good paper. 
Decision: Accept (Poster)