The Flip Side of the Reweighted Coin: Duality of Adaptive Dropout and Regularization

1 Introduction . In machine learning , it is often valuable for models to be parsimonious or sparse for a variety of reasons , from memory savings and computational speedups to model interpretability and generalizability . Classically , this is achieved by solving a regularized empirical risk minimization ( ERM ) problem of the form minimize w Lpwq ` λΩpwq , ( 1 ) where L is the data-dependent empirical risk and Ω is a sparsity-inducing regularization penalty . Ideally , Ωpwq is equal to } w } 0 : “ # ti : wi ‰ 0u or a function of } w } 0 [ 6 ] , in which case an appropriately chosen λ balances the trade-off between the suboptimality of L and the sparsity of the solution . However , the use of } w } 0 directly as a penalty is difficult , as it makes local search impossible , so alternative approaches are necessary for high-dimensional problems . The classical solution to this is to relax the problem using a smooth surrogate Ω that approximates } w } 0 [ 5 , 18 ] . When we know the regularization penalty Ω , we can understand the types of solutions they encourage ; however , many popular methods for sparsifying deep ( neural ) networks do not appear to fit the form ( 1 ) . For example , variational dropout [ 27 , 32 ] , “ ` 0 regularization ” [ 29 ] , and magnitude pruning [ 45 ] are the only methods compared in a recent large-scale study by Gale et al . [ 15 ] , and all achieve sparsity by adaptively masking or scaling the parameters of the network while optimizing the loss function in w. Although these methods are well-motivated heuristically , it is unclear how to make a principled choice between them aside from treating them as black boxes and comparing their empirical performance . However , if these methods could be cast as solutions to a regularized ERM problem , then we could compare them on the basis of their regularization penalties . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We show that , in fact , these sparsity methods do correspond to regularization penalties Ω , which we can obtain , compute , and compare . As we show in Figure 1 , these penalties bear striking resemblance to classical sparsity-inducing penalties such as the LOGSUM [ 9 ] penalty and the minimax concave penalty ( MCP ) [ 44 ] . More broadly , we analyze adaptive dropout methods , which apply the dropout technique of Srivastava et al . [ 35 ] with adaptive parameters . By considering adaptive dropout in the linear setting , we uncover a duality between adaptive dropout methods and regularized ERM . We make this connection via the “ η-trick ” [ 4 ] , a tool with a long history of application in sparse optimization via iteratively reweighted least squares ( IRLS ) ( see the history presented by Daubechies et al . [ 11 ] ) . We prove that all adaptive dropout methods whose amount of dropout varies monotonically with the magnitudes of the parameters induce an effective subquadratic ( and hence sparsity-inducing [ 5 ] ) penalty Ω . This further supports the experimental evidence that such methods excel at inducing sparsity . We also demonstrate how to use our result to determine the effective penalty for adaptive dropout methods , using as examples the sparsity methods listed above as well as the standout method [ 3 ] , which has an effective penalty on the layer activations , rather than the parameters , explaining why the standout method sparsifies activations . We then numerically compute the effective penalties1 and plot them together in Figure 1 . We validate our new theory by applying variational dropout on the task of deep network sparsification , and we show that the performance is similar to non-dropout methods based on the same effective penalty . This suggests that not only is considering the effective penalty a sound means for comparing sparsity methods , but also that classical regularized ERM is itself an effective means for inducing sparsity in deep networks and serves as a valuable tool for future work on deep network sparsity . For theoreticians , this work provides a general framework for obtaining the effective regularization penalty for adaptive dropout algorithms , enabling regularized ERM analysis tools to be applied to these methods . For practitioners , this work enables the application of classical regularization intuition when choosing between sparsifying approaches . For methods developers , this work provides a strong baseline against which new adaptive dropout methods should be compared : the effective penalty Ω . 2 Background . Notation . We denote the extended reals and the non-negative reals by R “ R Y t´8,8u and R ` “ r0,8q , respectively . The operator diag p¨q takes a vector and returns a matrix with that vector along the diagonal entries or takes a matrix and returns the same matrix with all non-diagonal entries set to zero . The matrix J denotes a matrix of all ones . We denote element-wise multiplication and exponentiation by d ; i.e. , ru d vsj “ ujvj and rUdpsij “ Upij . Division of vectors and scalar functions of vectors denote element-wise operations . Order statistics of the magnitudes of 1Our code is available at https : //github.com/dlej/adaptive-dropout . elements in a vector u P Rd are denoted by upjq such that |up1q| ě |up2q| ě . . . |updq| . The function σ : RÑ r0 , 1s denotes the sigmoid σptq “ 1 { p1 ` e´tq . 2.1 Dropout as Tikhonov Regularization . We can understand adaptive dropout methods by breaking them down into two components : the regularizing effect of dropout itself and the effect of adaptively updating the dropout-induced regularization . For the former , it is well-known [ 40 , 39 , 35 , 32 ] that dropout induces Tikhonov-like regularization for standard binary and Gaussian dropout . The same result holds more broadly given a second moment condition , which we need in order to be able to consider methods like HARDCONCRETE “ ` 0 regularization ” [ 29 ] . In general , we use the term dropout to refer to methods that mask or scale the weights w by independent “ mask ” variables s „ MASKpαq with sampling parameters α P r0 , 1sd . We emphasize that in this formulation , it is the parameters that are masked , rather than the nodes of a network as in the original formulation of dropout [ 35 ] , although in linear regression there is no difference . We assume that the masks are unbiased ( i.e. , that E rsjs “ 1 ) , and that E “ s2j ‰ “ α´1j . Note that with the independence assumption , this implies that Covpsq “ diag pαq´1 ´ I . Both standard unbiased binary dropout , where sj take value α´1j with probability αj and 0 otherwise , as well as Gaussian dropout with sj „ N ` 1 , α´1j ´ 1 ˘ , satisfy these assumptions . Dropout solves the optimization problem minimize w Es rLpsdwqs ( 2 ) via stochastic optimization such as gradient descent where each iteration performs a partial optimization of Lps d wq with a random s. It is most insightful to consider when L is the loss for linear regression of target variables y P Rn given data X P Rnˆd : Lpwq “ 1 2n } y ´Xw } 22 . ( 3 ) In this case , under our assumptions on s , we recover the known Tikhonov result [ e.g. , 32 ] Es rLpsdwqs “ Lpwq ` 1 2 wJ ˆ ´ diag pαq´1 ´ I ¯ d 1 n XJX ˙ w. ( 4 ) For simplicity , we assume that the data is standardized , or that diag ` 1 nX JX ˘ “ I , which is inexpensive to satisfy in practice , but our analysis can be extended to the general case . Under this assumption , we see that dropout elicits a diagonal Tikhonov regularization with scale diag pαq´1´ I . 2.2 The “ η-trick ” . The second component of adaptive dropout is the adaptive update of the dropout-induced regularization . To understand this , we introduce the so-called “ η-trick ” [ 4 ] applied to the regularization penalty in ( 1 ) . By introducing an auxiliary variable η P H Ď Rd ` , we can replace Ωpwq with a dual formulation as a function of η that majorizes Ωpwq and is quadratic in w for fixed η . In other words , we can find a function f : H Ď Rd ` Ñ R such that Ωpwq “ min ηPH 1 2 ´ wJdiag pηq´1 w ` fpηq ¯ . ( 5 ) If such a function f exists , then the regularized ERM problem ( 1 ) can be rewritten as a joint optimization in w and η of the dual regularized ERM formulation minimize w , η Lpwq ` λ 2 ´ wJdiag pηq´1 w ` fpηq ¯ . ( 6 ) This joint optimization can be performed , for example , by alternating minimization over w and η . For linear regression problems , this gives rise to the iteratively reweighted least squares ( IRLS ) algorithm , a popular method for sparse recovery [ 10 , 11 ] . We note that iteratively reweighted ` 1 schemes have also held a significant place in sparse optimization [ 47 , 9 , 42 ] , but because dropout regularization is quadratic , we limit our consideration to iteratively reweighted ` 2 ( Tikhonov ) regularization like ( 6 ) . Table 1 : Common regularization penalties have dual formulations that are often straightforward to obtain . See The simplest example of a penalty Ω that fits the form ( 5 ) is Ωpwq “ |w| , for which fpηq “ η , and it is used in the IRLS formulation of ` 1 regularization . More broadly , we can consider the class of functions for which Ω is concave in wd2 . These functions are said to be subquadratic , and are known to have a sparsity-inducing effect [ 5 ] . In fact , these are the only functions Ω that can fit this form . This is because wd2 ÞÑ ´2Ωpwq is the Legendre–Fenchel ( LF ) transform of ´ηd´1 ÞÑ fpηq , and thus Ω is concave in wd2 . By the Fenchel–Moreau theorem , there is conversely a unique function f satisfying ( 5 ) that is convex in ´ηd´1 , which can be obtained by the LF transform . That is , Ω and f are dual functions . Additionally , for differentiable penalties , the minimizing η given a fixed w is pηpwq “ p2∇wd2Ωpwqq d´1 , ( 7 ) which means that if Ωpwq has a closed-form expression , then so does the minimizer pηpwq . The simplicity of this form means that iteratively reweighted Tikhonov regularization is simple to implement for any differentiable penalty , leading to its popularity as a sparse optimization technique . We enumerate a few penalties along with their dual formulations in Table 1 . For separable penalties , we list the scalar function applied to a single element . Where H is restricted , we include it with fpηq . Hard Thresholding . Of particular note is HARDTHRESH , the indicator penalty of level sets of } w } 0 , which gives rise to the iterative hard thresholding ( IHT ) algorithm [ 7 ] . We remark that to the best of our knowledge , what we show in Table 1 is the first characterization of IHT as an iteratively reweighted ` 2 solution to a regularized ERM problem , rather than as a heuristic algorithm . This connection is important for understanding the MAGNITUDEPRUNING [ 45 ] algorithm in Section 4.4 .

Summary: This paper examines stochastic perturbation methods, such as dropout, through the lens of the "eta-trick".  This trick allows the optimization objective to be re-written as a joint objective in terms of the original parameters and auxiliary parameters eta.  This new objective consists of the original loss in addition to a quadratic penalty in the parameters and a formulation-specific function of eta.  The paper shows that any adaptive dropout method that updates the dropout parameters monotonically as a function of the weight magnitudes is equivalent to regularized risk minimization with a subquadratic penalty, which is known to induce sparsity.  The paper analyses standout, variational dropout, L0 regularization, and magnitude pruning from this perspective, deriving the corresponding penalty term (Table 2) and allowing for comparison.  The paper then experimentally compares variants of variational dropout, finding that performance is well-described by the eta-trick reformulation since stochastic variants suffer from the introduced noise.