The Flip Side of the Reweighted Coin: Duality of Adaptive Dropout and Regularization

1 Introduction . In machine learning , it is often valuable for models to be parsimonious or sparse for a variety of reasons , from memory savings and computational speedups to model interpretability and generalizability . Classically , this is achieved by solving a regularized empirical risk minimization ( ERM ) problem of the form minimize w Lpwq ` λΩpwq , ( 1 ) where L is the data-dependent empirical risk and Ω is a sparsity-inducing regularization penalty . Ideally , Ωpwq is equal to } w } 0 : “ # ti : wi ‰ 0u or a function of } w } 0 [ 6 ] , in which case an appropriately chosen λ balances the trade-off between the suboptimality of L and the sparsity of the solution . However , the use of } w } 0 directly as a penalty is difficult , as it makes local search impossible , so alternative approaches are necessary for high-dimensional problems . The classical solution to this is to relax the problem using a smooth surrogate Ω that approximates } w } 0 [ 5 , 18 ] . When we know the regularization penalty Ω , we can understand the types of solutions they encourage ; however , many popular methods for sparsifying deep ( neural ) networks do not appear to fit the form ( 1 ) . For example , variational dropout [ 27 , 32 ] , “ ` 0 regularization ” [ 29 ] , and magnitude pruning [ 45 ] are the only methods compared in a recent large-scale study by Gale et al . [ 15 ] , and all achieve sparsity by adaptively masking or scaling the parameters of the network while optimizing the loss function in w. Although these methods are well-motivated heuristically , it is unclear how to make a principled choice between them aside from treating them as black boxes and comparing their empirical performance . However , if these methods could be cast as solutions to a regularized ERM problem , then we could compare them on the basis of their regularization penalties . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We show that , in fact , these sparsity methods do correspond to regularization penalties Ω , which we can obtain , compute , and compare . As we show in Figure 1 , these penalties bear striking resemblance to classical sparsity-inducing penalties such as the LOGSUM [ 9 ] penalty and the minimax concave penalty ( MCP ) [ 44 ] . More broadly , we analyze adaptive dropout methods , which apply the dropout technique of Srivastava et al . [ 35 ] with adaptive parameters . By considering adaptive dropout in the linear setting , we uncover a duality between adaptive dropout methods and regularized ERM . We make this connection via the “ η-trick ” [ 4 ] , a tool with a long history of application in sparse optimization via iteratively reweighted least squares ( IRLS ) ( see the history presented by Daubechies et al . [ 11 ] ) . We prove that all adaptive dropout methods whose amount of dropout varies monotonically with the magnitudes of the parameters induce an effective subquadratic ( and hence sparsity-inducing [ 5 ] ) penalty Ω . This further supports the experimental evidence that such methods excel at inducing sparsity . We also demonstrate how to use our result to determine the effective penalty for adaptive dropout methods , using as examples the sparsity methods listed above as well as the standout method [ 3 ] , which has an effective penalty on the layer activations , rather than the parameters , explaining why the standout method sparsifies activations . We then numerically compute the effective penalties1 and plot them together in Figure 1 . We validate our new theory by applying variational dropout on the task of deep network sparsification , and we show that the performance is similar to non-dropout methods based on the same effective penalty . This suggests that not only is considering the effective penalty a sound means for comparing sparsity methods , but also that classical regularized ERM is itself an effective means for inducing sparsity in deep networks and serves as a valuable tool for future work on deep network sparsity . For theoreticians , this work provides a general framework for obtaining the effective regularization penalty for adaptive dropout algorithms , enabling regularized ERM analysis tools to be applied to these methods . For practitioners , this work enables the application of classical regularization intuition when choosing between sparsifying approaches . For methods developers , this work provides a strong baseline against which new adaptive dropout methods should be compared : the effective penalty Ω . 2 Background . Notation . We denote the extended reals and the non-negative reals by R “ R Y t´8,8u and R ` “ r0,8q , respectively . The operator diag p¨q takes a vector and returns a matrix with that vector along the diagonal entries or takes a matrix and returns the same matrix with all non-diagonal entries set to zero . The matrix J denotes a matrix of all ones . We denote element-wise multiplication and exponentiation by d ; i.e. , ru d vsj “ ujvj and rUdpsij “ Upij . Division of vectors and scalar functions of vectors denote element-wise operations . Order statistics of the magnitudes of 1Our code is available at https : //github.com/dlej/adaptive-dropout . elements in a vector u P Rd are denoted by upjq such that |up1q| ě |up2q| ě . . . |updq| . The function σ : RÑ r0 , 1s denotes the sigmoid σptq “ 1 { p1 ` e´tq . 2.1 Dropout as Tikhonov Regularization . We can understand adaptive dropout methods by breaking them down into two components : the regularizing effect of dropout itself and the effect of adaptively updating the dropout-induced regularization . For the former , it is well-known [ 40 , 39 , 35 , 32 ] that dropout induces Tikhonov-like regularization for standard binary and Gaussian dropout . The same result holds more broadly given a second moment condition , which we need in order to be able to consider methods like HARDCONCRETE “ ` 0 regularization ” [ 29 ] . In general , we use the term dropout to refer to methods that mask or scale the weights w by independent “ mask ” variables s „ MASKpαq with sampling parameters α P r0 , 1sd . We emphasize that in this formulation , it is the parameters that are masked , rather than the nodes of a network as in the original formulation of dropout [ 35 ] , although in linear regression there is no difference . We assume that the masks are unbiased ( i.e. , that E rsjs “ 1 ) , and that E “ s2j ‰ “ α´1j . Note that with the independence assumption , this implies that Covpsq “ diag pαq´1 ´ I . Both standard unbiased binary dropout , where sj take value α´1j with probability αj and 0 otherwise , as well as Gaussian dropout with sj „ N ` 1 , α´1j ´ 1 ˘ , satisfy these assumptions . Dropout solves the optimization problem minimize w Es rLpsdwqs ( 2 ) via stochastic optimization such as gradient descent where each iteration performs a partial optimization of Lps d wq with a random s. It is most insightful to consider when L is the loss for linear regression of target variables y P Rn given data X P Rnˆd : Lpwq “ 1 2n } y ´Xw } 22 . ( 3 ) In this case , under our assumptions on s , we recover the known Tikhonov result [ e.g. , 32 ] Es rLpsdwqs “ Lpwq ` 1 2 wJ ˆ ´ diag pαq´1 ´ I ¯ d 1 n XJX ˙ w. ( 4 ) For simplicity , we assume that the data is standardized , or that diag ` 1 nX JX ˘ “ I , which is inexpensive to satisfy in practice , but our analysis can be extended to the general case . Under this assumption , we see that dropout elicits a diagonal Tikhonov regularization with scale diag pαq´1´ I . 2.2 The “ η-trick ” . The second component of adaptive dropout is the adaptive update of the dropout-induced regularization . To understand this , we introduce the so-called “ η-trick ” [ 4 ] applied to the regularization penalty in ( 1 ) . By introducing an auxiliary variable η P H Ď Rd ` , we can replace Ωpwq with a dual formulation as a function of η that majorizes Ωpwq and is quadratic in w for fixed η . In other words , we can find a function f : H Ď Rd ` Ñ R such that Ωpwq “ min ηPH 1 2 ´ wJdiag pηq´1 w ` fpηq ¯ . ( 5 ) If such a function f exists , then the regularized ERM problem ( 1 ) can be rewritten as a joint optimization in w and η of the dual regularized ERM formulation minimize w , η Lpwq ` λ 2 ´ wJdiag pηq´1 w ` fpηq ¯ . ( 6 ) This joint optimization can be performed , for example , by alternating minimization over w and η . For linear regression problems , this gives rise to the iteratively reweighted least squares ( IRLS ) algorithm , a popular method for sparse recovery [ 10 , 11 ] . We note that iteratively reweighted ` 1 schemes have also held a significant place in sparse optimization [ 47 , 9 , 42 ] , but because dropout regularization is quadratic , we limit our consideration to iteratively reweighted ` 2 ( Tikhonov ) regularization like ( 6 ) . Table 1 : Common regularization penalties have dual formulations that are often straightforward to obtain . See The simplest example of a penalty Ω that fits the form ( 5 ) is Ωpwq “ |w| , for which fpηq “ η , and it is used in the IRLS formulation of ` 1 regularization . More broadly , we can consider the class of functions for which Ω is concave in wd2 . These functions are said to be subquadratic , and are known to have a sparsity-inducing effect [ 5 ] . In fact , these are the only functions Ω that can fit this form . This is because wd2 ÞÑ ´2Ωpwq is the Legendre–Fenchel ( LF ) transform of ´ηd´1 ÞÑ fpηq , and thus Ω is concave in wd2 . By the Fenchel–Moreau theorem , there is conversely a unique function f satisfying ( 5 ) that is convex in ´ηd´1 , which can be obtained by the LF transform . That is , Ω and f are dual functions . Additionally , for differentiable penalties , the minimizing η given a fixed w is pηpwq “ p2∇wd2Ωpwqq d´1 , ( 7 ) which means that if Ωpwq has a closed-form expression , then so does the minimizer pηpwq . The simplicity of this form means that iteratively reweighted Tikhonov regularization is simple to implement for any differentiable penalty , leading to its popularity as a sparse optimization technique . We enumerate a few penalties along with their dual formulations in Table 1 . For separable penalties , we list the scalar function applied to a single element . Where H is restricted , we include it with fpηq . Hard Thresholding . Of particular note is HARDTHRESH , the indicator penalty of level sets of } w } 0 , which gives rise to the iterative hard thresholding ( IHT ) algorithm [ 7 ] . We remark that to the best of our knowledge , what we show in Table 1 is the first characterization of IHT as an iteratively reweighted ` 2 solution to a regularized ERM problem , rather than as a heuristic algorithm . This connection is important for understanding the MAGNITUDEPRUNING [ 45 ] algorithm in Section 4.4 .

Summary: For linear regression, the paper shows that several popular heuristics used in deep learning correspond to regularized empirical risk minimization with sub-quadratic penalties, thereby, arguing that these methods indeed induce sparsity. The analysis leverages the so-called \eta-trick, where the penalty is replaced with its variational (minimization) form. Empirical results are also provided to support the theory.