Dropout: Explicit Forms and Capacity Control

1 INTRODUCTION . Dropout is a popular regularization technique for training deep neural networks that aims at “ breaking co-adaptation ” among neurons by randomly dropping them at training time ( Hinton et al. , 2012 ) . Dropout has been shown effective across a wide range of machine learning tasks , from classification ( Srivastava et al. , 2014 ; Szegedy et al. , 2015 ) to regression ( Toshev & Szegedy , 2014 ) . Notably , it is considered as a major component in the design of AlexNet ( Krizhevsky et al. , 2012 ) , which won the prominent ImageNet challenge in 2012 with a significant margin and helped transform the field of computer vision . Following the empirical success of dropout , there have been several studies in recent years that focus on understanding theoretical underpinnings of dropout ( Baldi & Sadowski , 2013 ; Wager et al. , 2013 ; McAllester , 2013 ; Van Erven et al. , 2014 ; Helmbold & Long , 2015 ; Gal & Ghahramani , 2016 ; Gao & Zhou , 2016 ; Mou et al. , 2018 ; Bank & Giryes , 2018 ; Cavazza et al. , 2018 ; Mianjy et al. , 2018 ) . However , none of these works adequately address the following basic question : how does dropout control the capacity of deep neural networks ? In this paper , we provide an answer to this question . We focus on the task of deep regression using squared ` 2-loss which yields state-ofthe-art results in human pose estimation ( Toshev & Szegedy , 2014 ) , facial landmark detection , age estimation ( Lathuilière et al. , 2019 ) , and more . We give precise generalization bounds for deep regression with dropout ; our bounds leverage recent results in understanding generalization in deep learning and help explain practice with meaningful bounds . Along the way we also recover the state-of-the-art bounds for matrix completion . The key contributions in this paper are as follows . 1 . We introduce dropout for matrix completion , a procedure that randomly drops the columns of the factors during training . Interestingly , this algorithmic procedure induces a datadependent regularizer that , in expectation , equals weighted trace-norm – a complexity measure that enjoys strong generalization guarantees ( Foygel et al. , 2011 ) . 2 . For two-layer neural networks with ReLU activation , when the input distribution is symmetric and isotropic , we show that dropout induces the ` 2 path-norm which has been shown to provide scale-sensitive generalization guarantees in deep neural networks ( Neyshabur et al. , 2015 ) . 3 . We study deep regression , where a feed-forward neural network is trained with dropout under squared loss . With no further assumptions on the data distribution and the network architecture , we show that the dropout regularizer is a data-dependent measure whose expected value serves as a strong complexity measure for networks trained with dropout . In particular , we show that a network with a small dropout regularizer enjoys a small generalization gap . 4 . We empirically evaluate our theoretical findings for matrix completion and deep regression on real world datasets MovieLens , Fashion MNIST and CIFAR-10 . 1.1 NOTATION . We denote matrices , vectors , scalar variables and sets by Roman capital letters , Roman small letters , small letters , and script letters , respectively ( e.g . X , x , x , and X ) . For any integer d , we represent the set { 1 , . . . , d } by [ d ] . For any vector x ∈ Rd , diag ( x ) ∈ Rd×d represents the diagonal matrix with diagonal elements equal to x , and √ x is the elementwise squared root of x . Let ‖x‖ represent the ` 2-norm of vector x , and ‖X‖ , ‖X‖F , and ‖X‖∗ represent the spectral norm , the Frobenius norm , and the nuclear norm of matrix X , respectively . Similarly , given a positive definite matrix C , we denote the Mahalonobis norm as ‖x‖2C = x > Cx . The standard inner product is represented by 〈· , ·〉 , for vectors or matrices , where 〈X , X′〉 = Tr ( X > X′ ) . 2 MATRIX SENSING . We begin with understanding dropout for matrix sensing , a problem which arguably is an important instance of a matrix learning problem with lots of applications , and is well understood from a theoretical perspective . Here is the problem setup . Let M∗ ∈ Rd2×d0 be a matrix with rank r∗ : = Rank ( M∗ ) . Let A ( 1 ) , . . . , A ( n ) be a set of measurement matrices of the same size as M. The goal of matrix sensing is to recover the matrix M∗ from n observations of the form yi = 〈M∗ , A ( i ) 〉 such that n d2d0 . A natural approach to solve this problem is to enforce the rank constraints implicitly using the Burer-Monteiro factorization and to solve the following empirical risk minimization problem : min U∈Rd2×d1 , V∈Rd0×d1 L̂ ( U , V ) : = 1 n n∑ i=1 ( yi − 〈UV > , A ( i ) 〉 ) 2 . ( 1 ) When d1 r∗ , there exist many “ bad ” empirical minimizers , i.e. , those with a large true risk . However , recently , Li et al . ( 2018 ) showed that under restricted isometry property , despite the existence of such poor ERM solutions , gradient descent with proper initialization is implicitly biased towards finding solutions with minimum nuclear norm – this is an important result which was first conjectured and empirically verified by Gunasekar et al . ( 2017 ) . However , for the most part , modern machine learning systems employ explicit regularization techniques . In fact , as we show in the experimental section , the implicit bias due to ( stochastic ) gradient descent does not prevent it from blatant overfitting in the matrix completion problem . We argue that in such cases , there is a need for “ suitable ” explicit regularization , such as trace-norm Recht et al . ( 2010 ) ; Candès & Recht ( 2009 ) , weighted trace-norm Foygel et al . ( 2011 ) , or max norm Srebro & Shraibman ( 2005 ) . We propose solving the ERM problem ( 1 ) with algorithmic regularization due to dropout , where at training time , columns of U and V are dropped uniformly at random . As opposed to the implicit effect of gradient descent , this dropout heuristic explicitly regularizes the empirical objective . It is then natural to ask , in the case of matrix sensing , if dropout also biases the ERM towards certain low norm solutions . To answer this question , we begin with the observation that dropout can be viewed as an instance of SGD on the following objective : L̂drop ( U , V ) = 1 n n∑ j=1 EB ( yj − 〈UBV > , A ( j ) 〉 ) 2 , ( 2 ) where B ∈ Rd1×d1 is a diagonal matrix whose diagonal elements are Bernoulli random variables distributed as Bii ∼ 11−pBer ( 1− p ) . In this case , it is easy to show that for any p ∈ [ 0 , 1 ) : L̂drop ( U , V ) = L̂ ( U , V ) + p 1− p R̂ ( U , V ) , R̂ ( U , V ) = d1∑ i=1 1 n n∑ j=1 ( u > i A ( j ) vi ) 2 where R̂ ( U , V ) is a data-dependent term that captures the explicit regularizer due to dropout ( see Proposition 1 in the Appendix ) . Provided that the sample size n is large enough , the explicit regularizer is well concentrated around its mean ( see Lemma 2 in the Appendix ) . Further , given that we seek a minima of L̂drop , it suffices to consider the factors with the minimal value of the regularizer among all that yield the same empirical loss . This motivates studying the the following distribution-dependent induced regularizer : Θ ( M ) : = min UV > =M R ( U , V ) , where R ( U , V ) : = EA [ R̂ ( U , V ) ] . Surprisingly , for a wide range of random measurements , Θ ( · ) turns out to be a “ suitable ” regularizer . Here , we instantiate two important examples ( see Proposition 2 in the Appendix ) . Gaussian measurements . For all j ∈ [ n ] , let A ( j ) be standard Gaussian matrices . Then Θ ( M ) = 1 d1 ‖M‖2∗ is the standard trace-norm regularization studied in Srebro et al . ( 2005 ) ; Bach ( 2008 ) ; Candès & Tao ( 2009 ) . Matrix completion . For all j ∈ [ n ] , let A ( j ) be an indicator matrix whose ( i , k ) -th element is selected randomly with probability p ( i ) q ( k ) , where p ( i ) and q ( k ) denote the probability of choosing the i-th row and the j-th column , respectively . Then Θ ( M ) = 1 d1 ‖diag ( √p ) UV > diag ( √q ) ‖2∗ is the weighted trace-norm studied by Srebro & Salakhutdinov ( 2010 ) and Foygel et al . ( 2011 ) . These observations are specifically important because they connect dropout , an algorithmic heuristic in deep learning , to strong complexity measures that are empirically effective as well as theoretically well understood . To illustrate , here we give a generalization bound for matrix completion with dropout in terms of the value of the explicit regularizer at the minimum of the empirical problem . Theorem 1 . Without loss of generality , assume that d2 ≥ d0 and ‖M∗‖ ≤ 1 . Furthermore , assume that mini , j p ( i ) q ( j ) ≥ log ( d2 ) n√d2d0 . Let ( U , V ) be a minimizer of the dropout ERM objective in equation ( 2 ) , and assume that maxi ‖U ( i , : ) ‖2 ≤ γ , maxi ‖V ( i , : ) ‖2 ≤ γ . Let α be such that R̂ ( U , V ) ≤ α/d1 . Then , for any δ ∈ ( 0 , 1 ) , the following generalization bounds holds with probability at least 1− 2δ over a sample of size n : L ( U , V ) ≤ L̂ ( U , V ) + C ( 1 + γ ) √ αd2 log ( d2 ) n + C ′ ( 1 + γ2 ) √ log ( 2/δ ) 2n as long as n = Ω ( ( d1γ 2/α ) 2 log ( 2/δ ) ) , where C , C ′ are some absolute constants . The proof of Theorem 1 follows from standard generalization bounds for ` 2 loss ( Mohri et al. , 2018 ) based on the Rademacher complexity ( Bartlett & Mendelson , 2002 ) of the class of functions with weighted trace-norm bounded by √ α , i.e.Mα : = { M : ‖ diag ( √ p ) M diag ( √ q ) ‖2∗ ≤ α } . A bound on the Rademacher complexity of this class was established by Foygel et al . ( 2011 ) . The technicalities here include showing that the explicit regularizer is well concentrated around its expected value , as well as deriving a bound on the supremum of the predictions . A few remarks are in order . We require that the sampling distributions be non-degenerate , as specified by the condition mini , j p ( i ) q ( j ) ≥ log ( d2 ) n√d2d0 . This is a natural requirement for bounding the Rademacher complexity ofMα , as discussed in Foygel et al . ( 2011 ) . We note that for large enough sample size , R̂ ( U , V ) ≈ R ( U , V ) ≈ Θ ( UV > ) = 1 d1 ‖diag ( √p ) UV > diag ( √q ) ‖2∗ , where the second approximation is due the fact that the pair ( U , V ) is a minimizer . That is , compared to the weighted trace-norm , the value of the explicit regularizer at the minimizer roughly scales as 1/d1 . Hence the assumption R̂ ( U , V ) ≤ α/d1 in the statement of the corollary . In practice , for models that are trained with dropout , the training error L̂ ( U , V ) is negligible ( see Figure 1 for experiments on the MovieLens dataset ) . Moreover , given that the sample size is large enough , the third term can be made arbitrarily small . Having said that , the second term , which is Õ ( γ √ αd2/n ) , dominates the right hand side of generalization error bound in Theorem 6 . The assumption maxi ‖U ( i , : ) ‖2 ≤ γ , maxi ‖V ( i , : ) ‖2 ≤ γ is motivated by the practice of deep learning ; such max-norm constraints are typically used with dropout , where the norm of the vector of incoming weights at each hidden unit is constrained to be bound by a constant ( Srivastava et al. , 2014 ) . In this case , if a dropout update violates this constraint , the weights of the hidden unit are projected back to the constraint norm ball . In proofs , we need this assumption to give a concentration bound for the empirical explicit regularizer , as well as bound the supremum deviation between the predictions and the true values . We remark that the value of γ also determines the complexity of the function class . On one hand , the generalization gap explicitly depends on and increases with γ . However , when γ is large , the constraints on U , V are milder , so that L̂ ( U , V ) can be made smaller . Finally , the required sample size heavily depends on the value of the explicit regularizer at the optima ( α/d1 ) , and hence , on the dropout rate p. In particular , increasing the dropout rate increases the regularization parameter λ : = p1−p , thereby intensifies the penalty due to the explicit regularizer . Intuitively , a larger dropout rate p results in a smaller α , thereby a tighter generalization gap can be guaranteed . We show through experiments that that is indeed the case in practice .

Summary: This work studies the effect of regularization through dropout on generalization bounds for Matrix Sensing and linear regression with deep neural network task. For both matrix sensing task and linear regression task, authors derive an explicit regularization term due to dropout. Authors give elegant interpretations of dropout regularizer. For matrix sensing: dropout can be thought of as the inducing trace-norm regularization when matrices are standard gaussian. For matrix completion: dropout can be thought of as weighted trace-norm regularizer.