DASGrad: Double Adaptive Stochastic Gradient

1 INTRODUCTION AND MOTIVATION . Stochastic gradient descent ( SGD ) is a widely used optimization method , and currently through backpropagation this algorithm has propelled the success of many deep learning applications . The Deep Learning community has particularly adopted variants of adaptive moment methods for SGD that specialize in high-dimensional features and non convex objectives , examples include ADAGRAD , ADADELTA , RMSPROP , ADAM and AMSGRAD ( Duchi et al . ( 2011 ) ; Zeiler ( 2012 ) ; Tieleman & Hinton ( 2012 ) ; Kingma & Ba ( 2014 ) ; Reddi et al . ( 2018 ) ) . All these adaptive moment methods relied on the efficient use of the information of the geometry of the problem to improve the rate of convergence . In parallel to the previous ideas , adaptive probabilities methods for SGD , traditionally focusing on convex objectives , have shown advantages over its uniform sampling baselines , by allowing a more efficient use of the gradient information ( Zhu ( 2018 ) ; Shen et al . ( 2016 ) ; Bottou et al . ( 2016 ) ; Csiba et al . ( 2015 ) ; Stich et al . ( 2017 ) ) . Adaptive probabilities were introduced to the Deep Learning community by Hinton ( 2007 ) through the discriminative fine tuning procedure . The method was further explored in a range of applications like object detection , reinforcement learning and curriculum learning ( Shrivastava et al . ( 2016 ) ; Schaul et al . ( 2015 ) ; Bengio et al . ( 2009 ) ) . In these examples the combination of adaptive moments and adaptive probabilities methods was implicit and its analysis as a pure optimization technique was still an open question . In this paper we analyze the asymptotic convergence properties of combining adaptive probabilities and adaptive moments . To our knowledge such family has not yet been introduced as an optimization procedure . We will refer to this family of optimization algorithms as Double Adaptive Stochastic Gradient DASGRAD . We show that the improvements of DASGRAD depend on its flexibility to control the variance of the adaptive moments methods . We prove the theoretical guarantees and improvements in a convex setting and validate these observations empirically in convex and deep learning objectives . Finally we demonstrate the generalization properties of the algorithm with a simple extension to importance weight transfer learning . 2 ADAPTIVE GRADIENT METHODS . Notation . In order to facilitate the proofs and reading process we introduce some simplified notation that will be common to the analyzed algorithms . Let a , b ∈ Rd andM ∈ Sd+ , then the multiplication of vector a by the inverse of M will be M−1a = a/M . Let √ a be the element-wise square root of vector a , a2 the element-wise square , a/b the element-wise division , and max ( a , b ) the elementwise max of vector a and vector b . Finally for any natural n the set { 1 , . . . , n } is denoted as [ n ] . Let T = { ( xi , yi ) } ni=1 be a training set ; let f : Θ ×X × Y → R be a differentiable function that represents the empirical risk of an agent over T for the parameters θ ∈ Θ , with Θ ⊆ Rd a convex feasible set of parameters ; let Sd+ the set of positive definite matrices in Rd×d , for a given matrix M ∈ Sd+ and parameter θ′ ∈ Θ ; let ΠΘ , M be the projection operator over the feasible domain defined by ΠΘ , M ( θ′ ) = arg minθ∈Θ ||M1/2 ( θ − θ′ ) || . 1 For the iterative stochastic optimization algorithmA , let it be a sampled index i at step t drawn from the training set indices [ n ] , with it ∼ pt and pt ∈ ∆n+ = { p ∈ Rn : pi > 0 Σipi = 1 } . We denote the evaluated risk f ( θ , xi , yi ) = fi ( θ ) , the complete gradient∇f ( θt ) = 1nΣit∇fit ( θt ) and the stochastic gradient∇fit ( θt ) , analogous a full descent directionmt = 1nΣitmit and a stochastic descent directionmit . Stochastic Optimization Framework . To analyze the convergence of the stochastic optimization algorithm A we use the convex optimization setting where we assume that the objective function is convex with bounded gradients , that is ||∇fi ( θ ) ||∞ ≤ G for all i ∈ [ n ] , θ ∈ Θ , and finally the parameter space Θ has bounded diameter , that is ||θ − θ′||∞ ≤ D for all θ , θ′ ∈ Θ . For our purposes , the algorithm A at time t chooses a distribution over the training set p̂t ∈ ∆n+ , obtains a training example it ∼ p̂t and its importance weights ŵit = ( 1/n ) /pit , then updates its parameters θt ∈ Θ using the available data at time t and the importance weights ŵit to unbias the direction of the gradients . After the update , the algorithm incurs in a loss from an unknown function f ( θt ) . To assess the performance of the algorithm after T steps we use the expected regret , which measures the difference of the loss at time t and the loss for optimal fixed parameter , along the possible trajectories induced by the chosen probabilities . R ( A ) = ∑T t=1 En [ fi ( θt ) −minθEn [ fi ( θ ) ] ] The goal is to design an algorithm A that has sub linear expected regret R ( A ) T = O ( T ) , which in turn implies that the algorithm will converge on average to the optimal parameter . Algorithm 1 : Double Adaptive Methods Input : θ1 ∈ Θ , step size { αt > 0 } Tt=1 , functions { φt , ψt } Tt=1 for t = 1 to T do Choose p̂t ∈ ∆n+ , and sample it ∼ p̂t Calculate git = ∇fit ( θt ) and ŵit = ( 1/n ) /p̂it mit = φt ( gi1 , . . . , git ) and V̂it = ψt ( gi1 , . . . , git ) θ̂t+1 = θt − αtŵit V̂ −1/2 it mit θt+1 = ΠΘ , V̂ 1/2it ( θ̂t+1 ) Algorithm 1 constitutes the general family of double adaptive gradient methods . This algorithm comprehends the classical stochastic gradient descent , adaptive moment methods family Zeiler ( 2012 ) ; Tieleman & Hinton ( 2012 ) ; Kingma & Ba ( 2014 ) ; Reddi et al . ( 2018 ) , and second order methods Duchi et al . ( 2011 ) , varying the averaging functions of the past gradients with φt : Θ t → Rd , and approximating the Hessian matrix with the functions ψt : Θt → Sd+ . 1The projection operator enables the algorithm to deal with constrained optimization over compact convex domains that are equivalent to common regularization techniques like ridge and LASSO . Adaptive Probabilities Methods . The classic stochastic gradient descent algorithm is recovered with the following step size , sampling probabilities and functions : αt = α/ √ t pit = 1/n for all t ∈ [ T ] , i ∈ [ n ] φt ( gi1 , . . . , git ) = git ψt ( gi1 , . . . , git ) = I ( SGD ) Adaptive probabilities methods can be obtained simply by allowing the algorithm to choose a different probability p̂t at any time t : αt = α/ √ t p̂t ∈ ∆n+ for all t ∈ [ T ] φt ( gi1 , . . . , git ) = git ψt ( gi1 , . . . , git ) = I ( AP-SGD ) Significant improvements in the convergence rate of the algorithm can be obtained by cleverly choosing and computing such probabilities that in turn enables the algorithm to use data in a more efficient manner Stich et al . ( 2017 ) . Fixed importance sampling is the case when p̂t = p for all t ∈ [ T ] . Adaptive Moments Methods . Duchi et al . propelled interest and research on adaptive algorithms . In their work they noticed that SGD lacked good convergence behavior in sparse settings , and proposed a family of algorithms that allowed the methods to dynamically incorporate information about the geometry of the data Duchi et al . ( 2011 ) . Following huge gains obtained with ADAGRAD , the deep learning community proposed variants based on exponential moving average functions for ψt like ADADELTA , RMSPROP , ADAM and most recently AMSGRAD ( Zeiler ( 2012 ) ; Tieleman & Hinton ( 2012 ) ; Kingma & Ba ( 2014 ) ; Reddi et al . ( 2018 ) ) . The first algorithm ADAGRAD is obtained by the following proximal functions : αt = 1/ √ t pit = 1/n for all t ∈ [ T ] , i ∈ [ n ] φt ( gi1 , . . . , git ) = git ψt ( gi1 , . . . , git ) = 1 t diag ( Σtτ=1g 2 iτ ) ( ADAGRAD ) The ADAM/AMSGRAD algorithm is obtained by setting : αt = 1/ √ t pit = 1/n ∀t ∈ [ T ] , i ∈ [ n ] φt ( gi1 , . . . , git ) = Σ t τ=1β1 ( t ) τgiτ vit = ( 1− β2 ) Σtτ=1βt−τ2 g2iτ v̂it = max ( v̂it−1 , vit ) ψt ( gi1 , . . . , git ) = diag ( v̂it ) ( ADAM/AMSGRAD ) Fortunately a very simple and computationally efficient way to implement ADAM is given by a recursion . RMSPROP is the particular case of ADAM when β1 = 0 and without maximum operator for the second moments vector , while ADAM is recovered without the maximum operator . Double Adaptive Methods . The key idea behind both the adaptive probabilities methods and adaptive moment methods is the efficient use of the information available in the training data to improve the convergence of the algorithms . In the case of adaptive moment methods the diagonal approximations of the Hessian matrix use the information about the geometry of the problem while the adaptive sampling methods , the probabilities p̂it , prioritize the examples with the highest impact on the learning progress . As these improvements rely on complementary sources of information we can combine them into a general framework described by the double adaptive methods in Algorithm 1 . To analyze the theoretical improvement guarantees of the double adaptive methods we first we extend the adaptive moments convergence guarantees to the stochastic case with uniform sampling and then we compare them to the convergence guarantees using optimal probabilities . 3 CONVERGENCE ANALYSIS . 3.1 CONVERGENCE OF ADAPTIVE MOMENTS METHODS . We first provide a regret bound of ADAM for weakly convex objectives with uniform probabilities adapting the arguments in Reddi et al . ( 2018 ) and Kingma & Ba ( 2014 ) . Then we extend these results to the adaptive probabilities case . Theorem 1 . Let { θt } Tt=1 be the sequence obtained with ADAM , then the regret bound is : R ( ADAM ) ≤ T∑ t=1 1 2αt ( 1− β1t ) En [ ||V̂ 1/4it ( θt − θ ∗ ) ||2 − ||V̂ 1/4it ( θt+1 − θ ∗ ) ||2 ] + T∑ t=1 αt 2 ( 1− β1t ) En [ ||V̂ −1/4it mit || 2 ] + T∑ t=1 αtβ1t 2 ( 1− β1t ) ||V̂ −1/4t mt−1||2 + T∑ t=1 β1t 2αt ( 1− β1t ) ||V̂ 1/4t ( θt − θ∗ ) ||2 Corollary 1.1 . Following the sequence { θt } Tt=1 of ADAM with step size αt = α/ √ t , averaging parameters β1 = β11 , β1t ≤ β1 for all t ∈ [ T ] , γ = β1/ √ β2 < 1 and uniform probabilities pit = 1/n . If we assume that Θ has bounded diameter D , ||∇fit ( θ ) ||∞ ≤ G for all t ∈ [ T ] and θ ∈ Θ , then the expected regret bound is : R ( ADAM ) ≤ D 2 √ T 2α ( 1− β1 ) En [ ||v̂1/4iT || 2 ] + α √ 1 + log ( T ) 2 ( 1− β1 ) 2 √ ( 1− β2 ) ( 1− γ ) d∑ h=1 || ¯| g |1 : T , h || + αGd 2α ( 1− β1 ) 3 √ ( 1− β2 ) ( 1− γ ) + D2 2α ( 1− β1 ) T∑ t=1 √ tβT−t1 ||v̂ 1/4 t ||2

Summary: In this paper, the doubly adaptive stochastic gradient method (DASGrad) is introduced via augmenting adaptive moment methods with adaptive (as opposed to uniform) probabilities for data sampling. The convergence of the proposed method is analyzed in terms of regret bound and is compared to similar results for ADAM. The method is validated with experiments on both convex and non-convex objectives as well as in applications to transfer learning.