Linear Convergence and Implicit Regularization of Generalized Mirror Descent with Time-Dependent Mirrors

1 INTRODUCTION . Recent work has established the optimization and generalization benefits of over-parameterization in machine learning ( Belkin et al. , 2019 ; Liu et al. , 2020 ; Zhang et al. , 2017 ) . In particular , several works including Vaswani et al . ( 2019 ) ; Du et al . ( 2018 ) ; Liu et al . ( 2020 ) ; Li & Liang ( 2018 ) have demonstrated that over-parameterized models converge to a global minimum when trained using stochastic gradient descent and that such convergence can occur at a linear rate . Independently , other work , such as Gunasekar et al . ( 2018 ) , have characterized implicit regularization of overparameterized models , i.e. , the properties of the solution selected by a given optimization method , without proving convergence . Recently , Azizan & Hassibi ( 2019 ) ; Azizan et al . ( 2019 ) simultaneously proved convergence and analyzed approximate implicit regularization for mirror descent ( Beck & Teboulle , 2003 ; Nemirovsky & Yudin , 1983 ) . In particular , by using the fundamental identity of stochastic mirror descent ( SMD ) , they proved that SMD converges to an interpolating solution that is approximately the closest one to the initialization in Bregman divergence . However , these works do not provide a rate of convergence for SMD and assume that there exists an interpolating solution within in Bregman divergence from the initialization . In this work , we provide sufficient conditions for linear convergence and obtain approximate implicit regularization results for generalized mirror descent ( GMD ) , an extension of mirror descent that introduces ( 1 ) a potential-free update rule and ( 2 ) a time-dependent mirror ; namely , GMD with invertible φ : Rd → Rd and learning rate η is used to minimize a real valued loss function , f , according to the update rule : φ ( t ) ( w ( t+1 ) ) = φ ( t ) ( w ( t ) ) − η∇f ( w ( t ) ) . ( 1 ) We discuss the stochastic version of GMD ( SGMD ) in Section 3 . GMD generalizes both mirror descent and preconditioning methods . Namely , if for all t , φ ( t ) = ∇ψ for some strictly convex function ψ , then GMD corresponds to mirror descent with potential ψ ; if φ ( t ) = G ( t ) for some invertible matrix G ( t ) ∈ Rd×d , then the update rule in equation ( 1 ) reduces to w ( t+1 ) = w ( t ) − ηG ( t ) −1 ∇f ( w ( t ) ) and hence represents applying a pre-conditioner to gradient updates . The following is a summary of our results : 1 . We provide a simple proof for linear convergence of GMD under the Polyak-Lojasiewicz inequality ( Theorem 1 ) . 2 . We provide sufficient conditions under which SGMD converges linearly under an adaptive learning rate ( Theorems 2 and 3 ) 1 . 3 . As corollaries to Theorems 1 and 3 , in Section 5 we provide sufficient conditions for linear convergence of stochastic mirror descent as well as stochastic preconditioner methods such as Adagrad ( Duchi et al. , 2011 ) . 4 . We prove the existence of an interpolating solution and linear convergence of GMD to this solution for non-negative loss functions that locally satisfy the PL * inequality ( Liu et al. , 2020 ) . This result ( Theorem 4 ) provides approximate implicit regularization results for GMD : GMD converges linearly to an interpolating solution that is approximately the closest interpolating solution to the initialization in ` 2 norm in the dual space induced by φ ( t ) . 2 RELATED WORK . Recent work ( Azizan et al. , 2019 ) established convergence of stochastic mirror descent ( SMD ) for nonlinear optimization problems . It characterized the implicit bias of mirror descent by demonstrating that SMD converges to a global minimum that is within epsilon of the closest interpolating solution in Bregman divergence . The analysis in Azizan et al . ( 2019 ) relies on the fundamental identity of SMD and does not provide explicit learning rates or establish a rate of convergence for SMD in the nonlinear setting . The work in Azizan & Hassibi ( 2019 ) provided explicit learning rates for the convergence of SMD in the linear setting under strongly convex potential , again without a rate of convergence . While these works established convergence of SMD , prior work by Gunasekar et al . ( 2018 ) analyzed the implicit bias of SMD without proving convergence . A potential-based version of generalized mirror descent with time-varying regularizes was presented for online problems in Orabona et al . ( 2015 ) . That work is primarily concerned with establishing regret bounds for the online learning setting , which differs from our setting of minimizing a loss function given a set of known data points . A potential-free formulation of GMD for the flow was presented in Gunasekar et al . ( 2020 ) . The Polyak-Lojasiewicz ( PL ) inequality ( Lojasiewicz , 1963 ; Polyak , 1963 ) serves as a simple condition for linear convergence in non-convex optimization problems and is satisfied in a number of settings including over-parameterized neural networks ( Liu et al. , 2020 ) . Work by Karimi et al . ( 2016 ) demonstrated linear convergence of a number of descent methods ( including gradient descent ) under the PL inequality . Similarly , Vaswani et al . ( 2019 ) proved linear convergence of stochastic gradient descent ( SGD ) under the PL inequality and the strong growth condition ( SGC ) , and Bassily et al . ( 2018 ) established the same rate for SGD under just the PL inequality . Soltanolkotabi et al . ( 2019 ) also used the PL inequality to establish a local linear convergence result for gradient descent on 1 hiddden layer over-parameterized neural networks . Recently , Xie et al . ( 2020 ) established linear convergence for a norm version of Adagrad ( AdagradNorm ) using the PL inequality , while Wu et al . ( 2019 ) established linear convergence for AdagradNorm in the particular setting of over-parameterized neural networks with one hidden layer . An alternate analysis for Adagrad-Norm for smooth , non-convex functions was presented in Ward et al . ( 2019 ) , resulting in a sub-linear convergence rate . 1We also provide a fixed learning rate for monotonically decreasing gradients∇f ( w ( t ) ) . Instead of focusing on a specific method , the goal of this work is to establish sufficient conditions for linear convergence by applying the PL inequality to a more general setting ( SGMD ) . We arrive at linear convergence for specific methods such as mirror descent and preconditioned gradient descent methods as corollaries . Moreover , our local convergence results provide an intuitive formulation of approximate implicit regularization for GMD and thus mirror descent . Namely , instead of resorting to Bregman divergence , we prove that GMD converges to an interpolating solution that is approximately the closest interpolating solution to the initialization in ` 2 norm in the dual space induced by φ ( t ) . 3 ALGORITHM DESCRIPTION AND PRELIMINARIES . We begin with a formal description of SGMD . Let fi : Rd → R denote real-valued , differentiable loss functions and let f ( x ) = 1n ∑n i=1 fi ( x ) . In addition , let φ ( t ) : Rd → Rd be an invertible function for all non-negative integers t. We solve the optimization problem arg min x∈Rd f ( x ) using stochastic generalized mirror descent with learning rate η2 : φ ( t ) ( w ( t+1 ) ) = φ ( t ) ( w ( t ) ) − η∇fit ( w ( t ) ) , ( 2 ) where it ∈ [ n ] is chosen uniformly at random . As described in the introduction , the above algorithm generalizes both gradient descent ( where φ ( x ) = x ) and mirror descent ( where φ ( t ) ( x ) = ∇ψ ( x ) for some strictly convex potential function ψ ) . In the case where φ ( t ) ( x ) = G ( t ) x for an invertible matrix G ( t ) ∈ Rd×d , the update rule in equation ( 2 ) reduces to : w ( t+1 ) = w ( t ) − ηG ( t ) −1 ∇fit ( w ( t ) ) Hence , when φ ( t ) is an invertible linear transformation , Equation ( 2 ) is equivalent to pre-conditioned gradient descent . We now present the Polyak-Lojasiewicz inequality and lemmas from optimization theory that will be used in our proofs3 . Polyak-Lojasiewicz ( PL ) Inequality . A function f : Rd → R is µ-PL if for some µ > 0 : 1 2 ‖∇f ( x ) ‖2≥ µ ( f ( x ) − f ( x∗ ) ) ∀x ∈ Rd , ( 3 ) where x∗ ∈ Rd is a global minimizer for f . A useful variation of the PL inequality is the PL * inequality introduced in Liu et al . ( 2020 ) which does not require knowledge of f ( x∗ ) . Definition . A function f : Rd → R is µ-PL * if for some µ > 0 : 1 2 ‖∇f ( x ) ‖2≥ µf ( x ) ∀x ∈ Rd , ( 4 ) A function that is µ-PL * is also µ-PL when f is non-negative . Additionally , we will typically assume that f is L-smooth ( with L-Lipschitz continuous derivative ) . Definition . A function f : Rd → R is L-smooth for L > 0 if for all x , y ∈ Rd : ‖∇f ( x ) −∇f ( y ) ‖≤ L‖x− y‖ . If φ ( t ) ( x ) = x for any t and x ∈ Rd then SGMD reduces to SGD . If f is L-smooth and satisfies the PL-Inequality , then SGD converges linearly to a global minimum ( Bassily et al. , 2018 ; Karimi et al. , 2016 ; Vaswani et al. , 2019 ) . Moreover , the following lemma ( proven in Appendix A ) shows that the PL * condition implies the existence of a global minimum x∗ for non-negative , L-smooth f . Lemma 1 . If f : Rd → R is µ-PL * , L-smooth and f ( x ) ≥ 0 for all x ∈ Rd , then gradient descent with learning rate η < 2L converges linearly to x ∗ satisfying f ( x∗ ) = 0 . 2The framework also allows for adaptive learning rates by using η ( t ) to denote a time-dependent step size . 3We assume all norms are the 2-norm unless stated otherwise . Hence , in cases where the loss function is nonnegative ( for example the squared loss ) , we can remove the usual assumption about the existence of a global minimum , x∗ , and instead assume that f satisfies the PL * inequality . We now reference standard properties of L-smooth functions ( Zhou , 2018 ) , which will be used in our proofs . Lemma 2 . If f : Rd → R is L-smooth , then for all x , y ∈ Rd : ( a ) f ( y ) ≤ f ( x ) + 〈∇f ( x ) , y − x〉+ L 2 ‖y − x‖2 , ( b ) ‖∇f ( x ) ‖2≤ 2L ( f ( x ) − f ( x∗ ) ) . The following lemma relates µ and L ( the proof is in Appendix B ) . Lemma 3 . If f : Rd → R is µ-PL and L-smooth , then µ ≤ L. Using Lemma 2b in place of the strong growth condition ( i.e . Ei [ ‖∇fi ( x ) ‖2 ] ≤ ρ‖∇f ( x ) ‖2 ) yields slightly different learning rates when establishing convergence of stochastic descent methods ( as is apparent from the different learning rates between Bassily et al . ( 2018 ) and Vaswani et al . ( 2019 ) ) . The following simple lemma will be used in the proof of Theorem 3 . Lemma 4 . If f ( x ) = 1n ∑n i=1 fi ( x ) where fi : Rd → R are Li-smooth , then f is supi Li-smooth . Note that there could exist some other constant L′ < supi Li for which f is L ′-smooth , but this upper bound suffices for our proof of Theorem 3 . Lastly , we define and reference standard properties of strongly convex functions ( Zhou , 2018 ) , which will be useful in demonstrating how our GMD results generalize those for mirror descent . Definition . For α > 0 , a differentiable function , ψ : Rd → R , is α-strongly convex if for all x , y , ψ ( y ) ≥ ψ ( x ) + 〈∇ψ ( x ) , y − x〉+ α 2 ‖y − x‖2 . Lemma 5 . If ψ : Rd → R is α-strongly convex , then for all x , y : ψ ( y ) ≤ ψ ( x ) + 〈∇ψ ( x ) , y − x〉+ 1 2α ‖∇ψ ( y ) −∇ψ ( x ) ‖2 . With these preliminaries in hand , we now present our proofs for linear convergence of SGMD using the PL-Inequality .

Summary: This paper studies the interesting property of generalized mirror descent (GMD) and its stochastic variant for nonconvex optimization problems. First, for GMD this paper shows the linear convergence under PL* condition (in Lemma 1) and finds out a new sufficient condition for the linear convergence (in Theorem 2). Next, this work tried to extend this result to a stochastic setting (in Theorem 3). Moreover, the implicit regularization of GMD is studied, which is an extension of the previous studies by [Azizan et al.].