Acceleration in Hyperbolic and Spherical Spaces

We further research on the acceleration phenomenon on Riemannian manifolds by introducing the first global first-order method that achieves the same rates as accelerated gradient descent in the Euclidean space for the optimization of smooth and geodesically convex ( g-convex ) or strongly g-convex functions defined on the hyperbolic space or a subset of the sphere , up to constants and log factors . To the best of our knowledge , this is the first method that is proved to achieve these rates globally on functions defined on a Riemannian manifoldM other than the Euclidean space . Additionally , for any Riemannian manifold of bounded sectional curvature , we provide reductions from optimization methods for smooth and gconvex functions to methods for smooth and strongly g-convex functions and vice versa . 1 INTRODUCTION . Acceleration in convex optimization is a phenomenon that has drawn lots of attention and has yielded many important results , since the renowned Accelerated Gradient Descent ( AGD ) method of Nesterov ( 1983 ) . Having been proved successful for deep learning Sutskever et al . ( 2013 ) , among other fields , there have been recent efforts to better understand this phenomenon Allen Zhu & Orecchia ( 2017 ) ; Diakonikolas & Orecchia ( 2019 ) ; Su et al . ( 2016 ) ; Wibisono et al . ( 2016 ) . These have yielded numerous new results going beyond convexity or the standard oracle model , in a wide variety of settings Allen-Zhu ( 2017 ; 2018a ; b ) ; Allen Zhu & Orecchia ( 2015 ) ; Allen Zhu et al . ( 2016 ) ; Allen-Zhu et al . ( 2017 ) ; Carmon et al . ( 2017 ) ; Cohen et al . ( 2018 ) ; Cutkosky & Sarlós ( 2019 ) ; Diakonikolas & Jordan ( 2019 ) ; Diakonikolas & Orecchia ( 2018 ) ; Gasnikov et al . ( 2019 ) ; Wang et al . ( 2016 ) . This surge of research that applies tools of convex optimization to models going beyond convexity has been fruitful . One of these models is the setting of geodesically convex Riemannian optimization . In this setting , the function to optimize is geodesically convex ( g-convex ) , i.e . convex restricted to any geodesic ( cf . Definition 1.1 ) . Riemannian optimization , g-convex and non-g-convex alike , is an extensive area of research . In recent years there have been numerous efforts towards obtaining Riemannian optimization algorithms that share analogous properties to the more broadly studied Euclidean first-order methods : deterministic de Carvalho Bento et al . ( 2017 ) ; Wei et al . ( 2016 ) ; Zhang & Sra ( 2016 ) , stochastic Hosseini & Sra ( 2017 ) ; Khuzani & Li ( 2017 ) ; Tripuraneni et al . ( 2018 ) , variance-reduced Sato et al . ( 2017 ; 2019 ) ; Zhang et al . ( 2016 ) , adaptive Kasai et al . ( 2019 ) , saddle-point-escaping Criscitiello & Boumal ( 2019 ) ; Sun et al . ( 2019 ) ; Zhang et al . ( 2018 ) ; Zhou et al . ( 2019 ) ; Criscitiello & Boumal ( 2020 ) , and projection-free methods Weber & Sra ( 2017 ; 2019 ) , among others . Unsurprisingly , Riemannian optimization has found many applications in machine learning , including low-rank matrix completion Cambier & Absil ( 2016 ) ; Heidel & Schulz ( 2018 ) ; Mishra & Sepulchre ( 2014 ) ; Tan et al . ( 2014 ) ; Vandereycken ( 2013 ) , dictionary learning Cherian & Sra ( 2017 ) ; Sun et al . ( 2017 ) , optimization under orthogonality constraints Edelman et al . ( 1998 ) , with applications to Recurrent Neural Networks Lezcano-Casado ( 2019 ) ; Lezcano-Casado & Martínez-Rubio ( 2019 ) , robust covariance estimation in Gaussian distributions Wiesel ( 2012 ) , Gaussian mixture models Hosseini & Sra ( 2015 ) , operator scaling Allen-Zhu et al . ( 2018 ) , and sparse principal component analysis Genicot et al . ( 2015 ) ; Huang & Wei ( 2019b ) ; Jolliffe et al . ( 2003 ) . However , the acceleration phenomenon , largely celebrated in the Euclidean space , is still not understood in Riemannian manifolds , although there has been some progress on this topic recently ( cf . Related work ) . This poses the following question , which is the central subject of this paper : Can a Riemannian first-order method enjoy the same rates as AGD in the Euclidean space ? In this work , we provide an answer in the affirmative for functions defined on hyperbolic and spherical spaces , up to constants depending on the curvature and the initial distance to an optimum , and up to log factors . In particular , the main results of this work are the following . Main Results : • Full acceleration . We design algorithms that provably achieve the same rates of convergence as AGD in the Euclidean space , up to constants and log factors . More precisely , we obtain the rates Õ ( L/ √ ε ) and O∗ ( √ L/µ log ( µ/ε ) ) when optimizing L-smooth functions that are , respectively , g-convex and µ-strongly g-convex , defined on the hyperbolic space or a subset of the sphere . The notation Õ ( · ) and O∗ ( · ) omits log ( L/ε ) and log ( L/µ ) factors , respectively , and constants . Previous approaches only showed local results Zhang & Sra ( 2018 ) or obtained results with rates in between the ones obtainable by Riemannian Gradient Descent ( RGD ) and AGD Ahn & Sra ( 2020 ) . Moreover , these previous works only apply to functions that are smooth and strongly g-convex and not to smooth functions that are only g-convex . As a proxy , we design an accelerated algorithm under a condition between of convexity and quasar-convexity in the constrained setting , which is of independent interest . • Reductions . We present two reductions for any Riemannian manifold of bounded sectional curvature . Given an optimization method for smooth and g-convex functions they provide a method for optimizing smooth and strongly g-convex functions , and vice versa . This allows to focus on designing methods for one set of assumptions only . It is often the case that methods and key geometric inequalities that apply to manifolds with bounded sectional curvatures are obtained from the ones existing for the spaces of constant extremal sectional curvature Grove et al . ( 1997 ) ; Zhang & Sra ( 2016 ; 2018 ) . Consequently , our contribution is relevant not only because we establish an algorithm achieving global acceleration on functions defined on a manifold other than the Euclidean space , but also because understanding the constant sectional curvature case is an important step towards understanding the more general case of obtaining algorithms that optimize g-convex functions , strongly or not , defined on manifolds of bounded sectional curvature . Our main technique for designing the accelerated method consists of mapping the function domain to a subset B of the Euclidean space via a geodesic map : a transformation that maps geodesics to geodesics . Given the gradient of a point x ∈M , which defines a lower bound on the function that is linear over the tangent space of x , we find a lower bound of the function that is linear over B , despite the map being non-conformal , deforming distances , and breaking convexity . This allows to aggregate the lower bounds easily . We believe that effective lower bound aggregation is key to achieving Riemannian acceleration and optimality . Using this strategy , we are able to provide an algorithm along the lines of the one in Diakonikolas & Orecchia ( 2018 ) to define a continuous method that we discretize using an approximate implementation of the implicit Euler method , obtaining a method achieving the same rates as the Euclidean AGD , up to constants and log factors . Our reductions take into account the deformations produced by the geometry to generalize existing Euclidean reductions Allen Zhu & Hazan ( 2016 ) ; Allen Zhu & Orecchia ( 2017 ) . Basic Geometric Definitions . We recall basic definitions of Riemannian geometry that we use in this work . For a thorough introduction we refer to Petersen et al . ( 2006 ) . A Riemannian manifold ( M , g ) is a real smooth manifoldM equipped with a metric g , which is a smoothly varying inner product . For x ∈M and any two vectors v , w ∈ TxM in the tangent space ofM , the inner product 〈v , w〉x is g ( v , w ) . For v ∈ TxM , the norm is defined as usual ‖v‖x def = √ 〈v , v〉x . Typically , x is known given v or w , so we will just write 〈v , w〉 or ‖v‖ if x is clear from context . A geodesic is a curve γ : [ 0 , 1 ] →M of unit speed that is locally distance minimizing . A uniquely geodesic space is a space such that for every two points there is one and only one geodesic that joins them . In such a case the exponential map Expx : TxM→M and inverse exponential map Exp−1x : M→ TxM are well defined for every pair of points , and are as follows . Given x , y ∈ M , v ∈ TxM , and a geodesic γ of length ‖v‖ such that γ ( 0 ) = x , γ ( 1 ) = y , γ′ ( 0 ) = v/‖v‖ , we have that Expx ( v ) = y and Exp−1x ( y ) = v. Note , however , that Expx ( · ) might not be defined for each v ∈ TxM . We denote by d ( x , y ) the distance between x and y . Its value is the same as ‖Exp−1x ( y ) ‖ . Given a 2-dimensional subspace V ⊆ TxM , the sectional curvature at x with respect to V is defined as the Gauss curvature of the manifold Expx ( V ) at x . Notation . Let M be a manifold and let B ⊆ Rd . We denote by h : M → B a geodesic map Kreyszig ( 1991 ) , which is a diffeomorphism such that the image and the inverse image of a geodesic is a geodesic . Usually , given an initial point x0 of our algorithm , we will have h ( x0 ) = 0 . Given a point x ∈ M we use the notation x̃ = h ( x ) and vice versa , any point in B will use a tilde . Given two points x , y ∈ M and a vector v ∈ TxM in the tangent space of x , we use the formal notation 〈v , y− x〉 def= −〈v , x− y〉 def= 〈v , Exp−1x ( y ) 〉 . Given a vector v ∈ TxM , we call ṽ ∈ Rd the vector of the same norm such that { x̃+ λ̃ṽ|λ̃ ∈ R+ , x̃+ λ̃ṽ ∈ B } = { h ( Expx ( λv ) ) |λ ∈ I ⊆ R+ } , for some interval I . Likewise , given x and a vector ṽ ∈ Rd , we define v ∈ TxM . Let x∗ be any minimizer of F : M→ R. We denote by R ≥ d ( x0 , x∗ ) a bound on the distance between x∗ and the initial point x0 . Note that this implies that x∗ ∈ Expx0 ( B̄ ( 0 , R ) ) , for the closed ball B̄ ( 0 , R ) ⊆ Tx0M . Consequently , we will work with the manifold that is a subset of a d-dimensional complete and simply connected manifold of constant sectional curvature K , namely a subset of the hyperbolic space or sphere Petersen et al . ( 2006 ) , defined as Expx0 ( B̄ ( 0 , R ) ) , with the inherited metric . Denote byH this manifold in the former case and S in the latter , and note that we are not making explicit the dependence on d , R and K. We want to work with the standard choice of uniquely geodesic manifolds Ahn & Sra ( 2020 ) ; Liu et al . ( 2017 ) ; Zhang & Sra ( 2016 ; 2018 ) . Therefore , in the case that the manifold is S , we restrict ourselves to R < π/2 √ K , so S is contained in an open hemisphere . The big O notations Õ ( · ) and O∗ ( · ) omit log ( L/ε ) and log ( L/µ ) factors , respectively , and constant factors depending on R and K. We define now the main properties that will be assumed on the function F to be minimized . Definition 1.1 ( Geodesic Convexity and Smoothness ) . Let F : M→ R be a differentiable function defined on a Riemannian manifold ( M , g ) . Given L ≥ µ > 0 , we say that F is L-smooth , and respectively µ-strongly g-convex , if for any two points x , y ∈M , F satisfies F ( y ) ≤ F ( x ) + 〈∇F ( x ) , y−x〉+ L 2 d ( x , y ) 2 , resp . F ( y ) ≥ F ( x ) + 〈∇F ( x ) , y−x〉+ µ 2 d ( x , y ) 2 . We say F is g-convex if the second inequality above , i.e . µ-strong g-convexity , is satisfied with µ = 0 . Note that we have used the formal notation above for the subtraction of points in the inner product . Comparison with Related Work . There are a number of works that study the problem of firstorder acceleration in Riemannian manifolds of bounded sectional curvature . The first study is Liu et al . ( 2017 ) . In this work , the authors develop an accelerated method with the same rates as AGD for both g-convex and strongly g-convex functions , provided that at each step a given nonlinear equation can be solved . No algorithm for solving this equation has been found and , in principle , it could be intractable or infeasible . In Alimisis et al . ( 2019 ) a continuous method analogous to the continuous approach to accelerated methods is presented , but it is not known if there exists an accelerated discretization of it . In Alimisis et al . ( 2020 ) , an algorithm presented is claimed to enjoy an accelerated rate of convergence , but fails to provide convergence when the function value gets below a potentially large constant that depends on the manifold and smoothness constant . In Huang & Wei ( 2019a ) an accelerated algorithm is presented but relying on strong geometric inequalities that are not proved to be satisfied . Zhang & Sra ( 2018 ) obtain a local algorithm that optimizes L-smooth and µ-strongly g-convex functions achieving the same rates as AGD in the Euclidean space , up to constants . That is , the initial point needs to start close to the optimum , O ( ( µ/L ) 3/4 ) close , to be precise . Their approach consists of adapting Nesterov ’ s estimate sequence technique by keeping a quadratic on TxtM that induces onM a regularized lower bound on F ( x∗ ) via Expxt ( · ) . They aggregate the information yielded by the gradient to it , and use a geometric lemma to find a quadratic in Txt+1M whose induced function lower bounds the other one . Ahn & Sra ( 2020 ) generalize the previous algorithm and , by using similar ideas for the lower bound , they adapt it to work globally , obtaining strictly better rates than RGD , recovering the local acceleration of the previous paper , but not achieving global rates comparable to the ones of AGD . In fact , they prove that their algorithm eventually decreases the function value at a rate close to AGD but this can take as many iterations as the ones needed by RGD to minimize the function . In our work , we take a step back and focus on the constant sectional curvature case to provide a global algorithm that achieves the same rates as AGD , up to constants and log factors . It is common to characterize the properties of spaces of bounded sectional curvature by using the ones of the spaces of constant extremal sectional curvature Grove et al . ( 1997 ) ; Zhang & Sra ( 2016 ; 2018 ) , which makes the study of the constant sectional curvature case critical to the development of full accelerated algorithms in the general bounded sectional curvature case . Additionally , our work studies g-convexity besides strong g-convexity . Another related work is the approximate duality gap technique Diakonikolas & Orecchia ( 2019 ) , which presents a unified view of the analysis of first-order methods for the optimization of convex functions defined in the Euclidean space . It defines a continuous duality gap and by enforcing a natural invariant , it obtains accelerated continuous dynamics and their discretizations for most classical first-order methods . A derived work Diakonikolas & Orecchia ( 2018 ) obtains acceleration in a fundamentally different way from previous acceleration approaches , namely using an approximate implicit Euler method for the discretization of the acceleration dynamics . The convergence analysis of Theorem 2.4 is inspired by these two works . We will see in the sequel that , for our manifolds of interest , g-convexity is related to a model known in the literature as quasar-convexity or weak-quasiconvexity Guminov & Gasnikov ( 2017 ) ; Hinder et al . ( 2019 ) ; Nesterov et al . ( 2018 ) .

Summary: This paper provides a generalization of AGD to constant sectional curvature spaces (or subsets of them), and proves the same global rates of convergence that hold in the Euclidean space. Additionally, they provide reductions for the bounded sectional curvature case. Their basic strategy involves the use of geodesic maps to accumulate local linear lower bounds, in a way that accounts for the geometric distortion incurred by the map.