Riemannian Adaptive Optimization Methods

Several first order stochastic optimization methods commonly used in the Euclidean domain such as stochastic gradient descent ( SGD ) , accelerated gradient descent or variance reduced methods have already been adapted to certain Riemannian settings . However , some of the most popular of these optimization tools − namely ADAM , ADAGRAD and the more recent AMSGRAD − remain to be generalized to Riemannian manifolds . We discuss the difficulty of generalizing such adaptive schemes to the most agnostic Riemannian setting , and then provide algorithms and convergence proofs for geodesically convex objectives in the particular case of a product of Riemannian manifolds , in which adaptivity is implemented across manifolds in the cartesian product . Our generalization is tight in the sense that choosing the Euclidean space as Riemannian manifold yields the same algorithms and regret bounds as those that were already known for the standard algorithms . Experimentally , we show faster convergence and to a lower train loss value for Riemannian adaptive methods over their corresponding baselines on the realistic task of embedding the WordNet taxonomy in the Poincaré ball . 1 INTRODUCTION . Developing powerful stochastic gradient-based optimization algorithms is of major importance for a variety of application domains . In particular , for computational efficiency , it is common to opt for a first order method , when the number of parameters to be optimized is great enough . Such cases have recently become ubiquitous in engineering and computational sciences , from the optimization of deep neural networks to learning embeddings over large vocabularies . This new need resulted in the development of empirically very successful first order methods such as ADAGRAD ( Duchi et al. , 2011 ) , ADADELTA ( Zeiler , 2012 ) , ADAM ( Kingma & Ba , 2015 ) or its recent update AMSGRAD ( Reddi et al. , 2018 ) . Note that these algorithms are designed to optimize parameters living in a Euclidean space Rn , which has often been considered as the default geometry to be used for continuous variables . However , a recent line of work has been concerned with the optimization of parameters lying on a Riemannian manifold , a more general setting allowing non-Euclidean geometries . This family of algorithms has already found numerous applications , including for instance solving Lyapunov equations ( Vandereycken & Vandewalle , 2010 ) , matrix factorization ( Tan et al. , 2014 ) , geometric programming ( Sra & Hosseini , 2015 ) , dictionary learning ( Cherian & Sra , 2017 ) or hyperbolic taxonomy embedding ( Nickel & Kiela , 2017 ; Ganea et al. , 2018a ; De Sa et al. , 2018 ; Nickel & Kiela , 2018 ) . A few first order stochastic methods have already been generalized to this setting ( see section 6 ) , the seminal one being Riemannian stochastic gradient descent ( RSGD ) ( Bonnabel , 2013 ) , along with new methods for their convergence analysis in the geodesically convex case ( Zhang & Sra , 2016 ) . However , the above mentioned empirically successful adaptive methods , together with their convergence analysis , remain to find their respective Riemannian counterparts . Indeed , the adaptivity of these algorithms can be thought of as assigning one learning rate per coordinate of the parameter vector . However , on a Riemannian manifold , one is generally not given an intrinsic coordinate system , rendering meaningless the notions sparsity or coordinate-wise update . Our contributions . In this work we ( i ) explain why generalizing these adaptive schemes to the most agnostic Riemannian setting in an intrinsic manner is compromised , and ( ii ) propose generalizations of the algorithms together with their convergence analysis in the particular case of a product of manifolds where each manifold represents one “ coordinate ” of the adaptive scheme . Finally , we ( iii ) empirically support our claims on the realistic task of hyperbolic taxonomy embedding . Our initial motivation . The particular application that motivated us in developing Riemannian versions of ADAGRAD and ADAM was the learning of symbolic embeddings in non-Euclidean spaces . As an example , the GloVe algorithm ( Pennington et al. , 2014 ) − an unsupervised method for learning Euclidean word embeddings capturing semantic/syntactic relationships − benefits significantly from optimizing with ADAGRAD compared to using SGD , presumably because different words are sampled at different frequencies . Hence the absence of Riemannian adaptive algorithms could constitute a significant obstacle to the development of competitive optimization-based Riemannian embedding methods . In particular , we believe that the recent rise of embedding methods in hyperbolic spaces could benefit from such developments ( Nickel & Kiela , 2017 ; 2018 ; Ganea et al. , 2018a ; b ; De Sa et al. , 2018 ; Vinh et al. , 2018 ) . 2 PRELIMINARIES AND NOTATIONS . 2.1 DIFFERENTIAL GEOMETRY . We recall here some elementary notions of differential geometry . For more in-depth expositions , we refer the interested reader to Spivak ( 1979 ) and Robbin & Salamon ( 2011 ) . Manifold , tangent space , Riemannian metric . A manifoldM of dimension n is a space that can locally be approximated by a Euclidean space Rn , and which can be understood as a generalization to higher dimensions of the notion of surface . For instance , the sphere S : = { x ∈ Rn | ‖x‖2 = 1 } embedded in Rn is an ( n−1 ) -dimensional manifold . In particular , Rn is a very simple n-dimensional manifold , with zero curvature . At each point x ∈M , one can define the tangent space TxM , which is an n-dimensional vector space and can be seen as a first order local approximation ofM around x . A Riemannian metric ρ is a collection ρ : = ( ρx ) x∈M of inner-products ρx ( · , · ) : TxM×TxM→ R on TxM , varying smoothly with x . It defines the geometry locally onM . For x ∈M and u ∈ TxM , we also write ‖u‖x : = √ ρx ( u , u ) . A Riemannian manifold is a pair ( M , ρ ) . Induced distance function , geodesics . Notice how a choice of a Riemannian metric ρ induces a natural global distance function onM . Indeed , for x , y ∈M , we can set d ( x , y ) to be equal to the infimum of the lengths of smooth paths between x and y inM , where the length ` ( c ) of a path c is given by integrating the size of its speed vector ċ ( t ) ∈ Tc ( t ) M , in the corresponding tangent space : ` ( c ) : = ∫ 1 t=0 ‖ċ ( t ) ‖c ( t ) dt . A geodesic γ in ( M , ρ ) is a smooth curve γ : ( a , b ) →M which locally has minimal length . In particular , a shortest path between two points inM is a geodesic . Exponential and logarithmic maps . Under some assumptions , one can define at point x ∈M the exponential map expx : TxM→M . Intuitively , this map folds the tangent space on the manifold . Locally , if v ∈ TxM , then for small t , expx ( tv ) tells us how to move inM as to take a shortest path from x with initial direction v. In Rn , expx ( v ) = x+ v. In some cases , one can also define the logarithmic map logx : M→ TxM as the inverse of expx . Parallel transport . In the Euclidean space , if one wants to transport a vector v from x to y , one simply translates v along the straight-line from x to y . In a Riemannian manifold , the resulting transported vector will depend on which path was taken from x to y . The parallel transport Px ( v ; w ) of a vector v from a point x in the direction w and in a unit time , gives a canonical way to transport v with zero acceleration along a geodesic starting from x , with initial velocity w . 2.2 RIEMANNIAN OPTIMIZATION . Consider performing an SGD update of the form xt+1 ← xt − αgt , ( 1 ) where gt denotes the gradient of objective ft1 and α > 0 is the step-size . In a Riemannian manifold ( M , ρ ) , for smooth f : M→ R , Bonnabel ( 2013 ) defines Riemannian SGD by the following update : xt+1 ← expxt ( −αgt ) , ( 2 ) where gt ∈ TxtM denotes the Riemannian gradient of ft at xt . Note that when ( M , ρ ) is the Euclidean space ( Rn , In ) , these two match , since we then have expx ( v ) = x+ v. Intuitively , applying the exponential map enables to perform an update along the shortest path in the relevant direction in unit time , while remaining in the manifold . In practice , when expx ( v ) is not known in closed-form , it is common to replace it by a retraction map Rx ( v ) , most often chosen as Rx ( v ) = x+ v , which is a first-order approximation of expx ( v ) . 2.3 AMSGRAD , ADAM , ADAGRAD . Let ’ s recall here the main algorithms that we are taking interest in . ADAGRAD . Introduced by Duchi et al . ( 2011 ) , the standard form of its update step is defined as2 xit+1 ← xit − αgit/ √√√√ t∑ k=1 ( gik ) 2 . ( 3 ) Such updates rescaled coordinate-wise depending on the size of past gradients can yield huge improvements when gradients are sparse , or in deep networks where the size of a good update may depend on the layer . However , the accumulation of all past gradients can also slow down learning . ADAM . Proposed by Kingma & Ba ( 2015 ) , the ADAM update rule is given by xit+1 ← xit − αmit/ √ vit , ( 4 ) wheremt = β1mt−1+ ( 1−β1 ) gt can be seen as a momentum term and vit = β2vit−1+ ( 1−β2 ) ( git ) 2 is an adaptivity term . When β1 = 0 , one essentially recovers the unpublished method RMSPROP ( Tieleman & Hinton , 2012 ) , the only difference to ADAGRAD being that the sum is replaced by an exponential moving average , hence past gradients are forgotten over time in the adaptivity term vt . This circumvents the issue of ADAGRAD that learning could stop too early when the sum of accumulated squared gradients is too significant . Let us also mention that the momentum term introduced by ADAM for β1 6= 0 has been observed to often yield huge empirical improvements . AMSGRAD . More recently , Reddi et al . ( 2018 ) identified a mistake in the convergence proof of ADAM . To fix it , they proposed to either modify the ADAM algorithm with3 xit+1 ← xit − αmit/ √ v̂it , where v̂ i t = max { v̂it−1 , vit } , ( 5 ) which they coin AMSGRAD , or to choose an increasing schedule for β2 , making it time dependent , which they call ADAMNC ( for non-constant ) . 3 ADAPTIVE SCHEMES IN RIEMANNIAN MANIFOLDS . 3.1 THE DIFFICULTY OF DESIGNING ADAPTIVE SCHEMES IN THE GENERAL SETTING . Intrinsic updates . It is easily understandable that writing any coordinate-wise update requires the choice of a coordinate system . However , on a Riemannian manifold ( M , ρ ) , one is generally not 1to be interpreted as the objective with the same parameters , evaluated at the minibatch taken at time t. 2a small ε = 10−8 is often added in the square-root for numerical stability , omitted here for simplicity . 3with mt and vt defined by the same equations as in ADAM ( see above paragraph ) . provided with a canonical coordinate system . The formalism only allows to work with certain local coordinate systems , also called charts , and several different charts can be defined around each point x ∈M . One usually says that a quantity defined using a chart is intrinsic toM if its definition does not depend on which chart was used . For instance , it is known that the Riemannian gradient gradf of a smooth function f : M → R can be defined intrinsically to ( M , ρ ) , but its Hessian is only intrinsically defined at critical points4 . It is easily seen that the RSGD update of Eq . ( 2 ) is intrinsic , since it only involves exp and grad , which are objects intrinsic to ( M , ρ ) . However , it is unclear whether it is possible at all to express either of Eqs . ( 3,4,5 ) in a coordinate-free or intrinsic manner . A tempting solution . Note that since an update is defined in a tangent space , one could be tempted to fix a canonical coordinate system e : = ( e ( 1 ) , ... , e ( n ) ) in the tangent space Tx0M ' Rd at the initialization x0 ∈M , and parallel-transport e along the optimization trajectory , adapting Eq . ( 3 ) to : xt+1 ← expxt ( ∆t ) , et+1 ← Pxt ( et ; ∆t ) , with ∆t : = −αgt √√√√ t∑ k=1 ( gk ) 2 , ( 6 ) where and ( · ) 2 denote coordinate-wise division and square respectively , these operations being taken relatively to coordinate system et . In the Euclidean space , parallel transport between two points x and y does not depend on the path it is taken along because the space has no curvature . However , in a general Riemannian manifold , not only does it depend on the chosen path but curvature will also give to parallel transport a rotational component5 , which will almost surely break the sparsity of the gradients and hence the benefit of adaptivity . Besides , the interpretation of adaptivity as optimizing different features ( i.e . gradient coordinates ) at different speeds is also completely lost here , since the coordinate system used to represent gradients depends on the optimization path . Finally , note that the techniques we used to prove our theorems would not apply to updates defined in the vein of Eq . ( 6 ) .

Summary: This paper presents Riemannian versions of adaptive optimization methods, including ADAGRAD, ADAM, AMSGRAD and ADAMNC. There are no natural coordinates on a manifold. Therefore, the authors resort to product of manifolds and view each manifold component as a coordinate. Convergence analyses for those methods are given. The the theoretical results and their Euclidean versions coincide. An experiment of embedding a tree-like graph into a Poincare model is used to show the performance of the Riemannian versions of the four methods.