No-regret Online Learning over Riemannian Manifolds

1 Introduction . The online optimization has been widely studied in the past decades in online routing , spam filtering , and machine learning [ 4 , 23 , 8 ] . Without a prior knowledge of loss functions , an online convex optimization algorithm predicts solutions before the loss function is revealed . In this paper , we consider the following Riemannian online convex optimization ( R-OCO ) problem , min xt∈K⊂M ft ( xt ) , t = 1 , 2 , . . . , T , ( 1 ) where M is a complete Riemannian manifold equipped with a Riemannian metric g and K is a geodesically convex ( g-convex ) subset ofM . Here , { ft } t=1,2 , ... , T is a sequence of unknown loss functions and every ft is a geodesically convex ( g-convex ) function with sufficient smoothness . The R-OCO problem ( 1 ) extends the online convex optimization in Euclidean spaces with potential applications in machine learning , such as online principal component analysis ( PCA ) , dictionary learning , and neural networks [ 28 , 19 , 25 ] . ∗Correspondence author ( Y. Hong , +86-10-82541888 ) 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The R-OCO problem ( 1 ) can be understood as a learning process of T rounds . At each round t = 1 , 2 , 3 , . . . , T , an online learner chooses a strategy xt from the g-convex subset K. Later or simultaneously , the adversary ( or nature ) produces a g-convex loss function ft : K → R of which the learner has no prior knowledge . Finally , the learner receives the feedback and suffers the loss ft ( xt ) . Generally , there are two types of information feedback . One is the full information feedback , where the entire function ft is revealed to the learner ; the other is the bandit feedback , where only the value ft ( xt ) is revealed . The goal of the R-OCO is to minimize the regret , defined as R ( T ) = T∑ t=1 ft ( xt ) −min x∈K T∑ t=1 ft ( x ) , which measures the difference between the cost by { xt } t=1 , ... , T and the best-fixed point chosen in hindsight . An algorithm is called no-regret [ 32 ] , if the regret of the algorithm goes sublinearly with the time horizon T . For carrying out optimization on a manifold , some classical methods treat the manifold as a subset of an ambient Euclidean space and employ Euclidean constrained optimization techniques . For instance , [ 30 ] presented an algorithm for the online PCA problem , where the variables were updated in an embedding Euclidean space and then projected onto a manifold . However , in practical applications , the dimension of an embedding Euclidean space can be too high ( e.g. , the Grassmann manifold [ 13 ] ) , and the projection can be expensive to compute ( e.g. , the manifold of symmetric positive definite ( SPD ) matrices [ 37 ] ) . An alternative approach termed Riemannian optimization makes use of intrinsic geometry of manifolds so that Riemannian optimization can optimize directly on the manifold as an unconstrained problem , and thus avoiding high dimension embedding and high computing cost for the projection . Furthermore , this viewpoint has shown benefits from the g-convexity , by which a nonconvex optimization problem can be converted into a g-convex one [ 6 ] . Consequently , it is important to take a Riemannian approach in our problem ( 1 ) . Although there were many existing algorithms for offline manifold optimization problems [ 2 , 31 , 5 ] , very few results were obtained about the Riemannian online optimization problem . [ 35 ] proposed an online algorithm for estimating hidden Markov chains on Hadamard homogeneous spaces and [ 9 ] analyzed Riemannian adaptive methods on products in the regret sense . More recently , [ 29 ] studied a zeroth-order online optimization problem on Hadamard manifolds with a sublinear assumption . Contribution This paper aims to design no-regret algorithms for the R-OCO problem in both full information feedback and bandit feedback . The contribution of this paper is summarized as follows : • We propose a Riemannian online gradient descent algorithm ( R-OGD ) for the R-OCO problem in the full information feedback , and then establish upper regret bounds of the R-OGD algorithm for g-convex and strongly g-convex functions . • We introduce a Riemannian bandit algorithm ( R-BAN ) for the R-OCO problem in the bandit feedback and then establish an upper regret bound for g-convex functions . Moreover , we develop a key technique to analyze the derivative of a local integration on homogeneous manifolds , which can be applied to estimate gradients in Riemannian optimization and beyond . • We focus on the worst-case regret and present a universal lower regret bound of R-OCO algorithms with g-convex losses on Hadamard manifolds , which matches the upper bound achieved by the R-OGD algorithm for g-convex functions . The established lower and upper bounds on the achievable bounds of R-OCO match their counterparts for Euclidean online convex optimization e.g. , [ 38 , 24 , 20 , 1 ] . We briefly list our results in Table 1 . Related Work The Euclidean online convex optimization was introduced in [ 38 ] . Inspired by the gradient descent method , [ 38 ] proposed the online gradient descent algorithm ( OGD ) of which the upper regret bound was proven to be O ( √ T ) . Then [ 24 ] proceeded with the study of the OGD algorithm and established an upper regret boundO ( log T ) for strongly convex functions . In addition , [ 1 ] gave a universal lower bound of O ( √ T ) for online algorithms , which indicated that the bounds in [ 38 ] and [ 24 ] are essentially optimal . In the bandit setting , [ 20 ] provided a detailed exposition of a one-point bandit algorithm . By modifying the gradient in the OGD algorithm to a randomized estimator , the upper regret bound attained O ( T 3 4 ) and O ( T 2 3 ) for convex loss functions and strongly convex loss functions , respectively . By extending the one-point bandit algorithm , [ 3 ] developed a multi-point bandit algorithm and presented upper regret bounds O ( √ T ) and O ( log T ) for convex and strongly convex loss functions . The Riemannian online algorithms proposed in this paper in the full information feedback and the one-point bandit feedback settings are extensions of the algorithms in ( [ 38 , 24 , 20 , 1 ] ) to Riemannian manifolds . Riemannian optimization has drawn much research attention in the past decades . Many basic algorithms in Euclidean spaces such as the gradient descent method , Newton ’ s method , and trustregion methods have been adapted into a Riemannian setting [ 2 , 5 , 31 ] . Some research of Riemannian stochastic optimization ( R-SO ) was intended to deal with time-varying optimization problems [ 12 , 36 , 37 , 35 ] . Among them , [ 36 ] provided the first global complexity analysis for the R-SGD algorithm on geodesically convex problems over Hadamard manifolds , and [ 35 ] proposed an online algorithm to deal with hidden Markov chains on Hadamard homogeneous spaces . When performed in batch , R-SO methods are to minimize the average regret in the case of knowing the prior distribution of the loss functions . In these sense , the R-SO can be viewed as a kind of R-OCO problems and R-OCO algorithms can handle settings without prior knowledge . The results about the R-OCO problem are quite limited . [ 7 ] proposed regularized online optimization methods via a Riemann–Lipschitz continuity condition , which focused on convex functions from an ambient Euclidean space . In the full information setting , [ 9 ] constructed regret upper bounds of Riemannian adaptive methods for g-convex functions , which required a product manifold structure . When the form of loss function was not available , [ 29 ] proposed a zeroth-order online algorithm on Hadamard manifolds for the tracking error in asymptotic sense as well as regret bounds by assuming the sublinearity of the term VT , which is a summation of distance between the minimizer of ft and ft+1 . In contrast , the regret bounds established for our online gradient-based/bandit Riemannian optimization algorithms are sublinear for any time , matching those for Euclidean online optimization . 2 Preliminaries . Riemannian manifolds A manifoldM is a topological space locally diffeomorphic to Euclidean spaces . The tangent space TxM is a linearization of manifoldM at point x . A vector field X is a map assigning every point x ∈ M with a tangent vector X ( x ) ∈ TxM , which can be also viewed as differential operators over smooth functions onM , i.e. , the operation X ( f ) defines a function X ( f ) ( x ) = limt→0 1 t ( f ( ξ ( t ) ) − f ( x ) ) onM , where ξ is a curve that starts at x with tangent vector X ( x ) . A Riemannian manifold is a smooth manifoldM equipped with a metric tensor g ( or called Riemannian metric ) , which defines an inner product 〈· , ·〉x in every tangent space TxM of x ∈ M. The Riemannian metric g brings a distance structure onM . A curve is a geodesic if it locally minimizes the length , which is an analog of a straight line in Euclidean spaces . On Riemannian manifolds , a geodesic is uniquely determined by the start point and initial tangent vector . In this way , the exponential map expx : TxM→M is defined by mapping a vector X ∈ TpM to γ ( 1 ) ∈ M for the geodesic γ such that γ ( 0 ) = x and γ̇ ( 0 ) = X . Curvature reflects the geometry of manifolds . We focus on sectional curvature , which is the Gauss curvature of a two-dimensional submanifold . Following [ 36 ] , we consider the Hadamard manifold , which is a simply connected and complete manifold with nonpositive sectional curvature . The Cartan-Hadamard theorem [ 11 ] shows that the exponential map expx ( · ) is a diffeomorphism from the tangent space TxM∼= Rn to the manifoldM . Therefore , the exponential map has an global inverse exp−1x ( · ) on Hadamard manifolds , and the distance d ( x , y ) can be expressed as ‖ exp−1x ( y ) ‖x . Isometries of Riemannian manifolds have been widely studied in differential geometry [ 11 , 10 ] . An isometry φ : M→M is a diffeomorphism preserving distance , i.e. , d ( x , y ) = d ( φ ( x ) , φ ( y ) ) for all x , y ∈ M. It is remarked that all isometries of a Riemannian manifold form a Lie group G. A Riemannian manifold is a homogeneous manifold if the group of isometries G acts onM transitively , i.e. , for any points x , y ∈ M there exists an isometry such that φ ( x ) = y . Some properties of homogeneous manifolds are used in our work , for example , the Killing field that represents the infinitesimal symmetry of isometries . We leave those properties in the appendix , due to the space limitation . Function Classes A set K is called geodesically convex ( g-convex ) if , for any points x , y ∈ K , there admits a geodesic γ ⊂ K connecting x and y . A function f : K → R is called geodesically convex ( or g-convex ) if for any geodesic γ : [ 0 , 1 ] →M , f ( γ ( t ) ) ≤ ( 1− t ) f ( γ ( 0 ) ) + tf ( γ ( 1 ) ) , ∀t ∈ [ 0 , 1 ] . The g-convexity has some equivalent conditions . When f is differentiable , which means that there exists a gradient ∇f ( x ) such that 〈∇f ( x ) , X〉 = X ( f ) ( x ) for any vector field X , the g-convexity is equivalent to f ( y ) ≥ f ( x ) + 〈∇f ( x ) , exp−1x ( y ) 〉 , ∀x , y ∈M . Furthermore , if f is twice differentiable , the g-convexity is equivalent to Hess ( f ) ( x ) ( X , X ) = X ( X ( f ( x ) ) − ( ∇XX ) f ( x ) ≥ 0 , ∀x ∈M , ∀X ∈ TxM , where ∇ is the Levi-Civita connection ofM , which is an analog of the differential of vector fields in Euclidean spaces ( see [ 17 ] ) . A differentiable function f : M→ R is geodesically µ-strongly convex ( or µ-strongly g-convex ) if there exists a constant µ > 0 such that for any x , y ∈M , f ( y ) ≥ f ( x ) + 〈∇f ( x ) , exp−1x ( y ) 〉+ µ 2 d2 ( x , y ) . We term a function to be geodesically L-Lipschitz ( or g-L-Lipschitz ) if there exists a constant L > 0 such that , |f ( y ) − f ( x ) | ≤ L · d ( x , y ) , ∀x , y ∈M , When f is differentiable , the g-L-Lipschitzness is equivalent to ‖∇f ( x ) ‖ ≤ L , ∀x ∈M .

Summary: This paper takes online learning algorithms designed for the Euclidean space (algorithms for online convex optimization with convex or strongly convex functions in the full information case. Or the convex case with bandit feedback) and generalizes them so they work in Hadamard manifolds with the corresponding notions of geodesic convexity. In the bandit case the manifolds are also assumed to be homogeneous.   A lower bound of \Omega(\sqrt{T}) on the expected regret is provided.   The regret bounds are greater than those in the Euclidean counterparts and in the bandit case (the main contribution) they are greater by a factor that depends on the dimension and that is not given in the paper ( its value is not worked out). This fact makes the claim in the abstract "All the obtained regret bounds match the corresponding results in Euclidean spaces." be false and misleading, in my opinion.