A No-go Theorem for Robust Acceleration in the Hyperbolic Plane

1 Introduction . Convex optimization [ 24 ] undergirds much of machine learning , theoretical computer science , operations research and statistics . Geodesically convex optimization is a natural generalization that replaces Euclidean space with a Riemannian manifold and we require that the function we want to minimize is convex along geodesics [ 31 , 1 , 5 ] . It turns out that many optimization problems of interest , while non-convex in the Euclidean view , become geodesically convex when equipped with the right geometry . Some notable examples : The fastest known algorithms for computing Brascamp-Lieb constants [ 17 , 3 ] , and solving related problems like the null cone problem [ 8 , 6 , 7 ] , exploit geodesic convexity . In machine learning , it arises in matrix completion [ 9 , 28 , 33 ] , dictionary learning [ 11 , 26 ] , robust subspace recovery [ 39 ] , mixture models [ 18 ] and optimization under orthogonality constraints [ 14 ] . In statistics , some basic problems like estimating the shape of an elliptical distribution [ 35 , 16 ] or estimation matrix normal models [ 29 , 4 ] are best viewed through the lens of geodesic convexity . In recent years there has been significant effort to adapt the key tools and ideas in convex optimization to the Riemannian setting . This includes giving new deterministic [ 37 ] , stochastic [ 21 , 30 ] , variancereduced [ 25 , 36 ] , projection-free [ 34 ] , adaptive [ 20 ] and saddle-point escaping [ 12 , 27 ] first-order methods . Many new ingredients are needed because the traditional analyses in the Euclidean setting rely on the linear structure . Still , one of the key challenges has remained elusive thus far : Is there a Nesterov-like accelerated gradient method for geodesically convex functions on a Riemannian manifold ? This question is particularly natural in settings where the curvature is non-positive , since it inherits many useful properties of Euclidean space such as having unique geodesics between any pair of ∗Department of Mathematics , Massachusetts Institute of Technology . Email : luh @ mit.edu . This work was supported in part by a Fannie and John Hertz Foundation Fellowship . †Department of Mathematics , Massachusetts Institute of Technology . Email : moitra @ mit.edu . This work was supported in part by a Microsoft Trustworthy AI Grant , NSF CAREER Award CCF-1453261 , NSF Large CCF1565235 , a David and Lucile Packard Fellowship , an Alfred P. Sloan Fellowship and an ONR Young Investigator Award . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . points . There has been notable partial progress . Zhang and Sra [ 38 ] were among the first to clearly articulate this question . They gave a method that achieves Nesterov-like acceleration if you start sufficiently close to the optimum . Since then , the aim has been to develop methods that achieve global acceleration . Ahn and Sra [ 2 ] gave a partial answer by giving a method that converges strictly faster than gradient descent and eventually accelerates . Martínez-Rubio [ 23 ] gave a method that achieves global acceleration but at the expense of having hidden constants that depend exponentially on the diameter of the space . In this work , our main contribution is to dash these hopes by showing that acceleration is impossible even in the simplest of settings where we want to minimize a smooth and strongly geodesically convex function over the hyperbolic plane . Our proof assumes that the gradient oracle returns an answer that has just an exponentially small amount of noise . In comparison , in the Euclidean setting it is possible to achieve Nesterov-like acceleration with an inverse polynomial amount of noise . Of course , in realistic settings some amount of noise is usually unavoidable . Theorem 1 . Given access to a δ-noisy gradient oracle , any algorithm for finding a point within distance r/5 of the minimum of a 1-strongly convex and O ( r ) -smooth function in the hyperbolic plane that succeeds with probability at least 2/3 must make at least Ω ( r log r + log 1/δ ) queries in expectation . Here r is a bound on how far the optimum is from the origin . Comment 1 . While it may at first seem like a limitation to restrict to functions whose condition number depends on the radius , we show in Section 5 that in the hyperbolic plane this is inevitable in the sense that every geodesically convex function has a condition number that is at least linear in the radius . See Theorem 3 for the full version . The key intuition is short and simple : In negatively curved spaces , the volume of a ball grows so fast that information about the past gradients is not useful in the future . Indeed for discrete approximations to the hyperbolic plane , like a 4-regular tree , it is not hard to make this intuition precise . This intuition also helps clarify why existing acceleration results need to assume that you are already within a constant neighborhood of the optimum or depend badly on the radius . It is more challenging to reason about general algorithms that can make queries anywhere they like and not just at a discrete set of locations . The proof of our main result is based on an abstraction in terms of a game where a player wants to find a hidden item in a set and can make queries to a noisy oracle . We prove an information-theoretic lower bound for this game , and then connect it back to convex optimization over curved spaces . The key point is that balls in hyperbolic space grow exponentially with their radius , and this is in turn related to the size of the set in our game . We believe that our query lower bounds for the noisy oracle game can be more broadly applicable , even beyond the realm of optimization . Remark 1 . The natural open question in our work is to prove similar lower bounds when we are given an exact gradient oracle . However any such algorithm that depends on exact gradient computations , and fails with even an exponentially small amount of noise , would seem to be rather brittle to issues of practical concern , like rounding due to machine precision . This paper is structured as follows : in Section 2 we introduce the reader to the counterintuitive properties of the hyperbolic plane . Section 3 defines the noisy gradient problem and phrases it as a more general ‘ noisy query game ’ . Section 4 proves our crucial lower bound on general noisy query problems , and then applies it to prove our no-go theorem about acceleration in curved spaces . 2 The Hyperbolic Plane . This section serves as an introduction to the hyperbolic plane H2 , establishing the important facts we use for our proof as well as intuition for our main result . The first and most important fact about the hyperbolic plane is that it is large : Fact 1 . [ 10 ] The circumference and area of a hyperbolic circle are both exponential in its radius . For illustration , consider a tiling of the hyperbolic plane with congruent equilateral pentagons . As you can see , the number of pentagons at distance r from the origin grows exponentially in r. This gives intuition for our result : When attempting to minimize a function whose minimum lies somewhere within a ball of radius r , the hyperbolic plane forces you to search over a much larger area than any fixed dimension of Euclidean space would . This inherently makes it harder to exploit information from past queries about the function value and gradient , as you can in acceleration in the Euclidean case , when we are interested in reaching a point far away from the origin . 2.1 Convexity . Definition 1 . [ 32 , 1 , 31 ] A function from a manifold ( here , the hyperbolic plane H2 ) to R is geodesically convex if it is convex along any geodesic . This is equivalent to the Hessian of the function having nonnegative eigenvalues . We say that a function is α-strongly convex if the eigenvalues of its Hessian are bounded below by α . Moreover it is β-smooth if the eigenvalues of its Hessian are bounded above by β . Note that even though a function f : H2 → R may live in the hyperbolic plane , its gradient at any point , denoted by ∇f ( x ) , lives in a space isomorphic to R2 . And similarly its Hessian at that point is an ordinary 2× 2 matrix . This is because the gradient and Hessian are local properties of a function , and locally the hyperbolic plane looks Euclidean . This can be made formal through the concept of tangent spaces , but will not be needed for our purposes . In this paper we will prove lower bounds for minimizing arguably the simplest geodesically convex function , the distance squared function : Fact 2 . In the hyperbolic plane , the distance squared function x 7→ d ( x , x ? ) 2 is geodesically convex and its minimum is x ? . At distance r from x ? , this function is 1-strongly convex and ( r/ tanh r ) smooth . The strong convexity and smoothness come from the formula for its Hessian given in [ 15 ] . 2.2 Why Pirates Don ’ t Search for Treasure in the Hyperbolic Plane . The purpose of this subsection is to provide intuition for our main theorem and is not required to understand our results . Imagine a pirate who has buried treasure in the desert somewhere at distance 100 away from her . She does not remember exactly where the treasure is , but is in possession of a compass which points towards it . The compass ’ reading has error , though : on the order of 10−16 degrees . ( This setting is analogous to an algorithm able to make noisy queries to the gradient of some function . ) In the Euclidean plane , the pirate could easily find the treasure : take a compass bearing , walk 100 steps , and dig . An error of 10−16 degrees would literally be subatomic . However , if the pirate attempted this strategy in the hyperbolic plane , she would end up at distance just over 190 from her treasure . She would have started to walk away from the treasure after just a few steps ! ( Specifically , a constant number of steps that scales with log ( 1/10−16 ) . ) Therefore , she would have to repeatedly look at her compass every few steps in order to make progress towards the treasure . This is why gradient descent has only linear convergence in hyperbolic spaces . Of course , this falls short of explaining why no algorithm converges faster . One of the authors ( though we won ’ t say who ) is indebted to the video game HyperRogue [ 22 ] for giving intuition about the hyperbolic plane . One level of this game features a similar scenario with pirates and compasses . 3 Optimization with a Noisy Gradient Oracle . In this section we define the main model we will be interested in . Moreover we recall convergence bounds in the Euclidean case , particularly those that continue to hold in the presence of a small amount of noise , as a point of comparison . Definition 2 . The radius-r gradient optimization model is as follows : There is an unknown differentiable function f whose minimum is within distance r of the origin . An algorithm may query points x within distance 1000r of the origin.3 Upon querying x , the algorithm learns f ( x ) and the gradient ∇f ( x ) . Definition 3 . In the noisy version of the model , instead of learning the exact values of f ( x ) and ∇f ( x ) , the algorithm receives f ( x ) + z1 and ∇f ( x ) + z2 where z1 and z2 are noise . We do not require the noise to be of a specific form such as Gaussian , uniform , etc – we only require that the noise terms for different queries are independent . Definition 4 . We say noise is c-non-concentrated if on any query , the probability distribution function of the noise is everywhere bounded above by c. We say noise is C-precise if the noise term never has magnitude larger than C. In Euclidean space , a small amount of noise does not preclude acceleration . As a point of comparison , we restate the key result ( Theorem 7 ) from [ 13 ] : Theorem 2 ( from [ 13 ] ) . There is an algorithm that , given C-precise noisy oracle access to an L-smooth µ-strongly convex function f and its gradient , along with a starting point at distance d from the minimum of f , outputs after k oracle queries a point xk such that f ( xk ) −min f ≤ O ( Ld ·min ( 1/k2 , exp ( −k/2 √ µ/L ) ) + min ( k · poly ( C ) , √ L/µ ) ) ( The bound in the original paper is more precise , making big-O constants explicit and using a more refined notion of precision . ) This theorem implies that accelerated gradient descent works in Euclidean space even in the presence of noise , provided that the noise has magnitude at most some inverse polynomial in r. Contrast our main result : in hyperbolic space , accelerated gradient descent is impossible even with exponentially small noise .

Summary: (Here and below, I will write "smooth" in the sense of differentiability, and "gradient Lipschitz" for "smooth-in-the-optimization-sense")  The authors prove that in the noisy oracle model (so function values and gradients are noisy, not exact), there is no accelerated Riemannian gradient descent algorithm (Nesterov-type acceleration) that succeeds whp uniformly over all manifolds and all geodesically convex functions. To show this, they study the minimization of a particular smooth geodesically convex function on the hyperbolic plane $\mathbb{H}^2$, namely the the intrinsic distance to a fixed point ${x}^{\star}$ (i.e. ${x} \mapsto \mathrm{dist}({x}, {x}^{\star})$), and prove the lower bound with this function; they also establish that this function is not pathological among functions on the hyperbolic plane in the sense that its gradient-Lipschitz constant is of near-minimal size. The proofs use elementary properties of the hyperbolic plane -- gradient and Hessian expressions for the distance, and the fact that ball volumes in the hyperbolic plane are exponential in the radius -- as well as a reduction to a "noisy query game" problem on which the lower bound is established.