Acceleration via silver step-size on Riemannian manifolds with applications to Wasserstein space
Authors: Jiyoung Park
Abhishek Roy
Jonathan W. Siegel
Anirban Bhattacharya
Venue: NeurIPS 2025 poster ()
Keywords: Riemannian optimization
Acceleration
Wasserstein space
Primary area: optimization
TL;DR: We propose a new class of vector transport based gradient descent enabling silver stepsize acceleration on Riemannian manifolds, yielding provable accelerated gradient methods for potential functional optimization in Wasserstein space.

Abstract: There is extensive literature on accelerating first-order optimization methods in an Euclidean setting. Under which conditions such acceleration is feasible in Riemannian optimization problems is an active area of research. Motivated by the recent success of silver stepsize methods in the Euclidean setting, we undertake a study of such algorithms in the Riemannian setting. We provide the new class of algorithms determined by the choice of vector transport that allows the silver stepsize acceleration on Riemannian manifolds for the function classes associated with the corresponding vector transport. As a core application, we show that our algorithm recovers the standard Wasserstein gradient descent on the 2-Wasserstein space and, as a result, provides the first provable accelerated gradient method for potential functional optimization problems in the Wasserstein space. In addition, we validate the numerical strength of the algorithm for standard benchmark tasks on the space of symmetric positive definite matrices.

Acceleration via silver stepsize on Riemannian manifolds with applications to Wasserstein space
Jiyoung Park
Abhishek Roy Department of Statistics Texas A&M University abhishekroy@tamet edu
Department of Statistics Texas A&M University =ldydd155 100tamu....... edu
Jonathan W. Siegel Department of Mathematics Texas A&M University jwsiegel 1aam...tmmm....mmmmmmmmmmmmm edu
Anirban Bhattacharya
Department of Statistics
Texas A&M University anirbanb@stat. tamu, edu
Abstract
There is extensive literature on accelerating first-order optimization methods in an Fuclidean setting. Under which conditions such acceleration is feasible Riemannian optimization problems is an active area research. Motivated by the recent success of silver stepsize methods in the Euclidean setting, we undertake study of such algorithms in the Riemannian setting. We provide the new class of algorithms determined by the choice of vector transport that allows the silver stepsize acceleration on Riemannian manifolds for the function classes associated with the corresponding vector transport. As core application, we show that our algorithm recovers the standard Wasserstein gradient descent on the 2-Wasserstein space and, as a result, provides the first provable accelerated gradient method for potential functional optimization problems in the Wasserstein space. addition, we validate the numerical strength of the algorithm for standard benchmark tasks the space of symmetric positive definite matrices.
1 Introduction
Consider the Riemannian optimization problem
min f(), ZeN
(1.1)
where N C M is geodesically convex subset of Riemannian manifold M, and f N - R is continuously differentiable geodesically convex functional. popular approach to solve (1.1) is via Riemannian gradient descent (RGD) [ZS16] given by,
Tn
expx,. (~1pp Grad ((x))
(1.2)
where (-) is the exponential map at x, Th. is the stepsize at iteration n, and Grad denotes the Riemannian gradient. It is known that for geodesically convex and smooth functionals f, constant stepsize RGD has (l/nn) convergence rate as in Euclidean spaces [ZS16].
natural follow-up question is whether one can find first-order algorithms that achieve an accelerated convergence rate. This is motivated by the success accelerated first-order methods in Euclidean settings, most notably Nesterov's method [Nes83], which uses momentum achieve an O(1nn²) rate for convex and smooth objectives. Extensive efforts have been made to achieve the same
39th Conference Neural Information Processing Systems (NeurIPS 2025).

---

accelerated rate using similar acceleration in Riemannian optimization problems under various settings [LSC+17, 17, ZS18, AS20, Sie21, AOBL2I, CB22, MR22, KY22, HMJG23]. However, these works typically rely on additional constraints, stronger assumptions, or modifications to the basic gradient descent update (1.2). For example, [LSC 17] involves an intractable nonlinear operator. The analysis [HMJG23) relies on a submanifold structure and establishes acceleration only in the asymptotic regime. All the other algorithms require both upper and lower sectional curvature bounds. We refer [dST21] for general survey of momentum-basee based acceleration methods, and to [KY22, Sections 1, 2] for Riemannian variants.
On the other hand, there is a line of work showing that, in the Euclidean case, an accelerated convergence rate possible by using a carefully designed dynamic stepsize schedule without any modification to vanilla gradient descent. This idea goes back to [You53]; for quadratic functions, choosing The be Chebyshev stepsizes gradient descent achieves the O(1/r²) rate. Generalizing this idea to general convex and smooth functions, [AIt18, AP24b, AP24c, BA24] introduced the silver stepsize schedule-a carcfully designed stepsize sequence that guarantees an improved convergence rate of O(1/nlo8a where acceleration, this method significantly outperforms constant stepsize gradient descent and shows that standard gradient descent, with a carefully designed stepsize schedule, can achieve meaningful acceleration. Its further studics-including the optimality of the stepsize and generalization to arbitrary number of iterations are active area of research in the field of optimization [Gri24, GSW24). Motivated by the success of the dynamic stepsize schedule in the Euclidean case, in this work, we pose the following question:
1 + V2. While slower than the O(1/n²) rate of Nesterov's 's
Is it possible accelerate Riemannian gradient descent by using the dynamic stepsize schedule?
Main contribution Towards addressing the above question, we make the following contributions.
1. We introduce a family of algorithms, vector-transported Riemannian gradient descent (VTRGD), parameterized by the choice of vector transport VT. This framework enables silver step-size acceleration on Riemannian manifolds for function classes defined relative VT. To formalize these classes, we define the notions of VT -yT-gdddesic convexity (resp. L L-smoothness) with base point E M.
2. We show if function is (i) VT -geodesically L-smooth with base b, and (ii) VT -eooddeesi convex with all base, then VTRGD with silver stepsize schedule achieves the accelerated convergence rate of geodesically strongly convex with condition number w. These rates match the corresponding rates in the Euclidean case, and provides the acceleration with minimal assumptions on the manifold. In particular, we avoid the curvature assumption and diameter assumption typically required for momentum- based accelerated RGDs.
"), and the rate of exp( (oo0OOOnnnoos,, (o,, when is in addition
3. We show when VT = r, the parallel transport, then P-geodesic convexity and -ggeddssic smoothness are satisfied for some non-trivial Riemannian optimization problems. In par- ticular, we show our algorithm coincides with the classical Wasserstein gradient descent 2- Wasserstein space, a space where previous accelerated algorithms fail. Hence, our method provides the first provable accelerated result for (usual) Wasserstein gradient de- scents, particularly for potential function optimization problem. We also provide numerical illustrations in the space of symmetric positive definite matrices.
Background
Riemannian manifolds. this section, we review basic concepts Riemannian manifolds while deferring more rigorous details Appendix A. At a point æ on manifold M, tangent vectors are the velocity vectors of smooth curves on M that pass through The tangent space T.M is the vector space consisting of all such tangent vectors A Riemannian manifold is a manifold equipped

---

with an inner product ( for each tangent space T,M, called a Riemannian metric. 2o E M, the distance d(x,, y) is the infimum of the length of all piecewise continuously differentiable curves from x: to y. A Riemannian gradient of the differentiable function f : M - IR at & is a tangent vector Grad f()) E T.M satisfying d. (x) - (Grad ʃ [(x), v)2 for v E TN M. Here, dz is directional derivative of f at a along the direction u. For (ar, v) E TM, where TM == LL T, M denotes the tangent bundle, a smooth curve TV [0, 1] + M with Yo (0) æ and (0) = is called (constant speed) geodesic if it has the locally minimum length with zero acceleration. The exponential map expx T.M- - M is map defined by expx(t) You (1). (x) transports the point x: in the direction of the tangent vector v, following the geodesic Yv- It is known that exp, is local diffeomorphism in some neighborhood U of E T.M. Hence, exp, allows the inverse on U, which is called the logarithmic map log, exp= (U) - T..M. While the exponential and logarithmic maps are always locally well-defined, they may not be globally well-defined.
A parallel transport ((r) I + ( M is way to transport tangent vector along the curve Y parallely. If y is geodesic curve such that r(0) Z, (1) y. then we simply denote [( as rУ (geodesic) parallel transport from T..M to Ty M. One can generalize the notion of the parallel transport by vector transport [AMS08, GGH+21, WDPY24]. For any æ, E M, a vector transport VT : TyMM- + TyM is an operator which maps a tangent vector v E T.M to another tangent space M, satisfying VT = id. Typical examples include the adjoint of the differential of the exponential map VT = (dexp,, Mogg y and parallel transports VT rV
Lastly, we introduce the notion of geodesic convexity and smoothness.
Definition 2.1 (Geodesic convexity). We say N C M is a geodesically convex subset of M for all I,y E N there exists a geodesic "y such that 7(0) = x, 711) = y. and (t) E N for all t E [0, 1]. We say a differentiable function f N - IR is geodesically a-strongly convex for all x, y E N
fly) f(x) + (Grad (1z), v)
2 d (x.
If the above inequality holds with 0, then f is said to be geodesically convex. Definition 2.2 (Geodesic smoothness). We say f is geodesically L-smooth for all x. y E M
L 2 (x..))
f(v) S f(x) + (Grad f()) log, v)
Some literature use the L-Lipschitz continuity of the gradient function as the definition of geodesic L-smoothness which in fact implies Definition 2.2 (see Definition 1. and Lemma A. 18). However, we adopt the above definition used in [KY22, Section 3.1], as it is more directly related to our VT extension introduced in Section 3.
Silver stepsize in Euclidean space In this section, we present the silver stepsize schedule [AP24c] for Euclidean optimization problem. Consider the problem (1.1) where N = Rd, and is convex and L-smooth. A standard approach is gradient descent, which updates via Zn+1 In V (z,) for fixed stepsize 17. In contrast, the silver stepsize schedule is sequence of dynamic stepsizes {mm hReN. For n = 2* - where &t E IN, (nn mee is given by the following inductively constructed sequence:
n
(++1) 1) = (n) (k) 1 + p² p²-1-=1000001 m,,
(2.
where p 1 /2. We set 1/u = p- 1. For example, for 1,2, 3, n7 (8) has the following form:
(2) [v2]. (2 = [22.2. n/ = [V2,2, 22, 22.
In Euclidean optimization, the silver stepsize was recently shown to improve the convergence rate of the gradient descent from ((1/n) to 0(1 011/nl08: P) [AP24c]. The philosophy of the silver stepsize extended to arbitrary n, with the name long stepsize [Gri24, GSW24J. For the brevity of the paper, for this work we focus on the silver stepsize.
3

---

3 Silver stepsize VTRGD: Assumptions and Preliminaries
In this section, we state the assumptions on the manifold and objective function required to solve problem (1.1) using silver stepsize RGD (1.2). Assumption 3.1 (Assumptions for Riemannian manifold).
1. M is a complete Riemannian manifold, i.e., any two points are connected by some geodesic.
2. N C M is open, geodesically convex subset.
3. Exponential maps and logarithmic maps are all well-defined and computationally tractable
N.
Assumption 3.2 (Assumptions on the objective). We make the following assumptions on f N R.
1. f is geodesically convex and has a global minimizer 2+ N.
2. All the iterates of our algorithms are well defined and remain inside N.
3. For given linear vector transport VT, There exists a constant L >> 0 and b E N such that for all T4y in the RGD trajectory, i, = 0, 1,2, ,.,
Qiss= 2L(f(x;) f(æ;)) 2L (VT Grad f(z;), logo x; log, 23),
(3.1)
|| Grad ((x;) VT Grad f(rrz) || 0.
Remark 3.3. Assumptions 3.1 and 3.2, excluding equation (3.1), are standard in the Riemannian optimization literature [AOBL21, KY22, HMJG23], and ensure well- bell-behaved RGD iterates. While we additionally assume (3.1), we do nol require the curvature bound or diameter bound on N typically assumed in (momentum-baseed RGD algorithms.
Some comments on (3.1) are in order. an Euclidean space, since all tangent spaces are the same, i.e., VT id, (3.1) holds for any Ii, Tzj b E Rd if and only if f convex and L-smooth [Nes14, Theorem 2.1.5]. However, on Riemannian manifolds, (3.1) does not directly match with standard geodesic convexity and smoothness. To provide the interpretation of (3.1) in Riemannian manifold in Euclidean space, introduce notions of convexity and smoothness, which dub VT -goddesic convexity (smoothness).
Definition 3.4 (VT (VT-geodesic convexity). A functional :N R is called VT -goodesically convex with base b E M for all x, N, we have,
f(v) 2 f(x) (VT's Grad f(x), log, log, 2))
(3.2)
I is culled VT -Tgeoddesically convex if (3.2) holds for all b E N. Definition 3.5 (VT--godesie I-smooth). A functional I M IR is called VT -T-eeddesically L-smooth with base b E M for all x, y M we have,
f(v) < f(x) + (VT² Grad (x), logs y log, x)), + |logs log, xl1
(3.3)
f is called -goodessicallly L-smooth if (3.3) holds for all b E M.
We interpret VT -gggedessic convexity (resp. L-smoothness) for some representative choices of VT. One canonical example VT; = (dexpx)) x' For such VT, the VT -geedees convexity (resp. L-smoothness) with base b equivalent to the (Euclidean) convexity (rexp. L-smoothness) of function F(v) = (expp.(v) (v)) defined on the tangent space. Another natural choice of VT the parallel transport I' In the optimal transport literature, I l'-gedessic convexity already popular concept, namely generalized geodesic convexity [AGS08, San14, SKL20, DBCS23] which is broadly applied for studying the proximal operator, l--oonvrgeence in 2- Wasserstein space, and Riemannian

---

TM
lop.
M10
Figure I: Geometric illustration of VT -Tggedessic convexity. Usual geodesic convexity (resp. L- smoothness) means for any T; y E N, the function convex (resp. T-smooth) along the geodesic curve 7i (t), On the other hand, VT -T-gedeesic convexity (L-smoothness) with base b implies the function is convex along curve 7/2 (2) Note curve 2/2 exists for general cases (see Remark C.7).
gradient methods [CMRS20, ACGS21]. Motivated from this fact, when VT we will dub --geodesic convexity (resp. smoothness) by generalized geodesic convexity (resp. smoothness). Since VT I will be our main application in Section 5, we introduce more detail about generalized geodesic convexity in Appendix C.
the function VT-geodesically convex (resp. L-smooth), then it is geodesically convex (resp. L-smooth) by taking b = x. However, function being VT -gggeddesically convex (resp. L-smooth) with single base b not a strictly stronger condition than geodesic convexity (resp. L-smoothness) and is incomparable. For example, for VT r' i.e. parallel transport, the function + _ (x,p) on non-Euclidean Hadamard manifold, manifold with non-positive curvature, is generalized geodesically 1-smooth with base while it is not geodesically smooth (see Example C.10 and [CK25]).
Remark 3.6 (Geometric interpretation VT -geodesic convexity). Intuitively, for any three points I, y, N, VT -e-gdeeesic convexity requires f to be convex along a curve from x to y, where the initial velocity is measured in the langent space at a third point b M (see Figure 1 for the detail). This generalizes standard geodesic convexiry. which corresponds to the special case
We now establish the relationship between (3.1) and convexity and smoothness, as in Euclidean space. Proposition 3.7 provides sufficient condition for (3.1).
Proposition 3.7. Let be VT -geoedessia T-smooth with base E N, VT -goodeesscaally convex, and for all E N, define 2 = expb - (VT), Grad f(y) - V Grad (x)) + +logo v) E N. Then satisfies f3r1) all x,, N.
We provide the proof in Appendix B. The condition a E N is technical and generally requires case-specific verification. We emphasize that, as a special case of Proposition 3.7 with b ie., under standard geodesic smoothness, one obtains a Riemannian analogue of the co-coercivity type inequality [Nes14, 14, Theorem 2. 2.1.5). a fundamental property of convex T-smooth functions in Euclidean space. See Appendix 1. 1 for further discussion.
At first glance, (3.1) and the corresponding Proposition 3.7 may appear be merely technical assumption imposed for the convenience of the proof. However, we will later demonstrate in Section 5 that (3.1) admits meaningful applications, particularly when VT = Г. In this regard, our notion of VT -geodesccc convexity and smoothness is not introduced solely for technical convenience; rather, can be viewed as the generalization of existing concepts and allows practical applications.
4 Main Results
In this section, present main convergence results for silver stepsize acceleration on Riemannian manifolds.
5

---

Proposed Algorithm: VTRGD While classical RGDs are defined by the exponential map be- tween consecutive updates as in (1.2), such construction turned out to be incapable to achieve the convergence for dynamic stepsize methods like silver stepsize. The main reason because dynamic stepsize methods require the relationship between non-consecutive itereates, which not feasible in standard RGD (1.2). To overcome this bottleneck, we propose Vector Transported RGD (VTRGD). VTRGD defined as follows. First, choose vector transport VT. Next, choose an arbitrary base point E M which satisfies (3.1) (e.g., base b that makes f VT "T-geddesically L-smooth with base b). Then, VTRGD makes the following update:
Zn+1 = exps (log, In I -
T7 VT Grad f(xx)
(4.1)
Since we assumed that the exponential map, logarithmic map, and Riemannian gradients are tractable in Assumption 3.1, (4.1) provides a tractable algorithm whenever VT tractable. particular, we focus on the case when ?h the silver stepsize. The below theorem is our main theorem, showing that silver stepsize VTRGD achieves the accelerated convergence rate.
Theorem 4.1. Let Assumption 3.1. 3.2 be true and 2 1. Then, for VTRGD (4.1) with silver stepsizes Tp/ L (2.1), we have,
f( ) fxx,) S ruL Ilogs - logg 11.
Tk = (1+ V 4p2k -
We provide the full proof in Appendix B.2. Since T x n2 loga RC n-----ee 1.2716 Theorem 4.1 shows an accelerated convergence rate than constant stepsize RGD on Riemannian manifolds, which is O(n ) [KY22, Appendix DJ. For n # 2 - 1, the error may oscillate and not always remain in low regime. However, the constraint n 2 is not a practical issue, since the number of iterations can be freely chosen. Moreover, while the theorem is formally stated for n 2 1, the oscillation happens when we encounter the huge stepsize, which is known priori. Thus, long as the iteration count avoids these spike points (specifically, when = 2*), the iterates remain stable. Our numerical experiments confirm this behavior; see Appendix D.
Strongly convex case While the Euclidean silver stepsize method allows the strong convexity variant [AP24b], it relies heavily on the use of the interpolating inequality [AP24b, Eqaution (2.6)] which does not seem to have the direct interpretation in Riemannian setting, as well as suffers the saturation mon--accelratiing regime after the certain number of the iterates. On the other hand, it turns out that silver stepsize VTRGD can be extended VT-geodesically a-strongly convex functionals with base b by the use of well-known restarting method [OC15]. The method proceeds as follows:
1. Perform steps of gradient descent starting from initial point 2o to obtain Zm--
2. Restart from Im with the stepsize reset Th, and run m additional steps obtain T2m
3. Repeat this process times, each time restarting from the most recent iterate with the stepsize reset to 7oo. The total number of iteration will be ml.
For fixed n, choosing m and appropriately yields the optimal convergence rate for strongly convex objectives. Notably, this approach remains valid in the Riemannian setting with silver stepsize VTRGD.
Theorem 4.2. Consider the same setting of Theorem 4.1. addition, let f be geodesically c-stronggy convex with base with the condition number consider the above restarting scheme with m = 2² - 1, so that the total number of iteration is n = t(2k - -1). Then, for any E N and n = 1(2k - -1),
L oz. Set k²* [log, k] 1. For any & IN,
1on, |llog, Fn log, x. 11: S exp (- (olog(p/2))) (alll///////// //////n,,, ?) Iloogs vp logg x- 11:
In particular, the algorithu finds an c-approximate solution, i.e., ||log,, Zn log, xx 13 within O (N log(1/e)) iterations.

---

We provide the proof in Appendix B.2. Although the left-hand side is ||log, Jx, - logs, x:x I1 rather than the usual objective (xn; 2.), on any non-negatively curved Riemannian manifold it always upper-bounds the squared distance (see [2S16, Lemma 5]). Hence, Theorem 4.2 yields a convergence rate for the squared distance at least on non-negatively curved Riemannian manifolds. Since constant stepsize RGD finds an r-approximate solution for strongly convex objectives Or log(1 (e)) iterations [KY22, Appendix D], our algorithm achieves an improved rate in the strongly convex setting as well. While the algorithm requires pick the certain number of iterations, again in practice this is not problematic as can freely choose the number of iterates, Also, unlike [AP24b]. the restarting method avoids suffering saturation regime.
We conclude this section with a remark on the choice of b. Although the convergence rates of our VTRGD algorithms in Theorems 4.1 and 4.2 are independent of the choice of the base point the choice of b still influences the algorithm's performance through the constant factor.
Applications
At first glance, (3.1) and (4.1) may appear be merely technical assumptions and consequences introduced for the convenience of the proof, without further implications. However, in this section, we demonstrate that VTRGD in fact coincides with the standard Riemannian gradient descent in the 2- Wasserstein space under the canonical choice of VT = r.
5.1 Optimization the 2-Wasserstein Space
key advantage of our algorithm over existing methods is that it achieves the acceleration without requiring the curvature bounds. This feature makes our analysis particularly suitable for the 2- Wasserstein space, which admits Riemannian structure but lacks an upper curvature bound (see Lemmas A.33 and A.42). While the acceleration has been studied in the continuous-time setting [CCT18, WL22], no discrete-time algorithm with provable acceleration guarantee was previously available. To the best of our knowledge, our method provides the first theoretically guaranteed accelerated algorithm the Wasserstein space for widely used family of functionals, the potential functional.
We briefly introduce the 2- Wasserstein geometry (see Appendix A.2 for details). Let P2.ac (R*) denote the set of probability measures Rd with finite second moments and absolutely continuous with respect to the Lebesgue measure, (1² be the space of square-integrable functions from Rd + Rd under μ E (R**), and T& denotes a pushforward of μ by T. For any v/ E P2. (Rd), the 2- Wasserstein metric is defined as:
min W2 (w, := TEC*(K) Twe
Ezzn [IT(x)) =||]
(5.1)
The well-definedness (precisely, the existence of the minimum T') is guaranteed by Brenier Theorem [Bre91]. The map Tu. achieving the minimum in (5.1) is called an optimal transport map from j to v. The metric space (P2.cc(")) W2). called the 2- Wasserstein space, admits a Riemannian structure with tangent space T,.P2,ac inner product. The exponential map defined by exp,, (v) (id # and the logarithmic map defined by log,
(1) and the Riemannian metric given by the (²(
Tu, id. The Wasserstein gradient at μ is defined as the operator satisfying 0 (( = (Gradw, F(H), vo), where N4 is any sufficiently regular curve of measures with #o = μ and D2 E C2(p)) is the velocity vector satisfying the continuity equation +. = 0. This definition is the exact analogy of the definition
Riemannian gradient that is defined by d,, (x:) = (Grad f [xx) v). A natural vector transport 2- 2-Wasserstein space a transport map, i.e. VT Te.u where Tu,a is map satisfying (Tu,)) = μ.. If Tu., is the optimal transport map. this is in fact un-projected parallel transport in 2-Wasserttein space. In particular, when one uses the optimal transport map, Definition 3.4, 3.5 becomes generalized geodesic convexity (resp. smoothness) in 2- Wasserstein space [AGS08, San14, SKL20J.

---

appin
Serae
Saral
000.
Figure 2: Comparison between silver stepsize method and RGD for potential functional optimization
in BW (R"), with different convexity parameters. We set L = 2 |0gg [===)) and n = E(2 - 1),
where k* being the optimal sub-iterate derived in Theorem 4.2. Columns: From left to right, each column corresponds to = 10 100, 10", 1013.
Perhaps surprisingly, VTRGD recovers the standard Wasserstein gradient descent (WGD) when VT is chosen to be any transport map, since for all b P2,ac((2) and any transport map To,un
Th Gradw, F(un) To,p.) = (id In Gradw, F(pm))#un L
Mn+1 (To,pm
(5.2)
L
Hence in 2-Wasserstein space, silver stepsize Wasserstein gradient descent guarantees the accelerated convergence, analogous to Theorem 4. and 4.2.
Corollary 5.1 (Acceleration by silver stepsize WGD). Let P2 on 2- W-Wassrsttinn space. Set T. = 2* - 1, and let pn be WGD update (5.2) with The being the silver stepsize. F sutisfies (3.1) with some b E (R") and VT Ta and all pai and (id - + (Gradw, F(u.) Gradw, F(Hz) ) o Tpmsss) #p. admit the densities, we get
R be functional
F(Hn) - -((,.) S L [Tb.po -To.p. 11:
Suppose F. is, in addition, geodesically a-stroogly convex with the condition number K. = L/ Then, under the same setup as in Theorem 4.2,
W (p p.) S Cxexp (- (-log(p/2)n//x/ee [Th,pro To. 11?
In particular, the algorithum finds an e-approximate solution in squared 2- Wasserstein distance within O Lo llo((/c)) iterations.
In particular, when the condition (3.1) holds for all b E P2, (R*), one can substitute the right hand sides [[Tb. - Tb,, -- 11, W22 (Ho. pa) by choosing b μr and letting Tu.... be an optimal transport map. While Corollary 5.1 may appear to be straightforward application of main theorems, it in fact requires careful attention to the underlying geometry, since the 2- 2-Wasserstein space is not geodesically complete [PZ20]. This obstacle can nevertheless be resolved by exploiting specific properties of the 2- 2-Wasserttein geometry. The proof is deferred to Appendix B.3.
Remark 5.2 (Optimality is not needed). In classical Wasserstein gradient descent, one typically needs to impose the condition id - 3m Gradw, F(Jr,) ) to be convex, which restricts the size of steps to be less or equal to L/L; the convexity condition is essential for the gradient update to be the optimal transport map. However, our proof does not use the optimality of the update step. Therefore, we do not require the map id To. Gradw, F(un) to be convex, and it justifies the choice of large
stepsizes.
Remark 5.3 (Simplification of the regularity condition). The regularity condition-existene of densities among iterates-can be simplified follows. the transport map THi.ss E CLi (Rd),, and Gradw, F(u) = Vh for some E Clo (Rd),, which is in fact the case for many of Wasserstein gradient algorithms,, then the regularity condition boils down to I 9h V Gradw, F(p) and (), V Gradw, F(u,) Gradw, F(p,) Tus,p) being invertible. See Appendix B.3 for the detail.
Motivated by Corollary 5.1, provide the numerical experiments WGD. We set M P2,ac and N as the Bures- Wasserstein space BW (RR), the space of non-singular Gaussian distributions
8

---

in R equipped with Wasserstein geometry. (RRR)) is geodesically convex subset of (R*) [LCB Moreover, BW BWR") can be identified with a product Riemannian manifold of mean vectors and covariance matrices, i.e., R x (PD(d) where (d)(d) denotes the space of symmetric positive definite matrices of Raxd, We introduce more detail of (W(R) geometry in Appendix 1.2.1. As our objective functional, we consider the potential energy functional:
V(+) == [V=))(
where V: Rd IR. Such functional appears frequently in applications, for example as the component of variational inference. Using the explicit formula of Gradpw V(m. Σ) [DBCS23) in (5.2), we obtain the following silver stepsize RGD in BW for V(u):
Th E ExxN [VV))).
Mn+1 Myx
(5.3)
En+1 = (I-
(1- Ex~N(mm T=- VV)())) En (1- SL.
In
7h
The following proposition shows that the potential functional is an instance of Corollary 5.1 whenever V is convex and L-smooth.
Proposition 5.4. If V is convex and L-smooth in RC, then V satisfies (3.1) with any b P2.ac (R4) and any transport map VT To under both the Wasserstein and Bures- Wasserstein geome- tries. In parficular, the results in Corollary 5.1, with F = hold by substituting ( (Jo: H.) for [T - Tb, lb::
We provide the proof in Appendix B.3. By Corollary 5.1, the silver stepsize WGD (5.3) achieves the accelerated convergence rate whenever 1/ (VV())) is invertible. For our experiment, we considered quadratic (xx) = 1/ (a - m,)"E, (a
- m.) defined on RR10, with m., Σ. being a randomly generated vector and symmetric positive definite matrix respectively. Since V strongly-convex quadratic function, by Proposition 5.4 is generalized geodesically a-strongly convex and geodesically I--mmoohh with T. 1/A (E.) and
1/> (E..). To study the effect of the condition number L/a, fix 1, and vary Small corresponds to convex case, and larger a stands for the strongly convex case. We choose 1/ L as the stepsize for constant stepsize WGD [ZS 6,, KY22J. Figure 2 shows that the silver stepsize WGD outperforms constant stepsize WGD in both convex and strongly convex case. We provide further implementation detail (c.g., the specific distributions of m, and Σx) and additional experiments under various settings (e.g., different random seeds, number of iterations, stochastic gradients, and comparisons with various constant stepsizes) in Appendix D.
Beyond quadratic potential, even in the absence of convexity and smoothness guarantees, we observed that our algorithm yields numerical improvements for other optimization problems in the 2- Wasserstein space. Specifically, we present additional experiments on logistic regression potential and mean field training of neural networks in Appendix D.2.1 and D.2.2.
5.2 Optimization on Symmetric Positive definite matrices
While our main motivation is 2- Wasserstein space, we also provide the numerical experiments for the optimization problem in symmetric positive definite matrix (SPD) with VT I again. The tangent space is (y)(d), the space of symmetric matrices of Raxd On this space, there is a natural choice of the metric, called affine invariant metric. The metric is defined by:
dai(A, B) = log 1-
=
This metric coincides with the Fisher-Rao metric between multivariate Gaussian distributions with fixed mean and covariance matrices A and [Nie23]. Additionally, the Fréchet mean of SPD matrices with respect to d coincides with the geometric mean and plays an important role in many applications, such as diffusion tensor imaging [FAP+00, PFA05, BH06, Ngu22, KPB25]. This metric induces the complete non posiiiivell curved manifold on SPD(d) with the curvature bound [-1/2, 0] [CB23]. While there is problem setup which fully satisfies assumption this space

---

Es Lyna wiw.o
Figure 3: Optimization of [(X) = Ir(CX) log del X, with the different condition number on C. We set = for VTRGD (4.1). We set L = 2 for both experiments. We plot f - -minn where fmin the minimum value over all experiments. Left: k = 10. Right: K = 10
(see Proposition D.I), validate the wide applicability of our method we provide experiments on representative benchmark problem which does not satisfy assumptions. In addition, we provide more experiments with various setups SPD(d) in Appendix D.2.3, including setup which does satisfies our assumption.
Linear minus log-determinant The linear minus log-determinant is widely studied problem, as can be considered as a generalization of linear semidefinite programming (SDP) problems as well as the log-likelihood of the Wishart distribution [BV04, WSTI0, HMJG21, MM24J. The problem is formulated as follows: for C SPD(d), one optimizes the function
f(X) = tr(CC) log det X. (E)
min
Unfortunately, this problem does not satisfy our desired assumptions. practice, however, standard RGD nontheless applied to this problem [HMJG21]. We d 50. Since the problem is not geodesically L-smooth, must choose L manually; set L = 2, which ensures the stability of both algorithms. For VTRGD with the silver stepsize (4.1), set the base point to b I. The results are summarized in Figure 3.
Codes for our experiments can be at http://hhhh..ccco/llooo/////1000 Silver.
6 Conclusion
In this work, we provide the new concept of convexity and smoothness on Riemannian manifolds, namely vector transported geodesic convexity and smoothness. Based on these new concepts, we propose vector transported Riemannian gradient descent (VTRGD) method (4.1), which enables silver stepsize acceleration be feasible for the functions satisfying the VTT-geddesic convexity and VT-geodesic smoothness with base. Albeit under different notion of convexity and smoothness, algorithm is the first tractable accelerated algorithm in Riemannian manifolds, without imposing the curvature assumption diameter assumption. As our core application, in 2-Wasserstein space VTRGD coincides with standard gradient descent in 2-Wasserstein space. Particularly, our framework yields the acceleration for potential energy functional optimization problem.
We conclude the paper with some open questions: The implication of VT-gcodesic convexity and smoothness is not direct. We are unaware how restrictive or favorable these conditions will be. That said, since this notion is well studied in 2-Wasserstein geometry and there are nontrivial examples that satisfy these assumptions, it seems worthy of further investigation. 2. Whether can obtain silver stepsize acceleration of standard RGD for general problems remains open question. 3. We restricted our attention to deterministic gradient descent. Stochastic or proximal version still remains the open. 4. There are extensions of the silver stepsize; showing for arbitrary iterates n or showing the optimality [Gri24, GSW24]. Also, for specific classes of functions in the Euclidean setting, [AP24a] proposed a stepsize schedule for gradient descent that achieves the fully accelerated rate ((1/1²), matching that of momentum methods. Extending these ideas the Riemannian setting would be an intriguing direction for future work.
10

---

Acknowledgments
JP acknowledges Bofan Wang for catching an algebraic flaw in an earlier version of the manuseript. JP additionally acknowledges Hyunwoong Chang and Jaewook Suh for helpful discussions on this topic. JP acknowledges support from NSF DMS-2210689 and NSF DMS-2424305. AB acknowledges NSF DMS-22 10689 for supporting this project. JS ackowledges support from NSF DMS- DMSS242430S and the ONR MURI grant N00014-20-1-2787

---

References
[ACGS2I] Jason Altschuler, Sinho Chewi, Patrik Robert Gerber, and Austin J Stromme. Averaging on the bures wassersesnnn manifold: dimension- free convergence of gradient descent. In Advances in Neural Information Processing Systems, 2021.
[AG08] Luigi Ambrosio and Nicola Gigli. Construction of the parallel transport the wasserstein space. Methods Appl. Anal., 15(1):1-30, 2008.
[AGS08] Luigi Ambrosio, Nicola Gigli, and Giuseppe Savaré. Gradient Flows in Metric Spaces and in the Space of Probability Measures. Lectures in Mathematics ETH Zürich. Springer Science & Business Media, 2ed edition, 2008.
[Alt18] Jason M. Altschuler. Greed, hedging, and acceleration convex optimization. Master's thesis, 2018.
[AMS08] P.-A. Absil, R. Mahony, and R. Sepulchre. Optimization Algorithms on Matrix Manifolds. Princeton University Press, 2008.
[AOBL20] Foivos Alimisis, Antonio Orvieto, Gary Becigneul, and Aurelien Lucchi. A continuous- time perspective for modeling acceleration in riemannian optimization. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, 2020.
[AOBL21] Foivos Alimisis, Antonio Orvieto, Gary Becigneul, and Aurelien Lucchi. Momentum improves optimization on riemannian manifolds. In Proceedings of The 24th Interna- tional Conference on Artificial Intelligence and Statistics, volume 130 Proceedings of Machine Learning Research, 2021.
[AP24a] Jason M. Altschuler and Pablo A. Parrilo. Acceleration by random stepsizes: Hedging, equalization, and the aresine stepsize schedule, 2024.
[AP24b] Jason M. Altschuler and Pablo A. Parrilo. Acceleration by stepsize hedging: Multi-step descent and the silver stepsize schedule. J. ACM, 2024.
[AP24c] Jason M. Altschuler and Pablo A. Parrilo. Acceleration by stepsize hedging: Silver stepsize schedule for smooth convex optimization. Mathematical Programming, 2024.
[AS20] Kwangjun Ahn and Suvrit Sra. From nesterov's estimate sequence to riemannian acceleration. In Proceedings Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, 2020.
[BA24] Jinho Bok and Jason M. Altschuler. Accelerating proximal gradient descent via silver stepsizes, 2024.
[Bar93] A.R. Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information Theory, 39(3):930-945, 1993.
[BH06] Rajendra Bhatia and John Holbrook. Riemannian geometry and matrix geometric means. Linear Algebra and its Applications, 413, 2006.
[BJL19] Rajendra Bhatia, Tanvi Jain, and Yongdo Lim. On the wasseraassnnt distance between positive definite matrices. Expositiones Mathematicae, 191
[Bou23] Nicolas Boumal. An introduction to optimization on smooth manifolds. Cambridge University Press, 2023.
[Bre91] Yann Brenier. Polar factorization and monotone rearrangement of vector-valued func- tions. Communications Pure and Applied Mathematics, 44, 1991.
12

---

[BV04] Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, Cambridge, 2004.
[CB [CB18] Lénaïc Chizat and Francis Bach. On the global convergence of gradient descent for over- parameterized models using optimal transport. In Proceedings of the 32nd International Conference Neural Information Processing Systems, NIPS' 18, page 3040-3050.
Curran Associates Inc., 2018.
[CB22] Christopher Criscitiello and Nicolas Boumal. An accelerated first-order method for non-convex optimization on manifolds. Foundations of Computational Mathematics, 2022.
[CB23] Christopher Criscitiello and Nicolas Boumal. An accelerated first-order method for non-convex optimization on manifolds. Foundations of Computational Mathematics, 2023.
[CCT18] José A. Carrillo, Young-Pil Choi, and Oliver Tse. Convergence to equilibrium in wasserstein distance for damped euler equations with interaction forces. Commumications in Mathematical Physics, 365(1):32--361, oct 2018.
[Che24] Sinho Chewi. Log-concave sampling, 2024.
[CK25] Christopher Criscitiello and Jungbin Kim. Horospherically convex optimization hadamard manifolds part i: Analysis and algorithms, 2025.
[CL20] Yifan Chen and Wuchen Li. Optimal transport natural gradient for statistical manifolds with continuous sample space. Information Geometry, 3:1-32, 2020.
[CMRS20] Sinho Chewi, Tyler Maunu, Philippe Rigollet, and Austin Stromme. Gradient descent algorithms for Bures- Wasserstein barycenters. Jacob Abernethy and Shivani Agarwal, editors, Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pages 1276-1304. PMLR, 09-12 Jul 2020.
(DBCS23) Michael Ziyang Diao, Krishna Balasubramanian, Sinho Chewi, and Adil Salim. Forward- backward gaussian variational inference via jko in the bures-wasserstein space. In International Conference on Machine Learning, pages 7960-7991. PMLR, 2023.
[dST21] Alexandre d' Aspremont, Damien Scieur, and Adrien Taylor. Acceleration methods. Foundations and Trends@ in Optimization, 5(1-2): 2021.
[FAP+05] Pierre Fillard, Vincent Arsigny, Xavier Pennec, Paul M. Thompson, and Nicholas Ayache. Extrapolation of sparse tensor fields: Application to the modeling of brain variability. Lecture Notes in Computer Science, 3565, 2005.
[FRF22] Xavier Fernandez-Reeeee and Alessio Figalli. The Continuous Formulation of Shallow Neu- rul Networks as Wasserstein-Type Gradient Flows, pages 29-57. Springer International Publishing, 05 2022.
[GGH+21] Reza Godaz, Benyamin Ghojogh, Reshad Hosseini, Reza Monsefi, Fakhri Karray, and Mark Crowley. Vector transport free riemannian lbfgs for optimization on symmetric positive definite matrix manifolds. In Asian Conference on Machine Learning (ACML) 2021, Proceedings of Machine Learning Research, 2021. Virtual, p. 8.
[Gri24] Benjamin Grimmer. Provably faster gradient descent via long steps. SIAM Journal on Optimization, 34(3):2588-24 2608, 2024.
[GSW24] Benjamin Grimmer, Kevin Shu, and Alex L. Wang. Composing optimized stepsize schedules for gradient descent, 2024.
13

---

[HMJG21] Andi Han, Bamdev Mishra, Pratik Kumar Jawanpuria, and Junbin Gao. On riemannian optimization over positive definite matrices with the bures- wasserstein geometry. In Advances Neural Information Processing Systems, volume 34. Curran Associates, Inc., 2021.
[HMJG23] Andi Han, Bamdev Mishra, Pratik Jawanpuria, and Junbin Gao. Riemannian accelerated gradient methods via extrapolation. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 Proceedings of Machine Learning Research, 2023.
[HS20] Reshad Hosseini and Suvrit Sra. An alternative to em for gaussian mixture models: batch and stochastic riemannian optimization. Mathematical Programming, 181:187-223, 2020.
[JKO98] Richard Jordan, David Kinderlehrer, and Felix Otto. The variational formulation of the fokker-planck equation. SIAM J. Math. Anal., 29(1):1-17,jn 1998.
[KPB25] Jakwang Kim, Jiyoung Park, and Anirban Bhattacharya. Robust estimation in metric spaces: Achieving exponential concentration with a fr' echet median. In The 28th International Conference on Artificial Intelligence and Statistics, 2025.
[KY22] Jungbin Kim and Insoon Yang. Accelerated gradient methods for geodesically convex Tractable algorithms and convergence analysis. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research. PMLR, 2022.
optimization:
[LCB [2] Marc Lambert, Sinho Chewi, Francis Bach, Silvère Bonnabel, and Philippe Rigollet. Variational inference via wasserstein gradient flows. Advances in Neural Information Processing Systems, 35:14434-14444 2022.
[Lee12] John M. Lee. Introduction to Smooth Manifolds, volume 218 of Graduate Texts in Mathematics. Springer, 2nd edition, 2012.
[Lee18] John M. Lee. Introduction to Riemannian Manifolds. Springer International Publishing, 2nd edition, 2018.
[Lot07] John Lott. Some geometric calculations on wasserstein space, 2007.
[LSC 17] Yuanyuan Liu, Fanhua Shang, James Cheng, Hong Cheng, and Licheng Jiao. Accelerated first-order methods for geodesically convex optimization riemannian manifolds. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[MM24] Yura Malitsky and Konstantin Mishchenko. Adaptive proximal gradient method for convex optimization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[MMN18] Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences of the United States of America, 115:E7665 E7671, 2018.
[MR22] David Martinez-Rubio. Global riemannian acceleration in hyperbolic and spherical spaces. In Proceedings of The 33rd International Conference on Algorithmic Learning Theory, volume 167 of Proceedings of Machine Learning Research, 2022.
[Nes83] Yurii Nesterov. A method for solving the convex programming problem with conver- gence rate O(1/k²). Proceedings of the USSR Academy of Sciences, 269, 1983.
[Nes Yurii Nesterov. Introductory Lectures on Convex Optimization: A Basic Course. Springer Publishing Company, Incorporated, I edition, 2014.
14

---

[Ngu22] Xuan Son Nguyen. A gyrovector space approach for symmetrie positive semi-defing matrix learning. In Computer Vision - ECCV 2022. Springer Nature Switzerland, 2022.
[Nic23] Frank Nielsen. A simple approximation method for the fisher-rao distance between multivariate normal distributions. Entropy, 25, 2023.
[OC15] Brendan O'Donoghue and Emmanuel Candès, Adaptive restart for accelerated gradient schemes. Foundations of Computational Mathematics, 15:715-732, 2015.
[Ott1] Felix Otto. The geometry of dissipative evolution equations: The porous medium
equation. Communications in Partial Differential Equations, 26, 2001.
[PFA05] Xavier Pennec, Pierre Fillard, and Nicholas Ayache. A riemannian framework for tensor computing. International Journal of Computer Vision, 66, 2005.
[PZ20] Victor M. Panaretos and Yoav Zemel. The wasserstein space. In Invitation to Statistics in Wasserstein Space, SpringerBriefs in Probability and Mathematical Statistics, pages 37-57. Springer, Cham, Cham, 2020.
[San14] Filippo Santambrogio. Introduction to optimal transport theory, 2014.
[SBS21] Lena Sembach, Jan Pablo Burgard, and Volker Schulz. A riemannian newton trust-region method for fitting gaussian mixture models. Statistics and Computing. 32(8), 2021.
[Sie21] Jonathan W. Siegel. Accelerated optimization with orthogonality constraints. Journal of Computational Mathematics, 39, 2021.
[SKL20] Adil Salim, Anna Korba, and Giulia Luise. The wasserstein proximal gradient algorithm. In Advances in Neural Information Processing Systems, 2020.
[SM25] Jaewook Suh and Shiqian Ma. An adaptive and parameter-free nesterov's accelerated gradient method for convex optimization, 2025.
[Tak09] Asuka Takatsu. On wasserstein geometry of the space of gaussian measures, 2009.
[THG17] Adrien B. Taylor, Julien M. Hendrickx, and François Glineur. Smooth strongly convex interpolation and exact worst-case performance of first-order methods. Mathematical Programming, 161(1-2):37-3445,22 2017.
[TP22] Yann Thanwerdas and Xavier Pennec. O(n)-invariant riemannian metrics on spd matrices. Linear Algebra and its Applications, 661, 2022.
[Vi108] Cédric Villani. Optimal transport: Old and new. 2008
[WDPY24] Xuejie Wang, Kangkang Deng, Zheng Peng, and Chengeheng Yan. New vector transport operators extending a riemannian cg algorithm to generalized stiefel manifold with low-rank applications. Journal of Computational and Applied Mathematics, 451, 2024.
[WE20] Stephan Wojtowytsch and Weinan E. Can shallow neural networks beat the curse of dimensionality? mean field training perspective. IEEE Transactions on Artificial Intelligence, 1(2):121-129, 2020.
[WL22] Yifei Wang and Wuchen Li. Accelerated information gradient flow, 2022.
[Woj20] Stephan Wojtowytsch. On the convergence of gradient descent training for two-layer relu-networks in the mean field regime, 2020.
[WSTI0] Chengjing Wang, Defeng Sun, and Kim-Chuan Toh. Solving log-determinant opti- mization problems by newton-cg primal proximal point algorithm. SIAM Journal on Optimization, 209))3299 2010.
15

---

[You53] David Young. On richardson's method for solving linear systems with positive definite matrices. Journal of Mathematics and Physics, 1953.
[7S16] Hongyi Zhang and Suvrit Sra. First-order methods for geodesically convex optimization. In 29th Annual Conference on Learning Theory, volume 49 of Proceedings of Machine Learning Research, 2016.
[ZS18] Hongyi Zhang and Suvrit Sra. An estimate sequence for geodesically convex opti- mization. In Proceedings of the 31st Conference On Learning Theory, volume 75 of Proceedings of Machine Learning Research, 2018
16

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Abstract and introduction clearly states main theorems 4.1: achieving the acceleration using silver stepsize under Riemannian manifolds via algorithmic modifications. In addition, our main application and experiments are well-stated as well.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Section 3 and 4, clarified that the algorithmic modification and some of the assumptions which are less standard. That said, Section 5 presents numerical results demonstrating that our method remains effective even when some of these assumptions are relaxed. Finally, Section 6 mentions directions that may refine or extend the current approach.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
17

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We explicitly stated the assumptions in our main body (Assumption 3.1, 3.2). We provide the full proof in the Appendix B.2, B.3. We also provided the full reference for the results we cited.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the sufficient detail of the experiment we conducted in Section 5 and Appendix D. We also provided the code by the public Github link.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
18

---

While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (c.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model limited in some way (e.g., registered users), but it should possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide the full code in the public Github link, which can reproduce the results of our experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: ctaaps.cc/publcc/uuiii. http://ooooSumuisiioeeeooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We provided the data generating process of the simulation data and explicit algorithm in Section 5 and Appendix D.
19

---

Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide the result of multiple experiments and statistical significance in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We stated our computational resource in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
20

---

9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httppp///eaarrpp..cppp/////chccc
Answer: [Yes]
Justification:
Our research is conducted in compliance with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There no societal impact of the work performed. Our focus purely on the algorithmic aspects, rather than societal applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymemn of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Our research does not involve the high risk for misuse.
Guidelines:
21

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We mentioned and cited the packages we used in Appendix D.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi paperswithcode thcode. com/ datasses has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
22

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non stanntand components.
Please refer to our LLM policy fttpp::/nneurpee http://pss.peeeeeeeennne///////e for what should or should not be described.
23

---

Appendix
A Preliminaries
24
Deferred proofs
33
Generalized geodesic convexity and smoothness
43
Implementation detail and additional experiments
48
Changes from the Submitted Version
56
A Preliminaries
A.1 Riemannian geometry
In this uppendix, we introduce key concepts in Riemannian geometry briefly discussed in Section We mainly mention the known results, and omit the proof and well-definedness of definitions. For detail, interested reader find relevant material in textbooks, e.g., [Lee12, Lee 18, Bou23].
A Riemannian manifold is manifold equipped with an inner product for each tangent space, called Riemannian metric.
Definition A.1 (Riemannian manifold). Riemannian manifold (M, g) is a real smooth manifold equipped with a Riemannian metric which assigns to each M a positive-defiite inner product Gp(u, w) (v. w)p on the tangent spuce T.M.
Often, this tangent space M,MM is conveniently expressed in the form of the vector field, which takes point in manifold as input and returns tangent space vector at that point. Formally, the vector field of M is defined as follows:
Definition A.2 (Vector field). A map X (M) + C (M) is called smooth vector field if it is derivation, i.e., X satisfies
X. = x(0))x((0))+))(xx
Here M is the imput of the function.
As the name derivation indicates, one can think of the vector field as directional derivative along the direction of the vector field. The following familiar example may help.
Example A.3 (Vector field in Rd). For f E O (Rd), p E Rd, and u E Rd, think of directional derivative of f at p along direction v. dp) If fix and view f as variable input, then v E T,, can be identified with the functional f > fp).. In other words, by defining Xp(f) = d dpp( the value of vector field Xp at each point p E M can be identified as tangent vector E T.R...
From on, will write X vector field, and this will mean function Xp()) d where Xp. For the definition of directional derivative in general manifolds, refer to [Lee12]. We write X(M) as a set of smooth vector fields on M.
One the fundamental structure of manifold is an affine connection, a concept that connects tangent spaces of different points of the manifold.
Definition A.4 (Affine connection), Let M be a manifold, and X(M) be the set of all smooth vector fields on M. An operator V.. (MM) x x(M) + X(X) called an affine connection if for all 8 ( () and X, Y X(M) it satisfies the following properties:
1. VxXY f i.e. linear in the first variable.

---

2. (dxf)) + XVYY, that is. V satisfies the Leibniz rule in the second variable.
In the case of Riemannian manifolds, we have a natural connection induced from the Riemannian metric, called Levi-Civita connection.
Definition A.5 (Levi-Civita connection). For Riemannian manifold (M, g). let ((M) be a set of smooth vector field on M. The Levi-Civita connection the unique affine connection V. X(M) {MM + xMM), satisfying the following properties:
1. VxY Vy [, YY i.e. it is torsion-free. Here, [(,-] denotes Lie bracket.
2. X (g(Y, 2)) g VVV.V. + gYY Vx that is, the connection is compatible with the metric 9-
The choice of the affine connection determines multiple geometric concepts. One fundamental concept is geodesic curve, which is constant speed curve on the manifold.
Definition A.6 (Geodesic). A smooth curve y [0, 1] - M is called geodesic curve if V = 0.
Riemannian manifold is called complete if any two points are connected by some geodesic. We will always assume M a complete Riemannian manifold.
We say a Riemannian submanifold C M is totally geodesic for every v E TM, the geodesic with respect to M, Te. lies entirely in M.
Equipped with the notion of geodesic, one can define the exponential map and logarithmic map on Riemannian manifold.
Definition A.7 (Exponential map, logarithmic map). Let pet E M.
1. For any M,M, and (0) Then, one cun define a map (xp,(v) == (1). This map is called the exponential map.
define a geodesic curve %o 10, 1] -> M such that "fu (0) p
2. It is known that the exponential map is a local diffeomorphism on U, the open neighborhood of E Ty M. Therefore, one can define log, == exp, (4) for E (U). This map is called the logarithmic map.
To understand the notions of the exponential map and logarithmic map, we illustrate these concepts in the Euclidean case. In the Fuclidean space, exp,(v) + u and log, 2 = p. In other words, the exponential map moves p along the tangent direction v, and the logarithmic map returns the tangent direction from p q.
Note the logarithmic map is only defined locally. While our analysis assumed the global existence of the logarithmic map over the geodesically convex subset N (Assumption 3.1), whether there is global logarithmic map is not always guaranteed.
Another geometrie concept induced from the connection a covariant derivative, a notion of differentiation of the vector field along the curve [Bou23, Definition 5.28, 5.29].
Definition A.8 (Vector field along the curve). Let y [0 M be smooth curve. map Z [0, TM is called vector field ((t) vector fields ry as X(y)..
E M M for all [0, We write the set of
Definition A.9 (Covariant derivative). Lety =0,1] + M be smooth curve and V be affine connection. Then, the covariant derivative is the unique operator D, X(7) - K(7) satisfying the following properties for all Y, Z E X(7), W E x(M),g E c² ([0, 1J) and a, b E R:
1. Di(aY + = aDD
2. Di(g2) (
3. (D,(W ())) = V. W for all t [0.
25

---

If V. is the Levi-Civita connection, then the covariant derivative also satisfies
d YY, = (V,
dt
Parallel transport is a notion of transporting vectors between different tangent space parallely. The parallel transport is uniquely determined by the covariant derivative [Bou23, Defintion 10.33, 10.35].
Definition A.10. vector field Z E X(7) called parallel if D,Z = 0.
Definition A.11 (Parallel transport). Let : [0, + M be a smooth curve. The parallel transport of the tangent vector at Ty(to) M the langent vector Tyyt,) M along the curve 7y is the map
T(7) Ty(to) M + M
defined by ((7) = 7(4) for the paraltel vector field E X(7)).
We collect some properties of the parallel transport. Proposition A.12. [Bou23, Proposition 10.36]
1. [(1) is linear map.
2. ((1) (()) = r(7)
3. (()) P(7) id.
4. (v, () w),()) = "
When is chosen to be the geodesic curve such that r(0) and 7(1) v. we denote the parallel transport I'(1) as L... When context is clear, we will denote L as the (geodesic) parallel transport from x to
Remark A.13 (Properties of geodesic parallel transport). By Proposition A.12, a geodesic parallel transport Ny satisfies the following properties:
1. m is linear map.
2. r r id.
3. (v, w), = (I
Note the second property is dropped, geodesics from y and y to do not necessarily be in the same curve.
Remark A.13 the key properties of parallel transport used in our analysis. These properties play a pivotal role when we define the parallel transport 2- Wasssersteinn space (Proposition A.30).
The last geometric concept induced from the Levi-Civita connection is curvature.
Definition A.14 (Riemannian curvature). The Riemannian curvature tensor R(.,)) X(M) x
X(M) x X(M) + x(M) is defined by the following formula:
R(X, V Vv 1
where [ denotes a Lie bracket.
The key geometric quantity in our analysis is sectional curvature, which generalizes Gaussian curvature in 2-dimensional surface.
Definition A.15 (Sectional curvature). Let E M, and denote Exp set of two-dimensional subspaces in T M. The sectional curvature K: Ep + IR is defined by the following formula:
(R(u, v)v, u), {u, v)
K(app) ||ll ||0|
where {u, o. is basis of dp.
26

---

Note that we can write this sectional curvature as a function of two linearly independent vectors in M as well. In particular, if u, u are orthonormal, then K (u, = (R(u, v)v, u),-
Riemannian manifold is called flat if for all p and dp sectional curvature K(Øp) positively curved if K (ap)) 0, and negatively curved if (app) < O.
A.1.1 Functional properties of functions on Riemannian manifolds
In this appendix, we introduce additional functional properties of functions on Riemannian manifold. We begin with introducing the notion of geodesically convex set.
Definition A.16. [Bou23, Definition 1 1.2] Let (M, g) be complete Riemannian manifold. N C M is called geodesically convex subset of M for all y E N, there exists geodesic y [0, M such that (()) = 7(1) y. and (t) E N for all E [0.
Next, we introduce the notion of geodesic convexity and smoothness.
Definition A.17 (Geodesic convexity and smoothness). Let f N - R be a differentiable function.
1. f is called geodesically a-strongly convex for all x, E
f(y) 2 f(x) (Grad f(x), v/, + 5/2 (x,
Ifa we say f is geodesically convex.
2. f is called geodesically L-smooth for all x, J.YEN E
|| Grad f(y) Grad f(x))l, S Ld(z.y).
Now, we show the key inequality induced from the geodesic L-smoothness. This is often called descent lemma.
Lemma A.18 (Descent lemma). If is geodesically L-smooth, then for all y E N
I.
f(y) S f(x) (Grad (xx), log, v) 2 (x,
Proof. y [0. + M be geodesic curve such that x(0) x, (1) = y. By the definition of the Riemannian logarithmic map, get 7/(0) log, y. By Fundamental Theorem of Calculus and properties of the parallel transport,
f(v) ((x(1)))=))))))))))))))))))))))) f(>(0)) + S. d = f(x) + S/' (mx1))
de ( f(x) + S. ((G())) dt 7(0) Grad ((((t))))//) ((0) dt f(x) + J.' (Tite Grad f(r(t)), 8) dt.
Then, by subtracting f(x) + (Grad (xx), log,r y) from the both hand sides,
f(y) f(x) (Grad f f(r), v\
(I% f(x(t) Grad f(z),
| Grad f(r(t)) - Grad f f(x) log vll dt
Ed(r)(t) x) (d( v) LL((x.)) ( idt
L (2,
For (i) used Cauchy-Schwartz inequality, and for (ii) we used L-smoothness property. For (iii) used the fact that the geodesic curve satisfies d(x, x()) td(æ, y) due to the constant speed property.
27

---

Note we used this descent lemma as the definition of the geodesic L-smoothness in the main body. This is because we wanted to make the correspondence with VT-geodesic smoothness. For our VT -gegedeesic smoothness, Lipschitz gradient type condition no longer implies (3.3) for general VT, unless (t) = (VT (()) (0) for a curve 7, where * denotes the adjoint operator. On the other hand, it turns out that results only require (3.3) rather than Lipschitz gradient type condition, thus we stayed with Definition 3.5 to pursue the generality.
A.1.2 Product Riemannain manifold
In Appendix A.2.1, will encounter product manifold. To that end, present preliminary facts here. We omit the details and simply list few useful results. For more information product Riemannian manifolds, we refer the reader to [Lee18].
Definition A.19 (Product Riemannian manifold), A product Riemannian manifold is a manifold M = Mi x M2 such that each (M1,5) 91) and (M2, 92) are Riemannian manifolds, and the Riemannian metric g is defined by the product metric:
9 ((11, X2); (2, (X1, Yi) 92(X2,
Product Riemannians manifold have useful properties that make the computation easier.
Theorem A.20 (Levi-Civita connection of product Riemannian manifold). The
con- nection of product Riemannian manifold (M,9) g) (M1.91) x (M2, 92) satisfies the following property:
V (xx, X2) = 1. Y1 + 2.
The following corollary is direct consequence of the definition of Riemannian curvature, Lie bracket, and Theorem A.20. Corollary A.21 (Riemannian curvature of product Riemannian manifold).
R ((X1, X2), (Yi, (2)) (Z1, Z2) = (x X1) @ (x(2,,Y2)4
Lastly, we obtain the following collorary, which will play an important role in our later section. Corollary A.22 (Sectional curvature of product Riemannian manifold), Let (u1, u2), (U1, V2) be orthonormal vectors in T.M. Write A; := ||u/|| ||:i1 (u(u, v.)2. Then,
K ((u1, u2), (v1, 21)) A, (u1, u1) + A2 v2).
Proof. From Definition A.15, Definition A.19, and Corollary A.21, we have K ((u1, ug), (V1, 02)) = g (R((u,, u2). (Ui u2) (U1, 02). (u1, uz)) g ((Ri (ui )ui, Ro (u2, U2 U2), (ii, u2)) 91 (R1 (u1, U2)0s, u1) g2 R2 (u2, u2) U2) = A₁ K2 (w1, v1) + ( 22).
In particular, if K₁ 0, i.e., of the spaces is flat, the the curvature behavior of the product manifold is entirely determined by K2. This will be the case in Appendix A.2.1.
A.2 Wasserstein geometry
In this appendix, we introduce the core concept of Wasserstein geometry, which is one of our key application. We write the space of probability measures with finite pth moment on Rd by (,.)). Again, we mainly introduce the known results without proofs. For interested readers, we refer to [Vi108, AGS08, San14, Che24].
For μ, VE PR let ((,,v) be set of couplings of μ and Wasserstein distance between μ and are defined follows.
28

---

Definition A.23 (Wasserstein metric), Let p. E Py(R*). Denote V), to be set of coupling measures of and v. Wasserstein distance between μa and is defined as follows:
WP (H, == inf Err,y)= ||| yll'].
TET(A.))
This is known to be a well-defined metric. A metric space (Pp(R"), Wp) is called p- Wasserstein space.
2- Wasserstein space is typically more interesting space compared to other p- Wasserstein spaces due to its geometric properties. [Bre91, JKO98, Ott1] found out that if we restrict our attention to the probability measures which are absolutely continuous with respect to Lebesgue measure and have a finite second moment, denoted by (2.ac((*)) (R2). then (P2,ac W2) endows a richer geometric properties. Specifically, while (R W2) is not precisely a Riemannian manifold, its geometry almost same to the non-negatively curved Riemannian manifold.
The reason (R"),, W2) endows Riemannian structure is rooted from the following theorem [Bre91]:
Theorem A.24 (Brenier Theorem). If pr, E P2.ac(RR) then
min (2(p TELP(p) TRu=V
Egru [|T(x) - =1(2
min TELP(p) Tay=
| - id] (a:
Denote the minima as Tp.y Then Tu is a gradient of some convex function & on Rd p.a.e. Furthermore, TH, Tu.u = id. The minima Tpp is called the optimal transport map from to v.
Theorem A.24 gives notion of tangent direction at μ.
Definition A.25 (Riemannian metric in 2- 2-Wasserstein space). For μ E W2(R"), tangent space
of μ. is P2. { (v v E C² (w)) (²)) Here, C22 (R2((R)) is a set of compactly supported smooth functions on Rd. The Riemannian metric is defined as L (j))-innr product. In other words, (v, w)μ Ex~p (V(x), w(x))].
Remark A.26 (Interpretation of the tangent space). By Brenier theorem, Tp = Vó. For arbitrary O, it follows that X(Tu. - id) = V(XΦ - > x space T aa can interpreted as the set of scaled displacement fields A(Tu.-id) If X I and v, then MTT,, iX) X(Y -X), which corresponds to directions in the usual Euclidean sense. From this perspective, the tangent space is naturally constructed to represent Euclidean directions at the level of individual particles.
(R This implies that the tangent
One can naturally define a geodesic curve (R2,ac(R4), W2), by pushforwarding the interpolation between particles to the measure space.
Definition A.27 (Geodesic in Wasserstein space). A geodesic curve 7: [0, (2. such that (0) and 7(1 = can be defined as follows:
((t) = ((1-((1))) - ((1-t)id+ + (Tp,.)
The exponential map and logarithmic map are then defined accordingly. Definition A.28 (Exponential map and Logarithmic map in Wasserstein space). For p.v (Rd) and u E L (ni), exponential map and logarithmic map of (P2 (Rd), W2) are defined as follows:
expp(v) = (w + id)
(v) = =T - id.
favorable property of 2- Wassersstii space is that the exponential map (and accordingly logarithmic map) is globally well-defined (14,, i.e., 2- Wassersteii space satisfies Assumption 3.1.
This Riemannian structure induces 2- Wasserstein metric. Observe the Riemannian distance in- duced from the above structure coincides with the Wasserstein distance; d(u, = log, w/l2
[|T =d] W (p,v).
One can define geodesic parallel transport as well.
29

---

Definition A.29. [AG08. Parallel transport] For μa, E (2 and E T Pu
T2 : (U o T...).
Here, II. a projection operator (²()) - T.P2 a.
This definition of parallel transport is not entirely satisfactory, as involves the operator 1I. which lacks explicit form. However, recall our analysis only requires the properties of parallel transport in Remark 131 It turns out that even if we drop II. and consider T transport onto L2 (1),, the corresponding parallel transport still has properties in Remark A.13, which are sufficient for our analyses.
Tw. parallel
Proposition A.30 (Transfer lemma). For μl, E (R and v E (1),,, define Te a Tup Then,
1. D is linear operator on (1).
2. DV D = id.
3. (v, w) = (T r/
Proof. Property 1 is direct: for u, (M) and a, R. (a + bw) av Tp,, + bw o Tu,
H e + n
Property 2 is from Theorem A.24.
Property 3 is direct consequence of the change of the measure formula:
(v, (), - (u(x), w()) d(Te.)) = (u Tem(z)) w (v,a = (T r
Therefore, by Proposition A.30, transport 1 and (2) the tangent space for analysis. In fact, such parallel transport and tangent space are sufficient for other first-order Wasserstein gradient flow analyses as well (e.g.. [AGS08, SKL20J).
the wn-projected parallel transport Te, parallel
Now, we introduce sectional curvature in 2- Wasserstein space. To establish this, we introduce the continuity equation and the notion of covariant derivative in the 2-Wasserstein space.
Definition A.31 (Continuity equation). Let μr be a flow in (R.)). For given μt, there exists vector field v, L (P() such that
d.p div((per))
Such vt is called (velocity) vector field of the flow pt.
One can think of U, velocity at μr, and plays similar role as (1) in Riemannian manifolds. Definition A.32 (Covariant derivative). A covariant derivative of we Tu. P2 along curve μ4 defined by the following formula:
Me
With We
V
Vn, W:
Im
lim h0
h
Here, I' is parallel transport defined in Definition A.29, and is vector field of the flow pt-
We are ready to introduce the result that 2- 2-Wasserstein space is non-negatively curved.
Lemma A.33. Let Ut, Wi be orthonormal elements in T (R4). Then, the sectional curvature of the subspace spanned by these Two tangent vectors is as follows:
Ky, (v,, we) = V we V Vel.
where the first V is Euclidean gradient, and the second V is covariant derivative.
30

---

We refer to [AG08, Proposition 7.2] or [Lot07, Corollary 5.13] for the derivation.
The last ingredients we need for the analysis of the Wasserstein space are notions of gradient. The concept defined as an analogous manner the Riemannian case. Again, we omit the detail and just present the result.
Wasserstein gradient is defined analogously to the formula dx.(()) = (Grad (a), e) in Riemannian manifold.
Definition A.34 (Wasserstein gradient). For functional F (2 R, the Wasserstein gradient of F at fo is an element of L (po) satisfying the following equation:
0.F(())) = (Gradw, F(po), vo)
Here vt is vector field of the flow μt. One has the following explicit formula:
Gradw, F()) 6F(1))
81z
Here, V is Euclidean gradient and 5F(u) is the first variation. Ap
Here, the role of y((0) is changed to uo. For the derivation we refer to [Che24, Theorem 1.4.1].
A.2.1 Bures- Wasserstein geometry
In this appendix, we briefly introduce Bures- Wasserstein space (W(RR) a space of Gaussian measures equipped with W2 metric. Main takeaways of this appendix are as follow:
1. BW(R)) is product Riemannian manifold with non-negative sectional curvature. 2. (RR)) is geodesically convex subset of (R W2) and totally geodesic submani- fold, In this regard, we can take N BW for our algorithm.
3. This example shows how one can parameterize the transport map to make the algorithm implementable as in Equation (5.3).
4. This example confirms that BW (R"), and therefore the 2- Wasserstein space, do not admit the curvature upper bound. Consequently, existing acceleration methods requiring the curvature upper bound are well-suited for solving the optimization problems in Wasserstein space.
Again, we briefly list the results. For detail, we refer to [Tak09, BJLI9, ACGS21, LCB+22, DBCS23].
Definition A.35 (Optimal transport map between Gaussian). The optimal transport map between 140 N(m,, En) and μn (m,,, En) is defined as follows:
Too (x) = m2 2 1/2 (2)
1/2 / / /212/
mo).
Definition A.35 saids the optimal transport map between Gaussians is an affine map. This fact provides two favorable results.
First, since affine transform of the Gaussian is also Gaussian, from Definition A.27 every geodesic interpolation between two Gaussians is also Gaussian. This shows BW(R") is geodesically convex subset of 2- Wasserstein space. In addition it implies BW(R)) is totally geodesic submanifold of 2- Wasserstein space [Lee18, 18, Exercise 8.4).
Second, we can identify = N(m, Σ) = (m, Σ) E Rd x SPD(d) and T.,B((R) (a, S) E Rd x Sym(d). Here, SPD(d) is the space of Rdxd symmetric positive definite matrices, and Sym (d) the space of Raxd symmetric matrices. By writing an affine map as T(x) + S(x m) for fixed m (which is the mean of p), any affine map starting μ N(m, Σ) be parameterized by (a, S). Under this identification,
view BW (Rd) space product Riemannian manifold
31

---

of Rd SPD(d) (Appendix A.1.2). Then one can parameterize every quantity in Appendix A.2 by this product manifold sense. For instance. the vector corresponding to the optimal transport map is (m1, 2 1/2 ( وn 1//3 1/2).
Then, we can define Riemannian metric, exponential map, logarithmic map, and SatresWasseerteinn gradient in terms of parameters as well.
Definition A.36 (Riemannian metric of Bures- Wasserstein space). Let (m, E). The Rieman- nian metric of BW (RR) is define by ((ao, So), (01, Si)), (wo, ai) (r(
Definition A.37. [LCB+22, Appendix B.3] Let μ4 (m,;22)) The exponential map and logarithm map in BW (R.) defined by
exppo ((a, 5)) N (a mo: (S + ()E +]);
log (11) (m² mg, E 1/1 (x) /,/
-).
Note the exponential map only defined when det(S 1) X 0 (more precisely, for all geodesic interpolations to be well-defined, needs - S). Hence, Bures-Wasserstein space is not geodesically complete.
Definition A.38. [LCB*22, 22, Appendix B.3] Kures-Wassersteie metric of the functional FF can be written function R x SPD(d), the space the mean and covariance. Then, for E R and E SPD(d).
Gradøw F(m, Σ) 2),
See [LCB+22, DBCS23] for further discussion.
Using the isometry between the function representation and the vector-matrix representation of BRR"W, we can define the following operation, which can be used to construct the (un-projected) parallel transport.
Definition A.39. For S) T,, (R and (b, R) E Tpp BW (RR), we have the following operation.
S) (b. R) = (a + Sb
In particular,
(m) (m) (a, 5) = (a, SET 1/2 (x/ Σn 12/ //10//////1//////////
Some works adopt an alternative definition of the Bures-Wasserstee Wasserstein metric; we make a remark that this definition is equivalent to the one we present here. This remark plays a pivotal role when we conduct actual calculation in BW (Rd) space (Appendix C).
Remark A.40 (Equivalent formulation of Masse-Wasserntiin metric). In some works (e.g.. [HMJG21J) BW (Rd) metric is defined as ((a, S), (b, R)) (a, R) + } (S))) where (S) is the Lyapunov operator defined via the solution of Ly(S)) ELz(S)= S. While it has the different form with what we introduced earlier, these two formulations turned out to equivalent: our formulation is from Wasserstein perspective, and the other formulation is from Riemannian perspective. In our setup, define the tangent vector directly parameterize the optimal transport map. That said, this does not directly fit with the Riemannian framework. For instance, if we con- sider the curve 7(t) exp,,(taa, defined by our exponential map, then the velocity at t = 0 is (() = (a, SY + ES), which does not coincide with the tangent vector (a, S). By contrast, under the Lyapunov operator based definition, the initial velocity exactly (()) = (a, S). However, since there is one-to-one correspondence between SΣ + ES and for a given Σ, one may regard these two definitions as equivalent by identifying the tangent vector with to S whenever the velocity (() SΣ + ES appears. One can change all corresponding quantities accordingly, and these two definitions turned be equivalent. We have chosen our formulation because it leads simpler algorithm (5.3) that avoids solving the Lyapunov equation.
32

---

Lastly, we end up with the analysis of the curvature of BW (R").. In particular, we show the result that even BW BW(R) space does not allow the curvature upper bound, indicating that the 2- Wasserstein space does not have the curvature upper bound as well.
By applying Corollary A.22 and the flatness of Euclidean space, we obtain the following result: Corollary A.41. For any f BW BW(R*) (R*) and ((),5), (6, R)} orthonormal vectors in T BWBW(R")
Rd x Sym(d),
K (BW(R ((a, S), (b, R))
tr(SER)) Sym, (R4xd) R)-
Therefore, to analyze the curvature of BW (R"), it is sufficient to analyze the space of positive definite matrices, without accounting for the mean component. In this regard, without the loss of generality we consider μ = (0, Σ). Then, since is symmetric positive definite matrix, it is diagonalizable, and therefore we can write and all real positive eigenvalues Aq. Then, it is known that Sym(d) spanned by the following orthonormal basis [TTkk99]:
PD(A,) with P being an orthogonal matrix
P(En + Ea
P(En- BnD PT Ar + +1 Sig
P(Ey + Ej:
VA1+ + Xa
Cig
VA, +
IsiJsa
where Fiy a matrix with only its (i,j) entry and O otherwise.
Using this orthonormal basis, we can characterize all of the sectional curvature in SPD(d) follows: Lemma A.4 [Tak09][Sectional curvature of Bures- Wasserstnnr space]
3A,
fis+,fis) = (A: + A3 (A1 + +a)
(i = Lor = d),
3d,d;
K(eass = (A: (2) + Ax)
(+
12drdj
Kieisyfis) (A; + 21)3
31,
K(fij, fai) =
k+
(A; + );) + Ac) An)
K (any other combinations) = 0.
This explicit form indicates that the curvature upper bound at μ depends on the smallest eigenavalue of the covariance matrix Σ. Since the space of symmetric positive definite matrices does not have the uniform positive eigenvalue lower bound, BW (RR)) does not have the uniform curvature upper bound. See [Tak09] for more discussions on the sectional curvature of BW (RR) space.
In general, the curvature of a submanifold and the curvature of its ambient manifold needs not be the same. However, if the submanifold is totally geodesic, by Gauss formula [Lee18, Theorem 8.2] and the fact that the second fundamental form vanishes [Lee18. Exereise 8.4], the curvature of the submanifold coincides to the curvature of the ambient manifold. Since BW(RR) is a totally geodesic submanifold of the 2- 2-Wasserttein space [CL.20], Lemma A.42 implies that 2- 2-Wasserstein space also does not have the sectional curvature upper bound.
B Deferred proofs
B.1 Deferred proofs for Section 3
Proof of Proposition 3.7. If f is VT -eoodeessyyllyy smooth with base b e M and VT -eooddessiyally convex, then we take
exp; ( (It) Grad f(y r Grad f(x)) + log, y)
33

---

2 E N from the condition we assumed. Then,
f()) f(v) [(x) f() f(2) f(v)
S (VT² Grad (xx) logg logo 2),, + (VT Grad [(v), logs logs 1)),
L
|llog,
logs vll
2
-VTT Grad f(x). logs x logs n),
|TT Grad f(x), logs = -ogo w)
(VTz, Grad (v), logo = logs v),
L
|logs = logs vllō
2
VT Grad f()), logs logs v),
1
|Tb Grad f(v) VT Grad f(x))](
2L
B.1.1 Additional discussions on Proposition 3.7
In Euclidean space, the important property of convex and T-smooth function is that the following inequality holds [Nes14, 14, Theorem 2.1.5]:
f() - -(=)f((=)(v)-) (V (z))
2L - vf()) 20.
(B.1)
Such inequality is sometimes referred as co-coercivity type inequality or interpolating inequality, and works as the core inequality for some optimization algorithms, e.g. algorithms based on Performance Estimation Problem (PEP) [THG17, AP24c] or some recent adaptive methods [SM25]. A natural Riemannian analogue of (B.1) would be written as follows:
1
f(v) f() - (Grad fz)), y)
||T Grad f(y Grad fx)]] 2 0.
(B.2)
2L
We show the same proof strategy in Appendix B.2 can be applied obtain the sufficient condition for (B.2) as the special case.
In fact, we provide more general result. We show, if f is generalized geodesically convex and for all x, E N, and f satisfies = == exp, (- (Grad f(v) - r Grad f(xx)) E N, then we show (B.2) is equivalent to geodesic L-smoothness. Hence, generalized geodesic convexity and geodesic smoothness, under additional technical assumption, imply (B.2).
For this result, we will use the L-Lipchitz gradient for the definition of geodesie L-smoothness, as defined in Definition A.17, Recall L-Lipchitz gradient implies quadratic upper bound (Definition 2.2) by Lemma A.18.
We need introduce the notion of co-coercivity.
Definition B.1 (Geodesic co-coercivity). A differentiable function f N + R is called geodesically co-coercive for all x, y E N
T Grad f(y) Grad log), v) 2 ||r Grad f(y) Grad (x)
The geodesic co-coercivity condition links L-smoothness and (B.2). The next lemma is a general version of Proposition 3.7, which shows the relationship between L-smoothness, co-coercivity, and (B.2). Lemma B.2. For a differentiable function f N The below relationship holds:
(B.2) 2 geodesic co-coercivity @ geodesic L-smoothness.
In addition, suppose for all x, y E N, satisfies exp, (- (Grad f(y) - ry Grad f()))
N. Then, i is generalized geodesically convex.
geodesic L-smoothness (B.2).
34

---

Proof. (i): By applying (B.2) for (z, y) and (y. and using Lemma C.9, one gets
f(y) [(x) - (Grad [(x); log, v) 2T. ||T Grad f(y) Grad [x) 0,
f(x) f(v) + (T Grad f(y), log, y) 2L Summing up two inequalities, one gets
||rg Grad f(y) Grad f(x)|l O.
(Ty Grad f(y) - Grad f(x)) log, v) 2 L [ry Grad f(y) - Grad f fx))
(ii): Using Cauchy-Setie Schwartz inequality on the co-coercivity condition, one gets
1 L, [rg Grad (vv Grad fx)]] ||r. Grad fwy Grad [x)) [|ogg yl
Since ||log yll = dx,, y), one gets the result.
(iii): We follow the same proof strategy in the proof of Proposition 3.7. Take expy (-z (Grad (v) T% Grad [x))). Write f(x) - f(v) f(x) [(=) + f(z) [(v). Then,
using generalized geodesic convexity with base y and Lemma A.18.
f(x) f(y) f(z) f(=) + f(=) f(v)
T Grad fxx) log, 2 - logy x)) + (Grad (Vv), 2) + T
2 |log z||?
ry Grad f(x), - (Grad f(y 1 Grad f(x)) log,,
1 Grad ((v r Grad [(x))
Grad (wy, -
1
+ 2L |Graad f(y - r Grad fx))11
T Grad (xx) log,, 2) ||Grad f(y) r Grad fx)
2L
1
(Grad fx), log, y)
||Lg Grad f(v) Grad f (=)]]
2L
Here, again used Lemma C.9 for the last equality. This is equivalent to the desired inequality.
B.2 Deferred proofs for Section 4
This appendix contains the proofs of Section 4. While the overall proofs follow [AP24c], to clarify that the Riemannian settings are properly taken into accout, we provide the full explicit proofs.
Before we proceed, we introduce simplified formulation of Qis:0- By simply expanding the squared norm term, one can write Qis:b as follows:
Qiz: = 2x(x)) 2f(1;) 2 (TT Grad f Iog, I; -oogg Tz),
- -||vT Grad f(x)) |
| Grad f f(x,)
1| 2, + (VTT, Grad f(z,), VT Grad ((x)))
This formulation will be used frequently for the rest of the proof.
In addition, we also use the following formula for the intermeidate calculation.
Lemma B.3. Set = 1. The The following equality holds.
zn Grad f*xn) 1. ||logo x. log, xo v0l1.
loga x., log, x, + (2ra)
-VTb
-4r2) -1 yTb Grad (1.
1/2
1
+ (VTO,
Grad / (Ta), logs x. logo Fin
n-1
n-1
th VT! Grad Gradd(
+2
Σ ? (VTu, Grad (11), x. log, Zi
1=0
35

---

Proof of Lemma B.3.
1 VT Grad f(x))
LHS ||log, x, log, zolls + |[log v. - logs Fn 11 +
4r
(log, logg 2n. VT Grad f(zn))
+ 7k
= ||logs 2. - logs 2olli + |llogb v. - logb I1l + ||log, Zn - -lggb In- 11b
(logs I. logo In logs In logo 1/1
VT Grad f(zx) || + (logs Ct logg 2n, VT on Grad f(xn))
+ 4r
@ |logg, x. logs xoll? + |loge v. logs -n- 11: + n VT Grad f(on
(logs x. - llog In- VT an-1 Grad f(nn-1)//
+ 2m/n=
VT Grad f(x)) + rb
[ (log, x, - log, Rn VT Grad f ((n))
+ 4
For (i), we used log, 24 log, 2;-1 = -7:-111 VTT Grad f(z1-1).
Now, the same decomposition on log, T- log, -n- by (log, Tx logs Fn-2) - (logs Fn-1 log, 22) and use log, H; log, -;-1 = 1/--11111111 VT procedure until zo yields the desired claim.
2c-1 Grad (xx-1). Iteratively conducting this
The following lemma is the main component of the proof of Theorem 4.1. Without loss of generality, set L = 1. We also set n 2 1 for some k E N.
Lemma B.4. Let the conditions of Theorem 4. I be true. Then, for suitably chosen Xij 2
Σ
AggRigis
- -f(xx= f(xx)) + |log log, zoll.
Uf......
(B.3)
10g, v. logg 2n +
1 VT Grad / (xx)
2r
Once we establish Lemma B.4, the proof of Theorem 4.1 is direct.
Proof of Theorem 4.1. First consider the case L By Lemma B.4 and (3.1), gets f(xx) f(x.) S |llog, x. log, xoll? This proves the desired convergence rate for T, = 1.
For general L. let g f.f. Then, by the linearity of the vector transport (which is assumed) and the Riemannian gradient, g satisfies (3.1) with L 1. By applying L case on g. one gets
1 1 (f(xn) f(=))) g(x,) (ax)) ) S r [log, zo log, 11.11-
L
To prove Lemma B.4, as in [AP24c] we will prove it by induction.
We begin with the base step the induction.
Base Step First, show (B.3) is valid for 1 (k = 1).
Lemma B.5. For any arbitrary initialization zo E N, consider the following VTRGD update (4.1)
Øt expo (l0g, xo TDTT Grad f (17))
where 710 = =p-1 11 Choose dez the same as in [AP24c, Example 2), i.e.,
A00 A X
Ao1
Ao+ Aie
0
A11
=
0
(B.4)
X
X,e)
0 1
2r1
Then, the equality (B.3) holds.
36

---

Proof of Lemma B.5. Observe the following calculation:
1
Σ Ad
AisQish = pQou + Q10.D + (p - 1)QQua + - 10Q00 +
Q+1.0
2ri
fax)) - -f(rL) 20 (VTo,, Grad (1i)), 2o logs 21) 1), 1||uTo Grad (xi) 2
l
p ||ITT Grad (xxo) 1 2p (VT Grad / (2)1 VT Grad ((0o)
20 (VT Grad f (Xo)) log, 11 log, 20 ) |Tu Grad ((x) 2 |TT Grad f(200)1
2 VT=, Grad f(11), VT Grad f(zo) (p 1) 11||TT |T Grad f f(zi))1
2 -0-1 (VT 20 Grad (1r), logs v. - logs logg.2)) 20),
(0 - 1) | Grad Muo)) ||
(VT x1
1 VTb Grad f (wai)l ||? Grad f(z1), log, 11 )11, 2r1
Th
fxx) f(x1)
+ +2 - 20/p) ( (VTT). Grad ((x),, VT Grad ((x0)
2ri + +2p) ||uT Grad ((aaf(a (2p 2npu
|T Grad (ro) l:
2(p - 1) (
(VT no Grad (100), (ogg), x. - -log 20),
1
(VTT mn
Grad ((x)), x. - logo 211
(
f(xx) - f f(11) nă IT Grad ( 113
1
|Tb Grad f (a)))l ||
ri
4r
-2o (VT Grad f(0o), logs x. - log, 200
(VT's, Grad f(x1), (01)),og, logs x. - logg n1);8
ri
R.S..
Here, for (i) used again log, Xi+1 log, æq -loog, Ty log, 14411 -/: Grad (x;). and for (ii) we used the explicit quantity of no. p. and r1. (iii) holds from Lemma B.3.
Induction step Lemma B.5 validates that (B.3) holds for the base case = 1. In this section, given
we have the inequality (B.3) for = 2 - 1 number of iterates, we show by merging two silver stepsizes, one can get (B.3) for 2n + += = 2k+1 - number of iterates.
Lemma B.6. Fix 2* 1. Take { C N sequence induced from the silver stepsize VTRGD. Suppose there exist x(4 2 0 such that (B.3) holds. Write
Oij- 1 1 m. + (1 + 2p) (k)
1j-n (1.=ntt...... 2n+1,++
where * n - 1 is understood to mean *. Define
2 (k+1) = oig + pTj (.
-2pm1 (i=a..jnt1
1 1 2p"
+ + p 2rk
1 ((=))=n) +
11 ((=+,=2n+11)
2rk+1
2rx
+ pll 2n 13 + p² (i=2n+1:jm))
+ (1 [i=n)==] + (2p - V2 x+1)1 (112))11,11
Then, X
(1+1) satisfies
f(x.) - f(zan11) + |loog x. log, 2ol1
(k+1) Qups
T641
37

---

logg, I- logb 32n
1
+
2rk+1
VT
22n+1
Grad (1
In particular, if Xig is chosen as Lemma B.5, then Xij 2 0 for all E IN and i,j = 0, .,.... 2h
1,*.
Proof of Lemma B.6. For the simplicity of the notation, we write gi = Grad f(x.).
From the construction of Jij, we have
Tit Vig.
E
10 (1) Qny - (1+
Σ
(R) x 1n Ritib.
LJ==0...22n+ 1,+
LJ=.,..,...
ijjntt 1 1,,
Since we assumed (B.3) in the induction, we have
Σ
Qiplip
1
(f(x.) - f(xn) + |loog x, log, zull?
rk
1 VT an Grad f (xxn)
logs 2, - -logb on +
2rk
b
1 + 2p (f(x.) f(x2n+1)) (1 + 2p) |logg x.
+
logo In 11
Grad f( 12
- (1 + 2p)
log, I. - -log, I2n+1 + 2Tk zan+1
VT
We subtract Eij aiyQ from the RHS. Using Lemma B.3, gets
RHS- - Qijei
15
2 2p2p)
1
1 1p
f(xx) ff(xn)
f(22++1)
Tk+1
rk
Tk
Tk
rk+1
2n
2n
2p
Wi
| Grad (1xi) 112 + 4p
Σ
" (log,, vi - -log, the VT x. Grad M(xi) )
$=
i==+1
2
|
x., Grad f(Z)) 11:
2mm
1
(108s T- - logo In, VT z, Grad ((E)))
4r
rk
1
1 2p)
| m2ni1 Grad f (122+1)
2 1/
4r11
Ar7
1 1 1ppp)
(log, - -logg 12n+1,
VTb
Grad f(z2+
Tk+1
Tk
12nt1
We now subtract the rest of terms in LHS. After careful calculations, one gets
RHS- Σ 2 (k+1)
Quyi  = 2p (1 + 1
VTb x.. Grad f(x,)) |1; + 2p Σ
2n
n² |TT Grad f(x:))
1
imn
2n
2p
Σ
Th (VT') Grad (xx)), logo - 1o88 x3: 1og).;))
(=++1
2n
+ 2p
Σ
Thi VTT Grad f(xi), log, 22n41 logo /
(=++1
2p
Σ
Th (VT', Grad f(x)), VT², Grad f(xn))
i=++1
38

---

2n
Σ 11 (VT', Grad f(x;), VT #2n-1 Grad f (122nt))
2p
i=++
+ 2p (VT s2n41 Grad (122+11) logs In - logs 12n+11
2pth (VTO,
on Grad f(xn), log, Tgn+1 - logb Fn
20(0 +1) (VT Grad ((x)), VT #2n+1 Grad I (122+11), A.
We show A 0, which implies the desired equality. To this end, note VT Grad f
(log, Ti+1 - log, æz). Plug-in this quantity to A. Then, the second, third, and fourth terms can be rewritten as
2n
VT Grad f(xi) 1:-
Σ m (VT z, Grad fxx,), on -logo 2: i=++1
2p
m
+ 2p
i=++1
2n
+ 2p
Σ T ( (TT Grad f (xx), logs #2n+1 - -logo 2:),,
1=++1
2n
p Σ
2 Iloggs Ti+l - log, xz 11 (logs Ti+1 logs Fi, logg Vin. logg 2i)s
f=++1
(log, 21+1 log, 24, logs 12n+1 log, ntt/
(B.5)
2n
|logs on -logo 12+1 |logo In - -logs x;11.
f=
+ +lllog, I2n-1 log, 11+1ll. - |llogs Ø2n+1 log, x; =11..
/ [1oggs In log, I2n 11 |logg In logo xn+1 11
|loge #2n+1 - logs In+1 11:)
-2p (logs Fn.
log, 1n+1, log, T2n-1 log, 1n+1)b
2n
m
Th (VT2, Grad ((x)), VT' Grad f f(x:))
2pm
i=++1
For (i) we used the telescoping sum, for (ii) we used the fact
||a bl12 - llaa- - c|| 116 el1 = -2 (a c, b-b- c), and for (iii) we used the fact log, #2n+1 - loga In+1 = Likewise, using the same fact,
Ei 1i VT Grad fa;))
2n
2p( VTbh z2n41 Grad f(F2n+1) log, In - loog Ø2n+1
2p Σ 7h (VT x2n-1 Grad (F2n+11) VT Grad f(x.)) b'
imn
2n
2pth (VT&,, 2n Grad (Ixn,, logs T2n+1 logs Znz
2p
17 (VT4, Grad f(xn), VT Grad f(xs)),
==
(B.6)
Pluggin-in Equation (B.5) and (B.6) into and using f/o + xk yields
A = 21 (1 + p²11) |PT Grad f(zm)) 1 2pm. (VT) sants Grad f(22)+1), VTb Grad f(x))
-pp (VT2,, Grad ((n), VT a., Grad f (zx))
39

---

2p(1 + p1-11 (VT*,, Grad f(xn), VT 22
Grad f(224+))),
= 0
by using the facts that ?/, = 1 + pk-1 This completes the induction argument.
For non-negativeness of the coefficients, the initialization is the same, then by [AP24c, Section 3.2] the non-negativeness is guaranteed as we selected the same coefficients.
Equipped with Lemma B.5 B.6, Lemma B.4 is direct.
Proof of Lemma B.4. Start with the coefficients in Lemma B.5. The coefficients are clearly non- negative, and (B.3) holds for k 1. Then, applying Lemma B.6 gives (B.3) for all k E N and 2 1.
B.2.1 Moving to strongly convex smooth functional: Restarting method
We now turn our attention to the geodesically strongly convex case. Although alternative silver stepsize scheme has been proposed for strongly convex, smooth problems in the Euclidean setting [AP24b], the co-coercivity condition it relies on does not carry over to geodesically strongly convex, smooth problems on Riemannian manifolds. In contrast, for convex, smooth functions the co- coercivity condition admits a natural Riemannian interpretation via generalized geodesic convexity and geodesic smoothness (see Proposition 3.7.
Nevertheless, noted in the main text, one still employ the silver stepsize in the convex, smooth setting by combining it with the restarting technique of [OC15]. Theorem 4.2 shows that applying the restarting method [OC15] to our silver stepsize RGD yields an algorithm that also applies to geodesically strongly convex problems.
Proof of Theorem 4.2. Since f is VT -geodesically a-strongly convex with base and x, is a mini- mizer,
a |logs Im logg x+ 12
f(xm) - f(x,) 2
2
Therefore, for 2h one
one gets
2 (f(xm) f(xx)) S 21TT |log, zo logs x.
|log, Zm log, x. 11.1. a
We first consider the case when r = 0, i.e., we exactly iterate m 2* silver stepsize gradient descent times, by restarting the algorithm from the very last update of the previous runs. The total number of iterations becomes n ml (2* 1)e. Then, one gets the following bound for number of iterations:
|log, Fn, - log, 11, (2kTx)2 llogs Tu logg 1
The term (2xTR) is the rate we obtain for this algorithm. Now, one can optimize the choice of k, to get the tightest convergence rate, by solving
min ( E,k
given (2* - -1)= =
Specifically, we plug-in A = [log, xi
+ Observe pk* +1 1 +
LOR N 1 + pk 2 pk. Then,
2n
2k
2KT
<
< <
24
+ 1
Now, since f. =x=-ii
(2kтк-) ) = exp lflog (2KTK. )) S exp
(11g)
2
< exp (-(10g
n Ton, 2
p
2k
40

---

which is the claimed rate.
For the e-approximate error, ||logs Im
logg I l1 I holds whenever
exp - (1og
P
n
|log, to - log, x. S
2
k loss
This is equivalent to
K
log 2
|loge 2o
logg x. 11e. lon nn, 2log2/lggg00 log(1/e))
2
(p)(p/ log
B.3 Deferred proofs for Section 5
This appendix contains the proofs for the results in Section 5.
Proof of Corollary 5.1. First, choose the base b E (R from the points that makes F being generalized geodesically L-smooth with base b. Then, if Proposition 3.7 holds, then the proof goes exactly same as in Theorem 4.1 and 4.2, once one substitutes the following quantities in the proof of Theorem 4.1 and 4.2 accordingly:
Set M N P2,ac (R*).
Change the Riemannian metric by (::)) (2))) Zzzp (1(1x), (xx))).
Set VT
ty. = Tb.u.. where Te,u, = Tb, In-1 Gradw, F(vn-1) o To Toppa- and Te
L
is an arbitrary transport map (e.g., optimal transport map).
Use cxp;(v) (id + #)
Take logs Tb,
id.
Set Grad ((x to Gradw, F(u). introduced in Definition A.34.
To substitute the left hand side of Theorem 4.2 by (2 R4),, one uses the fact that W2(,,,) & ||Th. id]l < Ii - Ib for any transport map To To, from the optimality the optimal transport map and the Kantorovich coupling.
One thing to clarify is that 2- Wasserstein space is not geodesically complete [PZ20], Assump- tions 3.1 and 3.2 are not direct. However, we claim that the regularity conditions we imposed ensure that all the proof ingredients in Theorem 4.1, 4.2, and Proposition 3.7 to be valid.
To check the claim, first Proposition 3.7 will remain valid the condition on z Proposition 3.7 satisfied. Under this specific geometry, the condition on can be written as for all p, v E (Rd))
= (-1 - (Gradw, Flv) Th, - Gradw, F(R) Tbyu) + To.,
#b
1 (Gradw, F(v) Gradw, F(H) Tom))
id
E (R
#v
The above equality holds since Te,, Tr,, Th when μ, u are the iterates from (5.2) (this because T.,. not the optimal transport map). Precisely, for j > i,
To.ay =(id -idd Gradw, F();-1)) (id- Gradw, F(ug-2)) .... (id - Gradw, Flue)) Th
=Tugies
Since Gradw, F(u), Gradw, F(u) Te,p (v),, the second moment condition is satisfied, and the regularity condition imposed ensures the absolute continuity.
41

---

Remaining part is whether the gradient iterates are well-defined in P2,ac(R)). which is not guaranteed anymore as the space is not geodesically complete. Again, the second moment condition is direct; since ,radw, F(H)) E L (j..), (id The Gradw, F(pa)) also has the second moment. In Ie Gradw, F(un) < dm where dm is Lebesugue measure, so certifies that the gradient iterates are staying (2.a However, there is still one more thing to clarify: Definition A.28 well defined only when all geodesic interpolations exist, which is stronger than merely requiring invertibility the endpoints. Such a condition guarantees the well definedness of geodesic interpolations between two points. Nevertheless, note that our proof' of Theorems 4.1 and 4.2 rely only on inequalities evaluated at the iterates themselves; no interpolating points are involved (e.g., the proofs do not invoke time integrals). Hence, we do not need the the all geodesic interpolations between An and (id - - Grad F(Jn)) to exist. Under this perspective, we can conclude that as long as the gradient iterates stay in (R))) all our proof ingredients remain valid.
addition, we imposed d(id I
In sum, as long the imposed regularity conditions hold, all our proof ingredients remain valid without the geodesic completeness. This completes the proof.
Remark B.7 (Proof for Bures-Wasseree Wasserstein space), The proof of Corollary 5.1 holds the same if we replace N to be (W BW (R4) is totally geodesic submanifold of P, (RC). This justifies our choice of N in Section 5.1.
Now, we prove the results in Remark 5.3.
Proofs on the statements in Remark 5.3. First, we show if dp & dm, I v being invertible implies d(id sVh) Hh. & dm for any fixed s 0, if h Clal (R4). Since Clul (RR), the map T(x) <Vh(x) is locally Lipschitz and differentiable a.c.
Write BR = ((0, R),, and for any nonnegative measurable function g consider gx(2) = min {g(v), ((ulsa). Then, by the change of variable formula and area formula,
I AR
(v( S gk T(x)dp(x)
[ ( gk T(T)u(d))
= - gx(y)
((x)
Σ
det(1 <v=()=)V)(())1' dy.
BR
ZEBROT (w)
We used the invertibility for the division by the Jacobian. Next, using monotone convergence theorem with R + oc, one gets
[ (((
gk(2)
Σ
p(x) dde vV2h(z)) dy.
((y)
Apply MCT one more time with & + oo to get
[ glu)d
g(v)
Σ
p(x) |det(I- -V dy.
Rdi
xET-1(y)
Now, for any measurable function g, can use the standard method in measure theory (spliting
g g to get
p(x))
sl))d(T
g(v)
Σ
det([ sV²h(x))]
Rd
*xT
for any measurable function g. This shows that (id vVh) admits the density m(x)
Σ Exer-1() [det(--<V<(a))'
For (id 1 (Vh- - Vh Te, < dm, the same argument as in the above holds as long as Te E (Rd).
In many applications, Gradw, F(P) Cld (RR) is weak condition. For example, potential energy functional application satisfies this long the potential function is convex and L-smooth.
42

---

which is typical case. In addition, the negative entropy functional H(x) J log also satisfies this condition as long the log density is Clod (R*). The regularity condition the optimal transport map Tu Cloc (R4) have been studied intensively the Monge-Ampere Equation literature, and depends on the relationship between densities of μ and v, but holds in many practical applications. For instance, in our Gaussian application, since the transport map is linear, it satisfies the desired regularity.
Next, we present a complete proof of Proposition 5.4.
Proof of Proposition 5.4. Since the argument is identical for both the 2-Wasserstein and Bu- ves-Wasserstein geometries, we only present the proof in the 2-Wasserstein case.
Fix aribtrary base b (R*)) Let To. and Th be any transport maps from b μ and respectively. From the condition that V is convex and L-smooth R" we have for any ~ b,
1 (VV( - (V
(z)( Tbu(()
20
2L
which is the standard incquality for convex L-smooth function R Take expectation over ~ b on the above inequality. The result follows from the fact Gradw, (+)( VV()), which is from [San14, Remark 7. 7.13] and Definition A.34. Since the result holds regardless of the choice of base b, it holds with any base b.
To substitute W (Mo, (t,) for ||Tb,po - notice the gradient update fen itself does not depend on b. Therefore, one can rewrite the result of Corollary 5.1 as follows:
Te.y. Ib. we notice the above result holds for any b. In addition,
V(u.) ((.)) < rk inf
|Too,to
To rul inf (E [TTo,r o Tppob id]
bEP2,ao
By the optimality of the transport plan, one has inf (R.)..((R.) IlTb... Too. id] (10 po)) which completes the proof.
Lastly, for strongly convex result, we claim V is geodesically a-strongly convex if V is a-strongly convex. Under the claim, the result is direct from Corollary 5.1. To show the claim, for any μ. v E (R write TH, the optimal transport map from j to v. From the strong convexity of V, for any x μ we get
(x)) 2 V(x) (VV(s), 2) [Tx,(()) x11²
Take the expectation over x 1 on the above inequality. Using the facts that Tu is the optimal transport map and Gradw, (-)) (V)..) lead to the claim.
Remark B.8. Note for the above proof we did not use the optimal transport map, and considered arbitrary transport map. Hence, our algorithm is readily applicable to this setting.
C Generalized geodesic convexity and smoothness
The notion of generalized geodesic convexity was originally introduced in optimal transport and has found various usages in Wasserstein geometry, including the theoretical analysis of the proximal operator in the 2- 2-Wasserstein space [AGS08, Lemma 9.2.7], [SKI.20, DBCS23), and its connection to I-convergencee [AGS08, Lemma 9.2.9]. To the best of our knowledge, this notion has not yet been explored in the Riemannian geometry literature. We therefore expect that introducing it in this context could provide new tools for analyzing proximal operators and I--oonvereence on Riemannian manifolds, as it has in the 2- Wassersttei setting-areas that, to date, remain underdeveloped.
In this appendix, we provide some examples of generalized geodesically convex functionals for readers who are not familiar with the concept.
43

---

First, recall the notion of generalized geodesic convexity, which is VT -geodesic convexity with VT = r. Generalized geodesic smoothness can be understood in analogous manner.
Definition C.1 (Generalized geodesic convexity). A differentiable function f N - R is called generalized geodesically a-strongly convex with base b E M if for all x, y E N
(()) 2 f(x) + (T Grad f(x), logg logs x), + |log, y log, oo
If a O, we suy f is generalized geodesically convex with base If is generalized geodesically a-strongly convex for all b M, then f called generalized geodesically a-strongly convex.
We start with the trivial example: Euclidean space.
Example C.2. differentiable, a-strongly convex function f Rd -> R generalized geodesically a-strongly convex.
Proof. In Euclidean space, (x) a-strongly convex, for all v, E Rd
and log, y = y--a Since f is differentiable and
f(u) 2 f(x) + (v)(x), y - r) a
z=1²
= f(x) fV x)) (v - b) -x b)) + a [1 b) - (x -b]1
2
Now, we move to nontrivial examples: non-Euclidean manifolds. As mentioned in the main body, this concept has already been widely discussed in the Wasserstein space. Therefore, there are some known examples in 2- Wasserstein space. We first introduce some generalized geodesically convex functionals in Wasserstein space: potential energy functional and internal energy functional.
Example C.3 (Potential energy). Consider a function V Rd
functional V(u) Ex~ul [V (X)] called potential functional. If V is a-strongly convex (L-smooth) in Rd, then V geodesically a-strongly convex (resp. L-smooth).
This is duplicate of Proposition 5.4.
Example C.4 (Internal energy). Let F (0) (-00, 00] be proper, lower semi-continuous convex function such that
F(s)
F(0) = 0, lim inf a10
00 for some a > d 2
sa
Consider functional Hy (R + R defined by
Hr(H) = f F(u(x))dr.
If the map sd=(s=d) is conver and non-increasing in (0, oo), then the functional Hr is generalized geodesically convex.
We refer to [AGS08, Proposition 9.3.9] for the proof.
Remark C.5. Some widely used choice of F satisfying the conditions are as follows:
1. F(s) = s slogss 8. This choice leads to Hr being the differential entropy functional.
2. For any 1, (()) 59,
3. For 2 1 1/d, F(3) m-
Now, we present examples Riemannian manifolds. We begin by providing sufficient conditions for generalized geodesic convexity, which turns out be useful in verifying the generalized geodesic convexity for a given functional.
44

---

Lemma C.6 (Criteria for generalized geodesic convexity), Fix N. For any z, E N, let yet) be any curve such that 7(0) = z. r(1) and ((0) I (logs y- log, z). If a differentiable functtio function f: f:N - R satisfies either one of the following conditions, then f is generalized geodesically convex with base E N.
1. Zeroth-order criterion: (1 (1-1)f(2)+ + (ʃ (1) all t E [0,
2. Second-order criterion: Le ()))) 2 0 for all 1 E (0,
Proof. 1. Zeroth-order criterion: ((t) == -1- + - (f (t).. Then, x(t) 2
and (00) 0 is the global minimizer. Since f is differentiable.
v(t)
d (f)
S d'(0+)= lim +-0+
f(y) f(z)
dt 1=
=f(v) - -f()) - (Grad (ax), 1o (log logs x))
f(v) f(x) (T² Grad (x) logo y log, x),-
2. Second-order criterion: By Taylor's theorem,
f(y) = f(x) + d
(f + / (1 (L-t) (fo o (t)dt dt ==-
d?
di2
2 fx) + (Grad (x), ( (1og, y - log, x)) = f(x) + (T) Grad f(x), log, y llog x)-
Remark C.7 (Existence 7). It is natural to ask whether such curve (t) exists. In fact, as long as the exponential map defined for sufficiently large neighborhood of x, there always exists curve satisfying the conditions. For example, in complete manifold, such curve always exists. Let v(t) g (logs log, x) 2 (log. y T (logs y -log,, x)), and define 7(t) exp, (u(t))- Observe 70) = and -(1) y. Furthermore, since the differential of the exponential map is the identity at the origin, by the chain rule
(0) dexp, (x0)) =1g llog, x))
In certain Riemannian manifolds with particularly well-behaving exponential map, simpler curves can be used. For instance, in the 2- Wasserstein space, more natural choice of curve is available. Fix a base π E (Rd).. For any μu, E a2 (Rd), let (() == exp,((1 t) log, 4 tlog, v) = - 1)T= + (ITzu)#t be a curve. Then, 70) = ,1, 71) = v, and the velocity vector field corresponding to r(t) is 24 = (TT,y - Ta,,) Tyy()) (DBCS23, Appendix B.2].
As specific example, consider the entropy functional SPD(d) space. This example will show how one can verify the generalized geodesic convexity using Lemma C.6.
Example C.8 (Entropy of Gaussian). Consider a functional H. SPD(d) + R defined by H(A) = - log det A. This functional in fact the entropy functional of the multivariate Gaussian distribation N0,, A) (up to an affine trunsformation). There are two natural Riemannian metrics in SPD(d) space (FAP* 05, PFA05, BH06, HMJG21, Ngu22, TP22, KPB25).
1. Affine invariant metric: dAI(A, = ||logA-1/220 /-/ and (S, A = tr(A- for S, R.R+ E Sym(d). This metric induces non-positively curved geome- try on SPD(d).
2. Bures- Wasserstein metric: drww (A, tr(A) + tr(B) 22 and (8, R), ir(SAR) for S, R Sym(d). This metric induces non-negatively curved geometry on SPD(d).
Both geometries originate from the geometry of zero-mean Gaussian distributions. The metric d' arises from the Fisher information metric associated with zero-mean Gaussians [Nie23], while the metric dew corresponds to the Wasserstein geometry of zero-mean Gaussians, described in Appendix 1.2.1. Under both geometries, ((A) is generalized geodesically convex.
45

---

Note that dew corresponds to the 2- W-sssersteinn distance between Gaussians, so the result for daw is special case of Example C.4. Nonetheless, we present the proof entirely in the language of Riemannian geometry to demonstrate that the notion of generalized geodesic convexity remains valid purely within the Riemannian framework.
Proof of Example C.8. In both cases, we apply the second-order criterion from Lemma C.6. The general strategy is construct a curve that satisfies the required conditions with respect fixed starting point, endpoint, and base point. The specific choice of curve should reflect the underlying geometry. Once the curve is chosen, compute the time derivative of the functional along the curve, this can be carried out entirely using matrix calculus, without explicitly invoking the Riemannian structure.
We will use N denote the arbitrary base point, and Mo, M1 to denote the starting point and the endpoint of the curve.
1. Affine invariant metric: We first construct curve satisfying the desired property. For simplicity, write X; := 3.N//M,,N-112 Define c(t) := At + BB² where
A = log 1 - log Xo.
= log(X, 1/2 X, 1/2) - (log x1 - llog Xo).
We now consider a curve on SPD(d) defined by
((t N/111/2222
Here, the exp usual matrix exponential, not the exponential map. We claim this is the desired curve in Lemma C.6.' First, M()) /2/22 Mo. M(1) X1/2X,N1 Mi. Now, we check (0)) Note
(' = (xx
= 12 x2/2 AX 1
Now, since Mo
N2/2X MoN-1 N1/2 1/2, Hence, (MoN-1)1//2
x1/2x] 1122 This leads to
M' 11
= -MMNN
This exactly coincides to Mo (log Mi- logn Mo) on (SPD(d),da) dar)?
Now, since we obtained the desired curve, we compute
2 H( First, observe
de
1
H(M)) = (log det exp(c(t)) + log det N + log det Xo)
2
1
(log exp tr(c(t)) + log det N + log det X²) (tr(c(tt) + log det N + log det Xu)
2
where we used the well-known matrix identity det exp(Y) exp exp tr(Y).
Thus,
d, H(M,) = - - -rt
di
--trlog(( (oogg(x010 1/2 X 1/2 1/2 + ir aroo X1) tr(log Xo)
log Xo log det X, log det X1 log det Xo- 0
'In fact, this curve is constructed as in Remark C.7. "For the formula of the parallel transport and Riemannian logarithmic map on (SPD, dar), see INgu22, Supplement 1.1].
46

---

from the well-known matrix identity tr IYg = log (Y). Hence, by the second order criterion of Lemma C.6, H. is generalized geodesically convex with base N.
Since the above result holds for arbitrary choice of N E SPD(d), we get the generalized geodesic convexity of H.
2. Bures- Wasserstein metric: We again start with constructing curve satisfying the desired properties. As noted in Remark A.40, in this setting we must match the tangent vector corresponding to (0) with Mo (logN M log,v Mo), rather than matching M0( directly. We consider y = (0, N), pig = [0, Mo), and μl = (0, M1). From Appendix A.2.1, the optimal transport map between O-mean Gaussians is linear map. Therefore, for any To, 11, we denote BLo.L, to be the matrix corresponding the optimal transport map between To Lo), ` L1), ie..
T T (1) BL x. Now, consider curve (PD(d) defined by
(()= = ((1 1)I + LBN,M, BM,,N) Mo ((1 - 1)I + (BN,M, BMO.N)"
Then, M()) Mo trivially and M() M1; for any X N(, Mo), on the one hand M, BM X Te.ms (X) NO, Mi), and on the other hand BN,M, N X
N²), meaning (BN,M, BM,,N) Mo M, BM,) M1. In addition, since (0) = BN BM Mo + Mo BM from the identifica- tion in Remark A.40 the tangent vector corresponding to (0) is Vo- = BN.M, BMo.N - -I = IN (Bx,M-- BN.,,)). Therefore, the curve M (t) satisfies the conditions in Lemma C.6. #5 Then, for all & 10,
(0, (BN, M. (M..N)Mo(BBN)
I Mo
Now, we compute
de H(M.). First, since Me- = A7 H(M) = -lo det((i)) 1/2 log det Mo.
d² H(M) d12
d²
log det (A₂) = -d tr (A,1A.) - d
d
(A, (A,'(BN,M BaS.N I)) A, (BN, BM - 11) = (r (A, (BN.M, BMo.N 1))
dt?
dt
dt
d
ur
BMo.N - -1)
dtt
tr (4TT(BN,.. BMONN - ( 1/2
tr (14:1/2( BMO.N - 1)2,1/2 20
which is the desired inequality. For (i), claim that A, 1/2 is well-defined as the principal square root for all & E (0, 1). This follows from the fact that both BN,M, Bar,,N are optimal transport maps and thus, by Brenier's Theorem A.24, they are non non-negative definite. Consequently, the product BN,M, BaN also has non-negative eigenvalues. Since A, is a convex combination of the identity matrix and a matrix with non-negative eigenvalues, it follows that all eigenvalues of A, are strictly positive on & E (0, 1). Hence, all eigenvalues of A, are positive for E (0, 1), and then A,1/2 is well-defined as the principal square root.
Again, since the inequality holds for arbitrary base N, we obtain the generalized geodesic convexity of H.
Lastly, we show the generalized geodesic smoothness with base b is not strictly stronger than geodesic smoothness. In particular, as mentioned in Section 3, we show the function f(x) 1/d2 (2 (x.P) for fixed p on Hadamard manifold M is generalized geodesically smooth with base p, while it is not geodesically smooth [CK25].
Before we show the result, we need the following lemma, which shows how logarithmic map changes under the parallel transport.
Lemma C.9. For all x, E N, let I'y be a parallel transport y induced from the geodesic connecting x and y. Then,
my log, - -log,, x.
'More precisely, since the second derivative is zero, functional H. is generalized geodesically linear. "While By,M, BM..N - I may not be symmetric, the formula on the right hand side still well-defined. Consequently, there is no harm in defining the curve via this formula.
47

---

This result is analogous result of y -(x y) in Euclidean case.
Proof. y [0, 1] - M be geodesic eurve such that 7(0) x and 7(1) = y- Then, by definition of logarithmic map, one gets 70(0) log, y.
Now, consider the reversed geodesic (l)) == 7(1 - t). Then, o' -1(1) logy By the property of the geodesic and the parallel transport,
r log, y / (0) -00))
-ogy x.
Now are ready to prove the following example.
Example C.10 (Generalized geodesic smoothness with base is not restrictive). The function f(x)
jd (x,p) on Hadamard manifold M is generalized geodesically 1-smooth with base p.
Proof. On Hadamard manifold, the exponential map is global diffeomorphism. Hence, we have f(z) log,, x |22 addition, the standard fact on Riemannian manifold is that Grad f(z) log,, [AOBL20, Section 4]. Now, show the inequality (3.3) with L 1. In fact, in this case it holds with equality. This can be verified by the below calculation:
f(y)
f(x) (T Grad f (xx) log, y - -log, x²
1
llg y/
|log, 2:
+ (I log, p. log, y log, 2)
2
@
2 1lgp y
1
|log 2:|
(log, x. logp log, x))
2
1
llog 8/
+ 2
|lo 2/|
- (log, x, log. 8)
2
log, y - log, 2
known that the above fxx not geodesically smooth [CK25]. Thus, Example C.10 shows that generalized geodesic smoothness with single base is not strictly stronger than standard geodesic smoothness.
D Implementation detail and additional experiments
This section includes implementation detail and more experiments of our algorithm under different settings. We conduct additional experiments on the problems in Section 5, to show the robustness of our algorithm. In particular, in this appendix we elaborate the following points that were briefly mentioned in the main body.
Because the silver stepsize schedule sometimes uses very large stepsizes, one might ask whether simply increasing RGD's constant stepsize could match its performance. We show this the case: using a constant stepsize above the critical threshold 2/ causes RGD to diverge, while silver stepsize shows the improved performance.
2. We conducted experiments using multiple random seeds and demonstrate that our algorithm's performances are statistically significant.
Furthermore, to demonstrate our method's versatility, we include experiments on an additional optimization problem in the Wasserstein space: the mean-field training of a two-layer neural network. This problem showcases the applicability of our algorithm, and of Wasserstein-based optimization more broadly, to neural network training. In addition, we provide additional experiments on SPD matrix space.
48

---

D.1 Implementation detail
All experiments in our paper were conducted on the free version of Google Colab using T4 GPU, Each task took no more than 5 minutes.
Wasserstein potential funcciionl optimization For the potential functional optimization problem in Section 5. we used Python packages numpy, scipy for the implementation. We generated m, from the uniform distribution on the unit cube [0. 1 For Σx, since we conducted experiments with fixed T, = and a = 10-1, 10-3, 10-7, 10-13 13, we have Amin = T/T. = and Amax 1/a. We placed d points evenly on log-scale over the interval [1/L, 1/o] and used those values as construct diagonal matrix A. Then, uniformly sampled orthogonal
the eigenvalues matrix P from the uniform distribution the orthogonal group ((d) (using Haar measure), and set E. PAP"" We used mo 0 and No the initialization for all experiments.
SPD space optimization As in Wasserstein potential optimization, for C' we placed d points evenly on log-scale over the interval [1/a, and used those values the diagonal matrix A. Then, again uniformly sampled the orthogonal matrix P and set C PAP We used 10 and 10s to make the desired condition number.
D.2
Additional experiments
D.2.1 Potential functional optimization
We conduct numerial experiments on two tasks: the same task as in Section 5.1, and logistic regression. To verify that our algorithm remains effective with a general choice of iteration count unless n is close to the spikes (c.g. n = 2²), we set the number of iterations n = 1500, which is neither of the form 2 nor close to 2 or 2 1. For the inner-iterations in the strongly convex setting for the restarting, we chose m 20 for a 10-1 and m 500 for 10-3, selecting values near the 2 in Theorem 4.2 while ensuring divisibility by 1,500. We compared our silver stepsize RGD with constant stepsize RGD using 1) 1/L (the standard choice), 1.99/L (just below the theoretical threshold), and 2.01/L (just above it). The experiment repeated over 100 random seeds, and we report the mean eπror curves along with 95% confidence intervals. Here, using different seeds can be understood as solving instances a stochastic optimization problem. this regard, comparing the errors across different seeds is a reasonable evaluation.
The results displayed in Figure 4. Figure 4 provides evidence supporting claims:
1. The algorithm performs well even when the number of iterations is not of the form 2 long is not close 2*
2. Our method not equivalent to simply increasing the constant stepsize in RGD; it consis- tently outperforms all tested stepsize choices. In particular, the large stepsize RGD, unlike silver stepsize RGD, diverges.
3. The performances of our algorithm are statistically significant.
In addition, we conduct the same experiments when gradient oracles are stochastic. The results are coherent with the previous observations at the cost of extra oscillations, given sufficient number of gradient samples. See Figure 5.
Lastly, we provide the Bures- Wasserstein gradient descent experiments on logistic regression potential. For response Y; {0, 13 and predictor X; RRd (i I,..., k), logistic regression model assumes Y, X₁ ~ Bermoulli(logistic{{ ()) independently across Assuming an improper prior on 6, the potential V takes the form
V(0) Σ log(1 +
49

---

Figure 4: Comparison between silver stepsize method and RGD for potential functional optimization in (W(R)) with different convexity parameters. For each task, we conduct 100 simulations with different seeds and plot the and 95% confidence interval of the error over the iterates. Columns: From left to right, each column corresponds to K 10 10%, 107, 1013.
Figure Same experiments as Figure 4, but with stochastic gradients. We set the number of gradient samples to be 100. Columns: From left to right, each column corresponds to 10 10%, 107, 10t3. 1013.
It is well-known that V is convex and 1/ |X |
-mmothh smooth. Thus, proposition 5.4 applies with L
We conduct the experiments on this logistic regression potential minimization problem. Note for this problem, the Wasserstein gradient (O)] (VAV does not allow the closed form solution. Hence, we used the Monte Carlo approximation for these quantities. The result is provided in Figure 6. This experiment only verifies the extendability of our method beyond the quadratic case, but also to the stochastic gradient given the sufficient number of gradient samples.
D.2.2 Mean-Field Two-Layer Network Training via Wasserstein gradient
Next, numerically demonstrate the effectiveness of our algorithms for two-layer neural network training. We first introduce the mean-field training formulation for two-layer neural network, which enables us to view neural network training as a Wasserstein optimization problem, and then present our experimental results. For further details, we refer the interested reader to [CB18 18, MMN18, Woj20, FRF22].
Problem formulation One way to interpret two-layer neural networks is to view their function space space of probability measures. In particular, adopt the Barron space formulation studied in [Bar93, WF20, Woj20J. In Barron space formulation, (possibly infinitely wide) two-layer neural network is represented as
[x(x) = E(a.w,b)~a [ (w x + b)]
where a denoting fixed activation function (e.g., ReLU), For instance, m-wwdthh two-layer neural network corresponds to fx.. where T.
-== (aa,,yyhh:)
This formulation enables us to view neural network training as an optimization over probability measures. In particular, becomes the following fisk-functional minimization problem:
argmin R(T)) == Ez- [z),
(D.I)
where f* is the target function, fx is the two-layer neural network, and é is loss function (c.g.. squared loss). The neural network fx. is the risk-functional minimizer and thus the desired solution. Since (D.1) is now just the optimization problem the Wasserstein space, it is possible consider Wasserstein gradient descent algorithms (5.2) to solve (D.1):
Tn+1
(id Th Gradw, R((T)) #T,
(D.2)
50

---

iteration.
Riam diver GE
bg VINLL (og vig.tneee
22
30
Figure 6: Comparison between silver stepsize method and RGD for potential functional optimization (RR) for logistic regression potential. We plotted log ((n)) - log Viin for the y-axis, where Vmnn is the minimum value among all experiments. We set = 20, d 10, and the number of gradient samples to be 100.
In practice, this update operates over the space of functions and is thus not directly implementable. Instead, one typically uses a particle approximation of the probability measure, i.e.,
1
m
(i)
171
where m is the number particles chosen by the user [SKL20, WL22J. Under this approximation, the Wasserstein gradient update becomes
nn+1 = (id Gradw, R(TT))
(m) 2 .6 ((m)) on Gradw, B( Ma (0(")) b("0)-
(m)
Using Definition A.34, it is known from [Woj20] that
Gradw, ((x)(a, we F
Therefore, the particle approximation of the Wasserstein gradient update for two-layer neural network takes the form
(a. (n+1) u.
(n+1) 06 g(xt1)) (af"), w. (())) 1/a If V (a(m) a0)") (6)) (,,
(D.3)
for = 1,..., m. Observe (D.3) exactly coincides with the standard gradient descent update of the
parameters.
In conclusion, the silver stepsize (and, respectively, constant stepsize) parameter updates in two- layer neural networks (D.3) can be interpreted as the particle approximation of silver stepsize (resp. constant stepsize) Wasserstein gradient descent (D.2) applied to the risk minimization problem (D.1). Hence, we consider applying silver stepsize WGD (5.2) on this problem.
Numerical experiments To evaluate the effectiveness of the silver stepsize for this task, we conduct experiments on learning a target function using a two-layer neural network with ReLU activation. Specifically, we consider the simple task of learning a univariate function f [-1, - R.. We consider two target functions:
1. (x) = 30 Ee afowwx + bt), i.e., a width two-layer neural network with fixed parameters a; w; by. Here, a the ReLU activation.
2. f*(x) sin(2
51

---

Emor difforent
Findl Aporoximation
10"1
0.2
ItE
10"7
Berl
0.
4L.mal
00
10-3
8 -1.1
1.2
10
-1.
107
Emor with difietett
Fruil Fandiian Approsimation
1.0
0
(142)
-10
Figure 7: Mean-field training (D.3) of two-layer neural networks. Rows: The first row is the results
from (xx) 30 Σi= az afo(w x + b*), and the second row is the results from (xx) = sin(2rx))
Columns: The first column is the training and test error curve, and the second column the function graph of the learned function.
We use N = 200 samples, with 70% of the data used for training and the remaining 30% for testing. The model is a two-layer neural network with width m = 100, trained using mean squared loss. We set the smoothness parameter
100, and the number of training iterations
2000.
Figure 7 shows the results of our experiments for solving (D.3) using different stepsize schedules. Consistent with previous findings, the silver stepsize algorithm outperforms constant stepsize RGDs with various stepsizes in solving (D.1). While the figure displays results for a specific random seed, we observed similar trends across multiple seeds.
D.2.3 Additional experiments on SPD space
As briefly mentioned in the main body, there is non-trivial family of functions that is favorable to use VTRGD algorithm (4.1), which are the functions of the form (X) p(log det X) for some yp : R -> R. The first favorable fact is that fx( is generalized geodesically convex (resp. L-smooth) on (SPD(d), dar) when is convex (resp. I-smooth).
Proposition D.1. For any convex function & : R - R, the functional fo(X) élog det X) defined on SPD(d) is generalized geodesically convex. Also, for any L-smooth function R + R, the function (X) = (log det X) is generalized geodesically Ld-smooth.
Proof. We will show for arbitrary base B E SPD(d), Definition 3.4 and 3.5 holds with VT=I.
First, we consider (XX) log del X. We first show for any X, Y, B SPD(d)
g G(X) = (T Grad g(X), log Y -lg X).
(D.4)
To verify this, first observe that Grad g X. This can be verified by the definition of Riemannian gradient. Riemannain gradient is defined by the operator satisfying for all H E &Sym(d
(x Grad g H) = (Grad g(X), H) = dgx (H) = tr((X H).
For the last inequality used the well-known formula for the derivative of log-determinant function. Since X e SPD(d), this implies Grad g(X) = X.
52

---

Next, we show x = X. To check this, from the definition of the parallel transport in (SPD(d), dAr) INgu22, Supplement 1.1].
rxx- - (BX (BX-))
= =.
Hence,
( X, 1,,X,og log log x) = (B, lo,nog Y) (B, logn X).
Now, for any matrix M, from the definition of the Riemannian metric on dar and logarithmic map [Ngu22, Supplement 1.1].
(B, logg M) = (r(B logg M) = trt(ttB// 1/2 B 1B
= tr(log(B-122M3 )
log det B -B-1/BMB MB 1B1 1/2) 2 log det B B-1/2 + log det M.
Therefore,
BT B1 logo A logg B) log del A - log det B = ((A) g(B)
which shows (D.4).
Now we are left with the & and v part. For Φh by the convexity of Φ
60 g - g 2 6 ( - ((X)) = o' (x(() (I Grad g(X), logn Y -log X) =(T Grad( (X), log Y logg x)
where the last equality is from the chain rule and linearity of the parallel transport and gradient. This shows @ g is E-geodesically convex with arbitrary base B.
For vh, first observe
g(Y) g(X (T Grad (X), logg - logg X) S | Grad g g(X)l |log Y logg XI|
< X |logg Y - logg X|| = tr(xttXX- xx X) |logg Y log XII = d [log Y log XII-
Then, using the L-smoothness vh.
L oo go) 2b g(X) < ('(((X))(o(()) g(()) ) + |l9(Y) 9(x))1
Ld
(T Grad(w (X), logn Y - log X) 2
||oon Y - -logn X||
which shows & o is r- T-geodesically Ld-smooth with arbitrary base B.
Note affine invariant metric yields the complete Riemannian manifolds on SPD(d) [PFA05], so Proposition D.1 combined with Proposition 3.7 yields (3.1) with any base b SPD(d), hence guarantee Theorem 4.1. On the other hand, since X + log det X is not L-smooth in matrix norm sense, Proposition D.I highlights the strength of VTRGD algorithms for optimization problems involving functions the form (X) yloog det.x).
Moreover, as in Wasserstein space, for this certain function VTRGD coincides with standard RGD. Proposition D.2. For the function of the form fx) = colog X), VTRGD algorithu coincides with standard RGD, i.e., any X, B SPD(d).
expg<log X 1 Grad fx)) expx (Grad fx)))
Proof. We directly compute (4.1) and compare. We write B the base point. First, from the calculations in the proof of Proposition D.I.
T Grad [o((,n) = r s (logdet X,, Grad(log det X,) s o'Ologder del X, X X = s'(log det X.)
Hence,
exp8 (logg Xn, 1 Grad log det Xn)
53

---

Figure 8: Comparison between silver stepsize method and RGD for entropy matching under 6- 6-pseudo Huber loss on (50( We set 2 as the total iteration number, and conducted 100 simulations under different seeds and initializations. Left: å 0.1. Middle: 0.5. Right: 6 1.
exp
(B 1/-1/21log Xn, (log det X, B,)) B1/2
log Tne (log x.,) Ba /2/2exp (B- 1/-1/2 (og X,, B, TT,B11 1/2) B1/2
e "log dot x.,) x,) Xn expx, (-mmip' (log det Xn) expx, (-1m Grad fi (Xx)).
The function ++ log X can be considered as the entropy of the zero-mean Gaussian distribu- tions (up to affine transform). Hence, functionals of the form yolog det :) appears in many practical applications, such entropy regularization.
Entropy matching
For the numerical experiment consider the problem
f6(X) -² (V++ (log det - -) -
8²
which corresponds to the entropy-matchine problem with target value under the pseudo-Huber loss. We adopt the Huber loss to examine how the algorithm's 's behavior depends on the curvature of the objective function (small 6 yields flatter function). Since pseudo-Huber loss is convex and 1-smooth, by Proposition D.I (AA) is generalized geodesically convex and generalized geodesically d-smooth. In addition, by Proposition D.2 we can apply silver stepsize directly to standard RGD. We conduced 100 experiments with T under different initializations, verify our algorithm's stability. The results are summarized in Figure
We additional provide two more benchmark problems on SPD(d) space. The Fréchet mean estimation problem and Gaussian mixture model problem. Note for these general problem we do not have Proposition D.2, so we again need to choose the base for VTRGD (4.1). We again chose = I for both experiments.
Fréchet mean estimation Fréchet mean estimation problem on the SPD space is widely studied problem for both theoretically interesting properties and practical application. Practically, under Affine invariant metric, Fréchet mean becomes geometric mean [FAP*00 05, PFA05] and therefore has many applications. Theoretically the Fréchet mean over SPD matrices is again SPD, so unlike typical matrix norm, this geometry is favorable when SPD constraint has to be involved when obtaining the means over SPD matrices; such constraint is common in covariance estimation problem [KPB25]. Hence, this problem is one of the standard benchmark for RGD methods [AOBL21, KY22J. The Fréchet mean estimation problem can be written for general manifold M as follows: for given (Pi....PP. M, the empirical Fréchet mean over p;
p. == argmin f(x) = Σ ((x,p)). ZEM
This problem is known to geodesically n-strongly convex, but geodesically L-smooth. While the squared distance function + (x. is generalized geodesically 1-smooth with base p. we discussed in Section 3, by summing these together there is single base that makes f to
54

---

Figure 9: Optimization of Barycenter problem. We plot f(x) fmin. where fuin is the minimum
value over all experiments. We set b = I for VTRGD (4.1). Left: = 100 and d = 10. We set L = 200 which guaranteed the stability. Right: n = 10 and = 100. We set L = 100 which again guaranteed the stability.
AD
10--27
Figure 10: Optimization of Gaussian mixture model fitting. We set for VTRGD (4.1). Left: Simple setting, where n 100, K 3, = 2, I. 20. Right: Complicated setting, where 1000, K 5, d 5, L 1000. The chosen guarantee the numerical stability.
be generalized geodesically L-smooth with base b. Hence, neither RGD or VTRGD guarantee the theoretical convergence for this problem. Still, we found out that our algorithm numerically outperforms the standard RGD for this problem.
With M SPD(d), We considered two settings: (n, d) (100, 10), and reversely (n, d) (10, 100), to observe whether dimension sample size matters. Again we chose b = / for VTRGD (4.1). We observed consistent strength of our method. The results are summarized in Figure 9.
Gaussian mixture model Another practical application of Riemannian optimization on SPD(d) is minimization of the negative likelihood in Gaussian mixture model, which is a classical problem in Statistics. [HS20] proposed the reformulated objective function for solving Gaussian mixture model fitting, which coincides to classical negative log likelihood minimization at the minima, and allows the use of Riemannian optimization algorithm [HS20, HMIG21, SBS21J. For given observations = (xi, 1)T E IRd for = I,...,n, n, the problem is formulated as follows:
n
e
argmin 0-((my) x
((0)
log
Σ
IN
(N(U): Sy)
-==1 eor
K
j=1
where
IN (v:: exp (1 1/ 1 v:1)
((2x)d det S
This problem can be considered as the product manifold (Rd)* K x (SPD(d)) One can conduct usual Euclidean gradient descent for vectors wy 's and conduct Riemannian methods in Sy's. Again, this problem not geodesically L-smooth, so neither our algorithm nor standard RGD do not allow the theoretical guarantee. Still, we found out that numerically our algorithm turned out be useful for this problem.
We considered two setting: simple setting (n = 100, d = 2, K = 3, L= 20) and complicated setting (n = 1000, = 5, K = 5, L = 1000). The quantities of L are chosen again to guarantee the stability of the algorithms, Again, we set = 1 for VTRGD 1). The results are aggregated in Figure 10.
55

---

E Changes from the Submitted Version
In the submitted and reviewed version of our paper, we subsequently identified two emrors. Below we document the resulting revisions as transparently as possible.
Error I: Algebraic misstep We originally claimed that standard RGD (1.2) with silver stepsize achieves the O(n log2 P) rate of convergence, when the manifold is non-negatively curved and the function satisfies (3.1) with zj. However, we identified an algebraic misstep when proving the induction step (Lemma 5,4 4 in the reviewed version)*. To address this issue, we made the following changes:
1. Algorithm modification. We replace standard RGD (1.2) by VTRGD (4.1). In contrast to standard RGD, VTRGD is defined relative to a specified vector transport, to which the algorithm and assumptions are tied.
2. Scope. The revised analysis longer requires nonnegative curvature; applies under vector transport-awarg assumptions the objective (3.1). Heuristically, the curvature requirement can be seen as transferred to vector transport-indexed incquality conditions.
3. Function class and assumptions. The admissible function class now tied to the chosen vector transport via (3. I); sufficient condition appears Proposition 3.7. For the canonical choice VT r (parallel transport), this replaces geodesic L-smoothness by generalized geodesic L-smoothness with a single base b. These conditions are incomparable; for instance, discussed in Example C.10, the function + d? dx( b) on Hadamard manifold satisfies the new but not the previous condition.
4. Numerical experiments: We replace the experiments on Rayleigh quotient maximization problem by experiments on benchmarks over symmetric positive definite (SPD) matrices, include a nonpositively curved setting where the revised analysis applies (Section Appendix D.2.3). In this setting we also identify an additional function class satisfying our assumptions (Propositions D.1, D.2), which did not appear in the previous setup.
Error I:: regularity condition Section 5.1, we noticed that we overlooked the geodesic incom- pleteness of Wasserstein space: along discrete updates, the update may leave the class of absolutely continuous densities, (e.g., for large step sizes the update map may be the gradient of convex potential). Consequently, Corollary 5.1 requires regularity condition ensuring that the iterates admit densities. In addition, the gradient update may not be the optimal transport map. We clarified these points in Remark 5.2, 5.3 and provide sufficient conditions relevant to our applications.
What remains unchanged or strengthened Despite these changes, some core points remain intact are further reinforced by the revision.
1. Our central application, potential function optimization on the 2- Wasserstein space, remains intact. In this setting, VTRGD with canonical vector transport VT = To recovers the standard Wasserstein gradient descent. Particularly, the potential energy satisfies the modified assumptions (Proposition 5.4). Thus, our main claim, the first acceleration result for Wasserstein gradient descent in Wasserstein space, still holds.
2. The Riemannian co-coercivity inequality, identified during the rebuttal as key technical component, is recovered as the special case of Proposition 3.7 when = y; see Ap- pendix 1.1.1.
3. Allowing negative curvature broadens applicability (e.g., the log-determinant objective on SPD manifolds verified in Proposition D. D.2).
'The error is in the last paragraph on p. 33 the reviewed version, when subtracting A. include the reviewed version in the supplementary materials.
56

---

Summary The revision corrects the errors via algorithmic modification (RGD (1.2) - VTRGD (4.1)) and additional regularity conditions (Corollary 5.1). The revised algorithm VTRGD is nor the standard RGD, and the updated assumptions are neither uniformly stronger nor weaker than those the original version. However, in some central applications VTRGD coincides with RGD, and those applications satisfy the newly introduced assumptions. Crucially, VTRGD coincides with standard RGD in our main application, the Wasserstein space, Hence, one of our main claims, the first acceleration result on Wasserstein gradient descent, remains valid, and the revision additionally includes log-determinant example on the SPD space, which was not covered before.
57


Reviews:
[{"review_id": "GpIe6vbNt3", "reviewer": "Reviewer_WD9n", "summary": "This work adapts the recent \"silver stepsize\" analysis for Euclidean convex optimization to the Riemannian setting. In particular, the authors focus on optimization over geodesically convex subsets of non-negatively curved manifolds, and develop some sufficient machinery to be able to extend the analysis to the setting of curved geometry. The authors then discuss the specific case of optimization over the Bures-Wasserstein space as a an example (since it carries a Riemannian structure, modulo some details). \n\nOne of the claimed key technical contributions relies on a crucial assumption (on $Q_{ij}$) made by the authors.", "questions": "* One of the key technical assumptions is 3.1. The need for precisely the same inequality has come up several times over the years, and to my knowledge, previous authors left it as an open problem (I feel if one does not assume it, then by paying an extra factor of \"L\" one should be able to get around this assumption, but then the constants might be wrong; however, this feeling is not a theorem, so please don't view it as such), while the current authors put it in as an assumption. That said, it is useful to have identified sufficient conditions (such as generalized geodesic convexity) for this assumption to hold. The authors should discuss the implications and limitations of this assumption more thoroughly, ultimately, because this seems to be one of the few \"truly Riemannian barriers\" that are in the paper.\n\nIn fact, I think if the authors could prove inequality 3.1 without invoking their sufficient condition for some nontrivial settings, that would be a valuable contribution on its own, and would add to the depth and maturity of the current work.\n\n* The numerical instability of the silver step size suggests that there may be a stronger rate for the \"lim inf\" hiding in there? Moreover, is this a problem that also plagues the Euclidean case? How to ameliorate that? Possibly, there is a Lyapunov function that the authors could exhibit, that is actually monotonically decreasing (for instance this Lyapunov function in the authors case might easily come from combining function values with a stepsize x tangent-space distance (because their generalized geodesic convexity relies on an object that relates to such a distance)\n\n*  The hypersphere example, while not geodesically convex, is known to satisfy a Riemannian PL inequality. From Euclidean results, it is known that PL functions cannot be solved at an accelerated rate, so that applies to the Riemannian case too. However, it is still possible that for the sphere, one can bypass that type of lower bound. Might the authors have some thoughts about that?", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "# Strengths\n1. The topic studied seems relevant\n2. The analysis is largely clean, and the authors point out clearly where and how they deal with the metric distortion.\n3. The application presented is relevant  (though, to be a bit critical, one must admit that at  this point using barycenter problems to highlight the performance of a Riemannian method is like continuing to use the  \"MNIST\" dataset as a benchmark).\n4. Localization of the metric distortion challenge to a condition like (3.1) -- more on this in the review below.\n\n# Weaknesses\n\n1. The adaptation from the Euclidean to the Riemannian case seems to not require any true heavy lifting (but this is not that big a weakness, as this view applies to a large body of Riemannian methods anyways). \n\n2. The functions that may satisfy the \"generalized geodesic convexity/smoothness\" inequality seems small (though appendix D does provide some interesting examples)\n\n3. The analysis is (strangely) limited to the nonnegative curvature case, and it would be valuable to see what happens for negative curvature.\n\n4. The empirical results suggest intense oscillations / numerical instability of the Riemannian silver step method, without further control or discussion or amelioration, this behavior can easily render the current contribution to be primarily of theoretical value.", "was_revised": true, "final_justification": "The authors have worked hard to answer all the questions and concerns. I would have upgraded my score, had the paper had sufficient novelty. The positive score that I have for the paper is thanks to its technical achievements, but ultimately, for a higher score, I would be pushed more by strong novelty (and to the authors: novelty is a subjective aspect, so I admit that you may not necessarily agree with my assessment). Nevertheless, the paper does deserve to be accepted, and the authors have been very responsive during the rebuttal process to tackle all other concerns that were raised.", "rebuttal": "We are grateful to the reviewer for their detailed feedback, which we will take into account when preparing the final version, if accepted.\n\n> Using barycenter problems to highlight the performance of a Riemannian method is like continuing to use the \"MNIST\" dataset as a benchmark.\n\nOur experiment addresses the Rayleigh quotient maximization problem, not the barycenter problem. It is a standard benchmark in Riemannian optimization [1, 2]. *We also included more complex problems, Bures-Wasserstein optmization (Section 6.1) and mean-field training of the neural network (Appendix E.2.3).*\n\n> The adaptation from the Euclidean to the Riemannian case seems to not require any true heavy lifting.\n\nWe faced two major challenges: \n\n1. Meaningful geometric interpretation of (3.1) is nontrivial. This required proving Proposition 3.8, which was possible by introducing the notion of generalized geodesic convexity.\n\n2. A key challenge in extending gradient methods to Riemannian manifolds is the lack of closed-form expressions for displacements between non-consecutive iterates. In Euclidean space, one has $x_l - x_k = -\\sum_{i=k}^{l-1}\\eta_i \\nabla f(x_i)$ for all $k < l$. Unlike constant/monotonically decreasing step-size gradient analyses (including the momentum), Euclidean silver step-size analyses employ this non-consecutive relationship heavily [4]. However, RGD only allows the accesses to consecutive terms like $\\log_{x_k} x_{k+1}$, making it difficult to adapt Euclidean silver stepsize analyses directly. Our main contribution was showing that, perhaps surprisingly, pairwise relationships alone suffice to recover the desired guarantees, which we elaborated in Remark 5.5.\n\n> The functions that may satisfy the generalized geodesic convexity/smoothness inequality seems small.\n\nWhile this condition is new and few functions are known to satisfy it, as the reviewer mentioned *we show in Appendix D.1 that it's non-vacuous with examples in popular spaces like Wasserstein, Bures–Wasserstein, and SPD matrices, along with verifiable criteria (Lemma D.6) and concrete examples (e.g., Example D.8).* The fact that it is not vacuous suggests that it is worth further investigation. We believe that continued study will help clarify the scope of this condition.\n\n> The analysis is (strangely) limited to the nonnegative curvature case, and it would be valuable to see what happens for negative curvature.\n\nOur proof does not readily extend to negative curvature case. In this case, the metric distortion inequality takes the form\n$$\n   \\\\|\\log_{x_{n+1}}x_*\\\\|^2 \\leq  \\\\|\\log_{x_n}x_*\\\\|^2 + \\zeta \\\\|\\log_{x_n}x_{n+1}\\\\|^2 - 2\\langle\\log_{x_n}x_{n+1}, \\log_{x_n}x_* \\rangle.\n$$\nfor some curvature-dependent constant $\\zeta > 1$ (Lemma 5.2, [2]). When the curvature is non-negative, $\\zeta = 1$.\n\nNaive application of our technique boils down to showing:\n\n$$ \\frac{f(x_*) - f(x_k)}{r_{k}} - \\frac{1}{4r_{k}^2}\\\\|\\nabla f(x_n)\\\\|^2 - \\frac{1}{r_{k}}\\langle\\log_{x_n}x_*,\\nabla f(x_n)\\rangle - \\zeta \\sum_{i=0}^{n-1}\\eta_{i}^{2} \\\\|\\nabla f(x_i)\\\\|^2 - 2\\sum_{i=0}^{n-1}\\eta_{i}\\langle \\nabla f(x_i), \\log_{x_i}x_*\\rangle  \\geq \\sum_{i,j = 0,\\dots, *}\\lambda_{ij}Q_{ij} \\geq 0. $$\n\nFor this to hold, $\\eta_i$ or/and $r_k$ and $\\lambda_{ij}$ should depend appropriately on $\\zeta$, which poses two challenges if $\\zeta > 1$.\n\n1. Due to dependencies between $r_k, \\eta_i, \\lambda_{ij}$, and $Q_{ij}$, adjusting these parameters jointly is a much harder problem. \n\n2. More importantly, since (5.1) is an inequality (unlike the equality in Euclidean case), coefficient matching technique used in Euclidean case fails, and proving $\\lambda_{ij} \\geq 0$ becomes harder. Our proof detoured this problem by showing that the nonzero coefficients actually coincide to the Euclidean ones. But when extending to negative curvature, since $\\lambda_{ij}$ changes, such technique no longer works.\n\nSince there are interesting non-negatively curved manifolds with broad applications, e.g., Wasserstein and Bures-Wasserstein space, we stayed with the non-negatively curved manifolds for this work. We defer the extension to negatively curved space to future work.\n\n> The empirical results suggest intense oscillations of the Riemannian silver step method ... render the current contribution to be primarily of theoretical value. ... is this a problem that also plagues the Euclidean case? How to ameliorate that? \n\n*The oscillation is expected under silver step-size scheme that involves occasional large step-sizes, as this step-size is designed to guarantee multi-step descent, rather than monotonic decrease in function value that one observes in gradient descent with constant/monotonically decreasing step-sizes (see [8]).* Similar behavior is observed in neural network training with large step-sizes as well [10].\n\nOn the other hand, *our experiments consistently show that the last iterates remain stable across multiple runs (see Figure 4) as the theory suggested*, illustrating statistical robustness of the algorithm.\n\nLastly, while [4] does not provide the experimental result for $n \\neq 2^k -1$, we observe similar oscillations in our experiments in Euclidean case too.\n\n> The need for the same inequality (3.1) has come up several times ... previous authors left it as an open problem.\n\nIn Proposition 3.8, we provided the sufficient condition for the inequality (3.1). Other than Proposition 3.8, we were unaware of study of this inequality in Riemannian setting (except the similar one in Lemma 2.3 of [3], but it is not the same) in any other works. We would be grateful if the reviewer can provide the reference.\n\n> By paying an extra factor of $L$ one should be able to get around generalized geodesic convexity assumption. \n\nWith only geodesic convexity and $L_0$-smoothness, a similar inequality can be obtained as shown in Lemma 2.3 of [3], but this is not precisely the same inequality needed. First, the constant $L = L_0d^2(x_i,x_j)$, now is not global and depends on the iterates. Second, the subtractive term is different from that in Equation (3.1). Since the precise form of Equation (3.1) is crucial for our proof, generalized geodesic convexity appears necessary at this moment. \n\n> The authors should discuss the implications and limitations of Equation (3.1) more thoroughly.\n\nWe interpret (3.1) via Proposition 3.8, as generalized geodesic convexity and geodesic smoothness. Geodesic smoothness is a standard concept, and we provide interpretations with examples in Appendix D.1 to illustrate generalized geodesic convexity.\n\n> If the authors could prove inequality 3.1 without invoking their sufficient condition for some nontrivial settings, that would be a valuable contribution on its own.\n\nEven in the Euclidean case, the proof of Equation (3.1) implicitly uses the notion of generalized geodesic convexity by introducing an auxiliary third point $z$. In this regard, we believe that deriving (3.1) without invoking generalized geodesic convexity would require fundamentally different techniques from those used in existing proofs.\n\n> Is there a possibility of a Lyapunov function type analysis?\n\nWe did not pursue a Lyapunov functional-based argument in this paper. While it will be interesting to explore Lyapunov functions that capture multi-step function-value descent, we defer it to future work.\n\n> The hypersphere example is known to satisfy a Riemannian PL inequality.\n\nRayleigh quotient maximization problem does not satisfy PL. Here the minimizer is $x_* = v_{\\max}$, the eigenvector corresponding to the largest eigenvalue of $H$. The Riemannian gradient is $\\nabla f(x) = (x^T Hx)x - Hx$. When  $x = v_k \\neq v_{\\max}$ is any eigenvector of $H$,  $\\| \\nabla f(x) \\|_{x}^2=0 < 2\\mu(f(x) - f(x_*))$, hence contradicting Riemannian PL inequality. \n\n> PL functions cannot be solved at an accelerated rate, so that applies to the Riemannian case too. However, it is still possible that for the sphere, one can bypass that type of lower bound. Might the authors have some thoughts about that?\n\nFirstly, since Rayleigh quotient problem does not satisfy Riemannian PL condition, the lower bound does not apply.\n\nSecondly, typically the lower bound analysis is obtained by the worst-case constructions. However, many functions do not exhibit worst-case behavior and may admit faster convergence rates than the lower bound suggests (e.g., Introduction of [9] comes up with one example).\n\nThirdly, we are cautious about extending Euclidean lower bounds to specific Riemannian manifolds (e.g., the sphere). The metric distortion inequality (Lemma C.1), which is an equality in Euclidean space, can be a strict inequality on some non-negatively curved Riemannian manifolds. Thus, at least in principle, some Riemannian manifolds may allow potentially sharper convergence rates.\n\nReferences\n\n[1] Alimisis et.al. Momentum improves optimization on riemannian manifolds. AISTATS 2021.\n\n[2] Kim et.al. Accelerated gradient methods for geodesically convex optimization: Tractable algorithms and convergence analysis. ICML 2022.\n\n[3] Ansari-Önnestam et.al. Adaptive Gradient Descent on Riemannian Manifolds with Nonnegative Curvature. Arxiv  2025.\n\n[4] Altschuler et.al. Acceleration by stepsize hedging II: Silver stepsize schedule for smooth convex optimization. Mathematical Programming. 2024.\n\n[5] Li et.al. A simple uniformly optimal method without line search for convex optimization. ArXiv 2023.\n\n[6] Shugart et.al. Negative Stepsizes Make Gradient-Descent-Ascent Converge. Arxiv 2025.\n\n[7] Zhang et.al. Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifolds. NIPS 2016.\n\n[8] Altschuler et.al. Acceleration by Stepsize Hedging I: Multi-Step Descent and the Silver Stepsize Schedule. Journal of the ACM 2024.\n\n[9] Grimmer et.al. Beyond Minimax Optimality: A Subgame Perfect Gradient Method. Arxiv 2025.\n\n[10] Cohen et.al. Gradient descent on neural networks typically occurs at the edge of stability. ICLR 2021."}, {"review_id": "6vO8xOBmFV", "reviewer": "Reviewer_Y9D4", "summary": "This paper investigates Riemannian Gradient Descent (RGD) with silver step sizes. It demonstrates that, for optimization problems defined on non-negatively curved Riemannian manifolds, where the objective function is generalized geodesically convex and geodesically smooth, vanilla RGD can achieve accelerated convergence when the step size sequence is carefully chosen—specifically, using silver step sizes. The derived convergence rate is approximately $\\mathcal{O}(1/n^{1.27})$, improving upon the known $\\mathcal{O}(1/n)$ rate for RGD with constant step sizes.\n\nIn the strongly convex setting, the paper shows that RGD with silver step sizes and restarts converges exponentially fast, similar to RGD with constant step sizes. However, it offers an improved dependence on the condition number: the convergence rate involves a coefficient scaling as $\\kappa^{0.786}$, compared to $\\kappa$ for constant step sizes.\n\nNumerical experiments include minimizing potential energy in the Bures--Wasserstein space and Rayleigh quotient maximization on the sphere. The results confirm that RGD with silver step sizes achieves acceleration over its constant-step-size counterpart.", "questions": "# Some questions and suggestions:\n- The Wasserstein space is not a bona fide Riemannian manifold. I wondered if the theorems 4.1 and 4.2 still applied.\n- The first experiment in the Wasserstein space: maybe also consider a more challenging form of $V$ instead of a quadratic function.\n- The second experiment on a sphere: why did not you choose another function that is generalized geodesically convex instead?\n\n# Minor points:\n- The second condition of Prop. 3.8: should add \"it holds\" before $z$, otherwise it looks a bit confusing.\n- Line 85: A word should be added at the beginning of the sentence starting with $\\exp_{x}(v)$.\n- Line 90: a sentence should be added to introduce/characterize parallel transport.\n- Equation (2.1): should say explicitly the square bracket represents concatenation.\n- Line 104: $\\eta_0$ should be $\\eta^{(0)}$. In equations after line 104, all indices are off by one unit.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths:\n\n- The theoretical results are noteworthy. To the best of my knowledge, achieving acceleration on Riemannian manifolds remains an active area of research--primarily focused on extensions of Nesterov-style acceleration--where progress has been limited due to challenges posed by the metric distortion of curved spaces. In contrast, this work explores a complementary direction: demonstrating that acceleration can be attained solely through the careful design of step sizes, similar to recent developments in the Euclidean setting.\n\n\n- The writing is of high quality. \n\n# Weaknesses:\n\n- The theory only guarantees for iteration counts of the form $n=2^k-1$.\n\n- One of the main motivations of the paper is to relax the assumption on upper bound and lower bound of the space curvature -- where the authors argue that the Wasserstein space is such an example. However, when working with the Wasserstein space, **deterministic** RGD usually cannot apply (except for very special and rare examples like the pontential energy), even for the practically important case of  KL divergence. Therefore, the motivation is somewhat limited. \n\n- The assumption of generalized geodesic convexity -- while being mild and standard in the Wasserstein space -- is strong and less standard in general Riemannian manifolds. Maybe this is the reason why this definition has not been introduced in the context of Riemannian optimization in the literature.\n\n- The numerical experiments, while illustrative, are somewhat limited in scope, focusing only on the Bures--Wasserstein space and the sphere. Moreover, the paper does not include comparisons with Nesterov-style Riemannian Gradient Descent, which would provide a valuable benchmark. Finally, it would be informative to include the theoretical convergence curve $y = 1/x^{1.27}$ in the plots as a reference, to better contextualize the empirical performance.", "was_revised": true, "final_justification": "The authors have addressed my concerns, I therefore keep my score.", "rebuttal": "We are grateful to the reviewer for their detailed feedback, which we will take into account when preparing the final version, if accepted.\n\n> The theory only guarantees for iteration counts of the form $n=2^k-1$.\n\n1. *In fact, under additional geodesic strong convexity, one can establish the same convergence rate with respect to $n$ for general $n \\neq \\ell (2^{k^{\\ast}} - 1)$*. One can proceed as follows: Consider $m = 2^{k^{\\ast}} -  1$ number of inner-iterations and $\\ell$ number of outer iteration, as in our restarting algorithm. Now, for general $ 0 < r < m$, observe the following bound:\n\n\\begin{align*}\nd^2(x_{m\\ell + r, x_*)} &\\leq 2d^2(x_{m \\ell + r - 1}, x_*) + 2 d^2(x_{m \\ell + r -1}, x_{m \\ell + r})\\\\\\\\\n&= 2d^2(x_{m \\ell + r - 1}, x_*) + 2 ||\\log_{x_{m\\ell + r - 1}} x_{m\\ell + r}||^2\\\\\\\\\n&= 2d^2(x_{m \\ell + r - 1}, x_*) +2 \\eta_{r-1}^2 ||\\nabla f(x_{m \\ell + r - 1})||^2\\\\\\\\\n&\\leq 2(1 + L^2 \\eta_{r-1}^2) d^2(x_{m \\ell + r - 1}, x_*).\n\\end{align*}\n\nHere, the first inequality is coming from triangular inequality, the second equality is coming from the definition of the  logarithmic map, and the last inequality comes from geodesic $L$-smoothness condition applied on $x_{m \\ell + r - 1}$ and $x_*$.\n\nInductively applying the above result, one gets\n$$d^2(x_{m\\ell + r}, x_*) \\leq 2 \\prod_{i=0}^{r-1} (1 + L^2 \\eta_{i}^2)d^2(x_{m \\ell}, x_*)$$\n\nNow, observe $\\prod_{i=0}^{r-1} (1 + L^2 \\eta_{i}^2) \\leq (1 + L^2(1 + \\rho^{\\lceil \\log_{\\rho} \\kappa \\rceil}))^{m}$, by the definition of the silver step-size, restarting scheme in Theorem 4.2, and $r < m$. This is just a constant with respect to $n := m\\ell + r$, since $m = 2^{\\lceil \\log_{\\rho} \\kappa \\rceil + 1} - 1$ is a fixed constant. And by Theorem 4.2 of our work, $d^2(x_{m \\ell}, x_*) \\leq \\exp(-Cm\\ell/\\kappa^{\\log_{\\rho} 2})d^2(x_0, x_*)$. Thus,\n$$d^2(x_{m\\ell + r}, x_*) \\leq C_{\\kappa} \\exp(-Cm\\ell/\\kappa^{\\log_{\\rho} 2})d^2(x_0, x_*).$$\nThis implies one still achieves the same $\\exp(-Cn/\\kappa^{\\log_{\\rho}2})$ convergence rate for any $m\\ell < n < m(\\ell + 1)$ for any $\\ell \\in \\mathbb{N}$, and therefore for general $n$. We will elaborate this in detail in the revision.\n\n2. On the other hand, as the reviewer mentioned, for generalized geodesically convex case the theory is only valid for $n = 2^k -1$. In the Euclidean setting, theoretical guarantees for general $n \\neq 2^k-1$ are derived from the analysis of the silver step-size under strong convexity [1]. However, interpretation of the co-coercivity condition used in [1] does not readily extend to the Riemannian setting, unlike the role played by Proposition 3.8. As a result, we cannot directly apply the strong convexity-based analysis to generalize the theory for arbitrary $n \\neq 2^k -1$ in Riemannian setup. That said, our numerical experiments show that the algorithm performs well even when $n \\neq 2^k - 1$. This suggests there is a possibility of a generalizing the proof for arbitrary $n$, though we do not currently have a complete answer. However, even in this case, the restriction $n = 2^k - 1$ is not severe, as the number of iteration is something one can choose freely, rather than some fixed parameter. \n\n> When working with the Wasserstein space, deterministic RGD usually cannot apply.\n\nWe agree that in Wasserstein space, a Monte Carlo approximation of the updates are often used rather than a closed form expression [8,9]. That said, we would like to make three remarks:\n\nFirst, in the literature of stochastic optimization, the analysis of deterministic algorithm is typically a prerequisite. Since a theoretical analysis of the deterministic silver step-size method on Riemannian manifolds was previously lacking, we considered this an essential first step. \n\nSecond, as the reviewer pointed out, in our main application, potential energy minimization, the deterministic form of the gradient is typically available, making our current formulation applicable.\n\nThird, the main challenge in the stochastic setting lies in controlling the variance. To do so, the common practice in Wasserstein gradient-based methods is to use Monte-Carlo approximation of the true gradient [8,9]. While omitted for clarity, our experiments show the algorithm remains effective with stochastic gradients given enough samples (e.g., 100 in our tests). If needed, we’re happy to include these results in the appendix for the camera-ready version. The interaction between large silver step-sizes and noisy/inexact gradients poses challenges for variance control (while maintaining the acceleration), even in Euclidean settings. We leave it to future work.\n\n> The assumption of generalized geodesic convexity is strong and less standard in general Riemannian manifolds. Maybe this is the reason why this definition has not been introduced in the context of Riemannian optimization in the literature.\n\nWe acknowledge that generalized geodesic convexity may indeed be a strong and potentially restrictive condition in general Riemannian manifolds. However, we would like to make two points:\n\nFirst, Example D.8 shows the concept of generalized geodesic convexity is not vacuous on general Riemannian manifolds beyond Wasserstein space. The Bures–Wasserstein (BW) space, although rooted in Wasserstein geometry, is a genuine Riemannian manifold in its own right. Moreover, the space of symmetric positive definite (SPD) matrices is a Riemannian manifold, without a direct connection to Wasserstein geometry. These examples illustrate that the notion of generalized geodesic convexity is not limited to Wasserstein space and may be broadly applicable across general Riemannian manifolds.\n\nSecond, the generalized geodesic convexity condition stems from the well-known Euclidean inequality (3.1), which has guided algorithm design in prior work [2,3,4]. In this regard, we thought extending this concept to Riemannian manifolds may offer a promising direction for developing analogous algorithms in the non-Euclidean setting.\n\n> The numerical experiments, while illustrative, are somewhat limited in scope, focusing only on the Bures--Wasserstein space and the sphere.\n\n*We also included the experiments on mean-field training of the two-layer neural network in Appendix E.2.3.*\n\n> The paper does not include comparisons with Nesterov-style Riemannian Gradient Descent, which would provide a valuable benchmark. Also, include theoretical convergence curve.\n\nOur primary objective in this work was to demonstrate acceleration relative to standard RGD, and for that reason, we thought including the comparison between Nesterov-type results might dilute the message. That said, if needed, we can include the comparison between Nesterov-type RGD, and the theoretical convergence curve in the final version.\n\n> The Wasserstein space is not a bona fide Riemannian manifold. I wondered if the theorems 4.1 and 4.2 still applied.\n\nYes. As discussed in the proof of Corollary 6.1, the same argument applies by directly substituting the corresponding quantities for the Wasserstein space. These substitutions are based on Otto calculus and do not require the Wasserstein space to be a Riemannian manifold in the strict sense.\n\n> The first experiment in the Wasserstein space: maybe also consider a more challenging form of $V$ instead of a quadratic function. \n\nWe used the quadratic potential energy since the target function $\\mathbb{E}[V(X)]$ admits the closed-form expression. Note that even solving the quadratic potential problem is nontrivial due to the underlying Riemannian structure. Instead, *to demonstrate the broader applicability of our method, we also included an additional experiment in mean-field neural network training in Wasserstein space in Appendix E.2.3.* If the reviewer has any specific suggestion for $V$, we are open to include additional experiments in the final version.\n\n> The second experiment on a sphere: why did not you choose another function that is generalized geodesically convex instead?\n\nThe main reason we chose this problem is because the Rayleigh quotient maximization problem is a widely recognized benchmark in the Riemannian optimization literature [5, 6]. In fact, even functions that just satisfy both geodesic convexity and geodesic smoothness are rare. For instance, on compact manifolds, globally geodesically convex functionals do not exist [7]. \n\n> Points regarding the notations.\n\nWe appreciate the reviewer for carefully noticing these typos. We will fix these typos when we prepare the camera-ready version.\n\nReferences\n\n[1] Altschuler et al. Acceleration by Stepsize Hedging I: Multi-Step Descent and the Silver Stepsize Schedule. ACM 2024.\n\n[2] Altschuler et al. Acceleration by stepsize hedging II: Silver stepsize schedule for smooth convex optimization. Mathematical Programming 2024.\n\n[3] Li et al. A simple uniformly optimal method without line search for convex optimization. ArXiv 2023.\n\n[4] Shugart et al. Negative Stepsizes Make Gradient-Descent-Ascent Converge. Arxiv 2025.\n\n[5] Alimisis et al. Momentum improves optimization on riemannian manifolds. AISTATS 2021.\n\n[6] Kim et al. Accelerated gradient methods for geodesically convex optimization: Tractable algorithms and convergence analysis. ICML 2022.\n\n[7] Jonathan W. Siegel. Accelerated Optimization with Orthogonality Constraints.\nJournal of Computational Mathematics, 2020.\n\n[8] Salim et al. The Wasserstein proximal gradient algorithm. NeurIPS 2020.\n\n[9] Bonet et al. Mirror and preconditioned gradient descent in wasserstein space. NeurIPS 2024."}, {"review_id": "Rd35Zv9qGE", "reviewer": "Reviewer_sSNV", "summary": "This paper analyzes the Riemannian gradient descent method under a varying silver stepsize schedule~\\cite{altschuler2024acceleration}, a strategy recently proposed and successfully applied in Euclidean spaces. The authors prove that, under a newly introduced notion of generalized geodesic convexity, the Riemannian gradient descent algorithm achieves an accelerated convergence rate on manifolds with non-negative curvature—consistent with the Euclidean case, i,e, $O(1/k^{\\alpha})$, $\\alpha \\approx 1.2716$.\n\nMoreover, by incorporating a restarting scheme, the authors extend the silver stepsize analysis to the setting of geodesically strongly convex functions, without requiring any modification to the stepsize itself.\n\nAs an application, the paper develops a theoretically accelerated gradient optimization method on Wasserstein spaces. Numerical experiments further demonstrate the effectiveness of the variable stepsize strategy on the Rayleigh quotient maximization problem over the sphere manifold.", "questions": "* Can the assumption of generalized geodesic convexity be relaxed to standard geodesic convexity?\n* What challenges arise in establishing convergence rate estimates when $n\\neq 2^{k}-1$?\n* Is it possible to achieve acceleration with silver step-size for geodesically $\\alpha$-strongly convex functions without employing a restarting strategy?", "limitations": "The present analysis is confined to manifolds with non-negative curvature and has not been extended to general manifolds. \nAdditionally, the method relies on the exponential and logarithmic maps, which may limit its applicability to some manifold-structured problems.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n* The paper introduces a new notion of generalized geodesic convexity and gives some examples.\n* The proposed Riemannian gradient descent method with silver stepsize is theoretically accelerated under non-negatively curved Riemannian manifolds.\n* The paper includes applications on the Wasserstein space and numerical experiments of Rayleigh quotient maximization on the sphere.\n\t\nWeaknesses:\n* The silver step-size schedule is not clearly explained.\n* Several key theoretical arguments, including Lemma 5.4 and its proof, are not clearly presented.\n* The study focuses on the exponential and logarithmic maps on the manifold, while omitting the consideration of the retraction map.", "was_revised": true, "final_justification": "I will keep my positive score, and my questions are all answered.", "rebuttal": "We are grateful to the reviewer for their detailed feedback, which we will take into account when preparing the final version, if accepted.\n\n> The silver step-size schedule is not clearly explained. \n\nSilver step-size is constructed recursively, starting from $n = 1$ where $\\eta_0 = \\rho - 1$ for $\\rho = 1+ \\sqrt{2}$. Then, given the sequence of the step-sizes at $n$ number of iterations, $\\eta_i$ for $i = 0, \\dots, n-1$, one constructs the new sequence of the length $2n + 1$ by \n\n\\begin{align*}\n    \\eta_i^{new} &=\\eta_i, \\qquad i = 0, \\dots, n-1,\\\\\\\\\n    \\eta_n^{new} &= 1 + \\rho^{k-1},\\\\\\\\\n    \\eta_i^{new} &= \\eta_{i-n-1}, \\qquad i = n+1, \\dots, 2n.\n\\end{align*}\n\nThen, one conducts the gradient descent by\n$x_{i+1} = \\exp_{x_i}(-\\eta_i \\nabla f(x_i))$\nWe will add more details about the silver step-size in the revised version.\n\n> Several key theoretical arguments, including Lemma 5.4 and its proof, are not clearly presented.\n\nWe will make our best effort to clarify the arguments in the revision. If the reviewer finds a specific part unclear, we would greatly appreciate it if the reviewer could point it out so that we can elaborate on it more thoroughly.\n\n> The study focuses on the exponential and logarithmic maps on the manifold, while omitting the consideration of the retraction map.\n\nIt is true our analysis is performed on exponential map and logarithmic map, and this is due to the use of metric distortion lemma which require the exact exponential map and logarithmic map. For the retraction map, one can consider the similar error analysis as in [1], while we did not pursue this generality.\n\n> Can the assumption of generalized geodesic convexity be relaxed to standard geodesic convexity?\n\nThe need of generalized geodesic convexity comes from deriving the key Equation (3.1). Therefore, avoiding the use of generalized geodesic convexity would depend on whether (a) one can derive Equation (3.1) without invoking the generalized geodesic convexity, or (b) one can prove the silver step-size acceleration without bringing Equation (3.1) type inequality. \n\nRegarding (a), there has been an attempt to derive a similar inequality under standard geodesic convexity and $L_0$-smoothness (Lemma 2.3. of [2]). However, this approach does not recover the exact inequality we need: First, a constant $L$ is of the form $L_0d^2(x_i,x_j)$, which depends on the specific points involved. Second, the structure of the subtractive term differs from that in Equation (3.1). \n\nRegarding (b), the current silver step-size analysis relies critically on the precise form of Equation (3.1) even in the Euclidean case. So generalized geodesic convexity appears necessary at this moment. \n\nMoreover, even in the Euclidean case, the proof of Equation (3.1) implicitly uses the notion of generalized geodesic convexity (which coincides with standard convexity in the Euclidean setting) by introducing an auxiliary third point $z$. In this regard, we believe that deriving (3.1) without invoking generalized geodesic convexity, if possible, would require fundamentally different techniques from those used in existing proofs.\n\n> What challenges arise in establishing convergence rate estimates when $n \\neq 2^k - 1$? Is it possible to achieve acceleration with silver step-size for geodesically strongly convex functions without employing a restarting strategy?\n\n1. *Under additionally geodesically strongly convex functional, one can establish the same convergence rate with respect to $n$ for general $n \\neq \\ell (2^{k^{\\ast}} - 1)$.* One can proceed as follows: Consider $m = 2^{k^{\\ast}} -  1$ number of inner-iterations and $\\ell$ number of outer iteration, as in our restarting algorithm. Now, for general $ 0 < r < m$, observe the following bound:\n\n\\begin{align*}\nd^2(x_{m\\ell + r, x_*)} &\\leq 2d^2(x_{m \\ell + r - 1}, x_*) + 2 d^2(x_{m \\ell + r -1}, x_{m \\ell + r})\\\\\\\\\n&= 2d^2(x_{m \\ell + r - 1}, x_*) + 2 \\|\\log_{x_{m\\ell + r - 1}} x_{m\\ell + r}\\|^2\\\\\\\\\n&= 2d^2(x_{m \\ell + r - 1}, x_*) +2 \\eta_{r-1}^2 \\|\\nabla f(x_{m \\ell + r - 1})\\|^2\\\\\\\\\n&\\leq 2(1 + L^2 \\eta_{r-1}^2) d^2(x_{m \\ell + r - 1}, x_*).\n\\end{align*}\n\nHere, the first inequality is coming from triangular inequality, the second equality is coming from the definition of the  logarithmic map, and the last inequality comes from geodesic $L$-smoothness condition applied on $x_{m \\ell + r - 1}$ and $x_*$.\n\nInductively applying the above result, one gets\n$$d^2(x_{m\\ell + r}, x_*) \\leq 2 \\prod_{i=0}^{r-1} (1 + L^2 \\eta_{i}^2)d^2(x_{m \\ell}, x_*)$$\n\nNow, observe $\\prod_{i=0}^{r-1} (1 + L^2 \\eta_{i}^2) \\leq (1 + L^2(1 + \\rho^{\\lceil \\log_{\\rho} \\kappa \\rceil}))^{m}$, by the definition of the silver step-size, restarting scheme in Theorem 4.2, and $r < m$. This is just a constant with respect to $n := m\\ell + r$, since $m = 2^{\\lceil \\log_{\\rho} \\kappa \\rceil + 1} - 1$ is a fixed constant. And by Theorem 4.2 of our work, $d^2(x_{m \\ell}, x_*) \\leq \\exp(-Cm\\ell/\\kappa^{\\log_{\\rho} 2})d^2(x_0, x_*)$. Thus,\n$$d^2(x_{m\\ell + r}, x_*) \\leq C_{\\kappa} \\exp(-Cm\\ell/\\kappa^{\\log_{\\rho} 2})d^2(x_0, x_*).$$\nThis implies one still achieves the same $\\exp(-Cn/\\kappa^{\\log_{\\rho}2})$ convergence rate for any $m\\ell < n < m(\\ell + 1)$ for any $\\ell \\in \\mathbb{N}$, and therefore for general $n$. We will elaborate this in detail in the revision.\n\n2. On the other hand, for generalized geodesically convex case, we are currently unaware how to avoid such constraint. In the Euclidean setting, theoretical guarantees for general $n \\neq 2^k-1$ are derived from the analysis of the silver step-size under strong convexity [3] without the use of restarting method. However, interpretation of the co-coercivity condition used in [3] does not readily extend to the Riemannian setting, unlike the role played by Proposition 3.8. As a result, we cannot directly apply the strong convexity-based analysis to generalize the theory for arbitrary $n \\neq 2^k -1$ in Riemannian setup. That said, our numerical experiments show that the algorithm performs well even when $n \\neq 2^k - 1$. This suggests that a generalization of the proof may be possible, though we do not currently have a complete answer. However, even in this case, we do not view the restriction $n = 2^k - 1$ as a critical drawback, as the number of iterations can be chosen freely in practice.\n\n> Paper Formatting Concerns\n\nWe appreciate the reviewer for carefully reviewing the paper and catching the typos, notation errors, and grammar errors. If accepted, we will fix these issues when we prepare the camera-ready version. \n\n> I believe the conclusion of Lemma D.6 is correct but the proof of the zeroth-order criterion is not sufficiently rigorous.\n\nWe appreciate the reviewer for noticing the missing detail. Your argument is correct, and the result holds due to the additional equality condition at $t = 0$, which was inadvertently omitted. Precisely, let $\\phi(t) = (1-t) f(x) + tf(y) - (f \\circ \\gamma)(t)$. Then, $\\phi(t) \\geq 0$ and $\\phi(0) = 0$ is the global minimizer. Therefore,\n\n$$\n0 \\leq \\phi'(0^{+}) = \\lim_{t \\to 0+}\\frac{\\phi(t)}{t} = f(y) - f(x) - \\lim_{t \\to 0+}\\frac{(f \\circ \\gamma)(t) - f(x)}{t} = f(y) - f(x) - \\frac{d}{dt}\\bigg\\vert_{t=0}(f \\circ \\gamma)(t).\n$$\n\nFrom here, one can proceed the same as in Lemma D.6. We will revise the statement to clarify this point if the paper is accepted.\n\nReferences\n\n[1] Han et al. Riemannian Accelerated Gradient Methods via Extrapolation. AISTATS 2023. \n\n[2] Ansari-Önnestam et al. Adaptive Gradient Descent on Riemannian Manifolds with Nonnegative Curvature. Arxiv 2025.\n\n[3] Altschuler et al. Acceleration by Stepsize Hedging I: Multi-Step Descent and the Silver Stepsize Schedule. ACM, 2024."}, {"review_id": "R3Rtn4RCyl", "reviewer": "Reviewer_zbh5", "summary": "This paper studies the silver step-sizes for Riemannian manifold optimization. Silver step-sizes are a recently proposed technique that can accelerate gradient descent without explicitly incorporating momentum terms. The authors extend this idea to the Riemannian setting and show that silver step-sizes can lead to improved worst-case complexity bounds for optimization on manifolds. The paper also discusses two applications of the proposed method: optimization on the 2-Wasserstein space and Rayleigh quotient maximization.", "questions": "The silver step-size was originally inspired by recent advances in the Performance Estimation Problem (PEP). In this paper, it is not clear whether PEP plays any role in the analysis or proof techniques. Could the authors clarify whether any part of the theoretical results, was informed or assisted by PEP? Is it possible to apply the PEP to the Riemannian setting considered in this paper, in order to derive a lower bound result?", "limitations": "Yes. This paper raises some open questions.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This paper addresses an important and fundamental problem: how to accelerate gradient descent methods on Riemannian manifolds. The authors propose using the recently introduced silver step-size to achieve acceleration in this setting. Both Riemannian manifold optimization and gradient-based methods are important areas of research.\n\nIn terms of presentation, I appreciate the clarity with which the paper defines the necessary preliminaries, as well as the additional preliminaries included in the appendix. The exposition is generally clear and accessible. However, as someone not deeply familiar with Riemannian optimization, I am not in a good position to verify the technical correctness of the mathematical arguments. \n\nAssuming the theoretical results are correct, I believe the contribution is significant. While the proposed step-size rule is directly borrowed from the Euclidean silver step-size literature, extending it to the Riemannian setting looks nontrivial. This generalization is meaningful, especially given that optimization on manifolds is less mature. From this perspective, the results presented in the paper are valuable.\n\nOne limitation is that, unlike in the Euclidean setting, the paper does not provide a corresponding lower bound to demonstrate the optimality of the obtained complexity result. Although this is unfortunate, it is understandable given the increased technical complexity of the manifold setting.\n\nAnother strength of the paper is the inclusion of two applications, in which the proposed method demonstrates improved empirical performance. Given that the practical value of the silver step-size in Euclidean settings is still valuable than its theoretical value, these positive experimental results are encouraging.", "was_revised": true, "final_justification": "The response is clear and helpful. The strengths and weaknesses of the paper are still clear and have been described. I maintain the previous assessment.", "rebuttal": "We are grateful to the reviewer for their detailed feedback, which we will take into account when preparing the final version, if accepted.\n\n> Unlike in the Euclidean setting, the paper does not provide a corresponding lower bound to demonstrate the optimality of the obtained complexity result. Is it possible to apply the PEP to the Riemannian setting considered in this paper, in order to derive a lower bound result?\n\nFirst, to the best of our knowledge, the optimality of silver step-size is not verified even in Euclidean space yet [1] (although it is conjectured to be the optimal [2]).\n\nSecond, the worst-case analysis in the Riemannian setting must account for both the function and the geometry of the manifold, unlike in the Euclidean case, where the analysis is function-centric. In particular, certain functions, such as the squared-distance function, are intrinsically tied to the geometric structure of the manifold [3]. These complexities make lower bound analysis substantially more challenging in the Riemannian context, even when using tools such as PEP. In particular, to use PEP, the key ingredient is the interpolating inequality. However, to the best of our knowledge, the only Riemannian version of the interpolating inequality we are aware of is Proposition 38 of [4], which requires to verify whether the condition holds for all $x \\in M$ instead of $x = x_j$ for $j = 1, \\dots, n$. Since such condition is hard to implement numerically, applying PEP to Riemannian setting does not seem direct at this stage (see also Introduction of [5]). We leave this as an open direction for future work.\n\n> Could the authors clarify whether any part of the theoretical results, was informed or assisted by PEP?\n\nIn this work, we did not directly utilize the PEP, as we were already aware of the target inequality (Lemma 5.2) from the Euclidean studies. However, certainly our analysis is based on the original silver step-size paper [1] which was heavily informed by PEP. \n\nReferences\n\n[1] Altschuler et al. Acceleration by stepsize hedging II: Silver stepsize schedule for smooth convex optimization. Mathematical Programming, 2024.\n\n[2] Grimmer et al. Composing Optimized Stepsize Schedules for Gradient Descent. Arxiv 2024.\n\n[3] Alimisis et al. A Continuous-time Perspective for Modeling Acceleration in Riemannian Optimization. AISTATS 2020.\n\n[4] Criscitiello et al. Curvature and complexity: Better lower bounds for geodesically convex optimization. COLT 2023.\n\n[5] Hu et al. Extragradient Type Methods for Riemannian Variational Inequality Problems. AISTATS 2024."}]

Author rebuttal:

Meta-review: The paper introduces a "silver step‑size" schedule for Riemannian gradient descent and shows that, on non‑negatively curved manifolds satisfying the newly defined generalized geodesic convexity and smoothness assumptions, this simple step‑size rule yields accelerated in the convex case and an improved condition‑number dependence in the strongly‑convex case. The technical contribution rests on "Proposition 3.8" (which has central to the discussion between authors and reviewers also), which provides a geometric interpretation that enables adapting the Euclidean silver‑step analysis to the curved setting although no closed‑form expressions for non‑consecutive iterates are available. The authors further demonstrate that their method provides the first provable accelerated algorithm various settings; concerns are limited to (i) the restriction to non‑negative curvature, (ii) the strength and unfamiliarity of the generalized geodesic convexity condition, (iii) modest experimental breadth, and (iv) lack of a lower‑bound optimality proof.  However it seems to be consensus that that these issues do not outweigh the solid theoretical advances and overall merit of the paper. I concur and recommend acceptance.
Decision: Accept (poster)