A general framework of Riemannian adaptive optimization methods with a convergence analysis
Authors: Hiroyuki Sakai
Hideaki Iiduka
Venue: Accepted by TMLR ()

Abstract: This paper proposes a general framework of Riemannian adaptive optimization methods. The framework encapsulates several stochastic optimization algorithms on Riemannian manifolds and incorporates the mini-batch strategy that is often used in deep learning. Within this framework, we also propose AMSGrad on embedded submanifolds of Euclidean space. Moreover, we give convergence analyses valid for both a constant and a diminishing step size. Our analyses also reveal the relationship between the convergence rate and mini-batch size. In numerical experiments, we applied the proposed algorithm to principal component analysis and the low-rank matrix completion problem, which can be considered to be Riemannian optimization problems. Python implementations of the methods used in the numerical experiments are available at https://github.com/iiduka-researches/202408-adaptive.

Published in Transactions on Machine Learning Research (01/2025)
A general framework of Riemannian adaptive optimization methods with a convergence analysis
Hiroyuki Sakai
sakai08155ccs http://.....mmtt........
Meiji University
Hideaki liduka Meiji University
idduka &ccs. mww.ssmmiiiia... ac.jp
Reviewed on OpenReview: https: /ttp://ppppp openreview. net/ formm?? id= knug LQFVoE
Abstract
This paper proposes general framework of Riemannian adaptive optimization methods. The framework encapsulates several stochastic optimization algorithms on Riemannian man- ifolds and incorporates the mini-batch strategy that is often used in deep learning, Within this framework, we also propose AMSGrad on embedded submanifolds of Euclidean space. Moreover, we give convergence analyses valid for both a constant and a diminishing step size. Our analyscs also reveal the relationship between the convergence rate and mini-batch sizc. In numerical experiments, we applied the proposed algorithm to principal component analy- sis and the low-rank matrix completion problem, which can be considered to be Riemannian optimization problems. Python implementations of the methods used in the numerical ex- periments available at httts::/ wwwpttthhu..cmmmiiiitaaeeeeeeeee com/iiduka researches/222408 http:eees//2220000aaattteeee
1 Introduction
Riemannian optimization (Absil et al. 2008 Sato 2021 has received much attention in machine learning.
For example, batch normalization (Cho & Lee 2017). representation lcarning Nickel & Kicla 2017). and the
low-rank matrix completion problem @Vandereycken 2013 Cambier & Absil 2016 Bounnl & Absil 2015)
can be considered optimization problems on Riemannian manifolds. This paper focuses on Riemannian adaptive optimization algorithms for solving stochastic optimization problems on Ricmannian manifolds. particular, we treat Riemannian submanifolds of Euclidean space (e.g., unit spheres and the Stiefel manifold).
In Euclidean settings, adaptive optimization methods are widely used for training deep neural networks. There are many adaptive optimization methods, such Adaptive gradient (AdaGrad) (Duchi et al. 2011). Adadelta [Zeiler 2012), Root mean square propagation (RMSProp) (Hinton et al. 2012). Adaptive mo- ment estimation (Adam) (Kingma & Ba 2015), Yogi (Zaheer et al. 2018), Adaptive mean square gradient (AMSGrad) (Reddi et al. 20188) AdaFom (Chen et al. 2201)), AdaBound (Luo et al.] 2019 Adam with decoupled weight decay (AdamW) (Loshchilov Hutter 2019 and AdaBelicf (Zhuang et al. 2020). Reddi
et al. (2018) proposed a general framework of adaptive optimization methods that encapsulates many of the
popular adaptive methods in Euclidean space.
Bonnabel 2013) proposed Riemannian stochastic gradient descent (RSGD), the most basic Riemannian stochastic optimization algorithm. In particular, Riemannian stochastic variance reduction algorithms, such as Riemannian stochastic varianco-reduced gradient (RSVRG) (Zhang et al.) 2016), Riemannian stochastic recursive gradient (RSRG) (Kasai et. al.) 2018), and Riemannian stochastic path-integrated differential es- timator (R-SPIDER) (Zhng et al. 20188 Zhou et al., 20099), are based on variance reduction methods in Euclidean space. There are several prior studies on Riemannian adaptive optimization methods for spe- cific Riemannian manifolds. In particular, Kasai al. (2019) proposed a Riemannian adaptive stochastic gradient am matrix manifolds (RASA). RASA is adaptive optimization method matrix

---

Published in Transactions on Machine Learning Research (01/2025)
manifolds (e.g., the Stiefel manifold or the Grassmann manifold), and gave convergence analysis under the upper-Hessian bounded and retraction T-smooth assumptions (see (Kasai et al., 2019 Section 4) for details). However, RASA is not a direct extension of the adaptive optimization methods commonly used in deep lcarning, and it works only for diminishing step sizes. RAMSGrad (Bécigneul & Ganca 2019) and modified RAMSGrad (Sakai & Tiduka 2021), direct extensions of AMSGrad, have been proposed as meth- ods that work on Cartesian products of Riemannian manifolds. In particular, Roy et al. (2018) proposod cRAMSProp and applied it to several Riemannian stochastic optimizations. However, they did not provide convergence analysis of it. More recently, Riemannian stochastic optimization methods, sharpness-awann minimization on Riemannian manifolds (Riemannian SAM) Yunn & Yang 2024) and Riemannian natural gradient descent (RNGD) (Hu et al. 2024), were proposed.
1.1 Motivations
Kasai et al. (019) presented useful results indicating RASA with a diminishing step size converges to a stationary point of contimuously differentiable function defined on matrix manifold. Meanwhile, other results have indicated that using constant step sizes makes the algorithm useful for training deep neural networks (Zaheer et al. 2018) and that the setting of the mini-batch size depends on the performance of the algorithms used to train deep neural networks (Smith et al. 2018 Sato Liduka, 2023). The main motivation of this paper is thus to gain an understanding of the theoretical performance relationship wet Ricmannian adaptive optimization methods using constant step sizes and ones using the mini-batch size. We are also interested in understanding the theoretical performance relationship between the Riemannian adaptive optimization methods using diminishing step sizes and those using the muini-btchh size.
1.2 Contributions
Our first contribution is to propose a framework of adaptive optimization methods on Riemannian subman- ifolds of Euclidean space (Algorithm D that is based on the framework (Reddi et al. 2018 Algorithm 1) proposed by Reddi, Kale and Kumar for Euclidean space. A key challenge in designing adaptive gradient methods over Riemannian manifolds is the absence of a coordinate system like that in Euclidean space. In this work, we specifically focus on manifolds that are embodded in Euclidean space, common setting in many applications. We address the issue by constructing algorithms that rely on projections onto the tangent space at each point on the manifold. Our framework incorporates the mini-batch strategy that is often used in deep learning. Important examples of Riemannian submanifolds of the Euclidean space include the unit sphere and the Sticfel manifold. Note that (Kasai et al.) 2019) focuses exclusively on the RASA algorithm, which is defined only on matrix manifolds. In contrast, our framework can be defined on any manifold embedded in Euclidean space. Moreover, within this framework, we propose AMSGrad on embedded submanifolds of Euclidean space (Algorithm 2 as a diroct extension of AMSGrad.
Our second contribution to give convergence analyses of the proposed algorithm CAleoriiAAAliiitt (Algorithm 1 by using mini-batee size 6 valid for both constant step size (Theorem 3.5) and diminishing step size (Theorem 3.6 In particular, Theorem 3.5 indicates that, under some conditions, the sequence (Ix) generated by the proposed algorithm with a constant step size a satisfies
K Σ
[[erad [(xx)12 =o (o (+
[lgrad f(xx)
min k=1....KK
We should note that Theorem 3.5 is an extension of the result in Zaheer et al. (0188 Corollary 2). Theorem 3.6 indicates that, under some conditions, the sequence (xx) generated by the proposed algorithm with diminishing step size O% satisfies
K K
((1+)) log K VK
min K Igrad f(xx))1
<
Igadd f(xx)12]
-0
Since the stochastic gradient (see 11) using large mini-batch sizes approximately the full gradient of f: intuitively, using large mini-batch sizes decreases the number of steps needed to find stationary points of
2

---

Published in Transactions on Machine Learning Research (01/2025)
f. Theorems 3.5 and 3.6 indicate that, the larger the mini-batch size is, the smaller the upper bound of 1
K Ek= B Bllggadd (arx))
becomes and the fewer the required steps become. Hence, Theorems 3.5 and 3.6
would match our intuition.
Theorem 3.5 indicates that, although the proposed algorithm with a constant step size a and constant mini- batch size b does not always converge, we can expect that converges with an increasing mini-batch size by such that by S bK+1- In practice, it has been shown that increasing the mini-batch size (Byrd et al., 2012; Balles et al., 2017; De et al. 2017 Smith et al.) 2018 Goyal et al. 2018) is useful for training deep neural networks with mini- optimizers. Motivated by Theorems 3.5 and 3.6 give convergence analyses of the proposed algorithm by using an increasing mini-batch size by valid for both a constant step size (Theorem 3.7) and diminishing step size (Theorem 3.8). Theorem 3.7 indicates that, under some conditions, the sequence (Zz) generated by the proposed algorithm with a constant step size a and an increasing mini-batch size by satisfies
[Igrad (orrad((x))1) (1x))11
1 K Igrad ( O
min
S K
That is, with increasing mini-batch sizes, it can find a stationary point of f, in contrast Theorem 3.5 Moreover, Theorem 3.8 indicates that, under some conditions, the sequence (2x))== generated by the pro- posed algorithm with diminishing step size og and an increasing mini-batch size 0y satisfies
K
VK (v*))
min
E [Igrad (112)
E
[Igrad ((x)) O
Ck=1 Of
That is, with increasing mini-batch sizes, it has better convergence rate than with constant mini-batch sizes (Theorem 3.6.
The third contribution is to numerically compare the performances of several methods based on Algorithm 1 including Algorithm 2 with the existing methods. In the numerical experiments, we applied the algorithms Lo principal component analysis (PCA) (Ksaai et al., | 2018: Roy et al. 2018 and the low-rank matrix completion (LRMC) problem [Boumal& & Absil 2015 Kasai et al. 2019 Hu et al., 2024), which can be considered to be Riemannian optimization problems. Based on our experiments, we can summarize the following key findings. For the PCA problem, RAMSGrad with constant step size minimizes the objective function in fewer comparable iterations compared with RSGD, RASA-L, RASA-R and RASA-LR, regardless of the dataset. For the LRMC problem, RAdam with dininishing step size minimizes the objective function in fewer or about the same number of iterations compared with RSGD, RASA-L and RASA-R, irrespective of the datasot. As reported in Sakai & Tiduka (2021), RAMSGrad demonstrates robust performance regardless of the initial step size. For the PCA and LRMC problems (where LRMC adopts a diminishing step size), a grid search was conducted, allowing the selection of an initial step size that yielded even better results. This likely contributed to the strong performance of R.MSGrad across datasets. However, for the LRMC problem, RAMSGrad may underperform relative to RASA, depending on the dataset. It is worth noting that the objective function in the LRMC problem is complex, and assumptions such as retraction L-smooth may not hold in this case. This could explain the less favorable results observed for RAMSGrad in this particular setting. Moreover, to support our theoretical analyses, we show that increasing the batch size decreases the number of steps needed to a certain index. In particular, Table (resp. Table 2) supports the result shown in Theorem 3.5 demonstrating that the proposed algorithms with a constant (resp. diminishing) step size require fewer iterations to reduce the gradient norm below the threshold as the batch size increases. Tables [] and confirm that larger batch sizes lead to faster convergence in terms of the number of iterations, aligning with our theoretical analysis. Similar tables (Tables II 16) for the remaining datasets and experiments can be found in Appendix E. Note that the choice of batch size plays a critical rolc in balancing the computational cost per iteration and the total number of iterat ions required for convergence. This is related to the concept of the critical batch size, as discussed in [akai & liduka (2024), which minimizes a stochastic first-order oracle (SFO). However, estimating this optimal batch size is challenging due to the dependency on unknown parameters such as the gradient Lipschitz constant and the variance of the stochastic gradient. To address this challenge, an effective approach may involve dynamically increasing the batch size at regular intervals
3

---

Published in Transactions on Machine Learning Research (01/2025)
during optimization, such doubling or tripling it after fixed number of iterations. This method has theoretical guarantees for convergence (see Theorems 3.7 and 3.8) and can adaptively approach an efficient balance between computational cost and convergence speed.
2 Mathematical preliminaries
Let R2 be a d-dimensional Euclidean space with inner product (x, y)2 == =x" y, which inducos the norm I1-l12- Let R++ be the set of positive real numbers, Le., R4+ == {ar E R x > 03. La denotes a x d identity matrix. For square matrices X, Y RRdxd, we write X Y (resp. X positive-semidefiitte) matrix. For two matrices X and of the same dimension, XOY denotes the Hadamard product, i.c., clement-wise product. Let max(X, Y) be the dlement-wise maximum. Let S (resp. S4, + be the set of d x d symmetric (resp. symmetric posilive-semideffnite, symmetric positive-defnite) matrices, := {XX Rdxd X = O} and S == [X be the set of d x d diagonal matrices. Let Oa be the orthogonal group, i.e, O2 == {X E Rdxd X = la). We denote the trace of square matrix A by tr(A). Let O be Landau's symbol; i.e. for a sequence (2x) (#x)R=o- we write F& = (g(()) as k oc there exist a constant > 0 and an index kn such that |.xl S C (l) for all k ko-
3 v) if X is positive-defintt (resp.
i.., S" : {X E pdxd| X = =}, s
{X E iddd X > O}. Let D
In this paper, focus Riemannian manifolds (Sakai 1996) embedded in Euclidean space. Let M be embedded submanifold of Rd. Morcover, let T.M be the tangent space at point x E M and TM be the tangent bundle of M. Let O, be the zero element of T.M. The inner product (5-)2 of a Euclidean space Rd induces a Riemannian metric (s )) of M at x E M according to (E. n) = (E, 1)2 = E for &, E M C T_R²= = Rd The norm of 1 E T.M is defined as ||7/, -nnT Il Let P. T.R. = Rd - T.M be the orthogonal projection onto T3 M (see) Absil et al. (2008)). For smooth map F M - N between two manifolds M and N, DF(x): ; T + T denotes the derivative of F at E M. The Riemannian gradient grad fx) of a smooth function f M + R at x E M is defined as a unique tangent vector at x satisfying (grad fx)) = (n())] for any 7 E T.M.
Definition 2.1 (Retraction). Let M be a manifold. Any smooth map R TM + M is called relraction on M if has the following properties.
(0,) x for all x E M;
With the canonical identificatio To.T,M = T,M, (O.) i : T.M - T.M for all x E M,
where R2 denotes the restriction of R T.M.
2.1 Examples
The unit sphere gd-1 := fx E Rd ||-||| 1} is an embedded manifold of Ted The tangent space T55*-1 at x E gd-1 is given by T,S"-1 = {n E R 17 TTx x = 0}. The induced Riemannian metric on gd-1 is given by (E. = (5,1)) = ETn n for E, T The orthogonal projection P, : Rd T, onto the langent space T,s"-1 is given by (n) = (La xx ") for E sd-1 and 17 T,5d-1
An important example is the Stiefel manifold (Absil et al. 2008 Chapter 3.3.2), which is defined as St(p, : (X E TRxp x = 1p for n 2 p. (t,p.n) is an embedded manifold of Raxdd The langent space Tz St(p. at X E St(p,n) n) is given by
Tx St(p,n) = (n R
X X 1 O}.
The induced Riemannian metric on nt(p,n) is given by (5.n)x tr(E m) for E, Tx St(p, The orthogonal projection onto the tangent space Tx St(p,n) is given by Px = X symm(X 1) for X e s),,,n, v E Tx St(p,n), where sym(A) = (A + A 2)/2 The Stiefel manifold St(p, n) reduces to the orthogonal groups when n = p, i... St(p.p) = Op.
Morcover, we will also consider the Grassmann manifold (Asii et al. 2008 Stip, Stpp,))/O n)/Op. Let X (t(p, n) be a representative of [X] [XQ Q E Op)
Chapter 3.4.4) Gr(p, n) == n)) We denote the
4

---

Published in Transactions on Machine Learning Research (01/2025)
horizontal lift of 7 Tix] Gr(p, n) at X by 7x Tx n). The Riemannian metric of the Grassmann manifold Gr(p, n) endowed with (E, x] = (Ex, fxx22 for E, E Txx Gr(p, The orthogonal projection onlo the langent space Tix] Gr(p, n) is defined through
P(n(7)= = (In - XX MX,
for [X] E Gr(p, n) and E Tixj Gr(p, m). Note that while the Grassmann manifold itself is not an embedded submanifold, in practical implementations, computations involving it is often performed via the Stiefel manifold. This allows the proposed method to be applied. Such an approach was used in previous studies, including Kasai et al. (2019): Hu al. (2024).
2.2 Riemannian stochastic optimization problem
We focus on minimizing an objective function f .M - of the form,
N f(x) = N f1(x),
where f; is a smooth function for i I. We use the mini-batch strategy as follows (see liduka 20244 for detail). Sk.i is a random variable generated from the i-th sampling at the k-th iteration, and 8k = (sk,1...., 5k,b,) independent of (I=) where by (5 N) is the batch size. To simplify the notation, we denote the expoctation Es, with respect to 83 by Ex. From the independence of 81, $2, ..... ... 8k, we can define the total expectation E by E,E2-.. ... Eg. We define the mini-batch stochastic gradient grad fB, (#x) of f at the k-th iteration by
by Σ grad Sac, (zK)-
grad Sm, (mx) = by:
(1)
Our main objective is to find stationary point v, E M satisfying grad f f(x)) = Os,,-
2.3 Proposed general framework of Riemannian adaptive methods
Reddi et al. 2018) provided a general framework of adaptive gradient methods in Euclidean space. We devised Algorithm by generalizing that framework to an embedded manifold of Rd. The main difference from the Euclidean selting is computing the projection of H my onto the langent space Tz. M by the orthogonal projection Paz. Algorithm [ requires sequences of maps, (øw) and (vx))= such that du T,, M x ... x Ta, M -> Rd and 4/x Tr, M x ... Tı, M Dd S + respectively. Note that Algorithm is still abstract because the maps (ox) and (wx))e are not specified. Algorithm! [ is the extension of general framework in Euclidean space proposed by Reddi, Kale and Kumar (Reddi et al. 2018). In the Euclidean setting (i.e., M = Rd), the orthogonal projection Pz. yields an identity map and this corresponds to the Euclidean version the general framework.
By setting the maps (0a) and (1n)) Algorithm 1 to those used by an adaptive optimization method in Euelidean space, any adaptive optimization method can be extended to Riemannian manifolds. In the following, we show some examples of extending the optimization algorithm in Euclidean space to Riemannian manifolds by using Algorithm
Here, SGD is the most basic method; it uses
de gk.
. la-
Algorithm 1 with these maps corresponds to RSGD (Bonnabel 2013) in the Riemannian setting. AdaGrad Duuhi al. 2011), the first adaptive gradient method in Euclidean space that propelled research adaptive methods, uses the sequences of maps ( . = yh and
Up 1k 1 Ih ON,
(r . ) diag( VUk.I...., vkkd) + ela.
5

---

Published in Transactions on Machine Learning Research (01/2025)
Algorithm 1 The general framework of Riemannian adaptive oplimization methods an embedded sub-
manifold of Rd
Require: Initial point 21 E M, retraction R TM -> M, step sizes (ax) (ox) C + sequences of maps (du))=1 (1b))==1
Ensure: Sequence (rx) M. k +.
loop
gk grad [Ba (I+).
4:
my = 0x( 9kk) E Rd
5:
H2 0x(91...., 9k) E D n S
6:
Fk+1 = Ros (- P.,. (H5-ms))
7:
k + 11
8: end loop
where 00 = 0 E R2 and 0. Here, we will denote the i-th component of uk by Uh The exponential moving average variant of AdaGrad often used in doep-learning training. The basic variant is RMSProp (Hinton et al. 2012), which uses the sequences of maps Φk (91....,99)) = Is and
= 8- 1 + (1 82)9n. + 9k.
the (g), , Ik) = diag( /k.ll...., (uk,a) + eld,
where 60 E Rd and e > O. Both Algorithm 1 with these maps and cRMSProp (Roy et al. 2018 can be considered extensions of RMSProp to Riemannian manifolds. They differ from each other in that parallel transport is needed to compute the search direction of cRMSProp, but it is not needed in method.
Adamn Kingma & Ba 2015 is one of the most common variants; it uses the sequence of maps,
m& B1 81m=-1 + (1 B1)gk, Φ6(gi, ...0....:00 9kk
mg
(2)
1 3/+11
and
Uk B2 + B2)ga gki Ūx 1 - $2 k
44 9hh) = diag(( ii........ Vūna)) d) + cld.
(3)
where my = E Id and vo = E Rd. 31 = 0.9, B2 0.999 and = 10-8 are typically recommended values. Moreover, within the general framework (Algorithm 1 we propose the following algorithm as an extension of AMSGrad (Reddi et al. 2018) in Euclidean space.
3 Convergence analysis
3.1
Assumptions and useful lemmas
We make the following Assumptions 3.1 assumees the boundedness of the gradient. (A4)
A1
A4)
(A and (A2)
include the standard conditions. (A3)
is assumption on the Lipschitz continuity of the gradienl
A5 assumes that lower bound exists.
A Assumption 3.1. Let ( be a sequence generated by Algorithem
(A1) Exlgrad (xx)) grad (xxx for all and = 1,... .....,bu. br.
(A2) There erists o² > such that
Ex [|arad ser. (xxx) grad (xxX 000
for all k and i 1, 1,..., by-

---

Published in Transactions on Machine Learning Research (01/2025)
Algorithm AMSGrad on an embedded submanifold of TR
Require: Initial point 21 E M, retraction R TM + M, step sizes (au )== R mini-batch sizes (bx)k=1 C R++ hyperparameters B1, E [0, 1), e 0.
Ensure: Sequence (= C M.
1: Set mig = O, 2o = 0 and 80
k ++11. 1.
loop
gk grad fB2 (#k)-
mh 81mk-1 1 + (1 B1 )mk.
6:
= B2 + (1 32)gk @ gk.
7:
6 max( (ox((xx111 (,xx(0-1, Ux).
8:
H₂ = diag( v 0b.,1...,, v (k,d) + cld.
1k+1 = Ris - arp P.z (H]
9:
10:
k + k 1.
11: end loop
(A3) There erists G, B 0 such that llgrad f(xx) S G and ||grad f', (rx)ll2 S B for all k: 2 1.
(A4) There erists constant I. > 0 such that
[f o Ra)(n)(n) Df(()) & & 1mlle
for all x E M, T.M.
(A5) is bounded below by E R.
Assumptions A1 and (22) are satisfied when the probability distribution is uniform (sce Appendix F
when the full gradient is used, i.e., when the batch size equals the total number of data. [A1] and (A2 have
been used to analyze adaptive methods in Euclidean space (Zaheer et al. 2018 Pages 3, 12, and 16) and on
the Hadamard manifold (Sakai & liduka 2024 (2.2) and (2.3)). Assumptions (A3)
(33)(A5)
[A5) are often used in
both Euclidean spaces and Riemannian manifolds. Intuitively, in practical applications, these assumpptions
are satisfied because the search space of the algorithm is bounded. Tf ʃ is unbounded on an unbounded search space X, then [A5) does not hold. Although there is a possibility such that [A4) is satisfied, [A2) would
not hold, even when the probability distribution is uniform. When of is unbounded, there is minimizer of ʃ. Hence, [A5 is essential in order to analyse opt imizaatoioon methods. Moreover, [A4) is needed in order to
use a retraction L-smooth (Proposition 3.2) that is an extension of the Euclidean-type descent lemma. [A3]
holds when the manifold is compact (Absil et al. 2008 or through a slight modification of the objective function and the algorithm (Kasai et al. 2018
It known that if Assumption 3.1
A4 holds, so does the following Proposition 3.2 This property is known retraction L-smoothness (see Huang et al. (2015): Kasai et al. (2018) for details).
Proposition 3.2. Suppose that Assumption 3.1 (44) holds. Then,
L 2 |lnle
f(Ra(n) S f(x (grad f(x),
for all M and 1j E T.M.
3.2 Convergence analysis of Algorithm
The main difficulty in analyzing the convergence of adaptive gradient methods is due to the stochastic momentum my = 01(91,..... gk). As a way to overcome this challenge in Euclidean space, Zhou al. (2024): Yan et al. (2018): Chen al. (2019) defined a new sequence 2k:
B1
2k Fz +
(Tk I& 1k-1)
31
7

---

Published in Transactions on Machine Learning Research (01/2025)
However, since the embedded submanifold is not closed under addition in Euclidean space, this strategy does not work in the Riemannian setting. Therefore, by following the policy Za et al. (2018), let us analyze the case in which Φx (x1......99))) 9k- To simplify the notation, we denote the i-th component of 9N (resp. Uks (x) by I (resp. Ukii: 6k,s)
Lemma 3.3. Suppose that Assumption 3.1 (A4) holds. If @el99....... 9k) Ok and H vla for all ls 2 and some v > 0, then the sequence (Ix) ( k= C generated by Algorithm salisfies
La
9k)2 + f(zx+1) S f(xx) + (grad f(zx), -a (
2 11
for all 1.
Proof. See Appendix
hold. Moreover, let us assume that 0x+1 S Okk (g.g...... gk) = gki a = ax+1H and there erist μ. >> such that pla 3 H vla for all k 2 Then, the sequence (xk) M generated by Algorithm satisfies
Theorem 3.4. Suppose that Assumptions 3.1 (A1)
AA5)
K
Layu²
K
at
Ork
1
E
SC
+ C2
Σ
2
bk
1=
for some conslant C1, C2 > 0.
Remark: Since Algorithm
2 satisfies 0K+1, == max(fx,5, Uk+1,i) > ŌR,i, it logether with 0x+1 Sag, leads to ax/(v /x +e) 2 ax+1/(1 0k41, te), Moreover, from Lemma A.3
S Vōn,s + E'
B B+c
which implies (B + (B+c)-11, 3 H 3 E -
la. Therefore, Algorithm
2 satisfies the assumption ag
ak41Hg41 k41 and pla I H I vla with = (B + +) and v
Proof. We denote grad f(x) by g(xx). First, let us consider the case of k From Lemma 3.3 we have
Laj
f(x2) s f(xi)
+
11 11.
2
By laking E|1-] of both sides, we obtain
Laj
[xff(z2) < f(x1) (o 9 +
EE
|1 11
2
$ + (g(r)), 91l); + Lafu²
o
+ ll((=1)12)
2
where the second inequality comes from Lemma get
A.2
By taking E[|] of both sides and rearranging terms, we
Lay
Laço²
[1(()))
<
(()))) -a (11 911)2
2
2b,
By adding G1 to both sides, we obtain
Lay2
Lafa2
E
<[lo(=>)11
{f(x)) - [[((2)) +
+ (g(xx), [ + a.pu?
2
26,
Co
8

---

Published in Transactions on Machine Learning Research (01/2025)
Here, we note that Next, let us consider the case of h: 2 2. From Lemma
anNe [1o(=))13] s sauu??
Therefore, we have
Layua
Lafoz
a1 p
E [|( 2 f(a) B[(())]
+ +
(4)
2
26
3.3 we have
f(*++1) S f(L)) + - 19k) + (o(ox), (ok- (0k-1H - - 1)x)) + Lag
v
11 l
2
for all k x22 From Assumption 3.1 (A3)
Lemma C.1 and ax-1 anlli - O, we have
((*++1) S (ax) + (g(*x), oz H -11)), GB CBtr(ac-HHi) on H Lazua
11 11
By taking [+ of both sides, we obtain
[x[((+++1]] & (*x) (g(xx)) -0k- - Bh (g
+ (GBEE[tr(aa- | - H 1) + Lagu²
Ex
11gb 1
2
< f(x)) a (g(0x), H --- 9(xn))
+ LGRRE H ar H + Loguz
0²
+ (l((2a)11))
2 by
where the first incquality comes from the independence of H for sy and the second inequality comes from Here, since H H ž pld and ag S a4-1, follows that
Lemmas A.1 and A.2
Or- (g(ex), Hz -aap (1(x))))
which implies
[e[[(x++1]] S ((xx) an (u Loxu? [19(2x))1 (B au.l.
Lafo²u2 20x
By taking [[[ of both sides, we have
[[f(xx+1) s B[f(=))] - as (u (a- Layi?
Laço2
[[((=))]1]] + GBE(tr(a--HH anll
2hv.
Rearranging the above inequality gives us
Laxu? E [1)((x))] s {Bf((+)] B[f(x++1) GBE|trQs-14H111 axH +
Lakor
ak (n-
(5)
2bg
By summing 5 from k K, we have
k
Lagu²
Σ
[|(()))1
2
2
K
La
<
[[f(x2)] - E[f(x GBE
ax x1
2by.
k=

---

Published in Transactions on Machine Learning Research (01/2025)
Since uld 3 HK 3 vla for all l 2 1, it follows that tr(a, 2 aivd and tr(ag 0. Here, we note
that (xx((+)) 2 J., from Assumption (A3 Therefore, we have
Lagee
K
Lazo
Ol
[1(()))] |1 s [[((22)] J+ + GBayed +
(6)
2
2bk
2=2
Here, by adding both sides of and 6 we have
ak
μ
Layuz
E [19(2=)
- f(11) - + Co GBauvd
Loz
Σ
akk
2
k=1
C
C2
This completes the proof.
Our convergence analysis (Theorem 3.4) allows the proposed framework (Algorithm to use both constant and diminishing steps sizes. Theorems 3.5 and 3.6 are convergence analyses of Algorithm with constant and diminishing steps sizes and a fixed mini-batch size, respectively.
Thcorem 3.5. Under the assumptions in Theorem 3.4 and assuming that the constant step size a == a salisfies 0 < a 2uL-1v=2 and the mini-batch size bk b salisfies 0 <b S N, the sequence (xt) M generated by Algorithm salisfies
K Σ
E [ggrad [
(
= +T)
K
k=1
Proof. We denote grad f(x)) by g(xx). From Theorem
3.4
we obtain
Law2
Cza² b
a
[1((
S K
+
K
2
Since 0 < a < 2 it follows that (2ap - La²v²)/2 > 0. Therefore, dividing both sides by
(2ah La?v2)/2 gives
2C2a² 1 2ay La? K 20p La?u2 b"
20₁
E [1o(==)11
K
This completes the proof.
Theorem 3.6. Under the assumplions in Theorem 3.4 and assuming that the diminishing step size on == a/Vk satisfies a E (0, and the mini-bater batch size by satisfies < b I N, the sequence (zx) C M generated by Algorithm satisfies
1 log K E |ggrad f(o))1 = ((1+
K
VK
Proof. We denote grad fIx2 by (xxx). Since (ak) satisfics a + 0 (Ic - 00), there exists natural number ko 2 such that, for all ho 2 1, k 2 ko. then 0 < as < 2 Therefore, we obtain
Laguz
0<p-
< 44
for all & 2 Ko. From Theorem 3.4 we have
K
Laquz
C2 K
kg
Laguu
Σ
On 1
E [1o(xx))
St +
ak
-
2
b
2
k==ko
k=
k=
10

---

Published in Transactions on Machine Learning Research (01/2025)
for all K 2 Nu- Since (a is monotone decreasing and ax > 0, obtain
Lamo u²
C2
Rp-1
ak p
E [|((=))1 S S +
a
La
[1(()))11
2
Dividing both sides of this inequality by 2- Kax (2 Lak 1²)
yields
Σ B
[1(
2 S Kax (2p - Lavio v²)
Ci
C2
ak k=1
No-1
La 2
K
ko
2
C1+
La LaE
[1((Tn))12]
Kax
2p Lono u2
k=1
2C2
bkok
24r - Lonk v²
O
From this and ok = a/v/ < 1, we obtain
1
C4 C3
E
+
Okk
2
E
K
KÆx
b
Kax
k=1
k=1
1
hn
1
C4 K
C3 +
E [1(
+
Σ
a
avK
b
From a E (0, 1), we have that
K
K
K
< +
= 1 + log K.
1=1
1=1
Therefore,
K
kn 11
1
Σ
E
[1(
<
1
C3s +
Σ
E
+ b b
Ca
+ C4 log K
K
avK
k=1
This completes the proof.
Theorems 3.7 and 3.8 are convergence analyses of Algorithm 1 with constant and diminishing steps sizes and increasing
sizes, respectively.
Theorem 3.7. Under the assumptions in Theorem 3.4 and assuming that the constant step size on = a
satisfies a < 2ul-iv" and the mini-batch size by satisfies by S and E bx < oc, the sequence (#x) C M generated by Algorithm 1 satisfies
min
[[grad I (Tx)
Igrad [xx = 0 K
K
K
1=1
Proof. We denote grad f(xx) by 9(xx). From Theorem 3.4
we obtain
k
Lava
C1 K
Cza² 2
K
1
E
<
(8)
K
K
bk
A=1
k=
11

---

Published in Transactions on Machine Learning Research (01/2025)
Since 0 < 2p--'v=3, it follows that (2au La?v2)/2 > 0. Therefore, dividing both sides of by
(2am L/2 gives
K Σ [[(([a)11]] $
201
1 + 2C20²
K
20p - La?
K
2ap - La
This completes the proof.
An example of by such that Ex=1 bil oo used in Theorem 3.7 is such that the mini-batch size is multiplied by 8 > every step, i.e.,
b, 8 bo.
(9)
For example, the mini-bated size defined by 9 with 8 makes batch size double each step. Moreover, the mini-batch size defined by (9) and the diminishing step size Og az/ w//kk, where E (0. 1], satisfy that < oo (see the assumption in Theorem 3.8).
Theorem 3.8. Under the assumptions in Theorem 3.4 and assuming that the diminishing step size as
a/vI satisfies E (0, and the mini-balch size by salisfies 0 < by S N and Ex=1af c0, the sequence (#w) C. M generated by Algorithm 1 satisfies
K Σ
min
Iggrad fxxx)
s
agE Igrad f(x))12 - (VK)
K
Ekml Orl
1=1
Proof. We denote grad fxxx by ((xx). Since (ax) satisfies as + 0 (I: + 20), there exists natural
number ko 2 such that, for all h: 2 if ko: then 0 < ar < 2p1 Therefore, we obtain
Lagu2
< 44
2
for all k > ko.
From Theorem 3.4
we have
K Σ
Laguz
ka-1
Laguz
ak 1
E
[1((
s
k
Σ
agk
E 119(2x))1
2
by
Am No
km
km
for all K 2 ko- Since (ax )
is monotone decreasing and ag > 0, we obtain
Lakn 22
k
K
hg 11
Σ
arE [[b[[a)]|]] SC + C2
La [[l[aa)]]
bk
k=kp
Dividing both sides of this inequality by 2-1
Lank, u2) > 0 yields
K
anE [1x(=x))12]
2
Ci+C
La
2p Long UZ
by
k=1
No -1
C1
Σ
La
2
[1,(
2μ Loug uz
1=1
202
O
2
2p Loce 22
bk
R=1
C4
12

---

Published in Transactions on Machine Learning Research (01/2025)
From this and Σ k= bb -1 co, we have that
K ag [[[(=))1|]
k--1 s SC'
K C2
az.
Σ agE
Σ
bk
1=1
which implies that
F
[lo(a>)]1
C
Σx km
Rm
OCe
k=1
02k
From ag == a===a/V; we obtain
K
K
K++ dl
a
a
= +
Vk
Vt
k=1
k=1
1
Therefore,
K
C6
Σ
agE
[la(=x)11
$
Og k=1
This completes the proof.
4 Numerical experiments
We experimentally compared our general framework of Riemannian adaptive optimization methods (Algo-
rithms
!
with several choices of (0n) and (ba)s= with the following algorithms:
RSGD
Bonnabel
2013 Algorithm with (x(11....... 9k) = gk and th (91...... .9k) = Ia-
RASA-LR, RASA-L, RASA-R Kasai et al. 2019. Algorithm 1): B = 0.99.
RAdam: Algorithm with (on) defined by 2 ( defined by 3 [1 =.9, 32 -0.999 and
e= 10-8.
RAMSGrad: Algorithm 2 with 31 = 0.9, 3, = 0.999 and = 10 10-8
We experimented with both constant and diminishing step sizes. For each algorithm, searched in the set 110-1,10-2..... ".00=,..,,,,".... for the best initial step size a (both constant and diminishing). Note that the constant (resp. diminishing) step size was determined to be ag = a (resp. 0* = =//Ak) for all 2 1. For each experiment, each data set was split into a training set and lest set. The experiments used a MacBook Air (M1, 2020) and macOS Monterey version 12.2 operating system. The algorithms were written in Python 3.12.1 with the NumPy 1.26.0 package and the Matplotlib 3.9.1 package. The Python implementations of the methods used in the numerical experiments available at httttps: httttss://gtthttt httpps///githhb..cmiiiaeeeeeee 202408-adaptive
4.1 Principal component analysis
We applied the algorithms to principal component analysis (PCA) problem (Kasai et al. 2018 Roy et al. 2018 For N given data points II,...,IN Rnn and (5 n),, the PCA problem is equivalent to minimizing
1 S(U) = N
Σ 1=
I
UUT x4 2
(10)
13

---

Published in Transactions on Machine Learning Research (01/2025)
the Stiefel manifold St(p, n). Therefore, the PCA problem can be considered to be optimization problem on the Stiefel manifold. In the experiments, we set to 10 and the batch size b to 2 We used the QR-based retraction on the Stiefel manifold St(p,n) (Absil et al. 2008 Example 4.1.3), which is defined by
Rx(n) qf(X + n),
for X E St(p, n) and 17 E Tx St(p, n), where qf(-) returns the Q-factor of the QR decompasition.
We evaluated the algorithms on training images of the MNIST dataset (LeCun et al., 1998) and the COTL100 dataset (Nene et al. 1996). The MNIST dataset contains 28 x 28 gray-scale images of handwritten digits. It has a training set of 60,000 images and test set of 10,000 images. We transformed every image into a 784-dimensional vector and normalized its pixel values to lie in the range of [0, 1]. Thus, we set N 60000 and n = 784. The COIL100 dataset contains 7,200 normalized color camcra images of the 100 objects taken from different angles. As in the previous study (Kasai et al. 2019), we resized them lo 32x32 pixels. Moreover, we split them into an 80% training set and 20% test set. Thus, we N 5760 and 1024.
Figure ((a) (resp. Figure I(b) shows the performances of the algorithms with constant (resp. diminishing)
step size for the objective function values defined by (10) with respect to the number of iterations the
training set of the MNIST dataset, while Figures ((a) and ((b) present those on the test set. Moreover,
Figures 5(a) and 5(b] show those on the training set of the COIL100 dataset, while Figures 6(a] and 6(b)
present those on the test set. Figure 3(a) (resp. 3(b] presents the performances of the algorithms with a constant (resp. diminishing) step size for the norm of the gradient of objective function defined by (00) with
respect to the number of iterations on the training set of the MNIST dataset, while Figures [[a) and 4(b)
present those the test set of the MNIST dataset. Moreover, Figures 7(a) and 7(b) show those the training set of the COIL100 dataset, Figures 8(a) and ((b) present those on the test set. The experiments were performed for three random initial points, and the thick line plots the average of all experiments. The area bounded by the maximum and minimum values is painted the same color as the corresponding line. Note that the upper part of Figures ((a) 3(b) ((a) and 4(b) are cut off. The initial values the gradient norm these figures are approximately 38, highlighting that the gradient norm was minimized during the optimization process. Furthermore, our results differ from those presented in Kasai al. (2019) because of three key factors. Tn Kasai et al. (2019), the mini-batch size was fixed at 10, while in experiments we used 20, Furthermore, the scope of the grid search differs from that of Kasai et al. 2201)). Moreover, we experimented with a doubly increasing batch size every 100 steps from an initial batch size of b 2 and constant and diminishing step sizes. The results are given in Appendix
RSGD
12
RSG3
RAdum
TAcum
Obeedive fondionvaal
3x
1x/22
300
202
100
203
350
500
Baration
number teneatoon
(a) constant learning rate
(b) diminishing learning rate
Figure 1: Objective function value defined by MNIST datasets.
10 versus number of iterations on the training set of the
Figures [(a] and ((a) indicate that RAdam and RAMSGrad (Algorithm 2 performed comparably to RASA- LR the sense of minimizing the objective function value. Figures [(b) and [(b indicate that RAdam and RAMSGrad (Algorithm 2 outperformed RASA-L and RASA-R. Figures 3(a) and ((a) show that RAMSGrad (Algon Algorihhm performed better than RASA-LR in the sense of minimizing the full gradient norm of the
14

---

Published in Transactions on Machine Learning Research (01/2025)
R3GD
RAMSScnd
NASA.L
4X
1A5A
NASA RASA-LR
NASA N.
1x13
400
Number of terations
Rumber Ieeeriass
constant learning rate
(b) diminishing learning
Figure 2: Objective function value defined by datasets.
versus number of iterations on the test of the MNIST
NATT wwe.ootteeeeo.oo grdd the aijettee lactinn
107
NAMSO/d
RASA-A RASA-LR
RASA-E 4 RASA-UR
the unod unt the
(a) constant learning rate
(b) diminishing learning rate
Figure 3: Norm of the gradient of objective function defined by training set of the MNIST datasets.
10
versus number of iterations on the
10
121
RACA-A RASH-L
RACA-5 RASA-AR
1x12
1x12
3x10
2x-17
100
202
300
203
(a) constant learning rate
(b) diminishing learning rate
Figure 4: Norm of the gradient of objective function defined by set of the MNIST datasets.
10 versus number of iterations on the test
15

---

Published in Transactions on Machine Learning Research (01/2025)
RIGD
Akown
Raroum
RAMSScnd
RASA.L
1A5A
NASA in RASA-LR
NASA N.
10
10
4x12
100
400
560
Number of Rerations
fumber of feerrlass
constant learning rate
(b) diminishing learning
Figure Objective function value defined by versus number of iterations on the training set of the COIL100 datasets.
NAMSOLd
RASA-A RASA-LE
RASA-2 4 PASA-UR
10
131
4x122
202 (a) constant learning rate
203
300
(b) diminishing learning rate
Figure 6: Objective function value defined by datasets.
10 versus number of iterations on the test set of the COIL100
REGD
17
ESG3
TAGUI
Nom ofeee groteen the orpertie dieeton
HAMSOrOd
RASA
RAMSOrLd
Nom grad est the adlectio
100
200
200
202
300
400
300
Number Rocuborra
Numbor teoiioos
constant learning rate
(b) diminishing learning rate
Figure Norm of the gradient of objective function defined by training set of the COIL100 datasets.
10 versus number of iterations the
16

---

Published in Transactions on Machine Learning Research (01/2025)
FAcCOn RASA-IR
10
RASHHER
6X
12127
1x12
28
2x12
100
202
200
(a) constant learning rate
(b) diminishing learning rate
Figure 8: Norm of the gradient of objective function defined by 10 versus number of iterations on the test set of the COIL100 datasets.
objective function. Figures 5(a) and 6(a) indicate that RAdam and RAMSGrad (Algorithm 2 had the best performance in the sense of minimizing the objective function value. Figures 5(b) and 6(b) indicate that RAdam and RA (Algorithm performed comparably to RASA-LR. Figures and 8(a) shows that RAdam had the best performance in the sense of minimizing the full gradient norm of the objective function. Figures 7(b) and 8(b) indicate that RAdam and RAMSGrad (Algorithm performed comparably to RASA-R and RASA-LR.
Table compares performances across different batch sizes, showing the number of iterations and CPU time (in seconds) required by cach algorithm with constant step size to reduce the gradient norm below 2 when solving the PCA problem the MNIST dataset. As analysis (Theorem 3.5) indicates, increasing the batch size reduces the number of iterations needed to decrease the norm.
Table 1: Iterations and CPU time (seconds) required by each algorithm with constant step size to reduce the gradient norm below 2 for solving the PCA problem the MNIST dataset across different batch sizes. f_" indicates cases where the algorithm did not reach the threshold within the maximum allowed (1,000) iterations.
batch size b number of ilerations
CPU time (seconds)
256
RSGD
512
303
0.531
1024
149
0.394
256
391
0.532
RAdam
512
150
0.270
1024
126
0.349
256
190
0.257
RAMSGrad
512
140
0.251
1024
114
0.323
256
493
1.228
RASA-L
512
442
1.291
1024
373
1.476
256
397
0.662
RASA-R
512
371
0.809
1024
300
0.901
256
RASA-LR
512
289
0.862
1024
113
0.447
17

---

Published in Transactions on Machine Learning Research (01/2025)
Table 2 compares performances across different batch sizes, showing the number of iterations and CPU time (in seconds) required by each algorithm with a diminishing step size to reduce the gradient norm below 2 when solving the PCA problem on the MNIST dataset. As our analysis (Theorem 3.6) indicates, increasing the batch size reduccs the number iterations needed to decrease the norm. Similar tables (Tables for the remaining dalasets and experiments can be found in Appendix E
Table 2: Iterations and CPU time (seconds) required by each algorithm with a diminishing step size to reduce the gradient norm below 2 for solving the PCA problem on the MNIST dataset across different batch sizes.
batch size b number itcrations
CPU time (scconds)
256
239
0.315
RSGD
512
140
0.327
1024
85
0.229
256
292
0.434
RAdam
512
189
0.349
1024
0.280
256
234
0.324
RAMSGrad
512
224
0.437
1024
141
0.384
256
409
1.008
RASA-L
512
370
1.133
1024
282
1.102
256
235
0.399
RASA-R
512
194
0.418
1024
156
0.472
256
92
0.229
RASA-LR
512
0.192
1024
36
0.145
4.2
Low-rank matrix completion
We applied the algorithms to the low-rank matrix completion (LRMC) problem (Boumal & Absil
2015 Kasai
2019 Hu et al.] 2024). The LRMC problem aims to recover low-rank matrix from an incomplete et al.
matrix X = (X.)) E ROXN, We denote the set of observed entries by Ω C {l...., n2 (I,...,,N) i.e. (i.j) e
Ω if and only X,y known. Here, we defined the orthogonal projection Pa, R - R :a (a. such that the j-th element Po. (a) is aj if (i, E Q and 0 otherwise. Moreover, we defined qi Buxp R + R
as
(U,,z) arg min [| (Ua x)1l2: DERP
(11)
for By partitioning X (I), (II,...,NN) IN), the rank-p LRMC problem is equivalent to minimizing
1 f(U)
PP (Uq((U)) - 2:)
(12)
2N
the Grassmann manifold Gr(p, n). Thercfore, the rank-p LRMC problem be considered to be
optimization problem on the Grassmann manifold (see Hu el al. 2024 Section 1) Kasai et al. 2001 Section 6.3) for details).
We evaluated the algorithms on the MovieLons-11 1M datasots (Harper & Konstan 2015 and the Jeste datasets for recommender systems. The MovieLens- 1M datasets contains 1,000,209 ratings given by 6,040
http:///goooplmn.coc///attssstss
18

---

Published in Transactions on Machine Learning Research (01/2025)
users 3,952 movies. Moreover, split them into 80% training set and 20% test set. Thus, we set N 4832 and n 3952. The Jester dataset contains ratings of 100 jokes given by 24,983 users with scores from -00 to 10. In addition, we split them into an 80% training set and a 20% test set. Thus, we set N = 19986 and n 100.
In the experiments, we set p to 10 and the batch size to 28. We used numpy w.w.llinlglaa.... 1stsc solve the least squares problem [1I.. We used retraction based on polar decomposition on the Grassmann manifold Gr(p,n) n) (Absil al. 2008 Example 4.1.3), which is defined through
(nx(n) = (X [xp + )x
for [X] Gr(p,n) and of E Tix] n).
Figure ((a) (resp. Figure 9(b) shows the performances of the algorithms with constant (resp. diminishing) step size for objective function values defined by (12) with respect to the nunber of iteralions on the training set of the MovieLens-1MM dataset, while Figures 10(a) and 10(b] present those the Jester dataset. Moreover, Figures [3(a) and [3(b) show those on the training set of the Jester dataset, while Figures 14(a) and [4(b) present those on the test set of the Jester dataset. Figure [1(a] (resp. [i(b]] shows the performances of the algorithms with a constant (resp. diminishing) step size for the norm of the gradient of the objective function defined by (10) with respect the number of iterations on the training set of the MovieLens-1M dataset, while Figures 12(a) and [2(b] present those on the test. set of the MovieLens-IM dataset. Moreover, Figures [5 and [b(b) show those the training set of the Jester dataset, while Figures 16(a] and 16(b) present those the test set of the Jester dataset. The experiments were performed for three random initial points, and the thick line plots the average results of all experiments. The area bounded by the maximum and minimum values is painted the same color as the corresponding line. Moreover, we experimented with a doubly increasing batch size every 20 steps from an initial batch size b = 26 and constant and diminishing step sizes. The results are given in Appendix G
RSGD
RSGO
Hadbm
nAMSGind
RAMSC/ad
RASA.
RASA
RASA- R
BASA IL
Obpesvve fund
1x
5x100
Number of terations
Numbar fteattonss
constant learning rate
(b) diminishing learning rate
Figure Objective function value defined by MovieLens-1e 1M datasets.
12 versus number of iterations on the training set of the
Figures [(a] and 10(a) indicate that RAMSGrad (Algorithm 2 performed better than RASA-L and RASA-
LR in the sense of minimizing the objective function value. Figures 9(b) and 10(b) show that RAdam and RAMSGRad (Algorithm 2 performed comparably to RASA-L and RASA-LR. Figures (1 and 12(a) indicate that R.Adam performed comparably to RASA-R in the sense of minimizing the full gradient norm of the objective function. Figures [I(bb and show that RAdam outperformed RASA-R. Figures 13 a [3(b] 14(a) and 14(b] indicate that RAMSGrad (Algorithm 2 performed better than RASA-LR in the sense of minimizing the objective function value. Figures 15(a) and 16(a) indicate that RAdam performed comparably to RASA-L and RASA-R in the sense of minimizing the full gradient norm of the objective function. Figures R.
15(b) and 16(b) show that RAMSGrad (Algorithm 2 outperformed RASA-L and RASA-
19

---

Published in Transactions on Machine Learning Research (01/2025)
RSGD
RAMSScdd
19
NASA.L
1A5A
NASA in RASA-LR
NASA A.
1X
6X
Number of terations
humber of Ieerlons
constant learning rate
(b) diminishing learning
Figure 10: Objective function value defined by MovieLens-1M 1M datasets.
versus number of itcrations on the set of the
10
NATT the grad vet the aejettie hacton
HAMSO/dd
RASA-A RASA-LR
RASA-6
PASA-UA
prod crt
(a) constant learning rate
(b) diminishing learning rate
Figure 11: Norm of the gradient of objective function defined by versus number of ilerations on the training set of the MovieLens-1MM datasets.
RACA-R RASH-L
6x120
RASA-S
BASA-A
127
12
2x10
2x177
(a) constant learning rate
(b) diminishing learning rate
Figure 12: Norm of the gradient of objective function defined by set of the MovieLens- 1M datasets.
versus number of iterations on the test
20

---

Published in Transactions on Machine Learning Research (01/2025)
RSGD
4XP
Akown
RAMSScd
RASA.
1A5A
NASA
NASA
RASA-LR
Number of Rerations
humber fetrrooss
constant learning rate
(b) diminishing learning
Figure 13: Objective function value defined by Jester datasets.
12 versus number of iterations on the training of the
4 1P
RASA-A RASA-LR
RASA-6
PASA-UL
1x
(a) constant learning rate
(b) diminishing learning rate
Figure 14: Objective function value defined by datasets.
versus number of iterations on the test set of the Jester
ESG3
TACUII
Ram oftec pra the oldesive fancion
RAMSOrdd
ROSA
Semm the osgectio
REOD
RAGW
RANSCFAC
BASAL RASSA
20-1
16-1
150
Number Reutors
humbor tenutoos
constant learning rate
(b) diminishing learning rate
Figure 15: Norm of the gradient of objective function defined by training set of the Jester datasets.
12 versus number of iterations the
21

---

Published in Transactions on Machine Learning Research (01/2025)
RASA-IR
RAGWI RANSOraS
10
16a
(a) constant learning rate
(b) diminishing learning rate
Figure 16: Norm of the gradient of objective function defined by set of the Jester datasets.
12
versus number of iterations on the test
5 Conclusion
This paper proposed general framework of Riemannian adaptive optimization methods, which encapsu- lates several stochastic optimization algorithms on Riemannian manifolds. The framework incorporates the mini-batch strategy often used in deep learning. We also proposed RAMSGrad (Algorithm 2 that works on embedded submanifolds in Euclidean space within our framework. addition, we gave convergence analyses that are valid for both a constant and diminishing step size. The analyses also revealed the relationship between the convergence rate and mini-batch size. We numerically compared the RAMSGrad (Algorithm ? with the existing algorithms by applying them to principal component analysis and low-rank matrix comple- tion problems, which can be considered to be Riemannian optimization problems. Numerical experiments showed that the proposed method performs well against PCA. RAdam and RAMSGrad performed well for constant and diminishing step sizes especially on the COIL100 dataset.
Acknowledgments
We are sincerely grateful to the Action Editor, Stephen Becker, and the three anonymous reviewers for helping us improve the original manuscript. This work was supported by JSPS KAKENHI Grant, number JP23KJ2003.
References
P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization Algorithms on Matrir Manifolds. Prince- ton University Press, 2008.
Sheldon Axler. Linear Algebra Done Right. Springer, 4th edition, 2024.
Lukas Balles, Javicr Romero, and Philipp Hennig. Coupling adaptive batch sizes with learning rates, 2017.
Heinz H Bauschke and Patrick L Combettes. Conver Analysis and Monotone Operator Theory in Hilbert Spaces. Springer, 2011.
Gary Bécigncul and Octavian-Eugen Ganca. Riemannian adaptive optimization methods. Proceedings of The International Conference on Learning Representations, 2019.
Silvere Bonnabel. Stochastic gradient descent on Riemannian manifolds. IEEE Transoctions Automatic Control, 58(9) 2013.
Nicolas Boumal and P-A Absil. Low-ram rank matrix completion via preconditioned optimization on the Grass- mann manifold. Linear Algebra and its Applications, 475:200-25 209,000-399, 2015.
22

---

Published in Transactions on Machine Learning Research (01/2025)
Richard H. Byrd, Gillian M. Chin, Jorge Nocedal, and Yuchen Wu. Sample size selection in optimization methods for machine learning. Mathemalical Programming, 134(1):127-155) 2012.
Léopold Cambier and P-A Absil. Robust low-rank matrix completion by Riemannian optimization. SIAM Journal on Scientific Computing, 38(5):S440 S4600, 2016.
Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong. On the convergence of a class of adam-type algorithms for non-convex optimization. Proceedings of The International Conference on Learning Representations, 2019.
Minhyung Cho and Jachyung Leo. Riemannian approach batch normalization. Advances in Neural Information Processing Systems, 30, 2017.
Soham De, Abhay Yadav, David Jacobs, and Tom Goldstein. Automated inference with adaptive batches. Proceedings of the 20th International Conference on Artificial Intelligence and Statislics, volume 54 of Proceedings of Machine Learning Research, pp. 1504-1513. PMLR, 2017.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, pp. 2121-2159, 2011.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training imagenet in hour, 2018.
F Maxwell Harper and Joseph A Konstan. The MovieLen datasets: History and context. Acm transactions on interactive intelligent systems, 5(4):1-19, 2015.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. COURSERA: Newral Networks for Machine Learning, pp. 26-31, 2012.
Jiang Hu, Ruicheng Ao, Anthony Man-Cho So, Minghan Yang, and Zaiwen Wen. Riemannian natural gradient methods. SIAM Journal on Scientific Computing, 46(1):A204-A2)) 2024.
Wen Huang, Kyle A Gallivan, and P-A Absil. A Broyden class of quasi-Newton methods for Riemannian optimization, SIAM Journal Optimization, 25(3):1660-166) 11116666-666,,, 2015.
Hideaki liduka. Theoretical analysis of Adam using hyperparameters close to one without Lipschitz smooth- ness. Numerical Algorithms,, (5:3):383 421, 2024.
Hiroyuki Kasai, Hiroyuki Sato, and Bamdev Mishra. Riemannian stochastic recursive gradient algorithm. In International Conference on Machine Learning, pp. 2516 2524. PMLR, 2018.
Hiroyuki Kasai, Pratik Jawanpuria, and Bamdev Mishra. Riemannian adaptive stochastic gradient algo- rithms on matrix manifolds. In International Conference on Machine Learning, pp. 3262 3271, PMLR, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Proceedings of The International Conference on Leurning Representations, pp. 15, 2015.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based lcarning applied to docu- ment recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2019.
Liangehen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate. In Proceedings of the 7th International Conference on Learning Representations, 2019.
Sameer A Nene, Shree K Nayar, Hiroshi Murase, et al. Columbia object image library (COIL-100). Technical Report CUCS-006-96, 1996.
23

---

Published in Transactions on Machine Learning Research (01/2025)
Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical representations. In Aduances in neural information processing systems, pp. 6338 6347, 2017.
Sashank Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of Adam and beyond. Proceedings of The International Conference on Learning Representations, pp. 1 1-23, 2018.
Soumava Kumar Roy, Zakaria Mhammedi, and Mehrtash Harandi. Geometry aware constrained optimization techniques for deep learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4460-4469, 2018.
Hiroyuki Sakai and Hideaki Tiduka. Riemannian adaptive optimization algorithm and its application to natural language processing. IEEE Transactions Cybernetics, 52(8):7328 7339, 2021.
Hiroyuki Sakai and Hideaki liduka. Convergence of Riemannian stochastic gradient descent on Hadamard manifold. Pacific Journal of Optimization, 2024. doi: https://doi.org/ wtt.oorgg//.....11 10.61208/ 11112/0////22---112 pjo-2024-005.
Takashi Sakai. Riemannian Geometry. American Mathematical Society, 1996.
Hiroyuki Sato. Riemannian Optimization and Its Applications. Springer, 2021.
Naoki Sato and Hideaki Tiduka. Existence and estimation of critical batch size for training generative adversarial networks with two time-scale update rule. In International Conference on Machine Learning, pp. 30080-30104. PMLR, 2023.
Samuel L. Smith, Pieter-Jan Kindermans, and Quoe V. Le. Don't decay the learning rate, increase the batch size. In International Conference on Learning Representations, 2018. URL forum?id-B1Yyl1xc22
net/
Bart Vandereycken. Low-rank matrix completion by Riemannian optimization. SIAM Journal on Optimiza- tion, 23(2)11214 2013.
Yan Yan, Tianbao Yang. Zhe Li, Qihang Lin, and Yi Yang. A unified analysis of stochastic momentum methods for deep learning. In Proceedings of the Sweent-Seventh International Joint Conference Arlificial Intelligence, pp. 2955-200200 2961, 2018.
Jihun Yun and Funho Yang. Riemannian SAM: Sharpness-aware minimization on riemannian manifolds. Advances in Neural Information Processing Systems, 36, 2024.
Manzil Zaheer, Sashank Reddi, Devendra Sachan, Satyen Kale, and Sanjiv Kumar. Adaptive methods for nonconvex optimization. Advances in newral information processing systems, 31, 2018.
Matthew D Zeiler. Adadclta: an adaptive lcarning rate method. ar preprint arXiv 2012.
Hongyi Zhang, Sashank J Reddi, and Suvrit Sra. Riemannian SVRG: Fast stochastic optimization on Riemannian manifolds. Advances in Neural Information Processing Systems, 29, 2016.
Jingzhao Zhang, Hongyi Zhang, and Suvrit Sra. R-SPIDER: A fast Riemannian stochastic optimization algorithm with curvature independent rate. arXiv preprint 18X1::811.04499 2018.
Dongruo Zhou, Jinghui Chen, Yuan Cao, Ziyan Yang, and Quanquan Gu. On the convergence of adaptive gradient methods for nonconvex optimization. Transactions on Machine Learning Research, 2024.
Pan Zhou, Xiao-Tong Yuan, and Jiashi Feng. Faster first-order methods for stochastic non-convex opti- mization on Riemannian manifolds. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 138-147. PMLR, 2019.
Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances in Neural Information Processing Systems, 14118795188806 2020.
24

---

Published in Transactions on Machine Learning Research (01/2025)
A
Useful Lemmas
Lemma A.1. Suppose that Assumption 3.1 (A1) holds. Let (0x)km be sequence generated by Algorithm
7 Then,
En grad fB, (#*)] grad fax)):
for all 2 1.
Proof. From 1 Assumption 3.1 (A1)
and the linearity of E&(T], we have
1
Es ggrad fB, (xx)]
Es grad fse, (xx)] grad f(xx).
bk
This completes the proof.
Lemma A.2. Suppose that Assumptions 3.1
and (A2) A1
hold. Let (Ik)R= be sequence generated by
Algoritimm 7 Then,
o²
Ex [Igrad fB, (xx)]12]
+ Iiara f3xx)11
bk
for all lo 2 1.
Proof. From ||a + b112 = -|aa12 + 2 (a, 2/2 + 1161 we obtain
Eg |Igrad fR, (xx) Ex ||grad fn, (#x) f f(x))1
2 [grad [1, (3xx) grad (**),, grad (xx))) E2 [Igrrad (xxx [2]
(13)
for all k 2 From and Assumption 3.1 (A1) the first term on the hight-hand side of 13 yields
b.
Ex [grad Sa, (Tx) - grad f(xx = En
grad Son. (IN) - grad ((xx)
bR 1=1
ER
Igrad Ssn,, grad S (ax))12
o²
bk
where the second equality comes from Assumption
3.1
A2
From Lemma A.1 the second term the
right-hand side of (13) yields
21, (grad fB. (ax) - grad f(xx), grad [xx)) 2 (Ex Igrad fB, (Iw)] grad f(*x), grad f(xx))
2 (grad f(xx) grad f (*x), grad f(xx))
0.
Therefore, we obtain
x² + llgrad (xx))
Ex [|grad Sa, (x=))
bx
for all l 2 1. This completes the proof.
25

---

Published in Transactions on Machine Learning Research (01/2025)
Lemma A.3. Suppose that Assumption 3.1 A8
holds.
Then, the sequence (xx) M generated by
Algorithm 2 solisfies
6,,i B²,
for all 21 and ......d.
Proof. Note that from Assumption 3.1 A3)
we have
g S 9k.1 +
gi.d = -119s S B
for all 1 2 1 and i 1, I....,dd d. The proof by induction. For k 1, from S 32 < 1, we have
11 = U1 == 82 + (1 B2)9ř (1 - B2)9, S g S B²
Suppose that 0x-14 S B². From U2- S 1 S B², have
Tk = 820h 1, (1 32)9k, S 32B² + +1 (1- +1-333)B2 BB.
Thus, induction ensures that Uk S B² for all k 2
B Proof of Lemma 3.3
Proof. We denote grad f(xx) by 9(xx). From Proposition 3.2 we have
5 + (g(ax). - as Px. HH 9n))2 + Pa gx) 9x)
for all A 2 1. Here, we note that the tangent space Tr, M of an embedded submanifold M in Euclidean
space is a subspace of Euclidean space. Therefore, according to the result in (Axler 2024 6.57 (a)), the
projection Paz is a linear map. From the linearity and symmetry of PEL. we obtain
- -arw),aapPe
(H 9x))2 = (Pax (Pox((())) -atti 9x)x = (g(xx), -aul1 -aulli'9h)g
From the symmetry of Pa, and Pz. Pay Px,, have
|| Pap (11 1b)ll ax P2, (1 9k) 11
of (Pz, (H gk ) Pa. (H 9x))2
at (H gk. Ps. (Pos (Hx 9k))) of (HE")) gk, Pz, (H=)) gk))
s of H g 2 Pas (H 9h) 112
Here, when Pz, (H 9x) # 0 Rd, follows that
| P.. (H²² gx) < af H "xll22 S azLu 11
gx)ll where the first inequality comes from Pz. (11 (145'gx)]; - || 91l122 (see Bauschke & Combettes 2011 Corollary 3.24 (ii)) and Axler (2024, 6.57 (h)) for details) and the second inequality comes from O < H =
vIg. On the other hand, this incquality clearly holds if Pz. (H²²) 9x)) = 0 E Rd. Therefore, we obtain
f(1+41) S f(xx) + (g(#x), -ax ( LaZr²
119x11
for all k 2 1. This completes the proof.
26

---

Published in Transactions on Machine Learning Research (01/2025)
C Linear algebra lemma
Lemma C.1. Let = (a),.. an)
an) an) E R", b = (6), (b)) E IR" and D = diag(d)...... dn) E St n D", If
||al|2 & A and [bb|| - B, then
a Db - ABtr(D).
Proof. From |all2 & A and |16||2 I B, we have |ai| v and |b.| S B for all 1,..., n. Therefore, we obtain
a Db Σ aadbb Σ lail-1bel
S AB Σ
di v ABtr(D).
i=1
This completes the proof.
D Details of numerical experiments
summarize the initial step size, batch size, hyperparameters, and CPU time (seconds) per iteration used in the numerical experiments. Table 3 (resp. Table O sunmarizes the details of solving the PCA problem for the MNIST dataset by using the algorithms with constant (resp. diminishing) step sizes. Table (resp. 6 summarizes the details of solving the PCA problem for the COIL100 dataset by using the algorithms with constant (resp. diminishing) step sizes. Table 7 (resp. Table () summarizes the details of solving the LRMC problem for the MovieLens-IM dataset by using the algorithms with constant (resp. diminishing) step sizes. Table g (resp. Table [0] summarizes the details of solving the LRMC problem for the Jester dataset by using the algorithms with constant (resp. diminishing) step sizes.
Tables 3 10
Table 3: Summary of the initial step size, batch size, CPU time (seconds) per iteration, and hyperparameters used the case of constant step sizes for the PCA problem on the MNIST dataset.
82
8
b
CPU time per iteration (seconds)
RSGD
10
210
1.4 x 10
RAdam
10 2
0.9
0.999
10 8
2
1.481 x 10
RAMSGrad
10-3
0.9
0.999
10-8
210
1.563 x 10-2
RASA-L
10
0.99
10 10-8
2
2.155 x 10
RASA-R
10
0.99
10
210
1.836 x 10
RASA-LR
10
0.99
10 10-8
210
2.374 x 10-2
Table 4: Summary of the initial step size, batch size, CPU time (soconds) per itcration, and hyperparamoters used the case of diminishing step sizes for the PCA problem on the MNIST dataset.
B1
82
b
CPU time per iteration (seconds)
RSGD)
10
210
1.19 x 10-2
RAdam
10-1
0.9
0.999
10-3
2
1.275 x 10
RAMSGrad
10-2
0.9
0.999
10-8
210
1.28 x 10-2
RASA-L.
10-2
0.99
10-8
210
1.803 x 10-2
RASA-R
10 10-2
0.99
10
10-8
210
1.629 x 10-2
RASA-LR
10-2
0.99
10-8
210
2.222 x 10-2
E Extended results on batch size comparisons
This appendix presents a detailed comparison using different batch sizes across various datasets and experiments. Tables [1 16 show the performance of the algorithms under varying batch sizes, highlighting the impact on convergence speed and computational cost. The results further s support our analyses (Thoorems 3.5 and 3.6 that. larger batch sizes lead to faster convergence.
27

---

Published in Transactions on Machine Learning Research (01/2025)
Table 5: Summary of the initial step size, batch size, CPU time (seconds) per iteration, and hyperparameters used in the case of constant step sizes for the PCA problem on the COIL100 datasct.
a
81
82
8
b
CPU time per iteration (seconds)
RSGD
10 2
2
3.937 x 10-2
RAdam
10
0.9
0.999
10-
2
5.256 x 10
RAMSGrad
10-3
0.9
0.999
10
2
4.531 x 10
RASA-L
10 10-3 3
0.99
10 10-8
8
2
5.697 x 10
RASA-R
10
0.99
10-8
10
8
2
5.13 x 10
RASA-LR
10
0.99
10 10-8
2
5.859 x 10-2
Table 6: Summary of the initial step size, batch size, CPU time (seconds) per iteration, and hyperparameters used in the case of diminishing step sizes for the PCA problem on the COIL100 dataset.
a
B1
82
3
e
b
CPU time per iteration (seconds)
RSGD
10 1
210
3.638 x 10-2
RAdam
10-2 2
0.9
0.999
10-8 8
210
5.339 x 10-2
RAMSGrad
10-3 3
0.9
0.999
10-8 8
2
4.033 x 10
RASA-L
10
0.99
10-3 8
2
6.805 x 10-2
RASA-R
10 2
0.99
10 8
2
5.781 x 10-2
RASA-LR
10-2
0.99
10- 8
210
6.938 x 10-2
Table 7: Summary of the initial step size, batch size, CPU time (seconds) per iteration, and hyperparameters used the case of constant step sizes for the LRMC problem on the MovieLens- 1M dataset.
a
31
32
b
CPU time per iteration (seconds)
RSGD
10-3 3
2
2.921 x 10
RAdam
10-3
0.9
0.999
10 s
2
2.813 x 10
RAMSGrad
10-3
0.9
0.999
10 8
2
2,806 x 10
RASA-L
10-3
0.99
10-8
28
3.834 x 10
RASA-R
10
0.99
10 8
2
8.512 x 10 10-2
RASA-LR
10
0.99
10 8
2a
1.285 x 10-2
Table& 8: Summary of the initial step size, batch size, CPU time (scconds) per iteration, and hyperparameters used in the case of diminishing step sizes for the LRMC problem on the MovieLens- 1M dataset.
a
31
32
3
b
CPU time per iteration (seconds)
RSGD
10-3
2
1.931 10
RAdam
10-3
0.9
0.999
10-3
2
1.953 x 10
RAMSGrad
10-3
0.9
0.999
10-3 5
2
2.302 10
RASA-L
10-3
0.99
10-3
2a
3.11 x 10
RASA-R
10-1
0.99
10-3
2a
1.231 x 10 10-2
RASA-LR
10-1
0.99
10-5
28
1.86 10-2
Table 9: Summary of the initial step size, batch size, CPU time (seconds) per iteration, and hyperparameters used in the case of constant step sizes for the LRMC problem the Jester dataset.
a
31
32
b
CPU time per iteration (seconds)
RSGD
10 3
2
6.03 x 10
RAdam
10-3 3
0.9
0.999
10 8
28
5.973 x 10 10-2
RAMSGrad
10 3
0.9
0.999
10 8
2a
5.966 x 10 10-2 2
RASA-L
10 3
0.99
10
2
5.955 x 10-2
RASA-R
10 3
0.99
10 8
2a
5.88 x 10 2
RASA-LR
10
0.99
10 8
2
5.958 x 10 2
28

---

Published in Transactions on Machine Learning Research (01/2025)
Table 10: Summary of the initial step size, batch size, CPU time (seconds) per iteration, and hyperparameters used in the case of diminishing step sizes for the LRMC problem on the Jester dataset.
81
82
B
b
CPU time per iteralion (seconds)
RSGD
10-3 3
28
5.815
10
RAdam
10
0.9
0.999
10
2
5.836
10
RAMSGrad
10 3
0.9
0.999
10- 5
2
5.885
10
RASA-L
10 3
0.99
10 10-8 8
2
5.746 10
RASA-R
10
0.99
10 10-5 5
2
5,853
10
RASA-LR
10 10-3
0.99
10-8
28
5.829
10
Table 11: Iterations and CPU time (seconds) required by each algorithm with constant step size reduce the gradient norm below for solving the PCA problem on the COIL100 dataset, across different batch sizes. A =_" indicates cases where the algorithm did not reach the threshold within the maximum allowed 1,000 iterations.
batch size b number of iterations
CPU time (seconds)
256
RSGD
512
1024
256
RAdam
512
19
0.178
1024
14
0.159
256
132
1.105
RAMSGrad
512
23
0.220
1024
14
0.158
256
297
3.558
RASA-L
512
122
1.636
1024
75
1.169
256
53
0.490
RASA-R
512
34
0.358
1024
29
0.372
256
797
9.741
RASA-LR
512
225
2.968
1024
60
0.936
29

---

Published in Transactions on Machine Learning Research (01/2025)
Table 12: Iterations and CPU time (seconds) required by each algorithm with a diminishing step size to reduce the gradient norm below 4 for solving the PCA problem on the COIL100 dataset across different batch sizes.
batch size b number of iterations
CPU time (seconds)
256
237
1.695
RSGD
512
171
1.526
1024
132
1.562
256
21
0.160
RAdam
512
17
0.153
1024
16
0.179
256
32
0.245
RAMSGrad
512
23
0.207
1024
0.247
256
94
1.082
RASA-L
512
52
0.670
1024
36
0.542
256
19
0.168
RASA-R
512
12
0.113
1024
9
0.123
256
36
0.420
RASA-LR
512
31
0.400
1024
27
0.411
Table 13: Iterations and CPU time (seconds) required by each algorithm with constant step size to reduce the gradient norm below 3 for solving the LRMC problem on the MovieLens-IM 1M dataset across different batch sizes. 4 indicates cases where the algorithm did not reach the threshold within the maximum allowed (300) iterations.
batch size b
number iterations
CPU time (seconds)
64
23
0.143
RSGD
128
20
0.249
256
21
0.481
64
45
0.292
RAdam
128
26
0.312
256
16
0.365
64
RAMSGrad
128
256
155
3.717
64
110
4.490
RASA-I
128
19
0.950
256
0.447
64
37
0.637
RASA-R
128
34
0.745
256
34
1.106
64
19
0.723
RASA-LR
128
0.424
256
0.232
30

---

Published in Transactions on Machine Learning Research (01/2025)
Table 14: Iterations and CPU time (seconds) required by each algorithm with a diminishing step size to reduce the gradient norm below solving the LRMC problem on the MovieLens-IN 1M dataset across different batch sizes. s." indicates cases where the algorithm did not reach the threshold within the maximum allowed (300) iterations.
batch size b number of iterations
CPU time (seconds)
64
108
0.691
RSGD
128
97
1.172
256
110
2.595
64
15
0.093
RAdam
128
13
0.149
256
11
0.238
64
50
0.322
RAMSGrad
128
32
0.385
256
22
0.503
64
8
0.299
RASA-L
128
7
0.305
256
0.261
64
RASA-R
128
256
64
0.874
RASA-LR
128
16
0.705
256
13
0.688
Table 15: Iterations and CPU time (seconds) required by each algorithm with constant step size to reduce the gradient norm below 1/4 for solving the LRMC problem on the Jester dataset across different batch sizes. 4_" indicates cases where the algorithm did not reach the threshold within the maximum allowed (300) iterations.
batch size b
number of ilerations
CPU time (seconds)
64
204
0.391
RSGD
128
210
0.785
256
204
1.500
64
37
0.070
RAdam
128
41
0.150
256
34
0.245
64
RAMSGrad
128
66
0.244
256
54
0.392
64
32
0.062
RASA-L
128
0.134
256
32
0.231
64
38
0.072
RASA-R
128
37
0.137
256
34
0.244
64
174
0.342
RASA-LR
128
48
0.184
256
43
0.312
31

---

Published in Transactions on Machine Learning Research (01/2025)
Table 16: Iterations and CPU time (seconds) required by each algorithm with a diminishing step size roduce the gradient norm below 1/2 for solving the LRMC problem on the Jester dataset across different batch sizes. 4_" indicates cases where the algorithm did not reach the threshold within the maximum allowed (300) iterations.
batch size b
number of iterations
CPU time (seconds)
64
RSGD
128
256
64
188
0.376
RAdam
128
225
0.840
256
167
1.232
64
21
0.039
RAMSGrad
128
18
0.064
256
19
0.136
64
184
0.372
RASA-L
128
249
0.943
256
178
1.325
64
219
0.427
RASA-R
128
256
195
1.441
64
0.008
RASA-LR
128
4
0.012
256
0.015
F Conditions under which Assumption 3.1
A22)
holds
We introduce the assumptions (Assumptions F.1
B1 and B2
under which Assumption 3.1
[22)
from
Section 3.1 holds, and demonstrate that Assumption 3.1
(A2) is valid under these assumptions (Lemma
2
Note that Assumptions F.1 (B1) and (B2) are simply the counterparts of Assumptions 3.1 [A4] and A5
for f, applied to each fo (i IN.,,,,, Assumption F.1. For all I,...,N,
(B1) There erists conslant L; > such that.
[DJ+ ((l)))DD((())))) s L, 117l2
for all x E M, E T.M.
(B2) ft bounded below by fa R.
Lemma F.2. Suppose that Assumptions F.1 B1 and (B2) hold. We assume that a random voriable sk,4 takes a value from 1 N following uniform distribution. Then, the sequence (xx) C M generated by Algoritium sotisfies
2 N
Es |lerad fee, (xx) grad f(x)) S L.M.
where
M, = s (0)
32

---

Published in Transactions on Machine Learning Research (01/2025)
Proof. From Assumption 3.1 AA5)
and Proposition 3.2 have
fo s fi Rex
1
grad f((zx))
Li
Ls
$ fa(x()) +
grad Ji(ork), T.j
grad [((xx)
+
grad Si(
2
I
1
1
f(((x))
Igrad fx l1 + 2L;
llgrad f(xx))))
Li
1
f((a))
||grad fi (3x))))
2L;
Therefore, we obtain
Igaad fi(xa)) s 22,(f((TR) - fa) s 2L,Mi,
(14)
where
M, == sup([((x)) k2 1} E [0, o0).
From
||a
bll2 ||all (aa + ||b12 we obtain
||arad foe, - grad f(xx) [2]]
Ex
ggaad Son. (Itx) 11/] - 2 [grad San, gra) (axx)) + Ex [Igrad s (1x))
=
Ex
N Igaad f1(2x) 2 (Ealgrad far, (xx)], grad f*x))) + llgrad fxx))11
Igrad (xx)) Igaad f(zx)
N Igad f*x))
Here, by using
14 it follows that
2 N N
Ex [| ggad (#x)) grad fIx))11 S Σ L.M.
G Experimental results on the PCA and LRMC with increasing batch sizes
33

---

Published in Transactions on Machine Learning Research (01/2025)
5x12
RSGD
Akdwn
RAMSScdd
NASA.
1A5A
NASA
NASA N.
RASA-LR
3x
200 Number of Rerations
fumber Ieeerians
constant learning rate
(b) diminishing learning
Figure 17: Objective function value defined by MNIST datasets.
10 versus number of iterations on the training of the
NAMSO/d
RASA-A RASA-LR
RASA-A
RASA-UR
1x
202
(a) constant learning rate
(b) diminishing learning rate
Figure 18: Objective function value defined by datasets.
versus number of iterations on the test set of the MNIST
10
REGD
RSG3
RAdUmI
TAGUI
Vim oeee proaten of the owg.rte Euectio
HAMSGred
RAMSOrLd
NASAIL
RASA
Norm grudiom est the wwbectio
100
200
200
202
400
500
Numbbr of Rocuborra
Number teoiioos
constant learning rate
(b) diminishing learning rate
Figure 19: Norm of the gradient of objective function defined by training set of the MNIST datasets.
10 versus number of iterations the
34

---

Published in Transactions on Machine Learning Research (01/2025)
RS60
RAdam
FAcau
RAMSOred
FAMrSCred
RASA-L
RASA-L
RASA-A
RASA-S 2
NASA
PA5A
28
Number Barations
Number Resustons
consttatt learning rate
(b) diminishing learming rate
Figure 20: Norm of the gradient of objective function defined by set of the MNIST datasets.
versus number of iterations on the test
REGD
RSGD
RAdar
RAcum
RAMSGmd
Osjecvve www.toonooooooo O9M
Ex12
103
200
4.00
503
100
203
350
403
500
Number Baratisss
number
constant learning rate
(b) diminishing learning rate
Figure 21: Objective function value defined by COIL100 datasets.
10 versus number of iterations the training set of the
HAMSGrdd
HAMSCrad
RASAVL
RASA-L
RACA-R
RASA-4
BASA-LA
PASA
10
131
6x:17
(a) constant learning rate
(b) diminishing learning rate
Figure 22: Objective function value defined by 10 versus number of iterations on the test set of the COTT.100 datasots.
35

---

Published in Transactions on Machine Learning Research (01/2025)
RSGD Akdwn RiMiGind NASA. RASA RASA-LR
RAroum
Nom of The SINTRATT thecteeeeeeeeeeeeeeeee Rucctis
RAMSScnd
1A5A NASA A.
Nom the gra.tentt www.ctteciiie.......ooo.oo......
Number of terations
humber Ieerriass
constant learning rate
(b) diminishing learning
Figure 23: Norm of the gradient of objective function defined by versus number of iterations on the training set of the COTL100 datasets.
REGD
HAMSGRR
HAMSCrDd
RASA-L
RASA-L
RASA-
BASA-A It
10
107
100
202
200
303
Number of terations
Numbir Iieeutlonss
learning
(b) diminishing learning
Figure 24: Norm of the gradient of objective function defined by of the COLL100 datasets.
versus number of iterations on the test
RACA-A RASH-L
7X10P
RASA-S RASA-A
1PP
constant learning rate
(b) diminishing learning rate
Figure 25: Objective function value defined by
12
versus number of iterations on the training set of the
MovieLens- 1M datasets.
36

---

Published in Transactions on Machine Learning Research (01/2025)
RIGD
0x
Akdwn
RAromn
RAMSSCd
NASA.L
1A5A
NASA in
NASA I.
RASA-LR
6X
1x12
5x12
Number of terations
humber deeerrans
constant learning rate
(b) diminishing learning
Figure 26: Objective function value defined by MovieLens-I1e 1M datasets.
versus number of itcrations on the set of the
WATT the grad weet the wwpettiee lacton
RASA-A RASA-LR
RASA-4
RASA-UR
Nerm prod ort the
(a) constant learning rate
(b) diminishing learning rate
Figure 27: Norm of the gradient of objective function defined by versus number of ilerations on the training set of the MovieLens-1MM datasets.
5x122
RACA-A RASH-L
RASA-5
BASA-A
1x12
(a) constant learning rate
(b) diminishing learning rate
Figure 28: Norm of the gradient of objective function defined by set of the MovieLens- 1M datasets.
versus number of iterations on the test
37

---

Published in Transactions on Machine Learning Research (01/2025)
RSGD
Akown
RAMSScdd
RASA.L
1ASA
NASA
NASA A.
RASA-LR
3x12
Number of Rerations
humber of feerroos
constant learning rate
(b) diminishing learning
Figure 29: Objective function value defined by Jester datasets.
12 versus number of iterations on the training of the
4912
4'
RAMSO/dd
RASA-A RASA-LR
RASA-6
PASA-UR
(a) constant learning rate
(b) diminishing learning rate
Figure 30: Objective function value defined by datasets.
versus number of iterations on the test set of the Jester
REGD
RSG3
RAdAm
TAUUMI
HAMSOred
Bum oftec grad can the olissiie fancion
ROSA RAMSOrdd
1000
NASAIL
Wtrm the osgectio
16-
Number of Recubors
humbor teuiooos
constant learning rate
(b) diminishing learning rate
Figure 31: Norm of the gradient of objective function defined by training set of the Jester datasets.
12 versus number of iterations the
38

---

Published in Transactions on Machine Learning Research (01/2025)
REGD
RSGD
RAdam
RAMSGmd
Born the www.errrretttt.oo tfe sbootiee trndian
Kom grident
30-1
104
20
40
20
42
60
30
100
Number Beratisns
Number
constant learning rate
(b) diminishing learning rate
Figure 32: Norm of the gradient of objective function defined by set of the Jester datasets.
12
versus number of iterations the test
39


Reviews:
[{"review_id": "vSJH8DeMt7", "reviewer": "Reviewer_CFpd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The article is well-written and easy to follow, with a particular care in the details of the statements and proofs. This really helps making the work understandable and accessible. The experiments are also well detailed, and easy to reproduce with the code provided by the authors.\n\nAs for the weaknesses, I identified two concerns:\n\n---The first is about the list of assumptions in Assumption 3.1. Given that these assumptions are essential to the theoretical results and their proofs, I think it is important to discuss them in more details. This includes, e.g., discussing when should one expect them in practice, what do they mean intuitively, whether they are satisfied or not in different application scenarios, etc. Otherwise, it is a bit difficult to understand whether these are just theoretical statements that almost never occur practically, or if one should expect them to be true in general (without experimental validation).\n\n---The second is about the final message of the experiments. While I appreciate the careful design of both applications, it remains unclear to me what the final message is. I think the authors should elaborate on a few principles that the plots indicate: in what context one optimization method is better or worse than another? How do the experiments help me for choosing the most appropriate optimization method in my own applications? Currently, the analysis of the results is purely comparative, and, as such, it is difficult to figure out what the final message on performance really is.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this article, the authors propose a general framework for stochastic optimization on Riemannian manifolds. They provide in Algorithm 1 a general scheme which, when applied with the right parameters, allows to emcompass previous approaches in the literature. Moreover, they also propose an adaptation of the classical AMSGrad on manifolds in Algorithm 2, and they prove convergence of both algorithms, in the form of decreasing upper bounds on the average of the sequence of expected gradient norms. Finally, they conclude the article with numerical experiments showing that on two different applications (PCA and low-rank matrix completion), using the authors' schemes indeed works well for both decreasing the objective loss functions as well as the gradient norms.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "---Appendix B. There is a square missing in the lhs of the first equation below equation (11).\n\n---Appendix D. There is a square missing in the middle term of the inequalities below \"it follows that\".\n\n---Appendix D. \"comes from $H_k^{-1} \\leq \\nu I_d$\": you should also recall that $H_k^{-1}$ is diagonal with positive coefficients, because otherwise the statement seems false ($B \\geq A$ for two general square matrices $A, B$ does not necessarily mean that $\\|Bx\\| \\geq \\|Ax\\|$).\n\n---Appendix D. \"From the linearity and symmetry of $P_{x_k}$\": is $P_{x_k}$ really linear? As far as I understand, it is an orthogonal projection onto an affine space, therefore an affine transformation that is not necessarily linear. I also do not see why the norm of $P_{x_k}(y)$ should be less than the norm of $y$ (which is used in the first inequality below \"it follows that\" with $y = H_k^{-1} g_k$).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FiggVjrmdA", "reviewer": "Reviewer_RAJx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: the proposed Riemannian adaptive gradient algorithms is easy to implement by leveraging the ambient Euclidean geometry of the embedded submanifold.  \n\nWeakness: \n1. The proposed algorithms are not convincingly superior to existing baselines, such as RASA and its variants. A key challenge in designing adaptive gradient methods over Riemannian manifolds is the absence of a coordinate system like that in Euclidean space. The paper would benefit from a clearer explanation of how the proposed methods address this issue.\n2. The convergence analysis is limited to a specific setting where no first-order momentum is used. Additionally, the results appear to closely follow the RASA paper, offering no significant advances. The convergence result with constant step size feels more like an intermediate result and does not guarantee obtaining a stationary point in expectation. It seems similar results would also apply to RASA. \n3. The writing could be improved, especially in providing intuitions or motivations for the algorithm’s effectiveness. On the theoretical side, the significance of the constant step size and the challenges it presents are not sufficiently explained. Some of the lemmas seem straightforward and could be omitted. In the numerical experiments, it is important to present results on both training and testing datasets for a more comprehensive evaluation.\n\nBased on all these comments, I consider that this paper does not meet the quality requirements for being published in TMLR.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes Riemannian adaptive gradient methods for solving optimization over compact submanifolds. Convergence analysis is presented, and numerical comparisons with existing algorithms are conducted.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Subsection 1.1. The two paragraphs overlap a lot. Please consider merging them more logically.\n2. Line 7 of Section 2. Missing definition of $O$.\n3. Line 9 of Section 2.1. Missing definition of tr.\n4. Line 11. Should $\\eta$ be any vector of $\\mathbb{R}^{n\\times p}$?\n5. Line 12. Missing a comma after i.e.\n6. Section 2.2. Is M a submanifold as introduced at the beginning of Section 2? In this case, this would exclude the Grassmann manifold.\n7. Second last line of Section 2.2. The local minimizer and the stationary point are distinct concepts in optimization. Why do you need to introduce both?\n8. Line 3 of Page 4. What is the definition of $\\mathcal{D}^d$?\n9. The sentence blow Algorithm 1. I am not sure what is claimed.\n10. Lemmas 3.2 and 3.3. These two lemmas are relatively simple and well-understood in literature.  It may not be necessary to include them as separate lemmas.\n11. Section 3.1. It is mentioned that averaging between iterations does not work in the Riemannian setting. Since the embedded submanifold is considered and addition is possible, please provide more explanation on why this is a challenge.\n12. Lemma 3.5. The result is straightforward and no need to introduce the lemma.\n13. Theorem 3.7. The proof appears quite similar to the one in RASA paper. In RASA, a diminishing step size is used to achieve exact convergence instead of convergence to a neighborhood. \n14. Theorem 3.9. There doesn’t seem to be much difficulty in establishing the convergence theorem based on the results from the RASA paper. Please clarify the novelty here.\n15. Line blow Equation (8). If I understand correctly, the principal component should be in a much smaller dimension of the original data dimension. So, U should be of dimension $p \\times r$, where r is the number of principal components.\n16. Section 4. Like RASA, the authors should present the convergence curves on both training and testing datasets.\n17. Figure 2. The gradient norm is quite large at the end of the iterations. It is hard to draw conclusions. \n18. Figure 3. The results on RAdam and RASA seem inconsistent with the RASA paper. Please add explanations.\n19. Section 4.2. The analysis is for the submanifold, is it applicable to the Grassmann manifold case?\n20. Section 4.2. RASA converges the fastest in the early stage but returns a worse solution than the proposed algorithms at the end. I am wondering whether a smaller (initial) step size would improve the quality of solutions. In addition, please split the training and testing datasets.", "broader_impact_concerns": "Not found", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OqgOK0zZXe", "reviewer": "Reviewer_jkpK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n1. The manuscript is well-written, and the motivation of the study is clear. \n2. The suggested concepts look interesting and promising for Riemannian adaptive optimization methods.\n3. The convergence analysis covers both constant and diminishing step sizes, offering a comprehensive theoretical understanding of the proposed framework. \n4. The authors demonstrate the framework’s applicability to practical optimization tasks by applying it to real problems such as PCA and LRMC. They provided a GitHub repository for reproducibility. \n5. The proof seems clear and correct, and the previous works are well-cited.\n\n## Weaknesses:\n\nWhile generally well-written, a few changes could improve the clarity of the manuscript. The proposed changes are outlined below and in the requested changes section:\n\n1. Extending the experiments to larger datasets, such as ImageNet or CIFAR-10, would strengthen the empirical validation and demonstrate the scalability of the proposed methods. Why did you choose these datasets? \n2. It would be helpful to include in the appendix summary tables for each experiment, detailing the hyperparameters used, such as the learning rate, mini-batch size, and the computation time per epoch. This would improve the clarity of the experimental setup and provide more insights into the computational efficiency of the proposed methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a general framework for Riemannian adaptive optimization methods, extending well-known stochastic optimization algorithms to Riemannian manifolds. A key contribution is the adaptation of AMSGrad on embedded submanifolds of Euclidean spaces. The authors provide rigorous convergence analyses for both constant and diminishing step sizes, investigating the relationship between convergence rates and mini-batch sizes. The framework is validated on practical tasks like principal component analysis (PCA) and low-rank matrix completion (LRMC). Python implementations are provided, enhancing the reproducibility of the work.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Some sections, particularly the mathematical preliminaries, are dense and difficult to follow. Additional clarifications or simplified explanations could improve accessibility.\n2. Displaying the loss on a logarithmic scale in the experiments would make the results clearer and allow for easier comparisons between algorithms.\n3. Including test accuracy in the experiments. \n\n### Typos:\nPage 2: “unite sphere” -> “unit sphere.” \n\nPage 9: “RAMGRad” ->  “RAMSGrad.”", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: As mentioned above, I find that there is sufficient novelty to the paper to be of interest to at least some of the TMLR audience. Since there have been no concerns about *correctness* of the results, therefore I'm pleased to find that the paper meets the TMLR criteria.

I have recommended the paper be Accepted with Minor Revisions so that the authors have an opportunity to expand on the conclusions from the numerical experiment.  In particular, a paragraph has been added in the Contributions that expands on this, but this could be further refined:

- the first part of the paragraph reports facts without analysis, e.g.., "RAMSGrad with a constant step size minimizes the objective
function in fewer or closer iterations compared with RSGD, RASA-L, RASA-R and RASA-LR, regardless
of the dataset".  Please add some insight into why this is
- referring to "AMSGrad" is a little confusing, since it's not clear if you're referring to the original algorithm from the literature, or if you refer to your extension. Adding "AMSGrad/Algorithm 2" would be more clear
- the very end of the paragraph does contain some high-level message, mainly that the algorithm converges faster with a larger batch size, but this is only a common-sense sanity check, as it's obvious.  It would instead be interesting to also say something about what the optimal batch size choice is, when taking into account both the work per iteration and the number of iterations.  The actual computation/work depends on implementation, and the convergence rates are only bounds, so this quantity is harder to analyze theoretically... hence it's a good question to answer *experimentally*.

Also, while this paragraph was added in the "Contributions" section, I didn't notice similar high-level commentary in the actual Numerics section. I'd suggest putting some high-level commentary there as well.

(Also, in the revision you can change the URL links as they no longer need to be anonymized)
Decision: Accept (poster)