SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ya-Ping Hsieh, Mohammad Reza Karimi, Andreas Krause ETH Zürich
{yaping.hsieh, mkarimi}@inf.ethz.ch, krausea@ethz.ch
Univ. Grenoble Alpes, CNRS, Inria, Grenoble INP, LIG & Archimedes RU, NKUA panayotis.mertikopoulos@imag.fr
Many modern machine learning applications – from online principal component analysis to covariance matrix identification and dictionary learning – can be formulated as minimization problems on Riemannian manifolds, typically solved with a Riemannian stochastic gradient method (or some variant thereof). However, in many cases of interest, the resulting minimization problem is not geodesically convex, so the convergence of the chosen solver to a desirable solution - i.e., a local minimizer - is by no means guaranteed. In this paper, we study precisely this question, that is, whether stochastic Riemannian optimization algorithms are guaranteed to avoid saddle points with probability 1. For generality, we study a family of retraction-based methods which, in addition to having a potentially much lower per-iteration cost relative to Riemannian gradient descent, include other widely used algorithms, such as natural policy gradient methods and mirror descent in ordinary convex spaces. In this general setting, we show that, under mild assumptions for the ambient manifold and the oracle providing gradient information, the policies under study avoid strict saddle points / submanifolds with probability 1, from any initial condition. This result provides an important sanity check for the use of gradient methods on manifolds as it shows that, almost always, the limit state of a stochastic Riemannian algorithm can only be a local minimizer.
Modern machine learning systems have achieved remarkable success in the efficient optimization of highly non-convex functions using straightforward Euclidean techniques like stochastic gradient descent. A widely accepted hypothesis to explain this phenomenon is that, when the learning system under study – e.g., a neural network – is sufficiently expressive, local minimizers are essentially as good as global ones [21, 35]; by this token, a training algorithm can attain satisfactory performance by simply evading saddle points of the model's loss surface.
This observation has sparked a far-reaching research thread examining the behavior of various algorithms around saddle points in non-convex functions. Informally, these studies aim to tackle two fundamental questions:
$Q_{1}$ : When does a given scheme, like stochastic gradient descent, avoid saddle points?
$Q_{2}$ : Can we augment a given scheme so that it efficiently escapes saddle points?
Of the above questions,  $Q_{1}$  focuses on explaining the empirical success of commonly used schemes, while the resolution of  $Q_{2}$  usually revolves around proposing new schemes with desirable escape guarantees. These complementary perspectives have been extensively studied over the past decade, 37th Conference on Neural Information Processing Systems (NeurIPS 2023).
leading to a fairly complete understanding of how and when a Euclidean (stochastic) algorithm escapes saddle points, cf. [6, 28, 29, 41, 42, 46, 47, 53, 54] and references therein.
In parallel to the above, the recent surge of interest in Riemannian optimization has prompted a closer examination of Riemannian methods, thereby motivating an extension of  $\mathbf{Q}_1$  and  $\mathbf{Q}_2$  to a manifold setting – itself due to a wide range of breakthrough applications to machine learning and data science, from natural language processing, and signal processing to dictionary learning and robotics [48, 52, 64, 65]. As a result, there is an increasing demand for a comprehensive exploration of various spaces, such as the  $d$ -dimensional torus, Grassmannian or Stiefel manifolds, hyperbolic spaces, and many others.
Unfortunately, in a proper Riemannian setting, only  $\mathrm{Q}_2$  has received sufficient scrutiny thus far. Recent works by Crisciello & Boumal [22] and Sun et al. [66] have shown that standard Riemannian deterministic algorithms can be augmented via the injection of an infinitesimal amount of noise (proportional to the method's desired accuracy), to achieve comparable escape guarantees in terms of oracle complexity as the corresponding Euclidean methods [30]. To the best of our knowledge, all existing results for  $\mathrm{Q}_1$  concern deterministic methods [28, 41, 42, 53] which are significantly limited in scope in large-scale machine learning applications because of their prohibitively high per-iteration cost.
Our results and techniques. In view of the above, our paper aims to provide a general answer to  $\mathbf{Q}_1$  for a broad class of Riemannian stochastic optimization methods – including Riemannian stochastic gradient descent, its retraction-based and/or optimistic variants, etc. Concretely, we focus throughout on a flexible template of Riemannian Robbins-Monro (RRM) schemes [34, 59] which, in addition to the specific algorithms of interest mentioned above, also includes a range of Euclidean methods that can be analyzed efficiently from a Riemannian viewpoint.
Informally, our main result may be stated as follows:
Under any stochastic Riemannian Robbins-Monro method, the probability of converging to a strict saddle point (or a submanifold thereof) is zero.
This statement provides firm grounds for accepting the output of a stochastic Riemannian optimization method as valid, as it shows that saddle points are avoided with probability 1. In the context of stochastic methods, our result builds on a series of foundational results by Pemantle [54] and Brandière & Duflo [20] who focused on hyperbolic traps (isolated saddle points with invertible Hessian). These results were subsequently extended by Benaim & Hirsch [12] to a more general class of unstable sets, but this analysis remained grounded in a flat, Euclidean setting. The connecting tissue of our analysis with these works is the notion of an asymptotic pseudotrajectory (APT), which allows us to couple the long-run behavior of discrete-time RRM methods to that of an associated Riemannian gradient flow. [This discrete-to-continuous comparison is crucial for our analysis in order to apply center stable manifold techniques [63] to the RRM framework.] However, this comes at a significant cost, as establishing the APT property in a Riemannian setting is quite challenging. To achieve this, we employ a set of techniques recently developed by [33] which allow us to make this comparison precise and establish the desired avoidance result.
2 Background on Riemannian Manifolds
We begin with a brief overview of some basic definitions from Riemannian geometry and optimization, solely intended to set notation and terminology; our presentation roughly follows the masterful account of Lee [43, 44], to which we refer the reader for a comprehensive introduction to the topic.
Le  $\mathcal{M}$  be a  $d$ -dimensional, geodesically complete Riemannian manifold. Throughout the sequel, the tangent space to  $\mathcal{M}$  at a point  $x \in \mathcal{M}$  will be denoted by  $\mathcal{T}_x\mathcal{M}$ , and we will write  $\dot{\gamma}(t) \in \mathcal{T}_{\gamma(t)}\mathcal{M}$  for the velocity vector to a smooth curve  $\gamma: \mathbb{R} \to \mathcal{M}$  at time  $t \in \mathbb{R}$ . We will also write  $\langle \cdot, \cdot \rangle_x$  for the metric at  $x \in \mathcal{M}$ ,  $\| \cdot \|_x$  for the associated norm, and  $\mathrm{dist}(\cdot, \cdot)$  for the induced distance function on  $\mathcal{M}$ , the latter being defined via the minimization of the length functional  $\mathcal{L}[\gamma] = \int \| \dot{\gamma}(t) \|_{\gamma(t)} dt$ .
<sup>1</sup>We recall here that a strict saddle manifold is a set of critical points each of which has at least one negative Hessian eigenvalue. Such manifolds include ridge hypersurfaces and other connected sets of non-isolated saddle points that are common in the loss landscapes of high-dimensional machine learning models, so this result has significant cutting power in this regard.
Given a point  $x \in \mathcal{M}$  and a tangent vector  $z \in \mathcal{T}_x\mathcal{M}$ , the (necessarily unique) geodesic emanating from  $x$  along  $z$  will be denoted by  $\gamma_z$ , and we define the exponential map at  $x$  as  $\exp_x(z) = \gamma_z(1)$  for all  $z \in \mathcal{T}_x\mathcal{M}$  (recall here that  $\mathcal{M}$  is assumed complete, so this map is well-defined for all  $x \in \mathcal{M}$  and all  $z \in \mathcal{T}_x\mathcal{M}$ ). Whenever well-defined, the inverse of  $\exp_x$  will be written as  $\log_x\colon \mathcal{M} \to \mathcal{T}_x\mathcal{M}$ , with the understanding that the domain of  $\log_x$  is actually the largest neighborhood of  $x \in \mathcal{M}$  on which the restriction of  $\exp_x$  is a (global) diffeomorphism; by definition, we have  $\log_x(\exp_x(z)) = z$  for all  $z$  for which the relevant quantities are well-defined. Finally, given a pair of points  $x,x' \in \mathcal{M}$  and a tangent vector  $z \in \mathcal{T}_x\mathcal{M}$ , we will write  $\Gamma_{x\to x'}(z)$  for the vector obtained by parallel transporting  $z$  along any minimizing geodesic connecting  $x$  and  $x'$ .
In this general context, we will be interested in solving the Riemannian optimization problem
$$
\operatorname {m i n i m i z e} _ {x \in \mathcal {M}} f (x) \tag {Opt}
$$
for some smooth objective function  $f\colon \mathcal{M}\to \mathbb{R}$ . We will also respectively write
$$
v (x) := - \operatorname {g r a d} f (x) \quad \text {a n d} \quad H (x) := \operatorname {H e s s} (f (x)) \tag {1}
$$
for the negative (Riemannian) gradient and the (Riemannian) Hessian of  $f$  at  $x$ . Finally, in terms of solutions of (Opt), we will focus on the avoidance of strict saddle points of  $f$ , i.e., points  $\hat{x} \in \mathcal{M}$  for which
$$
v (\hat {x}) = 0 \quad \text {a n d} \quad \lambda_ {\min } (H (\hat {x})) <   0 \tag {2}
$$
where  $\lambda_{\mathrm{min}}$  denotes the minimum eigenvalue of the tensor in question. We will also say that a smooth compact component (in the sense of manifolds) of critical points of  $f$  is a strict saddle manifold if there exist constants  $c_{\pm} > 0$  such that all negative eigenvalues of  $H(\hat{x})$ ,  $\hat{x}\in S$ , are bounded from above by  $-c_{-} < 0$ , and any positive eigenvalues (if they exist) are bounded from below by  $c_{+} > 0$ .
To differentiate the above from the Euclidean setting, when  $\mathcal{M}$  is a real space equipped with the Euclidean metric, we will instead write  $\nabla f$  and  $\nabla^2 f$  for the (ordinary) gradient and Hessian matrix of  $f$ . In this case, as is customary, we will not distinguish between primal and dual vectors.
3 Core algorithmic framework
For generality, our avoidance analysis will be carried out in an abstract stochastic approximation framework which includes several popular Riemannian optimization algorithms – from ordinary Riemannian (stochastic) gradient descent, to its retraction-based variants, optimistic methods, etc. For concreteness, we start with the general template below, and we present a (nonexhaustive!) series of representative examples right after.
3.1. The Riemannian Robbins-Monro template. The Riemannian Robbins-Monro (RRM) framework that we will consider for solving (Opt) is an iterative family of methods which directly extends the seminal stochastic approximation scheme of Robbins & Monro [59] to a manifold setting by replacing vector addition with the Riemannian exponential. Roughly following [33], we will focus on the abstract update rule
$$
X _ {n + 1} = \exp_ {X _ {n}} \left(\gamma_ {n} \hat {v} _ {n}\right) \tag {RRM}
$$
1.  $X_{n}\in \mathcal{M}$  denotes the state of the algorithm at each iteration  $n = 1,2,\ldots$
2.  $\hat{v}_n\in \mathcal{T}_{X_n}\mathcal{M}$  is a surrogate for the (negative) gradient  $v(X_{n})$  of  $f$  at  $X_{n}$  (defined in detail below).
3.  $\gamma_{n} > 0$  is the method's step-size (discussed in detail in Section 4).
In the above, the defining element of (RRM) is the sequence of "surrogate gradients"  $\hat{v}_n$ ,  $n = 1,2,\ldots$ , so this will be our first object of interest. Formally, letting  $\mathcal{F}_n$  denote the history of  $X_{n}$  up to stage  $n$  (inclusive), we will write
$$
\hat {v} _ {n} := v \left(X _ {n}\right) + U _ {n} + b _ {n} \tag {3}
$$
$$
U _ {n} := \hat {v} _ {n} - \mathbb {E} \left[ \hat {v} _ {n} \mid \mathcal {F} _ {n} \right] \quad \text {a n d} \quad b _ {n} := \mathbb {E} \left[ \hat {v} _ {n} \mid \mathcal {F} _ {n} \right] - v (X _ {n}), \tag {4}
$$
as the random error and the offset of  $\hat{v}_n$  relative to  $v(X_n)$  respectively. It will also be convenient to introduce the total error  $W_{n} = \hat{v}_{n} - v(X_{n}) = U_{n} + b_{n}$ , which captures both random and systematic fluctuations in  $\hat{v}_n$ , and which measures the total deviation of  $\hat{v}_n$  from  $v(X_{n})$ .
Two points are worth noting here: First,  $\hat{v}_n$  is not adapted to  $\mathcal{F}_n$ , so  $U_n$  is random relative to  $\mathcal{F}_n$ ; on the other hand,  $b_n$  is  $\mathcal{F}_n$ -measurable, so it is deterministic relative to  $\mathcal{F}_n$ . This brings us to the second important point regarding  $\hat{v}_n$ : given the systematic offset term  $b_n$  in  $\hat{v}_n$ , the latter should not be seen as the output of a gradient oracle for  $v(X_n)$ . In particular,  $b_n$  is intended to capture possible corrective terms, deviations from the exponential mapping, different algorithmic update structures (such as optimism), etc. We make this distinction precise below.
3.2. Specific algorithms and examples. In the series of examples that follow, we will assume that the optimizer can access  $f$  via a stochastic first-order oracle (SFO) returning noisy gradients of  $f$  at the evaluation point. Formally, following Nesterov [50], an SFO is a black-box mechanism which, when queried at  $x \in \mathcal{M}$ , returns a (negative) stochastic gradient of the form
$$
V (x; \theta) = v (x) + \operatorname {e r r} (x; \theta) \tag {SFO}
$$
where the seed  $\theta \in \Theta$  is a random variable taking values in some complete probability space  $\Theta$ , and  $\operatorname{err}(x; \theta)$  is an umbrella error term capturing all sources of uncertainty in the model.
The archetypal example of an SFO occurs when  $f$  is itself a stochastic expectation of the form  $f(x) = \mathbb{E}[F(x;\theta)]$  for some random function  $F\colon \mathcal{M}\times \Theta \to \mathbb{R}$  – the so-called stochastic optimization framework. In this case,  $V$  is typically given by  $V(x;\theta) = -\mathrm{grad}_xF(x;\theta)$ , so, under standard assumptions for exchanging differentiation and expectation, we have  $\mathbb{E}[V(x;\theta)] = v(x)$ . Extrapolating from this basic framework, our only assumption for the moment will be that  $\mathbb{E}[\mathrm{err}(x;\theta)] = 0$ ; for a detailed discussion of the required assumptions for (SFO), see Section 4.
In practice, (SFO) will be queried repeatedly at a sequence of states  $X_{n}$ ,  $n = 1,2,\ldots$ , with a different random seed  $\theta_{n}$  drawn i.i.d. from  $\Theta$ . In this manner, we obtain the following specific algorithms as special cases of (RRM):
Algorithm 1 (Riemannian stochastic gradient descent). Following Bonnabel [17], the Riemannian stochastic gradient descent (RSGD) algorithm queries (SFO) at  $X_{n}$  and proceeds as
$$
X _ {n + 1} = \exp_ {X _ {n}} \left(\gamma_ {n} V \left(X _ {n}; \theta_ {n}\right)\right). \tag {RSGD}
$$
As such, (RSGD) can be seen as an RRM scheme with  $\hat{v}_n = V(X_n;\theta_n)$  or, equivalently,  $U_{n} =$  err  $(X_{n};\theta_{n})$  and  $b_{n} = 0$
A key factor limiting the applicability of (RSGD) is that the exponential map  $\exp_{X_n}(\cdot)$  could be prohibitively expensive to compute in practice, even for relatively low-dimensional manifolds. On that account, a popular alternative to (RSGD) is to employ a retraction map [2, 19], that is, a smooth mapping  $\mathcal{R}\colon \mathcal{T}\mathcal{M}\to \mathcal{M}$  that agrees with the exponential map up to first order, namely
$$
\mathcal {R} _ {x} (0) = x \quad \text {a n d} \quad \left. \frac {d}{d t} \right| _ {t = 0} \mathcal {R} _ {x} (t z) = z \quad \text {f o r a l l} (x, z) \in \mathcal {T M}. \tag {Rtr}
$$
With this machinery in hand, we obtain the following retraction-based variant of (RSGD):
Algorithm 2 (Retraction-based stochastic gradient descent). By replacing the exponential map in (RSGD) with a retraction, we obtain the retraction-based stochastic gradient descent scheme
$$
X _ {n + 1} = \mathcal {R} _ {X _ {n}} \left(\gamma_ {n} V \left(X _ {n}; \theta_ {n}\right)\right). \tag {Rtr-SGD}
$$
This algorithm does not seem immediately related to the RRM template – and, indeed, the whole point of introducing a retraction was to get rid of the exponential map in (RRM). The expressive power of (RRM) can be seen in the fact that, despite this apparent disconnect, (Rtr-SGD) can still be expressed as a special case of (RRM) in a fairly straightforward fashion.
To do so, define the "forward-backward" gradient mapping
$$
\hat {v} _ {n} := \frac {1}{\gamma_ {n}} \log_ {X _ {n}} \left(\mathcal {R} _ {X _ {n}} \left(\gamma_ {n} V \left(X _ {n}; \theta_ {n}\right)\right)\right) \tag {5}
$$
with the proviso that the Riemannian logarithm in (5) is well-defined (we discuss the conditions under which this holds later in the paper). Under this definition, (Rtr-SGD) can be recast as a special case of (RRM) by running the latter with the surrogate gradient sequence  $\hat{v}_n$  of Eq. (5). To streamline our presentation, we defer the discussion about the inherent error  $W_{n} = \hat{v}_{n} - v(X_{n})$  to Appendix A.
As we mentioned before, retraction-based algorithms typically exhibit significantly lower per-iteration complexity compared to geodesic methods, resulting in their remarkable success in practical machine learning applications [2, 19]. In addition, as we show below, the use of a retraction mapping allows us to provide a unified perspective for several classical algorithms which, at first sight, might seem completely unrelated. An important example is provided by the (stochastic) mirror descent (MD) family of algorithms [49]:
Algorithm 3 (Stochastic mirror descent). Let  $\mathcal{M}$  be an open convex subset of  $\mathbb{R}^M$  and let  $h\colon \mathcal{M}\to \mathbb{R}$  be a  $C^2$ -smooth, strongly convex Legendre function on  $\mathcal{M}$ , that is,  $\| \nabla h(x)\| \rightarrow \infty$  whenever  $x\rightarrow \operatorname {bd}(\mathcal{M})$  [cf. 60, Chap. 26]. Then, the stochastic mirror descent (SMD) algorithm unfolds as
$$
X _ {n + 1} = \mathcal {P} _ {X _ {n}} \left(\gamma_ {n} V \left(X _ {n}; \theta_ {n}\right)\right) \tag {SMD}
$$
where  $V(X_{n};\theta_{n})$  is the output of an SFO query for  $\nabla f(X_n)$  at  $X_{n}$ , and  $\mathcal{P} \colon \mathcal{M} \times \mathbb{R}^{M} \to \mathcal{M}$  is the so-called prox-mapping associated to  $h$  [7-9, 31], viz.
$$
\mathcal {P} _ {x} (y) = \arg \max  _ {x ^ {\prime} \in \mathcal {M}} \left\{\left\langle \nabla h (x) + y, x ^ {\prime} \right\rangle - h \left(x ^ {\prime}\right) \right\} \quad \text {f o r a l l} x \in \mathcal {M}, y \in \mathbb {R} ^ {M}. \tag {6}
$$
where  $\langle \cdot ,\cdot \rangle$  stands for the ordinary Euclidean inner product in  $\mathbb{R}^M$
Now, even though the notation in (SMD) is reminiscent of (Rtr-SGD), the definition (6) of  $\mathcal{P}$  does not bear any resemblance to a geodesic exponential or a retraction – and, indeed, its origins are starkly different. However, as we show below,  $\mathcal{P}$  can indeed be seen as a retraction relative to a specific Riemannian structure on  $\mathcal{M}$ , the Hessian Riemannian (HR) metric associated to  $h$  [3, 16, 24, 45].
To make this precise, the first step is to note that the basic recursive structure  $x^{+} = \mathcal{P}_{x}(y)$  of (SMD) can be rewritten as
$$
x ^ {+} = \mathcal {P} _ {x} (y) = \nabla h ^ {*} (\nabla h (x) + y) \tag {7}
$$
where  $h^{*}(y) = \max_{x\in \mathcal{M}}\{\langle y,x\rangle -h(x)\}$  denotes the convex conjugate of  $h$ , and we have used Danskin's theorem [62] to write  $\arg \max_{x\in \mathcal{M}}\{\langle y,x\rangle -h(x)\} = \nabla h^{*}(y)$ . Then, if we endow  $\mathcal{M}$  with the Hessian Riemannian metric  $g(x) = \nabla^2 h(x)$ , the Riemannian gradient of  $f$  relative to  $g$  becomes  $\operatorname{grad}f(x) = [\nabla^2 h(x)]^{-1}\nabla f(x)$ ; more generally, given a cotangent (dual) vector  $y$  to  $\mathcal{M}$  at  $x$ , the corresponding tangent (primal) vector will be  $z = g(x)^{-1}y = [\nabla^2 h(x)]^{-1}y$ . In view of this, by inverting the relation  $z = g(x)^{-1}y$ , the abstract mirror descent recursion (7) can be rewritten as
$$
x ^ {+} = \mathcal {R} _ {x} (z) := \mathcal {P} _ {x} (g (x) z). \tag {8}
$$
Now, to proceed, consider the curve
$$
\gamma (t) = \mathcal {R} _ {x} (t z) = \mathcal {P} _ {x} (t g (x) z) = \nabla h ^ {*} (\nabla h (x) + t g (x) z), \tag {9}
$$
so, by definition,  $\gamma (0) = x$ . In addition, by a direct differentiation, we readily obtain
$$
\dot {\gamma} (0) = \nabla^ {2} h ^ {*} (\nabla h (x)) g (x) z = z \tag {10}
$$
where we used the standard identity  $\nabla^2 h^* (\nabla h(x)) = [\nabla^2 h(x)]^{-1}$  [9, 61]. This shows that the map  $\mathcal{R}_x(z) = \mathcal{P}_x(g(x)z)$  is, in fact, a retraction, so (SMD) is a special case of (Rtr-SGD) - and hence, of the general stochastic approximation template (RRM).
Remark. Even though elements of the above ideas are implicit in previous works on mirror descent and Hessian Riemannian metrics [3, 5, 16, 58, 68], to the best of our knowledge, this is the first time that (SMD) is formalized as a retraction-based (Hessian) Riemannian scheme.
Algorithm 4 (Riemannian optimistic gradient). Moving forward, an important algorithm for solving online optimization problems and games is the so-called optimistic gradient method - originally pioneered by Popov [56] and subsequently popularized by Rakhlin & Sridharan [57]. In the Euclidean case, this method introduces an interim, "optimistic" correction to gradient dynamics and updates as
$$
X _ {n} ^ {+} = X _ {n} + \gamma_ {n} V \left(X _ {n - 1} ^ {+}; \theta_ {n - 1}\right) \tag {OG}
$$
$$
X _ {n + 1} = X _ {n} + \gamma_ {n} V \left(X _ {n} ^ {+}; \theta_ {n}\right)
$$
where, as usual,  $V$  is an SFO for the (negative) gradient  $\nabla f$  of  $f$ . This idea can then be directly transported to a manifold setting [33], leading to the Riemannian optimistic gradient method
$$
X _ {n} ^ {+} = \exp_ {X _ {n}} \left(\gamma_ {n} V \left(X _ {n - 1} ^ {+}; \theta_ {n - 1}\right)\right), \tag {ROG}
$$
$$
X _ {n + 1} = \exp_ {X _ {n}} \left(\Gamma_ {X _ {n} ^ {+} \rightarrow X _ {n}} \left(\gamma_ {n} V \left(X _ {n} ^ {+}; \theta_ {n}\right)\right)\right).
$$
Importantly, the recursion (ROG) may be seen as a special case of (RRM) by setting  $\hat{v}_n = (1 / \gamma_n)\cdot$ $\Gamma_{X_n^+\to X_n}(\gamma_nV(X_n^+;\theta_n))$  or, equivalently  $U_{n} = \Gamma_{X_{n}^{+}\to X_{n}}(\mathrm{err}(X_{n}^{+};\theta_{n}))$  and  $b_{n} = \Gamma_{X_{n}^{+}\to X_{n}}(v(X_{n}^{+})) - v(X_{n})$ . We defer the details of this calculation to the Appendix A.
Algorithm 5 (Natural gradient descent). Our last example concerns the influential natural gradient descent (NGD) method of Amari [4], a stochastic optimization scheme for Euclidean spaces, but adapted to the local geometry defined by a strictly convex function  $h$ . Specifically, NGD queries an SFO and proceeds as
$$
X _ {n + 1} = X _ {n} - \gamma_ {n} (\operatorname {g r a d} f (X _ {n}) + \operatorname {e r r} (X _ {n}; \theta_ {n})) \tag {NGD}
$$
where  $\operatorname{grad}f(x) \coloneqq [\nabla^2 h(x)]^{-1}\nabla f(x)$  denotes the Riemannian gradient of  $f$  relative to Hessian Riemannian metric  $g(x) = \nabla h^2 (x)$  on  $\mathbb{R}^M$ . It is well known that (NGD) can be seen as a retraction-based Riemannian scheme [17], and may thus be integrated directly within the framework of (RRM); we defer the details to Appendix A. Importantly, (NGD) also includes the celebrated natural policy gradient [32] which plays an important role in reinforcement learning.