Nonconvex-nonconcave min-max optimization on Riemannian manifolds
Authors: Andi Han
Bamdev Mishra
Pratik Jawanpuria
Junbin Gao
Venue: Accepted by TMLR ()

Abstract: This work studies nonconvex-nonconcave min-max problems on Riemannian manifolds. We first characterize the local optimality of nonconvex-nonconcave problems on manifolds with a generalized notion of local minimax points. We then define the stability and convergence criteria of dynamical systems on manifolds and provide necessary and sufficient conditions of strictly stable equilibrium points for both continuous and discrete dynamics. Additionally, we propose several novel second-order methods on manifolds that provably converge to local minimax points asymptotically. We validate the empirical benefits of the proposed methods with extensive experiments.

Published in Transactions on Machine Learning Research (08/2023)
Nonconvex-nonconcave min-max optimization on Riemannian manifolds
Andi Han University of Sydney
andi.han Banddeyyyeeeeee edu.
Bamdev Mishra Microsoft India
bemmdemm@mmicrroofff..comm
Pratik Jawanpuria Microsoft India
Junbin Gao University of Sydney
junbin.gao www.neyyaasyyy.oooomoommmmmmmm edu.
Reviewed on OpenReview: https: // opw.r/veeee..eeeee/em/o net/ forum? id- EDVIHPZNFo
Abstract
This work studies nonconvex-moncconaa min-max problems Riemannian manifolds. We first characterize the local optimality of nonconvex-nonconcar problems on manifolds with a generalized notion of local minimax points. We then define the stability and convergence criteria of dynamical systems on manifolds and provide necessary and sufficient conditions of strictly stable equilibrium points for both continuous and discrete dynamics. Additionally, we propose several novel second-order methods on manifolds that provably converge local minimax points asymptotically. We validate the empirical benefits of the proposed methods with extensive experiments.
1 Introduction
Min-max optimization has been the central focus in variety of machine learning applications, including generative adversarial networks (GANs) (Goodfellow et al.
2020). adversarial training (Madry et al. 2017).
optimal transport (Lin et al. 2020a Jawanpuria et al. 2021
low-rank matrix/ matrix/tensor tensor completion (Jawanpuria
& Mishra 2018 Nimishakavi al.
2018), robust learning [Fl Ghaoui Lebret 1997), non-linear feature
learning [Rakotomamonjy
al. 2008 Aflalo et al., 2011
Jawanpuria et al.] 2011 2015b), online learning Jawanpuria al. 2015a fair inference When the objective function is
Bubeck et al. 2012). multi-task learning (Jawanpuria & Nath 2012
Madras et al.] 2018), and reinforcement learning (Busoniu et al. 2008
convex-concave, the well-known minimax theorem (Neumann 1928 Sion 1958 guarantees the existence of a
global (Nash) saddle point and there exist algorithms that converge to such global solutions Tsengg 1995
Nemirovski 2004 Mokhtari et al. 2020a).
Despite the noteworthy progress solving convex-coneave min-max problems, many real-life applications do not have the convex-concave structure. For general nonconvex-noncooeevee min-max problems, even the proper notions of local optimality are critically debated. Under such settings, the order of optimization matters, c.g., in GANs and adversarial training, and this renders the notion of Nash saddle points improperly. The recent work by Jin et al. 2020 introduces the notion of (local) minimax points, which better suits the sequential nature of the problems and bears a close relation to the stable limiting points of the gradient. descent ascent dynamics (with timescale separation).
Another research direction on -ononnee-nooccocaeeee optimization is to formulate min-max problems on Riemannian manifolds (Zhang et al. 2022c Jordan et al. 2022 Han et al. 2023b: Huang & Gao 2023). The

---

Published in Transactions on Machine Learning Research (08/2023)
generalized Sion's minimax theorem (Zhang et al., 2022c] allows many intractable sunconvex-noneeonawww problems to be solved efficiently on Riemannian manifolds, provided the objectives satisfy a generalized notion of convexity and concavity along geodesics on manifolds. Nevertheless, there exists several problems on manifolds without such generalized convexity or concavity structures, c.g., the robust Fréchct mean (Zhang
et al. 2022e|, projection robust. Wasserstein distance (Lin al. 2020a), and orthonormal GAN (Müler et al. 2019 problems, to name few. is, therefore, natural to ask:
how to define local optimality for general conconver-noncconcan problems Riemannian manifolds and what are algorithms that can converge to such local optimality?
We attempt to address the above questions by generalizing the notion of (local) minimax points to Riemannian manifolds and studying the behaviors of various algorithms on manifolds relating to such optimal points. To this end, our contributions are as follows.
We generalize the concepts of minimax points for defining local optimality for nononnev--ooooncaee min-max problems on Riemannian manifolds from the Euclidean space and explore various properties. We also illustrate examples of such points.
We introduce notions of stability and convergence for both continuous and discrete dynamical systems on manifolds. We then show the relation between the limiting behaviors of the Riemannian gradient descent ascent method and the generalized minimax points.
We propose several novel second-order methods and prove that they asymptotically converge to local minimax points.
We demonstrate the efficacy of the proposed algorithms against existing baselines on a variety of honconvex-noncomeave problems, including the robust Fréchet mean, robust maximum likelihood estimation, projection robust Wasserstein distance, and orthonormal GAN problems.
2
Related works
In the Euclidean space, most studies focus on (stroolyyooneexxtttonyyyoyoooo -coonage set settings Nemirovski
2004 Mokhtari et al. 2020a b Golowich et al. 2020 Yoon & Ryu 2021 Zhang et al. 2022b However, without the convex-concave structure, the problem is in general intractable. Thus, efforts have been devoted to solving structured
problems, including assuming (strong)-concaviity in the max-variable Lin et al. 2020b c), imposing variational inequality conditions on the min-max al.] 2021 Lee & Kim 2021), and assuming a PL-like Abernethy et al. 2019). For general sanconvex-nonconent settings, existing works have defined various notions of local optimality, such local Nash saddle points (Daskalakis &
(Tekumparampil el al. 2019
gradient operator (Song al.
2020 Diakonikolas
condition (Nouiched et al. 2019
Panageas 2018 Adolphs et al.
Mazumdar et al. 2020). and local minimax points (Jin et al. 2020 Zhang et al. 2022a), and have analyzed (local) convergence of algorithms to such local solutions (Daskalakis
2019
& Panageas 2018 Wang et al.
2019 Zhang et al.] 2020 Fiez et al.) 2020 Gao et al. 2022 Zhang et al,
2022a The work DDasallakis et al. 2021) studies the complexity of identifying and solving constrained
nonconvex nonconcave problems. Daskalakis et al. (2021), in particular, focus on approximate local Nash saddle points and linear convex constraints.
On Riemannian manifolds, min-max problems have attracted increasing attention. Zhang et al. 2022c generalize Sion's minimax thcorem to Riemannian manifolds and propose Riemannian extra-gradient method (namely RCEG) for of both Riemannian gradient descent ascent (RGDA) and RCEG for geodesic- geodesic-(strongly isstroglyy-conveene soonve--eeddsscc. sarrongly--concat scttings. Han et al. (2023b) extend the Hamiltonian gradient methods to Riemannian min-max optimization and show convergence under the Riemannian PL condition on a proxy function. Huang & Gao (2023): Wu et al. (2023) consider the (stronoyyoooonyy min-max problems manifolds.
problems. Jordan et. al.) (2022) complete the analysis
2

---

Published in Transactions on Machine Learning Research (08/2023)
3 Preliminaries and notations
This section provides a summary of concepts we require in this work with a focus on notations. We defer the detailed introduction to Appendix B
A Riemannian manifold M is a manifold with a Riemannian metric on each tangent space T.M, V2 e M, i.c., (u,u)) u)= for any u,v E T.M. The induced norm denoted as || lla. We also use TM to denote the langent bundle, the disjoint union of all langent spaces. The geodesics 7: [0, + M generalizes the straight line segment, and the exponentiol map Exp; T.M + .M generalizes the vector addition in Euclidean space. Riemannian distance dist(=1, 22) is the length of the shortest geodesic connecting 2 22 E .M. The (smooth) inverse of the exponential map (when exists), namely the logarithm map, is denoted as Log., :M + Tr,M. Parallel transport #i : TT2 - M transports tangent vectors between tangent spaces. The covariant derivative induced from the Riemannian connection is denoted as V, which allows differentiation of vector fields on manifolds.
A Riemannian product manifold M = MM x My admits a Riemannian manifold structure with induced Riemannian metric from My. For a linear operator H on T.M, can decompose its operation on H [u] Hzy[o] where Haz TyMs. + H [u] + H [u
HED Hey
{ = (u, E TaMa x TyMy as H(S) TzMa; Hyy TyMy + TyyMy Hry TyMy + TTMM, linear operator G between tangent spaces, we denote G as its adjoint operator.
Hyz
Hyy
Hys : TrMM + My are linear operators. For a
For bifunction f Me MM R, we denote VaJ(I, y). (,,( as the Euclideon (partial) gradient and gradzf(x, v), graddd as the Riemannian (partial) gradient. Furthermore, be the Rieman- nian cross-derivatives defined as gradzy/ v)lu D w)[u] for any E TyMM and similarly for grad (a., y). Geodesic (strong) convenity is a generalized notion of convexity along geodesics on mani- folds. A function is called geodesic linear if it is both geodesic convex and geodesic concave. A bifunction fz. y) is called geodesic (strongly) convex (strongly) concave if f is geodesic (strongly) convex in and geodesic (strongly) concave in y-
let
Hess y), H (x,,) y) be the Riemannian Hessian and f graddy fxx g
Other notations. We use a Abb: = min(a, bj and a avb:= ma bj for a, E R. We use [m] == (1, (1,..,,. n for n E Z+, the set of positive integers. Suppose A E [n] are real eigenvalues of a self-adjoint operator. We order the eigenvalues as a 2 d2 2 .... 2 An. We denote || -112. |l || as the Euclidean norm and the Frobenius norm, respectivcly. We denote logm() (.) as the principal matrix logarithm and define log det(-) log(det(-)).
4 Minimax point for nonconvex-noncooncaw min-max problems on manifolds
We consider the min-max problems on Riemannian manifolds as
min max f v). M NEM,
where f is at least twice differentiable and in general nonconccve in both Euclidean and geodesic senses. Under such settings, the generalized Sion's minimax theorem does not hold (Zhang et al. (022c), and the order of optimization matters. Without loss of generality, we assumee minimization takes place first, followed by maximization.
Recently, Zhang et al. (2022c): Jordan et al. (2022) have studied global saddle points on manifolds. Below, the classic definitions of global and local Nash saddle points are generalized from the Euclidean space to general manifolds.
Definition (Global and local Nash saddle point). A point (a*, (x","") is called a global Nash saddle point if, for any (z,y) E X x V, we have f(x".y) S f(x","" S f(),y"). A point (x*,y") is called a local Nash saddle point there exists e <> such that for any (x, satisfying dist(x,a*)= S c, dist v*) S c, we have [(" .y) & f(* , y²) < (xx
The conditions of optimality for local saddle points on manifolds are similar those in Fuclidean space.
3

---

Published in Transactions on Machine Learning Research (08/2023)
Proposition 1 (First-order and second-order conditions of local Nash saddle points). If point (x*, (x*) is local Nash saddle point of a bwice continuously differentiable function J, then it salisfies (1) grad ,x",1") 0, grady f ,y²)= =0, 0, and (2) fess, .3" = 0 and fess, ,y²) 0. (x""(""")))))))""""""")))))""")_
As shown in Jin et al. (020), the Nash saddle point, be it local or global, may not exist in general for snncoveex-nonconcen functions. Hence, Jin et al.] (2020) have proposed the notion of global and local minimar points in the Euclidean setting. In the following, we generalize it to Riemannian manifolds.
Definition 2 (Global minimax point). Consider two subsets X' x y MM x My, (x", ) is called global minimaz point of f if, for any (z,y) E X' x V' it satisfies f(z", ,) S f(x="=") S maxy/ey f(x, v/). Equivalently, (x",y") satisfies y² = arg maxyey (xx,,) and x² = arg minzex ((x)), where <(x) == maxxey faz y).
Definition (Local minimax point). A point (z", is called a local minimaz point of f: X' x + R if there exists a constant óg and a function h satistying lims- h() = for any 8 E (0, 5o) and any (a. v) N XxX satisfying (ast, x²) S 5, disstyy.) y²) 6, we have f(x",)) v) S f=x", S maxy" i (sh())) [(x, y")- Proposition 2 (Equivalent characterization of local minimaxily). Suppose J is condinnous on X x y. A point (x is local minimar point if and only if y" is a local marimum of function f(x",8) ,2) and there
)
exists a 5g > 0 such that x² is a local minimum of 95(2)) = maxy' wn 158 f(z, v') for any ō E (0, 50)-
A game-thcoretic intuition for minimax points is that the max-player always maximizes its payoff based on the observation of the min-player's action, while the min-player chooses an overall optimal strategy without knowing the action of the max-player. In particular, (Jin et al. 2020 Proposition 11) show that there always exists global minimax point due to the extreme value theorem on compact domain, unlike the case for Nash saddle points. However, the local minimax point may not exist. In addition, any local Nash saddle point is a local minimax point.
Next, we provide first-order and second-order characterizations of local minimaxity on general Riemannian manifolds, where we make use of the Riemannian Hessian and cross derivatviv operators on tangent spaces.
Proposition 3 (First-order and second-order conditions). Suppose f is continuous and at least twicc differentiable and is (x", ,yy) is local minimaz point, then il salisfies grad_J(x", v²) y²)=00 0, grad,J(x", .v") = O. In addition, (x". .y" satisfics Hess,y(x" ,y") 3 0. If further f < 0, then H fx" - gyad ,y²) o [Hess,, ,,"''=""== grad fr fx=
For completeness, we show the sufficient first-order and socond-order conditions for local Nash and minimax saddle points in Proposition [5 (Appendix). We now discuss several examples to elucidate the concepts of local minimax points on Riemannian manifolds.
Proposition 4 (Examples of local minimax points). Let $4 == (X E Rdxdd X = -XX > 0} be the symmetric positive definite (SPD) manifold endowed with the affine-invariant metric (detailed in Appendir C.2 Consider X, Y E 34 and define the set S = {A E det(A))
(1) Any (X+, Y-) E St x St is global Nash saddle point and hence a local minimax point to function f(X, Y) c (log det (x)) o2 log log det( cs(log det(Y))? for any C1, 0. (2) Any (X", YY 5* S is local minimax point but not local Nash saddle point to function f(X, Y) = (log det (2 2 + C2 log de) X log det( ca(log det( for any 01,02,03 c3 > 0 such
that 02 2 2//0.
(3) Any (X* Y S x [I] is local minimaz point but not local Nash saddle point to function f(x.1 Y) = -c (log det (2) + e2 log det(X) det - c3 [|]loom())l? for any C1, C2 cs > 0 such
that 02 N Vd
2/10
Proposition makes use of "quadratic" functions (log dc) and logmm I| on the SPD manifold. (log de)( is quadratic due to geodesic linearity of log det(-) (Vishnoi) 2018). [|logi(Y)) is equivalent to dist? (I, Y), i.e. the squared Riemannian distance. Hence, the functions are analogs of quadratic functions in Euclidcan space. However, we highlight that the example functions of the form (2) in Proposition are geodesic concave but not necessarily strongly concave in Y. unlike the quadratic functions in Euclidean space. A consequence is that the local minimax point is not strict and there exists a continuum of such points.
4

---

Published in Transactions on Machine Learning Research (08/2023)
5 Stability and convergence of dynamics on manifolds
In this section, we introduce the notions of stability and asymptotic convergence rate of both continuous and discrete dynamics on manifolds, which are essential for subsequent convergence analysis around minimax points. Given the analysis is local around an equilibrium point, we make the following standard assumption. Assumption 1. Dynamics considered in this paper are contained in neighborhood of an equilibrium point where the exponential map has smooth inverse.
Such an assumption allows the linearization of arbitrary dynamics manifolds around the equilibrium point, in similar vein as the linearization of nonlinear dynamics in Euclidean space.
5.1 Stability of continuous dynamics
We consider a continuous autonomous dynamical system *(t) (=z(t) on a Riemannian manifold M, where a(t) E M, 2(t) M represents the time-derivatives, and G M + TM is vector field. We say 2" is equilibrium point (or fixed point) if it satisfies G(=*)=0. =0
In Euclidean space, i.e., z(t) E Rd, the stability of the dynamical system can be characterized via the (real- parts) of Jacobian cigenvalues of G(z*) at equilibrium (Khalil 2002). Nevertheless, on general Riemannian manifolds, the derivative of G(z*) is not necessarily self-adjoint, and its eigenvalue/etie seeuvaalue/eggevvetto may not exist due to the lack of well-defined complex cigenvalue/cigann eigennvccor on tangent spaces.
Here, we approach the notion of stability via linearization of the dynamics and Lyapunov analysis.
Proposition 5. Consider a dynamies =(1) = G(=(1)) on M, with an equilibrium point. z". Let V(=(1) = (Log.- (z(t)), H [[Loo.-
be a Lyapunov function for some self-adjoint positive definite operator T.-M. Then, we have d (Zoz)-z((t) = "G(="))oog,
d [=())(<(t))]]] (z(t)) Llg (x(t))l==))) and de VV((t)
(Log,- (z(1), (H VG(=*) + (VG(=")) (L) + o(I[Log,- -=(()1o,,--(())])))))
From Proposition 5 we see any dynamical system be linearized around fixed point, which is sufficient for the asymptotic analysis in subsequent sections. In addition, z (=(t)) < 0 if and only if H VG(2*) + (VG(2"))fo is negative definite for some positive definite operator H. This leads to the notion of equilibrium stability on manifolds, which generalizes the classic notion of Lyapunov (asymptotic) stability in Euclidean space.
Definition 4 (Strictly stable equilibrium). Consider any (autonomous) dynamical system on a Riemannian manifold M as #(() G(=(()). An equilibrium point =" is called strietly stable if and only if H o VG(=*) + (VG(=*)) o H is negative definite for some positive definite self-adjoint operator H on T.
We next show that a strictly stable cquilibrium can be cquivalently characterized via the cigenvalues of a matrix representation of VG(=") + (VG(=*)) o H. The matrix representation obtained with respect to an orthonormal basis on tangent space. In this paper, this is also followed for any linear operator on a tangent space.
Proposition 6. if and only if the matria representation of GG(2)) in an orthonormal basis of T.-M has all real parts of eigenvalues negative.
For a dynamical system ε(l) = G(z(()) on a manifold M, an equilibrium x" is strictly stable
Remark 1. Proposition is mainly due the energy preservation under an orthonormal coordinate transform. We highlight that this result independent of the choice of orthonormal basis because, as shown in the proof, the change-of-basis transformation forms an isomorphism between the dynamics under different. coordinate systems and the stability can be studied under any basis.
f
5.2 Stability of discrete dynamics
In this section, we consider the asymptotic stability of a discrete dynamical system on manifolds, namely 2k+1 Expsy (n) G(zz)) for some stepsize 17 > with a fixed point x* satisfying G(²*) 0. This corresponds to the geometric Fuler discretization of the continuous dynamics ²(l) G(=(()) discussed in the previous
5

---

Published in Transactions on Machine Learning Research (08/2023)
section. The following proposition shows that similar to the continuous case, the discrete dynamics is also locally linear around fixed point.
Proposition 7. Consider a discrele dynamicaal system on M as žk+1 = Exps, (1) Gzx)) for some vj > O. Define the Lyapunov function ((z)) (Log.- (zx), ), H [og.. (z+)l) for some self aelf-adjoint positive definite operator H T.-M. Then Logg- (2k+1) = (id + 1f V)V)(=")) [Log.- (2x)] + V(z)) (Logs- (2x), ((id n/V H (id n vVG(=")) [Log. (z+))) + (++)).)))+
and ((*+11)
Hence, to ensure decrease in the Lyapunov function near the fixed point 2", require (id + VV H (id + + V=())) H to be negative definite for some positive definite H. Here, id represents the identity operator. Following similar arguments in Proposition O this is equivalent to requiring the matrix representation of id + V==)) in an orthonormal basis to have a spectral radius less than one.
Proposition 8. For discrete dynamical system 2 Exp,,, (n ((z))), fixed point z" is called strictly stable if and only if the matria representation of id + 17 G)²) has all the eigenvalues with magnitude less than one for some 1j > 0.
5.3 Stability equivalence and asymptotic convergence
The next proposition proves an equivalence between Propositions 6 and
Proposition 9. For discrete dynamics 2k+1 Exp:, 24 (1) ((+)) on monifolds, the following are equivalent.
(1) xn is a strictly stable equilibrium point.
(2) All the eigenvalues of the matrix representation of id + G)())) have magnitude lower than one for some 1) 0.
(3) All the eigenvalues of the matrin representation of G have real parts negative.
Proposition g allows us define the asymptotic convergence rate of dynamical systems on Riemannian manifolds.
Definition 5 (Asymptotic convergence rate). Consider a dynamical system œ = Exp,, (n G(zx) on M. Suppose 2" is a strictly stable fixed point. We say = max; |1 + maaa is the asymptotic convergence rate of the system for some 1) > such that [1 + 174,1 < 1 for all i, where A, are the eigenvalues of the matrix representation of VG(²") in an orthonormal basis.
Remark 2. We remark that the notions of stability and asymptotic convergence are generalizations of the counterparts in Euclidean space via coordinate representation. The notion of asymptotic convergence on manifolds follows from the lincarization in Proposition 7 where the dynamics evolve according to Log.- (2++1) around a local neighborhood of =* Such a claim formalized in the following result, where the asymptotic convergence rate related to the local linear convergence rate.
(id + (Togg)))))
Proposition 10. Consider the discrele dynamical system #k+1 = Exps, (1 G(zx)) Let == - 1-1- < be the asymptotic converyence rate of the system. Then, there erists a local neighborhood u C MM around z" such that for any 20 E U, it satisfies dist(22, 2²) S - x dist (est(22,2))) 2²)
for some constant C > 0.
In the next section, separated Riemannian gradient descent ascent (TSRGDA) method.
the above results to analyze the asymptotic convergence behavior of timescale-
5.4 Asymptotic analysis of TSRGDA
In the Euclidean setup, Jin et al. (2020) analyze the limit points of (limescale-sepuraatel) gradient descent ascent (TSGDA) and establish a relationship between stable limit points of TSGDA with local minimax point and Nash saddle points. We attempt to genoralize this analysis to the Riemannian gradient descent ascent Figure D we show a schematic view of an iterative
(RGDA) method on manifolds Huang & Gao 2023 algorithm on manifolds.
We consider timescale-soparatee RGDA (TSRGDA) with different stepsizes for descent and ascent dircctions, i.e.
(-17 gradg (Tx, 31:) [ggrad, grad (I Wk)
TSRGDA: Zb+1 = Exp,, (epp,,() (++)) = ExPax
(1)
6

---

Published in Transactions on Machine Learning Research (08/2023)
T M
Exps2 (E)
M
Figure 1: Schematic view of Riemannian optimization. direction & E M with the goal to reach z². This scheme encompasses all the proposed methods considered in this paper.
The algorithm iteratively updates & following a
where (IK, vh) and r > 0 is the separation parameter. From Proposition it is equivalent to studying the continuous version of such dynamics (1) - 0) as (() = G, (x(t))
- grad,f( fx((t,, v()()),((t)) ( ) grad,f(x(t),, v(t))
Let TSRGDA denote the set of strictly stable equilibrium points of dynamics ( with parameter т, and LNS {= (z. v) gra) 0, f() > 0, (c)),f(2) < 0}. i.c., the set of strictly local Nash saddle points and LMiniMax == {z (xr, v) grad,f(=) 0, (Hess, f Hess be the set of strictly local minimax points. The following result claims for any fixed timescale separation parameter т, there exist certain functions, where TSRGDA may converge to unmeaning points, which are neither local Nash saddle nor local minimax points.
f I
gradye ())) O, Hessy/((=) 0}
Proposition 11. For any ftred T, we have LNS C TSRGDA, while there evists some functions such that TSRGDA & LNS. Furthermore, for any fized T, there crists some functions such that LMiniMax TSRGDA and TSRGDA & LMiniMax.
The proof of Proposition II relies on Proposition where we work with matrix representation of VG, (z). It has been further shown by Jin et al.) (2020) that provided sufficiently small, the stable equilibrium points of TSRGDA follow-up work (Fiez & Ratliff (0221) further derives a non-asymptotic construction of an upper bound to T, which we extend to the Riemannian manifold settings as follows.
strictly local minimax points (up to some degenerate points with respect to Hess,f(z)). A
Proposition 12. Suppose z is non-degenerale crilical point of J, i.e., grad,I(=) 0, Hess (=) is non-degenerate. Then there exists > 0 such that for all (0, TSRGDA if and only if E LMiniMax.
In addition, the asymptotic convergence rate is derived for TSRGDA, motivated by Zhang al. 2020
Theorem 1 (Asymptotic convergence of TSRGDA). Suppose z" E LMiniMax and for some sufficiently small, x" E TSRGDA. Then the asymptotic convergence rate of TSRGDA to z" is given by max; |1 + mpll, where de)dd are the eigenvalues of (="))) in an orthonormal basis, and n min; (- ( 1/1)), with R(-) denoting the real part.
Remark 3. From (Jin et al. 2020 Lemma 40), we see when T - O, we have the eigenvalues of matrix representation of (x*) asymptotically splits and converges to {-rA Jed and {- ddy where X1 2 2 Xd, > 0, 11 ... 2 Pa, > 0 are cigenvalues of (Hess gradzy, Hess f grad and -Hess, f="), respectively. In this limiting case, we the convergence rate reduces to max; 1 777 Ail V max, |1 mu4 with 17
-2 "Ai

---

Published in Transactions on Machine Learning Research (08/2023)
6 Second-orde methods for nonconvex-nonconcave problems on Riemannian manifolds
This section proposes various second-order methods that are guaranteed converge to the local minimax point, without requiring sufficiently small stepsize (i.c., sufficiently small) for the minimization variable as in TSRGDA. We also analyze few recent Riemannian min-max algorithms proposed in the literature.
6.1 Riemannian follow-the-ridge and total gradient descent ascent methods
Riemannian follow-the-ridge. The Riemannian followethe-ridge (RFR) generalizes the idea proposed by Wang et al. (2019). Based on the definition of a strictly local minimax point (x".v") xt is a local minimum of fax (x)) where vp : X + y is an implicit function such that grad, (xr, v/xx)) 0 for any z in a neighborhood of x² (due to implicit function theorem on manifolds, i.e., Theorem 5 in Appendix). In order to ensure the condition grad, (Ix. (xxx)) = O is satisfied (approximately) at every iteration, a correction term for the y player is added as D2 (rx) (I*+1)) Rs Logn, (M++1). Specifically, each iteration starts by updating Tg as Fk+1 Exp2a (-yrgrad, (#x, 9k)) and then updates y as Mk+1 = Expys ys (n grad, 8k) (z [Log,, (1++1)]) where the latter term is computed following again the implicit function theorem on manifolds, as ((x)))) [Log), (#x+1)] = yp ([Hess, grad (zx, [grad, 2x]]. Suppose at iteration k we have grad, (ark: yk) = then we can show yk+1 = ((x+11) with grad, f (x(++))) = O.
Using similar notations as before, we write the dynamics compactly
rid
RFR: 2x41 Exps, (xp,,, ((Gre((
GRER(EK) = (*(Hss f grad ya (2x)
Riemannian total gradient descent ascent. Total gradient descent ascent Fiezz et al. 2020 is closcly
related to the follow-the-ridge update, where both methods have been shown to precondition the GDA update
with preconditioner being the transpose of each other (Zhang et al. 2020). The update of the Riemannian version is given by
10-10t (-rid 7 (grad? Hoss, (o) f)x))
RTGDA:
Zk++ = ExxP.. GHIGDA (zgggaad
GRTGDA(ZL) =
0
id
It can be seen that GRiGDA (zx) = (G)F due to that gradzy, = (grad, yzr f)f, Hence, RFR and RTGDA
share similar convergence properties as stated below.
Theorem 2 (Asymptotic convergence of RFR and RTGDA). For any T > 0, the following three statements are equivalent: (1) 2 is a strictly local mánimar point; (2) 2" is a strictly stable fired point of RFR; and (9) z¹ is strictly stable fixed point of RTGDA. Moreover, the asymptotic convergence rate of RFR and RTGDA to striclly local minimar point p max; T/rdal V max, muuh for " TALVEL where Xi, Fy are the
2
same eigenvalues in Theorem 1
we see RTGDA and RFR achieve the same (limiting) asymptotic convergence rate with an arbitrary choice of which is in general numerically favorable.
Remark 4. Compared to TSRGDA in Theorem and Remark 3
6.2 Riemannian Newton gradient descent and Newton follow-the-ridge.
The Euclidean Newton gradient descent method (Zhang et al. 2020) updates the max player with Newton step, which allows the maximization step to be solved accurately and is more favorable when the maximization problem is ill-conditioned, i.e., H has a large eigenvalue gap. The Riemanuian Newton gradient descent. (RNGD) given by
RNGD: Z6+1 Exp., ( (- ( Mk+1 = Expy,. (- n/Hess, fzz))gra,, ((x41,
where the negative sign the max variable is because the Newton step is the same for both minimization and maximization problems.
8

---

Published in Transactions on Machine Learning Research (08/2023)
Another related algorithm, known as the Newton follow-the-ridge |Gao et al. 2022) method, presents
first-order approximation to the maximization step of the Newton gradient descent. On general Rieman- nian manifolds, by Taylor approximation of gradyf(# 8k), we have grad, f yk) s gradyf((z)
yrgrad [gg]gad rollow (RNFR) as
[gradg (=x)] where substituting this approximation to RNGD yields the Riemannian Newton
RNFR: 2k+1 = Ex (unee((e))graadf(z.)))) GANFR(ZE) = (r(Hosss - rid
grad yo. ffr (2x) -CHess f f(2x))
We remark that Gao et al. (022) use Ç'id (Hess, f(z+) in place of - - y [(xx
for the max variable. Since we motivate our RNFR R from Taylor approximation, we decide to exclude Ç'id, where the subsequent analysis does not differ significantly. In addition, based on the discussion in the previous section, we propose Riemannian Newton total gradient descent ascent (RNTGDA) by simply taking the adjoint of GRNER- Specifically, let GRNTGDA (ZETGDA(Z)).........).. (CRNrR(*<)) we directly consider the update on general Riemannian manifolds as
RNTGDA: Ek+1 = Exps, (1 (1 GRNTGDA
(zx) Expx, (GRNFR(EE))) (arad f
Below, we present the convergence guarantees for RNFR and RNTGDA.
Theorem 3 (Asymptotic convergence of RNFR and RNTGDA). For any t > 0, the following three statements are equivalent: (1) 2" is strictly local minimaz point; (2) x² is strictly stable fized point of RNFR; and (3) x" is strictly stable fixed point of RNTGDA. When choosing Ç (= 1/n, the asymptotic convergence rate of RNFR and RNTGDA to a strictly local minimar point is p max; [1 - TTT where < 2 and 7;5 are the eigenvalues of (Hess-J~ grad?, Hess, o (z) (=*)
7X1
Remark 5. Compared to the convergence of RFR and RIGDA, the asymptotic convergence rate of RNER and RNTGDA docs not depend on the cigenvalues of (2*". Thus, when Hess,, is ill-conditioned, we obtain an improved convergence rate with the Newton step for the max player.
6.3 Riemannian Hamiltonian and consensus methods (RHM/RCON)
Riemannian Hamiltonian and consensus methods (Han et al. 2023b) are motivated by the minimization of gradient norm as proxy problem to the original max problem. The updates of the methods are
RHM/RCON: 2k+1 Exp2, (2x)) Exp.,
(VG(()))
(2)
where G(zx) = (f-grad_ffaa f(x) grady, (zx)) and 17 > 0,3 2.300 0. We also let H(zx)) = 3G((=) (VC())))C
Particularly the update (2b is called the Riemannian Hamiltonian method (RHM) when B = and called
the Riemannian consensus method (RCON) when 3 > 0. The mothods are shown to converge lincarly to stationary points of the min-max problem, provided the Riemannian Hamiltonian function, i.e., ||G(zz) satisfies the general Riemannian PL condition.
The below result shows that for general nonconvex-nonconees functions, RHM/RCON may converge to undesirable stable fixed points.
Proposition 13. The strictly stable fized points of RHM and RCON may contain local minima, mazima, minimax, or mazimin points.
7 Experiments
In this section, we evaluate the proposed sccond-order algorithms - RFR, RTGDA, RNGD, RNFR, and RNTGDA (proposed and analyzed in Section ( on a variety problems and compare them against the below baselines. The code is available
RGDA (Jordan al. 2022 : It first-order method with same stepsize for the min and max variables.
9

---

Published in Transactions on Machine Learning Research (08/2023)
TSRGDA (studied in Section 5.4 variant of the RGDA method but with different stepsizes for
min and max variables, i.e., timescale separated.
RCEG (Zhang et al. 2022c Jordan et al. 2022): The Riemannian corrected extra-gradient method
a first-order method where the scarch direction is modified by taking gradient from the past iterate.
RHM (Han et al. 2023b): the Riemannian Hamiltonian method with fixed stepsize.
RCON (Han et al. 2023b): the Riemannian consensus method with fixed stepsize. It based on
the RHM search direction but now combined with the gradient descent accent direction.
All our experiments are done in Matlab with the Manopt package (Boumal et al. 2014) except for the GAN experiments (Section 7.4), where we use the Geoopt package with Pytorch (Kochurov al.) 2020). For the proposed sccond-order methods, the Hessian inverse is computed via conjugate gradient tangent space (Boumal 2020 Algorithm 6.2), with practical implementation details given in Appendix C For all the experiments, stop the algorithms either when the optimality gap (or gradient norm wherever applicable) falls below a threshold or the maximum iteration count is reached. The experiments in Section 7.1 show that RTGDA and RNTGDA empirically perform similarly to RFR and RNFR, respectively (this also follows from the theoretical analysis Sections 6.1 and (.2). Therefore, for all other subsequent experiments, we remove RTGDA and RNTGDA from our comparisons.
7.1 Quadratic optimization
We evaluate the algorithms on the functions discussed in Proposition 4
Logdet quadratic problem. The first example is the following quadratic optimization problem with the log-determinant (logdet) function, i.e.,
min max (logddet(X))) + ca log det log det(Y) + cg deo (Y))² XESt YES
(3)
for C1, 02, C3 E TR. We consider both noncomvex-eenbesiccoooe as well as geodesii-coovvx--cnnee settings, i.e,, C1 = -1, = 5, = -1 and с₁ = I, c2 = 10, c3 = -1. We respectively label the two problems as and geodesic convex-concave logdet quadratic
LD:NCGC and LD: GCC, representing manconvex-geodesiocooee problems. The Riemannian gradient and second-order derivatives for the problem are derived in Lemma
From Proposition 4
that the local minimax points (X', Y*) of problem LD:NCGC satisfy det(X")
deu(Y*) 1. For the problem GCC, the global saddle points are shown to also satisly the same conditions as in (Han et al. 2023b Proposition 8). Hence for both problems, we use the same criterion det(X) - 1 +
(Yc for measuring the optimality gap as in Han et al. (003b)).
Riemannian distance quadratic problem. We now consider a modification to where (log det (Y))² is replaced with d1st? Y), i.e.,
min max (log ( 02 det (X) log cadist Y), X Xead Yead
(4)
where we recall the Riemannian distance on dd with the affine-invariant metric is dist" B) [llogm(A (aggm(A-1/BAA -1/3)/- For this problem instance, we consider the setting 6 = 11 03 = -1, which we label as RD: :NCCC which is a nonconve--oodsiiccssrroooo gradient and second-order derivatives are given in Lemma 5 From Proposition 4 we the local minimax
problem. The Riemannian d
points "X,Y") satisfy det(X*) 1, Y = We use det X 1| + dist? (Y*, I)) to measure the optimality gap.
Experiment setup and results. For experiments, we consider d = 30 and lune slepsizes for all the methods. For RCON, we also tune y after fixing the same stepsize as RHM. We fix Ç = for RNER and RNTGDA. The results are plotted in Figure [ where we show the optimality gap as function of both iteration counts and time. For the RD: NCGC problem, we also plot the gradient norm against time. Examining the plots, we make the following observations.
10

---

Published in Transactions on Machine Learning Research (08/2023)
REDA REES RAN
MPR 3850 38'664
Optinally goi
RMG0 RMTOOM
Optimally gap
Optimallly gap
RCEG BOW
1550
Optimallty gep
R3G0
1078
RAM
RAT0OM
Iteration
Time
Iteration
Time
(a)
LD:GCC (optgap-iter)
(b) LD GCC (optgap-time)
(c) LD:NCGC (optgap-iter)
(d) LD: LD:NCGC (optgap-time)
20DA
Optimality gap
4000
Optimality gap
KCED
8000
Gradnom
Iteration
Time
Time
(e) RD: NCGC (optgap-iter)
(f) RD:NCGC (optgap-time) (g) RD:NCGC (gradnorm-time)
Figure 2: Experiments on the quadratic optimization problems
3 and (4) on the SPD manifold. We observe
faster convergence proposed second-order methods against the benchmarks particularly for nonconvex- nonconcave settings (LD:NCGC, RD: RD:NCGC).
For all the problem instances, our proposed second-order methods perform competitively among the baselines. They perform the best in terms of both iteration and runtime for the two nonconvex- geodesic-concave problems NCD: :D:NCGC).
We observe very similar convergence behavior of RFR and RTGDA well as RNFR and RNTGDA.
This empirical observation matches the theoretical analysis as they share the same preconditioner
(up to transpose) and have the same asymptotic convergence rate.
For the geodesic convex-concave problem (LD: Figure 2a), we see RHM/RCON outperforms other methods in terms of the number of iterations. As the proxy problem for RHM/RCON satisfies the Riemannian PL condition as demonstrated in (Han et al. 2023b Proposition 8), RHM and
RCON have better convergence rates.
Among the methods that use first-order information, we observe that TSRGDA outperforms RGDA and RCEG. This suggests using different stepsizes for minimization and maximization updates is essential to improve the convergence.
7.2 Robust Fréchet mean
The robust Fréchet mean (RFM) computation problem on the SPD manifold (Zhang et al., 2022c Han el al., 2023b) can be formulated as the task of finding the Fréchet mean (of a set of SPD matrices) with eigenvalues bounded away from zero, i.e.
min max x Mx dist (M, M,M,), MES XES*
(5)
where 0 is the regularization parameter and Sd fx Rd xx 1} is the sphere manifold. This problem is geodesic strongly convex in M and in general nonconcave for x. We derive the Riemannian gradient and second-order derivatives in Lemma 6 We use the gradient norm as a measure of distance to optimality.
Experiment setup and results. We follow the experimental sotting in Zhang et al. (022c) Han al. (203b)) for generating the sample SPD matrices with bounded eigenvalues in Ag, An). We choose d 50, 40, An 0.2, λ₁ 4.5, a 0.1 as in Han et al. (023bb
11

---

Published in Transactions on Machine Learning Research (08/2023)
1GDA 2C16
RODA REGS RCCON RER
RGDA
RCEG 1nk
2C06
Gradromm
RC35 378
Graddorm
Gradnomm
Gradnamm
RCD6
523 1R-R
276.
2473
2NTA
20
Iteration
Time
ieeraioo
Time
(a) RFM (Gradnorm-iter)
(b) RFM (Gradnorm-time)
(c) RMLE (Gradnorm-iter)
(d) RMLE (Gradnorm-time)
Figure 3: Experiments on the robust Fréchct mcan (RFM) and robust maximum likelihood cstimation (RMLE)
problems. For RFM, we observe that RNGD and RNFR achieve the fastest convergence. The other converging method RHM, although its convergence is slow, All other baselines converge to undesired points that are not stationary. For RMLE, the only two converging algorithms are RFR and RNFR, while first-order methods (RGDA, TSRGDA, and RCEG), RNGD, and RHM/RCON either do not converge converge slowly.
From Figures 3a and 3b we see that RNGD and RNFR outperform others both in number of iterations required and runtime. RHM converges as well, but slowly. RFR and all other first-order methods fail converge suggesting that the Newton step provides better-conditioned maximization update.
7.3 Robust maximum likelihood estimation
The robust maximum likelihood estimation problem (Bertsimas & Nohadani 2019 Zhang al., 2022ch is formulated follows. Given observed data matrix X E Raxd where n, d,d respectively the number of samples and feature dimension, the objective is
nd (og(2x) log det(E)
1 n
min max BES" PERd DESt
- xx: 5 E 'x; 5 p1)
(6)
where we choose the constraint set for the perturbation as a sphere, iee, the perturbed samples are unit-
distance away from the original ones. We adopt the reformulation (Hosseini & Sra 2020 Theorem 1)
transform problem (6) to be geodesic concave in the maximization variable. Tf y X 1 Rd-1 for all
be the augmented data, then equivalent problem to 6 is
max log det(() 8 Seat+1
min
y: S - (y: 8).
where we ignore the constants. Although the problem is geodesic concave in S, it is in general nonconvex in 8. The Riemannian gradient and second-order derivatives are derived in Lemma
Experiment setup and results. We use the fisheriris dataset from Matlab, which consists of n 150 samples in d 4 dimension. We augment the given data matrix with column of ones and solve the reformulated problem 7 We observe in Figures 3c 3d that RFR and RNFR converge faster, whereas others
either do not converge converge slowly.
7.4 Orthonormal generative adversarial networks
Lastly, we evaluate the proposed second-order methods on the application of generative adversarial network (GAN), which optimizes a game between generator and discriminator network. In particular, we consider the orthonormal GAN (Brock et al. 2019 Müller et al. 2019), where the weights of the discriminator are constrained to be orthonormal. Such a constraint has shown to improve training stability and generalization capability (Müller et al. 2019). To this end, we formulate the objective of the orthonormal GAN (OGAN) problem as
min (W2) WW n
max
Σ Am1
log (o(D(x:))) + log (1 (D-O(D((())))))
12

---

Published in Transactions on Machine Learning Research (08/2023)
(a) TSRGDA
(b) RCEG
(c) RHM
(d) RCON
(e) RFR
(f) RNGD
(g) RNFR
Figure 4: OGAN on a mixture of Gaussians. The top row plots samples generated from the generator and the bottom row plots prediction from the discriminator on the entire 2d domain where the pixel intensity represents the predicted probability. Although all the algorithms achieve good sample quality for the generator, only the discriminator optimized by RNGD and RNFR is fooled completely by the generator, suggesting that the Newton step is crucial for stabilizing training.
10'
10h
10-1
10-1
Gradnorm
Gradnorm
30-2
20
20:1
20
TSRGDA
RFR
TSAGOA
10m-
RCEG
RNGD
30-
RCEG
RNGD
10-5
10-3
RCON
RCON
10-4
10a4
0
10000
2000
3000
4000
5000
50
160
150
200
Iteration
Time
(a) OGAN )tradnorm-iter)
(b) OGAN (Gradnorm-time)
Figure 5: Experiments on orthonormal generative adversarial network (DGAN) on mixture of Gaussians where we see RNGD and RNFR converge within a few iterations, while the others fail to converge.
where we denote (()) as the sigmoid function, as the input samples, 2, as the input noise, usually following a Gaussian distribution, We also use D and G to represent the discriminator and generator networks, respectively. The weights of I-th layer of generator and Ê-the layer of discriminator are denoted as WC, wP, and rtd (with r S d) represents the set of column orthonormal matrices (i.e., the Stiefel manifold).
Experiment setup and results. We consider the mixture of Gaussians example as in Zhang et al. (2020) and use the same network architecture where both the generator and discriminator have 3 hidden layers, each with 256 units and ReLU activation. The dimension of the prior is 100 and the batch size is set as 512. Following Han et al. 2023b), we parameterize the orthonormal weights only for the second last layer of the discriminator. To examine the local convergence, we use the pre-trained network parameters in Zhhng et al. (020) (after projecting to the manifold) as initialization. For RFR, RNGD, and RNFR, we use 20 iterations for the conjugate gradient method (that is used compute the Hessian inverse). We fix the stepsize for the generator be 0.001 for all methods and tune the stepsize of the discriminator. We set the maximum number of iterations 5000 for RGDA, RCEG, RHM, and RCON, 200 for RFR, and 100 for RNGD and RNER.
Figures 5a and 5b show the gradient norm in terms of both itcration and runtime, respectively, averaged over five random runs. We observe that RNGD and RNFR converge within a few ilerations while other methods do not. The significant convergence speed justifies the additional per-iteration cost in computing the Hessian inverse. We further plot the generated samples and discriminator prediction Figure [ We notice that despite generating high-quality samples by all methods, only RNGD and RNFR trick the discriminator predict uniformly almost everywhere.
13

---

Published in Transactions on Machine Learning Research (08/2023)
8 Concluding remarks
This work studies noncovee--nonconeave min-max Riemannian optimization and introduces the notion of minimax points for defining local optimality on manifokds. It generalizes the analysis from the Euclidean space. We also quantify the stability and asymptotic convergence for dynamics on manifolds that allow studying the limiting behaviors of various existing algorithms under the nonconvex-nonconee As an additional novel contribution, we propose class of second-order methods with provable asymptotic convergence local minimax points. The experiments show the merits of the proposed methods. In partieular, for many nonconvex nonconcave problems robust Fréchet mean, robust maximum likelihood estimation, and orthonormal generative adversarial networks - the only methods converging to the local minimax are the proposed second-order methods.
regimes.
nonconvve-nonccoeewwv
To the best our knowledge, this is the first work that systematically studies nonconvex now.oonccve min-max optimization problems on manifolds. However, there remain many open challenges and questions. One question is whether con-asymptotic convergence to local minimax points can be shown for the methods proposed manifolds. Convergence under stochastic settings there exist various other notions of local optimality for min-max problems in Euclidean space. It would be interesting to study those on manifolds and verify the (nov-eoonvereence of existing methods to such optimality. Similarly, developing acceleration schemes for solving min-max optimization on manifolds is a challenging but interesting research problem.
be also interesting direction. Furthermore,
References
Jacob Abernethy, Kevin A Lai, and Andre Wibisono. Last-iterate convergence rates for min-max optimization. arv: 100,007000 2019.
P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization algorithms on matrix manifolds. In Oplimizalion Algorithms on Matria Manifolds. Princeton University Press, 2009.
Leonard Adolphs, Hadi Daneshmand, Aurelien Lucchi, and Thomas Hofmann. Local saddle point optimization: A curvature exploitation approach. In International Conference on Artificial Intelligence and Statistics, pp. 186-495. PMLR, 2019.
Jonathan Aflalo, Aharon Ben-Tal Tal, Chiranjib Bhattacharyya, Jagarlapudi Saketha Nath, and Sankaran Raman. Variable sparsity kernel learning. J. Mach. Learn. Res., 12(17):565-592, 2011.
Foivos Alimisis, Antonio Orvieto, Gary Bécigneul, and Aurelien Lucchi. A continuous-time perspective for modeling acceleration in Riemannian optimization. In International Conference Artificial Intelligence and Statistics, pp. 1297-1307. PMLR, 2020.
Dimitris Bertsimas and Omid Nohadani. Robust maximum likelihood estimation. INFORMS Journal Computing, 31(3):445-455 458, 2019.
Rajendra Bhatia. Positive definite matrices. In Positive Definite Matrices. Princcton university press, 2009.
Nicolas Boumal. An introduction to optimization on smooth manifolds. Available online, Mag, 3, 2020.
Nicolas Boumal, Bamdev Mishra, P-A Absil, and Rodolphe Sepulchre. Manopt, a Matlab toolbox for optimization on manifolds. The Journal of Machine Learning Research, 15(1):1455 1459, 2014.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
Sébastien Bubeck, Nicolo Ciaa-Bianchi, al. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and TrendsS in Machine Learning, 5(1):1-122, 2012.
Lucian Busoniu, Robert Babuska, and Bart De Schutter. A comprehensive survey multiagent reinforccment learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 38 (2):156-115) 172, 2008.
14

---

Published in Transactions on Machine Learning Research (08/2023)
Constantinos Daskalakis and Ioannis Panageas. The limit points of (optimistic) gradient descent in min-max optimization. Aduances in Newral Information Processing Systems, 31, 2018.
Constantinos Daskalakis, Stratis Skoulakis, and Manolis Zampetakis. The complexity of constrained min-max optimization. In Annual ACM SIGACT Symposium on Theory of Computing, pp- 1466-1478, 2021.
Jelena Diakonikolas, Constantinos Daskalakis, and Michael I Jordan. Efficient methods for structured nonconvex--nooconcavv min-max optimization. In International Conference on Artificial Intelligence and Statistics, pp. 2746-272227 2254.. PMLR, 2021.
Ahmed Douik and Babak Hassibi. Manifold optimization over the set of doubly stochastic matrices: A second-order geometry. IEEE Transactions on Signal Processing, 67(22):5661-5774 2019.
Laurent El Ghaoui and Hervé Lebret. Robust solutions to least-squares problems with uncertain data. SIAM Journal on matrir analysis and applications, 18(4):1035-1064, 1997.
Tanner Fiez and Lillian Ratliff. Local convergence analysis of gradient descent ascent with finite timoscale separation. In International Conference on Learning Represenlation, 2021.
Tanner Fiez, Benjamin Chasnov, and Lillian Ratliff. Implicit learning dynamics in stackelberg games:
Equilibria characterivation, convergence analysis, and empirical study. In International Conference on Machine Learning, pp. 3133-3144. PMLR, 2020.
Yihang Gao, Huafeng Liu, Michael K Ng, and Mingjie Zhou, HessianFR: An efficient Ressian-based follow- the-ridge algorithm for minimax optimization. 0rXiv:2205.1000 2022.
Kenneth R. Garren. Bounds for the Eigenvalues of Motriz, volume 4373. National Aeronautics and Space Administration, 1968.
Noah Golowich, Sarath Pattathil, Constantinos Daskalakis, and Asuman Ozdaglar. Last iterate is slower than averaged iterate in smooth convex-concave saddle point problems. In Conference on Learning Theory, pp. 1758- 1784. PMLR, 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commumications of the ACM, 63(11):
144, 2020.
Andi Han, Bamdev Mishra, Pratik Jawanpuria, and Junbin Gao. Riemannian accelerated gradient methods via extrapolation. In International Conference on Artificial Intelligence and Statistics, pp. 1554 1585. PMLR, 2023a.
Andi Han, Bamdev Mishra, Pratik Jawanpuria, Pawan Kumar, and Junbin Gao. Riemannian Hamiltonian methods for min-max optimization on manifolds. SIAM Journal on Optimization, 3397) 2023b.
Reshad Hosseini and Suvrit Sra. An alternative to EM for Gaussian mixture models: batch and stochastic Riemannian optimization. Mathematical Programming, 181(1):187-223, 2020.
Feihu Huang and Shangqian Gao. Gradient descent ascent for minimax problems on Riemannian manifolds. IEEE Trunsactions on Pullern Analysis and Machine Intelligence, 2023.
Minhui Huang, Shiqian Ma, and Lifeng Lai. Riemannian block coordinate descent method for computing the projection robust Wasserstein distance. In International Conference on Machine Learning, pp. 4446 4455. PMLR, 2021.
P. Jawanpuria, N. T. V. Satya Dev, and B. Mishra. Efficient robust optimal transport: formulations and algorithms. In JEEE Conference on Decision and Control, 2021.
Pratik Jawanpuria and Bamdev Mishra. A unified framework for structured low-rank matrix learning. In International Conference on Machine Learning, 2018.
15

---

Published in Transactions on Machine Learning Research (08/2023)
Pratik Jawanpuria and Jagarlapudi Saketha Nath. A convex feature learning formulation for latent task structure discovery. In International Conference on Machine Learning, 2012.
Pratik Jawanpuria, Jagarlapudi Saketha Nath, and Ganesh Ramakrishnan. Efficient rule ensemble learning using hierarchical kernels. In International Conference on Machine Learning, 2011.
Pratik Jawanpuria, Maksim Lapin, Matthias Hein, and Bernt Schiele. Efficient output kernel learning for multiple tasks. In Advances in Neural Information Processing Systems, 2015a.
Pratik Jawanpuria, Jagarlapudi Saketha Nath, and Ganesh Ramakrishnan. Generalized hierarchical kernel learning. J. Mach. Learn. Res., 16(20):617-633 2015b.
Chi Jin, Praneeth Netrapalli, and Michael Jordan. What is local optimality in nonconvex- nonconcave minimax optimization? In International Conference on Machine Learning, pp. 4880-4889. PMLR, 2020.
nonconvee-nnnnooncave
Michael Jordan, Tianyi Lin, and Mataanooul-Vasileiii PlatakisGGaragkoouie First-order algorithms for min-max optimization in geodesic metric spaces. Advances in Neural Information Processing Systems, 2022.
Hassan K Khalil. Nonlinear systems. Prentice-Hall, Upper Saddle River, NJ, 2002.
Max Kochurov, Rasul Karimov, and Serge Kozlukov. Geoopt: Riemannian optimization in pytorch. 0rXiv:2005.0281 2020.
Jifí Lebl. Basic analysis: Introduction to real analysis. volumn II, 2009.
Suchool Lee and Donghwan Kim. Fast extra gradient methods for smooth structured nonconvex-nonconee minimax problems. Advances in Neural Information Processing Systems, 34:22588 22600, 2021.
Tianyi Lin, Chenyon Fan, Nhat Ho, Marco Cuturi, and Michael Jordan. Projection robust Wasserstein distance and Riemannian optimization. Aduances in Neural Information Processing Systems, 33:9383 9397, 2020a.
Tianyi Lin, Chi Jin, and Michael Jordan. On gradient descent ascent for nonconvex-concave minimax problems. In International Conference on Machine Learning, pp. 6083 6093. PMLR, 2020b.
Tianyi Lin, Chi Jin, and Michael I Jordan. Near-optimal algorithms for minimax optimization. In Conference on Learning Theory, pp. 2738 2779. PMLR, 2020c.
David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. Learning adversarially fair and transferable representations. In International Conference Machine Learning, pp. 3384 3393. PMLR, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial atlckks. arXiv:1706.06083, 2017.
Eric Mazumdar, Lillian J Ratliff, and Shankar Sastry. On gradient-based learning in continuous games. STAM Journal on Mathemalics of Data Science, 2(1):103 111110331111-11 2020.
Jorma K Mcrikoski and Ravinder Kumar. Inequalities for spreads of matrix sums and products. Applied Mathemalics E-Noles, (1950-159):9, 2004.
Bamdev Mishra, NTV Satyadev, Hiroyuki Kasai, and Pratik Jawanpuria. Manifold optimization for non-linear optimal transport problems. ar arXiv:2103.000 2021.
Aryan Mokhtari, Asuman Ozdaglar, and Sarath Pattathil. A unified analysis of extra-gradient and optimistic gradient methods for saddle point problems: Proximal point approach. In International Conference Artificial Intelligence and Statistics, pp. 1497-1507. PMLR, 2020a.
Aryan Mokhtari, Asuman E Ozdaglar, and Sarath Pattathil. Convergence rate of o(1kk) for optimistic gradient and extragradient methods in smooth convex-concave saddle point problems. SIAM Journal on Oplimization, (2330 3251, 2020b.
16

---

Published in Transactions on Machine Learning Research (08/2023)
Jan Müller, Reinhard Klein, and Michael Weinmann. Orthogonal Wasserstein GANs. arXi::911113300, 2019.
Arkadi Nemirovski.
with rate of convergence o(1/t) for variational inequalities with Lipschitz continuous monotone operators and smooth convex-concave saddle point problems. STAM Journal Optimization, 15(1):229-25) 251, 2004.
v Neumann. Zur theorie der gesellschaftsspieee. Mathematische Annalen, 100(1):295 320, 1928.
Madhav Nimishakavi, Pratik Jawanpuria, and Bamdev Mishra. A dual framework for low-rank tensor completion. In Conference on Neural Information Processing Systems (NeurIPS), 2018.
Maher Nouiehed, Maziar Sanjabi, Tianjian Huang, Jason D Lee, and Meisam Razaviyayn. Solving a class of non-convex min-max games using iterative first order methods. Advances in Neural Information Processing Systems, 32, 2019.
François- Pierre Paty and Marco Cuturi. Subspace robust Wasserstein distances. In International conference on machine learning, pp. 5072 5081. PMLR, 2019.
Alain Rakotomamonjy, Francis Bach, Stéphane Canu, and Yves Grandvalet. Simplemkl. J. Mach. Learn. Res., ((83):2491-22) 2221, 2008.
Maurice Sion. On general minimax theorems. Pacific J. Math., 8(4):171-176 1958.
Chaobing Song, Zhengyuan Zhou, Yichao Zhou, Yong Jiang, and Yi Ma. Optimistic dual extrapolation for coherent non-monotone variational inequalities. Advonces in Newral Information Processing Systems, 33: 14303-14314 2020.
smooth minimax optimization. Advances in Newral Information Processing Systems, 32, 2019. Kiran Thekumparampil, Prateck Jain, Pranecth Notrapalli, and Scwoong Oh. Efficient algorithms for
Paul Tseng. On linear convergence of iterative methods for the variational incquality problem. Journal of Computational and Applied Mathematics, 60(1-2):237-252 1995.
Nisheeth K Vishnoi. Geodesic convex optimization: Differentiation on manifolds, geodesics, and convexity. arXiv:1806.00 0633,, 2018.
Yuanhao Wang, Guodong Zhang, and Jimmy Ba. solving minimax optimization locally: follow-the-ridge approach. International Conference on Learning Representations, 2019.
Xidong Wu, Zhengmian Hu, and Heng Huang. Decentralized riemannian algorithm for nonconvex minimax problems. XxXi::2302.0882, 2023.
TaeHo Yoon and Ernest K Ryu. Accelerated algorithms for smooth convex-coneave minimax problems with O(1/²²) rate on squared gradient norm. In International Conference Machine Leurning, pp. 12098 12109. PMLR, 2021.
Guojun Zhang, Kaiwen Wu, Pascal Poupart, and Yaoliang Yu. Newton-type methods for minimax optimization. arXiv:2006. 14592, 2020.
Guojun Zhang, Pascal Poupart, and Yaoliang Yu. Optimality and stability in non-convex smooth games. Journal of Machine Learning Research, 23:35-1, 2022a.
Junyu Zhang, Mingyi Hong, and Shuzhong Zhang. On lower iteration complexity bounds for the convex concave saddle point problems. Mathematical Programming, 194(1-2):901 935, 2022b.
Peiyuan Zhang, Jingzhao Zhang, and Suvrit Sra. Minimax in geodesic metric spaces: Sion's theorem and algorithms. arxiv:220 06950, 2022c.
17

---

Published in Transactions on Machine Learning Research (08/2023)
A
Organization of Appendix
The appendix sections are organized as follows.
Appendix B provides a preliminary introduction to Riemannian geometry and Riemannian opti- mization.
Appendix C discusses the practical implementation of the proposed algorithms in the main text, including the evaluation of the Hessian inverse and ingredients for the Riemannian geometries considered in this work.
includes additional experimental results the problem of computing projection robust Wasserstein distance.
Appendix D
Appendix
E derives some useful lemmas for the proofs of the main results.
Appendices present the proofs for the main results.
B Preliminaries on Riemannian geometry and Riemannian optimization
This section introduces the basics of Riemannian geometry and optimization with a focus on the essential concepts required for this work. We refer readers to Absil et al. (2009); Boumal (0200) for a detailed exposition of the topics.
Riemannian geometry. A Riemannian manifold M is a smooth manifold equipped with smooth inner product structure, i.e., Riemannian metric on each tangent space T,M, Vz E M, which is denoted (u, u)s for any u, T.M. Riemannian metric induces a norm || = / u)2. We also use TM to denote the tangent. bundle, which is the disjoint union of all tangent spaces. A geodesic y [0, + M generalizes the straight line segment in Euclidean space, which is the locally shortest curve with zero acceleration. The exponential map, Exp. T&M- + M and for any u E T.M, (xp, is defined such that there exists a geodesic y with 7(0) = 2, 70) = u and 7(1) = (u), Riemannian distance dist(21, (2) is defined as the length of the distance minimizing geodesic connecting 21, 22 E M. In a uniquely geodesic neighborhood U, the exponential map has a smooth inverse, namely the logarithm map, i.e., Log.. u + M. In such neighbourhood, we can express dist(21, ||Logz, (22)lla, [[Log,, (=1) Parallel transport 1 M + M is linear isometric map between tangent spaces such that the vector field along curve connecting 21, 22 is parallel.
This paper also considers the product of Riemannian manifolds, i.e., = M x My, which also admits a Riemannian manifold structure, with the Riemannian metric induced from Ma, That is, for any 2 = (z, 3) M, (u, v), (u', r') T,M, ((u,)), (ud,v) (u', v')),
(u, ul)M= + (v, of) MM where we denote (,,)) (.))M,,(,)) MM as Riemannian metrics on MM,Ma, respectively. We will make use of a lincar operator on the tangent space of Riemannian product manifolds, i.e., H. T&M + T.M. We can write out its operation on (aa Hay { (u,u) T,M, x My H[E Hye Huy
e (y= Huz[u]] + Haylv!) where H TyMM
Hyyllu [u] + Hyylu)
Ta,,, Hyp My + My, Hay T,,M, + TyM2, H T.M, + Ty,M, are linear operators. Here we slightly abuse the block-matrix notation for representing the decomposition. For a linear operator T.T,M, + TTyM, between tangent spaces, we denote its adjoint operator (with respect to Riemannian metrics) G TyyMy + TaMa such that for any u, TyMM, My E MyyMy, (uy, Glum))? (uz, Gf (uy])>= We use V to denote the covariant derivative induced from the Riemannian connection. On product manifold, its Riemannian connection is derived from the product connection (Boumal 2020 Exercise 5.4).
Riemannian optimization. For a real-valued function ʃ -M- + IR, its Riemannian gradient gr) is the tangent vector that satisfies (g)adf(:), = Df(x)(E] for any ş E T.M. The Riemannian Hessian f T.M + T.M is a self-adjoint, lincar operator, defined as the covariant derivative of the Riemannian gradient, i.c., (2s (z). A geodesic convex set 24 S M is a subset where every two points are joined with geodesic that lies entirely in the set. A function f M R is called p-geodesic strongly
18

---

Published in Transactions on Machine Learning Research (08/2023)
convex in U is for any 22 E U1 there exists a geodesic y connecting them such that d'
dr 2 f for all E [0, 1] for some f 2 0. If μ 0, the function is called geodesic convex. A function is called geodesic linear if it is both geodesic convex and geodesic concave.
Riemannian min-max optimization. For bifunction J :M x My - IR, we denote (aI v): V V v) as the Euclidean (partial) gradient and grad, (æ, y), grad, (z, v) as the Riemannian (partial) gradient with respect to I, respectively. A bifunction f(x. y) is geodesic (strongly) convex (strongly) concave if it is geodesic (strongly) convex in x and geodesic (strongly) concave in y. For the bifunction [(x, y), its Riemannian Hessian is derived in (Han et al., 2023b Proposition 9), given by Hess(rra)) v)lE [eess,/(1,,) 3)
J y)
()) where gradzy (1, (2, v) MyM, - TaMa is the Riemannian cross deriva-
1
f v)
Hessy (2, v)
tives, computed as grad? [ v)] Dygradz (2;, ()] and similarly for grad [ v). It has been shown in (Han et al. adjomnnness ol Riemannian Hessian.
2023b. Proposition 10) that graday (x, v), grad fz, y) are adjoint operators due to solf-
C
Implementation and Riemannian geometries
C.1 Practical considerations
The second-order methods introduced in this section all require evaluating the Hessian inverse. In order ensure efficient computation in real applications, we adopt the following practical implementation strategies whenever necessary.
Conjugate gradient and regularization. In order to compute Hess, ffr, y) [v], we consider solving a linear system (f²s y)[s} Hess, f(x y) for 8 E My.M.. This can be solved via a conjugate gradient algorithm tangent space (Boumal, 2020 Section 6.3). To ensure (Hess, (z, y) is strictly positive definite, we add a regularization term to the linear system as ((Hess,, + (a, ()) sufficiently small c > 0. Such a strategy has also been considered in the Euclidean space (Wang et al. 2019
f [v] for some
Zhang et al. 2020 Gao et al. 2022). In all the experiments, we simply set e == 0.
Use of general retraction in place of exponential map. When the exponential map is inaccessible or computationally costly to evaluate, a general retraction (Retr T' + M) can be used to approximate the exponential map, usually to the first order. With retraction, the resulting dynamics is 24+1 Retr (17 G(zz)), and can be similarly linearized using inverse retraction (Retr-) : : + TMM, and Proposition 7 holds, i.e., Retr (2x+1) = (id + 1 (Retr""") + R(] (see Han et al. (2023a) for more details). Hence the same notions of stability and asymptotic convergence in Section 5.3 apply, and the convergence
analysis for various methods also holds.
For the experiments, we use the exponential map whenever such an operation is well-defined, e.g., on SPD and sphere manifolds. For the Stiefel and doubly stochastic manifolds, we use the retraction operation.
C.2 Riemannian geometries of manifolds of interest
Symmetric positive definite (SPD) manifold. The SPD manifold of size d is denoted as S = X
[ Radd T X,X 0) where the affine-invariant metric (U, V)x -
is considered
Bhatia 2,09), for U. E TxS4 The exponential map is given by Expx(U) = Xexpm X UU) where expm(()) denotes the principal matrix exponential. The corresponding logarithm map is given by logx (Y) Alogm((--y) Its Riemannian gradient of a real-valued function f is derived as gr)df(X) X xXX Dg - (UX- (UX-Igradf((X) "gradf(X))s = XXX +
and the Riemannian Hessian is (xsss(X) (X) [U]
(XVS(X)X}s where we use {A}s = A + A 1/2.
Sphere manifold. The sphere of size d is St = {x E Rd x manifold with the Euclidean metric. The exponential map is given by (xpx = cos(I|ull2)x sin sin(|[u||2) |म (I-xx 1 -x)xx)
T x = 1). The sphere manifold a Riemannian
and the logarithm map is Logx arccos(x y)
-X-xx -)y= The Riemannian gradient and Riemannian
19

---

Published in Transactions on Machine Learning Research (08/2023)
RODK
TM RER
RUM
Gradnorm
Gradnomm
GNG3
REGB
Iteraaioo
Time
aa) PRWD (Gradnorm-iter)
(b) PRWD (Gradnorm-time)
Figure 6: Experiments on the projection robust Wasserstein distance (PRWD) computation problem on the Tragmented hypercube. We see all methods, except RHM and RNFR, perform compelitively.
Hessian are respectively gradf(x) = (I xx (x) and Hesss = (I- [1-xx xx [ - XVVV (x)u for any u E TxS4.
Stiefel manifold. The Stiefel manifold rt(d.r) {X E gdxr X I with the Euclidean inner product. We consider the QR-based retraction in the experiment, which is Retrx (U) qf(X U) where qf(+) extracts the Q-factor from the QR. decomposition. Let the orthogonal projection to the tangent space be denoted (X( = U- X UJs. Then, the Riemannian gradient and Riemannian Hessian are given by (() = Px (X)) and [e = (xx - U(X T v(())s)).
Doubly stochastic manifold. The doubly stochastic manifold (or coupling manifold) between two discrete probability measures μu, v with marginals a E R "..be R" is the set IT = {T E Rmxa Túi > 0, = a.r 1m = b). It can be equipped with the Fisher information metric, defined as (U, V)r = EUg for any U, V v), The retraction is given by Retrr (U) Sinkhorn( o exp( where exp, O, are elementwise exponential, product, and division operations.
D Additional experiments on projection robust Wasserstein distance problem
We consider the problem of computing the subspace robust Wasserstein distance between two discrete
probability measures (Paty & Cuturi 2019
Lin et al. 2020a Huang et al. 2021 i.e.
T x; - U'y y +el (log(Tis) - -1)).
min DE)((LU) UESt(d.r)
max
i=
where j = Em ax == Σn b,dy, are discrete probability measures. We denote 11 1) {T E "x"" F5 > 0, ,, VjE 1, F. = a, Vi [m]] as the set of doubly stochastic matrices, which form a Riemannian manifold (Douik Hassibi 2019: Mishra et al. 2021). The Riemannian gradient and second-order derivatives are given in Lemma The problem is nonconvex-nonconeaw and hence, we track the gradient norm for comparing convergence behaviors of various algorithms.
Experiment setup and results. We consider the same fragmented hypercube example as in Lin al. (0200a): ; Han et al. (2023b), concerning a uniform distribution on [_1, and a push-forward map T(x) wheresign(() sign(-) denotes the elementwise sign of and e; are the i-th canonical basis in Rdd We consider the same choice of parameters in generating the samples as d = 30, 5, k 2, 100, 0.2. We exclude RCEG in the experiment because there is no well-defined logarithm map (or inverse retraction) operations for double stochastic manifold. In Figures 6a
++2ig (x)@ (E ez), where
we see that all methods, except RNFR and RHM, perform competitively. Even though we see matching convergence in iterations, TSRGDA is slightly superior due to its less costly per-iteration computation.
6b
E
Useful results
This section derives several useful results for the analysis presented in this paper. Appendices b1 and allow to relate the spectrum of any linear operator tangent space to its matrix representation via
20

---

Published in Transactions on Machine Learning Research (08/2023)
coordinate transform. This is crucial for quantifying local minimax points. Appendix E3 provides inverse and implicit function theorem manifolds, which proves to be useful for relating dynamical systems under different representations (i.e., Proposition 6
E.1 Spectrum of tangent space linear operators
First, we recall the definition of eigenvalues and eigenvectors of a tangent space operator.
Definition 6. For some x E M, let A : T&M + T.M be a lincar operator. A tuple (A, u) for some A R, T,M is called an eigenpair of A A(u] Au.
It important notice here that the cigenvalues need be real. Thus, based on Definition c a general non-sell-adjoinl operator can have eigenpair, unlike the case in Euclidean space where complex eigenvalues and cigenvectors well-defined.
The next lemma shows that for eigenpairs that exist for a linear operator (not necessarily self-adjoint), they can be equivalently written in terms of matrix representation of the operator on an orthonormal basis. It should be emphasized that only real cigenpairs are preserved when switching back and forth between the matrix representation and the abstract operator.
Lemma 1 (Eigenpairs in basis). Suppose we are given an orthonormal basis (E1, Ed) of a tangent space T.M. Then for some X E IR, u E T,M, we have Alu]= Au if and only if Aut = >ut where i = [(u, ei) Re,, Algglizli are the vector/matin representations in the basis.
Proof of Lemma ! Taking the inner product with e;, i L,... d for Alu] = Au on both sides gives (A[u), ex) Xūt, for all where i7, (u, ez) is the i-th component of vector i. Hence it remains to show the i-th component of Au is equal to (A[u), ez).. This can be seen from the linearity of A, i.e.
d Σ (e,, Aleyllaauu,es)) (et,1 3'=1
d Σ
(Au)
(u, (u,eyreyl); = (e,, Alu)). 3'=1
where we note u Dd (14, 2i)
Furthermore, extend such result to tangent space operators Riemannian product manifold.
Lemma (Eigenpairs in basis product manifold). Consider product Riemannian manifold M Mr My of dimension de dy. Let linear operator be T T,M T&M and denote its elements Tax TrM + T.,,Tyy TyM + My Tzy MyM + TrM, Tyz T.M- + TyM be linear operators, not necessarily self-adjoinl.
Suppose that we are given an orthonormal basis B, = (EI, ed,), B, = (Wi,.., Wa,). Also suppose that for some E E T.M such that (X,E) is an eigenpair of T.. Then, we have TzlEi = XE if and only if 1.,,,,.,,,, (u, ea,) and ū= [(v, wi wi)y..., (v. wa,)ul and T.
T.E NE where { = 117 Rd, where i = [(u, 01),,
Tam, Tay) E )xdd+t)) with T [(ei, Tex (e))))) Try (es, Tay (m))))) Tye = kww, T Tyz legl)oles
Tyz: Tyy]
Tny [(,, Lollw,lluk.
Proof of Lemma 2 If (A,E) is an eigenpair of Te, then T.k = xg and we set & = v) with Edz ei) v Σee (v, wy)y Substituting the expressions to the T2[E] and taking inner product with
21

---

Published in Transactions on Machine Learning Research (08/2023)
Twn (x]
+ Txyl
basis in B2, respectively yields T=
where
T [u]
+ T yy
d
Tex [u]
Tax
[0)
(ei, Tex
[u (une))ze)))
(e, Taw Σ (us woy w
(e,,Tov/
1=1
de
(4,
Σ
(u, ey Tox
(u, wz)) y Tayywos))
j=1
d,
dy
(e,, Tan [e ey)
Σ
fei, Tan [w5))= wa)v
j=l
j=1
[Tam [i]
+ Try
vi [1, dz),
Ty [1
+ T yu [1)
[T [
Tu
vi E (d + 1, 1,db,
=
where we apply the linearity of the operators. Hence, T2 T.c. Similarly taking the inner product for
the RHS (i.e., XE), we have Xě = Ag. For the converse, we can show if T XE, T&E = xg using similar arguments via lincarity of operators.
We also remind of a result for similar operators.
Proposition 14 (Similar operators). Consider a Räemannian manifold M and æ E M, let A, linear operulors on T,M. We say A, are similar if there erists a non-degenerule operalor such that A C. C- In addition, if A, B are similar, then they share the same (real) eigenwalues.
E.2
Coordinate transform
Lemma 3 (Coordinate transform isometric isomorphism). Given an orthonormal basis (e1, ea) on a tangent space T,M, the coordinate transform T.M - defined by Φ(u) = [u, es)= an isometric isomorphism (i.e., bijective isomelry) belween the langenl space and the Euclidean space.
Proof of Lemma 3 First, given the coordinate transform, we can write for any E T2M, u u= Et The mapping @ it is invertible and write the inverse of ( Σd The, such that ( (1) E (u, ci)2u= = and ( = In addition, we have for any u, e E T&M, ū, 7 Rd.
Σ Yu (i) = (u, Σ 2=1
(o)e))) (u, v)21
i=1
J,e,,
djed) Σ ū,06 (ū, 0)2.
where we use the orthonormality the basis.
E.3 Inverse and implicit function theorem on manifolds
Theorem 4 (Inverse function theorem on smooth manifolds (Boumal 2020 Theorem 4.16)). Let G: : M2 + My be a smooth map on two smooth manifolds My. If (G is invertible at x E Ma, then there erists an open subset U₂ C M2 that contains x and My C M, that contains G(x) such that Glu, Ur + Uy is a diffeomorphism.
Thcorem 5 (Implicit function theorem on smooth manifolds). Consider a differentiable function F : Ms My + TMy defined on the smooth product manifolds M MM. Suppose x" (x".y", E Mr My such that. F(x", ) and V F("", (s=",") v/" is invertâble where V, denotes the covariant derivative My-
22

---

Published in Transactions on Machine Learning Research (08/2023)
Then, there erists open subset U = U2 x Uy C M2 x My where x' Uuryy" E My and differentiable function V : MMx- + My such that (x²)) 5f and for all x E Uu, V(x) is the unique point in My such that (()) 0. In addition, we have D2 (x*( = ( )z2)) (x
Theorem 8.5.6) in Euclidean space. We start by defining a map F u Ma x TMy as F(x, v) == (z, F(x, (()y))) We recall that the tangent bundle TMy is also a smooth manifold, and therefore, Mr M. too smooth product manifold. The derivative of map F, for any & = v), TyM, is derived as DF( Tyu MrM2 x My
Proof of Theorem G The proof is motivated by (Lebl 2009
DF(() (D. (2, v)lu (D.. + v. (xr,)))))))
Evaluated at 2", it is easy to see when DF [{] 0 if and only if = 0 (because vz*(2*) is invertible). Thus, it follows that Dz(2) is invertible. By the inverse function theorem (Theorem 4h. there exists an open subset V that contains F(=*)= (x", 0) and Flu : 4 + V is invertible. That is, for any (z, v) E V, we have (z, v)) = (x, v). Let F-1 = (FT). F2 such that F(F (z. F (z,
F2 (2, = (2, v).
This leads to F(x, F5 (ar, v)) = v for any v E V. Set v yields F(x, F- (x, = O. Let ((x) == F3 (x,0)
and it can be verified that V (x) is one-to-one as Fz [z, 0) is one-to-one and hence the uniqueness.
Finally, we take derivative of F(x, (x)), which is the zero map. This loads to 0 0= F(z, [z] = D(F(e,,(<)\au) + vyF(z, (ex) (x))[u]
for any M.. Thus it follows that for x²,"" ((*),, V* (VV o,
F Proofs of Section
Proof of Proposilion 2020
The proof can be trivially adapted from the Euclidean counterpart. in Jin et al.
Proof of Proposition g (1) First, we prove the first-order condition. By Proposition 2 y² is the local maximizer
of ((x",)), and therefore (based on Proposition 4.5 in Boumal (2020)), we have grad, (x", ,²)==")=))) O. From the definition of a local minimax point, we know that f(" .y minn maxyodist (Sh()) I (x, w) for any 6 du- For any I' Exp.- (u) with ||u][a- S 5, denote y' = arg maX f f(z', v). Then,
0 0 f(a' v/) -
f(a, (/) f(x", ((x",/)) f(="(((" ,²)) - f(x", ,v")
$ f(a', ./) 5(x", = (grad, (x= ,')), u) + a(Iull==-) = (grad f( (x","");,)))"""))))",))) u).- + o([lu]la-)
Since the inequality holds for any u with ||u||a- + O, we have grad, f(x", ,y²) y²)=== 0.
(2) We now prove for the second-order condition. By the fact that y" is a local maximizer of f(x", v), we have by standard result Hoss, (x=",""""""""""""" ,y") = O. In addition, by Taylor expansion on f. we have
f(Exp (ux), Expy- (uy)) = f(x", "o"",,")=
2 (ugs Hess, f(x" v (uzll + (uz,, grad fx" "o"",, ,3 (uyl)
+ + [ug, (uy, Hess,, f(=", v² (uylyy + olllu=1la + 1(u,)ll-),
where we use the fact that Riemannian cross derivatives are adjoint with respect to the Riemannian metric. Since Hessy (x", "e=,,)") y²) is invertible, define h(6) = 21l[Hess,, f("", ,y o gradhu f("", "")"lp- that satisfies T(6)) 0 as 8 + O. Hence it can be shown that - ([Hessy/ (x" : yy"]]= 11 e grady. y² [u.] + o(I arg max blply (mxx)l))_mm f(Exp.- (uz), Exp,. (u,)) for any 1(6).. By definition of (x", 2²) as local minimax point,
05 max [(Exp,- (uz), Exp,- (1))) - f(x",1")
|1a (B)
|lalll- (()) [Exp,,. (uæ), Expy, (uy)) - f(=", .000,0 ,y²)
max
2 (ums (Hess, (2" ,y²) gradz (z, v) a [Hess, (xr w))l grad (x. vi) [um])=r- + a(Iuun 12.).
23

---

Published in Transactions on Machine Learning Research (08/2023)
Since the incquality holds for any ux, the proof is complete.
Proposition 15 (Sufficient conditions for local Nash saddle points and local minimax points). Let a point (x",y") satisfy: (1) grad,f(x", yy" = 0. grad, f(="/(/="""""" yy) = 0 and (2) Hess, f("", ((a")))) > 0, Hesssy f(" (20,, y²) x O.
Then, the point is (strict) local Nash saddle poinl.
If point (x*,y") satisfies gradzf(x", = 0, gr (2", ,²) = 0 and Hessy ,v²" < 0, Hess, y) - grad
v) o [Hess, (xr y)] grad (x, y) - 0, then (x", ,²") is a (strict) local minimaz point.
Proof of Proposition 15. The proof follows a similar vein from Proposition 3 and (Jin al. 2020 Proposition 20).
Proof of Proposition 4 (1) By the definition of global Nash saddle point, we see when (X', Y+) S f(X", Y") = 0 and it satisfies - (log det( f(x", Y) I f(X', Y") s (X,Y) = (log det(()) (x)) Then, from (Jin et al. 2020 Proposition 11), we can conclude (X", Y*) is a local minimax point.
(2) For the second claim, we follow the definition of local minimax point in Definition 3 It is easy to verify that when (X", Y") E S x S", -ca(log det (Y))² = f(x", Y) S f(X", Y") = for all Y E y. On the other hand, consider maxy Y) f(X, for X such that dist(X", XX S 8. First we can for any symmetric matrix V. We can show the Riemannian Hessian is negative semidefinite with respect to the Riemannian metric, ie., (Hess,, Y)[V], V)x = -2c (cc(tr(Y--V))) concave, e.g. when V=0), Hence by first-order conditions and compactness of the domain, we can show the maximum is attained either when grad,, [X, Y) = O or on the boundary of the constraint set. The Riemannian gradient can be derived as grad, Y) (cz log det 20g log det dY) Given Y E e$4 f
derive the Riemannian Hessian as H [V] = -eeettr(Y-
s O. Hence f(X, Y) is geodesic concave in Y (but not strongly
(X, Y) = 0 if and only if e2 log de) = log det (Y). The task is thus to show given certain 6, h(8), and X such that dist(X', X) < 8 we can always find Y satisfying dist (Y,, Y) S h(8) such that gradyf(X, Y) = 0.
To see this, first we recall that the Riemannian distance is
(iis?(A, (A, B) = ||logm( A - B)]l
log (Aa(A (B))
We notice that for any A S*, any B E S it satisfies log AedettAtt A- = log degd(t-A)) + log det (B) log det( In addition, we notice that log dBt = Σn (B)). Hence, suppose we choose Y such that (x)) -x)) for all E [d] Since both X", Y E S*, we can verify
log ((YY)-YYY) = i2
20g
log (edde(() YY) = log (Y))
C2 log det (x)=
C2 log det)
203
20g
which satisfies frad,,(XX Y) = O. Now we just need to show dist 2 (Y', Y) is bounded. To this end, we write
d ( (log (xx(())))x)))
o A dist dx*t (X", x). 4og
dist d Y) (log (X('))
Hence, can simply choose h(8)
2n3 5, and under such choice, we can always find Y given X under the constraints such that /Xa Y) O. Such choice leads to d(X) = (ixx: f(x, Y) Teg ca 2 O. Hence o(X) f(', Y*) 0.
- c les
a (log de) When C2 2 2/eic we have &
Finally, to show it is not local Nash saddle point, it suffices to see that (U, Hess, Y*) 20, (Ur()))² < O, which violates the second-order characterization of local Nash saddle points (Proposition
(3) For the third claim, first we see that f(X*, Y) S fX", (()),')= = 0 for all Y E y. Then, we consider (I fX, Y) for X with dist X)= < 6. From the standard result (e.g. (Alimisis et al. 2020 Lemma 2)), we notice that the f(,, [)] is negative definite for SPD manifold with affine-invariat.
24

---

Published in Transactions on Machine Learning Research (08/2023)
metric (because the curvature is negative). Hence f(X, Y) is geodesic strongly concave in Y. We similarly take the first-order conditions where g Y) is computed as
grad, (X, Y) = C2 log deggeel(())) + 2cgLogy (I) = C2 log det X + 20g Ylogm( Y ()s
0 02 log det (X) (X) A 20z A log( O))
where we consider the cigendecomposition of Y = OAO T (A is a positive diagonal matrix) and denote {A}s = (A+ + AT) 1/2 and logm(A) A) as the principal matrix logarithm of A E $4 We also use log(A) (A) as elementwise logarithm. can be verified that (X,, Y) 0 if and only if c2 log (X)A - -2a log(A) ) = 0. Given > O, this is equivalent to 02 log det = 20g log( Hence the AY( = exp(
ez log (X)) for
2ns
all if and only if f(X, = O. Given that dist 2 XI. Ei (10)(A((X))) 2 I 8² and suppose (Y) = exp( d2 log det( Then,
20g
de del (x))²
de2 d 4
L& E 8²
dist (1, Y) = Σ (log(A((Y)))
flog det
log log():(X)))² -
403
4cg
where we use the fact that (Ei= a;)² < d Ei of in the last inoquality. Let h(d) always ensure grad,/(X, Y) = by choosing Y with X; exp(( log (X)( for all
5e Then, we can
Under such choice of Y, we obtain log dY) gives
dea log det (X) and dist (I. Y. 20s
de def
(log detd())) which
de2
doz
((X)
max V:dist(I,Y) (L))s()))))
1(X, Y) - (log det (eeddee())) (x))² (log det (X))²
(ogdet(X))
203
4eg
dc3
=( 4c3
When de3
C1 2 0, we have o(X) 2 0 = f(X', Y*). Hence, the proof is complete.
G Proofs of Section 5
Proof of Proposition E First, we study the linearization of the dynamics Log,-(:(()) on T as follows.
d dl (z(1)) = DLog.- (=()))=(0)) = DLog.- (())))))) = (DLog,- (x(1)) [ ((=(0))))))
T - G(z(t) o(ILogg-- (=(t))ll=-), (8)
where the last equality is from the proof of (Han al. DLog.- (z(t)) o (=(() is locally identity.
2023a
Lemma 3) where it has been shown that
Next from Taylor expansion on G(z(t)), we obtain
T G(=(t) = VG(= ([(=""")[o [[oo
(9)
where we notice G(z*) = 0. Finally combining the results with gives the first result. Next,
di v(((t)) = (=Log..<x(t)) H| d Log.- (z(t))))
d
di
= 2( (z(t)), H (z()), (I VG + o Lo) (x())]]) +
+ (zo)))
where (VG(="))t represents the adjoint operator of VG(=*.
25

---

Published in Transactions on Machine Learning Research (08/2023)
Proof of Proposition 0 From Proposition at mal basis, which represents tangent vectors in terms of its coordinates. It can be easily verified that ø defines an isomorphism between the tangent space and the R Thus under the coordinate system, let u(l) = (( and then the dynamical system can be described as
consider the linear dynamical system locally d [o())(<(()) v F(=")) (Togs)()) Let Φ T..M + R be coordinate transform via an orthonor-
a(t) = (Do( (u(t)) o
with an equilibrium point = 602-), where 0g- denotes the 0 tangent vector at T. Consider a Lyapunov function V(Log (z(t)) = - Log,, (=(t))]) The induced Lyapunov function under the coordinate system is given by Vo(u(t)) = ( (u(tt)) We can show that
d dt Volu(t))
= (DV(o)'(u(tt)) (P(p))))) Do(o='(u(t))) (ur
d
(DV(o))(u(()) (F(=)) {( dt V((
where the second equality is due to the inverse function theorem. Hence we see the stability analysis of =(t) is equivalent to that of u(t).
Finally, notice that @ is linear and is independent of t. Hence, have a(() (Φ FF o o (u(())
[u(t)]
Let G of F(=)) Φ- be the matrix representation of F=*)) under the coordinate system. Hence the dynamical system of interest is completely described in Euclidean space as üit) G[u(t)]. By Definition 4 u² = 600--) is strictly stable if there exists a positive definite matrix H such that HG + G* И is negative definite. This is equivalent to requiring G to have all real parts of its cigenvalucs negative (sce for example Theorem 3.6)).
Khalil 2002
Proof of Proposition 7 The first result follows similarly from the proof of (Han et al. 2023a Lemma 3). And the second result follows by substituting the first result in the definition of the Lyapunov function.
Proof of Proposition The proof follows similarly as the proof of Proposition P and applies the equivalence between the eigenvalue and matrix definiteness characterization of stability (see for example (Khalil 2002 Exercise 3.52)).
Proof of Proposition First (2)++2) is verified in Proposition Now we verify the equivalence (2)++(3). First, when (2) holds, let X a; bąi be the eigenvalues of matrix representation of GG(=*.. Then (2) states [1+ra, 1bbii| = /(1 na, 2 17 < 1 for all i. Given 17 > 0, must have a, for all i. For the converse, when (3) holds, i.c., all a; < 0, then we can see (1 + n7,,)² 2 + 2?? = + 2na; + (a + b,) 1 + 2na; + o(n). Hence there always exists a sufficiently small n > 0 such that + 21ja + o(n) < and (2) is satisfied.
Proof of Proposition 10 We start from the linearization of the system on T as A = (id + [VG(>)))[A)) + o(I o(IAAxl==+) where we let A = (2g E T.MM. Then we consider the coordi- nate transform ó under an orthonormal basis where let As = d(Ax). By the isometric property of such coordinate transform (Lemma
we have the equivalent characterization of the system A Ae+1 (I + VVG + o[I Axll)) where we notice that | = | By the definition of the asymptotic convergence rate (Definition 5) and following (Wang et al. 2019 Proposition 4), we have [ < C(1 x/2)*11 A Finally, applying the inverse coordinate transform and using the isometric property again, have the desired result.
26

---

Published in Transactions on Machine Learning Research (08/2023)
H Proofs of Section 5.4
Proof of Proposition 11 To show LNS TSRGDA, it suffices to write out the matrix representation of V (2*).
We consider orthonormal basis on langent space of product manifold as B = B, x B, where B, = (el; 6d.) and By = (Wi,... Wa,), each orthonormal. From Lemma
we can write the matrix representation of 2
(=") -HH -TH Hye Hyy
where Haz [ee, Hess,, (e)))s))) Hay = [(es, (la,, (2 (=))))
yyp = , (2" H=ssyf(=*) 0 and by Lemma 1 Hy == [w,, Hoss,, ( (w)) (wy])) Now since 2" LNS, f > 0 and all (real) eigenvalues of Hess,f(2"), (=²s, are equal to that of Hz., Hy, respectively. Thus we have Hz > 0, Hyv < 0 in the matrix sense. Therefore, the following proof readily follows from that of (Jin et al. 2020 Proposition 26). For the converse, since Euclidean space is a special case of Riemannian manifold, it suffices to consider the counter-exane -exatee given in (Jin et al. 2020 Proposition 26).
For the second claim, the counter-examples given in Jin et al. 2020 Proposition 27) suffice for the purpose.
Proof of Proposilion 12
Similar to Proposition
11
we adopt the matrix representation and the subsequent.
proof follows from F lez & Ratlifi 2021 Theorem 1).
Proof of Theorem The asymptotic convergence rate is readily obtained from Definition 5 The requirement 1) is to ensure the convergence rate is smaller than 1, i.e., |1 + mpsl < 1, for all 14,
]
Proofs of Section 6
Proof of Theorem First from Proposition g we start by writing the covariant derivative at 2" for RTGDA as (GRTGDA (Grd GRIGDA (2" [Hess where
(HroDa( (-rid 7 (grad o Hess, ()(ss Hesss I (22) gract / (2*)
gr (2²)) Hess, (2*)
id
- T (Hess,, f o Hess, f grad (2*) Hessy f(*))
grad (x*)
By transforming the operator into its matrix representation and using Lemma 2 for eigenvalue equivalence in basis, we see the eigenvalues of GErGDA [Hess) (z*)] are precisely the eigenvalues of f grad Hess, f o gradj,..) minimax point, then Hess,,((z*) x 0 and (Hess, grad Hess,, f o grad).. > O, which suggests eigenvalues of GRIGDA (2*) [Hess fflless((>> (2*)] are all negative. The converse is also true for the same reason.
and (=") (due to the block triangular structure). Hence if 2 is a strictly local
For the RFR, the covariant derivative
is given by (Gnnn|gradd/))
Gnrn(=") (Hess= where gradt f(2))
(Hess, f ((F)ss
id grad (2)) ())
(THess Pgradhy/
(RFR(=")(Hess) [(=)))
Hess,
which is similar to the following operator
(1Hesss; 'I fo grad (z*) id
-id id grad
-id 0) GRER(z (HHs) (HHess, f grad grad ((*) id
0
- grad Hess, f (n) rgradBy (2*)
Hess, ((²*)
By the eigenvalue property of similar operators (Proposition 14] as well similar arguments for RTGDA, the eigenvalues of GRFR(2") [Hessf(z")) are the eigenvalues of - (Hess,f- grad", o Hess, grad" (2*) and Hessy](2"") Thus the same equivalence between strictly local minimax point and strictly stable fixed point.
27

---

Published in Transactions on Machine Learning Research (08/2023)
To obtain the asymptotic convergence rate, consider the discrete case where the dynamics are given by and #k+1 = ExP GErGDA ( (=))) First consider RTGDA and denote A as the eigenvalues of V ([RTGDA [grad/[]) The asymplotic convergence rate is given by PRIGDA max; |1 + nd'e| for some 7) > 0 such that [1 + ndel 1 for all t. Let X and My be the eigenvalues of (Hess, - grad Hess, -1 gradz and ( respectively. Hence for strictly local minimax point x- we have At H4 > 0 and we can see d E {- U {-
#k+1 Exps,. (17 Gnrn(<x)lgradd/
Thus the couvergence rate PRTGDA = max; [1 - 7r,, V max; |1 mHl-- The requirement on such that PRTGDA < is again 1 FAIVPi This proves the convergence rate for RTGDA. For RFR, the proof is exactly the same.
]
Proof of Theorem
The derivative of (ENFF([rad]] at x* is given by
([GRNFR[gradd = CRNER( (e))S
-rid = (r(Hesss If o f) -CHess, '5(())
0
Haase grad yo f2²)
f(2")
grad H
which is similar to
id (Hess, f grad yar ff (z*) id)
0 GRNER(= (=s)) (Hess,, fo grad (2t) id (2²))
-id
0
-T grad? o Hess, If grad (x*) rgradd,
-(id
Thus, the eigenvalues of V ([RNnnn(grad//)e grad (rad)))((")) (=*) and -çid. If 2² is a strict local minimax, then we have all cigenvalues of V(GnNNFigraa ()r)d))(=)))) negalive, which suggests x* is a strictly stable point. On the other hand, if" is a strictly stable fixed point, then -7 -H(ess, grad? o Hess," fo gradzz)(=*) < 0 and hence (GRNFR [grad [[Gnnr[[grad/]]
are the union of eigenvalues of -HessH,,HHH------- -He-Heess, Hess y fo
has all eigenvalues negative. Similar as before, the asymptotic convergence rate is given by max; 1 - 7T A/| V max; [1 - 75 where X; are the cigenvalues of ( grad? o Hess, f o grad Suppose we choose Ç 1/n. then the convergence rate simplifies to max, 1 - v/ where r) < Fλ1 to ensure the rate is less than one.
For the RNTGDA algorithum, we derive
V ((RNTTDDA [grad = GRNTADA (3" (z") HHess (2²)
(-rid rid rggradhy o Hess, f)(zt)
Hess, ff (2
gradiy fz*)
-CHess, f(z²)
gradd yz f(2))
Hess, (3*)
T (Hess, gradk o Hess, y If grad? yx 1 (2*) -CHess
-1 ʃ( ((=))rrad ( f(*))
-iid
which has the same eigenvalues as ((G))))lgrad()((*))) Thus the same results follow.
Proof of Proposition 15 We compute the derivative of H at stationary points 8 as
VH(=') (Bid (VG(2")))) VG(=").
First, we analyze the asymptotic behavior of RHM (i.c., when 3 = 0). We see RHM converges to a stationary point as long as VG(=") is invertible, and the asymptolic convergence rate indeed linear that depends on the spectrum of (2*s) More superficially for RHM, id + (=)) = id 7 n(VG(2²)) VG(2")
jd=n(Hesssf(="))) As long as Hess f==*) is invertible, then we see 2" is strictly stable fixed point of RHM. Let 6, be the cigenvalues of Hessf(>*). Then the asymptotic convergence rate is given by PRHM max, |1 n and 1) < m7. However, because the only condition on z" being a strictly stable fixed point is that Hess (==) is invertible. It. may happen that =" is a local minimum, local maximum, local minimax or local maximin.
For RCON with 3 > 0, we have id + 1/ (2*)) id + GD)(((> my( (VG(2))) VG(2*). To analyze the asymptotic behavior, we first consider the real part of eigenvalues of VH(="), i.e., ((()))())"))) From
28

---

Published in Transactions on Machine Learning Research (08/2023)
classic results Garren
1968
real parts of cigenvalues, have Amin(1]s - R(((A)) >mmxx({A)s)
where we denote (A)s = (4 + A/)/2 for any linear operator A. Then we see
V (VH()))) -Hesss(="))) (-Hess,((3))
0
Hcss,
0
Further have from Merikoski & Kumar
2004
Theorem 1)
max
(=)))ss $ min of (mmin vy A min pt))
Amin (VH()=))))))))))))))))))))))"())
max o7 - (mmxx max ue)
where we let Uj Ae be the eigenvalues of Hess, and ( respectively. A sufficient condition for 2 to be strictly stable fixed point of RCON is thus min; 8, + { Uy A ming pt) > 0. Similar the case of RHM, there are many such points as long as 82 is sufficiently larger than the negative -Hees, f(*)., On the contrary, a sufficient condition for to be unstable is where max; 6? ((max, U2 max; (e) < 0.
spectrum of f
J Proofs of Section 7
Lemma 4. The Riemannian gradient and second-order derivatives for the logdet quadratic problem f(X, = ci clog + 02 log (X) log det (Y) det)
are derived as follows, where we let (X, Y) for simplicity.
grad,,(()) (2c) log det((et(tXX + log det (Y)), f(Z) (2c3 log det(Y))))))))) (Y) + log (e) Hess, / [Z) 20 20 tatttrXX--o UUX, Hess,, [Z)[V] -2cgtr(Y-
grad? my (Z)]VV = o2tr(Y- grad? f = (xtr((X'U))
Proof of Lemma
The Riemannian gradient is given by
grad = XV X(2e; log (X) c2 log dY) -)X
(2c) log det
(X) (X) + log (Y))
and similarly for Y. For Riemannian Hessian Hess (()) for any U E Tx + use the formula Heessff(Z)[ = f() - (UX--Iurdd
which gives
Hess, [(Z)(U] 2x (UXX + 2c, log (e) log (UX (UX-" = 2entr((X-U))
The cross derivatives are given from the definition as ff = [VV - Similar arguments hold for [Z)s and grad
Lemma 5. The Riemannion gradient and second-order derivatives for problem i.e.,
Y)X,Y) c1 (x) C2 log dot dct( cydist YI
are derived as follows.
grad_f(Z) (2c, log det X) 02 log det (Y))x,
g( = e2 log ( X)Y 2cgLogy (I) e2 lot 20g Ylogm(Y-")
Hess, f (Z) = 20,tr(Xx (X (
Hess, [Z) = -2ss (Vlogo( Dlogm D + 2ca VV Ylogm (
g (Z) -V)),
29

---

Published in Transactions on Machine Learning Research (08/2023)
Proof of Lemma Given we only change the quadratic term Y, the only differences come in the compu- tation of grad and (Z)) It. has been shown by Alimisis et al. (2020) that the Riemannian gradient of the squared Riemannian distance grad dist (A, B) 2Log (B). Substituting expression for grad, In terms of the Riemannian Hessian, i.e., Hess for any V E Ty we use the same formula as in Lemma 4 which leads to the desired expression.
we obtain the
]
Lemma 6. The Riemannian gradient and Riemannian second-onder derivatives of the robust Fréchet mean (RFM) problem 5
J(M,x)=xX =x Mx a
dist (M,M,)
are computed as
2a
(radM/(M,X
= Mxx MM +
Jogin(MM; MM)s,
gradz/(M, x) =2(1 xx xxT]Mx
n
20
(foss/((,xx) {Uxx M)s
Dlogm (MM logm(MM,,
-UM (logm(MM, }M]}]
[lessx/(M,,X) = 2(1 xx JM 2(x Mx)v
(f = Mvx M Mxv
gf 2(I -xx Ux
Proof of Lemma 0 First, we recall that the Riemannian gradient and Hessian on a sphere are given by x fx)vv.
gradg(x) = (I - xx Vg(x) and for any v E T2S4, Hxssg(x)[v) = (I-xs xx xx V> Substituting the results /M/ x) 2Mx and V², x) 2Mv in the Riemannian gradient and Hessian, obtain the desired expressions. Then for gradx.M/ (M, UU DMgrad, x) U) 2(I xx Uxx
The Riemannian gradient for M can be derived as
2a
2x
ffa = Mxx M Loog (M.) = Mxx M
Mllgmm [ MM)}s.
We highlight that the second term the expression of grad (M, x) can be costly to evaluate because M-1 needs to be computed every iteration. To simplify the computation, we can show {Mlogm(M (M-)M:))s [loogm(MM; MMs ") where M.¹ can be pre-computed.
To see why this is the case, we consider the generalized eigenvalue problem M,Ø AMΦ. Since M,M, E S A 0 and have M-'M, XΦ and '. MM >- Φ. Written compactly, let + be the matrix collecting & and thus we have M- $A and M, ' M -A-1ø-1 where A 0 is the diagonal matrix collecting the generalized eigenvalues. Then
Mlogm(MM'MM M$ (og(A)==1=== - -1--0----1
- log - M) = - (MMM)M)
Hence, we can see (Mlogm(M |M)]s = -(logm M)s, and therefore,
2a
gradmf(M, = -Mxx TAxXTM- T
flogm (MMM
n
30

---

Published in Transactions on Machine Learning Research (08/2023)
We derive gradM (M,x) x)vv ( Mvx M Mxv 'M. It only remains derive Hessm /M,, x)[,,, which is
Hess (M,
Da (M, 1 {UM 'grad (M,
20
2[Uxxx MM)s (Uxx M)s MMss
[Dlgg M + logmo (MMMMM,))
n
-IUM- (MMMMMMMM
20
M}s (Uxx [UxxTM}s +
Dlogm (MMM + logm(MM,
UM (logmMMMM
M)
The Riemannian gradient and second-order derivatives expressions for the robust marimum likelihood estimation problem
Lemma 7.
n f( = log det 2
Σ
[y: 8) 6)
1=1
are derived as
(o,d = -5- (Σ
yi - no).
gradg/(ō,S) = -
"s
yy: 8)(y; 88)
2 1=1
(=1
(rad),s((o,SS(55 = -I - 58 SSS11
Σ yi nō). gradg,s(66,S]( = Σ (ō - -8) 1s
1=1
(6,s)f( Σ (SS-
5)(y, -6) -s)
n Hess& ( nI 56 5 5" 33S--1 (S-1 1-
< (yx-no)) nō))ō
Proof of Lemma 7 From the definition of the Riemannian gradient for 8d $4, the Riemannian gradient can be easily derived, and similarly for the cross derivatives. For the Riemannian Hessian,
Hessg (6,,)) Degradg S -Ss-ig f(
Th ( " 1-"5--"-
Σ i=1
Ss Ss -y, - 5)(y, 5) 5)T}s
1 Σ
SS (y: -)) -8)Tss
i=1
(ess) -I- - 58 )331(, 8 Vof(6, - 1 8 (S
Σ 1=1
y: no n
The proof is complete.
Lemma 8. The Riemannion gradient and Riemannian second-order derivatives of Projection robust. Wasser- stein distance, i.e.,
( x; U +l -1))
1(,
mm
31

---

Published in Transactions on Machine Learning Research (08/2023)
are computed as follows. First, the Riemannian gradient and Riemannian Hession be computed as long as the Fuclidean gradient and Hession are compuled, i.e.,
VuS(T,U) 2 (x diag( Tl + diag(T didig 1m)Y - x -
Vr.(T, U)) = (diag XUU X ) + + AediagYYUU - -XUU Y1) + c log(T),
mn
mn ViS(T, V
X
X Y Y diag(T 1m)Y -
VIT =
where we denote 2 as elementuise division. In addition, the cross derivatives are given by
X grady [ 2 (I- ( x diag X d'diag(T 1mm - x Dolli = PTT
where Pr(V) the orthogonal projection of V Rmxx to the langend space TrI
To derive the gradient and second-order derivatives, we first rewrite the objective follows. Let X E Roxd Y Raxd be the data matrix. Then the objective is
Proof of Lemma
1 (r, diag XXUU X +1 diag dUU YUUTYY) 2XUU Y YT) (o(T,log(T) 1 1m
f(r, U)
mn
The Euclidean gradients are given by
2
Fuf((.U) = X T diag (T,, Y diag(T -X
1
Vrf(I, = (daa (diag(XUU') XUU 11, + 1 ding YUU Y -XXU Y + e log(T).
mn
From the derivations of Riemannian gradients of the Stiefel manifold and doubly stochastic manifold, have graduf(T, U) (I -U Vu')V,)(U, and f U) = (T Vrf(T, U)) = (Vrf(T,
al 1m where a E R", B e 12 are solutions from the following linear system, i.e.,
a@aat (T Vrf(T, U)) Bobtr a (T @ Vr/(T, U)) 1n.
For the second-order derivatives, first we see that it is casy to derive the Riemannian Hessian Hessu f(T, U)( and Hessr [T, U) T) for any U TuSt(d, I Tr v) as long the Fuclidean Hessian can be derived. To this end, obtain
VUST U)Ú] 2
(x diag(T1,) Y Yddigg( 1m) -
mn
VH, T = cTT r.
For the cross-derivatives, we first compute
UT mn - (x [ (XTdiag(P1,)) + Y (Tdiag( - - -
To derive gradf, where we omit the evaluation of function at (U,T) for clarity. Then we have
fufdd (T', U)] we let fi = -
0 and f2 = -
aoooV 1mm
Dufi(ū Dua o a rD. (T Do Du /ū) =
Duf2[U] Du D- (r DUV 1m 0.
32

---

Published in Transactions on Machine Learning Research (08/2023)
Thus can compute Dua [Ū] and Duß[U] from the above linear system. Then we have
(fT, = o (DuVrf(ū) - Dua[U]1] - (Io
where DU (U) -
.mn (diaa J y + (Y(U) ex(UÚTssY]] This is
equivalent to gradi U) = Pr @ Du DvVU..
]
33


Reviews:
[{"review_id": "g5DvbzpDJd", "reviewer": "Reviewer_Aujm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "See Below", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "See below", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I have had the opportunity to go through the paper in question and found the concept to be quite interesting. However, I must express my concerns regarding the presentation of the work. From a reader's perspective, I found the paper's language and structure a bit challenging to navigate.\n\nA particular issue I observed is related to the writing style. While the mathematics within the paper are correct, the lack of contextualization and intuition prior to each proof creates a barrier for the reader. To be able to appreciate the content fully, readers would greatly benefit from a more explanatory approach.\n\nMoreover, a significant aspect that is missing from the paper is the visual representation of their algorithm. By incorporating geometrical aspects and some illustrative plots, the authors could make their work more digestible and didactic. This will not only enhance the readability of the paper but also engage the audience in a much more effective manner.\n\nIn my previous experiences with TMLR, the emphasis has been on mathematical accuracy as a sufficient condition for publication. While I understand the importance of correctness, I firmly believe that effective communication and accessibility of the work to the wider scientific community are equally crucial.\n\nThus, In my thorough perusal of your work, I have identified a few areas that I believe could be significantly improved to enhance the reader's understanding and the overall clarity of the manuscript.\n\nFirstly, the inclusion of illustrative diagrams or plots which depict the geometrical facets of the algorithm would provide a substantial visual aid and support to the reader. This would not only clarify the working mechanism of the method but also aid in breaking the monotony of a text-intensive explanation. The addition of such visual elements is highly encouraged.\n\nSecondly, it is imperative to introduce textual guidance or \"foreshadowing\" before each lemma in the appendix. This approach will provide a roadmap to readers on what to expect in the upcoming proofs. Without this navigational aid, readers might find themselves lost in a sea of symbols and equations, thereby failing to grasp the significance of the proofs. It's essential to remember that we're communicating a complex theory, not merely demonstrating algebraic prowess.\n\nLastly, I would like to see a future challenges section integrated into the manuscript. This section should consolidate observations from the experimental comparisons between different methods, delineating which aspects of these comparisons have been sufficiently elucidated by your theory or extant literature. More importantly, it should highlight which areas are yet to be adequately explained, posing them as open problems for future work. This would not only underline the current limitations of the study but also open new avenues for further research, demonstrating an appreciation for the ongoing evolution of knowledge in this area.\n\nRemember, an academic manuscript should not only be a conduit of information but also a tool that educates and stimulates curiosity in the reader. Thus, adhering to these suggestions will significantly enhance your manuscript's pedagogical value and its scholarly impact.\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wRtlHJ5Spy", "reviewer": "Reviewer_z28z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper includes a wide range of results, which provides a big picture for the topic.  \n\n2. The examples in experiments show the meaningfulness of such a setting, even though most results are extended from Euclidean space. I believe the setting may have more applications in the future. \n\nWeakness: I have several major concerns about the paper. \n\n1. I think the paper missed an important reference: \n\n    [1] Tanner Fiez, Lillian J. Ratliff. Local Convergence Analysis of Gradient Descent Ascent with Finite Timescale Separation. ICLR 2021.\n\n    [1] shows the local behavior GDA when the separation $\\tau$ does not need to converge to 0, while similar results in [Jin et al., 2020] need $\\tau \\rightarrow 0$.\n\n2. Regarding the main result of Theorem 1:\n\n     a.  The complete proof of Theorem 1 is not presented in the paper. It states that the result of stable points follows from [Jin et al., 2020]. The theorem in the paper holds for a finite $\\tau_0 > 0$. However, based on my understanding, Theorem 28 in [Jin et al., 2020] requires $\\tau \\rightarrow 0$. \n\n     b. The second half of Theorem 1 states TSRGDA converges to local minmax points with a rate. However, as the stable equilibrium of TSRGDA is a superset of local minmax points, how can we guarantee that it only finds local minmax points? I also wonder why it does not depend on the initial point. If the algorithm starts from a point with derivatives 0 but not a local minmax point, will it stop there?\n\n     c. I didn't follow what the $\\delta$ is in this theorem, and how it affects the relationship between local minmax points and stable points.\n\n     d. I suggest adding the requirement for $\\eta$ in the theorem.\n\n3. At the beginning of Section 6, it states that \"without requiring a sufficiently small stepsize for the minimization variable as in TSRGDA\".  I didn't follow this sentence. If Theorem 1 is correct, there exists $\\tau>0$ for TSRGDA. In Theorem 2 and 3, there are also requirements for $\\eta$ and $\\tau$ (hidden in the proof. I suggest writing them in the statement of the theorems). \n\n4. For experiments, I wonder why some algorithms stop earlier, e.g., the blue line in Figure 1(d). I also suggest using some different colors in the experiments, since some of them are similar to each other.  \n\n5. I also think it is necessary to compare with the following papers. \n\n   [2] Constantinos Daskalakis, Stratis Skoulakis, Manolis Zampetakis: The Complexity of Constrained Min-Max Optimization. STOC 2021. \n\n   [3] Gidel et al. Negative momentum for improved game dynamics. AISTATS 2019.\n\n   [3] gives a bilinear minmax problem that GDA with any time separation might not converge. I wonder if such a pathological example is excluded by the assumption in this paper. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper focuses on solving nonconvex-nonconcave min-max optimization problems on Riemannian manifolds.\n\n1. Section 4 provides the definitions and necessary conditions of local solutions and illustrates them in examples in Prop 4. This is an extension of previous results to Riemannian manifolds.\n\n2. Section 5 first provides the definitions and characterization for strictly stable points for continuous and discrete dynamics, and proves the relationship between them. After that, it discussed the relationship between the limit point of TSRGDA and local minmax points and provides a local convergence rate. \n\n3. Section 6 further considers several second-order methods, and provides a relationship between their stable fix points and local minmax points.\n\n4. Section 7 conducts numerical experiments to compare the algorithms above. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weakness part above. ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "g1gnXcpLww", "reviewer": "Reviewer_yZgL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:** This is an excellent work that studies a nonconvex-nonconcave minimax optimization problem on Riemannian manifolds. This problem is challenging and under-explored but has important applications including robust Fréchet mean, projection robust Wasserstein distance, and orthonormal GAN. This work makes abundant and non-trivial novel extensions from Euclidean to Riemannian manifolds as summarized above in the ''Summary Of Contributions''.  The lit review, preliminary knowledge, novel notions, theories and experiments are presented clearly in a well-organized structure. Honestly, I am a novice in Riemann manifold. However, after reading this paper, I gain much knowledge about Riemann manifold. I did not check the proof but I can see that the proposed notions and theorems in this work are reasonable as they can reduce to what I already know on Euclidean space. Four (abundant) experiments cover both synthetic and application settings and the results are convincing. \n\n**Weaknesses:** As a novice in Riemannian manifolds, I cannot find any major weaknesses of this paper. A few details could be clarified more, and some typos could be corrected, as I listed in the ''Requested Changes'' below. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work extends the notions of local minimax points and stability of continuous and discrete dynamics for a nonconvex-nonconcave minimax optimization from Euclidean space to Riemannian manifolds. Equivalent conditions of stability are also given. Then this work obtains asymptotic convergence rates of the existing first-order TSRGDA algorithm and the proposed second-order algorithms.  ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Requested changes or answers (critical for acceptance):** \n\n(1) The discriminator prediction in Figure 3 looks confusing to me. Could you explain the meaning? For example, does the discriminator classify between the real points from the Gaussian mixture and fake points from noise $z_i$? Does the pixel strength mean the probability of being classified as real points? \n\n(2) In the RNGD algorithm, why do you use $-\\eta$ not $+\\eta$ to maximize over $y$? \n\n(3) The algorithm proposed by (Gao et al., 2022) is called Hessian FR instead of Newton follow-the-ridge. In that algorithm, the update rule of $y$ is $y_{t+1}=y_t+\\eta_{y1}\\nabla_y f(x_t,y_t)-\\eta_{y2}H_{yy}^{-1}\\nabla_y  f(x_t,y_t)+\\eta_x H_{yy}^{-1}H_{yx}\\nabla_x  f(x_t,y_t)$. Therefore, shouldn't we use $\\text{id}-\\zeta\\text{Hess}_y^{-1}f(z_k)$ to replace $-\\zeta\\text{Hess}_y^{-1}f(z_k)$ in your RNFR. \n\n**Requested changes or answers (not critical for acceptance):**\n\t\n(4) It seems that the global minimax point on the Riemann manifold is also proposed by your work, yes? If so, why not add that to the abstract, the research question, and the list of contributions in the Introduction? \n\n(5) In Theorem 1, if $x$ and $y$ have different dimensionality, then $\\text{grad}_{yx}^2 f(z)$ corresponds to a non-square matrix in Euclidean space and is thus not invertible so that LMiniMax=TSRGDA, right? \n\t\t\n(6) Typo: In the final paragraph of related works, add ''s'' after the verbs, including ''generalizes'', ''completes'', ''extends''. \n\n(7) Typo in Theorem 1: $\\mu_1\\ge \\cdots\\ge \\mu_{d_y}>0$. ", "broader_impact_concerns": "I did not find out any ethical concerns. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on solving min-max optimization problems on Riemannian manifolds, where the objective function is generally nonconvex and nonconcave in both the Euclidean and geodesic contexts. The authors (i) provide definitions of local optimality for such nonconvex-nonconcave min-max problems on Riemannian manifolds, (ii) establish stability and (local) asymptotic convergence rates for both continuous and discrete dynamical systems on these manifolds, and (iii) propose second-order methods that are provably guaranteed to asymptotically converge to local minimax points. The results are supported by detailed proofs and experiments involving a variety of nonconvex-nonconcave problems. 

Nonconvex-nonconcave min-max optimization problems on Riemannian manifolds can be challenging and have remained relatively under-explored. This study introduces novel extensions from the Euclidean to Riemannian manifolds, as elaborated above. The reviewers have acknowledged these contributions and have either recommended acceptance or are leaning toward acceptance. I concur with the reviewers' assessment and recommend accepting.
Decision: Accept (poster)