Title: Finite-Time Analysis of Stochastic Nonconvex Nonsmooth  Optimization on the Riemannian Manifolds
Venue: NeurIPS 2025
Accepted: yes
Keywords: Riemannian Optimization

Nonsmooth Optimization

Nonconvex Optimization
Abstract: This work addresses the finite-time analysis of nonsmooth nonconvex stochastic optimization under Riemannian manifold constraints. We adapt the notion of Goldstein stationarity to the Riemannian setting as a performance metric for nonsmooth optimization on manifolds. We then propose a Riemannian Online to NonConvex (RO2NC) algorithm, for which we establish the sample complexity of $O(\epsilon^{-3}\delta^{-1})$ in finding ($\delta,\epsilon$)-stationary points. This result is the first-ever finite-time guarantee for fully nonsmooth, nonconvex optimization on manifolds and matches the optimal complexity in the Euclidean setting. When gradient information is unavailable, we develop a zeroth order version of RO2NC algorithm (ZO-RO2NC), for which we establish the same sample complexity. The numerical results support the theory and demonstrate the practical effectiveness of the algorithms.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an Riemannian adaptation of the Online to NonConvex (O2NC) algorithm, namely the RO2NC algorithm. The proposed RO2NC algorithm introduces appropriate adaptions of retraction and vector transport to implement the essential algorithmic framework of the O2NC on an embedded manifold setting. The finite-time analysis of the proposed RO2NC algorithm for solving the optimization problem $\\arg\\min_{x\\in\\mathcal{M}} f(x)$ on a Riemannian manifold $\\mathcal{M}$ is presented with appropriate assumptions: \n* Assumption 2.4 demands a bounded second fundamental form of the manifold.\n* Assumption 2.5 proposes some constraints on the retraction map as an alternative of the Riemannian geodesic map.\n* Assumption 2.6 and 2.7 propose standard constraints on the non-convex objective function.\n\nIn particular, Assumption 2.4 should be highlighted, as it is a stronger assumption than the typical assumption on a bounded sectional curvature on a manifold, i.e., Assumption 2.4 yields a bounded sectional curvature but not vice-verse. Note that this Assumption 2.4 CANNOT be relaxed in the finite-time analysis. The finite-time analysis is presented on RO2NC algorithm with the following settings:\n* Theorem 3.1: Moving points with a retraction map, and moving tangent vectors with the parallel translation along a minimal geodesic.\n* Theorem 3.2: Moving points with a retraction map, and moving tangent vectors with the orthogonal projection (in the ambient space).\n* Theorem 4.2: Moving points with a retraction map and moving tangent vectors with the parallel translation, with the estimated gradient in use.\n\nThe analysis states that all of the above settings converge (in the sense of an adaptation of the Goldstein Stationarity criterion on the subdifferential set) with the theoretically optimal converging rate. Moreover, the rates seem to be in the same order under appropriate choices of parameters. Some numerical experiments are conducted (with different tuning parameters) on the sparse principal component problem on $n-1$ dimensional sphere. The reported results demonstrate converging trends. However, the converging rates or the finally converged stationary neighborhood are not addressed nor demonstrated.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, this paper has a clear structure and it reads well. Necessary background and related literature are reviewed and addressed appropriately. The effort in making appropriate adaptations of the O2NC and the Goldstein Stationarity criterion to the Riemannian setting is worth noting. The presented finite-time analyses are nontrivial yet not entirely convincing to me. In particular, the derivation that appeals to Theorem 2.8 is problematic to me, and the specific questions are elaborated in the following section. It hurts the credibility of this paper. The experimental section is less ideal. In particular, the presented RO2NC algorithm is not compared with existing algorithms, e.g., the basic stochastic gradient method, which makes it hard to interpret the optimal converging rates claimed in previous analyses. The demonstrated results stagger around the objective values of $0.1$, which does not demonstrate the converged stationary neighborhood.\n\nQUESTIONS:\n\\textbf{Most important question:} Theorem 2.8 states that \\|\\mathrm{proj}_{T_y\\mathcal{M}}(v) - P_{x,y}^g(v)\\|<C\\|v\\|\\mathrm{dist}(x,y)$ where $\\mathrm{proj}_{T_y\\mathcal{M}}$ is the orthogonal projector in the ambient space and P_{x,y}^g:T_x\\mathcal{M}\\to T_y\\mathcal{M} is the parallel translation along a minimal geodesic from x to y. Note that the parallel translation is path-dependent, i.e., the parallel translation along a different path that connects x and y (other than a minimal geodesic) is different from P_{x,y}^g. However, in the derivations that appeal to Theorem 2.8, e.g., line 863 and line 904, a geodesic polygon with vertices x\\to y_1\\to \\cdots\\to y_n\\to x is involved, where n \\geq 2. Take line 904 as an example that traversals the vertices x\\to y_1\\to y_2\\to x with the vector v parallel translated along the path, Theorem 2.8 should not be applicable to the composed parallel translations P_{y_2,x}^g\\circ P_{y_1,y_2}^g\\circ P_{x,y_1}^g. However, the derivation seems to be concluding as follows:\n\\begin{equation*}\n\t\\|\\mathrm{proj}_{T_x\\mathcal{M}}(v) - P_{y_2,x}^g\\circ P_{y_1,y_2}^g\\circ P_{x,y_1}^g(v)\\| < C\\|v\\|(\\mathrm{dist}(x,y_1)+\\mathrm{dist}(y_1,y_2) + \\mathrm{dist}(y_2,x)).\n\\end{equation*}\nDo the above bounds hold for any geodesic triangle or polygon with more vertices? If so, please state it appropriately as this is not what Theorem 2.8 says. If that is not the case, I do not see a workaround to get to the claimed results of the finite-time analysis.\n\nOther concerning questions are listed below:\n* In all of the derivations associated with the parallel translation, except for the one that appeals to Theorem 2.8, only the isometric nature of the parallel translation is used. Is the analysis applicable to any other isometric vector transport?\n* The vector transport by orthogonal projection is much less restrictive compared to the parallel translation, yet they both obtain the same converging rates as claimed. Are there any insights to explain this less intuitive result?\n* In Assumption 2.5, the constraint $\\|d/dt \\mathrm{Retr}_x(t\\xi)\\|\\leq \\|\\xi\\|$ is odd to me. We usually see assumptions about the $\\|O(t^2)\\|$ term in $\\mathrm{Retr}_x(t\\xi) = x + t\\xi + O(t^2) + o(t^2)$. Is the assumption proposed in this paper met in general? \n* The numerical experiment section is weak in its current state. It is only showing different converging trends (depends on the algorithmic parameter) that stagger around $0.1$. \n\nThe less concerning questions are listed below:\n* The online to non-convex framework even in the Euclidean setting is not that straightforward, not to mention its adaptation in the Riemannian setting that involves retraction and vector transport. It would be nice to include some pictures for visual aid.\n* Line 320 to Line 321 is confusing. Does that mean the analysis is based on the parallel translation, and the analysis does not apply to the implementation?\n* Line 328: What is a two-point bandit feedback?\n* Typos: (Line 889) the square of $\\|\\Delta_t-u_t\\|$ is missing; (Line 906) Assumption 2.5.\n\nLIMITATIONS:\nThis paper discussed its limitations, i.e., how the curvature of a manifold affects the analysis, is an open question.\nThe main problem of this paper is stated in the \"Questions\" box.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission studies the problem of minimizing a nonconvex and nonsmooth stochastic function $f$ over a Riemannian manifold $\\mathcal M$. The development is similar to that in the Euclidean in the sense that one must work with the Goldstein subdifferential to obtain meaningful results. Explicitly, the notion of stationarity used herein requires defining the $\\delta$-subdifferential at $x\\in\\mathcal M$ as the closed convex hull of the set $\\cup_{y\\in\\mathrm{cl}(B(x,\\delta))}P^g_{y,x}(\\partial f(y))$ where $P^g_{y,x}$ is the parallel transport taking the tangent space at $y$ to the tangent space at $x$. Then, a point is $(\\delta,\\epsilon)$-stationary once its $\\delta$-subdifferential contains a point which has norm at $\\epsilon$.\n\nWith these definitions in place, the paper enumerates a number of technical assumptions related to 1) boundedness of the second fundamental form of $\\mathcal M$, 2) the quality of the retraction in approximating the exponential mapping, 3) Lipschitz continuity of the deterministic part of the objective, 4) unbiased of the stochastic oracle and boundedness of the second moment of the gradient.\n\nThe proposed method is inspired by the Online to NonConvex (O2NC) approach and uses retractions for the updates to ensure that the iterates remain constrained to the manifold and uses parallel transport or projections to compare points from different tangent spaces. The corresponding algorithm (RO2NC) is studied under both of these settings (parallel transport or projections) yielding similar theoretical guarantees (though different techniques are used in their analysis).\n\nFinally, a zeroth order method is proposed which admits the same overall iteration complexity.\n\nSTRENGTHS AND WEAKNESSES:\nThe main problem at hand, nonconvex and nonsmooth optimization over Riemannian manifolds is of a great interest. This is, to my knowledge, the first paper to provide rigorous iteration complexity results and does so with zeroth and first order methods. While the overall approach is similar to O2NC, the modifications required to conform to the Riemannian setting are not straightforward. Furthermore, I find the paper to be well-written overall and the development to be clear.\n\nI believe that the main limitations of this work regard the required assumptions; a concern which I expand upon in the questions below. \nI also note that some experimental validation of this theory is provided, but is contained in the supplement. I believe it would be relevant to include the experiment in the main text.\n\nQUESTIONS:\n1. For Assumption 2.5, the only examples provided are the polar retraction for the Stiefel manifold and the actual exponential mapping. Consequently, this assumption appears to be quite strong. Are there any other examples where this assumption holds?\n\n2. In Assumption 2.7, we require an unbiased estimate for the gradient. Some recent works in the Euclidean setting have managed to bypass the assumption that the oracle always returns the minimal norm subgradient. As such, I wonder if such techniques could be applied in the Riemannian setting as well.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This submission presents the first finite-time analysis of nonsmooth, nonconvex stochastic optimization problems under Riemannian manifold constraints. This fills a theoretical gap where previous literature had only considered smooth or asymptotic cases.

This submission has made following contributions: 

- Goldstein Stationarity on Manifolds: The authors adapt the Goldstein stationarity to the Riemannian setting, forming a meaningful convergence criterion for nonsmooth optimization on manifolds.

- It proposes the RO2NC Algorithm, which is an adaptation of the Online-to-NonConvex method to the Riemannian setting.

- It establishes strong theoretical results: a sample complexity of O(δ⁻¹ϵ⁻³), matching the best-known Euclidean bounds.

STRENGTHS AND WEAKNESSES:
I am not familiar with this field and cannot comment too much on the strength and weakness. But it seems to me this work is a solid contribution to the optimization community. The following contributes make sense to me:

- Adapting the notion of Goldstein stationarity to the Riemannian setting. 
- Adapting the O2NC algorithm to Riemannian setting. 
- A strong theoretical performance bound is established. 


Weakness:

- The empirical evaluation for this method is limited. I am sure how it indeed performs in realistic applications.

QUESTIONS:
I am just wondering its performance in realistic applciations.

LIMITATIONS:
NA
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents finite-time results for optimizing nonconvex, nonsmooth functions over Riemannian manifolds. The approach is built upon the recent online-to-nonconvex framework and existing work in Riemannian gradient sampling. The authors also discuss the zero-order case, drawing on known results from the Euclidean setting. While the core ideas may not be surprising, difficulties can arise when generalizing analysis from the Euclidean to the manifold setting.\n\nSTRENGTHS AND WEAKNESSES:\nI have two technical questions:\n* In Assumption 2.7, I don't understand the rationale behind the notation grad F(x,\\nu), which seems not defined. If it is meant to be the Riemannian gradient of function x \\mapsto F(x, \\nu) for fixed \\nu, then one possible issue is that why we can write E_\\nu [grad F] = grad f. This implicitly assumes F(., \\nu) is differentiable at x for D-a.s. \\nu, needs a careful justification, and may not be true even in the Euclidean setting.\n* Another issue is on the evaluation of g_{k,t} in Algorithm 1. How can you ensure the function F(c, v_{k,t}) is actually (Riemannian) differentiable at the point w_{w,t}? If it is not, then such an evaluation seems rather intractable. This issue may be due to the fact that the curve s_{k,t} \\mapsto Retr_{x_kt}(s_{k,t} \\Delta_{k,t}) is not sufficiently generic, which allows F(., \\nu_{k,t}) to be non-differentiable at any point along it.\n\n\nOther comments:\n* the objective f is not defined in Eq.1\n* L55: weakly convexity -> weak convexity\n* L157: I cannot see why B(x,\\delta) is an open ball.\n* Definition 2.1. It may be better to clarify the meaning of \\lim_l grad f(x_l), since {grad f(x_l)}_l are in different tangent spaces.\n* For the known facts mentioned in Section 2.2, it would be better to provide concrete citations for verification and reference; e.g., L178--L182 and L185--L189. \n* Definition 2.3. Why is the function f defined on an Euclidean space?\n* the function f may need to be assumed to be lower bounded on manifold M; otherwise, C1, C3, C5 may not be upper bounded wrt T.\n* In Lemma 2.9, what are {w_t}_t? You may need to define them before using them to define \\nabla_t. Besides, from the proof, you may need additional assumption like dist(w_t, y) <= \\delta. Also, RO2NC is not defined here, hence please try to avoid use that term in the proof.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}