Title: Adaptive Riemannian ADMM for Nonsmooth Optimization: Optimal Complexity without Smoothing
Venue: NeurIPS 2025
Accepted: yes
Keywords: ADMM

manifold optimization

iteration complexity

adaptivity
Abstract: We study the problem of minimizing the sum of a smooth function and a nonsmooth convex regularizer over a compact Riemannian submanifold embedded in Euclidean space. By introducing an auxiliary splitting variable, we propose an adaptive Riemannian alternating direction method of multipliers (ARADMM), which, for the first time, achieves convergence without requiring smoothing of the nonsmooth term. In contrast to conventional Riemannian ADMM methods that require exactly solving a nested subproblem at each iteration, our approach involves only one Riemannian gradient evaluation and one proximal update per iteration. Through careful and adaptive coordination of the stepsizes and penalty parameters, we establish an optimal iteration complexity of order $\mathcal{O}(\epsilon^{-3})$ for finding an $\epsilon$-approximate KKT point, matching the complexity of existing smoothing technique-based Riemannian ADMM methods. Extensive numerical experiments on sparse PCA and robust subspace recovery demonstrate that our  ARADMM consistently outperforms state-of-the-art  Riemannian ADMM variants in convergence speed and solution quality.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of minimizing the sum of a smooth function and a non-smooth convex regularizer over a compact Riemannian submanifold embedded in Euclidean space. The authors propose an adaptive Riemannian alternating direction method of multipliers (ARADMM), which achieves convergence for the first time without requiring the smoothing of the non-smooth term. In contrast to conventional Riemannian ADMM methods that require solving a subproblem exactly at each iteration, the proposed approach involves only one Riemannian gradient evaluation and one proximal update per iteration. A sophisticated dual step size design ensures the convergence of the algorithm. Finally, extensive numerical experiments demonstrate the superiority of ARADMM in terms of both convergence speed and solution quality.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\nThe structure of the paper is well-organized, and the logical flow is clear and coherent. The authors present, for the first time, an algorithm that achieves the convergence complexity of smoothing-based methods without the need for smoothing techniques. The numerical experiments demonstrate impressive results, further validating the proposed method. Additionally, the clever dual step size design provides a guarantee for controlling the boundedness of the Lagrangian multipliers, which seems to be an innovative contribution.\n\n### Weaknesses\nWhile the overall logic of the paper is clear, there are some areas that could be improved:\n- Some equation numbers, such as those in equations (11) and (13), appear to be unnecessary. These redundant labels impact readability.\n- In line 60, the discussion on “the best-known complexity achieved by smoothing-based Euclidean and Riemannian ADMM methods” could benefit from the addition of references to support the claim.\n- The usage of $\\tilde{\\mathcal{O}}$ should be explained for readers who are less familiar with complexity notation.\n- In line 172, the formula should be $\\lambda_k - \\lambda_{k+1}$, and the final inequality should be an equality.\n- The equality in equation (18) should be defined by the symbol used in the paper.\n- In line 405, the definitions of various parameters should reference their initial definitions, and in line 431, the symbol $D$ should refer back to its previous definition for clarity.\n- The paper would benefit from reducing typographical errors. There are some typographical errors:\n    - In equations (1) and (3), the periods should be replaced with commas.\n    - In line 24, \"a linear mappings\" should be corrected to \"a linear mapping.\"\n    - In Proposition 2.1, the second inequality has an extra space.\n    - In line 430, $Q$ should be $\\alpha$.\n    - In line 439, \"Corollary\" should be changed to \"Theorem.\"\n\nQUESTIONS:\n- Regarding the dual step size $\\beta_k$, could you clarify the inspiration behind this choice? Is there any similar approach discussed in previous literature? Is there a similar step size design in the Euclidean case? Additionally, what is the role of the constant $\\log^2 2$ in $\\beta_k$? It seems that its critical function is not evident in the proof. A clearer explanation of this would help highlight the originality and significance of your work.\n- The choice of $c_\\tau$ appears to be a delicate adjustment in practice, especially since the Lipschitz constant of the function may not be easily known. In your experiments, did you test different values of $c_\\tau$? How do the results vary with different selections of $c_\\tau$? Clarifying this aspect would further demonstrate the practical applicability of your algorithm.\n- I noticed that in the Sparse PCA experiments, you compared your method with ManPG. As far as I know, the method AManPG in reference [19] achieves faster and better results for Sparse PCA. Why was this method not included in your comparison? Additionally, it seems that comparisons with ALM-based algorithms could also be relevant, such as those in [41] and [Zhang, C. et al.]$^1$. Furthermore, the experimental dimensions appear to be limited to less than 1000. Have you conducted experiments with larger dimensions? Providing an explanation for the exclusion of these comparisons, or additional experiments, would help us better understand the advantages of your proposed algorithm.\n- In the proof section, is there a complete proof or a reference for Lemma A.1? Also, the condition $c_\\rho \\geq 1$ seems to be incorrect and should be $c_\\rho \\leq 1$, which matches the value used in your experiments. Please carefully review the corresponding locations and the relationship between $\\tau_k$ and $\\rho_k$. Furthermore, equation (29) seems to be missing a $\\lambda_0$, and the summation symbol in the second inequality should have the index $l=0$ instead of $l=1$. In line 421, the first equation should be an inequality, and in line 418, $\\ell_k$ should be replaced with $\\ell_{k-1}$.\n\nReference\n1. Zhang, C., Xiao, R., Huang, W., & Jiang, R. (2024). Riemannian Trust Region Methods for SC 1 Minimization. Journal of Scientific Computing, 101(2), 32.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an adaptive Riemannian alternating direction method of multipliers to solve the problem of minimizing the sum of a smooth function and a non-smooth convex regularizer over a compact Riemannian submanifold embedded in Euclidean space. Unlike smoothed ADMM, the proposed method directly addresses the original nonsmooth problem. Through adaptive coordination of the step size and penalty parameters, ensuring the Lagrangian multiplier difference is bounded by the primal iterate difference, then establish an optimal iteration complexity of order $O(\\epsilon^{-3})$ for finding an $\\epsilon$-approximate KKT point. Numerical experiments on sparse PCA and robust subspace recovery demonstrate that ARADMM consistently outperforms state-of-the-art Riemannian ADMM variants in convergence speed and solution quality.\n\nSTRENGTHS AND WEAKNESSES:\nThe idea presented in this paper is great. The innovation lies in adaptively selecting dual step sizes and penalty parameters, explicitly bounding multiplier differences by the norms of corresponding primal iterate differences. This is crucial for ensuring convergence. But there are still some details missing in the writing, mainly in the theoretical proof part. The major concern also lies in the theoretical proofs of the main result.\n\nQUESTIONS:\n*) In line 6, RADMM proposed by Jiaxiang Li et al. in [28] also involves only one Riemannian gradient evaluation and one proximal update per iteration.\n\n*) In Section 1.2, the indicator function mentioned is not used in this paper and is recommended to be removed.\n\n*) Vector transport is an important concept in manifold optimization, which is used in this paper. Its definition should be explained in Section 2.1.\n\n*) Why does the inequality below Line 172 hold? Since the Lambda is a vector, does the inequality mean for entrywise inequality?\n*) Why does the second equation in (30) hold?\n\n*) Lemma A.1 is missing a proof or reference. Does the vector transport used in Lemma 1 need to be isometry, and if not, can the requirement of isometry in line 419 be relaxed to \\|\\mathcal{T}_{x}^{y}(u)\\| \\leq L \\|u\\|, where there L is some constant?\n\n*)  Lemma B.1., B.2., and Theorem B.1. and 3.1. all require c_\\rho\\geq 1, but the parameter c_\\rho is set to 0.6 in the numerical experiments for robust subspace recovery. Moreover, line 408 and line 420 both require c_\\rho\\leq 1, but line 418 needs \\rho_{k-1} = c_{\\rho}(k-1)^{1/3}>1 . I think the parameter c_{\\rho} was chosen incorrectly and needs to be adjusted. \n\n*)  During the proof of Theorem B1, there were several writing errors, which made reading difficult. Such as in the first inequality of the proof, \\rho_{k}->\\rho_{k+1}, \\rho_{k-1}->\\rho_{k}, (\\ell_h+\\lambda_{max})^2 is lost in the proof, and so on.\n\n*)  Some symbols do not indicate their meaning before they are used. In Definition 2.1, the meaning of the symbol T\\mathcal{M} is not specified. In inequality (44), the meanings of symbols f_* , h_* and \\mathcal{D} are not specified. Moreover, the symbol \\mathcal{D} also appears in line 142, but their meanings are different.\n\n*)  There are many typos in this paper; for example, in line 114. It is suggested to check again.\n\nLIMITATIONS:\nThis paper already states its limitation, i.e., its current analysis does not exploit structural properties like the Kurdyka–Łojasiewicz (KL) inequality, which could lead to improved convergence rates.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a Riemannian ADMM algorithm for optimizing the sum of a smooth function and the composition of a non smooth function with a linear operator, all of that restricted to be in a Riemannian manifold embedded in R^n. Access to the gradient of the former function and the proximal operator of the latter is assumed. Author show that their algorithm achieves a KKT point up to an \\epsilon in O(\\epsilon^{-3}). Experients are provided.\n\nSTRENGTHS AND WEAKNESSES:\nThis work is generally well written and the result seems interesting, but I missed an explanation comparing the analysis of this paper and the one of the Euclidean approach that this work is based on. There are a couple of minor problems / typos in the analysis:\n\nThe way the statement of Theorem 3.1 is phrased is inaccurate, since if the actual guarantee you prove is what you state, then the minimum of each of the three objectives could happen at different indices k, and it could be that for no single index the three terms are small at the same time. However, taking a look at your proof you can fix it and claim that you have a k such that the sum of all three terms is small.\n\nThere is an index problem in the proof of Theorem B.1. At the beginning of the proof, a property is shown for $\\mathcal{L}$ with rho having an index a unit less than the indices of x, y, \\lambda. However, between line 430 and 431 the indices for all of those variables become the same. This should be fixed.\n\nOne thing to note is that the method is claimed to be adaptive but the algorithm requires access to most problem constants like the smoothness of f, the operator norm of the linear map. The adaptivity refers to how the dual step size can adapt depending on the previous iterates but still depends on these constants.\n\nQUESTIONS:\nWhat are the main differences and new techniques of this work with respect to its Euclidean counterpart?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study nonsmooth nonconvex optimization over Riemannian Submanifolds. Canonical examples of such a problem include sparse and/or nonnegative PCA. An adaptive ADMM algorithm is proposed which notably does not necessitate smoothing the nonsmooth term. By carefully adapting the dual step size, ARADMM controls the bound on the dual variables and the difference between successive dual variables. This yields an optimal oracle / iteration complexity of O(e-3) (under certain flexible assumptions e.g. Lipschitzness and smoothness of the smooth term and convexity and Lipschitzness of the nonsmooth term). Compared to existing ADMM-based frameworks for nonsmooth optimization over Riemannian Manifolds (e.g. MADMM), the proposed method only requires a single projection (via the retraction defined on M) and avoids nested iterative procedures or complicated subroutines (in addition to not modifying the original problem, e.g. by smoothing). \n\nNumerically, the authors compare RADMM to four ADMM-based methods and two first-order nonsmooth manifold optimization algorithms. On Sparse PCA, RADMM achieves a marked improvement over other methods in terms of solution quality and empirical convergence rate. On a robust subspace recovery task formulated via dual principal component pursuit, RADMM also outperforms relevant ADMM-based methods and a recently proposed energy minimization scheme.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths** This is a well-written paper that contributes a new practically designed and provably convergent ADMM-based algorithm for nonsmooth nonconvex optimization on smooth manifolds. The derived method notably does not necessitate smoothing the nonsmooth terms in the objective while maintaining numerical efficacy.\n\nNumerically, the method shines over relevant ADMM and first-order methods for optimization over smooth manifolds in terms of simplicity of implementation, solution quality, iteration complexity, and runtime. \n\n\n**Weaknesses** I am admittedly not up to date with the literature, but it seems that the author is missing some key references, including Wang, Yin, Zheng Global Convergence of ADMM in Nonconvex Nonsmooth Optimization, which also addresses nonsmooth nonconvex optimization of an ADMM-based method. It would be good to contextualize the work the authors have done with this important paper. Similarly, only two applications are discussed. It would strengthen the paper to include more experiments - e.g. nonnegative PCA as discussed in the introduction, or problems on manifolds other than the Stiefel manifold.\n\nQUESTIONS:\nSee weaknesses. I would like some contextualization of your framework with respect to some existing relevant missing references and more experimental results if possible.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}