ADMM for Nonsmooth Composite Optimization under Orthogonality Constraints

## Abstract

Abstract We consider a class of structured, nonconvex, nonsmooth optimization problems under orthogonality constraints, where the objectives combine a smooth function, a nonsmooth concave function, and a nonsmooth weakly convex function. This class of problems finds diverse applications in statistical learning and data science. Existing ADMMs for addressing these problems often fail to exploit the specific structure of orthogonality constraints, struggle with nonsmooth functions and nonconvex constraint sets, or result in suboptimal oracle complexity. We propose OADMM , an Alternating Direction Method of Multipliers (ADMM) designed to solve this class of problems using efficient proximal linearized strategies. Two specific variants of OADMM are explored: one based on Euclidean Projection ( OADMM-EP ) and the other on Riemannian retraction ( OADMM-RR ). We integrate a Nesterov extrapolation strategy into OADMM-EP and a monotone Barzilai-Borwein strategy into OADMM-RR to potentially accelerate primal convergence. Additionally, we adopt an over-relaxation strategy in both OADMM-EP and OADMM-RR for rapid dual convergence. Under mild assumptions, we prove that OADMM converges to the critical point of the problem with a provable convergence rate of 𝒪 ​ ( 1 / ϵ 3 ) 𝒪 1 superscript italic-ϵ 3 \mathcal{O}(1/\epsilon^{3}) . We also establish the convergence rate of OADMM under the Kurdyka-Lojasiewicz (KL) inequality. Numerical experiments are conducted to demonstrate the advantages of the proposed method.

## 1 Introduction

This paper mainly focuses on the following nonsmooth composite optimization problem under orthogonality constraints (‘$\triangleq$’ means define):

$$ $\displaystyle\min_{{\mathbf{X}}\in\mathbb{R}^{n\times r}}\,F({\mathbf{X}})\triangleq f({\mathbf{X}})-g({\mathbf{X}})+h({\mathcal{A}}({\mathbf{X}})),~{}s.t.~{}{\mathbf{X}}^{\mathsf{T}}{\mathbf{X}}={\mathbf{I}}_{r}.$ (1) $$

Here, $n\geq r$, ${\mathcal{A}}({\mathbf{X}})\in\mathbb{R}^{m\times 1}$, and ${\mathbf{I}}_{r}$ is a $r\times r$ identity matrix. For conciseness, the orthogonality constraints ${\mathbf{X}}^{\mathsf{T}}{\mathbf{X}}={\mathbf{I}}_{r}$ in Problem ([1](#S1.E1)) is rewritten as ${\mathbf{X}}\in\mathcal{M}\in\mathbb{R}^{n\times r}$, with $\mathcal{M}$ representing the Stiefel manifold in the literature [, ].

We impose the following assumptions on Problem ([1](#S1.E1)) throughout this paper. ($\mathbb{A}$-i) $f({\mathbf{X}})$ is $L_{f}$-smooth, satisfying $\|\nabla f({\mathbf{X}})-\nabla f({\mathbf{X}}^{\prime})\|_{\mathsf{F}}\leq L_{f}\|{\mathbf{X}}-{\mathbf{X}}^{\prime}\|_{\mathsf{F}}$ holds for all ${\mathbf{X}},{\mathbf{X}}^{\prime}\in\mathbb{R}^{n\times r}$. This implies: $|f({\mathbf{X}})-f({\mathbf{X}}^{\prime})-\langle\nabla f({\mathbf{X}}^{\prime}),{\mathbf{X}}-{\mathbf{X}}^{\prime}\rangle|\leq\tfrac{L_{f}}{2}\|{\mathbf{X}}-{\mathbf{X}}^{\prime}\|_{\mathsf{F}}^{2}$ (cf. Lemma 1.2.3 in []). We also assume that $f({\mathbf{X}})$ demonstrates $C_{f}$-Lipschitz continuity, with $\|\nabla f({\mathbf{X}})\|_{\mathsf{F}}\leq C_{f}$ for all ${\mathbf{X}}\in\mathcal{M}$. The convexity of $f({\mathbf{X}})$ is not assumed. ($\mathbb{A}$-ii) The function $g(\cdot)$ is convex and $C_{g}$-Lipschitz continuous, though it may not be smooth. ($\mathbb{A}$-iii) The function $h(\cdot)$ lacks both smoothness and convexity but exhibits $C_{h}$-Lipschitz continuity. Additionally, it is weakly convexity with constant $W_{h}\geq 0$, which implies that the function $h({\mathbf{y}})+\frac{W_{h}}{2}\|{\mathbf{y}}\|_{2}^{2}$ is convex for all ${\mathbf{y}}\in\mathbb{R}^{m\times 1}$. ($\mathbb{A}$-iv) The mapping ${\mathcal{A}}({\mathbf{X}}):\mathbb{R}^{n\times r}\rightarrow\mathbb{R}^{m\times 1}$ is $C_{A}$-Lipschitz continuous, satisfying $\|{\mathcal{A}}({\mathbf{X}})-{\mathcal{A}}(\mathbf{X^{\prime}})\|\leq C_{A}\|{\mathbf{X}}-{\mathbf{X}}^{\prime}\|_{\mathsf{F}}$ for all ${\mathbf{X}},\mathbf{X^{\prime}}\in\mathbb{R}^{n\times r}$. Additionally, ${\mathcal{A}}({\mathbf{X}})$ is $L_{A}$-smooth, ensuring the existence of the gradient mapping $\nabla{\mathcal{A}}({\mathbf{X}}):\mathbb{R}^{n\times r}\rightarrow\mathbb{R}^{m\times(nr)}$ and $\|\nabla{\mathcal{A}}({\mathbf{X}})-\nabla{\mathcal{A}}(\mathbf{X^{\prime}})\|_{\mathsf{F}}\leq L_{A}\|{\mathbf{X}}-{\mathbf{X}}^{\prime}\|_{\mathsf{F}}$. We further assume that $\|{\mathcal{A}}({\mathbf{X}})\|\leq R_{A}$ for all ${\mathbf{X}}\in\mathcal{M}$.

Problem ([1](#S1.E1)) represents an optimization framework that plays a crucial role in a variety of statistical learning and data science models. These models include sparse Principal Component Analysis (PCA) [, ], deep neural networks [], orthogonal nonnegative matrix factorization [], range-based independent component analysis [], and dictionary learning [].

### 1.1 Related Work

$\blacktriangleright$ Optimization under Orthogonality Constraints. Solving Problem ([1](#S1.E1)) is challenging due to the computationally expensive and non-convex orthogonality constraints. Existing methods can be divided into three classes. (i) Geodesic-like methods [, , , ]. These methods involve calculating geodesics by solving ordinary differential equations, which can introduce significant computational complexity. To mitigate this, geodesic-like methods iteratively compute the geodesic logarithm using simple linear algebra calculations. Efficient constraint-preserving update schemes have been integrated with the Barzilai-Borwein (BB) stepsize strategy [, ] for minimizing smooth functions under orthogonality constraints. (ii) Projection and retractions methods [, ]. These methods maintain orthogonality constraints through projection or retraction. They reduce the objective value by using its current Euclidean gradient direction or Riemannian tangent direction, followed by an orthogonal projection operation. This projection can be computed using polar decomposition or singular value decomposition, or approximated with QR factorization. (iii) Multiplier correction methods [, , ]. Leveraging the insight that the Lagrangian multiplier associated with the orthogonality constraint is symmetric and has an explicit closed-form expression at the first-order optimality condition, these methods tackle an alternative unconstrained nonlinear objective minimization problem, rather than the original smooth function under orthogonality constraints.

$\blacktriangleright$ Optimization with Nonsmooth Objectives. Another challenge in addressing Problem ([1](#S1.E1)) stems from the nonsmooth nature of the objective function. Existing methods for tackling this challenge fall into three main categories. (i) Subgradient methods [, , ]. Subgradient methods, analogous to gradient descent methods, can incorporate various geodesic-like and projection-like techniques. However, they often exhibit slower convergence rates compared to other approaches. (ii) Proximal gradient methods []. These methods use a semi-smooth Newton approach to solve a strongly convex minimization problem over the tangent space, finding a descent direction while preserving the orthogonality constraint through a retraction operation. (iii) Operator splitting methods [, , ]. These methods introduce linear constraints to break down the original problem into simpler subproblems that can be solved separately and exactly. Among these, ADMM is a promising solution for Problem ([1](#S1.E1)) due to its capability to handle nonsmooth objectives and nonconvex constraints separately and alternately. Several ADMM-like algorithms have been proposed for solving nonconvex problems [, , , , , , ], but these methods fail to exploit the specific structure of orthogonality constraints or cannot be adapted to solve Problem ([1](#S1.E1)).

$\blacktriangleright$ Detailed Discussions on Operator Splitting Methods. We list some popular variants of operator splitting methods for tackling Problem ([1](#S1.E1)). Initially, two natural splitting strategies are used in the literature:

$$ $\displaystyle\textstyle\min_{{\mathbf{X}},{\mathbf{y}}}\,F_{1}({\mathbf{X}},{\mathbf{y}})\triangleq f({\mathbf{X}})-g({\mathbf{X}})+h({\mathbf{y}})+\mathcal{I}_{\mathcal{M}}({\mathbf{X}}),~{}s.t.~{}{\mathcal{A}}({\mathbf{X}})={\mathbf{y}}$ (2) $\displaystyle\textstyle\min_{{\mathbf{X}},{\mathbf{Y}}}\,F_{2}({\mathbf{X}},{\mathbf{Y}})\triangleq f({\mathbf{X}})-g({\mathbf{X}})+h({\mathcal{A}}({\mathbf{X}}))+\mathcal{I}_{\mathcal{M}}({\mathbf{Y}}),~{}s.t.~{}{\mathbf{X}}={\mathbf{Y}}.$ (3) $$

(a) Smoothing Proximal Gradient Methods (SPGM, [, ]) incorporate a penalty (or smoothing) parameter $\mu\rightarrow 0$ to penalize the squared error in the constraints, resulting in the subsequent minimization problem [, , ]: $\min_{{\mathbf{X}},{\mathbf{y}}}~{}F_{1}({\mathbf{X}},{\mathbf{y}})+\tfrac{1}{2\mu}\|{\mathcal{A}}({\mathbf{X}})-{\mathbf{y}}\|_{2}^{2}$. During each iteration, SPGM employs proximal gradient strategies to alternatively minimize w.r.t. ${\mathbf{X}}$ and ${\mathbf{y}}$. (b) Splitting Orthogonality Constraints Methods (SOCM, []) use the following iteration scheme: ${\mathbf{X}}^{t+1}\approx\arg\min_{{\mathbf{X}}}F_{2}({\mathbf{X}},{\mathbf{Y}}^{t})+\langle{\mathbf{Z}}^{t},{\mathbf{X}}-{\mathbf{Y}}^{t}\rangle+\tfrac{\beta}{2}\|{\mathbf{X}}-{\mathbf{Y}}\|_{\mathsf{F}}^{2}$, ${\mathbf{Y}}^{t+1}\in\arg\min_{{\mathbf{Y}}}F_{2}({\mathbf{X}}^{t+1},{\mathbf{Y}})+\langle{\mathbf{Z}}^{t},{\mathbf{X}}^{t+1}-{\mathbf{Y}}\rangle+\tfrac{\beta}{2}\|{\mathbf{X}}^{t+1}-{\mathbf{Y}}\|_{\mathsf{F}}^{2}$, and ${\mathbf{Z}}^{t+1}={\mathbf{Z}}^{t}+\beta({\mathbf{X}}^{t+1}-{\mathbf{Y}}^{t+1})$, where $\beta$ is a fixed penalty constant, and ${\mathbf{Z}}^{t}$ is the multiplier associated with the constraint ${\mathbf{X}}={\mathbf{Y}}$ at iteration $t$. (c) Similarly, Manifold ADMM (MADMM, []) iterates as follows: ${\mathbf{X}}^{t+1}\approx\arg\min_{{\mathbf{X}}}F_{1}({\mathbf{X}},{\mathbf{y}}^{t})+\langle{\mathbf{z}}^{t},{\mathcal{A}}({\mathbf{X}})-{\mathbf{y}}^{t}\rangle+\tfrac{\beta}{2}\|{\mathcal{A}}({\mathbf{X}})-{\mathbf{y}}^{t}\|_{\mathsf{F}}^{2}$, ${\mathbf{y}}^{t+1}\in\arg\min_{{\mathbf{y}}}F_{1}({\mathbf{X}}^{t+1},{\mathbf{y}})+\langle{\mathbf{z}}^{t},{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}\rangle+\tfrac{\beta}{2}\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}\|_{\mathsf{F}}^{2}$, and ${\mathbf{z}}^{t+1}={\mathbf{z}}^{t}+\beta(A({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1})$, where ${\mathbf{z}}^{t}$ is the multiplier associated with the constraint $A({\mathbf{X}})-{\mathbf{y}}=\mathbf{0}$ at iteration $t$. (d) Like MADMM, Riemannian ADMM (RADMM, []) operates using the first splitting strategy in Equation ([2](#S1.E2)). In contrast, it employs a Riemannian retraction strategy to solve the ${\mathbf{X}}$-subproblem and a Moreau envelope smoothing strategy to solve the ${\mathbf{y}}$-subproblem.

Contributions. We compare existing methods for solving Problem ([1](#alg1)) in Table [1](#S1.T1). Our main contributions are summarized as follows. (i) We introduce OADMM, a specialized ADMM for structured nonsmooth composite optimization problems under orthogonality constraints in Problem ([1](#S1.E1)). Two specific variants of OADMM are explored: one based on Euclidean Projection (OADMM-EP) and the other on Riemannian Retraction (OADMM-EP). Notably, while previous works primarily focus on convex function $h(\cdot)$, linear transformation ${\mathcal{A}}$, and $g({\mathbf{X}})=0$, we extend these by considering weakly convex function $h(\cdot)$, nonlinear mapping ${\mathcal{A}}$, and convex $g({\mathbf{X}})$. (ii) OADMM could demonstrate fast convergence. We integrate a Nesterov extrapolation strategy [] into OADMM-EP and employ a Monotone Barzilai-Borwein (MBB) stepsize strategy [] in OADMM-RR to potentially accelerate primal convergence. Additionally, both OADMM-EP and OADMM-RR utilize an over-relaxation strategy for rapid dual convergence [, , ]. (iii) By introducing a novel Lyapunov function, we establish the convergence of OADMM to critical points of Problem ([1](#S1.E1)) within an oracle complexity of $\mathcal{O}(1/\epsilon^{3})$, aligning it with the best-known oracle complexities to date [, ]. This is achieved through the use of a decreasing step size for updating primal and dual variables. In contrast, RADMM employs a small constant step size for such updates, resulting in a sub-optimal oracle complexity of $\mathcal{O}(\epsilon^{-4})$ []. (iv) We establish the convergence rate of OADMM under the Kurdyka-Łojasiewicz (KL) inequality.

**Table 1: Comparison of existing methods for solving Problem ([1](#alg1)).**
| Reference | $h({\mathcal{A}}({\mathbf{X}}))$ | $g({\mathbf{X}})$ | Notable Features | Complexity | Conv. Rate |
| --- | --- | --- | --- | --- | --- |
| RSG [] | weakly convex; nonlinear ${\mathcal{A}}$ | empty | – | $\mathcal{O}(\epsilon^{-4})$ | unknown |
| ManPG [] | convex $h(\cdot)$, $h({\mathcal{A}}({\mathbf{X}}))=\|{\mathbf{X}}\|_{1}$ | empty | difficult subproblem | $\mathcal{O}(\epsilon^{-2})$ | unknown |
| SPGM [] | convex $h(\cdot)$; linear ${\mathcal{A}}$ | empty | – | $\mathcal{O}(\epsilon^{-3})$ | unknown |
| SOCM [] | convex $h(\cdot)$; linear ${\mathcal{A}}$ | empty | $\sigma=1$, $\alpha_{x}=0$ | unknown | unknown |
| MADMM [] | convex $h(\cdot)$; linear ${\mathcal{A}}$ | empty | $\sigma=1$, $\alpha_{x}=0$ | unknown | unknown |
| RADMM [] | convex $h(\cdot)$; linear ${\mathcal{A}}$ | empty | $\sigma=1$, $\alpha_{x}=0$ | $\mathcal{O}(\epsilon^{-4})$ | unknown |
| OADMM-EP[ours] | weakly convex $h(\cdot)$, nonlinear ${\mathcal{A}}$ | convex | $\sigma\in[1,2)$, $\alpha_{x}>0$ | $\mathcal{O}(\epsilon^{-3})$ | Theorem [5.6](#S5.Thmtheorem6) |
| OADMM-RR[ours] | weakly convex $h(\cdot)$, nonlinear ${\mathcal{A}}$ | convex | $\sigma\in[1,2)$, MBB | $\mathcal{O}(\epsilon^{-3})$ | Theorem [5.6](#S5.Thmtheorem6) |

## 2 Technical Preliminaries

This section provides some technical preliminaries on Moreau envelopes for weakly convex functions and manifold optimization. Notations, technical preliminaries, and relevant lemmas are detailed in Appendix Section [A](#A1).

### 2.1 Moreau Envelopes for Weakly Convex Functions

We provide the following useful definitions.

###### Definition 2.1 .

For a proper convex, and Lipschitz continuous function $h({\mathbf{y}}):\mathbb{R}^{m\times 1}\mapsto\mathbb{R}$, the Moreau envelope of $h({\mathbf{y}})$ with the parameter $\mu>0$ is given by $h_{\mu}({\mathbf{y}})\triangleq\min_{\breve{{\mathbf{y}}}}h(\breve{{\mathbf{y}}})+\frac{1}{2\mu}\|\breve{{\mathbf{y}}}-{\mathbf{y}}\|_{2}^{2}$.

###### Definition 2.2 .

A function $h({\mathbf{y}})$ is $W$-weakly convex if $(h({\mathbf{y}})+\tfrac{W}{2}\|{\mathbf{y}}\|_{2}^{2})$ is convex.

We show some useful properties of Moreau envelope for weakly convex functions [].

###### Lemma 2.3 .

Let $h:\mathbb{R}^{m\times 1}\mapsto\mathbb{R}$ to be a proper, $W_{h}$-weakly convex, and lower semicontinuous function. Assume $\mu\in(0,W_{h}^{-1})$. We have:

(a) The function $h_{\mu}(\cdot)$ is $C_{h}$-Lipschitz continuous.

(b) The function $h_{\mu}(\cdot)$ is continuously differentiable with gradient $\nabla h_{\mu}({\mathbf{y}})=\frac{1}{\mu}({\mathbf{y}}-\mathbb{P}_{\mu}({\mathbf{y}}))$ for all ${\mathbf{y}}$, where $\mathbb{P}_{\mu}({\mathbf{y}})\triangleq\arg\min_{\breve{{\mathbf{y}}}}h(\breve{{\mathbf{y}}})+\frac{1}{2\mu}\|\breve{{\mathbf{y}}}-{\mathbf{y}}\|_{2}^{2}$. This gradient is $\max(\mu^{-1},\frac{W_{h}}{1-\mu W_{h}})$-Lipschitz continuous. In particular, when $\mu\in(0,\frac{1}{2W_{h}}]$, $\nabla h_{\mu}({\mathbf{y}})$ is $(\mu^{-1})$-Lipschitz continuous.

###### Lemma 2.4 .

(Proof in Appendix [B.1](#A2.SS1)) If $h({\mathbf{y}})$ is $W_{h}$-weakly convex and $\mu\in(0,\tfrac{1}{W_{h}})$, then $h_{\mu}({\mathbf{y}})$ is $(\tfrac{W_{h}}{1-\mu W_{h}})$-weakly convex.

###### Lemma 2.5 .

(Proof in Appendix [B.2](#A2.SS2)) Assume $0<\mu_{2}<\mu_{1}<\frac{1}{W_{h}}$, and fixing ${\mathbf{y}}\in\mathbb{R}^{m\times 1}$. We have: $0\leq h_{\mu_{2}}({\mathbf{y}})-h_{\mu_{1}}({\mathbf{y}})\leq\min\{\tfrac{\mu_{1}}{2\mu_{2}},1\}\cdot(\mu_{1}-\mu_{2})C_{h}^{2}$.

###### Lemma 2.6 .

(Proof in Appendix [B.3](#A2.SS3)) Assume $0<\mu_{2}<\mu_{1}\leq\frac{1}{2W_{h}}$, and fixing ${\mathbf{y}}\in\mathbb{R}^{m\times 1}$. We have: $\|\nabla h_{\mu_{1}}({\mathbf{y}})-\nabla h_{\mu_{2}}({\mathbf{y}})\|\leq(\tfrac{\mu_{1}}{\mu_{2}}-1)C_{h}$.

Remarks. (i) Lemmas [2.4](#S2.Thmtheorem4), [2.5](#S2.Thmtheorem5), and [2.6](#S2.Thmtheorem6) presented in this paper are novel. (ii) The upper bound in Lemma [2.5](#S2.Thmtheorem5) is slightly better than the bound established in Lemma 4.1 of [].

###### Definition 2.1 .

###### Definition 2.2 .

###### Lemma 2.3 .

###### Lemma 2.4 .

###### Lemma 2.5 .

###### Lemma 2.6 .

### 2.2 Manifold Optimization

We define the $\epsilon$-stationary point of Problem ([1](#S1.E1)) as follows.

###### Definition 2.7 .

(First-Order Optimality Conditions, [, , ]) The solution $(\ddot{{\mathbf{X}}},\ddot{{\mathbf{y}}},\ddot{{\mathbf{z}}})$ with $\ddot{{\mathbf{X}}}\in\mathcal{M}$ is called an $\epsilon$-stationary point of Problem ([1](#S1.E1)) if: ${\rm{Crit}}(\ddot{{\mathbf{X}}},\ddot{{\mathbf{y}}},\ddot{{\mathbf{z}}})\leq\epsilon^{2}$, where ${\rm{Crit}}({\mathbf{X}},{\mathbf{y}},{\mathbf{z}})\triangleq\|{\mathcal{A}}({\mathbf{X}})-{\mathbf{y}}\|^{2}_{2}+\|\partial h({\mathbf{y}})-{\mathbf{z}}\|_{2}^{2}+\|{\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}}\mathcal{M}}\left(\nabla f({\mathbf{X}})-\partial g({\mathbf{X}})+{\rm{mat}}([\nabla{\mathcal{A}}({\mathbf{X}})]^{\mathsf{T}}{\mathbf{z}})\right)\|_{\mathsf{F}}^{2}$. Here, according to [], for all $\mathbf{\Delta}\in\mathbb{R}^{n\times r}$, we have: ${\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}}\mathcal{M}}(\mathbf{\Delta})=\mathbf{\Delta}-\tfrac{1}{2}{\mathbf{X}}(\mathbf{\Delta}^{\mathsf{T}}{\mathbf{X}}+{\mathbf{X}}^{\mathsf{T}}\mathbf{\Delta})$.

The proposed algorithm is an iterative procedure. After shifting the current iterate ${\mathbf{X}}\in\mathcal{M}$ in the search direction, it may no longer reside on $\mathcal{M}$. Therefore, we must retract the point onto $\mathcal{M}$ to form the next iterate. The following definition is useful in this context.

###### Definition 2.8 .

A retraction on $\mathcal{M}$ is a smooth map: ${\sf{Retr}}_{{\mathbf{X}}}(\mathbf{\Delta})\in\mathcal{M}$ with ${\mathbf{X}}\in\mathcal{M}$ and $\mathbf{\Delta}\in\mathbb{R}^{n\times r}$ satisfying ${\sf{Retr}}_{{\mathbf{X}}}(\mathbf{0})={\mathbf{X}}$, and $\lim_{{\rm{T}}_{{\mathbf{X}}}\mathcal{M}\ni\mathbf{\Delta}\rightarrow\mathbf{0}}\tfrac{\|{\sf{Retr}}_{{\mathbf{X}}}(\mathbf{\Delta})-{\mathbf{X}}-\mathbf{\Delta}\|_{\mathsf{F}}}{\|\mathbf{\Delta}\|_{\mathsf{F}}}=0$ for any ${\mathbf{X}}\in\mathcal{M}$.

Remark. Several retractions on the Stiefel manifold have been explored in literature, such as those based on Polar decomposition and QR decomposition [, ].

The following two lemmas concerning the retraction operator are useful for our subsequent analysis.

###### Lemma 2.9 .

([]) Let ${\mathbf{X}}\in\mathcal{M}$ and $\mathbf{\Delta}\in{\rm{T}}_{{\mathbf{X}}}\mathcal{M}$. There exists positive constants $\{k_{1},k_{2}\}$ such that the following two inequalities hold: $\|{\sf{Retr}}_{{\mathbf{X}}}(\mathbf{\Delta})-{\mathbf{X}}\|_{\mathsf{F}}\leq k_{1}\|\mathbf{\Delta}\|_{\mathsf{F}}$, and $\|{\sf{Retr}}_{{\mathbf{X}}}(\mathbf{\Delta})-{\mathbf{X}}-\mathbf{\Delta}\|_{\mathsf{F}}\leq k_{2}\|\mathbf{\Delta}\|_{\mathsf{F}}^{2}$.

###### Lemma 2.10 .

(Proof in Appendix [B.4](#A2.SS4)) Let ${\mathbf{X}}\in\mathcal{M}$ and $\mathbf{\Delta}\in\mathbb{R}^{n\times r}$, we have $\|{\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}}\mathcal{M}}(\mathbf{\Delta})\|_{\mathsf{F}}\leq\|\mathbf{\Delta}\|_{\mathsf{F}}$.

Furthermore, we present the following two insightful lemmas.

###### Lemma 2.11 .

(Proof in Appendix [B.5](#A2.SS5)) Let $\rho\in(0,+\infty)$, ${\mathbf{G}}\in\mathbb{R}^{n\times r}$, and ${\mathbf{X}}\in\mathcal{M}$. We define ${\mathbf{D}}_{\rho}\triangleq\rho{\mathbf{X}}{\mathbf{G}}^{\mathsf{T}}{\mathbf{X}}+(1-\rho){\mathbf{X}}{\mathbf{X}}^{\mathsf{T}}{\mathbf{G}}-{\mathbf{G}}$. It follows that: (a) $\langle{\mathbf{G}},{\mathbf{D}}_{\rho}\rangle\leq-\tfrac{1}{\max(1,2\rho)}\|{\mathbf{D}}_{\rho}\|_{\mathsf{F}}^{2}$. (b) $\min(1,2\rho)\|{\mathbf{D}}_{1/2}\|_{\mathsf{F}}\leq\|{\mathbf{D}}_{\rho}\|_{\mathsf{F}}$. (c) $\|{\mathbf{D}}_{\rho}\|_{\mathsf{F}}\leq\max(1,2\rho)\|{\mathbf{D}}_{1/2}\|_{\mathsf{F}}$. (d) $\|{\mathbf{D}}_{1}\|_{\mathsf{F}}\leq\max(1,1/\rho)\|{\mathbf{D}}_{\rho}\|_{\mathsf{F}}$.

###### Lemma 2.12 .

(Proof in Appendix [B.6](#A2.SS6)) Consider the following optimization problem: $\min_{{\mathbf{X}}\in\mathcal{M}}f({\mathbf{X}})$, where $f({\mathbf{X}})$ is differentiable. For all ${\mathbf{X}}\in\mathcal{M}$, we have: ${\rm{dist}}(\mathbf{0},\partial I_{\mathcal{M}}({\mathbf{X}})+\nabla f({\mathbf{X}}))\leq\|\nabla f({\mathbf{X}})-{\mathbf{X}}\nabla f({\mathbf{X}})^{\mathsf{T}}{\mathbf{X}}\|_{\mathsf{F}}$.

Remarks. The matrix ${\mathbf{D}}_{\rho}\in\mathbb{R}^{n\times r}$ in Lemma [2.11](#S2.Thmtheorem11) is closely related to the search descent direction of the proposed OADMM-RR algorithm. While one can set $\rho$ to typical values such as $1$ or $1/2$, we consider the setting $\rho\in(0,\infty)$ to enhance the versatility of OADMM-RR, aligning with [, ].

###### Definition 2.7 .

###### Definition 2.8 .

###### Lemma 2.9 .

###### Lemma 2.10 .

###### Lemma 2.11 .

###### Lemma 2.12 .

## 3 The Proposed OADMM Algorithm

This section provides the proposed OADMM algorithm for solving Problem ([1](#S1.E1)), featuring two variants, one is based on Euclidean Projection (OADMM-EP) and the other on Riemannian Retraction (OADMM-RR).

Initially, we consider the following optimization problem:

$$ $\displaystyle\min_{{\mathbf{X}},{\mathbf{y}}}\,f({\mathbf{X}})-g({\mathbf{X}})+h_{\mu}({\mathbf{y}})+\mathcal{I}_{\mathcal{M}}({\mathbf{X}}),~{}s.t.~{}{\mathcal{A}}({\mathbf{X}})={\mathbf{y}},$ (4) $$

where $\mu\rightarrow 0$, and $h_{\mu}({\mathbf{y}})$ is the Moreau Envelope of $h({\mathbf{y}})$. Importantly, $h_{\mu}({\mathbf{y}})$ is $(\mu^{-1})$-smooth when $\mu\leq\frac{1}{2W_{h}}$, according to Lemma [2.3](#S2.Thmtheorem3). We define the augmented Lagrangian function of Problem ([4](#S3.E4)) as follows:

$$ $\displaystyle\mathcal{L}({\mathbf{X}},{\mathbf{y}},{\mathbf{z}};\beta,\mu)=f({\mathbf{X}})-g({\mathbf{X}})+h_{\mu}({\mathbf{y}})+\mathcal{I}_{\mathcal{M}}({\mathbf{X}})+\langle{\mathbf{z}},{\mathcal{A}}({\mathbf{X}})-{\mathbf{y}}\rangle+\tfrac{\beta}{2}\|{\mathcal{A}}({\mathbf{X}})-{\mathbf{y}}\|_{2}^{2}.$ (5) $$

Here, ${\mathbf{z}}$ is the dual variable for the equality constraint, $\mu$ is the smoothing parameter linked to the function $h({\mathbf{y}})$, $\beta$ is the penalty parameter associated with the equality constraint, and $\mathcal{I}_{\mathcal{M}}({\mathbf{X}})$ is the indicator function of the set $\mathcal{M}$. We detail iteration steps of OADMM in Algorithm [1](#alg1). In simple terms, OADMM updates are performed by minimizing the augmented Lagrangian function $\mathcal{L}({\mathbf{X}},{\mathbf{y}},{\mathbf{z}};\beta,\mu)$ over the primal variables $\{{\mathbf{X}}^{t},{\mathbf{y}}^{t}\}$ at a time, while keeping all other primal and dual variables fixed. The dual variables are updated by gradient ascent on the resulting dual problem.

Figure: Algorithm 1 OADMM: The Proposed ADMM for Solving Problem ([1](#S1.E1)).

We have the following remarks on Algorithm [1](#alg1).

- (a)
We use an increasing penalty and decreasing smoothing update scheme throughout all iteration $t$ with $\beta^{t}\rightarrow\infty$ and $\mu^{t}\propto\tfrac{1}{\beta^{t}}\rightarrow 0$.
- (b)
For both variants, we use over-relaxation strategy with $\sigma\in(1,2)$ to update the dual variable ${\mathbf{z}}$. This is expected to obtain faster convergence according to past studies [, , ].
- (c)
For updating the primal variable ${\mathbf{X}}$, we employ different strategies, resulting in distinct variants of OADMM. In OADMM-EP, we utilize the proximal linearized method, which is based on Euclidean projection, and incorporate a Nesterov extrapolation strategy with parameter $\alpha_{x}\in(0,1)$ to minimize the augmented Lagrangian function over ${\mathbf{X}}$.
- (d)
In contrast, in OADMM-RR, we set $\alpha_{x}=0$ and utilize a Monotone Barzilai-Borwein (MBB) strategy [, ] to minimize the augmented Lagrangian function over ${\mathbf{X}}$. This approach could achieve fast convergence, as the parameter $b^{t}$ can be dynamically adjusted to capture the curvature information of the problem. For instance, we can set $b^{t}={\langle\mathbf{S}^{t},\mathbf{S}^{t}\rangle}/{\langle\mathbf{S}^{t},\mathbf{Z}^{t}\rangle}$ or $b^{t}={\langle\mathbf{S}^{t},\mathbf{Z}^{t}\rangle}/{\langle\mathbf{Z}^{t},\mathbf{Z}^{t}\rangle}$, where $\mathbf{S}^{t}={\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}$ and $\mathbf{Z}^{t}={\mathbf{D}}_{1}^{t-1}-{\mathbf{D}}_{1}^{t}$. Here, $-{\mathbf{D}}_{1}^{t}$ corresponds to the Riemannian gradient []. Importantly, a line search strategy is employed to ensure monotonic decrease in the augmented Lagrangian function [, ].
- (e)
Interesting, though we employ a Nesterov extrapolation strategy to update the primal variable ${\mathbf{y}}$ with $\alpha_{y}\in[0,1)$, it is noteworthy that this operation is equivalent to the un-accelerated version with $\alpha_{y}=0$. This equivalence arises due to the consistent form of the first-order optimality condition, as demonstrated in Claim (a) of Lemma [4.1](#S4.Thmtheorem1).
- (f)
The parameters $\gamma$ and $\delta$ respectively represent the decay rate and the sufficient decrease parameter which are typically used in line search procedures [, ].

$\blacktriangleright$ Choose Suitable parameters for Algorithm [1](#alg1). The parameters $\{p,\beta^{0},\xi,\chi,\theta,\alpha_{x},\alpha_{y},\sigma\}$ play a critical role in determining the convergence of the algorithm. We make the following assumptions for the parameters $\{p,\beta^{0},\xi,\chi,\theta,\alpha_{x},\alpha_{y},\sigma\}$.

###### Assumption 3.1 .

For all $t$, we assume $\mu^{t}\triangleq\chi/\beta^{t}<\frac{1}{2W_{h}}$, where $\beta^{t}=\beta^{0}(1+\xi t^{p})$ and $p,\xi\in(0,1)$.

###### Assumption 3.2 .

We assume that $\varepsilon_{x}\triangleq\tfrac{1}{2}\varepsilon_{x}^{\prime}\underline{\rm{L}}>0$, where $\varepsilon_{x}^{\prime}\triangleq(\theta-1-\alpha_{x}-\theta\alpha_{x})-(1+\xi)(\alpha_{x}+\theta\alpha_{x})$. Notably, this assumption solely applies to OADMM-EP, not to OADMM-RR.

###### Assumption 3.3 .

We assume that $\varepsilon_{y}\triangleq\tfrac{1}{2}-\tfrac{W^{\star}_{h}}{2\beta^{0}}-c_{1}c_{0}>0$, where $c_{1}\triangleq\tfrac{2\sigma^{2}}{\chi^{2}(2-\sigma)^{2}}$, $c_{0}\triangleq 1+\tfrac{\xi}{\sigma^{2}}+\tfrac{\xi}{2\sigma^{2}}$, and $W^{\star}_{h}={W_{h}}/(1-W_{h}\mu^{0})$.

Remarks. (i) Assumption [3.1](#S3.Thmtheorem1) guarantees that the function $h_{\mu}({\mathbf{y}})$ is $(\mu^{-1})$-smooth, as detailed in Lemma refer to Lemma [2.3](#S2.Thmtheorem3)). This Assumption is satisfied when $\beta^{0}\geq 2\chi W_{h}$. (ii) Assumption [3.2](#S3.Thmtheorem2) can be met by constraining $\alpha_{x}\in[0,\frac{\theta-1}{(\theta+1)(\xi+2)})$ for OADMM-EP. (iii) As for Assumption [3.3](#S3.Thmtheorem3), given $\xi\in(0,1)$ and $\sigma\in[1,2)$, we have $c_{0}=1+\frac{\xi}{\sigma^{2}}+\frac{\xi}{2\sigma^{2}}\leq 1+\tfrac{1}{1}+\tfrac{1}{2}=2.5$. We conclude that Assumption [3.3](#S3.Thmtheorem3) is achievable if $\beta^{0}\geq 2\chi W_{h}$, and both $\beta^{0}$ and $\chi$ are sufficiently large.

$\blacktriangleright$ Boundedness of the Solutions. We obtain the following lemma.

###### Lemma 3.4 .

(Proof in Section [C.1](#A3.SS1)) For all $t\geq 1$, we have: (a) $\|{\mathbf{X}}^{t}\|_{\mathsf{F}}\leq\overline{\rm{X}}$, (b) $\|{\mathbf{z}}^{t}\|\leq\overline{\rm{z}}$, (c) $\|{\mathbf{y}}^{t}\|\leq\overline{\rm{y}}$. Here, $\overline{\rm{X}}\triangleq\sqrt{r}$, $\overline{\rm{z}}\triangleq\|{\mathbf{z}}^{0}\|+\tfrac{\sigma C_{h}}{2-\sigma}$, $\overline{\rm{y}}\triangleq R_{A}+\tfrac{1}{\beta^{0}}C_{h}+\tfrac{1}{\beta^{0}}\overline{\rm{z}}$.

$\blacktriangleright$ Properties of $\beta^{t}$ and the Lipschitz constant of the smooth function $\mathcal{S}^{t}({\mathbf{X}},{\mathbf{y}}^{t})$ w.r.t. ${\mathbf{X}}$. We present the subsequent valuable lemma.

###### Lemma 3.5 .

(Proof in Appendix [C.2](#A3.SS2)) (a) It holds that $\beta^{t+1}\leq\beta^{t}(1+\xi)$. (b) We define $\mathcal{S}^{t}({\mathbf{X}},{\mathbf{y}}^{t})$ as in Algorithm [1](#alg1). The function $\mathcal{S}^{t}({\mathbf{X}},{\mathbf{y}}^{t})$ is $\ell(\beta^{t})$-smooth w.r.t. ${\mathbf{X}}$, where $\ell(\beta^{t})\triangleq\beta^{t}(C_{A}^{2}+L_{A}R_{A}+L_{A}\overline{\rm{y}})+L_{f}+L_{A}\overline{\rm{z}}$. Also, we obtain: $\beta^{t}\underline{\rm{L}}\leq\ell(\beta^{t})\leq\beta^{t}\overline{\rm{L}}$, where $\underline{\rm{L}}\triangleq L_{A}\overline{\rm{y}}+C_{A}^{2}+L_{A}R_{A}$, and $\overline{\rm{L}}\triangleq\underline{\rm{L}}+(L_{f}+L_{A}\overline{\rm{z}})/\beta^{0}$.

$\blacktriangleright$ Solving the ${\mathbf{y}}$-Subproblem. We show how to solve the ${\mathbf{y}}$-subproblem as in Problem ([7](#S3.E7)) using the following lemma.

###### Lemma 3.6 .

(Proof in Appendix [C.3](#A3.SS3)) Assume $\beta^{t}>\tfrac{W_{h}}{1-W_{h}\mu^{t}}$ and $\mu^{t}\in(0,\tfrac{1}{2W_{h}})$. Problem ([7](#S3.E7)) reduces to a strongly convex minimization problem. It follows that: (a) ${\mathbf{y}}^{t+1}=\frac{1}{1/\mu^{t}+\beta^{t}}(\tfrac{1}{\mu^{t}}\breve{{\mathbf{y}}}^{t+1}+\beta^{t}\mathbf{b})$, where $\breve{{\mathbf{y}}}^{t+1}=\arg\min_{\breve{{\mathbf{y}}}}~{}h(\breve{{\mathbf{y}}})+\tfrac{\beta^{t}}{2(1+\mu^{t}\beta^{t})}\|\breve{{\mathbf{y}}}-\mathbf{b}\|_{2}^{2}$. (b) $\beta^{t}(\mathbf{b}-{\mathbf{y}}^{t+1})\in\partial h(\breve{{\mathbf{y}}}^{t+1})$. (c) $\|{\mathbf{y}}^{t+1}-\breve{{\mathbf{y}}}^{t+1}\|\leq\mu^{t}C_{h}$.

Remarks. Lemma [3.6](#S3.Thmtheorem6) parallels Lemma 1 in [], though its results cannot be directly applied in this scenario since $W_{h}$ is not necessarily zero.

###### Assumption 3.1 .

###### Assumption 3.2 .

###### Assumption 3.3 .

###### Lemma 3.4 .

###### Lemma 3.5 .

###### Lemma 3.6 .

## 4 Oracle Complexity

This section details the oracle complexity of Algorithm [1](#alg1).

Initially, we define the sequence associated with the potential function (or Lyapunov function) as follows:

$$ $\displaystyle\Theta^{t}=L({\mathbf{X}}^{t},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\beta^{t},\mu^{t})+\mu^{t}C_{h}^{2}+\Gamma^{t}_{\mu}+\Gamma^{t}_{z}+\Gamma_{{\sf x}}^{t},$ (8) $$

where $L({\mathbf{X}}^{t},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\beta^{t},\mu^{t})$ is defined in Equation ([5](#S3.E5)), and the terms $\{\Gamma^{t}_{\mu},\Gamma^{t}_{z},\Gamma_{{\sf x}}^{t}\}$ are specified as follows:

$$ $\displaystyle\Gamma^{t}_{\mu}$ $\displaystyle\triangleq$ $\displaystyle\tfrac{4c_{0}C_{h}^{2}\sigma^{2}}{\beta^{0}(2-\sigma)^{2}}\cdot\tfrac{1}{t}$ (9) $\displaystyle\Gamma^{t}_{z}$ $\displaystyle\triangleq$ $\displaystyle c_{0}\cdot\tfrac{1-\sigma}{\sigma}\cdot\tfrac{1}{\beta^{t}}\|{\mathbf{z}}^{t}-{\mathbf{z}}^{t-1}\|_{2}^{2}$ (10) $\displaystyle\Gamma_{{\sf x}}^{t}$ $\displaystyle\triangleq$ $\displaystyle\tfrac{\alpha_{x}(\theta+1)\ell(\beta^{t})}{2}\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}^{2}.$ (11) $$

Here, for OADMM-RR, we set $\alpha_{x}=0$, resulting in $\Gamma_{{\sf x}}^{t}=0$ for all $t$.

We have the following useful lemma, derived using the first-order optimality condition of $\mathbf{y}^{t+1}$.

###### Lemma 4.1 .

(Proof in Section [D.1](#A4.SS1), Bounding Dual using Primal) We define $\{c_{0},c_{1}\}$ in Assumption [3.3](#S3.Thmtheorem3). For all $t\geq 1$, we have: (a) ${\mathbf{z}}^{t-1}-\nabla h_{\mu^{t-1}}({\mathbf{y}}^{t})=\tfrac{1}{\sigma}({\mathbf{z}}^{t-1}-{\mathbf{z}}^{t})$. (b) $\|{\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}\|_{2}^{2}\leq\tfrac{\sigma-1}{\sigma}(\|{\mathbf{z}}^{t}-{\mathbf{z}}^{t-1}\|_{2}^{2}-\|{\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}\|_{2}^{2})+\tfrac{\sigma}{2-\sigma}\|\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})-\nabla h_{\mu^{t-1}}({\mathbf{y}}^{t})\|_{2}^{2}$. (c) $\tfrac{c_{0}}{\beta^{t}}\|{\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}\|_{2}^{2}\leq c_{0}c_{1}\beta^{t}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}+\Gamma^{t}_{\mu}-\Gamma^{t+1}_{\mu}+\Gamma^{t}_{z}-\Gamma^{t+1}_{z}$. (d) $\|\partial h(\breve{{\mathbf{y}}}^{t+1})-{\mathbf{z}}^{t+1}\|=(\sigma-1)\beta^{t}\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|$.

Remarks. (i) With the choice $\sigma=1$, we have: $\nabla h_{\mu^{t-1}}({\mathbf{y}}^{t})={\mathbf{z}}^{t}$, and $\|{\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}\|\leq\|\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})-\nabla h_{\mu^{t-1}}({\mathbf{y}}^{t})\|$. (ii) Claim (a) of Lemma [4.1](#S4.Thmtheorem1) implies that the optimality condition is independent of the choice of $\alpha_{y}$. Therefore, although a Nesterov extrapolation strategy is used to update variable ${\mathbf{y}}$, it reverts to the unaccelerated version with $\alpha_{y}=0$. This phenomenon is expected, as the original minimization problem w.r.t. ${\mathbf{y}}$ in Equation [5](#S3.E5) has a closed-form solution, making the extrapolation strategy unnecessary.

The subsequent lemma demonstrates that the sequence $\{\Theta^{t}\}_{t=1}^{\infty}$ is always lower bounded.

###### Lemma 4.2 .

(Proof in Section [D.2](#A4.SS2)) It holds that $\Theta^{t}\geq\underline{\rm{\Theta}}$ for all $t\geq 1$. Here, $\underline{\rm{\Theta}}\triangleq F(\bar{{\mathbf{X}}})-\mu^{0}C_{h}^{2}-C_{h}(R_{A}+\overline{\rm{y}})-\tfrac{\overline{\rm{z}}^{2}}{2\beta^{0}}$, where $\bar{{\mathbf{X}}}$ is the optimal solution for Problem ([1](#S1.E1)).

The following lemma is useful for our subsequent analysis, applicable to both OADMM-EP and OADMM-RR.

###### Lemma 4.3 .

(Proof in Appendix [D.3](#A4.SS3), Sufficient Decrease for Variables $\{{\mathbf{y}},{\mathbf{z}},\beta,\mu\})$ We define $\varepsilon_{z}=\xi$. We have $L({\mathbf{X}}^{t+1},{\mathbf{y}}^{t+1},{\mathbf{z}}^{t+1};\beta^{t+1},\mu^{t+1})-L({\mathbf{X}}^{t+1},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\beta^{t},\mu^{t})\leq-\varepsilon_{y}\beta^{t}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}-\varepsilon_{z}\beta^{t}\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|_{2}^{2}+\Gamma_{\mu}^{t}-\Gamma_{\mu}^{t+1}+\Gamma_{{\sf z}}^{t}-\Gamma_{{\sf z}}^{t+1}+(\mu^{t}-\mu^{t+1})C_{h}^{2}$, where $\varepsilon_{y}$ is defined in Assumption [3.3](#S3.Thmtheorem3).

In the remaining content of this section, we provide separate analyses for OADMM-EP and OADMM-RR.

###### Lemma 4.1 .

###### Lemma 4.2 .

###### Lemma 4.3 .

### 4.1 Analysis for OADMM-EP

Using the optimality condition of $\mathbf{X}^{t+1}$, we derive the following lemma.

###### Lemma 4.4 .

(Proof in Appendix [D.4](#A4.SS4), Sufficient Decrease for Variable ${\mathbf{X}}$) We have $L({\mathbf{X}}^{t+1},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\beta^{t},\mu^{t})-L({\mathbf{X}}^{t},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\beta^{t},\mu^{t})\leq-\varepsilon_{x}\beta^{t}\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}^{2}+\Gamma_{{\sf x}}^{t}-\Gamma_{{\sf x}}^{t+1}$, where $\varepsilon_{x}$ is defined in Assumption [3.2](#S3.Thmtheorem2).

Combining the results from Lemmas [4.2](#S4.Thmtheorem2),[4.3](#S4.Thmtheorem3), and [4.4](#S4.Thmtheorem4), we arrive at the following lemma.

###### Lemma 4.5 .

(Proof in Appendix [D.5](#A4.SS5)) We have: (a) $\beta^{t}\{\varepsilon_{z}\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|_{2}^{2}+\varepsilon_{y}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}+\varepsilon_{x}\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}^{2}\}\leq\Theta^{t}-\Theta^{t+1}$. (b) $\sum_{t=1}^{T}\beta^{t}\{\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|_{2}^{2}+\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}^{2}+\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}\}\leq W_{1}$, where $W_{1}\triangleq\tfrac{1}{\min(\varepsilon_{y},\varepsilon_{x},\varepsilon_{z})}\cdot(\Theta^{1}-\underline{\rm{\Theta}})$.

Finally, we have the following theorem regarding the oracle complexity of OADMM-EP.

###### Theorem 4.6 .

(Proof in Appendix [D.6](#A4.SS6)) Let $p=1/3$. We have: $\frac{1}{T}\sum_{t=1}^{T}{\rm{Crit}}({\mathbf{X}}^{t+1},\breve{{\mathbf{y}}}^{t+1},{\mathbf{z}}^{t+1})\leq\mathcal{O}(T^{-2/3})$. In other words, there exists $\bar{t}\leq T$ such that:
$\frac{1}{T}\sum_{t=1}^{T}{\rm{Crit}}({\mathbf{X}}^{t+1},\breve{{\mathbf{y}}}^{t+1},{\mathbf{z}}^{t+1})\leq\epsilon^{2}$, provided that $T\geq\mathcal{O}(1/\epsilon^{3})$.

Remarks. The oracle complexity of OADMM-EP matches the best-known complexities currently available to date [, ].

###### Lemma 4.4 .

###### Lemma 4.5 .

###### Theorem 4.6 .

### 4.2 Analysis for OADMM-RR

Using the properties of the line search procedure for updating the variable $\mathbf{X}^{t+1}$, we deduce the following lemma.

###### Lemma 4.7 .

(Proof in Appendix [D.7](#A4.SS7), Sufficient Decrease for Variable ${\mathbf{X}}$) We define $\varepsilon_{x}\triangleq\min(1,2\rho)^{2}\delta\beta^{0}\overline{\gamma}\gamma\underline{b}>0$, and $\overline{\gamma}\triangleq(\tfrac{1}{2}-\delta)/(\overline{g}k_{2}\overline{b}+\tfrac{1}{2}\overline{\rm{L}}k_{1}\overline{b})>0$. We have: (a) For any $t\geq 0$, if $j$ is large enough such that $\gamma^{j}\in(0,\overline{\gamma})$, then the condition in Equation ([6](#S3.E6)) is satisfied. (b) It follows that: $L({\mathbf{X}}^{t+1},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\beta^{t},\mu^{t})-L({\mathbf{X}}^{t},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\beta^{t},\mu^{t})\leq-\tfrac{\varepsilon_{x}}{\beta^{t}}\|{\mathbf{D}}^{t}_{1/2}\|^{2}_{\mathsf{F}}$. Here, $\overline{g}\triangleq C_{f}+C_{g}+4C_{A}\overline{\rm{z}}$, $\{k_{1},k_{2}\}$ are defined in Lemma [2.9](#S2.Thmtheorem9), and $\{\delta,\gamma,\overline{b},\underline{b},\rho\}$ are defined in Algorithm [1](#alg1).

Combining the results from Lemmas [4.2](#S4.Thmtheorem2),[4.3](#S4.Thmtheorem3), and [4.7](#S4.Thmtheorem7), we obtain the following lemma.

###### Lemma 4.8 .

(Proof in Appendix [D.8](#A4.SS8)) We have: (a) $\beta^{t}\{\varepsilon_{z}\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|_{2}^{2}+\varepsilon_{y}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}+\varepsilon_{x}\|\tfrac{1}{\beta^{t}}{\mathbf{D}}^{t}_{1/2}\|^{2}_{\mathsf{F}}\}\leq\Theta^{t}-\Theta^{t+1}$. (b) $\sum_{t=1}^{T}\beta^{t}\{\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|_{2}^{2}+\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}+\|\tfrac{1}{\beta^{t}}{\mathbf{D}}^{t}_{1/2}\|^{2}_{\mathsf{F}}\}\leq W_{2}$, where $W_{2}\triangleq\tfrac{1}{\min(\varepsilon_{y},\varepsilon_{x},\varepsilon_{z})}\cdot(\Theta^{1}-\underline{\rm{\Theta}})$.

Finally, we derive the following theorem on the oracle complexity of OADMM-RR.

###### Theorem 4.9 .

(Proof in Appendix [D.9](#A4.SS9)) Let $p=1/3$. We have: $\frac{1}{T}\sum_{t=1}^{T}{\rm{Crit}}({\mathbf{X}}^{t+1},\breve{{\mathbf{y}}}^{t+1},{\mathbf{z}}^{t+1})\leq\mathcal{O}(T^{-2/3})$. In other words, there exists $\bar{t}\leq T$ such that:
$\frac{1}{T}\sum_{t=1}^{T}{\rm{Crit}}({\mathbf{X}}^{t+1},\breve{{\mathbf{y}}}^{t+1},{\mathbf{z}}^{t+1})\leq\epsilon^{2}$, provided that $T\geq\mathcal{O}(1/\epsilon^{3})$.

Remarks. Theorem [4.9](#S4.Thmtheorem9) mirrors Theorem [4.6](#S4.Thmtheorem6), and OADMM-RR shares the same oracle complexity as OADMM-EP.

###### Lemma 4.7 .

###### Lemma 4.8 .

###### Theorem 4.9 .

## 5 Convergence Rate

This section provides convergence rate of OADMM-EP and OADMM-RR.

Our analyses are based on a non-convex analysis tool called KL inequality [, , , ]. It is worth noting that the work of [] establishes a convergence rate for the iterates with diminishing stepsizes by further exploring the KL exponent, while our study establishes the convergence rate for the proposed OADMM algorithms.

For notation convenience, we define ${\mathbbm{w}}\triangleq\{{\mathbf{X}},{\mathbf{X}}^{-},{\mathbf{y}},{\mathbf{z}}\}$ and ${\mathbbm{u}}\triangleq\{\beta,\beta^{-},\mu,t\}$. We let ${\mathbbm{w}}^{t}\triangleq\{{\mathbf{X}}^{t},{\mathbf{X}}^{t-1},{\mathbf{y}}^{t},{\mathbf{z}}^{t}\}$ and ${\mathbbm{u}}^{t}\triangleq\{\beta^{t},\beta^{t-1},\mu^{t},t\}$. We denote ${\mathbbm{w}}^{\infty}$ as a limiting point of Algorithm [1](#alg1). For both OADMM-EP and OADMM-RR, we define the Lyapunov function as:

$$ $\displaystyle\Theta({\mathbbm{w}};{\mathbbm{u}})\triangleq L({\mathbf{X}},{\mathbf{y}},{\mathbf{z}};\beta,\mu)+v_{1}\tfrac{(\beta^{-})^{2}}{\beta}\|{\mathcal{A}}({\mathbf{X}})-{\mathbf{y}}\|_{2}^{2}+v_{2}\ell(\beta)\|{\mathbf{X}}-{\mathbf{X}}^{-}\|_{\mathsf{F}}^{2}+\mu C_{g}^{2}+K_{\mu}t^{-1},$ $$

where $v_{1}\triangleq c_{0}(1-\sigma)\sigma$, $v_{2}\triangleq\tfrac{1}{2}\alpha_{x}(\theta+1)$. Clearly, we have $\Theta^{t}=\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t})$. Additionally, we define:

$$ $\displaystyle{\small e_{t}\triangleq\left\{\begin{array}[]{ll}\|{\mathbf{y}}^{t}-{\mathbf{y}}^{t-1}\|+\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|+\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}},&\hbox{{\sf OADMM-EP};}\\ \|{\mathbf{y}}^{t}-{\mathbf{y}}^{t-1}\|+\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|+\|\tfrac{1}{\beta^{t}}{\mathbf{D}}^{t-1}_{1/2}\|_{\mathsf{F}},&\hbox{{\sf OADMM-RR}.}\end{array}\right.}$ (14) $$

We make the following additional assumptions.

###### Assumption 5.1 .

(Kurdyka-Łojasiewicz Inequality []). Consider a semi-algebraic function $\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t})$ w.r.t. ${\mathbbm{w}}^{t}$ for all $t$, where ${\mathbbm{w}}^{t}$ is in the effective domain of $\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t})$. There exist $\tilde{\eta}\in(0,+\infty)$, $\tilde{\sigma}\in[0,1)$, a neighborhood $\Upsilon$ of ${\mathbbm{w}}^{\infty}$, and a continuous and concave desingularization function $\varphi(s)\triangleq\tilde{c}s^{1-\tilde{\sigma}}$ with $\tilde{c}>0$ and $s\in[0,\tilde{\eta})$ such that, for all ${\mathbbm{w}}^{t}\in\Upsilon$ satisfying $\Theta({\mathbbm{w}}^{t},{\mathbbm{u}}^{t})-\Theta({\mathbbm{w}}^{\infty},{\mathbbm{u}}^{\infty})\in(0,\tilde{\eta})$, it holds that: $\varphi^{\prime}(\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t})-\Theta({\mathbbm{w}}^{\infty};{\mathbbm{u}}^{\infty}))\cdot{\rm{dist}}(\mathbf{0},\partial\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t}))\geq 1$. Here, ${\rm{dist}}^{2}(\mathbf{0},\partial\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t}))\triangleq{\rm{dist}}^{2}(\mathbf{0},\partial_{{\mathbf{X}}}\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t}))+{\rm{dist}}^{2}(\mathbf{0},\partial_{{\mathbf{X}}^{-}}\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t}))+{\rm{dist}}^{2}(\mathbf{0},\partial_{{\mathbf{y}}}\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t}))+{\rm{dist}}^{2}(\mathbf{0},\partial_{{\mathbf{z}}}\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t}))$.

###### Assumption 5.2 .

The function $g({\mathbf{X}})$ is $L_{g}$-smooth such that $\|\nabla g({\mathbf{X}})-\nabla g({\mathbf{X}}^{\prime})\|_{\mathsf{F}}\leq L_{g}\|{\mathbf{X}}-{\mathbf{X}}^{\prime}\|_{\mathsf{F}}$ holds for all ${\mathbf{X}}\in\mathcal{M}$ and ${\mathbf{X}}^{\prime}\in\mathcal{M}$.

Remarks. Semi-algebraic functions, including real polynomial functions, finite combinations, and indicator functions of semi-algebraic sets, commonly exhibit the KL property and find extensive use in applications.

We present the following lemma regarding subgradient bounds for each iteration.

###### Lemma 5.3 .

(Proof in Section [E.1](#A5.SS1), Subgradient Bounds) For both OADMM-EP and OADMM-RR, there exists a constant $K_{1}>0$ such that: ${\rm{dist}}(\mathbf{0},\partial\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t}))\leq\beta^{t}K_{1}(e_{t+1}+e_{t}+e_{t-1})$.

Remarks. Lemma [5.3](#S5.Thmtheorem3) significantly differs from prior work that used a constant penalty due to the crucial role played by the increasing penalty.

We provide the following helpful lemma.

###### Lemma 5.4 .

(Proof in Section [E.2](#A5.SS2)) For OADMM-RR, it holds that: $\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}\leq c_{0}\|\tfrac{1}{\beta^{t}}{\mathbf{D}}_{1/2}^{t}\|_{\mathsf{F}}$, where $c_{0}\triangleq k_{1}\beta^{0}\overline{b}\overline{\gamma}\max(2\rho,1)$.

The following theorem establishes a finite length property of OADMM.

###### Theorem 5.5 .

(Proof in Section [E.3](#A5.SS3), A Finite Length Property) We define $\varphi_{t}\triangleq\varphi(\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t})-\Theta({\mathbbm{w}}^{\infty};{\mathbbm{u}}^{\infty}))$, where $\varphi(\cdot)$ is the desingularization function defined in Assumption [5.1](#S5.Thmtheorem1). We have the following recursive inequality for both OADMM-EP and OADMM-RR: $e_{t+1}^{2}\leq(e_{t+1}+e_{t}+e_{t-1})\cdot K_{2}(\varphi_{t}-\varphi_{t+1})$, where $K_{2}=\tfrac{3K_{1}}{\min(\varepsilon_{z},\varepsilon_{y},\varepsilon_{x})}$, and $K_{1}$ is defined in Lemma [5.3](#S5.Thmtheorem3). The sequence $\{{\mathbbm{w}}^{t}\}_{t=1}^{\infty}$ has the finite length property that $\sum_{t=1}^{\infty}e_{t+1}\leq C_{{\sf e}}<+\infty$, where $C_{{\sf e}}\triangleq e_{1}+e_{0}+7K_{2}\varphi_{1}$.

Remarks. The finite length property in Theorem [5.5](#S5.Thmtheorem5) represents much stronger convergence results compared to those outlined in Theorems [4.6](#S4.Thmtheorem6) and [4.9](#S4.Thmtheorem9).

Finally, we establish the convergence rate of OADMM with exploiting the KL exponent $\tilde{\sigma}$.

###### Theorem 5.6 .

(Proof in Section [E.4](#A5.SS4), Convergence Rate) We fix $p=1/3$. We define $d_{t}\triangleq\sum_{i=t}^{\infty}e^{i+1}$. We have $d_{t+1}\leq\textstyle d_{t-2}-d_{t+1}+K_{4}(\beta^{t}(d_{t-2}-d_{t+1}))^{\frac{1-\tilde{\sigma}}{\tilde{\sigma}}}$, where $K_{4}\triangleq 4K_{3}\tilde{c}\cdot(K_{1}\tilde{c}(1-\tilde{\sigma}))^{\frac{1-\tilde{\sigma}}{\tilde{\sigma}}}$. There exists $t^{\prime}$ such that for all $t\geq t^{\prime}$, we have:

- (a)
If $\tilde{\sigma}\in(\tfrac{1}{4},\tfrac{1}{2}]$, then we have $\|{\mathbf{X}}^{t}-{\mathbf{X}}^{\infty}\|_{\mathsf{F}}\leq\mathcal{O}(1/\exp(t^{1-u}))$, where $u=\frac{p(1-\tilde{\sigma})}{\tilde{\sigma}}\in(\tfrac{1}{3},1)$.
- (b)
If $\tilde{\sigma}\in(\tfrac{1}{2},1)$, then we have: $\|{\mathbf{X}}^{t}-{\mathbf{X}}^{\infty}\|_{\mathsf{F}}\leq\mathcal{O}(1/(t^{(1-p)/\tau}))$, where $\tau=\frac{2\tilde{\sigma}-1}{1-\tilde{\sigma}}\in(0,\infty)$.

Remarks. Under the KL inequality assumption, with the desingularizing function chosen in the form of $\varphi(s)\triangleq\tilde{c}s^{1-\tilde{\sigma}}$ with $\tilde{\sigma}\in(0,1)$, OADMM converges with a sublinear rate when $\tilde{\sigma}\in(\tfrac{1}{4},\tfrac{1}{2}]$, and converges with an inverse power-law rate when $\tilde{\sigma}\in(\tfrac{1}{2},1)$ for the gap $\|{\mathbf{X}}^{t}-{\mathbf{X}}^{\infty}\|_{\mathsf{F}}$. Notably, inverse power-law convergence is slower than sublinear convergence.

###### Assumption 5.1 .

###### Assumption 5.2 .

###### Lemma 5.3 .

###### Lemma 5.4 .

###### Theorem 5.5 .

###### Theorem 5.6 .

## 6 Applications and Numerical Experiments

This section evaluates the effectiveness of OADMM on the sparse PCA problem.

We compare OADMM-EP and OADMM-RR against four state-of-the-art optimization algorithms: (i) RADMM: ADMM using Riemannian retraction with fixed and small stepsizes []. (ii) SPGM-EP: Smoothing Proximal Gradient Method using Euclidean projection []. (iii) SPGM-EP: SPGM using Riemannian retraction []. (iv) Sub-Grad: Subgradient methods using Euclidean projection [, ].

Our experiments reveal that the proposed OADMM-EP and OADMM-RR demonstrate similar performance and generally outperform the other methods by achieving lower objective function values. Due to space constraints, detailed
experiment results are provided in the Appendix Section [F](#A6).

## 7 Conclusions

This paper introduces OADMM, an Alternating Direction Method of Multipliers (ADMM) tailored for solving structured nonsmooth composite optimization problems under orthogonality constraints. OADMM integrates a Nesterov extrapolation strategy or a Monotone Barzilai-Borwein (MBB) stepsize strategy for potentially accelerated primal convergence, alongside an over-relaxation stepsize strategy for rapid dual convergence. We adjust the penalty parameter and smoothing parameter at a controlled rate. Additionally, we develop a novel Lyapunov function to rigorously analyze the oracle complexity of OADMM, and establish the convergence rate of OADMM. Finally, to demonstrate the efficacy of our approach, we conduct a series of experimental evaluations.

## Appendix A Notations, Technical Preliminaries, and Relevant Lemmas

### A.1 Notations

In this paper, lowercase boldface letters signify vectors, while uppercase letters denote real-valued matrices. The following notations are utilized throughout this paper.

- •
$[n]$: $\{1,2,...,n\}$
- •
$\|\mathbf{x}\|$: Euclidean norm: $\|\mathbf{x}\|=\|\mathbf{x}\|_{2}=\sqrt{\langle\mathbf{x},\mathbf{x}\rangle}$
- •
${\rm{vec}}({\mathbf{X}})$ : ${\rm{vec}}({\mathbf{X}})\in\mathbb{R}^{nr\times 1}$, the vector formed by stacking the column vectors of ${\mathbf{X}}$
- •
${\rm{mat}}(\mathbf{x})$ : ${\rm{mat}}(\mathbf{x})\in\mathbb{R}^{n\times r}$, Convert $\mathbf{x}\in\mathbb{R}^{nr\times 1}$ into a matrix with ${\rm{mat}}({\rm{vec}}({\mathbf{X}}))={\mathbf{X}}$
- •
${\mathbf{X}}^{\mathsf{T}}$ : the transpose of the matrix ${\mathbf{X}}$
- •
$\mathbf{0}_{n,r}$ : A zero matrix of size $n\times r$; the subscript is omitted sometimes
- •
${\mathbf{I}}_{r}$ : ${\mathbf{I}}_{r}\in\mathbb{R}^{r\times r}$, Identity matrix
- •
$\mathcal{M}$: Orthogonality constraint set (a.k.a., Stiefel manifold: $\mathcal{M}=\{{\mathbf{X}}\in\mathbb{R}^{n\times r}\,|\,{\mathbf{X}}^{\mathsf{T}}{\mathbf{X}}=\mathbf{I}_{r}\}$.
- •
${\mathbf{X}}\succeq\mathbf{0}(\text{or}~{}\succ\mathbf{0})$ : the Matrix ${\mathbf{X}}$ is symmetric positive semidefinite (or definite)
- •
${\rm{tr}}(\mathbf{A})$ : Sum of the elements on the main diagonal $\mathbf{A}$: ${\rm{tr}}(\mathbf{A})=\sum_{i}\mathbf{A}_{i,i}$
- •
$\|{\mathbf{X}}\|$ : Operator/Spectral norm: the largest singular value of ${\mathbf{X}}$
- •
$\|{\mathbf{X}}\|_{\mathsf{F}}$ : Frobenius norm: $(\sum_{ij}{{\mathbf{X}}_{ij}^{2}})^{1/2}$
- •
$\|{\mathbf{X}}\|_{1}$: Absolute sum of the elements in ${\mathbf{X}}$ with ${\mathbf{X}}=\sum_{ij}|{\mathbf{X}}_{ij}|$
- •
$\|{\mathbf{X}}\|_{[k]}$: $\ell_{1}$ norm the the $k$ largest (in magnitude) elements of the matrix ${\mathbf{X}}$
- •
$\partial F({\mathbf{X}})$ : (limiting) Euclidean subdifferential of $F({\mathbf{X}})$ at ${\mathbf{X}}$
- •
$\text{dist}(\Xi,\Xi^{\prime})$ : the distance
between two sets with $\text{dist}(\Xi,\Xi^{\prime})\triangleq\inf_{{\mathbf{X}}\in\Xi,{\mathbf{X}}^{\prime}\in\Xi^{\prime}}\|{\mathbf{X}}-{\mathbf{X}}^{\prime}\|_{\mathsf{F}}$
- •
$\|\partial F({\mathbf{X}})\|_{\mathsf{F}}$: $\|\partial F({\mathbf{X}})\|_{\mathsf{F}}=\inf_{\mathbf{Y}\in\partial F({\mathbf{X}})}\|\mathbf{Y}\|_{\mathsf{F}}={\rm{dist}}(\mathbf{0},\partial F({\mathbf{X}}))$.
- •
$\ell(\beta^{t})$: the smoothness parameter of the function $\mathcal{S}^{t}({\mathbf{X}},{\mathbf{y}}^{t})$ w.r.t. ${\mathbf{X}}$.
- •
$\mathcal{I}_{\mathcal{M}}(\mathbf{{\mathbf{x}}})$ : the indicator function of a set $\mathcal{M}$ with $\mathcal{I}_{\mathcal{M}}(\mathbf{{\mathbf{x}}})=0$ if $\mathbf{{\mathbf{x}}}\in\mathcal{M}$ and otherwise $+\infty$.
- •
$\theta$: the proximal parameter
- •
$\chi$: the decrease factor for $\mu^{t}$
- •
$\sigma$: the over-relaxation parameter with $\sigma\in[1,2)$
- •
$\alpha_{x},\alpha_{y}$: the Nesterov extrapolation parameter
- •
$\rho$: the search descent parameter with $\rho\in(0,\infty)$
- •
$\gamma$: the decay rate parameter in the line search procedure with $\gamma\in(0,1)$
- •
$\delta$: the sufficient decrease parameter in the line search procedure with $\delta\in(0,\tfrac{1}{2})$
- •
$p$: the exponent parameter used in the penalty update rule: $\beta^{t}=\beta^{0}(1+\xi t^{p})$
- •
$\xi$: the growth factor parameter used in the penalty update rule: $\beta^{t}=\beta^{0}(1+\xi t^{p})$

### A.2 Technical Preliminaries

Non-convex Non-smooth Optimization. Given the potential non-convexity and non-smoothness of the function $F(\cdot)$, we introduce tools from non-smooth analysis [, ]. The domain of any extended real-valued function $F:\mathbb{R}^{n\times r}\rightarrow(-\infty,+\infty]$ is defined as $\text{dom}(F)\triangleq\{{\mathbf{X}}\in\mathbb{R}^{n\times r}:|F({\mathbf{X}})|<+\infty\}$. At ${\mathbf{X}}\in\text{dom}(F)$, the Fréchet subdifferential of $F$ is defined as $\hat{\partial}{F}({\mathbf{X}})\triangleq\{\bm{\xi}\in\mathbb{R}^{n\times r}:\lim_{\mathbf{Z}\rightarrow{\mathbf{X}}}\inf_{\mathbf{Z}\neq{\mathbf{X}}}\frac{{F}(\mathbf{Z})-{F}({\mathbf{X}})-\langle\bm{\xi},\mathbf{Z}-{\mathbf{X}}\rangle}{\|\mathbf{Z}-{\mathbf{X}}\|_{\mathsf{F}}}\geq 0\}$, while the limiting subdifferential of ${F}({\mathbf{X}})$ at ${\mathbf{X}}\in\text{dom}({F})$ is denoted as $\partial{F}({\mathbf{X}})\triangleq\{\bm{\xi}\in\mathbb{R}^{n}:\exists{\mathbf{X}}^{t}\rightarrow{\mathbf{X}},{F}({\mathbf{X}}^{t})\rightarrow{F}({\mathbf{X}}),\bm{\xi}^{t}\in\hat{\partial}{F}({\mathbf{X}}^{t})\rightarrow\bm{\xi},\forall t\}$. The gradient of $F(\cdot)$ at ${\mathbf{X}}$ in the Euclidean space is denoted as $\nabla{F}({\mathbf{X}})$. The following relations hold among $\hat{\partial}{F}({\mathbf{X}})$, $\partial{F}({\mathbf{X}})$, and $\nabla{F}({\mathbf{X}})$: (i) $\hat{\partial}{F}({\mathbf{X}})\subseteq\partial{F}({\mathbf{X}})$.
(ii) If the function $F(\cdot)$ is convex, $\partial{F}({\mathbf{X}})$ and $\hat{\partial}{F}({\mathbf{X}})$ represent the classical subdifferential for convex functions, i.e., $\partial{F}({\mathbf{X}})=\hat{\partial}{F}({\mathbf{X}})=\{\bm{\xi}\in\mathbb{R}^{n\times r}:F(\mathbf{Z})\geq F({\mathbf{X}})+\langle\bm{\xi},\mathbf{Z}-{\mathbf{X}}\rangle,\forall\mathbf{Z}\in\mathbb{R}^{n\times r}\}$. (iii) If the function $F(\cdot)$ is differentiable, then $\hat{\partial}{F}({\mathbf{X}})=\partial{F}({\mathbf{X}})=\{\nabla F({\mathbf{X}})\}$.

Optimization with Orthogonality Constraints. We introduce some prior knowledge of optimization involving orthogonality constraints []. The nearest orthogonality matrix to any arbitrary matrix $\mathbf{Y}\in\mathbb{R}^{n\times r}$ is determined as $\mathbb{P}_{\mathcal{M}}(\mathbf{Y})=\breve{{\mathbf{U}}}\breve{\mathbf{V}}^{\mathsf{T}}$, where $\mathbf{Y}=\breve{{\mathbf{U}}}{\rm{Diag}}(\mathbf{s})\breve{\mathbf{V}}^{\mathsf{T}}$ represents the singular value decomposition of $\mathbf{Y}$. We use $\mathcal{N}_{\mathcal{M}}({\mathbf{X}})$ to denote the limiting normal cone to $\mathcal{M}$ at ${\mathbf{X}}$, thus defined as $\mathcal{N}_{\mathcal{M}}({\mathbf{X}})=\partial\mathcal{I}_{\mathcal{M}}({\mathbf{X}})=\{\mathbf{Z}\in\mathbb{R}^{n\times r}:\langle\mathbf{Z},{\mathbf{X}}\rangle\geq\langle\mathbf{Z},\mathbf{Y}\rangle,\,\forall\mathbf{Y}\in\mathcal{M}\}$. Moreover, the tangent and normal space to $\mathcal{M}$ at ${\mathbf{X}}\in\mathcal{M}$ are respectively denoted as $\mathrm{T}_{{\mathbf{X}}}\mathcal{M}$ and $\mathrm{N}_{{\mathbf{X}}}\mathcal{M}$. We have: $\mathrm{T}_{{\mathbf{X}}}\mathcal{M}=\{\mathbf{Y}\in\mathbb{R}^{n\times r}|{\mathcal{A}}_{X}(\mathbf{Y})=\mathbf{0}\}$ and $\mathrm{N}_{{\mathbf{X}}}\mathcal{M}=2{\mathbf{X}}\mathbf{\Lambda}\,|\,\mathbf{\Lambda}=\mathbf{\Lambda}^{\mathsf{T}},\mathbf{\Lambda}\in\mathbb{R}^{r\times r}\}$, where ${\mathcal{A}}_{{\mathbf{X}}}(\mathbf{Y})\triangleq{\mathbf{X}}^{\mathsf{T}}\mathbf{Y}+\mathbf{Y}^{\mathsf{T}}{\mathbf{X}}$ for $\mathbf{Y}\in\mathbb{R}^{n\times r}$ and ${\mathbf{X}}\in\mathcal{M}$.

### A.3 Relevant Lemmas

###### Lemma A.1 .

Let $\mathbf{a},\mathbf{b}\in\mathbb{R}^{n}$, and $\gamma\geq 0$ be any constant. We have: $-\|\mathbf{a}-\gamma\mathbf{b}\|_{2}^{2}\leq(\gamma-1)\|\mathbf{a}\|_{2}^{2}-\gamma(\gamma-1)\|\mathbf{b}\|_{2}^{2}$.

###### Proof.

We have: $-\|\mathbf{a}-\gamma\mathbf{b}\|_{2}^{2}=-\|\mathbf{a}\|_{2}^{2}-\|\gamma\mathbf{b}\|_{2}^{2}+2\gamma\langle\mathbf{a},\mathbf{b}\rangle\leq-\|\mathbf{a}\|_{2}^{2}-\|\gamma\mathbf{b}\|_{2}^{2}+2\gamma\cdot(\tfrac{1}{2}\|\mathbf{a}\|_{2}^{2}+\tfrac{1}{2}\|\mathbf{b}\|_{2}^{2})=(\gamma-1)\|\mathbf{a}\|_{2}^{2}-(\gamma^{2}-\gamma)\|\mathbf{b}\|_{2}^{2}$.
∎

###### Lemma A.2 .

Assume $t\geq 1$ and $p\in(0,1)$. We have: $\tfrac{t^{p}-(t-1)^{p}}{1+(t-1)^{p}}\leq\tfrac{1}{t}$.

###### Proof.

We let $t\geq 1$ and $p\in(0,1)$.

First, we define $f(t)\triangleq t^{p}-2(t-1)^{p}-1$. We have $\nabla f(t)=pt^{p}-2p(t-1)^{p-1}=p(t-1)^{p-1}\{(\tfrac{t}{t-1})^{p-1}-2\}\leq p(t-1)^{p-1}\{(\tfrac{t}{t-1})-2\}\leq p(t-1)^{p-1}\{\tfrac{1}{2}-2\}\leq 0$. This implies that $f(t)$ is decreasing. Noting that $f(1)=0$, we conclude that

$$ $\displaystyle f(t)\triangleq t^{p}-2(t-1)^{p}-1\leq 0.$ (15) $$

Second, we have:

$$ $\displaystyle g(t)\triangleq t^{p}-\tfrac{t^{p}+1}{t+1}-(t-1)^{p}\overset{\text{\char 172}}{\leq}t^{p}-\tfrac{t^{p}}{t}-(t-1)^{p}\overset{}{=}t^{p}((1-\tfrac{1}{t})-(1-\tfrac{1}{t})^{p})\overset{\text{\char 173}}{\leq}0,$ (16) $$

where step ① uses $\tfrac{t^{p}+1}{t+1}\geq\tfrac{t^{p}}{t}$ as $t\geq t^{p}$; step ② uses $a\leq a^{p}$ for all $a\in(0,1)$ and $p\in(0,1)$.

Finally, we derive the following results:

$$ $\displaystyle\tfrac{t^{p}-(t-1)^{p}}{1+(t-1)^{p}}\cdot t\overset{\text{\char 172}}{\leq}\tfrac{t}{t+1}\cdot\tfrac{1+t^{p}}{1+(t-1)^{p}}\overset{\text{\char 173}}{\leq}\tfrac{1+t^{p}}{2+2(t-1)^{p}}\overset{\text{\char 174}}{\leq}\tfrac{1+1+2(t-1)^{p}}{2+2(t-1)^{p}}=1,$ $$

where step ① uses Inequality ([16](#A1.E16)); step ② uses $\tfrac{t}{t+1}\leq\tfrac{1}{2}$; step ③ uses Inequality ([15](#A1.E15)).

∎

###### Lemma A.3 .

Let $\beta^{t}=\beta^{0}(1+\xi t^{p})$ and $\mu^{t}\propto\tfrac{1}{\beta^{t}}$. For all $t\geq 1$, we have: $(\tfrac{\mu^{t-1}}{\mu^{t}}-1)^{2}\leq\frac{2}{t}-\frac{2}{t+1}$.

###### Proof.

We derive:

$$ $\displaystyle\textstyle(\tfrac{\mu^{t-1}}{\mu^{t}}-1)^{2}$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\textstyle(\tfrac{\beta^{t}}{\beta^{t-1}}-1)^{2}$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle\textstyle(\tfrac{1+\xi t^{p}}{1+\xi(t-1)^{p}}-1)^{2}=(\tfrac{\xi t^{p}-\xi(t-1)^{p}}{1+\xi(t-1)^{p}})^{2}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\textstyle(\tfrac{t^{p}-(t-1)^{p}}{1+(t-1)^{p}})^{2}$ $\displaystyle\overset{\text{\char 175}}{\leq}$ $\displaystyle\textstyle(\tfrac{1}{t})^{2}$ $\displaystyle\overset{\text{\char 175}}{\leq}$ $\displaystyle\textstyle\frac{2}{t}-\frac{2}{t+1},$ $$

where step ① uses $\mu^{t}\propto\tfrac{1}{\beta^{t}}$; step ② uses $\beta^{t}=\beta^{0}(1+\xi t^{p})$; step ③ uses $\tfrac{\xi}{1+\xi a}<\tfrac{1}{1+a}$ for all $a\geq 0$ when $\xi\in(0,1)$; step ④ uses Lemma [A.2](#A1.Thmtheorem2); step ⑤ uses the fact that $\frac{1}{t^{2}}\leq\tfrac{2}{t}-\tfrac{2}{t+1}$ for all $t\geq 1$.

∎

###### Lemma A.4 .

Assume that ${\mathbf{X}}_{{\sf c}}^{t}={\mathbf{X}}^{t}+\alpha_{x}({\mathbf{X}}^{t}-{\mathbf{X}}^{t-1})$, where $\alpha_{x}\in[0,1)$. We have:

(a) $\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t}_{{\sf c}}\|_{\mathsf{F}}\leq\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}$.

(b) $\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}_{{\sf c}}\|_{\mathsf{F}}\leq\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}+\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}$.

(c) $\|{\mathcal{A}}({\mathbf{X}}_{{\sf c}}^{t})-{\mathbf{y}}^{t}\|\leq\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|+C_{A}\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}$.

(d) $\|\nabla{\mathcal{A}}({\mathbf{X}}^{t}_{{\sf c}})\|_{\mathsf{F}}\leq\overline{A}\triangleq C_{A}+2r$.

(e) $\|[\nabla{\mathcal{A}}({\mathbf{X}}^{t-1}_{{\sf c}})]^{\mathsf{T}}{\mathbf{z}}^{t-1}-[\nabla{\mathcal{A}}({\mathbf{X}}^{t})]^{\mathsf{T}}{\mathbf{z}}^{t}\|_{\mathsf{F}}\leq\overline{A}\|{\mathbf{z}}^{t}-{\mathbf{z}}^{t-1}\|+\|{\mathbf{z}}^{t}\|L_{A}\{\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}+\|{\mathbf{X}}^{t-1}-{\mathbf{X}}^{t-2}\|_{\mathsf{F}}\}$.

###### Proof.

(a) We have:

$$ $\displaystyle\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t}_{{\sf c}}\|_{\mathsf{F}}\overset{\text{\char 172}}{=}\alpha_{x}\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}\overset{\text{\char 173}}{\leq}\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}},$ $$

where step ① uses ${\mathbf{X}}^{t}_{{\sf c}}={\mathbf{X}}^{t}+\alpha_{x}({\mathbf{X}}^{t}-{\mathbf{X}}^{t-1})$; step ② uses $\alpha_{x}\in[0,1)$.

(b) We have:

$$ $\displaystyle\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}_{{\sf c}}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}-\alpha_{x}({\mathbf{X}}^{t}-{\mathbf{X}}^{t-1})\|_{\mathsf{F}}$ (17) $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}+\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}},$ $$

where step ① uses ${\mathbf{X}}^{t}_{{\sf c}}={\mathbf{X}}^{t}+\alpha_{x}({\mathbf{X}}^{t}-{\mathbf{X}}^{t-1})$; step ② uses the triangle inequality and $\alpha_{x}\in[0,1)$.

(c) We have:

$$ $\displaystyle\|{\mathcal{A}}({\mathbf{X}}^{t}_{{\sf c}})-{\mathbf{y}}^{t}\|$ $\displaystyle=$ $\displaystyle\|{\mathcal{A}}({\mathbf{X}}^{t}_{{\sf c}})-{\mathbf{y}}^{t}-{\mathcal{A}}({\mathbf{X}}^{t})+{\mathcal{A}}({\mathbf{X}}^{t})\|$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|+\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathcal{A}}({\mathbf{X}}^{t}_{{\sf c}})\|$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|+C_{A}\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t}_{{\sf c}}\|$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|+C_{A}\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|,$ $$

where step ① uses the triangle inequality; step ② uses $\|{\mathcal{A}}({\mathbf{X}})-{\mathcal{A}}({\mathbf{X}}^{\prime})\|\leq C_{A}\|{\mathbf{X}}-{\mathbf{X}}^{\prime}\|_{\mathsf{F}}$; step ③ uses Claim (a) of this lemma.

(d) We have:

$$ $\displaystyle\|\nabla{\mathcal{A}}({\mathbf{X}}^{t}_{{\sf c}})\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\|\nabla{\mathcal{A}}({\mathbf{X}}^{t}+\alpha_{x}({\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}))\|_{\mathsf{F}}$ (18) $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\|\nabla{\mathcal{A}}({\mathbf{X}}^{t})\|_{\mathsf{F}}+\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle C_{A}+2r\triangleq\overline{A},$ $$

where step ① uses ${\mathbf{X}}^{t}_{{\sf c}}={\mathbf{X}}^{t}+\alpha_{x}({\mathbf{X}}^{t}-{\mathbf{X}}^{t-1})$; step ② uses the triangle inequality and $\alpha_{x}\in[0,1)$; step ③ uses $\|\nabla{\mathcal{A}}({\mathbf{X}}^{t})\|_{\mathsf{F}}\leq C_{A}$ which is implied by $\|{\mathcal{A}}({\mathbf{X}})-{\mathcal{A}}({\mathbf{X}}^{\prime})\|\leq C_{A}\|{\mathbf{X}}-{\mathbf{X}}^{\prime}\|_{\mathsf{F}}$, and $\|{\mathbf{X}}^{t}\|_{\mathsf{F}}\leq r$.

(e) We have:

$$ $\displaystyle\|\nabla{\mathcal{A}}({\mathbf{X}}^{t-1}_{{\sf c}})^{\mathsf{T}}{\mathbf{z}}^{t-1}-[\nabla{\mathcal{A}}({\mathbf{X}}^{t})]^{\mathsf{T}}{\mathbf{z}}^{t}\|_{\mathsf{F}}$ (19) $\displaystyle=$ $\displaystyle\|\nabla{\mathcal{A}}({\mathbf{X}}^{t-1}_{{\sf c}})^{\mathsf{T}}{\mathbf{z}}^{t-1}-[\nabla{\mathcal{A}}({\mathbf{X}}^{t-1}_{{\sf c}})]^{\mathsf{T}}{\mathbf{z}}^{t}+[\nabla{\mathcal{A}}({\mathbf{X}}^{t-1}_{{\sf c}})]^{\mathsf{T}}{\mathbf{z}}^{t}-[\nabla{\mathcal{A}}({\mathbf{X}}^{t})]^{\mathsf{T}}{\mathbf{z}}^{t}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\|\nabla{\mathcal{A}}({\mathbf{X}}^{t-1}_{{\sf c}})^{\mathsf{T}}{\mathbf{z}}^{t-1}-[\nabla{\mathcal{A}}({\mathbf{X}}^{t-1}_{{\sf c}})]^{\mathsf{T}}{\mathbf{z}}^{t}\|_{\mathsf{F}}+\|[\nabla{\mathcal{A}}({\mathbf{X}}^{t-1}_{{\sf c}})]^{\mathsf{T}}{\mathbf{z}}^{t}-[\nabla{\mathcal{A}}({\mathbf{X}}^{t})]^{\mathsf{T}}{\mathbf{z}}^{t}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\|{\mathbf{z}}^{t}-{\mathbf{z}}^{t-1}\|\cdot\|\nabla{\mathcal{A}}({\mathbf{X}}^{t-1}_{{\sf c}})\|_{\mathsf{F}}+\|{\mathbf{z}}^{t}\|\cdot\|\nabla{\mathcal{A}}({\mathbf{X}}^{t})-\nabla{\mathcal{A}}({\mathbf{X}}^{t-1}_{{\sf c}})\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\|{\mathbf{z}}^{t}-{\mathbf{z}}^{t-1}\|\overline{A}+\|{\mathbf{z}}^{t}\|\cdot L_{A}\cdot\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}_{{\sf c}}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 175}}{\leq}$ $\displaystyle\|{\mathbf{z}}^{t}-{\mathbf{z}}^{t-1}\|_{\mathsf{F}}\overline{A}+\|{\mathbf{z}}^{t}\|_{\mathsf{F}}\cdot L_{A}\cdot\{\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}+\|{\mathbf{X}}^{t-1}-{\mathbf{X}}^{t-2}\|_{\mathsf{F}}\}$ $$

where step ① uses the triangle inequality; step ② uses the norm inequality; step ③ uses Inequality ([18](#A1.E18)), and the fact that $\|\nabla{\mathcal{A}}({\mathbf{X}})-\nabla{\mathcal{A}}({\mathbf{X}}^{\prime})\|_{\mathsf{F}}\leq L_{A}\|{\mathbf{X}}-{\mathbf{X}}^{\prime}\|_{\mathsf{F}}$; step ④ uses Inequality ([17](#A1.E17)).

∎

###### Lemma A.5 .

Let $\mathbf{P},\tilde{\mathbf{P}}\in\mathbb{R}^{n\times r}$, and ${\mathbf{X}},\tilde{{\mathbf{X}}}\in\mathcal{M}$. We have:

$$ $\displaystyle\|{\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}}\mathcal{M}}(\mathbf{P})-{\rm{Proj}}_{{\rm{T}}_{\tilde{{\mathbf{X}}}}\mathcal{M}}(\tilde{\mathbf{P}})\|_{\mathsf{F}}\leq 2\|\mathbf{P}-\tilde{\mathbf{P}}\|_{\mathsf{F}}+2\sqrt{r}\|\mathbf{P}\|\|{\mathbf{X}}-\tilde{{\mathbf{X}}}\|_{\mathsf{F}}.$ $$

###### Proof.

First, we obtain:

$$ $\displaystyle\|{\mathbf{X}}\mathbf{P}^{\mathsf{T}}{\mathbf{X}}-\tilde{{\mathbf{X}}}\tilde{\mathbf{P}}^{\mathsf{T}}\tilde{{\mathbf{X}}}\|_{\mathsf{F}}$ (20) $\displaystyle=$ $\displaystyle\|({\mathbf{X}}-\tilde{{\mathbf{X}}})\mathbf{P}^{\mathsf{T}}{\mathbf{X}}+\tilde{{\mathbf{X}}}\mathbf{P}^{\mathsf{T}}({\mathbf{X}}-\tilde{{\mathbf{X}}})+\tilde{{\mathbf{X}}}(\mathbf{P}-\tilde{\mathbf{P}})^{\mathsf{T}}\tilde{{\mathbf{X}}}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\|{\mathbf{X}}-\tilde{{\mathbf{X}}}\|_{\mathsf{F}}\|\mathbf{P}^{\mathsf{T}}{\mathbf{X}}\|+\|\tilde{{\mathbf{X}}}\mathbf{P}^{\mathsf{T}}\|\|{\mathbf{X}}-\tilde{{\mathbf{X}}}\|_{\mathsf{F}}+\|\tilde{{\mathbf{X}}}(\mathbf{P}-\tilde{\mathbf{P}})^{\mathsf{T}}\tilde{{\mathbf{X}}}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle 2\sqrt{r}\|\mathbf{P}\|\|{\mathbf{X}}-\tilde{{\mathbf{X}}}\|_{\mathsf{F}}+\|\mathbf{P}-\tilde{\mathbf{P}}\|_{\mathsf{F}},$ $$

where step ① uses the triangle inequality; step ② uses $\|\mathbf{A}\mathbf{B}\|_{\mathsf{F}}\leq\|\mathbf{A}\|\cdot\|\mathbf{B}\|_{\mathsf{F}}$, and $\|\tilde{{\mathbf{X}}}\|\leq 1$.

Second, we have:

$$ $\displaystyle\|{\mathbf{X}}{\mathbf{X}}^{\mathsf{T}}\mathbf{P}-\tilde{{\mathbf{X}}}\tilde{{\mathbf{X}}}^{\mathsf{T}}\tilde{\mathbf{P}}\|_{\mathsf{F}}$ (21) $\displaystyle=$ $\displaystyle\|({\mathbf{X}}-\tilde{{\mathbf{X}}}){\mathbf{X}}^{\mathsf{T}}\mathbf{P}+\tilde{{\mathbf{X}}}({\mathbf{X}}-\tilde{{\mathbf{X}}})^{\mathsf{T}}\mathbf{P}+\tilde{{\mathbf{X}}}\tilde{{\mathbf{X}}}^{\mathsf{T}}(\mathbf{P}-\tilde{\mathbf{P}})\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\|{\mathbf{X}}-\tilde{{\mathbf{X}}}\|_{\mathsf{F}}\|{\mathbf{X}}^{\mathsf{T}}\mathbf{P}\|+\|\tilde{{\mathbf{X}}}\|\cdot\|{\mathbf{X}}-\tilde{{\mathbf{X}}}\|_{\mathsf{F}}\cdot\|\mathbf{P}\|+\|\tilde{{\mathbf{X}}}\tilde{{\mathbf{X}}}^{\mathsf{T}}\|\cdot\|\mathbf{P}-\tilde{\mathbf{P}}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle 2\sqrt{r}\|\mathbf{P}\|\|{\mathbf{X}}-\tilde{{\mathbf{X}}}\|_{\mathsf{F}}+\|\mathbf{P}-\tilde{\mathbf{P}}\|_{\mathsf{F}},$ $$

where step ① uses the triangle inequality; step ② uses $\|\mathbf{A}\mathbf{B}\|_{\mathsf{F}}\leq\|\mathbf{A}\|\cdot\|\mathbf{B}\|_{\mathsf{F}}$, and $\|\tilde{{\mathbf{X}}}\|\leq 1$.

Finally, we derive:

$$ $\displaystyle\|{\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}}\mathcal{M}}(\mathbf{P})-{\rm{Proj}}_{{\rm{T}}_{\tilde{{\mathbf{X}}}}\mathcal{M}}(\tilde{\mathbf{P}})\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\|[\mathbf{P}-\tfrac{1}{2}{\mathbf{X}}\mathbf{P}^{\mathsf{T}}{\mathbf{X}}-\tfrac{1}{2}{\mathbf{X}}{\mathbf{X}}^{\mathsf{T}}\mathbf{P}]-[\tilde{\mathbf{P}}-\tfrac{1}{2}\tilde{{\mathbf{X}}}\tilde{\mathbf{P}}^{\mathsf{T}}\tilde{{\mathbf{X}}}-\tfrac{1}{2}\tilde{{\mathbf{X}}}\tilde{{\mathbf{X}}}^{\mathsf{T}}\tilde{\mathbf{P}}]\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\|\mathbf{P}-\tilde{\mathbf{P}}\|_{\mathsf{F}}+\tfrac{1}{2}\|{\mathbf{X}}\mathbf{P}^{\mathsf{T}}{\mathbf{X}}-\tilde{{\mathbf{X}}}\tilde{\mathbf{P}}^{\mathsf{T}}\tilde{{\mathbf{X}}}\|_{\mathsf{F}}+\tfrac{1}{2}\|{\mathbf{X}}{\mathbf{X}}^{\mathsf{T}}\mathbf{P}-\tilde{{\mathbf{X}}}\tilde{{\mathbf{X}}}^{\mathsf{T}}\tilde{\mathbf{P}}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\|\mathbf{P}-\tilde{\mathbf{P}}\|_{\mathsf{F}}+2\sqrt{r}\|\mathbf{P}\|\|{\mathbf{X}}-\tilde{{\mathbf{X}}}\|_{\mathsf{F}}+\|\mathbf{P}-\tilde{\mathbf{P}}\|_{\mathsf{F}}$ $$

where step ① uses ${\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}}\mathcal{M}}(\mathbf{\Delta})=\mathbf{\Delta}-\tfrac{1}{2}{\mathbf{X}}(\mathbf{\Delta}^{\mathsf{T}}{\mathbf{X}}+{\mathbf{X}}^{\mathsf{T}}\mathbf{\Delta})$ for all $\mathbf{\Delta}\in\mathbb{R}^{n\times r}$ []; step ② uses the triangle inequality; step ③ uses Inequalities ([20](#A1.E20)) and ([21](#A1.E21)).

∎

###### Lemma A.6 .

We let $p\in(0,1)$. We define $g(t)\triangleq\tfrac{1}{1-p}(t+1)^{(1-p)}-\tfrac{1}{1-p}-(1-p)t^{(1-p)}$. We have $g(t)\geq 0$ for all $t\geq 1$.

###### Proof.

We assume $p\in(0,1)$.

First, we show that $h(p)\triangleq(1-p)^{1/p}\leq\tfrac{1}{\exp(1)}$. Recall that it holds: $\lim_{p\rightarrow 0^{+}}(1+p)^{1/p}=\exp(1)$ and $\lim_{p\rightarrow 0^{+}}(1-p)^{1/p}=1/\exp(1)$. Given the function $h(p)$ is a decreasing function on $p\in(0,1)$, we have $h(p)\leq\lim_{p\rightarrow 0^{+}}(1-p)^{1/p}=\tfrac{1}{\exp(1)}$.

Second, we show that $f(q)=2^{q}-1-q^{2}\geq 0$ for all $q\in(0,1)$. We have $\nabla f(q)=\log(2)2^{q}-2q$, and $\nabla^{2}f(q)=2^{q}(\log(2))^{2}-2\leq 2(\log(2))^{2}-2\leq 0$, implying that the function $f(q)$ is concave on $q\in(0,1)$. Noticing $f(0)=f(1)=0$, we conclude that $f(q)\geq 0$.

We show that $g(t)$ is an increasing function. We have:

$$ $\displaystyle\nabla g(t)$ $\displaystyle=$ $\displaystyle\textstyle(t+1)^{-p}-(1-p)^{2}t^{-p}$ $\displaystyle=$ $\displaystyle\textstyle(t+1)^{-p}\cdot(1-(1-p)^{2}(\tfrac{t+1}{t})^{p})$ $\displaystyle\overset{\text{\char 172}}{\geq}$ $\displaystyle\textstyle(t+1)^{-p}\cdot(1-(1-p)^{2}2^{p})$ $\displaystyle\overset{\text{\char 173}}{\geq}$ $\displaystyle\textstyle(t+1)^{-p}\cdot(1-(\frac{2}{\exp(1)^{2}})^{p})$ $\displaystyle\overset{\text{\char 174}}{\geq}$ $\displaystyle\textstyle 0,$ $$

where step ① uses $\frac{t+1}{t}\leq 2$ for all $t\geq 1$; step ② uses $1-p\leq(\tfrac{1}{\exp(1)})^{p}$ for all $p\in(0,1)$; step ③ uses $\frac{2}{\exp(1)^{2}}\thickapprox 0.2707<1$. Finally, for all $t\geq 1$, we have:

$$ $\displaystyle g(t)\overset{\text{\char 172}}{\geq}g(1)=(1-p)^{-1}\cdot\{2^{(1-p)}-1-(1-p)^{2}\}\overset{\text{\char 173}}{\geq}0,$ $$

where step ① uses the fact that $g(t)$ is an increasing function; step ② uses $2^{q}-1-q^{2}\geq 0$ for all $q=1-p\in(0,1)$.

∎

###### Lemma A.7 .

Let $p\in(0,1)$. It holds that: $(1-p)T^{(1-p)}\leq\sum_{t=1}^{T}\tfrac{1}{t^{p}}\leq\tfrac{T^{(1-p)}}{1-p}$.

###### Proof.

We define $g(t)\triangleq\tfrac{1}{t^{p}}$ and $h(t)\triangleq\frac{1}{1-p}t^{(1-p)}$.

Using the integral test for convergence, we obtain: $\int_{1}^{T+1}g(x)dx\leq\sum_{t=1}^{T}g(t)\leq g(1)+\int_{1}^{T}g(x)dx$.

(a) We first focus on the lower bound. We derive:

$$ $\displaystyle\textstyle\sum_{t=1}^{T}t^{-p}$ $\displaystyle\geq$ $\displaystyle\textstyle\sum_{t=1}^{T}\int_{t}^{t+1}x^{-p}dx=\int_{1}^{T+1}x^{-p}dx$ $\displaystyle\overset{\text{\char 172}}{\geq}$ $\displaystyle\textstyle h(T+1)-h(1)=\tfrac{1}{1-p}(T+1)^{1-p}-\tfrac{1}{1-p}\overset{\text{\char 173}}{\geq}(1-p)T^{1-p},$ $$

where step ① uses $\nabla h(x)=x^{-p}$; step ② uses Lemma [A.6](#A1.Thmtheorem6).

(b) We now focus on the upper bound. We derive:

$$ $\displaystyle\textstyle\sum_{t=1}^{T}t^{-p}$ $\displaystyle\leq$ $\displaystyle\textstyle h(1)+\int_{1}^{T}x^{-p}dx\overset{\text{\char 172}}{=}1+h(T)-h(1)$ $\displaystyle=$ $\displaystyle\textstyle 1+\tfrac{1}{1-p}(T)^{1-p}-\tfrac{1}{1-p}\overset{}{=}\tfrac{T^{(1-p)}-p}{1-p}<\tfrac{T^{(1-p)}}{1-p},$ $$

where step ① uses $\nabla h(x)=x^{-p}$.

∎

###### Lemma A.8 .

Let $\{e_{t}\}_{t=1}^{\infty}$ and $\{p_{t}\}_{t=1}^{\infty}$ be any two nonnegative sequences satisfying $(e_{t+1})^{2}\leq(e_{t+1}+e_{t}+e_{t-1})(p_{t}-p_{t+1})$. For all $i\geq 1$, we have: $\sum_{t=i}^{\infty}e_{t+1}\leq e_{i}+e_{i-1}+7p_{i}$.

###### Proof.

We define $w_{t}\triangleq p_{t}-p_{t+1}$. We let $1\leq i<T$.

First, for any $i\geq 1$, we have:

$$ $\displaystyle\textstyle\sum^{T}_{t=i}w_{t}=\sum^{T}_{t=i}(p_{t}-p_{t+1})=p_{i}-p_{T+1}\overset{\text{\char 172}}{\leq}p_{i},$ (22) $$

where step ① uses $p_{i}\geq 0$ for all $i$.

Second, we obtain:

$$ $\displaystyle e_{t+1}$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\textstyle\sqrt{(e_{t+1}+e_{t}+e_{t-1})w_{t}}$ (23) $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\textstyle\sqrt{\tfrac{\alpha}{2}(e_{t+1}+e_{t}+e_{t-1})^{2}+\tfrac{1}{2\alpha}(w_{t})^{2}}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\textstyle\sqrt{\tfrac{\alpha}{2}}\cdot(e_{t+1}+e_{t}+e_{t-1})+\sqrt{\tfrac{1}{2\alpha}}w_{t},$ $$

where step ① uses $(e_{t+1})^{2}\leq(e_{t+1}+e_{t}+e_{t-1})(p_{t}-p_{t+1})$ and $w_{t}\triangleq p_{t}-p_{t+1}$; step ② uses the fact that $ab\leq\frac{\alpha}{2}a^{2}+\frac{1}{2\alpha}b^{2}$ for all $\alpha>0$; step ③ uses the fact that $\sqrt{a+b}\leq\sqrt{a}+\sqrt{b}$ for all $a\geq 0$ and $b\geq 0$.

Assume the parameter $\alpha$ is sufficiently small that $1-\sqrt{\tfrac{\alpha}{2}}-2\sqrt{\tfrac{\alpha}{2}}>0$. Telescoping Inequality ([23](#A1.E23)) over $t$ from $i$ to $T$, we obtain:

$$ $\displaystyle\textstyle\sum_{t=i}^{T}\sqrt{\tfrac{1}{2\alpha}}w_{t}$ $\displaystyle\geq$ $\displaystyle\textstyle(1-\sqrt{\tfrac{\alpha}{2}})\{\sum_{t=i}^{T}e_{t+1}\}-\sqrt{\tfrac{\alpha}{2}}\{\sum_{t=i}^{T}e_{t}\}-\sqrt{\tfrac{\alpha}{2}}\{\sum_{t=i}^{T}e_{t-1}\}$ $\displaystyle=$ $\displaystyle\textstyle(1-\sqrt{\tfrac{\alpha}{2}})\{e_{T}+e_{T+1}+\sum_{t=i}^{T-2}e_{t+1}\}-\sqrt{\tfrac{\alpha}{2}}\{e_{i}+e_{T}+\sum_{t=i}^{T-2}e_{t+1}\}$ $\displaystyle\textstyle-\sqrt{\tfrac{\alpha}{2}}\{e_{i-1}+e_{i}+\sum_{t=i}^{T-2}e_{t+1}\}$ $\displaystyle=$ $\displaystyle\textstyle(1-\sqrt{\tfrac{\alpha}{2}})(e_{T}+e_{T+1})-\sqrt{\tfrac{\alpha}{2}}(e_{i}+e_{T}+e_{i-1}+e_{i})+(1-\sqrt{\tfrac{\alpha}{2}}-2\sqrt{\tfrac{\alpha}{2}})\sum_{t=i}^{T-2}e_{t+1}$ $\displaystyle\overset{\text{\char 172}}{\geq}$ $\displaystyle\textstyle(1-\sqrt{\tfrac{\alpha}{2}})e_{T}-\sqrt{\tfrac{\alpha}{2}}(e_{i}+e_{T}+e_{i-1}+e_{i})+(1-\sqrt{\tfrac{\alpha}{2}}-2\sqrt{\tfrac{\alpha}{2}})\sum_{t=i}^{T-2}e_{t+1}$ $\displaystyle\overset{\text{\char 173}}{\geq}$ $\displaystyle\textstyle-2\sqrt{\tfrac{\alpha}{2}}(e_{i}+e_{i-1})+(1-\sqrt{\tfrac{\alpha}{2}}-2\sqrt{\tfrac{\alpha}{2}})\sum_{t=i}^{T-2}e_{t+1},$ $$

where step ① uses $e_{T+1}\geq 0$ and $1-\sqrt{\tfrac{\alpha}{2}}>0$; step ② uses $e_{T}\geq 0$ and $1-2\sqrt{\frac{\alpha}{2}}>0$. This leads to:

$$ $\displaystyle\textstyle\sum_{t=i}^{T-2}e_{t+1}$ $\displaystyle\leq$ $\displaystyle\textstyle(1-3\sqrt{\tfrac{\alpha}{2}})^{-1}\cdot\{2\sqrt{\tfrac{\alpha}{2}}(e_{i}+e_{i-1})+\sqrt{\tfrac{1}{2\alpha}}\sum_{t=i}^{T}w_{t}\}$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\textstyle(e_{i}+e_{i-1})+6.25\sum_{t=i}^{T}w_{t}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\textstyle(e_{i}+e_{i-1})+7p_{i},$ $$

step ① uses the fact that $(1-3\sqrt{\tfrac{\alpha}{2}})^{-1}\cdot 2\sqrt{\tfrac{\alpha}{2}}=1$ and $(1-3\sqrt{\tfrac{\alpha}{2}})^{-1}\cdot\sqrt{\tfrac{1}{2\alpha}}=6.25$ when $\alpha=0.08$; step ② uses Inequalities ([22](#A1.E22)). Letting $T\rightarrow\infty$, we conclude this lemma.

∎

###### Lemma A.9 .

Assume $a\geq 0$. Let $\{\ddot{\beta}^{t}\}_{t=1}^{T}$ be any nonnegative increasing sequence satisfying $\sum_{i=0}^{T}({1}/{\ddot{\beta}^{i}})\geq\mathcal{O}(T^{a})$. We have: $\sum_{i=1}^{T/3}({1}/{\ddot{\beta}^{3i}})\geq\mathcal{O}(T^{a})$.

###### Proof.

We have: $\textstyle\sum_{t=1}^{T/3}\tfrac{1}{\ddot{\beta}^{3t}}=\textstyle\tfrac{1}{3}\sum_{t=1}^{T/3}(\tfrac{1}{\ddot{\beta}^{3t}}+\tfrac{1}{\ddot{\beta}^{3t}}+\tfrac{1}{\ddot{\beta}^{3t}})\overset{\text{\char 172}}{\geq}\textstyle\tfrac{1}{3}\sum_{t=1}^{T/3}(\tfrac{1}{\ddot{\beta}^{3t}}+\tfrac{1}{\ddot{\beta}^{3t+1}}+\tfrac{1}{\ddot{\beta}^{3t+2}})=\mathcal{O}(\sum_{t=1}^{T}\tfrac{1}{\ddot{\beta}^{t}})\geq\mathcal{O}(T^{a})$, where step ① uses the fact that $\{\ddot{\beta}^{t}\}_{t=1}^{T}$ is increasing.

∎

###### Lemma A.10 .

Let $\{d_{t}\}_{t=0}^{\infty}$ and $\{\tilde{\beta}^{t}\}_{t=0}^{\infty}$ be any two nonnegative sequences satisfying $\tfrac{d_{t+1}}{d_{t-2}}\leq\tfrac{\tilde{\beta}^{t}+1}{\tilde{\beta}^{t}+2}$. Assume that $\sum_{i=0}^{T}({1}/{\tilde{\beta}^{i}})\geq\mathcal{O}(T^{a})$, where $T\geq 0$ and $a\geq 0$. We have: $d_{T}\leq\mathcal{O}({1}/{\exp(T^{a})})$.

###### Proof.

We define $\gamma^{t}\triangleq\tfrac{1}{\tilde{\beta}^{t}+2}\in(0,1)$.

We have: $\tfrac{d_{t+1}}{d_{t-2}}\leq 1-\gamma^{t}$, leading to:

$$ $\displaystyle d_{3j}$ $\displaystyle\leq$ $\displaystyle d_{0}(1-\gamma^{3-1})(1-\gamma^{6-1})(1-\gamma^{9-1})\ldots(1-\gamma^{3j-1})$ (24) $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle d_{0}(1-\gamma^{3})(1-\gamma^{6})(1-\gamma^{9})\ldots(1-\gamma^{3j-1}),$ $$

where step ① uses $\gamma^{t}\geq\gamma^{t+1}$.

(a) When $T$ is multiple of $3$ given by $T=3j$ for some integer $j\geq 0$, we have:

$$ $\displaystyle\textstyle d_{T}$ $\displaystyle=$ $\displaystyle\textstyle\exp(\log(d_{T}))$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\textstyle\exp(\log(d_{0}\cdot\prod_{i=1}^{T/3}(1-\gamma^{3i})))$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle\textstyle\exp(\log(d_{0})+\sum_{i=1}^{T/3}\log(1-\gamma^{3i}))$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\textstyle\exp(\log(d_{0})+\sum_{i=1}^{T/3}(-\gamma^{3i}))$ $\displaystyle\overset{\text{\char 175}}{\leq}$ $\displaystyle\textstyle\exp(\log(d_{0}))\times\{\exp(\sum_{i=1}^{T/3}(\gamma^{3i}))\}^{-1}$ $\displaystyle\overset{\text{\char 176}}{\leq}$ $\displaystyle\textstyle d_{0}\times\{\exp(\mathcal{O}(T^{a}))\}^{-1}=\mathcal{O}({1}/{\exp(T^{a})}),$ $$

where step ① uses Inequality ([24](#A1.E24)); step ② uses $\log(ab)=\log(a)+\log(b)$ for all $a>0$ and $b>0$; step ③ uses $\log(1-x)\leq-x$ for all $x\in(0,1)$, and $1-\gamma^{t}\in(0,1)$ for all $t$; step ④ uses $\exp(a+b)=\exp(a)\exp(b)$ for all $a>0$ and $b>0$; step ⑤ uses Lemma [A.9](#A1.Thmtheorem9) with $\ddot{\beta}^{t}=1/\gamma^{t}=\tilde{\beta}^{t}+2$.

(b) When $T$ is not a multiple of $3$, employing analogous strategies yields: $d_{T+1}=\mathcal{O}({1}/{\exp(T^{a})})$.

∎

###### Lemma A.11 .

Let $\tilde{\sigma}\in(\frac{1}{2},1)$, we define $\tau\triangleq\frac{2\tilde{\sigma}-1}{1-\tilde{\sigma}}\in(0,\infty)$. Let $\{d_{t}\}_{t=0}^{\infty}$ be a nonnegative decreasing sequence, and $\{\tilde{\beta}^{t}\}_{t=0}^{\infty}$ a nonnegative increasing sequence. Assume that $d_{t+1}^{\tau+1}\leq\tilde{\beta}^{t}(d_{t-2}-d_{t+1})$, and $\sum_{i=1}^{T}({1}/{\tilde{\beta}^{i}})\geq\mathcal{O}(T^{a})$. We have: $d_{T}\leq\mathcal{O}(1/(T^{a/\tau}))$.

###### Proof.

We define $\tau\triangleq\frac{2\tilde{\sigma}-1}{1-\tilde{\sigma}}\in(0,\infty)$. We let $\kappa>1$. We define $h(s)=\tfrac{1}{s^{\tau+1}}$.

We consider two cases for $h(d_{t+1})/h(d_{t-2})$.

Case (1). $h(d_{t+1})\leq\kappa h(d_{t-2})$. We define $\breve{h}(s)=-\tfrac{1}{\tau}\cdot s^{-\tau}$. We derive:

$$ $\displaystyle 1$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\textstyle\tilde{\beta}^{t}(d_{t-2}-d_{t})\cdot h(d_{t+1})$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\textstyle\tilde{\beta}^{t}(d_{t-2}-d_{t})\cdot\kappa h(d_{t-1})$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\textstyle\tilde{\beta}^{t}\kappa\int_{d_{t+1}}^{d_{t-2}}h(s)ds$ $\displaystyle\overset{\text{\char 175}}{=}$ $\displaystyle\textstyle\tilde{\beta}^{t}\kappa\cdot(\breve{h}(d_{t-2})-\breve{h}(d_{t}))$ $\displaystyle\overset{\text{\char 176}}{=}$ $\displaystyle\textstyle\tilde{\beta}^{t}\kappa\cdot\tfrac{1}{\tau}\cdot(d_{t}^{-\tau}-d_{t-2}^{-\tau})$ $$

where step ① uses $d_{t+1}^{\tau+1}\leq\tilde{\beta}^{t}(d_{t-2}-d_{t+1})$; step ② uses $h(d_{t+1})\leq\kappa h(d_{t-2})$; step ③ uses the fact that $h(s)$ is a nonnegative and increasing function that $(a-b)h(a)\leq\int_{b}^{a}h(s)$ for all $a,b\in[0,\infty)$; step ④ uses the fact that $\nabla\breve{h}(s)=h(s)$. This leads to:

$$ $\displaystyle d_{t+1}^{-\tau}-d_{t-2}^{-\tau}\geq\tfrac{\kappa^{-1}\tau}{\tilde{\beta}^{t}}.$ (25) $$

Case (2). $h(d_{t+1})>\kappa h(d_{t-2})$. We have:

$$ $\displaystyle h(d_{t+1})>\kappa h(d_{t-2})$ $\displaystyle\overset{\text{\char 172}}{\Rightarrow}$ $\displaystyle d_{t+1}^{-(\tau+1)}>\kappa\cdot d_{t-2}^{-(\tau+1)}$ (26) $\displaystyle\overset{\text{\char 173}}{\Rightarrow}$ $\displaystyle(d_{t+1}^{-(\tau+1)})^{\tfrac{\tau}{\tau+1}}>\kappa^{\tfrac{\tau}{\tau+1}}\cdot(d_{t-2}^{-(\tau+1)})^{\tfrac{\tau}{\tau+1}}$ $\displaystyle\overset{}{\Rightarrow}$ $\displaystyle d_{t+1}^{-\tau}>\kappa^{\tfrac{\tau}{\tau+1}}\cdot d_{t-2}^{-\tau},$ $$

where step ① uses the definition of $h(\cdot)$; step ② uses the fact that if $a>b$ (where $a,b>0$), then $a^{\nu^{\prime}}>b^{\nu^{\prime}}$ for any exponent $\nu^{\prime}\triangleq\tfrac{\tau}{\tau+1}\in(0,1)$. We further derive:

$$ $\displaystyle\textstyle d_{t+1}^{-\tau}-d_{t-2}^{-\tau}$ $\displaystyle\overset{\text{\char 172}}{\geq}$ $\displaystyle\textstyle(\kappa^{\tfrac{\tau}{\tau+1}}-1)\cdot d_{t-2}^{-\tau}$ (27) $\displaystyle\overset{\text{\char 173}}{\geq}$ $\displaystyle\textstyle(\kappa^{\tfrac{\tau}{\tau+1}}-1)\cdot d_{0}^{-\tau},$ $$

where step ① uses Inequality ([26](#A1.E26)); step ② uses $\tau>0$ and $d_{t-2}\leq d_{0}$ for all $t$.

In view of Inequalities ([25](#A1.E25)) and ([27](#A1.E27)), we have:

$$ $\displaystyle d_{t+1}^{-\tau}-d_{t-2}^{-\tau}$ $\displaystyle\geq$ $\displaystyle\textstyle\min(\tfrac{\kappa^{-1}\tau}{\beta^{t}},(\kappa^{\tfrac{\tau}{\tau+1}}-1)\cdot d_{0}^{-\tau})$ (28) $\displaystyle\geq$ $\displaystyle\textstyle\tfrac{1}{\tilde{\beta}^{t}}\cdot\underbrace{\min(\kappa^{-1}\tau,(\kappa^{\tfrac{\tau}{\tau+1}}-1)\cdot d_{0}^{-\tau}\tilde{\beta}^{0})}_{\mu_{0}}.$ $$

We now focus on Inequality ([28](#A1.E28)).

(a) $T$ is a multiple of $3$ given by $T=3j$ for some integer $j\geq 0$. Telescoping Inequality ([28](#A1.E28)) over $t=\{3,6,9,\ldots,T\}$, we have:

$$ $\displaystyle\textstyle d_{T}^{-\tau}-d_{0}^{-\tau}\geq\textstyle\mu_{0}\sum_{t=1}^{T/3}\tfrac{1}{\tilde{\beta}^{3t}}\overset{\text{\char 172}}{\geq}\mathcal{O}(T^{a}),$ $$

where step ① use Lemma [A.9](#A1.Thmtheorem9) with $\ddot{\beta}^{t}=\tilde{\beta}^{t}$. This leads to:

$$ $\displaystyle d_{T}=(d_{T}^{-\tau})^{-1/\tau}\leq\mathcal{O}(T^{a})^{-1/\tau}=\mathcal{O}(1/(T^{a/\tau})).$ $$

(b) When $T$ is not a multiple of $3$, the use of analogous strategies results in the same complexity outcome.

∎

###### Lemma A.1 .

###### Proof.

###### Lemma A.2 .

###### Proof.

###### Lemma A.3 .

###### Proof.

###### Lemma A.4 .

###### Proof.

###### Lemma A.5 .

###### Proof.

###### Lemma A.6 .

###### Proof.

###### Lemma A.7 .

###### Proof.

###### Lemma A.8 .

###### Proof.

###### Lemma A.9 .

###### Proof.

###### Lemma A.10 .

###### Proof.

###### Lemma A.11 .

###### Proof.

## Appendix B Proofs for Section 2

### B.1 Proof of Lemma 2.4

To finish the proof of this lemma. We first provide the following three useful lemmas.

###### Lemma B.1 .

Let $\lambda\in[0,1]$, ${\mathbf{p}}_{1},{\mathbf{p}}_{2}\in\mathbb{R}^{m\times 1}$, and ${\mathbf{p}}=\lambda{\mathbf{p}}_{1}+(1-\lambda){\mathbf{p}}_{2}$. A function $\check{h}({\mathbf{y}})$ is $W_{h}$-weakly convex if and only if:

$$ $\displaystyle\check{h}({\mathbf{p}})\leq\lambda h({\mathbf{p}}_{1})+(1-\lambda)\check{h}({\mathbf{p}}_{2})+\tfrac{\lambda(1-\lambda)W_{h}}{2}\|{\mathbf{p}}_{1}-{\mathbf{p}}_{2}\|_{2}^{2}.$ $$

###### Proof.

We omit the proof for brevity.

∎

###### Lemma B.2 .

Let $\kappa\in(0,1)$, and ${\mathbf{p}}_{1},{\mathbf{p}}_{2},{\mathbf{y}}_{1},{\mathbf{y}}_{2}\in\mathbb{R}^{m\times 1}$. We have:

$$ $\displaystyle\kappa\|{\mathbf{p}}_{1}-{\mathbf{p}}_{2}\|_{2}^{2}-\tfrac{\kappa}{1-\kappa}\|{\mathbf{y}}_{1}-{\mathbf{y}}_{2}\|_{2}^{2}\leq\|{\mathbf{y}}_{1}-{\mathbf{p}}_{1}\|_{2}^{2}+\|{\mathbf{y}}_{2}-{\mathbf{p}}_{2}\|_{2}^{2}-2\langle{\mathbf{y}}_{1}-{\mathbf{p}}_{1},{\mathbf{y}}_{2}-{\mathbf{p}}_{2}\rangle.$ $$

###### Proof.

For all vectors $\{{\mathbf{w}}_{1},{\mathbf{w}}_{2},{\mathbf{w}}_{3}\}$ of suitable size, we have:

$$ $\displaystyle 0$ $\displaystyle\geq$ $\displaystyle-\|\kappa{\mathbf{w}}_{1}+{\mathbf{w}}_{2}+{\mathbf{w}}_{3}\|_{2}^{2}$ $\displaystyle=$ $\displaystyle-\kappa^{2}\|{\mathbf{w}}_{1}\|_{2}^{2}-\|{\mathbf{w}}_{2}\|_{2}^{2}-\|{\mathbf{w}}_{3}\|_{2}^{2}-2\kappa\langle{\mathbf{w}}_{2},{\mathbf{w}}_{1}\rangle-2\kappa\langle{\mathbf{w}}_{3},{\mathbf{w}}_{1}\rangle-2\langle{\mathbf{w}}_{2},{\mathbf{w}}_{3}\rangle$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle(1-\kappa)\{\kappa\|{\mathbf{w}}_{1}\|_{2}^{2}-\|{\mathbf{w}}_{2}\|_{2}^{2}-\|{\mathbf{w}}_{3}\|_{2}^{2}-2\langle{\mathbf{w}}_{2},{\mathbf{w}}_{3}\rangle\}-\kappa\|{\mathbf{w}}_{1}+{\mathbf{w}}_{2}+{\mathbf{w}}_{3}\|_{2}^{2},$ $$

where step ① uses basic reduction, with details omitted for brevity. Dividing both sides by $(1-\kappa)$, we have:

$$ $\displaystyle\kappa\|{\mathbf{w}}_{1}\|_{2}^{2}-\tfrac{\kappa}{1-\kappa}\|{\mathbf{w}}_{2}+{\mathbf{w}}_{3}+{\mathbf{w}}_{1}\|_{2}^{2}\leq\|{\mathbf{w}}_{2}\|_{2}^{2}+\|{\mathbf{w}}_{3}\|^{2}-2\langle{\mathbf{w}}_{2},{\mathbf{w}}_{3}\rangle.$ $$

Applying this inequality with ${\mathbf{w}}_{1}={\mathbf{p}}_{1}-{\mathbf{p}}_{2}$, ${\mathbf{w}}_{2}={\mathbf{y}}_{1}-{\mathbf{p}}_{1}$, and ${\mathbf{w}}_{3}={\mathbf{p}}_{2}-{\mathbf{y}}_{2}$, we conclude this lemma.

∎

###### Lemma B.3 .

Let $\kappa\triangleq\mu W_{h}\in(0,1)$, $\lambda\in[0,1]$ and ${\mathbf{p}}_{1},{\mathbf{p}}_{2},{\mathbf{y}}_{1},{\mathbf{y}}_{2}\in\mathbb{R}^{m\times 1}$. Let ${\mathbf{y}}=\lambda{\mathbf{y}}_{1}+(1-\lambda){\mathbf{y}}_{2}$ and ${\mathbf{p}}=\lambda{\mathbf{p}}_{1}+(1-\lambda){\mathbf{p}}_{2}$. We have:

$$ $\displaystyle\tfrac{\lambda(1-\lambda)W_{h}}{2}\|{\mathbf{p}}_{1}-{\mathbf{p}}_{2}\|_{2}^{2}+\tfrac{1}{2\mu}\|{\mathbf{y}}-{\mathbf{p}}\|_{2}^{2}\leq\Psi,$ $$

where $\Psi\triangleq\tfrac{\lambda(1-\lambda)W_{h}}{2(1-\mu W_{h})}\|{\mathbf{y}}_{1}-{\mathbf{y}}_{2}\|_{2}^{2}+\tfrac{\lambda}{2\mu}\|{\mathbf{y}}_{1}-{\mathbf{p}}_{1}\|_{2}^{2}+\tfrac{1-\lambda}{2\mu}\|{\mathbf{y}}_{2}-{\mathbf{p}}_{2}\|_{2}^{2}$.

###### Proof.

Applying Lemma [B.2](#A2.Thmtheorem2) with $\kappa\in(0,1)$, we have:

$$ $\displaystyle\tfrac{\mu W_{h}}{2}\|{\mathbf{p}}_{1}-{\mathbf{p}}_{2}\|_{2}^{2}-\tfrac{\mu W_{h}}{2(1-\mu W_{h})}\|{\mathbf{y}}_{1}-{\mathbf{y}}_{2}\|_{2}^{2}$ $\displaystyle\leq$ $\displaystyle\tfrac{1}{2}\|{\mathbf{y}}_{1}-{\mathbf{p}}_{1}\|_{2}^{2}+\tfrac{1}{2}\|{\mathbf{y}}_{2}-{\mathbf{p}}_{2}\|_{2}^{2}-\langle{\mathbf{y}}_{1}-{\mathbf{p}}_{1},{\mathbf{y}}_{2}-{\mathbf{p}}_{2}\rangle.$ $$

By multiplying both sides by $\tfrac{\lambda(1-\lambda)}{\mu}$, we have:

$$ $\displaystyle\tfrac{\lambda(1-\lambda)W_{h}}{2}\|{\mathbf{p}}_{1}-{\mathbf{p}}_{2}\|_{2}^{2}-\tfrac{\lambda(1-\lambda)W_{h}}{2(1-\mu W_{h})}\|{\mathbf{y}}_{1}-{\mathbf{y}}_{2}\|_{2}^{2}$ $\displaystyle\leq$ $\displaystyle\tfrac{\lambda(1-\lambda)}{2\mu}\cdot\|{\mathbf{y}}_{1}-{\mathbf{p}}_{1}\|_{2}^{2}+\tfrac{\lambda(1-\lambda)}{2\mu}\cdot\|{\mathbf{y}}_{2}-{\mathbf{p}}_{2}\|_{2}^{2}-\tfrac{1}{\mu}\langle\lambda({\mathbf{y}}_{1}-{\mathbf{p}}_{1}),(1-\lambda)({\mathbf{y}}_{2}-{\mathbf{p}}_{2})\rangle.$ $$

Using the fact that ${\mathbf{y}}-{\mathbf{p}}=\lambda({\mathbf{y}}_{1}-{\mathbf{p}}_{1})+(1-\lambda)({\mathbf{y}}_{2}-{\mathbf{p}}_{2})$, we obtain:

$$ $\displaystyle\tfrac{1}{2\mu}\|{\mathbf{y}}-{\mathbf{p}}\|_{2}^{2}=\tfrac{1}{2\mu}\|\lambda({\mathbf{y}}_{1}-{\mathbf{p}}_{1})+(1-\lambda)({\mathbf{y}}_{2}-{\mathbf{p}}_{2})\|_{2}^{2}.$ (30) $$

By adding Inequality ([B.3](#A2.Ex81)) and Equality ([30](#A2.E30)) together, and using the definition of $\Psi$, we conclude this lemma.

∎

Finally, we proof the original lemma.

###### Proof.

We assume $\mu\in(0,\tfrac{1}{W_{h}})$.

We let ${\mathbf{y}}_{1}\in\mathbb{R}^{m\times d}$ and ${\mathbf{y}}_{2}\in\mathbb{R}^{m\times 1}$.

We define $h_{\mu}({\mathbf{y}}_{1})=\min_{{\mathbf{y}}}h({\mathbf{y}})+\tfrac{1}{2\mu}\|{\mathbf{y}}-{\mathbf{y}}_{1}\|_{2}^{2}$, and $\mathbb{P}_{\mu}({\mathbf{y}}_{1})=\arg\min_{{\mathbf{y}}}h({\mathbf{y}})+\tfrac{1}{2\mu}\|{\mathbf{y}}-{\mathbf{y}}_{1}\|_{2}^{2}$.

We define $h_{\mu}({\mathbf{y}}_{2})=\min_{{\mathbf{y}}}h({\mathbf{y}})+\tfrac{1}{2\mu}\|{\mathbf{y}}-{\mathbf{y}}_{2}\|_{2}^{2}$, and $\mathbb{P}_{\mu}({\mathbf{y}}_{2})=\arg\min_{{\mathbf{y}}}h({\mathbf{y}})+\tfrac{1}{2\mu}\|{\mathbf{y}}-{\mathbf{y}}_{1}\|_{2}^{2}$.

We define ${\mathbf{y}}\triangleq\lambda{\mathbf{y}}_{1}+(1-\lambda){\mathbf{y}}_{2}$, and ${\mathbf{p}}\triangleq\lambda\mathbb{P}_{\mu}({\mathbf{y}}_{1})+(1-\lambda)\mathbb{P}_{\mu}({\mathbf{y}}_{2})$, where $\lambda\in[0,1]$.

We derive the following inequalities:

$$ $\displaystyle\textstyle h_{\mu}({\mathbf{y}})$ (31) $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\textstyle\tfrac{1}{2\mu}\|{\mathbf{y}}-{\mathbf{p}}\|_{2}^{2}+h({\mathbf{p}})$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\textstyle\tfrac{1}{2\mu}\|{\mathbf{y}}-{\mathbf{p}}\|_{2}^{2}+\lambda h(\mathbb{P}_{\mu}({\mathbf{y}}_{1}))+(1-\lambda)h(\mathbb{P}_{\mu}({\mathbf{y}}_{2}))+\tfrac{\lambda(1-\lambda)W_{h}}{2}\|\mathbb{P}_{\mu}({\mathbf{y}}_{1})-\mathbb{P}_{\mu}({\mathbf{y}}_{2})\|_{2}^{2}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\textstyle\lambda h(\mathbb{P}_{\mu}({\mathbf{y}}_{1}))+(1-\lambda)h(\mathbb{P}_{\mu}({\mathbf{y}}_{2}))+\tfrac{\lambda}{2\mu}\|{\mathbf{y}}_{1}-\mathbb{P}_{\mu}({\mathbf{y}}_{1})\|_{2}^{2}+\tfrac{1-\lambda}{2\mu}\|{\mathbf{y}}_{2}-\mathbb{P}_{\mu}({\mathbf{y}}_{2})\|_{2}^{2}$ $\displaystyle+\tfrac{\lambda(1-\lambda)W_{h}}{2(1-\mu W_{h})}\|{\mathbf{y}}_{1}-{\mathbf{y}}_{2}\|_{2}^{2}$ $\displaystyle\overset{\text{\char 175}}{=}$ $\displaystyle\textstyle\lambda h_{\mu}({\mathbf{y}}_{1})+(1-\lambda)h_{\mu}({\mathbf{y}}_{2})+\tfrac{\lambda(1-\lambda)W_{h}}{2(1-\mu W_{h})}\|{\mathbf{y}}_{1}-{\mathbf{y}}_{2}\|_{2}^{2},$ $$

where step ① uses $h_{\mu}({\mathbf{y}})=\textstyle\min_{{\mathbf{y}}^{\prime}}\tfrac{1}{2\mu}\|{\mathbf{y}}-{\mathbf{y}}^{\prime}\|_{2}^{2}+h({\mathbf{y}}^{\prime})$; step ② uses the $W_{h}$-weakly convexity of $h({\mathbf{y}})$ and Lemma [B.2](#A2.Thmtheorem2); step ③ uses Lemma [B.3](#A2.Thmtheorem3) with ${\mathbf{p}}_{1}=\mathbb{P}_{\mu}({\mathbf{y}}_{1})$ and ${\mathbf{p}}_{2}=\mathbb{P}_{\mu}({\mathbf{y}}_{2})$; step ④ uses the definition of $h_{\mu}({\mathbf{y}})$ for any ${\mathbf{y}}$, and it follows that: $h_{\mu}({\mathbf{y}}_{1})=h(\mathbb{P}_{\mu}({\mathbf{y}}_{1}))+\tfrac{1}{2\mu}\|{\mathbf{y}}_{1}-\mathbb{P}_{\mu}({\mathbf{y}}_{1})\|_{2}^{2}$, and $h_{\mu}({\mathbf{y}}_{2})=h(\mathbb{P}_{\mu}({\mathbf{y}}_{2}))+\tfrac{1}{2\mu}\|{\mathbf{y}}_{2}-\mathbb{P}_{\mu}({\mathbf{y}}_{2})\|_{2}^{2}$. Considering Inequality ([31](#A2.E31)) and in conjunction with Lemma [B.2](#A2.Thmtheorem2), we establish that the function $h_{\mu}({\mathbf{y}})$ is $(\tfrac{W_{h}}{1-\mu W_{h}})$-weakly convex.

∎

###### Lemma B.1 .

###### Proof.

###### Lemma B.2 .

###### Proof.

###### Lemma B.3 .

###### Proof.

###### Proof.

### B.2 Proof of Lemma 2.5

###### Proof.

Assume $0<\mu_{2}<\mu_{1}<\frac{1}{W_{h}}$, and fixing ${\mathbf{y}}\in\mathbb{R}^{m\times 1}$.

We define $h_{\mu_{1}}({\mathbf{y}})\triangleq\min_{{\mathbf{v}}}h({\mathbf{v}})+\frac{1}{2\mu_{1}}\|{\mathbf{v}}-{\mathbf{y}}\|_{2}^{2}$, and $\mathbb{P}_{\mu_{1}}({\mathbf{y}})=\textstyle\arg\min_{{\mathbf{v}}}\,h({\mathbf{v}})+\tfrac{1}{2\mu_{1}}\|{\mathbf{v}}-{\mathbf{y}}\|_{2}^{2}$.

We define $h_{\mu_{2}}({\mathbf{y}})\triangleq\min_{{\mathbf{v}}}h({\mathbf{v}})+\frac{1}{2\mu_{2}}\|{\mathbf{v}}-{\mathbf{y}}\|_{2}^{2}$, and $\mathbb{P}_{\mu_{2}}({\mathbf{y}})=\textstyle\arg\min_{{\mathbf{v}}}\,h({\mathbf{v}})+\tfrac{1}{2\mu_{2}}\|{\mathbf{v}}-{\mathbf{y}}\|_{2}^{2}$.

By the optimality of $\mathbb{P}_{\mu_{1}}({\mathbf{y}})$ and $\mathbb{P}_{\mu_{2}}({\mathbf{y}})$, we obtain:

$$ $\displaystyle\textstyle{\mathbf{y}}-\mathbb{P}_{\mu_{1}}({\mathbf{y}})$ $\displaystyle\in$ $\displaystyle\textstyle\mu_{1}\partial h(\mathbb{P}_{\mu_{1}}({\mathbf{y}}))$ (32) $\displaystyle\textstyle{\mathbf{y}}-\mathbb{P}_{\mu_{2}}({\mathbf{y}})$ $\displaystyle\in$ $\displaystyle\textstyle\mu_{2}\partial h(\mathbb{P}_{\mu_{2}}({\mathbf{y}})).$ (33) $$

(a) We now prove that $0\leq h_{\mu_{2}}({\mathbf{y}})-h_{\mu_{1}}({\mathbf{y}})$. For any ${\mathbf{s}}_{1}\in\partial h(\mathbb{P}_{\mu_{1}}({\mathbf{y}}))$ and ${\mathbf{s}}_{2}\in\partial h(\mathbb{P}_{\mu_{2}}({\mathbf{y}}))$, we have:

$$ $\displaystyle h_{\mu_{1}}({\mathbf{y}})-h_{\mu_{2}}({\mathbf{y}})$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\tfrac{1}{2\mu_{1}}\|{\mathbf{y}}-\mathbb{P}_{\mu_{1}}({\mathbf{y}})\|_{2}^{2}-\tfrac{1}{2\mu_{2}}\|{\mathbf{y}}-\mathbb{P}_{\mu_{2}}({\mathbf{y}})\|_{2}^{2}+h(\mathbb{P}_{\mu_{1}}({\mathbf{y}}))-h(\mathbb{P}_{\mu_{2}}({\mathbf{y}}))$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\tfrac{1}{2\mu_{1}}\|{\mathbf{y}}-\mathbb{P}_{\mu_{1}}({\mathbf{y}})\|_{2}^{2}-\tfrac{1}{2\mu_{2}}\|{\mathbf{y}}-\mathbb{P}_{\mu_{2}}({\mathbf{y}})\|_{2}^{2}+\langle\mathbb{P}_{\mu_{1}}({\mathbf{y}})-\mathbb{P}_{\mu_{2}}({\mathbf{y}}),{\mathbf{s}}_{1}\rangle+\tfrac{W_{h}}{2}\|\mathbb{P}_{\mu_{2}}({\mathbf{y}})-\mathbb{P}_{\mu_{1}}({\mathbf{y}})\|_{2}^{2}$ $\displaystyle\overset{\text{\char 174}}{=}$ $\displaystyle\tfrac{1}{2\mu_{1}}\|\mu_{1}{\mathbf{s}}_{1}\|_{2}^{2}-\tfrac{1}{2\mu_{2}}\|\mu_{2}{\mathbf{s}}_{2}\|_{2}^{2}+\langle\mu_{2}{\mathbf{s}}_{2}-\mu_{1}{\mathbf{s}}_{1},{\mathbf{s}}_{1}\rangle+\tfrac{W_{h}}{2}\|\mu_{1}{\mathbf{s}}_{1}-\mu_{2}{\mathbf{s}}_{2}\|_{2}^{2}$ $\displaystyle\overset{\text{\char 175}}{\leq}$ $\displaystyle\tfrac{1}{2\mu_{1}}\|\mu_{1}{\mathbf{s}}_{1}\|_{2}^{2}-\tfrac{1}{2\mu_{2}}\|\mu_{2}{\mathbf{s}}_{2}\|_{2}^{2}+\langle\mu_{2}{\mathbf{s}}_{2}-\mu_{1}{\mathbf{s}}_{1},{\mathbf{s}}_{1}\rangle+\tfrac{1}{2\mu_{1}}\|\mu_{1}{\mathbf{s}}_{1}-\mu_{2}{\mathbf{s}}_{2}\|_{2}^{2}$ $\displaystyle\overset{}{=}$ $\displaystyle-\tfrac{\mu_{2}}{2}\|{\mathbf{s}}_{2}\|_{2}^{2}\cdot(1-\tfrac{\mu_{2}}{\mu_{1}})$ $\displaystyle\overset{\text{\char 176}}{\leq}$ $\displaystyle 0,$ $$

where step ① uses the definition of $h_{\mu_{1}}({\mathbf{y}})$ and $h_{\mu_{2}}({\mathbf{y}})$; step ② uses weakly convexity of $h(\cdot)$; step ③ uses the optimality of $\mathbb{P}_{\mu_{1}}({\mathbf{y}})$ and $\mathbb{P}_{\mu_{2}}({\mathbf{y}})$ in Equations ([32](#A2.E32)) and ([33](#A2.E33)); step ④ uses $W_{h}\leq\tfrac{1}{\mu_{1}}$; step ⑤ uses $1\geq\tfrac{\mu_{2}}{\mu_{1}}$.

(b) We now prove that $h_{\mu_{2}}({\mathbf{y}})-h_{\mu_{1}}({\mathbf{y}})\leq\min\{\tfrac{\mu_{1}}{2\mu_{2}},1\}\cdot(\mu_{1}-\mu_{2})C_{h}^{2}$. For any ${\mathbf{s}}_{1}\in\partial h(\mathbb{P}_{\mu_{1}}({\mathbf{y}}))$ and ${\mathbf{s}}_{2}\in\partial h(\mathbb{P}_{\mu_{2}}({\mathbf{y}}))$, we have:

$$ $\displaystyle h_{\mu_{2}}({\mathbf{y}})-h_{\mu_{1}}({\mathbf{y}})$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\tfrac{1}{2\mu_{2}}\|{\mathbf{y}}-\mathbb{P}_{\mu_{2}}({\mathbf{y}})\|_{2}^{2}-\tfrac{1}{2\mu_{1}}\|{\mathbf{y}}-\mathbb{P}_{\mu_{1}}({\mathbf{y}})\|_{2}^{2}+h(\mathbb{P}_{\mu_{2}}({\mathbf{y}}))-h(\mathbb{P}_{\mu_{1}}({\mathbf{y}}))$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\tfrac{1}{2\mu_{2}}\|{\mathbf{y}}-\mathbb{P}_{\mu_{2}}({\mathbf{y}})\|_{2}^{2}-\tfrac{1}{2\mu_{1}}\|{\mathbf{y}}-\mathbb{P}_{\mu_{1}}({\mathbf{y}})\|_{2}^{2}+\langle\mathbb{P}_{\mu_{2}}({\mathbf{y}})-\mathbb{P}_{\mu_{1}}({\mathbf{y}}),{\mathbf{s}}_{1}\rangle+\tfrac{W_{h}}{2}\|\mathbb{P}_{\mu_{2}}({\mathbf{y}})-\mathbb{P}_{\mu_{1}}({\mathbf{y}})\|_{2}^{2}$ $\displaystyle\overset{\text{\char 174}}{=}$ $\displaystyle\tfrac{\mu_{2}}{2}\|{\mathbf{s}}_{1}\|_{2}^{2}-\tfrac{\mu_{1}}{2}\|{\mathbf{s}}_{2}\|_{2}^{2}+\langle\mu_{1}{\mathbf{s}}_{2}-\mu_{2}{\mathbf{s}}_{1},{\mathbf{s}}_{1}\rangle+\tfrac{W_{h}}{2}\|\mu_{1}{\mathbf{s}}_{2}-\mu_{2}{\mathbf{s}}_{1}\|_{2}^{2}$ $\displaystyle=$ $\displaystyle-\tfrac{\mu_{2}}{2}\|{\mathbf{s}}_{1}\|_{2}^{2}-\tfrac{\mu_{1}}{2}\|{\mathbf{s}}_{2}\|_{2}^{2}+\mu_{1}\langle{\mathbf{s}}_{1},{\mathbf{s}}_{2}\rangle+\tfrac{W_{h}}{2}\|\mu_{1}{\mathbf{s}}_{2}-\mu_{2}{\mathbf{s}}_{1}\|_{2}^{2}$ $\displaystyle\overset{\text{\char 175}}{\leq}$ $\displaystyle\min\{-\tfrac{\mu_{1}}{2}\|{\mathbf{s}}_{2}\|_{2}^{2}+\mu_{1}\langle{\mathbf{s}}_{1},{\mathbf{s}}_{2}\rangle+\tfrac{1}{2\mu_{2}}\|\mu_{1}{\mathbf{s}}_{2}-\mu_{2}{\mathbf{s}}_{1}\|_{2}^{2}-\tfrac{\mu_{2}}{2}\|{\mathbf{s}}_{1}\|_{2}^{2},$ $\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}-\tfrac{\mu_{1}}{2}\|{\mathbf{s}}_{2}\|_{2}^{2}+\mu_{1}\langle{\mathbf{s}}_{1},{\mathbf{s}}_{2}\rangle+\tfrac{1}{2\mu_{1}}\|\mu_{1}{\mathbf{s}}_{2}-\mu_{2}{\mathbf{s}}_{1}\|_{2}^{2}-\tfrac{\mu_{2}}{2}\|{\mathbf{s}}_{1}\|_{2}^{2}\}$ $\displaystyle\overset{}{=}$ $\displaystyle\min\{(-\mu_{2}+\mu_{1})\cdot\tfrac{\mu_{1}}{2\mu_{2}}\|{\mathbf{s}}_{2}\|_{2}^{2},(\mu_{1}-\mu_{2})\langle{\mathbf{s}}_{1},{\mathbf{s}}_{2}\rangle-\tfrac{\mu_{2}}{2}\|{\mathbf{s}}_{1}\|_{2}^{2}+\tfrac{\mu_{2}^{2}}{2\mu_{1}}\|{\mathbf{s}}_{1}\|_{2}^{2}\}$ $\displaystyle\overset{\text{\char 176}}{\leq}$ $\displaystyle\min\{\tfrac{\mu_{1}}{2\mu_{2}}\|{\mathbf{s}}_{2}\|_{2}^{2}\cdot(\mu_{1}-\mu_{2}),(\mu_{1}-\mu_{2})\langle{\mathbf{s}}_{1},{\mathbf{s}}_{2}\rangle\}$ $\displaystyle\overset{\text{\char 177}}{\leq}$ $\displaystyle\min\{\tfrac{\mu_{1}}{2\mu_{2}}\cdot(\mu_{1}-\mu_{2}),(\mu_{1}-\mu_{2})\}\cdot C_{h}^{2}$ $\displaystyle\overset{}{=}$ $\displaystyle\min\{\tfrac{\mu_{1}}{2\mu_{2}},1\}\cdot(\mu_{1}-\mu_{2})\cdot C_{h}^{2},$ $$

where step ① uses the definition of $h_{\mu_{1}}({\mathbf{y}})$ and $h_{\mu_{2}}({\mathbf{y}})$; step ② uses the weakly convexity of $h(\cdot)$; step ③ uses the optimality of $\mathbb{P}_{\mu_{2}}({\mathbf{y}})$ and $\mathbb{P}_{\mu_{1}}({\mathbf{y}})$ in Equations ([32](#A2.E32)) and ([33](#A2.E33)); step ④ uses $W_{h}\leq\frac{1}{\mu_{1}}$ and $W_{h}\leq\frac{1}{\mu_{2}}$; step ⑤ uses $\mu_{2}\leq\mu_{1}$; step ⑥ uses $\|{\mathbf{s}}_{1}\|^{2}_{2}\leq C^{2}_{h}$, $\|{\mathbf{s}}_{2}\|^{2}_{2}\leq C^{2}_{h}$, and $\langle{\mathbf{s}}_{1},{\mathbf{s}}_{2}\rangle\leq\|{\mathbf{s}}_{1}\|_{2}\|{\mathbf{s}}_{2}\|_{2}\leq C_{h}^{2}$.

∎

###### Proof.

### B.3 Proof of Lemma 2.6

###### Proof.

Assume $0<\mu_{2}<\mu_{1}\leq\tfrac{1}{2W_{h}}$, and fixing ${\mathbf{y}}\in\mathbb{R}^{m\times 1}$.

Using the result in Lemma [2.3](#S2.Thmtheorem3), we establish that the gradient of $h_{\mu}({\mathbf{y}})$ w.r.t ${\mathbf{y}}$ can be computed as:

$$ $\displaystyle\nabla h_{\mu}({\mathbf{y}})=\mu^{-1}({\mathbf{y}}-\mathbb{P}_{\mu}({\mathbf{y}})).$ $$

The gradient of the mapping $\nabla h_{\mu}({\mathbf{y}})$ w.r.t. the variable $1/\mu$ can be computed as: $\nabla_{1/\mu}\left(\nabla h_{\mu}({\mathbf{y}})\right)={\mathbf{y}}-\mathbb{P}_{\mu}({\mathbf{y}})$. We further obtain:

$$ $\displaystyle\|\nabla_{1/\mu}\left(\nabla h_{\mu}({\mathbf{y}})\right)\|_{2}=\|{\mathbf{y}}-\mathbb{P}_{\mu}({\mathbf{y}})\|_{2}\overset{\text{\char 172}}{=}\mu\|\partial h(\mathbb{P}_{\mu}({\mathbf{y}}))\|_{2}\leq\mu C_{h}.$ $$

Here, step ① uses the optimality of $\mathbb{P}_{\mu}({\mathbf{y}})$ that: $\mathbf{0}\in\partial h(\mathbb{P}_{\mu}({\mathbf{y}}))+\tfrac{1}{\mu}(\mathbb{P}_{\mu}({\mathbf{y}})-{\mathbf{y}})$. Therefore, for all $\mu\in(0,\tfrac{1}{2W_{h}}]$, we have:

$$ $\displaystyle\frac{\|\nabla h_{\mu}({\mathbf{y}})-\nabla h_{\mu^{\prime}}({\mathbf{y}})\|_{2}}{|1/\mu-1/\mu^{\prime}|}\leq\mu C_{h}.$ $$

Letting $\mu=\mu_{1}$ and $\mu^{\prime}=\mu_{2}$, we have: $\|\nabla h_{\mu_{1}}({\mathbf{y}})-\nabla h_{\mu_{2}}({\mathbf{y}})\|_{2}\leq|1-\mu_{1}/\mu_{2}|C_{h}=(\mu_{1}/\mu_{2}-1)C_{h}$.

∎

###### Proof.

### B.4 Proofs for Lemma 2.10

###### Proof.

We let $\mathbf{\Delta}\in\mathbb{R}^{n\times r}$ and ${\mathbf{X}}\in\mathcal{M}$. We define $\mathbf{U}\triangleq\mathbf{\Delta}^{\mathsf{T}}{\mathbf{X}}\in\mathbb{R}^{r\times r}$.

We derive the following results:

$$ $\displaystyle\|{\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}}\mathcal{M}}(\mathbf{\Delta})\|_{\mathsf{F}}^{2}-\|\mathbf{\Delta}\|_{\mathsf{F}}^{2}$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\|\mathbf{\Delta}-\tfrac{1}{2}{\mathbf{X}}(\mathbf{\Delta}^{\mathsf{T}}{\mathbf{X}}+{\mathbf{X}}^{\mathsf{T}}\mathbf{\Delta})\|_{\mathsf{F}}^{2}-\|\mathbf{\Delta}\|_{\mathsf{F}}^{2}$ $\displaystyle\overset{}{=}$ $\displaystyle\tfrac{1}{4}\|{\mathbf{X}}(\mathbf{\Delta}^{\mathsf{T}}{\mathbf{X}}+{\mathbf{X}}^{\mathsf{T}}\mathbf{\Delta})\|_{\mathsf{F}}^{2}-\langle\mathbf{\Delta},{\mathbf{X}}(\mathbf{\Delta}^{\mathsf{T}}{\mathbf{X}}+{\mathbf{X}}^{\mathsf{T}}\mathbf{\Delta})\rangle$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle\tfrac{1}{4}\|\mathbf{\Delta}^{\mathsf{T}}{\mathbf{X}}+{\mathbf{X}}^{\mathsf{T}}\mathbf{\Delta}\|_{\mathsf{F}}^{2}-\langle\mathbf{\Delta},{\mathbf{X}}(\mathbf{\Delta}^{\mathsf{T}}{\mathbf{X}}+{\mathbf{X}}^{\mathsf{T}}\mathbf{\Delta})\rangle$ $\displaystyle\overset{\text{\char 174}}{=}$ $\displaystyle\tfrac{1}{4}\|\mathbf{U}+\mathbf{U}^{\mathsf{T}}\|_{\mathsf{F}}^{2}-\langle\mathbf{U}+\mathbf{U}^{\mathsf{T}},\mathbf{U}\rangle$ $\displaystyle\overset{\text{\char 175}}{=}$ $\displaystyle\tfrac{1}{4}\|\mathbf{U}+\mathbf{U}^{\mathsf{T}}\|_{\mathsf{F}}^{2}-\langle\mathbf{U}+\mathbf{U}^{\mathsf{T}},\mathbf{U}+\mathbf{U}^{\mathsf{T}}\rangle\cdot\tfrac{1}{2}$ $\displaystyle\overset{}{=}$ $\displaystyle-\tfrac{1}{4}\|\mathbf{U}+\mathbf{U}^{\mathsf{T}}\|_{\mathsf{F}}^{2}\leq 0,$ $$

where step ① uses ${\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}}\mathcal{M}}(\mathbf{\Delta})=\mathbf{\Delta}-\tfrac{1}{2}{\mathbf{X}}(\mathbf{\Delta}^{\mathsf{T}}{\mathbf{X}}+{\mathbf{X}}^{\mathsf{T}}\mathbf{\Delta})$ for all $\mathbf{\Delta}\in\mathbb{R}^{n\times r}$ []; step ② uses the fact that $\|{\mathbf{X}}\mathbf{P}\|_{\mathsf{F}}^{2}={\rm{tr}}(\mathbf{P}{\mathbf{X}}^{\mathsf{T}}{\mathbf{X}}\mathbf{P}^{\mathsf{T}})=\|\mathbf{P}\|_{\mathsf{F}}^{2}$ for all ${\mathbf{X}}\in\mathcal{M}$; step ③ uses the definition of $\mathbf{U}\triangleq\mathbf{\Delta}^{\mathsf{T}}{\mathbf{X}}$; step ④ uses the symmetric properties of the matrix $(\mathbf{U}+\mathbf{U}^{\mathsf{T}})$.

∎

###### Proof.

### B.5 Proof of Lemma 2.11

###### Proof.

We let ${\mathbf{G}}\in\mathbb{R}^{n\times r}$, ${\mathbf{X}}\in\mathcal{M}$, and $\rho\in(0,+\infty)$.

We define ${\mathbf{U}}\triangleq{\mathbf{G}}^{\mathsf{T}}{\mathbf{X}}$, and ${\mathbf{D}}_{\rho}\triangleq\rho{\mathbf{X}}{\mathbf{G}}^{\mathsf{T}}{\mathbf{X}}+(1-\rho){\mathbf{X}}{\mathbf{X}}^{\mathsf{T}}{\mathbf{G}}-{\mathbf{G}}$.

First, we have the following equalities:

$$ $\displaystyle\langle{\mathbf{G}},{\mathbf{D}}_{\rho}\rangle$ $\displaystyle=$ $\displaystyle\langle{\mathbf{G}},\rho{\mathbf{X}}{\mathbf{G}}^{\mathsf{T}}{\mathbf{X}}+(1-\rho){\mathbf{X}}{\mathbf{X}}^{\mathsf{T}}{\mathbf{G}}-{\mathbf{G}}\rangle$ (34) $\displaystyle\overset{}{=}$ $\displaystyle\rho{\rm{tr}}({\mathbf{G}}^{\mathsf{T}}{\mathbf{X}}{\mathbf{G}}^{\mathsf{T}}{\mathbf{X}})+(1-\rho){\rm{tr}}({\mathbf{G}}^{\mathsf{T}}{\mathbf{X}}{\mathbf{X}}^{\mathsf{T}}{\mathbf{G}})-\langle{\mathbf{G}},{\mathbf{G}}\rangle$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\rho{\rm{tr}}({\mathbf{U}}{\mathbf{U}})+(1-\rho){\rm{tr}}({\mathbf{U}}{\mathbf{U}}^{\mathsf{T}})-\langle{\mathbf{G}},{\mathbf{G}}\rangle,$ $$

where step ① uses ${\mathbf{U}}\triangleq{\mathbf{G}}^{\mathsf{T}}{\mathbf{X}}$.

Second, we derive the following equalities:

$$ $\displaystyle\|{\mathbf{D}}_{\rho}\|_{\mathsf{F}}^{2}$ $\displaystyle=$ $\displaystyle\langle\rho{\mathbf{X}}{\mathbf{G}}^{\mathsf{T}}{\mathbf{X}}+(1-\rho){\mathbf{X}}{\mathbf{X}}^{\mathsf{T}}{\mathbf{G}}-{\mathbf{G}},\rho{\mathbf{X}}{\mathbf{G}}^{\mathsf{T}}{\mathbf{X}}+(1-\rho){\mathbf{X}}{\mathbf{X}}^{\mathsf{T}}{\mathbf{G}}-{\mathbf{G}}\rangle$ (35) $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\rho^{2}{\rm{tr}}({\mathbf{U}}^{\mathsf{T}}{\mathbf{U}})+\rho(1-\rho){\rm{tr}}({\mathbf{U}}^{\mathsf{T}}{\mathbf{U}}^{\mathsf{T}})-\rho{\rm{tr}}({\mathbf{U}}^{\mathsf{T}}{\mathbf{U}}^{\mathsf{T}})$ $\displaystyle+(1-\rho)\rho{\rm{tr}}({\mathbf{U}}{\mathbf{U}})+(1-\rho)^{2}{\rm{tr}}({\mathbf{U}}{\mathbf{U}}^{\mathsf{T}})-(1-\rho){\rm{tr}}({\mathbf{U}}{\mathbf{U}}^{\mathsf{T}})$ $\displaystyle-\rho{\rm{tr}}({\mathbf{U}}{\mathbf{U}})-(1-\rho){\rm{tr}}({\mathbf{U}}{\mathbf{U}}^{\mathsf{T}})+\langle{\mathbf{G}},{\mathbf{G}}\rangle$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle(2\rho^{2}-1)\cdot{\rm{tr}}({\mathbf{U}}^{\mathsf{T}}{\mathbf{U}})-2\rho^{2}\cdot{\rm{tr}}({\mathbf{U}}{\mathbf{U}})+\langle{\mathbf{G}},{\mathbf{G}}\rangle,$ $$

where step ① uses ${\mathbf{U}}\triangleq{\mathbf{G}}^{\mathsf{T}}{\mathbf{X}}$ and ${\mathbf{X}}^{\mathsf{T}}{\mathbf{X}}={\mathbf{I}}_{r}$; step ② uses ${\rm{tr}}({\mathbf{U}}^{\mathsf{T}}{\mathbf{U}}^{\mathsf{T}})={\rm{tr}}({\mathbf{U}}{\mathbf{U}})$.

Third, we have:

$$ $\displaystyle{\rm{tr}}({\mathbf{G}}^{\mathsf{T}}{\mathbf{G}})-{\rm{tr}}({\mathbf{U}}^{\mathsf{T}}{\mathbf{U}})\overset{\text{\char 172}}{=}\langle{\mathbf{G}}{\mathbf{G}}^{\mathsf{T}},{\mathbf{I}}_{n}-{\mathbf{X}}{\mathbf{X}}^{\mathsf{T}}\rangle\overset{\text{\char 173}}{\geq}0,$ (36) $$

where step ① uses ${\mathbf{U}}\triangleq{\mathbf{G}}^{\mathsf{T}}{\mathbf{X}}$; step ② uses the fact that the matrix $({\mathbf{I}}_{n}-{\mathbf{X}}{\mathbf{X}}^{\mathsf{T}})$ only contains eigenvalues that are 0 0 or $1$.

(a) We now prove that $\langle{\mathbf{G}},{\mathbf{D}}_{\rho}\rangle\leq-\tfrac{1}{\max(1,2\rho)}\|{\mathbf{D}}_{\rho}\|_{\mathsf{F}}^{2}$. We discuss two cases. Case (i): $\rho\in(0,\frac{1}{2}]$. We have:

$$ $\|{\mathbf{D}}_{\rho}\|_{\mathsf{F}}^{2}+\langle{\mathbf{G}},{\mathbf{D}}_{\rho}\rangle\overset{\text{\char 172}}{=}\rho(1-2\rho)\cdot({\rm{tr}}({\mathbf{U}}{\mathbf{U}})-{\rm{tr}}({\mathbf{U}}{\mathbf{U}}^{\mathsf{T}}))\overset{\text{\char 173}}{\leq}0,$ $$

where step ① uses Inequalities ([34](#A2.E34)) and ([35](#A2.E35)); step ② uses $\rho(1-2\rho)\geq 0$ for all $\rho\in(0,\frac{1}{2}]$, and ${\rm{tr}}({\mathbf{U}}{\mathbf{U}})-{\rm{tr}}({\mathbf{U}}{\mathbf{U}}^{\mathsf{T}})\leq 0$ for all ${\mathbf{U}}\in\mathbb{R}^{r\times r}$.
Case (ii): $\rho\in[\frac{1}{2},\infty)$. We have:

$$ $\textstyle\|{\mathbf{D}}_{\rho}\|_{\mathsf{F}}^{2}+2\rho\langle{\mathbf{G}},{\mathbf{D}}_{\rho}\rangle\overset{\text{\char 172}}{=}(2\rho-1)({\rm{tr}}({\mathbf{U}}{\mathbf{U}}^{\mathsf{T}})-\langle{\mathbf{G}},{\mathbf{G}}\rangle)\overset{\text{\char 173}}{\leq}0,$ $$

where step ① uses Inequalities ([34](#A2.E34)) and ([35](#A2.E35)); step ② uses $2\rho-1\geq 0$ for all $\rho\in[\frac{1}{2},\infty)$, and Inequality([36](#A2.E36)). Therefore, we conclude that: $\langle{\mathbf{G}},{\mathbf{D}}_{\rho}\rangle\leq-\tfrac{1}{\max(1,2\rho)}\|{\mathbf{D}}_{\rho}\|_{\mathsf{F}}^{2}$.

(b) We now prove that $\|{\mathbf{D}}_{\rho}\|_{\mathsf{F}}\geq\min(1,2\rho)\|{\mathbf{D}}_{1/2}\|_{\mathsf{F}}$. We consider two cases. Case (i): $\rho\in(0,\frac{1}{2}]$. We have:

$$ $(2\rho)^{2}\|{\mathbf{D}}_{1/2}\|_{\mathsf{F}}^{2}-\|{\mathbf{D}}_{\rho}\|_{\mathsf{F}}^{2}\overset{\text{\char 172}}{=}(4\rho^{2}-1)\cdot({\rm{tr}}({\mathbf{G}}^{\mathsf{T}}{\mathbf{G}})-{\rm{tr}}({\mathbf{U}}^{\mathsf{T}}{\mathbf{U}}))\overset{\text{\char 173}}{\leq}0,$ $$

where step ① uses Inequality ([35](#A2.E35)); step ② uses $4\rho^{2}-1\leq 0$ for all $\rho\in(0,\frac{1}{2}]$, and Inequality ([36](#A2.E36)).
Case (ii): $\rho\in(\frac{1}{2},\infty)$. We have:

$$ $\|{\mathbf{D}}_{1/2}\|_{\mathsf{F}}^{2}-\|{\mathbf{D}}_{\rho}\|_{\mathsf{F}}^{2}\overset{\text{\char 172}}{=}(2\rho^{2}-\tfrac{1}{2})\cdot({\rm{tr}}({\mathbf{U}}{\mathbf{U}})-{\rm{tr}}({\mathbf{U}}^{\mathsf{T}}{\mathbf{U}}))\overset{\text{\char 173}}{\leq}0,$ $$

where step ① uses Inequalities ([34](#A2.E34)) and ([35](#A2.E35)); step ② uses $2\rho^{2}-\tfrac{1}{2}\geq 0$ for all $\rho\in(\frac{1}{2},\infty)$, and the fact that ${\rm{tr}}({\mathbf{U}}{\mathbf{U}})-{\rm{tr}}({\mathbf{U}}{\mathbf{U}}^{\mathsf{T}})\leq 0$ for all ${\mathbf{U}}\in\mathbb{R}^{r\times r}$. Therefore, we conclude that $\|{\mathbf{D}}_{\rho}\|_{\mathsf{F}}\geq\min(1,2\rho)\|{\mathbf{D}}_{1/2}\|_{\mathsf{F}}$.

(c) We now prove that $\|{\mathbf{D}}_{\rho}\|_{\mathsf{F}}\leq\max(1,2\rho)\|{\mathbf{D}}_{1/2}\|_{\mathsf{F}}$. We consider two cases. Case (i): $\rho\in(0,\frac{1}{2}]$. We have:

$$ $\|{\mathbf{D}}_{1/2}\|_{\mathsf{F}}^{2}-\|{\mathbf{D}}_{\rho}\|_{\mathsf{F}}^{2}\overset{\text{\char 172}}{=}(2\rho^{2}-\tfrac{1}{2})\cdot({\rm{tr}}({\mathbf{U}}{\mathbf{U}})-{\rm{tr}}({\mathbf{U}}^{\mathsf{T}}{\mathbf{U}}))\overset{\text{\char 173}}{\geq}0,$ $$

where step ① uses Inequality ([35](#A2.E35)); step ② uses $2\rho^{2}-\tfrac{1}{2}\leq 0$ for all $\rho\in(0,\frac{1}{2}]$, and the fact that ${\rm{tr}}({\mathbf{U}}{\mathbf{U}})-{\rm{tr}}({\mathbf{U}}{\mathbf{U}}^{\mathsf{T}})\leq 0$ for all ${\mathbf{U}}\in\mathbb{R}^{r\times r}$.
Case (ii): $\rho\in(\frac{1}{2},\infty)$. We have:

$$ $(2\rho)^{2}\|{\mathbf{D}}_{1/2}\|_{\mathsf{F}}^{2}-\|{\mathbf{D}}_{\rho}\|_{\mathsf{F}}^{2}\overset{\text{\char 172}}{=}(4\rho^{2}-1)\cdot({\rm{tr}}({\mathbf{G}}^{\mathsf{T}}{\mathbf{G}})-{\rm{tr}}({\mathbf{U}}^{\mathsf{T}}{\mathbf{U}}))\overset{\text{\char 173}}{\geq}0,$ $$

where step ① uses Inequalities ([34](#A2.E34)) and ([35](#A2.E35)); step ② uses $4\rho^{2}-1\geq 0$ for all $\rho\in(\frac{1}{2},\infty)$, and Inequality ([36](#A2.E36)). Therefore, we conclude that: $\|{\mathbf{D}}_{\rho}\|_{\mathsf{F}}\geq\min(1,2\rho)\|{\mathbf{D}}_{1/2}\|_{\mathsf{F}}$.

(d) We now prove that $\|{\mathbf{D}}_{1}\|_{\mathsf{F}}\leq\max(1,1/\rho)\|{\mathbf{D}}_{\rho}\|_{\mathsf{F}}$. We consider two cases. Case (i): $\rho\in(0,1]$. We have:

$$ $\textstyle\rho^{2}\|{\mathbf{D}}_{1}\|_{\mathsf{F}}^{2}-\|{\mathbf{D}}_{\rho}\|^{2}_{\mathsf{F}}\overset{\text{\char 172}}{=}\textstyle(1-\rho^{2})({\rm{tr}}({\mathbf{U}}^{\mathsf{T}}{\mathbf{U}})-\langle{\mathbf{G}},{\mathbf{G}}\rangle)\overset{\text{\char 173}}{\leq}0,$ $$

where step ① uses Inequalities ([34](#A2.E34)) and ([35](#A2.E35)); step ② uses $1-\rho^{2}\geq 0$, and Inequality ([36](#A2.E36)).
Case (ii): $\rho\in(1,\infty)$. We have:

$$ $\|{\mathbf{D}}_{1}\|_{\mathsf{F}}^{2}-\|{\mathbf{D}}_{\rho}\|^{2}_{\mathsf{F}}\overset{\text{\char 172}}{=}(2-2\rho^{2})({\rm{tr}}({\mathbf{U}}^{\mathsf{T}}{\mathbf{U}})-{\rm{tr}}({\mathbf{U}}{\mathbf{U}}))\overset{}{\leq}0,$ $$

where step ① uses Inequality ([35](#A2.E35)); step ② uses $4\rho^{2}-1\leq 0$ for all $\rho\in(0,\frac{1}{2}]$, and the fact that ${\rm{tr}}({\mathbf{U}}{\mathbf{U}})-{\rm{tr}}({\mathbf{U}}{\mathbf{U}}^{\mathsf{T}})\leq 0$ for all ${\mathbf{U}}\in\mathbb{R}^{r\times r}$. Therefore, we conclude that: $\|{\mathbf{D}}_{1}\|_{\mathsf{F}}\leq\max(1,1/\rho)\|{\mathbf{D}}_{\rho}\|_{\mathsf{F}}$.

∎

###### Proof.

### B.6 Proof of Lemma 2.12

###### Proof.

Recall that the following first-order optimality conditions are equivalent for all ${\mathbf{X}}\in\mathbb{R}^{n\times r}$:

$$ $\displaystyle\left(\mathbf{0}\in\partial\mathcal{I}_{\mathcal{M}}({\mathbf{X}})+\nabla f({\mathbf{X}})\right)\Leftrightarrow\left(\mathbf{0}\in{\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}}\mathcal{M}}(\nabla f({\mathbf{X}}))\right).$ (37) $$

Therefore, we derive the following results:

$$ $\displaystyle{\rm{dist}}(\mathbf{0},\partial\mathcal{I}_{\mathcal{M}}({\mathbf{X}})+\nabla f({\mathbf{X}}))$ $\displaystyle=$ $\displaystyle\inf_{\mathbf{R}\in\nabla f({\mathbf{X}})+\partial\mathcal{I}_{\mathcal{M}}({\mathbf{X}})}\|\mathbf{R}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\inf_{\mathbf{R}\in{\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}}\mathcal{M}}(\nabla f({\mathbf{X}}))}\|\mathbf{R}\|_{\mathsf{F}}$ $\displaystyle\overset{}{=}$ $\displaystyle\|{\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}}\mathcal{M}}(\nabla f({\mathbf{X}}))\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle\|\nabla f({\mathbf{X}})-\tfrac{1}{2}{\mathbf{X}}({\mathbf{X}}^{\mathsf{T}}\nabla f({\mathbf{X}})+\nabla f({\mathbf{X}})^{\mathsf{T}}{\mathbf{X}})\|_{\mathsf{F}}$ $\displaystyle\overset{}{=}$ $\displaystyle\|({\mathbf{I}}-\tfrac{1}{2}{\mathbf{X}}{\mathbf{X}}^{\mathsf{T}})(\nabla f({\mathbf{X}})-{\mathbf{X}}\nabla f({\mathbf{X}})^{\mathsf{T}}{\mathbf{X}})\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\|\nabla f({\mathbf{X}})-{\mathbf{X}}\nabla f({\mathbf{X}})^{\mathsf{T}}{\mathbf{X}}\|_{\mathsf{F}},$ $$

where step ① uses Formulation ([37](#A2.E37)); step ② uses ${\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}}\mathcal{M}}(\mathbf{\Delta})=\mathbf{\Delta}-\tfrac{1}{2}{\mathbf{X}}(\mathbf{\Delta}^{\mathsf{T}}{\mathbf{X}}+{\mathbf{X}}^{\mathsf{T}}\mathbf{\Delta})$ for all $\mathbf{\Delta}\in\mathbb{R}^{n\times r}$ []; step ③ uses the norm inequality $\|\mathbf{A}\mathbf{B}\|_{\mathsf{F}}\leq\|\mathbf{A}\|\|\mathbf{B}\|_{\mathsf{F}}$, and fact that the matrix ${\mathbf{I}}-\frac{1}{2}{\mathbf{X}}{\mathbf{X}}^{\mathsf{T}}$ only contains eigenvalues that are $\frac{1}{2}$ or $1$.

∎

###### Proof.

## Appendix C Proofs for Section 3

### C.1 Proof of Lemma 3.4

###### Proof.

We define $\overline{\rm{X}}\triangleq\sqrt{r}$, $\overline{\rm{z}}\triangleq\|{\mathbf{z}}^{0}\|+\tfrac{\sigma C_{h}}{2-\sigma}$, $\overline{\rm{y}}\triangleq R_{A}+\tfrac{1}{\beta^{0}}C_{h}+\tfrac{1}{\beta^{0}}\overline{\rm{z}}$, where $\sigma\in[1,2)$.

(a) Given ${\mathbf{X}}^{t+1}\in\mathcal{M}$, we have: $\|{\mathbf{X}}^{t}\|_{\mathsf{F}}\leq\overline{\rm{X}}\triangleq\sqrt{r}$.

(b) We show that $\|{\mathbf{z}}^{t}\|\leq\overline{\rm{z}}$. Using the first-order optimality condition of ${\mathbf{y}}^{t+1}$in Algorithm [1](#alg1) that: ${\mathbf{y}}^{t+1}=\arg\min_{{\mathbf{y}}}h_{\mu^{t}}({\mathbf{y}})+\frac{\beta^{t}}{2}\|{\mathbf{y}}-\mathbf{b}\|_{2}^{2}$, where $\mathbf{b}\triangleq{\mathbf{y}}_{{\sf c}}^{t}-\nabla_{{\mathbf{y}}}\mathcal{S}^{t}({\mathbf{X}}^{t+1},{\mathbf{y}}_{{\sf c}}^{t})/\beta^{t}$, for all $t\geq 0$, we have:

$$ $\displaystyle\mathbf{0}$ $\displaystyle=$ $\displaystyle\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})-{\mathbf{z}}^{t}+\beta^{t}({\mathbf{y}}^{t+1}-{\mathbf{y}}_{{\sf c}}^{t})$ (38) $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})-{\mathbf{z}}^{t}+\beta^{t}({\mathbf{y}}_{{\sf c}}^{t}-{\mathcal{A}}({\mathbf{X}}^{t+1}))+\beta^{t}({\mathbf{y}}^{t+1}-{\mathbf{y}}_{{\sf c}}^{t})$ $\displaystyle=$ $\displaystyle\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})-{\mathbf{z}}^{t}+\beta^{t}({\mathbf{y}}^{t+1}-{\mathcal{A}}({\mathbf{X}}^{t+1}))$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})-{\mathbf{z}}^{t}+\tfrac{1}{\sigma}({\mathbf{z}}^{t}-{\mathbf{z}}^{t+1}),$ $$

where step ① uses $\nabla_{{\mathbf{y}}}\mathcal{S}^{t}({\mathbf{X}}^{t+1},{\mathbf{y}})=-{\mathbf{z}}^{t}+\beta^{t}({\mathbf{y}}-{\mathcal{A}}({\mathbf{X}}^{t+1}))$; step ② uses ${\mathbf{z}}^{t+1}={\mathbf{z}}^{t}+\sigma\beta^{t}({\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1})$.

For all $t\geq 0$, we have:

$$ $\displaystyle\|(\sigma-1){\mathbf{z}}^{t-1}+{\mathbf{z}}^{t}\|=\sigma\|\nabla h_{\mu^{t-1}}({\mathbf{y}}^{t})\|\overset{\text{\char 172}}{\leq}\sigma C_{h},$ (39) $$

where step ① uses the fact that $h_{\mu}({\mathbf{y}})$ is $C_{h}$-Lipschitz continuous for all $\mu$ and ${\mathbf{y}}$, as shown in Lemma [2.3](#S2.Thmtheorem3). For all $t\geq 0$, we have:

$$ $\displaystyle\|{\mathbf{z}}^{t}\|$ $\displaystyle=$ $\displaystyle\|(\sigma-1){\mathbf{z}}^{t-1}+{\mathbf{z}}^{t}-(\sigma-1){\mathbf{z}}^{t-1}\|$ $\displaystyle\leq$ $\displaystyle\|(\sigma-1){\mathbf{z}}^{t-1}+{\mathbf{z}}^{t}\|+\|(\sigma-1){\mathbf{z}}^{t-1}\|$ $\displaystyle\leq$ $\displaystyle\sigma C_{h}+(\sigma-1)\|{\mathbf{z}}^{t-1}\|.$ $$

We further obtain:

$$ $\displaystyle t=1,~{}$ $\displaystyle\|{\mathbf{z}}^{1}\|\leq\sigma C_{h}+(\sigma-1)\|{\mathbf{z}}^{0}\|$ $\displaystyle t=2,~{}$ $\displaystyle\|{\mathbf{z}}^{2}\|\leq\sigma C_{h}+(\sigma-1)\|{\mathbf{z}}^{1}\|\leq\sigma C_{h}+(\sigma-1)\sigma C_{h}+(\sigma-1)^{2}\|{\mathbf{z}}^{0}\|$ $\displaystyle t=3,~{}$ $\displaystyle\|{\mathbf{z}}^{3}\|\leq\sigma C_{h}+(\sigma-1)\|{\mathbf{z}}^{2}\|\leq\sigma C_{h}+(\sigma-1)\sigma C_{h}+(\sigma-1)^{2}\sigma C_{h}+(\sigma-1)^{3}\|{\mathbf{z}}^{0}\|$ $\displaystyle\ldots$ $$

Using basic reduction, we obtain:

$$ $\displaystyle\textstyle\|{\mathbf{z}}^{T}\|$ $\displaystyle\leq$ $\displaystyle\textstyle(\sigma-1)^{T}\|{\mathbf{z}}^{0}\|+\sigma C_{h}\sum_{t=0}^{T-1}(\sigma-1)^{t}$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle(\sigma-1)^{T}\|{\mathbf{z}}^{0}\|+\sigma C_{h}\cdot\tfrac{1-(\sigma-1)^{T}}{1-(\sigma-1)}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\|{\mathbf{z}}^{0}\|+\tfrac{\sigma C_{h}}{2-\sigma}\triangleq\overline{\rm{z}},$ $$

where step ① uses the summation formula of a geometric sequence; step ② uses $(\sigma-1)^{T}\in(0,1)$.

(c) We show that $\|{\mathbf{y}}^{t}\|\leq\overline{\rm{y}}$. For all $t\geq 0$, we have:

$$ $\displaystyle\mathbf{0}$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})-{\mathbf{z}}^{t}+\tfrac{1}{\sigma}({\mathbf{z}}^{t}-{\mathbf{z}}^{t+1})$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})-{\mathbf{z}}^{t}+\beta^{t}({\mathbf{y}}^{t+1}-{\mathcal{A}}({\mathbf{X}}^{t+1})),$ $$

where step ① uses Equality ([45](#A4.E45)); step ② uses ${\mathbf{z}}^{t+1}={\mathbf{z}}^{t}+\sigma\beta^{t}({\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1})$. For all $t\geq 0$, we obtain:

$$ $\displaystyle\|{\mathbf{y}}^{t+1}\|$ $\displaystyle=$ $\displaystyle\|{\mathcal{A}}({\mathbf{X}}^{t+1})-\tfrac{1}{\beta^{t}}\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})+\tfrac{1}{\beta^{t}}{\mathbf{z}}^{t}\|$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\|{\mathcal{A}}({\mathbf{X}}^{t+1})\|+\tfrac{1}{\beta^{t}}\|\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})\|+\tfrac{1}{\beta^{t}}\|{\mathbf{z}}^{t}\|$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle R_{A}+\tfrac{1}{\beta^{0}}C_{h}+\tfrac{1}{\beta^{0}}\overline{\rm{z}}\triangleq\overline{\rm{y}},$ $$

where step ① uses the triangle inequality; step ② uses $\|{\mathcal{A}}({\mathbf{X}})\|\leq R_{A}$ for all ${\mathbf{X}}\in\mathcal{M}$, $\tfrac{1}{\beta^{t}}\leq\tfrac{1}{\beta^{0}}$, and $\|{\mathbf{z}}^{t}\|\leq\overline{\rm{z}}$.

∎

###### Proof.

### C.2 Proof of Lemma 3.5

###### Proof.

We define $\mathcal{S}^{t}({\mathbf{X}},{\mathbf{y}}^{t})\triangleq f({\mathbf{X}})+\langle{\mathbf{z}}^{t},{\mathcal{A}}({\mathbf{X}})-{\mathbf{y}}^{t}\rangle+\tfrac{\beta^{t}}{2}\|{\mathcal{A}}({\mathbf{X}})-{\mathbf{y}}^{t}\|_{2}^{2}$.

(a) We have:

$$ $\displaystyle\beta^{t+1}-\beta^{t}\cdot(1+\xi)\overset{\text{\char 172}}{=}\beta^{0}\xi(t+1)^{p}-\beta^{0}\xi t^{p}-\beta^{t}\xi\overset{\text{\char 173}}{\leq}\beta^{0}\xi-\beta^{t}\xi\overset{\text{\char 174}}{\leq}0,$ $$

where step ① uses $\beta^{t}=\beta^{0}(1+\xi t^{p})$; step ② uses $(t+1)^{p}-t^{p}\leq 1$ for all $p\in(0,1)$; step ③ uses $\beta^{0}\leq\beta^{t}$ and $\xi>0$.

(b) For any ${\mathbf{X}},{\mathbf{X}}^{+}\in\mathcal{M}$, we have:

$$ $\displaystyle\|\nabla{\mathcal{A}}({\mathbf{X}})^{\mathsf{T}}{\mathcal{A}}({\mathbf{X}})-\nabla{\mathcal{A}}({\mathbf{X}}^{+})^{\mathsf{T}}{\mathcal{A}}({\mathbf{X}}^{+})\|$ (40) $\displaystyle=$ $\displaystyle\|\nabla{\mathcal{A}}({\mathbf{X}})^{\mathsf{T}}\{{\mathcal{A}}({\mathbf{X}})-{\mathcal{A}}({\mathbf{X}}^{+})\}+\{\nabla{\mathcal{A}}({\mathbf{X}})-\nabla{\mathcal{A}}({\mathbf{X}}^{+})\}^{\mathsf{T}}{\mathcal{A}}({\mathbf{X}}^{+})\|$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\|\nabla{\mathcal{A}}({\mathbf{X}})^{\mathsf{T}}\{{\mathcal{A}}({\mathbf{X}})-{\mathcal{A}}({\mathbf{X}}^{+})\}\|+\|\{\nabla{\mathcal{A}}({\mathbf{X}})-\nabla{\mathcal{A}}({\mathbf{X}}^{+})\}^{\mathsf{T}}{\mathcal{A}}({\mathbf{X}}^{+})\|$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\|\nabla{\mathcal{A}}({\mathbf{X}})\|\cdot\|{\mathcal{A}}({\mathbf{X}})-{\mathcal{A}}({\mathbf{X}}^{+})\|+\|\nabla{\mathcal{A}}({\mathbf{X}})-\nabla{\mathcal{A}}({\mathbf{X}}^{+})\|\cdot\|{\mathcal{A}}({\mathbf{X}}^{+})\|$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle C_{A}C_{A}\|{\mathbf{X}}-{\mathbf{X}}^{+}\|_{\mathsf{F}}+L_{A}R_{A}\|{\mathbf{X}}-{\mathbf{X}}^{+}\|_{\mathsf{F}},$ $$

where step ① uses the triangle inequality; step ② uses the norm inequality that $\|\mathbf{A}\mathbf{B}\|\leq\|\mathbf{A}\|_{2}\cdot\|\mathbf{B}\|_{\mathsf{F}}\leq\|\mathbf{A}\|_{\mathsf{F}}\cdot\|\mathbf{B}\|_{\mathsf{F}}$ for all matrices $\mathbf{A}$ and $\mathbf{B}$; step ③ uses the $C_{A}$-Lipschitz continuity and $L_{A}$-smoothness of ${\mathcal{A}}({\mathbf{X}})$, and the fact that $\|{\mathcal{A}}({\mathbf{X}})\|\leq R_{A}$ for all ${\mathbf{X}}\in\mathcal{M}$.

Furthermore, we have:

$$ $\displaystyle\|\nabla_{{\mathbf{X}}}\mathcal{S}^{t}({\mathbf{X}}^{+},{\mathbf{y}}^{t})-\nabla_{{\mathbf{X}}}\mathcal{S}^{t}({\mathbf{X}},{\mathbf{y}}^{t})\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\|\beta^{t}{\rm{mat}}(\nabla{\mathcal{A}}({\mathbf{X}})^{\mathsf{T}}{\mathcal{A}}({\mathbf{X}})-\nabla{\mathcal{A}}({\mathbf{X}}^{+})^{\mathsf{T}}{\mathcal{A}}({\mathbf{X}}^{+}))+\nabla f({\mathbf{X}})-\nabla f({\mathbf{X}}^{+})$ $\displaystyle+{\rm{mat}}([\nabla{\mathcal{A}}({\mathbf{X}})-\nabla{\mathcal{A}}({\mathbf{X}}^{+})]^{\mathsf{T}}[{\mathbf{z}}^{t}-\beta^{t}{\mathbf{y}}^{t}])\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\|{\mathbf{X}}-{\mathbf{X}}^{+}\|_{\mathsf{F}}\cdot\{\beta^{t}C^{2}_{A}+\beta^{t}L_{A}R_{A}+L_{f}\}+\|[\nabla{\mathcal{A}}({\mathbf{X}})-\nabla{\mathcal{A}}({\mathbf{X}}^{+})]^{\mathsf{T}}[{\mathbf{z}}^{t}-\beta^{t}{\mathbf{y}}^{t}]\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\|{\mathbf{X}}-{\mathbf{X}}^{+}\|_{\mathsf{F}}\cdot\{\underbrace{\beta^{t}C_{A}^{2}+\beta^{t}L_{A}R_{A}+L_{f}+L_{A}(\overline{\rm{z}}+\beta^{t}\overline{\rm{y}})}_{\triangleq\ell(\beta^{t})}\},$ $$

where step ① uses the fact that $\nabla_{{\mathbf{X}}}\mathcal{S}^{t}({\mathbf{X}},{\mathbf{y}}^{t})=\nabla f({\mathbf{X}})+{\rm{mat}}(\nabla{\mathcal{A}}({\mathbf{X}})^{\mathsf{T}}({\mathbf{z}}^{t}-\beta^{t}({\mathcal{A}}({\mathbf{X}})-{\mathbf{y}}^{t})))$; step ② uses the triangle inequality, Inequality ([40](#A3.E40)), and $L_{f}$-smoothness of $f({\mathbf{X}})$; step ③ uses $L_{A}$-smoothness of ${\mathcal{A}}({\mathbf{X}})$, $\|{\mathbf{z}}^{t}\|\leq\overline{\rm{z}}$, and $\|{\mathbf{y}}^{t}\|\leq\overline{\rm{y}}$.

Noticing $\beta^{t}\geq\beta^{0}$ for all $t\geq 0$, we have: $\beta^{t}\underline{\rm{L}}\leq\ell(\beta^{t})\leq\beta^{t}\overline{\rm{L}}$, where $\underline{\rm{L}}\triangleq L_{A}\overline{\rm{y}}+C_{A}^{2}+L_{A}R_{A}$, and $\overline{\rm{L}}\triangleq\underline{\rm{L}}+(L_{f}+L_{A}\overline{\rm{z}})/\beta^{0}$.

∎

###### Proof.

### C.3 Proof of Lemma 3.6

###### Proof.

The proof of this lemma parallels that of Lemma 1 in [], though its results cannot be directly applied in this scenario since $W_{h}\neq 0$.

We consider the following optimization problem:

$$ $\displaystyle\bar{{\mathbf{y}}}=\arg\min_{{\mathbf{y}}}h_{\mu}({\mathbf{y}})+\tfrac{\beta}{2}\|{\mathbf{y}}-\mathbf{b}\|_{2}^{2},$ (41) $$

where $\beta=\beta^{t}$ and $\mu=\mu^{t}$. Given $\beta>\tfrac{W_{h}}{1-W_{h}\mu}$ and $h_{\mu}({\mathbf{y}})$ is $(\tfrac{W_{h}}{1-W_{h}\mu})$-weakly, Problem ([41](#A3.E41)) becomes strongly convex and has a unique optimal solution. Furthermore, Problem ([41](#A3.E41)) can be equivalently rewritten as:

$$ $\displaystyle\textstyle({\mathbf{y}}^{t+1},\breve{{\mathbf{y}}}^{t+1})=\arg\min_{{\mathbf{y}},\breve{{\mathbf{y}}}}h(\breve{{\mathbf{y}}})+\tfrac{1}{2\mu}\|\breve{{\mathbf{y}}}-{\mathbf{y}}\|_{2}^{2}+\tfrac{\beta}{2}\|{\mathbf{y}}-\mathbf{b}\|_{2}^{2},$ $$

We have the following first-order optimality conditions for $({\mathbf{y}}^{t+1},\breve{{\mathbf{y}}}^{t+1})$:

$$ $\displaystyle\tfrac{1}{\mu}({\mathbf{y}}^{t+1}-\breve{{\mathbf{y}}}^{t+1})$ $\displaystyle=$ $\displaystyle\beta(\mathbf{b}-{\mathbf{y}}^{t+1})$ (42) $\displaystyle\tfrac{1}{\mu}({\mathbf{y}}^{t+1}-\breve{{\mathbf{y}}}^{t+1})$ $\displaystyle\in$ $\displaystyle\partial h(\breve{{\mathbf{y}}}^{t+1}).$ (43) $$

(a) We have the following results:

$$ $\displaystyle\mathbf{0}$ $\displaystyle\overset{\text{\char 172}}{\in}$ $\displaystyle\partial h(\breve{{\mathbf{y}}}^{t+1})+\tfrac{1}{\mu}(\breve{{\mathbf{y}}}^{t+1}-{\mathbf{y}}^{t+1})$ (44) $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle\partial h(\breve{{\mathbf{y}}}^{t+1})+\tfrac{1}{\mu}(\breve{{\mathbf{y}}}^{t+1}-\tfrac{1}{1/\mu+\beta}(\tfrac{1}{\mu}\breve{{\mathbf{y}}}^{t+1}+\beta\mathbf{b}))$ $\displaystyle\overset{}{=}$ $\displaystyle\partial h(\breve{{\mathbf{y}}}^{t+1})+\frac{\beta}{1+\mu\beta}(\breve{{\mathbf{y}}}^{t+1}-\mathbf{b}),$ $$

where step ① uses Equality ([43](#A3.E43)); step ② uses Equality ([42](#A3.E42)) that ${\mathbf{y}}^{t+1}=\tfrac{1}{1/\mu+\beta}(\tfrac{1}{\mu}\breve{{\mathbf{y}}}^{t+1}+\beta\mathbf{b})$. The inclusion in ([44](#A3.E44)) implies that:

$$ $\displaystyle\breve{{\mathbf{y}}}^{t+1}=\arg\min_{{\mathbf{v}}}h(\breve{{\mathbf{y}}})+\tfrac{1}{2}\cdot\tfrac{\beta}{1+\mu\beta}\|\breve{{\mathbf{y}}}-\mathbf{b}\|_{2}^{2}.$ $$

(b) Combining Equalities ([42](#A3.E42)) and ([43](#A3.E43)), we have: $\beta(\mathbf{b}-{\mathbf{y}}^{t+1})\in\partial h(\breve{{\mathbf{y}}}^{t+1})$.

(c) In view of Equation ([43](#A3.E43)), we have: ${\mathbf{y}}^{t+1}-\breve{{\mathbf{y}}}^{t+1}=\mu\partial h(\breve{{\mathbf{y}}}^{t+1})$, leading to: $\|\breve{{\mathbf{y}}}^{t+1}-{\mathbf{y}}^{t+1}\|\leq\mu C_{h}$.

∎

###### Proof.

## Appendix D Proofs for Section 4

### D.1 Proof of Lemma 4.1

###### Proof.

We define $L({\mathbf{X}},{\mathbf{y}},{\mathbf{z}};\beta,\mu)\triangleq f({\mathbf{X}})-g({\mathbf{X}})+h_{\mu}({\mathbf{y}})+\langle{\mathbf{z}},{\mathcal{A}}({\mathbf{X}})-{\mathbf{y}}\rangle+\frac{\beta}{2}\|{\mathcal{A}}({\mathbf{X}})-{\mathbf{y}}\|_{2}^{2}$.

(a) Using the first-order optimality condition of ${\mathbf{y}}^{t+1}\in\arg\min_{{\mathbf{y}}}L({\mathbf{X}}^{t+1},{\mathbf{y}},{\mathbf{z}}^{t};\beta^{t},\mu^{t})$ in Algorithm [1](#alg1), for all $t\geq 0$, we have:

$$ $\displaystyle\mathbf{0}$ $\displaystyle=$ $\displaystyle\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})+\nabla q({\mathbf{y}}_{{\sf c}}^{t})+\beta^{t}({\mathbf{y}}^{t+1}-{\mathbf{y}}_{{\sf c}}^{t})$ (45) $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})-{\mathbf{z}}^{t}+\beta^{t}({\mathbf{y}}_{{\sf c}}^{t}-{\mathcal{A}}({\mathbf{X}}^{t+1}))+\beta^{t}({\mathbf{y}}^{t+1}-{\mathbf{y}}_{{\sf c}}^{t})$ $\displaystyle=$ $\displaystyle\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})-{\mathbf{z}}^{t}+\beta^{t}({\mathbf{y}}^{t+1}-{\mathcal{A}}({\mathbf{X}}^{t+1}))$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})-{\mathbf{z}}^{t}+\tfrac{1}{\sigma}({\mathbf{z}}^{t}-{\mathbf{z}}^{t+1}),$ $$

where step ① uses $\nabla q({\mathbf{y}})=-{\mathbf{z}}^{t}+\beta^{t}({\mathbf{y}}-{\mathcal{A}}({\mathbf{X}}^{t+1}))$; step ② uses ${\mathbf{z}}^{t+1}={\mathbf{z}}^{t}+\sigma\beta^{t}({\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1})$. Since $t$ can be arbitrarily chosen, we have:

$$ $\displaystyle\forall t\geq 1,~{}\mathbf{0}\in\nabla h_{\mu^{t-1}}({\mathbf{y}}^{t})-{\mathbf{z}}^{t-1}+\tfrac{1}{\sigma}({\mathbf{z}}^{t-1}-{\mathbf{z}}^{t}).$ (46) $$

(b) Combining Inequalities ([45](#A4.E45)) and ([46](#A4.E46)), for all $t\geq 1$, we have:

$$ $\displaystyle(1-\tfrac{1}{\sigma})({\mathbf{z}}^{t}-{\mathbf{z}}^{t-1})+\tfrac{1}{\sigma}({\mathbf{z}}^{t+1}-{\mathbf{z}}^{t})=\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})-\nabla h_{\mu^{t-1}}({\mathbf{y}}^{t}).$ $$

We further obtain:

$$ $\displaystyle{\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}=(1-\sigma)({\mathbf{z}}^{t}-{\mathbf{z}}^{t-1})+\sigma(\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})-\nabla h_{\mu^{t-1}}({\mathbf{y}}^{t}).$ $$

Given $\sigma\in[1,2)$, we have:

$$ $\displaystyle{\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}=(\sigma-1)\cdot(-{\mathbf{z}}^{t}+{\mathbf{z}}^{t-1})+(1-(\sigma-1))\cdot\tfrac{\sigma}{2-\sigma}(\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})-\nabla h_{\mu^{t-1}}({\mathbf{y}}^{t})).$ $$

Since $\|\cdot\|$ is convex, we derive the following inequalities:

$$ $\displaystyle\|{\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}\|_{2}^{2}\leq(\sigma-1)\|{\mathbf{z}}^{t}-{\mathbf{z}}^{t-1}\|_{2}^{2}+(1-(\sigma-1))\|\tfrac{\sigma}{2-\sigma}(\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})-\nabla h_{\mu^{t-1}}({\mathbf{y}}^{t})\|_{2}^{2}.$ $$

Adding both sides by $(1-\sigma)\|{\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}\|_{2}^{2}$, we have:

$$ $\displaystyle(2-\sigma)\|{\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}\|_{2}^{2}\leq(\sigma-1)(\|{\mathbf{z}}^{t}-{\mathbf{z}}^{t-1}\|_{2}^{2}-\|{\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}\|_{2}^{2})+\tfrac{\sigma^{2}}{2-\sigma}\|\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})-\nabla h_{\mu^{t-1}}({\mathbf{y}}^{t})\|_{2}^{2}.$ $$

Dividing both sides by $(2-\sigma)$, we have:

$$ $\displaystyle\|{\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}\|_{2}^{2}\leq\tfrac{\sigma-1}{2-\sigma}(\|{\mathbf{z}}^{t}-{\mathbf{z}}^{t-1}\|_{2}^{2}-\|{\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}\|_{2}^{2})+\tfrac{\sigma^{2}}{(2-\sigma)^{2}}\|\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})-\nabla h_{\mu^{t-1}}({\mathbf{y}}^{t})\|_{2}^{2}.$ (47) $$

(c) We have from Inequality ([47](#A4.E47)):

$$ $\displaystyle\|{\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}\|_{2}^{2}-\tfrac{\sigma-1}{2-\sigma}(\|{\mathbf{z}}^{t}-{\mathbf{z}}^{t-1}\|_{2}^{2}-\|{\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}\|_{2}^{2})$ (48) $\displaystyle\leq$ $\displaystyle\tfrac{\sigma^{2}}{(2-\sigma)^{2}}\|\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})-\nabla h_{\mu^{t-1}}({\mathbf{y}}^{t})\|_{2}^{2}$ $\displaystyle\overset{}{=}$ $\displaystyle\tfrac{\sigma^{2}}{(2-\sigma)^{2}}\|\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})-\nabla h_{\mu^{t}}({\mathbf{y}}^{t})+\nabla h_{\mu^{t}}({\mathbf{y}}^{t})-\nabla h_{\mu^{t-1}}({\mathbf{y}}^{t})\|_{2}^{2}$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\tfrac{\sigma^{2}}{(2-\sigma)^{2}}\|\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})-\nabla h_{\mu^{t}}({\mathbf{y}}^{t})\|_{2}^{2}+\tfrac{2\sigma}{2-\sigma}\|\nabla h_{\mu^{t}}({\mathbf{y}}^{t})-\nabla h_{\mu^{t-1}}({\mathbf{y}}^{t})\|_{2}^{2}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\tfrac{2\sigma^{2}}{(2-\sigma)^{2}}\cdot\tfrac{1}{(\mu^{t})^{2}}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}+\tfrac{2\sigma^{2}}{(2-\sigma)^{2}}\cdot C_{h}^{2}\cdot(\tfrac{\mu^{t-1}}{\mu^{t}}-1)^{2}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\tfrac{2\sigma^{2}}{(2-\sigma)^{2}}\cdot\tfrac{(\beta^{t})^{2}}{\chi^{2}}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}+\tfrac{2\sigma^{2}}{(2-\sigma)^{2}}\cdot C_{h}^{2}\cdot(\tfrac{2}{t}-\tfrac{2}{t+1})$ $\displaystyle\overset{\text{\char 175}}{=}$ $\displaystyle c_{1}(\beta^{t})^{2}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}+(\Gamma_{\mu}^{t}-\Gamma_{\mu}^{t+1})\cdot\tfrac{\beta^{0}}{c_{0}},$ $$

where step ① uses $\|\mathbf{A}+\mathbf{B}\|_{\mathsf{F}}^{2}\leq 2\|\mathbf{A}\|_{\mathsf{F}}^{2}+2\|\mathbf{B}\|_{\mathsf{F}}^{2}$; step ② uses the fact that the function $h_{\mu^{t}}({\mathbf{y}})$ is $\tfrac{1}{\mu^{t}}$-smooth w.r.t. ${\mathbf{y}}$ that:

$$ $\displaystyle\|\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})-\nabla h_{\mu^{t}}({\mathbf{y}}^{t})\|\leq\tfrac{1}{\mu^{t}}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|,$ $$

and the fact that

$$ $\displaystyle\|\nabla h_{\mu^{t}}({\mathbf{y}}^{t})-\nabla h_{\mu^{t-1}}({\mathbf{y}}^{t})\|\leq(\tfrac{\mu^{t-1}}{\mu^{t}}-1)C_{h}$ $$

which is due to Lemma [2.6](#S2.Thmtheorem6); step ③ uses Lemma [A.3](#A1.Thmtheorem3); step ④ uses the definitions of $\{c_{1},\Gamma_{\mu^{t}}\}$.

We obtain from Inequality ([48](#A4.E48)) for all $t\geq 1$:

$$ $\displaystyle\tfrac{c_{0}}{\beta^{t}}\|{\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}\|_{2}^{2}$ $\displaystyle\leq$ $\displaystyle\tfrac{c_{0}}{\beta^{t}}\{\tfrac{\sigma-1}{\sigma}(\|{\mathbf{z}}^{t}-{\mathbf{z}}^{t-1}\|_{2}^{2}-\|{\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}\|_{2}^{2})+c_{1}(\beta^{t})^{2}\|{\mathbf{y}}^{t}-{\mathbf{y}}^{t-1}\|_{2}^{2}+(\Gamma_{\mu}^{t}-\Gamma_{\mu}^{t+1})\cdot\tfrac{\beta^{0}}{c_{0}}\}$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\tfrac{c_{0}}{\beta^{t}}\cdot\tfrac{\sigma-1}{\sigma}\|{\mathbf{z}}^{t}-{\mathbf{z}}^{t-1}\|_{2}^{2}-\tfrac{c_{0}}{\beta^{t+1}}\cdot\tfrac{\sigma-1}{\sigma}\|{\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}\|_{2}^{2}+c_{1}(\beta^{t})^{2}\|{\mathbf{y}}^{t}-{\mathbf{y}}^{t-1}\|_{2}^{2}+\Gamma_{\mu}^{t}-\Gamma_{\mu}^{t+1}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\Gamma^{t}_{z}-\Gamma^{t+1}_{z}+c_{0}c_{1}\beta^{t}\|{\mathbf{y}}^{t}-{\mathbf{y}}^{t-1}\|_{2}^{2}+\Gamma_{\mu}^{t}-\Gamma_{\mu}^{t+1},$ $$

where step ① uses $\beta^{t+1}\geq\beta^{t}\geq\beta^{0}$, and $\Gamma_{\mu}^{t}\geq\Gamma_{\mu}^{t+1}\geq 0$; step ② uses the definition of $\Gamma^{t}_{z}$.

(d) We let $\breve{\mathbf{g}}\in\partial h(\breve{{\mathbf{y}}}^{t+1})$ and obtain:

$$ $\displaystyle\|\breve{\mathbf{g}}-{\mathbf{z}}^{t+1}\|$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\|\beta^{t}(\mathbf{b}-{\mathbf{y}}^{t+1})-{\mathbf{z}}^{t+1}\|$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle\|\beta^{t}{\mathbf{y}}_{{\sf c}}^{t}-\nabla_{{\mathbf{y}}}\mathcal{S}^{t}({\mathbf{X}}^{t+1},{\mathbf{y}}_{{\sf c}}^{t})-\beta^{t}{\mathbf{y}}^{t+1}-{\mathbf{z}}^{t+1}\|$ $\displaystyle\overset{\text{\char 174}}{=}$ $\displaystyle\|\beta^{t}{\mathbf{y}}_{{\sf c}}^{t}+{\mathbf{z}}^{t}-\beta^{t}({\mathbf{y}}_{{\sf c}}^{t}-{\mathcal{A}}({\mathbf{X}}^{t+1}))-\beta^{t}{\mathbf{y}}^{t+1}-{\mathbf{z}}^{t+1}\|$ $\displaystyle\overset{}{=}$ $\displaystyle\|{\mathbf{z}}^{t}-{\mathbf{z}}^{t+1}+\beta^{t}({\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1})\|$ $\displaystyle\overset{\text{\char 175}}{=}$ $\displaystyle(\sigma-1)\beta^{t}\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|,$ $$

where step ① uses the result in Lemma [3.6](#S3.Thmtheorem6) that $\beta^{t}(\mathbf{b}-{\mathbf{y}}^{t+1})\in\partial h(\breve{{\mathbf{y}}}^{t+1})$; step ② uses $\mathbf{b}\triangleq{\mathbf{y}}_{{\sf c}}^{t}-\nabla_{{\mathbf{y}}}\mathcal{S}^{t}({\mathbf{X}}^{t+1},{\mathbf{y}}_{{\sf c}}^{t})/\beta^{t}$, as shown in Algorithm [1](#alg1); step ③ uses $\nabla_{{\mathbf{y}}}\mathcal{S}^{t}({\mathbf{X}}^{t+1},{\mathbf{y}})=-{\mathbf{z}}^{t}+\beta^{t}({\mathbf{y}}-{\mathcal{A}}({\mathbf{X}}^{t+1}))$; step ④ uses ${\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}=\sigma\beta^{t}({\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1})$.

∎

###### Proof.

### D.2 Proof of Lemma 4.2

We assume $\mu\in(0,\frac{W_{h}}{2}]$. We have:

$$ $\displaystyle\Theta^{t}$ $\displaystyle=$ $\displaystyle L({\mathbf{X}}^{t},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\beta^{t},\mu^{t})+\tfrac{c_{0}}{\beta^{t}}\tfrac{1-\sigma}{\sigma}\|{\mathbf{z}}^{t}-{\mathbf{z}}^{t-1}\|_{2}^{2}+\mu^{t}C_{h}^{2}+\Gamma_{{\sf x}}^{t}$ $\displaystyle\geq$ $\displaystyle f({\mathbf{X}}^{t})-g({\mathbf{X}}^{t})+h_{\mu^{t}}({\mathbf{y}}^{t})+\langle{\mathbf{z}}^{t},{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\rangle+\tfrac{\beta^{t}}{2}\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|_{2}^{2}+0+0+0$ $\displaystyle=$ $\displaystyle f({\mathbf{X}}^{t})-g({\mathbf{X}}^{t})+h_{\mu^{t}}({\mathbf{y}}^{t})+\tfrac{\beta^{t}}{2}\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}+{\mathbf{z}}^{t}/\beta^{t}\|_{2}^{2}-\tfrac{\beta^{t}}{2}\|{\mathbf{z}}^{t}/\beta^{t}\|_{2}^{2}$ $\displaystyle\overset{\text{\char 172}}{\geq}$ $\displaystyle f({\mathbf{X}}^{t})-g({\mathbf{X}}^{t})+h_{\mu^{t}}({\mathcal{A}}({\mathbf{X}}^{t}))-C_{h}\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|-\tfrac{1}{2\beta^{t}}\|{\mathbf{z}}^{t}\|_{2}^{2}$ $\displaystyle\overset{\text{\char 173}}{\geq}$ $\displaystyle f({\mathbf{X}}^{t})-g({\mathbf{X}}^{t})+h({\mathcal{A}}({\mathbf{X}}^{t}))-\mu^{t}C_{h}^{2}-C_{h}\|{\mathcal{A}}({\mathbf{X}}^{t})\|-C_{h}\|{\mathbf{y}}^{t}\|-\tfrac{1}{2\beta^{t}}\|{\mathbf{z}}^{t}\|_{2}^{2}$ $\displaystyle\overset{\text{\char 174}}{\geq}$ $\displaystyle F(\bar{{\mathbf{X}}})-\mu^{0}C_{h}^{2}-C_{h}(R_{A}+\overline{\rm{y}})-\tfrac{\overline{\rm{z}}^{2}}{2\beta^{0}}\triangleq\underline{\rm{\Theta}},$ $$

where step ① uses the $L_{h}$-Lipschitz continuity of $h_{\mu^{t}}({\mathbf{y}})$, ensuring $h_{\mu^{t}}({\mathbf{y}}^{t})\geq h_{\mu^{t}}({\mathbf{y}})-C_{h}\|{\mathbf{y}}^{t}-{\mathbf{y}}\|_{2}^{2}$, with the specific choice of ${\mathbf{y}}={\mathcal{A}}({\mathbf{X}}^{t})$; step ② uses $h({\mathbf{y}})-h_{\mu}({\mathbf{y}})\leq\mu C_{h}^{2}$, which has been shown in Lemma [2.5](#S2.Thmtheorem5); step ③ uses $\mu^{t}\leq\mu^{0}$, $\beta^{t}\geq\beta^{0}$, $\|{\mathcal{A}}({\mathbf{X}})\|\leq R_{A}$ for all ${\mathbf{X}}\in\mathcal{M}$; $\|{\mathbf{y}}\|\leq\overline{\rm{y}}$, and $\|{\mathbf{z}}^{t}\|\leq\overline{\rm{z}}$.

### D.3 Proof of Lemma 4.3

###### Proof.

We define $L({\mathbf{X}},{\mathbf{y}},{\mathbf{z}};\beta,\mu)=f({\mathbf{X}})-g({\mathbf{X}})+h_{\mu}({\mathbf{y}})+\langle{\mathbf{z}},{\mathcal{A}}({\mathbf{X}})-{\mathbf{y}}\rangle+\frac{\beta}{2}\|{\mathcal{A}}({\mathbf{X}})-{\mathbf{y}}\|_{2}^{2}$.

(a) We focus on sufficient decrease for variable ${\mathbf{y}}$. Given the function $h_{\mu^{t}}({\mathbf{y}})$ is $W^{\star}_{h}$-weakly convex w.r.t ${\mathbf{y}}$ with $W^{\star}_{h}=\tfrac{W_{h}}{1-W_{h}\mu^{0}}$, we have:

$$ $\displaystyle h_{\mu^{t}}({\mathbf{y}}^{t+1})-h_{\mu^{t}}({\mathbf{y}}^{t})$ (49) $\displaystyle\leq$ $\displaystyle\langle{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t},\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})\rangle+\tfrac{W^{\star}_{h}}{2}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\langle{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t},{\mathbf{z}}^{t}+\tfrac{1}{\sigma}({\mathbf{z}}^{t+1}-{\mathbf{z}}^{t})\rangle+\tfrac{W^{\star}_{h}}{2}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle\langle{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t},{\mathbf{z}}^{t}+\beta^{t}({\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1})\rangle+\tfrac{W^{\star}_{h}}{2}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}$ $$

where step ① uses the first-order optimality condition of ${\mathbf{y}}^{t+1}$ as used in Lemma [4.3](#S4.Thmtheorem3) (Refer to Equation [45](#A4.E45)) that: $\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})={\mathbf{z}}^{t}-\tfrac{1}{\sigma}({\mathbf{z}}^{t}-{\mathbf{z}}^{t+1})$; step ② uses ${\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}=\sigma\beta^{t}({\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1})$.

We derive the following inequalities:

$$ $\displaystyle L({\mathbf{X}}^{t+1},{\mathbf{y}}^{t+1},{\mathbf{z}}^{t};\beta^{t},\mu^{t})-L({\mathbf{X}}^{t+1},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\beta^{t},\mu^{t})$ (50) $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle h_{\mu^{t}}({\mathbf{y}}^{t+1})-h_{\mu^{t}}({\mathbf{y}}^{t})+\langle{\mathbf{y}}^{t}-{\mathbf{y}}^{t+1},{\mathbf{z}}^{t}\rangle+\tfrac{\beta^{t}}{2}\|{\mathbf{y}}^{t+1}-{\mathcal{A}}({\mathbf{X}}^{t+1})\|_{2}^{2}-\tfrac{\beta^{t}}{2}\|{\mathbf{y}}^{t}-{\mathcal{A}}({\mathbf{X}}^{t+1})\|_{2}^{2}$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle h_{\mu^{t}}({\mathbf{y}}^{t+1})-h_{\mu^{t}}({\mathbf{y}}^{t})+\langle{\mathbf{y}}^{t}-{\mathbf{y}}^{t+1},{\mathbf{z}}^{t}+\beta^{t}({\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t})\rangle+\tfrac{\beta^{t}}{2}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\langle{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t},\beta^{t}({\mathbf{y}}^{t}-{\mathbf{y}}^{t+1})\rangle+\tfrac{W^{\star}_{h}}{2}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}+\tfrac{\beta^{t}}{2}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}$ $\displaystyle\overset{}{=}$ $\displaystyle\tfrac{W^{\star}_{h}-\beta^{t}}{2}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}$ $$

where step ① uses the definition of $L({\mathbf{X}},{\mathbf{y}},{\mathbf{z}};\beta,\mu)$; step ② uses the Pythagoras Relation that $\tfrac{1}{2}\|{\mathbf{y}}^{+}-{\mathbf{s}}\|_{2}^{2}-\tfrac{1}{2}\|{\mathbf{y}}-{\mathbf{s}}\|_{2}^{2}=\langle{\mathbf{y}}-{\mathbf{y}}^{+},{\mathbf{s}}\rangle+\tfrac{1}{2}\|{\mathbf{y}}^{+}-{\mathbf{y}}\|_{2}^{2}$ for all ${\mathbf{y}}\in\mathbb{R}^{m\times 1}$, ${\mathbf{y}}^{+}\in\mathbb{R}^{m\times 1}$, and ${\mathbf{s}}\in\mathbb{R}^{m\times 1}$; step ③ uses Inequality ([49](#A4.E49))

(b) We focus on sufficient decrease for variable ${\mathbf{z}}$. We have:

$$ $\displaystyle L({\mathbf{X}}^{t+1},{\mathbf{y}}^{t+1},{\mathbf{z}}^{t+1};\beta^{t},\mu^{t})-L({\mathbf{X}}^{t+1},{\mathbf{y}}^{t+1},{\mathbf{z}}^{t};\beta^{t},\mu^{t})$ (51) $\displaystyle=$ $\displaystyle\langle{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1},{\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}\rangle$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\tfrac{1}{\sigma\beta^{t}}\|{\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}\|_{2}^{2},$ $$

where step ① uses ${\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}=\sigma\beta^{t}({\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1})$.

(c) We focus on sufficient decrease for variable $\beta$. We have:

$$ $\displaystyle L({\mathbf{X}}^{t+1},{\mathbf{y}}^{t+1},{\mathbf{z}}^{t+1};\beta^{t+1},\mu^{t})-L({\mathbf{X}}^{t+1},{\mathbf{y}}^{t+1},{\mathbf{z}}^{t+1};\beta^{t},\mu^{t})$ (52) $\displaystyle=$ $\displaystyle\tfrac{\beta^{t+1}-\beta^{t}}{2}\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|_{2}^{2}$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\tfrac{\beta^{t+1}-\beta^{t}}{2\sigma^{2}\beta^{t}}\cdot\tfrac{1}{\beta^{t}}\|{\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}\|_{2}^{2}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\tfrac{\xi}{2\sigma^{2}}\cdot\tfrac{1}{\beta^{t}}\|{\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}\|_{2}^{2},$ $$

where step ① uses ${\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}=\sigma\beta^{t}({\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1})$; step ② uses Assumption LABEL:ass:beta:t1 that $\beta^{t+1}\leq\beta^{t}(1+\xi)$ for all $t$.

(d) We focus on sufficient decrease for variable $\mu$. We have:

$$ $\displaystyle L({\mathbf{X}}^{t+1},{\mathbf{y}}^{t+1},{\mathbf{z}}^{t+1};\beta^{t+1},\mu^{t+1})-L({\mathbf{X}}^{t+1},{\mathbf{y}}^{t+1},{\mathbf{z}}^{t+1};\beta^{t+1},\mu^{t})$ (53) $\displaystyle=$ $\displaystyle h_{\mu^{t+1}}({\mathbf{y}}^{t+1})-h_{\mu^{t}}({\mathbf{y}}^{t+1})$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\min\{\tfrac{\mu^{t}}{2\mu^{t+1}},1\}\cdot(\mu^{t}-\mu^{t+1})C_{h}^{2}$ $\displaystyle\overset{}{\leq}$ $\displaystyle(\mu^{t}-\mu^{t+1})C_{h}^{2},$ $$

where step ① uses Lemma [2.5](#S2.Thmtheorem5) under the condition that $0<\mu^{t+1}<\mu^{t}<\frac{1}{W_{h}}$.

Adding Inequalities ([50](#A4.E50)), ([51](#A4.E51)), ([52](#A4.E52)), and ([53](#A4.E53)) together, we have:

$$ $\displaystyle\textstyle L({\mathbf{X}}^{t+1},{\mathbf{y}}^{t+1},{\mathbf{z}}^{t+1};\beta^{t+1},\mu^{t+1})-L({\mathbf{X}}^{t+1},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\beta^{t},\mu^{t})$ $\displaystyle\textstyle-(\mu^{t}-\mu^{t+1})C_{h}^{2}+\varepsilon_{z}\beta^{t}\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|_{2}^{2}$ $\displaystyle\leq$ $\displaystyle\tfrac{W^{\star}_{h}-\beta^{t}}{2}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}+(1+\tfrac{\xi}{2\sigma^{2}})\cdot\tfrac{1}{\beta^{t}}\|{\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}\|_{2}^{2}+\varepsilon_{z}\beta^{t}\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|_{2}^{2}$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\tfrac{W^{\star}_{h}-\beta^{t}}{2}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}+\underbrace{(1+\tfrac{\varepsilon_{z}}{\sigma^{2}}+\tfrac{\xi}{2\sigma^{2}})}_{\triangleq c_{0}}\cdot\tfrac{1}{\beta^{t}}\|{\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}\|_{2}^{2}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\tfrac{W^{\star}_{h}/\beta^{t}-1}{2}\cdot\beta^{t}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}+c_{0}c_{1}\beta^{t}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}+\Gamma_{\mu}^{t}-\Gamma_{\mu}^{t+1}+\Gamma_{{\sf z}}^{t}-\Gamma_{{\sf z}}^{t+1},$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle-\underbrace{(\tfrac{1}{2}-\tfrac{W^{\star}_{h}}{2\beta^{0}}-c_{0}c_{1})}_{\triangleq\varepsilon_{y}}\beta^{t}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}+\Gamma_{\mu}^{t}-\Gamma_{\mu}^{t+1}+\Gamma_{{\sf z}}^{t}-\Gamma_{{\sf z}}^{t+1},$ $$

where step ① uses $\varepsilon_{z}\beta^{t}\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|_{2}^{2}=\tfrac{\varepsilon_{z}}{\sigma^{2}\beta^{t}}\|{\mathbf{z}}^{t+1}-{\mathbf{z}}^{t}\|_{2}^{2}$; step ② uses Claim ((b)) of Lemma [4.1](#S4.Thmtheorem1); step ③ uses $\tfrac{1}{\beta^{t}}\leq\tfrac{1}{\beta^{0}}$.

∎

###### Proof.

### D.4 Proof of Lemma 4.4

###### Proof.

We define $\mathcal{S}^{t}({\mathbf{X}},{\mathbf{y}})\triangleq f({\mathbf{X}})+\langle{\mathbf{z}}^{t},{\mathcal{A}}({\mathbf{X}})-{\mathbf{y}}\rangle+\tfrac{\beta^{t}}{2}\|{\mathcal{A}}({\mathbf{X}})-{\mathbf{y}}\|_{2}^{2}$.

We let ${\mathbf{G}}^{t}\in\nabla_{{\mathbf{X}}}\mathcal{S}^{t}({\mathbf{X}}_{{\sf c}}^{t},{\mathbf{y}}^{t})-\partial g({\mathbf{X}}^{t})$.

We define $\Gamma_{{\sf x}}^{t}\triangleq\tfrac{(\alpha_{x}+\theta\alpha_{x})\ell(\beta^{t})}{2}\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}^{2}$.

We define $\varepsilon_{x}^{\prime}\triangleq(\theta-1-\alpha_{x}-\theta\alpha_{x})-(1+\xi)(\alpha_{x}+\theta\alpha_{x})$.

First, we notice that ${\mathbf{X}}^{t}\in\mathcal{M}$ and ${\mathbf{X}}^{t+1}\in\mathcal{M}$. Using the optimality condition of ${\mathbf{X}}^{t+1}$, we have:

$$ $\displaystyle\langle{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t},{\mathbf{G}}^{t}\rangle+\tfrac{\theta\ell(\beta^{t})}{2}\|{\mathbf{X}}^{t+1}-{\mathbf{X}}_{{\sf c}}^{t}\|_{\mathsf{F}}^{2}\leq\langle{\mathbf{X}}^{t}-{\mathbf{X}}^{t},{\mathbf{G}}^{t}\rangle+\tfrac{\theta\ell(\beta^{t})}{2}\|{\mathbf{X}}^{t}-{\mathbf{X}}_{{\sf c}}^{t}\|_{\mathsf{F}}^{2}.$ (54) $$

Second, since the function $\mathcal{S}^{t}({\mathbf{X}},{\mathbf{y}}^{t})$ is $\ell(\beta^{t})$-smooth w.r.t. ${\mathbf{X}}$, we have:

$$ $\displaystyle\mathcal{S}^{t}({\mathbf{X}}^{t+1},{\mathbf{y}}^{t})-\mathcal{S}^{t}({\mathbf{X}}^{t},{\mathbf{y}}^{t})\leq\langle{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t},\nabla_{{\mathbf{X}}}\mathcal{S}^{t}({\mathbf{X}}^{t},{\mathbf{y}}^{t})\rangle+\tfrac{\ell(\beta^{t})}{2}\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}^{2}$ (55) $$

Third, given the function $g({\mathbf{X}})$ is convex, we have:

$$ $\displaystyle g({\mathbf{X}}^{t})-g({\mathbf{X}}^{t+1})\leq\langle{\mathbf{X}}^{t}-{\mathbf{X}}^{t+1},\partial g({\mathbf{X}}^{t})\rangle$ (56) $$

Fourth, we derive:

$$ $\displaystyle\langle{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t},\nabla\mathcal{S}^{t}({\mathbf{X}}^{t},{\mathbf{y}}^{t})-\nabla\mathcal{S}^{t}({\mathbf{X}}_{{\sf c}}^{t},{\mathbf{y}}^{t})\rangle$ (57) $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}\cdot\|\nabla\mathcal{S}^{t}({\mathbf{X}}^{t},{\mathbf{y}}^{t})-\nabla\mathcal{S}^{t}({\mathbf{X}}_{{\sf c}}^{t},{\mathbf{y}}^{t})\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}\cdot\ell(\beta^{t})\|{\mathbf{X}}^{t}-{\mathbf{X}}_{{\sf c}}^{t}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\alpha_{x}\ell(\beta^{t})\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}\cdot\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 175}}{\leq}$ $\displaystyle\tfrac{\alpha_{x}\ell(\beta^{t})}{2}\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}^{2}+\tfrac{\alpha_{x}\ell(\beta^{t})}{2}\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|^{2}_{\mathsf{F}},$ $$

where step ① uses the norm inequality; step ② uses the $\ell(\beta^{t})$-smoothness of $\mathcal{S}^{t}({\mathbf{X}},{\mathbf{y}}^{t})$; step ③ uses ${\mathbf{X}}_{{\sf c}}^{t}={\mathbf{X}}^{t}+\alpha_{x}({\mathbf{X}}^{t}-{\mathbf{X}}^{t-1})$; step ④ uses $ab\leq\frac{1}{2}a^{2}+\frac{1}{2}b^{2}$ for all $a\in\mathbb{R}$ and $b\in\mathbb{R}$.

Fifth, we have:

$$ $\displaystyle L({\mathbf{X}}^{t+1},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\mu^{t},\beta^{t})-L({\mathbf{X}}^{t},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\mu^{t},\beta^{t})$ (58) $\displaystyle=$ $\displaystyle\textstyle\mathcal{S}^{t}({\mathbf{X}}^{t+1},{\mathbf{y}}^{t})-\mathcal{S}^{t}({\mathbf{X}}^{t},{\mathbf{y}}^{t})+g({\mathbf{X}}^{t})-g({\mathbf{X}}^{t+1}).$ $$

By summing the five inequalities specified in ([54](#A4.E54)), ([55](#A4.E55)), ([56](#A4.E56)), ([57](#A4.E57)), and ([58](#A4.E58)), we obtain:

$$ $\displaystyle L({\mathbf{X}}^{t+1},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\mu^{t},\beta^{t})-L({\mathbf{X}}^{t},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\mu^{t},\beta^{t})$ $\displaystyle\leq$ $\displaystyle\textstyle\tfrac{1}{2}\ell(\beta^{t})\cdot\{(1+\alpha_{x})\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|^{2}_{\mathsf{F}}+\alpha_{x}\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}+\theta\|{\mathbf{X}}^{t}-{\mathbf{X}}_{{\sf c}}^{t}\|_{\mathsf{F}}^{2}-\theta\|{\mathbf{X}}^{t+1}-{\mathbf{X}}_{{\sf c}}^{t}\|_{\mathsf{F}}^{2}\}$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\textstyle\tfrac{\ell(\beta^{t})}{2}\cdot\{(1+\alpha_{x})\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|^{2}_{\mathsf{F}}+(\alpha_{x}+\theta\alpha_{x}^{2})\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}^{2}-\theta\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}-\alpha_{x}({\mathbf{X}}^{t}-{\mathbf{X}}^{t-1})\|_{\mathsf{F}}^{2}\}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\textstyle\tfrac{\ell(\beta^{t})}{2}\cdot\{(1+\alpha_{x})\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|^{2}_{\mathsf{F}}+(\alpha_{x}+\theta\alpha_{x}^{2})\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}^{2}$ $\displaystyle+\theta(\alpha_{x}-1)\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}^{2}-\theta\alpha_{x}(\alpha_{x}-1)\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}^{2}\}$ $\displaystyle\overset{\text{\char 174}}{=}$ $\displaystyle\textstyle\Gamma_{x}^{t}+\tfrac{\ell(\beta^{t})}{2}\cdot\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|^{2}_{\mathsf{F}}\cdot\{1+\alpha_{x}+\theta\alpha_{x}-\theta\}$ $\displaystyle\overset{}{=}$ $\displaystyle\textstyle\Gamma_{x}^{t}-\Gamma_{x}^{t+1}+\tfrac{1}{2}\cdot\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|^{2}_{\mathsf{F}}\cdot\{\ell(\beta^{t})(1+\alpha_{x}+\theta\alpha_{x}-\theta)+\ell(\beta^{t+1})(\alpha_{x}+\theta\alpha_{x})\}$ $\displaystyle\overset{\text{\char 175}}{\leq}$ $\displaystyle\textstyle\Gamma_{x}^{t}-\Gamma_{x}^{t+1}+\tfrac{1}{2}\cdot\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|^{2}_{\mathsf{F}}\cdot\{\ell(\beta^{t})(1+\alpha_{x}+\theta\alpha_{x}-\theta)+\ell(\beta^{t})(1+\xi)(\alpha_{x}+\theta\alpha_{x})\}$ $\displaystyle\overset{\text{\char 176}}{=}$ $\displaystyle\textstyle\Gamma_{x}^{t}-\Gamma_{x}^{t+1}-\tfrac{\ell(\beta^{t})}{2}\cdot\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|^{2}_{\mathsf{F}}\cdot\varepsilon_{x}^{\prime}$ $\displaystyle\overset{\text{\char 177}}{\leq}$ $\displaystyle\textstyle\Gamma_{x}^{t}-\Gamma_{x}^{t+1}-\tfrac{1}{2}\cdot\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|^{2}_{\mathsf{F}}\cdot\varepsilon_{x}^{\prime}\cdot\beta^{t}\underline{\rm{L}}$ $\displaystyle\overset{\text{\char 178}}{=}$ $\displaystyle\textstyle\Gamma_{x}^{t}-\Gamma_{x}^{t+1}-\varepsilon_{x}\cdot\beta^{t}\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|^{2}_{\mathsf{F}},$ $$

where step ① uses ${\mathbf{X}}_{{\sf c}}^{t}={\mathbf{X}}^{t}+\alpha_{x}({\mathbf{X}}^{t}-{\mathbf{X}}^{t-1})$; step ② uses Lemma [A.1](#A1.Thmtheorem1) with $\mathbf{a}={\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}$, $\mathbf{b}={\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}$, and $\alpha=\alpha_{x}$; step ③ uses the definition of $\Gamma_{x}^{t}$; step ④ uses the fact that $\ell(\beta^{t+1})\leq(1+\xi)\ell(\beta^{t})$, which is implied by $\beta^{t+1}\leq(1+\xi)\beta^{t}$; step ⑤ uses the definition of $\varepsilon_{x}^{\prime}$ with $\varepsilon_{x}^{\prime}>0$; step ⑥ uses Lemma [3.5](#S3.Thmtheorem5) that $\beta^{t}\underline{\rm{L}}\leq\ell(\beta^{t})\leq\beta^{t}\overline{\rm{L}}$; step ⑦ uses $\varepsilon_{x}\triangleq\tfrac{1}{2}\varepsilon_{x}^{\prime}\underline{\rm{L}}>0$.

∎

###### Proof.

### D.5 Proof of Lemma 4.5

###### Proof.

We let $t\geq 1$.

We define: $\Theta^{t}\triangleq L({\mathbf{X}}^{t},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\beta^{t},\mu^{t})+\Gamma^{t}_{z}+\mu^{t}C_{h}^{2}+\Gamma_{{\sf x}}^{t}+\Gamma_{\mu}^{t}$,

(a) Using Lemma [4.3](#S4.Thmtheorem3), we have:

$$ $\displaystyle L({\mathbf{X}}^{t+1},{\mathbf{y}}^{t+1},{\mathbf{z}}^{t+1};\beta^{t+1},\mu^{t+1})-L({\mathbf{X}}^{t+1},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\beta^{t},\mu^{t})+\Gamma_{\mu}^{t+1}-\Gamma_{\mu}^{t}+\Gamma_{{\sf z}}^{t+1}-\Gamma_{{\sf z}}^{t}$ (59) $\displaystyle\leq$ $\displaystyle-\varepsilon_{y}\beta^{t}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}-\varepsilon_{z}\beta^{t}\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|_{2}^{2}+(\mu^{t}-\mu^{t+1})C_{h}^{2}$ $$

Using Lemma [4.4](#S4.Thmtheorem4), we have:

$$ $\displaystyle L({\mathbf{X}}^{t+1},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\beta^{t},\mu^{t})-L({\mathbf{X}}^{t},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\beta^{t},\mu^{t})\leq-\varepsilon_{x}\beta^{t}\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}^{2}+\Gamma_{{\sf x}}^{t}-\Gamma_{{\sf x}}^{t+1}$ $$

Adding these two inequalities together and using the definition of $\Theta^{t}$, we have:

$$ $\displaystyle\Theta^{t}-\Theta^{t+1}$ $\displaystyle\geq$ $\displaystyle\varepsilon_{y}\beta^{t}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}+\varepsilon_{x}\beta^{t}\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}^{2}+\varepsilon_{z}\beta^{t}\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|_{2}^{2}$ $\displaystyle\geq$ $\displaystyle\min(\varepsilon_{y},\varepsilon_{x},\varepsilon_{z})\cdot\beta^{t}\cdot\{\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}+\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}^{2}+\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|_{2}^{2}\}$ $$

(b) Telescoping this inequality over $t$ from 1 to $T$, we have:

$$ $\displaystyle\textstyle\sum_{t=1}^{T}\beta^{t}\{\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}+\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}^{2}+\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|_{2}^{2}\}$ $\displaystyle\leq$ $\displaystyle\textstyle\tfrac{1}{\min(\varepsilon_{y},\varepsilon_{x},\varepsilon_{z})}\cdot\sum_{t=1}^{T}(\Theta^{t}-\Theta^{t+1})$ $\displaystyle=$ $\displaystyle\textstyle\tfrac{1}{\min(\varepsilon_{y},\varepsilon_{x},\varepsilon_{z})}\cdot(\Theta^{1}-\Theta^{T+1})$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\textstyle\tfrac{1}{\min(\varepsilon_{y},\varepsilon_{x},\varepsilon_{z})}\cdot(\Theta^{1}-\underline{\rm{\Theta}})\triangleq W_{1},$ $$

where step ① uses Lemma [4.2](#S4.Thmtheorem2) that $\Theta^{t}\geq\underline{\rm{\Theta}}$.

∎

###### Proof.

### D.6 Proof of Theorem 4.6

###### Proof.

We define

$$ $\displaystyle{\rm{Crit}}^{\star}({\mathbf{X}},{\mathbf{y}},{\mathbf{z}})$ $\displaystyle\triangleq\|{\mathcal{A}}({\mathbf{X}})-{\mathbf{y}}\|+\|\partial h({\mathbf{y}})-{\mathbf{z}}\|+\|{\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}}\mathcal{M}}\left(\nabla f({\mathbf{X}})-\partial g({\mathbf{X}})+{\rm{mat}}(\nabla{\mathcal{A}}({\mathbf{X}})^{\mathsf{T}}{\mathbf{z}})\right)\|_{\mathsf{F}}.$ $\displaystyle{\rm{Crit}}({\mathbf{X}},{\mathbf{y}},{\mathbf{z}})$ $\displaystyle\triangleq\|{\mathcal{A}}({\mathbf{X}})-{\mathbf{y}}\|^{2}_{2}+\|\partial h({\mathbf{y}})-{\mathbf{z}}\|^{2}_{2}+\|{\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}}\mathcal{M}}\left(\nabla f({\mathbf{X}})-\partial g({\mathbf{X}})+{\rm{mat}}(\nabla{\mathcal{A}}({\mathbf{X}})^{\mathsf{T}}{\mathbf{z}})\right)\|_{\mathsf{F}}^{2}.$ $\displaystyle{\mathbf{G}}_{1}$ $\displaystyle\triangleq\nabla f({\mathbf{X}}_{{\sf c}}^{t})-\partial g({\mathbf{X}}^{t})+{\rm{mat}}(\nabla{\mathcal{A}}({\mathbf{X}}_{{\sf c}}^{t})^{\mathsf{T}}({\mathbf{z}}^{t}+\beta^{t}{\mathcal{A}}({\mathbf{X}}_{{\sf c}}^{t})-{\mathbf{y}}^{t}))+\theta\ell(\beta^{t})({\mathbf{X}}^{t+1}-{\mathbf{X}}_{{\sf c}}^{t})$ $\displaystyle{\mathbf{G}}_{2}$ $\displaystyle\triangleq\nabla f({\mathbf{X}}^{t})-\partial g({\mathbf{X}}^{t})+{\rm{mat}}(\nabla{\mathcal{A}}({\mathbf{X}}^{t})^{\mathsf{T}}{\mathbf{z}}^{t}).$ $$

We first derive the following inequalities:

$$ $\displaystyle\|{\mathbf{G}}_{1}-{\mathbf{G}}_{2}\|_{\mathsf{F}}$ (60) $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\|\nabla f({\mathbf{X}}^{t})-\nabla f({\mathbf{X}}_{{\sf c}}^{t})+{\rm{mat}}((\nabla{\mathcal{A}}({\mathbf{X}}^{t})-\nabla{\mathcal{A}}({\mathbf{X}}_{{\sf c}}^{t}))^{\mathsf{T}}{\mathbf{z}}^{t})$ $\displaystyle-\beta^{t}{\rm{mat}}(\nabla{\mathcal{A}}({\mathbf{X}}_{{\sf c}}^{t})^{\mathsf{T}}({\mathcal{A}}({\mathbf{X}}_{{\sf c}}^{t})-{\mathbf{y}}^{t}))-\theta\ell(\beta^{t})({\mathbf{X}}^{t+1}-{\mathbf{X}}_{{\sf c}}^{t})\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle L_{f}\|{\mathbf{X}}^{t}-{\mathbf{X}}_{{\sf c}}^{t}\|_{\mathsf{F}}+\|{\mathbf{z}}^{t}\|\|\nabla{\mathcal{A}}({\mathbf{X}}^{t})-\nabla{\mathcal{A}}({\mathbf{X}}_{{\sf c}}^{t})\|_{\mathsf{F}}$ $\displaystyle+\beta^{t}\|\nabla{\mathcal{A}}({\mathbf{X}}_{{\sf c}}^{t})\|\|{\mathcal{A}}({\mathbf{X}}_{{\sf c}}^{t})-{\mathbf{y}}^{t}\|+\theta\ell(\beta^{t})\|{\mathbf{X}}^{t+1}-{\mathbf{X}}_{{\sf c}}^{t}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle(L_{f}+\overline{\rm{z}}L_{A})\|{\mathbf{X}}^{t}-{\mathbf{X}}_{{\sf c}}^{t}\|_{\mathsf{F}}+\beta^{t}C_{A}\|{\mathcal{A}}({\mathbf{X}}_{{\sf c}}^{t})-{\mathbf{y}}^{t}\|+\theta\ell(\beta^{t})\|{\mathbf{X}}^{t+1}-{\mathbf{X}}_{{\sf c}}^{t}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 175}}{\leq}$ $\displaystyle(L_{f}+\overline{\rm{z}}L_{A})\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}+\beta^{t}C_{A}\{\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|+C_{A}\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}\}$ $\displaystyle+\theta\ell(\beta^{t})\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}+\theta\ell(\beta^{t})\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 176}}{\leq}$ $\displaystyle(L_{f}+\overline{\rm{z}}L_{A}+\beta^{t}C_{A}^{2}+\theta\ell(\beta^{t}))\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}+\beta^{t}C_{A}\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|$ $\displaystyle+\theta\ell(\beta^{t})\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}},$ $$

where step ① uses the definitions of $\{{\mathbf{G}}_{1},{\mathbf{G}}_{2}\}$; step ② uses the triangle inequality, and the fact that $f({\mathbf{X}})$ is $L_{f}$-smooth; step ③ uses $\|{\mathbf{z}}^{t}\|\leq\overline{\rm{z}}$, $L_{A}$-smoothness of ${\mathcal{A}}({\mathbf{X}})$, and $C_{A}$-continuity of ${\mathcal{A}}({\mathbf{X}})$; step ④ uses $\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t}_{{\sf c}}\|_{\mathsf{F}}\leq\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}$, $\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}_{{\sf c}}\|_{\mathsf{F}}\leq\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}+\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}$, and $\|{\mathcal{A}}({\mathbf{X}}_{{\sf c}}^{t})-{\mathbf{y}}^{t}\|\leq\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|_{\mathsf{F}}+C_{A}\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}$, as shown in Lemma [A.4](#A1.Thmtheorem4); step ⑤ uses $\beta^{t}C_{A}\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|=\tfrac{\beta^{t}C_{A}}{\sigma\beta^{t-1}}\|{\mathbf{z}}^{t}-{\mathbf{z}}^{t-1}\|\leq\tfrac{(1+\xi)C_{A}}{\sigma}\|{\mathbf{z}}^{t}-{\mathbf{z}}^{t-1}\|$

We derive the following inequalities:

$$ $\displaystyle\|{\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}^{t}}\mathcal{M}}({\mathbf{G}}_{2})\|_{\mathsf{F}}$ (61) $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\|{\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}^{t}}\mathcal{M}}({\mathbf{G}}_{2})+{\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}^{t+1}}\mathcal{M}}({\mathbf{G}}_{1})\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle 2\|{\mathbf{G}}_{2}-{\mathbf{G}}_{1}\|_{\mathsf{F}}+2\sqrt{r}\|{\mathbf{G}}_{2}\|\cdot\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle 2(L_{f}+\overline{\rm{z}}L_{A}+\beta^{t}C_{A}^{2}+\theta\ell(\beta^{t}))\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}+2\beta^{t}C_{A}\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|$ $\displaystyle+2\theta\ell(\beta^{t})\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}+2\sqrt{r}(C_{f}+C_{g}+C_{A}\overline{\rm{z}})\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}$ $$

where step ① uses the optimality of ${\mathbf{X}}^{t+1}$ that:

$$ $\displaystyle\mathbf{0}={\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}^{t+1}}\mathcal{M}}({\mathbf{G}}_{1});$ $$

step ② uses the result of Lemma [A.5](#A1.Thmtheorem5) by applying

$$ $\displaystyle{\mathbf{X}}={\mathbf{X}}^{t},~{}\tilde{{\mathbf{X}}}={\mathbf{X}}^{t+1},~{}\mathbf{P}={\mathbf{G}}_{2},~{}\text{and}~{}\tilde{\mathbf{P}}={\mathbf{G}}_{1};$ (62) $$

step ③ uses Inequality ([60](#A4.E60)), and the fact that $\|{\mathbf{G}}_{2}\|=\|\nabla f({\mathbf{X}}^{t})-\partial g({\mathbf{X}}^{t})+{\rm{mat}}(\nabla{\mathcal{A}}({\mathbf{X}}^{t})^{\mathsf{T}}{\mathbf{z}}^{t})\|\leq\|\nabla f({\mathbf{X}}^{t})-\partial g({\mathbf{X}}^{t})+{\rm{mat}}(\nabla{\mathcal{A}}({\mathbf{X}}^{t})^{\mathsf{T}}{\mathbf{z}}^{t})\|_{\mathsf{F}}\leq C_{f}+C_{g}+C_{A}\overline{\rm{z}}$.

We consider the solution $({\mathbf{X}}^{t},\breve{{\mathbf{y}}}^{t},{\mathbf{z}}^{t})$ with $t\geq 1$. We have:

$$ $\displaystyle{\rm{Crit}}^{\star}({\mathbf{X}}^{t},\breve{{\mathbf{y}}}^{t},{\mathbf{z}}^{t})$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\|{\mathcal{A}}({\mathbf{X}}^{t})-\breve{{\mathbf{y}}}^{t}\|+\|\partial h(\breve{{\mathbf{y}}}^{t})-{\mathbf{z}}^{t}\|+\|{\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}^{t}}\mathcal{M}}({\mathbf{G}}_{2})\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|+\|{\mathbf{y}}^{t}-\breve{{\mathbf{y}}}^{t}\|+\|\partial h(\breve{{\mathbf{y}}}^{t})-{\mathbf{z}}^{t}\|+\|{\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}^{t}}\mathcal{M}}({\mathbf{G}}_{2})\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|+\mu^{t-1}C_{h}+(\sigma-1)\beta^{t-1}\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|$ $\displaystyle+2(L_{f}+\overline{\rm{z}}L_{A}+\beta^{t}C_{A}^{2}+\theta\ell(\beta^{t}))\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}+2\beta^{t}C_{A}\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|$ $\displaystyle+\{2\theta\ell(\beta^{t})+2\sqrt{r}(C_{f}+C_{g}+C_{A}\overline{\rm{z}})\}\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 175}}{\leq}$ $\displaystyle K_{z}\beta^{t-1}\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|+K_{x}\beta^{t-1}\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}+K_{x}\beta^{t}\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}+\mu^{t-1}C_{h},$ $$

where step ① uses the definition of ${\rm{Crit}}^{\star}({\mathbf{X}},{\mathbf{y}},{\mathbf{z}})$; step ② uses the triangle inequality; step ③ uses Claim (c) of Lemma [3.6](#S3.Thmtheorem6), Claim (d) of Lemma [4.1](#S4.Thmtheorem1), and Inequality ([61](#A4.E61)); step ④ uses $\frac{1}{\beta^{t}}\leq\frac{1}{\beta^{0}}$, and $\ell(\beta^{t})\leq\beta^{t}\overline{\rm{L}}$; step ⑤ uses the definitions of $\{K_{x},K_{z}\}$ that:

$$ $\displaystyle K_{z}$ $\displaystyle\triangleq$ $\displaystyle\tfrac{1}{\beta^{0}}+\sigma-1+2C_{A}(1+\xi)$ $\displaystyle K_{x}$ $\displaystyle\triangleq$ $\displaystyle\max(\tfrac{2L_{f}}{\beta^{0}}+\tfrac{2\overline{\rm{z}}L_{A}}{\beta^{0}}+2(1+\xi)C_{A}^{2}+2(1+\xi)\theta\overline{\rm{L}},\,2\theta\overline{\rm{L}}+2\sqrt{r}(C_{f}+C_{g}+C_{A}\overline{\rm{z}})).$ $$

We derive:

$$ $\displaystyle\textstyle\frac{1}{T}\sum_{t=1}^{T}{\rm{Crit}}({\mathbf{X}}^{t+1},\breve{{\mathbf{y}}}^{t+1},{\mathbf{z}}^{t+1})$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\textstyle\frac{1}{T}\sum_{t=1}^{T}({\rm{Crit}}^{\star}({\mathbf{X}}^{t+1},\breve{{\mathbf{y}}}^{t+1},{\mathbf{z}}^{t+1}))^{2}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\textstyle\frac{1}{T}\sum_{t=1}^{T}(K_{z}\beta^{t}\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|+K_{x}\beta^{t}\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}+K_{x}\beta^{t+1}\|{\mathbf{X}}^{t+2}-{\mathbf{X}}^{t+1}\|_{\mathsf{F}}+\mu^{t}C_{h})^{2}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\textstyle\frac{4}{T}\sum_{t=1}^{T}K_{z}^{2}\|\beta^{t}({\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1})\|_{2}^{2}+\frac{4}{T}\sum_{t=1}^{T}K^{2}_{x}\|\beta^{t}({\mathbf{X}}^{t+1}-{\mathbf{X}}^{t})\|^{2}_{\mathsf{F}}$ $\displaystyle\textstyle+\frac{4}{T}\sum_{t=1}^{T}K^{2}_{x}\|\beta^{t+1}({\mathbf{X}}^{t+2}-{\mathbf{X}}^{t+1})\|^{2}_{\mathsf{F}}+\frac{4}{T}\sum_{t=1}^{T}(\mu^{t}C_{h})^{2}\}$ $\displaystyle\overset{\text{\char 175}}{\leq}$ $\displaystyle\textstyle\frac{4}{T}\cdot\beta^{T}\cdot\sum_{t=1}^{T}K_{z}^{2}\beta^{t}\|({\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1})\|_{2}^{2}+\frac{4}{T}\cdot\beta^{T}\sum_{t=1}^{T}K^{2}_{x}\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|^{2}_{\mathsf{F}}$ $\displaystyle\textstyle+\frac{4}{T}\cdot\beta^{T+1}\sum_{t=1}^{T}K^{2}_{x}\beta^{t+1}\|{\mathbf{X}}^{t+2}-{\mathbf{X}}^{t+1}\|^{2}_{\mathsf{F}}+\frac{4}{T}\sum_{t=1}^{T}(\mu^{t}C_{h})^{2}\}$ $\displaystyle\overset{\text{\char 176}}{\leq}$ $\displaystyle\textstyle\mathcal{O}(\frac{\beta^{T}}{T})+\mathcal{O}(\frac{\beta^{T}}{T})+\mathcal{O}(\frac{\beta^{T}}{T})+\mathcal{O}(\frac{\sum_{t=1}^{T}(1/\beta^{t})^{2}}{T})$ $\displaystyle\overset{\text{\char 177}}{=}$ $\displaystyle\textstyle\mathcal{O}(T^{p-1})+\mathcal{O}(\frac{T^{1-2p}}{T})$ $\displaystyle\overset{\text{\char 178}}{=}$ $\displaystyle\textstyle\mathcal{O}(T^{-2/3}),$ $$

where step ① uses $a^{2}+b^{2}+c^{2}\leq(a+b+c)^{2}$ for all $a,b,c\geq 0$; step ② uses Inequality ([D.6](#A4.Ex268)); step ③ uses $(a+b+c+d)^{2}\leq 4(a^{2}+b^{2}+c^{2}+d^{2})$; step ④ uses $\langle\mathbf{a},\mathbf{b}\rangle\leq\|\mathbf{a}\|_{\infty}\cdot\|\mathbf{b}\|_{1}$; step ⑤ uses Claim (b) of Lemma [4.5](#S4.Thmtheorem5) and $\mu^{t}=\chi/\beta^{t}$; step ⑥ uses $\beta^{t}=\mathcal{O}(t^{p})$ and $\sum_{t=1}^{\infty}\frac{1}{(\beta^{t})^{2}}=\mathcal{O}(t^{1-2p})$; step ⑦ uses the choice $p=1/3$.

∎

###### Proof.

### D.7 Proof of Lemma 4.7

###### Proof.

Noticing $\alpha_{x}=0$, we have ${\mathbf{X}}_{{\sf c}}^{t}={\mathbf{X}}^{t}$ for all $t$.

We define $\mathcal{S}^{t}({\mathbf{X}},{\mathbf{y}})\triangleq f({\mathbf{X}})+\langle{\mathbf{z}}^{t},{\mathcal{A}}({\mathbf{X}})-{\mathbf{y}}\rangle+\tfrac{\beta^{t}}{2}\|{\mathcal{A}}({\mathbf{X}})-{\mathbf{y}}\|_{2}^{2}$.

We define ${\mathbf{G}}^{t}\in\nabla_{{\mathbf{X}}}\mathcal{S}^{t}({\mathbf{X}}^{t},{\mathbf{y}}^{t})-\partial g({\mathbf{X}}^{t})$.

We define $\eta^{t}\triangleq\tfrac{\beta^{0}}{\beta^{t}}b^{t}\gamma^{j}\in(0,\infty)$.

(a) Initially, we show that $\|{\mathbf{G}}^{t}\|_{\mathsf{F}}$ is always bounded for $t$ with ${\mathbf{X}}\in\mathcal{M}$. We have:

$$ $\displaystyle\|{\mathbf{G}}^{t}\|_{\mathsf{F}}$ $\displaystyle=$ $\displaystyle\|\nabla f({\mathbf{X}}^{t})-\partial g({\mathbf{X}}^{t})+{\rm{mat}}(\nabla{\mathcal{A}}({\mathbf{X}}^{t})^{\mathsf{T}}[{\mathbf{z}}^{t}+\beta^{t}({\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t})])\|$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\|\nabla f({\mathbf{X}}^{t})-\partial g({\mathbf{X}}^{t})+\nabla{\mathcal{A}}({\mathbf{X}}^{t})^{\mathsf{T}}[{\mathbf{z}}^{t}+\tfrac{\beta^{t}}{\sigma\beta^{t-1}}({\mathbf{z}}^{t}-{\mathbf{z}}^{t-1})]\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\|\nabla f({\mathbf{X}}^{t})\|_{\mathsf{F}}+\|\partial g({\mathbf{X}}^{t})\|_{\mathsf{F}}+\|\nabla{\mathcal{A}}({\mathbf{X}}^{t})\|_{\mathsf{F}}\cdot\{\|{\mathbf{z}}^{t}\|+\tfrac{\beta^{t}}{\sigma\beta^{t-1}}(\|{\mathbf{z}}^{t}\|+\|{\mathbf{z}}^{t-1}\|)\}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle C_{f}+C_{g}+C_{A}\cdot(\overline{\rm{z}}+(1+\xi)\overline{\rm{z}}+\overline{\rm{z}})$ $\displaystyle\overset{\text{\char 175}}{\leq}$ $\displaystyle C_{f}+C_{g}+4\overline{\rm{z}}C_{A}\triangleq\overline{g},$ $$

where step ① uses ${\mathbf{z}}^{t+1}={\mathbf{z}}^{t}+\sigma\beta^{t}({\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1})$; step ② uses the triangle inequality; step ③ uses $\|\nabla f({\mathbf{X}}^{t})\|_{\mathsf{F}}\leq C_{f}$, $\|\nabla g({\mathbf{X}}^{t})\|_{\mathsf{F}}\leq C_{g}$, $\|\nabla{\mathcal{A}}({\mathbf{X}}^{t})\|_{\mathsf{F}}\leq\|\nabla{\mathcal{A}}({\mathbf{X}}^{t})\|\leq C_{A}$, $\|{\mathbf{z}}^{t}\|\leq\overline{\rm{z}}$, $\tfrac{1}{\sigma}\leq 1$, $\beta^{t}\leq\beta^{t-1}(1+\xi)$; step ④ uses $\xi\leq 1$.

We derive the following inequalities:

$$ $\displaystyle L({\mathbf{X}}^{t+1},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\beta^{t},\mu^{t})-L({\mathbf{X}}^{t},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\beta^{t},\mu^{t})=\dot{\mathcal{L}}({\mathbf{X}}^{t+1})-\dot{\mathcal{L}}({\mathbf{X}}^{t})$ (63) $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\{\mathcal{S}^{t}({\mathbf{X}}^{t+1},{\mathbf{y}}^{t})-g({\mathbf{X}}^{t+1})\}-\{\mathcal{S}^{t}({\mathbf{X}}^{t},{\mathbf{y}}^{t})-g({\mathbf{X}}^{t})\}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\langle{\mathbf{G}}^{t},{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\rangle+\tfrac{1}{2}\ell(\beta^{t})\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}^{2}$ $\displaystyle\overset{\text{\char 174}}{=}$ $\displaystyle\tfrac{1}{2}\ell(\beta^{t})\|{\sf{Retr}}_{{\mathbf{X}}^{t}}(\eta^{t}{\mathbf{D}}^{t}_{\rho})-{\mathbf{X}}^{t}\|_{\mathsf{F}}^{2}+\langle{\mathbf{G}}^{t},{\sf{Retr}}_{{\mathbf{X}}^{t}}(\eta^{t}{\mathbf{D}}^{t}_{\rho})-{\mathbf{X}}^{t}-\eta^{t}{\mathbf{D}}^{t}_{\rho}\rangle+\langle{\mathbf{G}}^{t},\eta^{t}{\mathbf{D}}^{t}_{\rho}\rangle$ $\displaystyle\overset{\text{\char 175}}{\leq}$ $\displaystyle\tfrac{1}{2}\ell(\beta^{t})\|{\sf{Retr}}_{{\mathbf{X}}^{t}}(\eta^{t}{\mathbf{D}}^{t}_{\rho})-{\mathbf{X}}^{t}\|_{\mathsf{F}}^{2}+\overline{g}\|{\sf{Retr}}_{{\mathbf{X}}^{t}}(\eta^{t}{\mathbf{D}}^{t}_{\rho})-{\mathbf{X}}^{t}-\eta^{t}{\mathbf{D}}^{t}_{\rho}\|_{\mathsf{F}}-\tfrac{1}{2}\eta^{t}\|{\mathbf{D}}^{t}_{\rho}\|_{\mathsf{F}}^{2}$ $\displaystyle\overset{\text{\char 176}}{\leq}$ $\displaystyle\tfrac{1}{2}\ell(\beta^{t})k_{1}\|\eta^{t}{\mathbf{D}}^{t}_{\rho}\|_{\mathsf{F}}^{2}+\overline{g}k_{2}\|\eta^{t}{\mathbf{D}}^{t}_{\rho}\|_{\mathsf{F}}^{2}-\tfrac{1}{2}\eta^{t}\|{\mathbf{D}}^{t}_{\rho}\|_{\mathsf{F}}^{2}$ $\displaystyle\overset{\text{\char 177}}{=}$ $\displaystyle\eta^{t}\|{\mathbf{D}}^{t}_{\rho}\|_{\mathsf{F}}^{2}\cdot\{\tfrac{1}{2}\ell(\beta^{t})k_{1}\tfrac{\beta^{0}}{\beta^{t}}b^{t}\gamma^{j}+\overline{g}k_{2}\tfrac{\beta^{0}}{\beta^{t}}b^{t}\gamma^{j}-\tfrac{1}{2}\}$ $\displaystyle\overset{\text{\char 178}}{\leq}$ $\displaystyle\eta^{t}\|{\mathbf{D}}^{t}_{\rho}\|_{\mathsf{F}}^{2}\cdot\{\tfrac{1}{2}\overline{\rm{L}}k_{1}\overline{b}\gamma^{j}+\overline{g}k_{2}\overline{b}\gamma^{j}-\tfrac{1}{2}\}$ $\displaystyle\overset{\text{\char 179}}{\leq}$ $\displaystyle\eta^{t}\|{\mathbf{D}}^{t}_{\rho}\|_{\mathsf{F}}^{2}\cdot\{-\delta\},$ $$

where step ① uses the definitions of $\dot{\mathcal{L}}({\mathbf{X}}^{t})$ and $L({\mathbf{X}}^{t},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\beta^{t},\mu^{t})$; step ② uses the fact that the function $g({\mathbf{X}})$ is convex and the function $\mathcal{S}^{t}({\mathbf{X}},{\mathbf{y}}^{t})$ is $\ell(\beta^{t})$-smooth w.r.t. ${\mathbf{X}}$; step ③ uses ${\mathbf{X}}^{t+1}={\sf{Retr}}_{{\mathbf{X}}^{t}}(\eta^{t}{\mathbf{D}}^{t}_{\rho})$; step ④ uses the Cauchy-Schwarz Inequality, $\|{\mathbf{G}}^{t}\|_{\mathsf{F}}\leq\overline{g}$, and Claim (a) of Lemma [2.11](#S2.Thmtheorem11) that $\langle{\mathbf{G}}^{t},{\mathbf{D}}^{t}_{\rho}\rangle\leq-\tfrac{1}{\max(1,2\rho)}\|{\mathbf{D}}^{t}_{\rho}\|_{\mathsf{F}}^{2}$; step ⑤ uses Lemma [2.9](#S2.Thmtheorem9) with $\mathbf{\Delta}\triangleq\eta^{t}{\mathbf{D}}^{t}_{\rho}$ given that ${\mathbf{X}}^{t}\in\mathcal{M}$ and $\mathbf{\Delta}\in{\rm{T}}_{{\mathbf{X}}^{t}}\mathcal{M}$; step ⑥ uses $\eta^{t}\triangleq\tfrac{\beta^{0}}{\beta^{t}}b^{t}\gamma^{j}$; step ⑦ uses $\ell(\beta^{t})\leq\beta^{t}\overline{\rm{L}}$, $\beta^{0}\leq\beta^{t}$, and $b^{t}\leq\overline{b}$; step ⑧ uses the fact that $\gamma^{j}$ is sufficiently small such that:

$$ $\displaystyle\gamma^{j}\leq\frac{\tfrac{1}{2}-\delta}{\overline{g}k_{2}\overline{b}+\tfrac{1}{2}\overline{\rm{L}}k_{1}\overline{b}}\triangleq\overline{\gamma}.$ $$

Given Inequality ([63](#A4.E63)) coincides with the condition in Equation ([6](#S3.E6)), we have completed the proof of the first part of this lemma.

(b) We derive the following inequalities:

$$ $\displaystyle L({\mathbf{X}}^{t+1},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\beta^{t},\mu^{t})-L({\mathbf{X}}^{t},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\beta^{t},\mu^{t})$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle-\delta{\eta}^{t}\|{\mathbf{D}}^{t}_{\rho}\|_{\mathsf{F}}^{2}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle-\delta{\eta}^{t}\cdot\min(1,2\rho)^{2}\|{\mathbf{D}}^{t}_{1/2}\|^{2}_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 174}}{=}$ $\displaystyle-\delta\beta^{0}b^{t}\gamma^{j-1}\gamma\cdot\min(1,2\rho)^{2}\cdot\tfrac{1}{\beta^{t}}\|{\mathbf{D}}^{t}_{1/2}\|^{2}_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 175}}{\leq}$ $\displaystyle-\underbrace{\delta\beta^{0}\underline{b}\overline{\gamma}\gamma\cdot\min(1,2\rho)^{2}}_{\triangleq\varepsilon_{x}}\cdot\tfrac{1}{\beta^{t}}\|{\mathbf{D}}^{t}_{1/2}\|^{2}_{\mathsf{F}},$ $$

where step ① uses the stopping criteria in Inequality ([6](#S3.E6)); step ② uses Claim (c) of Lemma [2.11](#S2.Thmtheorem11) that $\|{\mathbf{D}}_{\rho}\|_{\mathsf{F}}\geq\min(1,2\rho)\|{\mathbf{D}}_{1/2}\|_{\mathsf{F}}$; step ③ uses the definition ${\eta}^{t}\triangleq\tfrac{\beta^{0}}{\beta^{t}}b^{t}\gamma^{j}$; step ④ uses $b^{t}\geq\underline{b}$, and the following inequality:

$$ $\displaystyle\gamma^{j-1}\geq\overline{\gamma},$ $$

which can be implied by the the first part of this lemma, and the stopping criteria in Inequality ([6](#S3.E6)).

∎

###### Proof.

### D.8 Proof of Lemma 4.8

###### Proof.

We let $t\geq 1$.

We define: $\Theta^{t}\triangleq L({\mathbf{X}}^{t},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\beta^{t},\mu^{t})+\Gamma^{t}_{z}+\mu^{t}C_{h}^{2}+0\times\Gamma_{{\sf x}}^{t}+\Gamma_{\mu}^{t}$,

(a) Using Lemma [4.3](#S4.Thmtheorem3), we have:

$$ $\displaystyle L({\mathbf{X}}^{t+1},{\mathbf{y}}^{t+1},{\mathbf{z}}^{t+1};\beta^{t+1},\mu^{t+1})-L({\mathbf{X}}^{t+1},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\beta^{t},\mu^{t})+\Gamma_{\mu}^{t+1}-\Gamma_{\mu}^{t}+\Gamma_{{\sf z}}^{t+1}-\Gamma_{{\sf z}}^{t}$ $\displaystyle\leq$ $\displaystyle-\varepsilon_{y}\beta^{t}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}-\varepsilon_{z}\beta^{t}\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|_{2}^{2}+(\mu^{t}-\mu^{t+1})C_{h}^{2}$ $$

Using Lemma [4.7](#S4.Thmtheorem7), we have:

$$ $\displaystyle L({\mathbf{X}}^{t+1},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\beta^{t},\mu^{t})-L({\mathbf{X}}^{t},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\beta^{t},\mu^{t})\leq-\varepsilon_{x}\beta^{t}\|\tfrac{1}{\beta^{t}}{\mathbf{D}}^{t}_{1/2}\|^{2}_{\mathsf{F}}+0\times\Gamma_{{\sf x}}^{t}-0\times\Gamma_{{\sf x}}^{t+1}$ $$

Adding these two inequalities together and using the definition of $\Theta^{t}$, we have:

$$ $\displaystyle\Theta^{t}-\Theta^{t+1}$ $\displaystyle\geq$ $\displaystyle\varepsilon_{y}\beta^{t}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}+\varepsilon_{x}\beta^{t}\|\tfrac{1}{\beta^{t}}{\mathbf{D}}^{t}_{1/2}\|^{2}_{\mathsf{F}}+\varepsilon_{z}\beta^{t}\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|_{2}^{2}$ $\displaystyle\geq$ $\displaystyle\min(\varepsilon_{y},\varepsilon_{x},\varepsilon_{z})\cdot\beta^{t}\cdot\{\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}+\|\tfrac{1}{\beta^{t}}{\mathbf{D}}^{t}_{1/2}\|^{2}_{\mathsf{F}}+\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|_{2}^{2}\}$ $$

(b) Telescoping this inequality over $t$ from 1 to $T$, we have:

$$ $\displaystyle\textstyle\sum_{t=1}^{T}\beta^{t}\{\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}+\|\tfrac{1}{\beta^{t}}{\mathbf{D}}^{t}_{1/2}\|^{2}_{\mathsf{F}}+\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|_{2}^{2}\}$ $\displaystyle\leq$ $\displaystyle\textstyle\tfrac{1}{\min(\varepsilon_{y},\varepsilon_{x},\varepsilon_{z})}\cdot\sum_{t=1}^{T}(\Theta^{t}-\Theta^{t+1})$ $\displaystyle=$ $\displaystyle\textstyle\tfrac{1}{\min(\varepsilon_{y},\varepsilon_{x},\varepsilon_{z})}\cdot(\Theta^{1}-\Theta^{T+1})$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\textstyle\tfrac{1}{\min(\varepsilon_{y},\varepsilon_{x},\varepsilon_{z})}\cdot(\Theta^{1}-\underline{\rm{\Theta}})\triangleq W_{1},$ $$

where step ① uses Lemma [4.2](#S4.Thmtheorem2) that $\Theta^{t}\geq\underline{\rm{\Theta}}$.

∎

###### Proof.

### D.9 Proof of Theorem 4.9

###### Proof.

We define

$$ $\displaystyle{\rm{Crit}}^{\star}({\mathbf{X}},{\mathbf{y}},{\mathbf{z}})$ $\displaystyle\triangleq$ $\displaystyle\|{\mathcal{A}}({\mathbf{X}})-{\mathbf{y}}\|+\|\partial h({\mathbf{y}})-{\mathbf{z}}\|+\|{\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}}\mathcal{M}}\left(\nabla f({\mathbf{X}})-\partial g({\mathbf{X}})+{\rm{mat}}(\nabla{\mathcal{A}}({\mathbf{X}})^{\mathsf{T}}{\mathbf{z}})\right)\|_{\mathsf{F}},$ $\displaystyle{\rm{Crit}}({\mathbf{X}},{\mathbf{y}},{\mathbf{z}})$ $\displaystyle\triangleq$ $\displaystyle\|{\mathcal{A}}({\mathbf{X}})-{\mathbf{y}}\|^{2}_{2}+\|\partial h({\mathbf{y}})-{\mathbf{z}}\|^{2}_{2}+\|{\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}}\mathcal{M}}\left(\nabla f({\mathbf{X}})-\partial g({\mathbf{X}})+{\rm{mat}}(\nabla{\mathcal{A}}({\mathbf{X}})^{\mathsf{T}}{\mathbf{z}})\right)\|_{\mathsf{F}}^{2},$ $\displaystyle{\mathbf{G}}_{1}$ $\displaystyle\triangleq$ $\displaystyle\nabla f({\mathbf{X}}^{t})-\partial g({\mathbf{X}}^{t})+{\rm{mat}}([\nabla{\mathcal{A}}({\mathbf{X}}^{t})]^{\mathsf{T}}{\mathbf{z}}^{t}),$ $\displaystyle{\mathbf{G}}_{2}$ $\displaystyle\triangleq$ $\displaystyle\beta^{t}{\rm{mat}}([\nabla{\mathcal{A}}({\mathbf{X}}^{t})]^{\mathsf{T}}({\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t})).$ $$

We let ${\mathbf{G}}\in{\mathbf{G}}_{1}+{\mathbf{G}}_{2}=\partial\dot{\mathcal{L}}({\mathbf{X}}^{t})$, where $\dot{\mathcal{L}}({\mathbf{X}})\triangleq L({\mathbf{X}},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\beta^{t},\mu^{t})$.

We further obtain:

$$ $\displaystyle{\mathbf{D}}^{t}_{1/2}$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\tfrac{1}{2}{\mathbf{X}}^{t}{\mathbf{G}}^{\mathsf{T}}{\mathbf{X}}^{t}-\tfrac{1}{2}{\mathbf{X}}^{t}[{\mathbf{X}}^{t}]^{\mathsf{T}}{\mathbf{G}}-{\mathbf{G}}$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle(\tfrac{1}{2}{\mathbf{X}}^{t}{\mathbf{G}}_{1}^{\mathsf{T}}{\mathbf{X}}^{t}-\tfrac{1}{2}{\mathbf{X}}^{t}[{\mathbf{X}}^{t}]^{\mathsf{T}}{\mathbf{G}}_{1}-{\mathbf{G}}_{1})+(\tfrac{1}{2}{\mathbf{X}}^{t}{\mathbf{G}}_{2}^{\mathsf{T}}{\mathbf{X}}^{t}-\tfrac{1}{2}{\mathbf{X}}^{t}[{\mathbf{X}}^{t}]^{\mathsf{T}}{\mathbf{G}}_{2}-{\mathbf{G}}_{2})$ $\displaystyle\overset{\text{\char 174}}{=}$ $\displaystyle-{\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}^{t}}\mathcal{M}}({\mathbf{G}}_{1})-{\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}^{t}}\mathcal{M}}({\mathbf{G}}_{2})$ $$

where step ① uses the definition ${\mathbf{D}}^{t}_{\rho}\triangleq\rho{\mathbf{X}}^{t}{\mathbf{G}}^{\mathsf{T}}{\mathbf{X}}^{t}-(1-\rho){\mathbf{X}}^{t}[{\mathbf{X}}^{t}]^{\mathsf{T}}{\mathbf{G}}-{\mathbf{G}}$, where ${\mathbf{G}}\in\partial\dot{\mathcal{L}}({\mathbf{X}}^{t})$, as shown in Algorithm [1](#alg1); step ② uses ${\mathbf{G}}\in{\mathbf{G}}_{1}+{\mathbf{G}}_{2}$; step ③ uses the fact that ${\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}}\mathcal{M}}(\mathbf{\Delta})=\mathbf{\Delta}-\tfrac{1}{2}{\mathbf{X}}(\mathbf{\Delta}^{\mathsf{T}}{\mathbf{X}}+{\mathbf{X}}^{\mathsf{T}}\mathbf{\Delta})$ for all $\mathbf{\Delta}\in\mathbb{R}^{n\times r}$ []. This leads to:

$$ $\displaystyle\|{\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}^{t}}\mathcal{M}}({\mathbf{G}}_{1})\|_{\mathsf{F}}$ $\displaystyle=$ $\displaystyle\|{\mathbf{D}}^{t}_{1/2}-{\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}^{t}}\mathcal{M}}({\mathbf{G}}_{2})\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\|{\mathbf{D}}^{t}_{1/2}\|_{\mathsf{F}}+\|{\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}^{t}}\mathcal{M}}({\mathbf{G}}_{2})\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\|{\mathbf{D}}^{t}_{1/2}\|_{\mathsf{F}}+\|{\mathbf{G}}_{2}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 174}}{=}$ $\displaystyle\|{\mathbf{D}}^{t}_{1/2}\|_{\mathsf{F}}+\beta^{t}\|[\nabla{\mathcal{A}}({\mathbf{X}}^{t})]^{\mathsf{T}}({\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t})\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 175}}{\leq}$ $\displaystyle\|{\mathbf{D}}^{t}_{1/2}\|_{\mathsf{F}}+\beta^{t}C_{A}\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|,$ $$

where step ① uses the triangle inequality; step ② uses Lemma [2.10](#S2.Thmtheorem10) that $\|{\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}}\mathcal{M}}(\mathbf{\Delta})\|_{\mathsf{F}}\leq\|\mathbf{\Delta}\|_{\mathsf{F}}$ for all $\mathbf{\Delta}\in\mathbb{R}^{n\times r}$; step ③ uses the definition of ${\mathbf{G}}_{2}\triangleq\beta^{t}[\nabla{\mathcal{A}}({\mathbf{X}}^{t})]^{\mathsf{T}}({\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t})$; step ④ uses $C_{A}$-Lipschitz continuity of ${\mathcal{A}}({\mathbf{X}})$ that $\|{\mathcal{A}}({\mathbf{X}})-{\mathcal{A}}({\mathbf{X}}^{\prime})\|\leq C_{A}\|{\mathbf{X}}-{\mathbf{X}}^{\prime}\|_{\mathsf{F}}$ for all ${\mathbf{X}},{\mathbf{X}}^{\prime}\in\mathbb{R}^{n\times r}$.

We consider the solution $({\mathbf{X}}^{t},\breve{{\mathbf{y}}}^{t},{\mathbf{z}}^{t})$ with $t\geq 1$. We have:

$$ $\displaystyle{\rm{Crit}}^{\star}({\mathbf{X}}^{t},\breve{{\mathbf{y}}}^{t},{\mathbf{z}}^{t})$ (64) $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\|{\mathbf{y}}^{t}-\breve{{\mathbf{y}}}^{t}\|+\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|+\|\partial h(\breve{{\mathbf{y}}}^{t})-{\mathbf{z}}^{t}\|+\|{\rm{Proj}}_{{\rm{T}}_{{\mathbf{X}}^{t}}\mathcal{M}}({\mathbf{G}}_{1})\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\|{\mathbf{y}}^{t}-\breve{{\mathbf{y}}}^{t}\|+\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|+(\sigma-1)\beta^{t-1}\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|+\|{\mathbf{D}}^{t}_{1/2}\|_{\mathsf{F}}+\beta^{t}C_{A}\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\mu^{t-1}C_{h}+\{\tfrac{1}{\beta^{0}}+1+(1+\xi)C_{A}\}\cdot\beta^{t-1}\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|+\|{\mathbf{D}}^{t}_{1/2}\|_{\mathsf{F}},$ $$

where step ① uses the definition of ${\mathbf{G}}_{1}$, and the triangle inequality; step ② uses Claim (d) of Lemma [4.1](#S4.Thmtheorem1); step ③ uses Claim (c) of Lemma [3.6](#S3.Thmtheorem6) and $\sigma-1\in[0,1)$.

We define $K_{z}\triangleq\tfrac{1}{\beta^{0}}+1+(1+\xi)C_{A}$. We derive:

$$ $\displaystyle\textstyle\frac{1}{T}\sum_{t=1}^{T}{\rm{Crit}}({\mathbf{X}}^{t+1},\breve{{\mathbf{y}}}^{t+1},{\mathbf{z}}^{t+1})$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\textstyle\frac{1}{T}\sum_{t=1}^{T}({\rm{Crit}}^{\star}({\mathbf{X}}^{t+1},\breve{{\mathbf{y}}}^{t+1},{\mathbf{z}}^{t+1}))^{2}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\textstyle\frac{1}{T}\sum_{t=1}^{T}(K_{z}\beta^{t}\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|+C_{h}\mu^{t}+\|{\mathbf{D}}^{t+1}_{1/2}\|_{\mathsf{F}})^{2}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\textstyle\frac{3}{T}\sum_{t=1}^{T}(\frac{3}{T}\sum_{t=1}^{T}K_{z}^{2}(\beta^{t}\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|)^{2}+\tfrac{3}{T}\sum_{t=1}^{T}(C_{h}\mu^{t})^{2}+\tfrac{4}{T}\sum_{t=1}^{T}\beta^{t}\cdot\tfrac{1}{\beta^{t}}\|{\mathbf{D}}^{t+1}_{1/2}\|_{\mathsf{F}}^{2}$ $\displaystyle\overset{\text{\char 175}}{\leq}$ $\displaystyle\textstyle\frac{3\beta^{T}}{T}\sum_{t=1}^{T}K_{z}^{2}\beta^{t}\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|_{2}^{2}+\tfrac{3C_{h}^{2}}{T}\sum_{t=1}^{T}(\mu^{t})^{2}+\tfrac{3\beta^{T}}{T}\sum_{t=1}^{T}\tfrac{1}{\beta^{t}}\|{\mathbf{D}}^{t+1}_{1/2}\|_{\mathsf{F}}^{2}$ $\displaystyle\overset{\text{\char 176}}{=}$ $\displaystyle\textstyle\mathcal{O}(\frac{\beta^{T}}{T})+\mathcal{O}(\frac{\beta^{T}}{T})+\mathcal{O}(\frac{\sum_{t=1}^{T}(1/\beta^{t})^{2}}{T})+\mathcal{O}(\frac{\beta^{T}}{T})$ $\displaystyle\overset{\text{\char 177}}{=}$ $\displaystyle\textstyle\mathcal{O}(T^{p-1})+\mathcal{O}(\frac{T^{1-2p}}{T})$ $\displaystyle\overset{\text{\char 178}}{=}$ $\displaystyle\textstyle\mathcal{O}(T^{-2/3}),$ $$

where step ① uses $a^{2}+b^{2}+c^{2}\leq(a+b+c)^{2}$ for all $a,b,c\geq 0$; step ② uses Inequality ([64](#A4.E64)); step ③ uses $(a+b+c)^{2}\leq 3(a^{2}+b^{2}+c^{2})$; step ④ uses $\langle\mathbf{a},\mathbf{b}\rangle\leq\|\mathbf{a}\|_{\infty}\cdot\|\mathbf{b}\|_{1}$; step ⑤ uses Claim (b) of Lemma [4.8](#S4.Thmtheorem8) and $\mu^{t}=\chi/\beta^{t}$; step ⑥ uses $\beta^{t}=\mathcal{O}(t^{p})$ and $\sum_{t=1}^{\infty}\frac{1}{(\beta^{t})^{2}}=\mathcal{O}(t^{1-2p})$; step ⑦ uses the choice $p=1/3$.

∎

###### Proof.

## Appendix E Proofs for Section 5

### E.1 Proof of Lemma 5.3

###### Lemma E.1 .

For both OADMM-EP and OADMM-RR, we define the Lyapunov function as: $\Theta({\mathbf{X}},{\mathbf{X}}^{-},{\mathbf{y}},{\mathbf{z}};\beta,\beta^{-},\mu)\triangleq L({\mathbf{X}},{\mathbf{y}},{\mathbf{z}};\beta,\mu)+v_{1}\tfrac{(\beta^{-})^{2}}{\beta}\|{\mathcal{A}}({\mathbf{X}})-{\mathbf{y}}\|_{2}^{2}+v_{2}\ell(\beta)\|{\mathbf{X}}-{\mathbf{X}}^{-}\|_{\mathsf{F}}^{2}$. We have:

$$ $\displaystyle({\mathbf{d}}_{{\mathbf{X}}},{\mathbf{d}}_{{\mathbf{X}}^{-}},{\mathbf{d}}_{{\mathbf{y}}},{\mathbf{d}}_{{\mathbf{z}}})\in\partial\Theta({\mathbf{X}}^{t},{\mathbf{X}}^{t-1},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\beta^{t},\beta^{t-1},\mu^{t}),$ (65) $$

where ${\mathbf{d}}_{{\mathbf{X}}}\triangleq\mathbb{A}^{t}+\{\beta^{t}+2v_{1}\tfrac{(\beta^{t-1})^{2}}{\beta^{t}}\}\cdot{\rm{mat}}([\nabla{\mathcal{A}}({\mathbf{X}}^{t})]^{\mathsf{T}}({\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}))+2v_{2}\ell(\beta^{t})({\mathbf{X}}^{t}-{\mathbf{X}}^{t-1})$, ${\mathbf{d}}_{{\mathbf{X}}^{-}}\triangleq-2v_{2}\ell(\beta^{t})({\mathbf{X}}^{t}-{\mathbf{X}}^{t-1})$, ${\mathbf{d}}_{{\mathbf{y}}}\triangleq\nabla h_{\mu^{t}}({\mathbf{y}}^{t})-{\mathbf{z}}^{t}+({\mathbf{y}}^{t}-{\mathcal{A}}({\mathbf{X}}^{t}))\cdot(\beta^{t}+2v_{1}\tfrac{(\beta^{t-1})^{2}}{\beta^{t}})$, ${\mathbf{d}}_{{\mathbf{z}}}\triangleq{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}$. Here, $\mathbb{A}^{t}\triangleq\partial I_{\mathcal{M}}({\mathbf{X}}^{t})+\nabla f({\mathbf{X}}^{t})-\nabla g({\mathbf{X}}^{t})+{\rm{mat}}([\nabla{\mathcal{A}}({\mathbf{X}}^{t})]^{\mathsf{T}}{\mathbf{z}}^{t})$.

###### Proof.

These results are based on basic deductions.
∎

###### Lemma E.2 .

For OADMM-EP, we define $\{{\mathbf{d}}_{{\mathbf{X}}},{\mathbf{d}}_{{\mathbf{X}}^{-}},{\mathbf{d}}_{{\mathbf{y}}},{\mathbf{d}}_{{\mathbf{z}}}\}$ as in Lemma [E.1](#A5.Thmtheorem1). It holds that:

$$ $\displaystyle\textstyle\tfrac{1}{\beta^{t}}\|{\mathbf{d}}_{{\mathbf{X}}}\|_{\mathsf{F}}\leq$ $\displaystyle~{}\textstyle r_{1}\mathcal{X}^{t}+r_{2}\mathcal{Z}^{t}+r_{3}\mathcal{X}^{t-1}+r_{4}\mathcal{Z}^{t-1},$ $\displaystyle\textstyle\tfrac{1}{\beta^{t}}\|{\mathbf{d}}_{{\mathbf{X}}^{-}}\|_{\mathsf{F}}\leq$ $\displaystyle~{}\textstyle r_{5}\mathcal{X}^{t},$ $\displaystyle\textstyle\tfrac{1}{\beta^{t}}\|{\mathbf{d}}_{{\mathbf{y}}}\|\leq$ $\displaystyle~{}\textstyle r_{6}\mathcal{Y}^{t+1}+r_{7}\mathcal{Z}^{t+1}+r_{8}\mathcal{Z}^{t},$ $\displaystyle\textstyle\tfrac{1}{\beta^{t}}\|{\mathbf{d}}_{{\mathbf{z}}}\|\leq$ $\displaystyle~{}\textstyle r_{9}\mathcal{Z}^{t}.$ $$

Here, $\mathcal{X}^{t}\triangleq\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}$, $\mathcal{Y}^{t}\triangleq\|{\mathbf{y}}^{t}-{\mathbf{y}}^{t-1}\|$, $\mathcal{Z}^{t}\triangleq\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|$.
Also, we define: $r_{1}\triangleq L_{g}/\beta^{0}+L_{f}/\beta^{0}+\theta\overline{\rm{L}}+L_{A}\overline{\rm{z}}/\beta^{0}+2v_{2}\overline{\rm{L}}$, $r_{2}\triangleq\overline{\rm{A}}\sigma+(1+2v_{1})C_{A}$, $r_{3}\triangleq L_{f}/\beta^{0}+\theta\overline{\rm{L}}+L_{A}\overline{z}/\beta^{0}+\overline{\rm{A}}C_{A}$, $r_{4}\triangleq\overline{\rm{A}}$, $r_{5}\triangleq 2v_{2}\overline{\rm{L}}$, $r_{6}\triangleq\chi$, $r_{7}=1$; $r_{8}\triangleq 1+2v_{1}$, $r_{9}\triangleq\tfrac{1}{\beta^{0}}$.

###### Proof.

We define $\mathbb{A}^{t}\triangleq\partial I_{\mathcal{M}}({\mathbf{X}}^{t})+\nabla f({\mathbf{X}}^{t})-\nabla g({\mathbf{X}}^{t})+{\rm{mat}}([\nabla{\mathcal{A}}({\mathbf{X}}^{t})]^{\mathsf{T}}{\mathbf{z}}^{t})$.

We define $r_{1}^{\prime}\triangleq\tfrac{L_{g}}{\beta^{0}}+\tfrac{L_{f}}{\beta^{0}}+\theta\overline{\rm{L}}+\tfrac{\overline{\rm{z}}L_{A}}{\beta^{0}}$, $r_{2}^{\prime}\triangleq\overline{\rm{A}}\sigma$, $r_{3}^{\prime}\triangleq\tfrac{L_{f}}{\beta^{0}}+\theta\overline{\rm{L}}+\tfrac{\overline{\rm{z}}L_{A}}{\beta^{0}}+\overline{\rm{A}}C_{A}$, and $r_{4}^{\prime}\triangleq\overline{\rm{A}}$.

First, using the optimality of ${\mathbf{X}}^{t+1}$ for OADMM-EP, we have:

$$ $\displaystyle\textstyle\partial I_{\mathcal{M}}({\mathbf{X}}^{t+1})$ $\displaystyle\ni$ $\displaystyle\textstyle-\theta\ell(\beta^{t})({\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}_{{\sf c}})+\nabla g({\mathbf{X}}^{t})-\nabla\mathcal{S}({\mathbf{X}}^{t}_{{\sf c}};\beta^{t})$ $\displaystyle=$ $\displaystyle-\theta\ell(\beta^{t})({\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}_{{\sf c}})+\nabla g({\mathbf{X}}^{t})-\nabla f({\mathbf{X}}^{t}_{{\sf c}})-{\rm{mat}}([\nabla{\mathcal{A}}({\mathbf{X}}^{t}_{{\sf c}})]^{\mathsf{T}}[{\mathbf{z}}^{t}+\beta^{t}({\mathcal{A}}({\mathbf{X}}^{t}_{{\sf c}})-{\mathbf{y}}^{t})]).$ $$

Second, we obtain:

$$ $\displaystyle\tfrac{1}{\beta^{t}}\|\mathbb{A}^{t}\|_{\mathsf{F}}$ (67) $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\tfrac{1}{\beta^{t}}\|\nabla g({\mathbf{X}}^{t-1})-\nabla g({\mathbf{X}}^{t})+\nabla f({\mathbf{X}}^{t})-\nabla f({\mathbf{X}}^{t-1}_{{\sf c}})-\theta\ell(\beta^{t-1})({\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}_{{\sf c}})$ $\displaystyle+{\rm{mat}}([\nabla{\mathcal{A}}({\mathbf{X}}^{t})]^{\mathsf{T}}{\mathbf{z}}^{t}-[\nabla{\mathcal{A}}({\mathbf{X}}^{t-1}_{{\sf c}})]^{\mathsf{T}}{\mathbf{z}}^{t-1}-\beta^{t-1}\nabla{\mathcal{A}}({\mathbf{X}}^{t-1}_{{\sf c}})^{\mathsf{T}}({\mathcal{A}}({\mathbf{X}}^{t-1}_{{\sf c}})-{\mathbf{y}}^{t-1}))\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\tfrac{1}{\beta^{t}}L_{g}\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}+\tfrac{1}{\beta^{t}}(L_{f}+\theta\ell(\beta^{t-1}))\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}_{{\sf c}}\|_{\mathsf{F}}$ $\displaystyle+\tfrac{1}{\beta^{t}}\overline{\rm{A}}\|{\mathbf{z}}^{t}-{\mathbf{z}}^{t-1}\|+\tfrac{1}{\beta^{t}}\|{\mathbf{z}}^{t}\|\cdot L_{A}\cdot\{\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}+\|{\mathbf{X}}^{t-1}-{\mathbf{X}}^{t-2}\|_{\mathsf{F}}\}$ $\displaystyle+\tfrac{1}{\beta^{t}}\beta^{t-1}\overline{\rm{A}}\{\|{\mathcal{A}}({\mathbf{X}}^{t-1})-{\mathbf{y}}^{t-1}\|+C_{A}\|{\mathbf{X}}^{t-1}-{\mathbf{X}}^{t-2}\|_{\mathsf{F}}\}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\tfrac{L_{g}}{\beta^{0}}\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}+(\tfrac{L_{f}}{\beta^{0}}+\theta\overline{\rm{L}})\{\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}+\|{\mathbf{X}}^{t-1}-{\mathbf{X}}^{t-2}\|_{\mathsf{F}}\}$ $\displaystyle+\overline{\rm{A}}\sigma\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|+\tfrac{\overline{\rm{z}}L_{A}}{\beta^{0}}\cdot\{\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}+\|{\mathbf{X}}^{t-1}-{\mathbf{X}}^{t-2}\|_{\mathsf{F}}\}$ $\displaystyle+\overline{\rm{A}}\{\|{\mathcal{A}}({\mathbf{X}}^{t-1})-{\mathbf{y}}^{t-1}\|+C_{A}\|{\mathbf{X}}^{t-1}-{\mathbf{X}}^{t-2}\|_{\mathsf{F}}\}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle r^{\prime}_{1}\mathcal{X}^{t}+r^{\prime}_{2}\mathcal{Z}^{t}+r^{\prime}_{3}\mathcal{X}^{t-1}+r^{\prime}_{4}\mathcal{Z}^{t-1},$ $$

where step ① uses Equality ([E.2](#A5.Ex339)); step ② uses the triangle inequality, the $L_{f}$-Lipschitz continuity of $\nabla f({\mathbf{X}})$ for all ${\mathbf{X}}$; the $L_{g}$-Lipschitz continuity of $\nabla g({\mathbf{X}})$, the upper bound of $\|[\nabla{\mathcal{A}}({\mathbf{X}}^{t})]^{\mathsf{T}}{\mathbf{z}}^{t}-[\nabla{\mathcal{A}}({\mathbf{X}}^{t-1}_{{\sf c}})]^{\mathsf{T}}{\mathbf{z}}^{t-1}\|_{\mathsf{F}}$ as shown in Claim (e) of Lemma [A.4](#A1.Thmtheorem4), the upper bound of $\|\nabla{\mathcal{A}}({\mathbf{X}}^{t-1}_{{\sf c}})\|_{\mathsf{F}}$ as shown in Claim (d) of Lemma [A.4](#A1.Thmtheorem4), and the upper bound of $\|{\mathcal{A}}({\mathbf{X}}_{{\sf c}}^{t})-{\mathbf{y}}^{t}\|$ as shown in Claim (c) of Lemma [A.4](#A1.Thmtheorem4); step ③ uses the upper bound of $\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}_{{\sf c}}\|_{\mathsf{F}}$; the upper bound of $\|{\mathbf{z}}^{t}\|$, and ${\mathbf{z}}^{t}-{\mathbf{z}}^{t-1}=\sigma\beta^{t-1}({\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t})$.

Third, using the optimality of ${\mathbf{y}}^{t}$, we have:

$$ $\displaystyle\mathbf{0}$ $\displaystyle=$ $\displaystyle\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})-{\mathbf{z}}^{t}+\beta^{t}({\mathbf{y}}^{t+1}-{\mathbf{y}}_{{\sf c}}^{t})$ (68) $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})-{\mathbf{z}}^{t}+\beta^{t}({\mathbf{y}}_{{\sf c}}^{t}-{\mathcal{A}}({\mathbf{X}}^{t+1}))+\beta^{t}({\mathbf{y}}^{t+1}-{\mathbf{y}}_{{\sf c}}^{t})$ $\displaystyle=$ $\displaystyle\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})-{\mathbf{z}}^{t}+\beta^{t}({\mathbf{y}}^{t+1}-{\mathcal{A}}({\mathbf{X}}^{t+1})),$ $$

where step ① uses $\nabla_{{\mathbf{y}}}\mathcal{S}^{t}({\mathbf{X}}^{t+1},{\mathbf{y}})=-{\mathbf{z}}^{t}+\beta^{t}({\mathbf{y}}-{\mathcal{A}}({\mathbf{X}}^{t+1}))$; step ② uses ${\mathbf{z}}^{t+1}={\mathbf{z}}^{t}+\sigma\beta^{t}({\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1})$.

(a) We bound the term $\tfrac{1}{\beta^{t}}\|{\mathbf{d}}_{{\mathbf{X}}}\|_{\mathsf{F}}$. We have:

$$ $\displaystyle\tfrac{1}{\beta^{t}}\|{\mathbf{d}}_{{\mathbf{X}}}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\textstyle\tfrac{1}{\beta^{t}}\|\mathbb{A}^{t}+(\beta^{t}+2v_{1}\tfrac{(\beta^{t-1})^{2}}{\beta^{t}})\cdot{\rm{mat}}([\nabla{\mathcal{A}}({\mathbf{X}}^{t})]^{\mathsf{T}}({\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}))+2v_{2}\ell(\beta^{t})({\mathbf{X}}^{t}-{\mathbf{X}}^{t-1})\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\textstyle\tfrac{1}{\beta^{t}}\|\mathbb{A}^{t}\|_{\mathsf{F}}+(1+2v_{1})\cdot\|[\nabla{\mathcal{A}}({\mathbf{X}}^{t})]^{\mathsf{T}}({\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t})\|_{\mathsf{F}}+2v_{2}\overline{\rm{L}}\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\textstyle r^{\prime}_{1}\mathcal{X}^{t}+r^{\prime}_{2}\mathcal{Z}^{t}+r^{\prime}_{3}\mathcal{X}^{t-1}+r^{\prime}_{4}\mathcal{Z}^{t-1}+(1+2v_{1})C_{A}\mathcal{Z}^{t}+2v_{2}\overline{\rm{L}}\mathcal{X}^{t},$ $\displaystyle\overset{\text{\char 175}}{=}$ $\displaystyle\textstyle r_{1}\mathcal{X}^{t}+r_{2}\mathcal{Z}^{t}+r_{3}\mathcal{X}^{t-1}+r_{4}\mathcal{Z}^{t-1},$ $$

where step ① uses the definition of ${\mathbf{d}}_{{\mathbf{X}}}$ in Lemma [E.1](#A5.Thmtheorem1); step ② uses the triangle inequality, $\beta^{t-1}\leq\beta^{t}$, and $\ell(\beta^{t})\leq\beta^{t}\overline{\rm{L}}$; step ③ uses Inequality ([67](#A5.E67)), the $C_{A}$-Lipschitz continuity of ${\mathcal{A}}({\mathbf{X}})$; step ④ uses the definitions of $\{r_{1},r_{2},r_{3},r_{4}\}$.

(b) We bound the term $\tfrac{1}{\beta^{t}}\|{\mathbf{d}}_{{\mathbf{X}}^{-}}\|_{\mathsf{F}}$. We have:

$$ $\displaystyle\tfrac{1}{\beta^{t}}\|{\mathbf{d}}_{{\mathbf{X}}^{-}}\|_{\mathsf{F}}\overset{\text{\char 172}}{=}\tfrac{1}{\beta^{t}}\|-2v_{2}\ell(\beta^{t})({\mathbf{X}}^{t}-{\mathbf{X}}^{t-1})\|_{\mathsf{F}}\overset{\text{\char 173}}{\leq}2v_{2}\overline{\rm{L}}\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}}=2v_{2}\overline{\rm{L}}\mathcal{X}^{t},$ (69) $$

where step ① uses the definition of ${\mathbf{d}}_{{\mathbf{X}}^{-}}$ in Lemma [E.1](#A5.Thmtheorem1); step ② uses $\ell(\beta^{t})\leq\beta^{t}\overline{\rm{L}}$.

(c) We bound the term $\tfrac{1}{\beta^{t}}\|{\mathbf{d}}_{{\mathbf{y}}}\|_{\mathsf{F}}$. We have:

$$ $\displaystyle\tfrac{1}{\beta^{t}}\|{\mathbf{d}}_{{\mathbf{y}}}\|$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\tfrac{1}{\beta^{t}}\|\nabla h_{\mu^{t}}({\mathbf{y}}^{t})-{\mathbf{z}}^{t}+({\mathbf{y}}^{t}-{\mathcal{A}}({\mathbf{X}}^{t}))\cdot(\beta^{t}+2v_{1}\tfrac{(\beta^{t-1})^{2}}{\beta^{t}})\|$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\tfrac{1}{\beta^{t}}\|\nabla h_{\mu^{t}}({\mathbf{y}}^{t})-{\mathbf{z}}^{t}\|+(1+2v_{1})\|({\mathbf{y}}^{t}-{\mathcal{A}}({\mathbf{X}}^{t})\|$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\tfrac{1}{\beta^{t}}\|\nabla h_{\mu^{t}}({\mathbf{y}}^{t})-\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})\|+\tfrac{1}{\beta^{t}}\|\nabla h_{\mu^{t}}({\mathbf{y}}^{t+1})-{\mathbf{z}}^{t}\|+(1+2v_{1})\mathcal{Z}^{t}$ $\displaystyle\overset{\text{\char 175}}{\leq}$ $\displaystyle\tfrac{1}{\beta^{t}}\cdot\tfrac{1}{\mu^{t}}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|+\|{\mathbf{y}}^{t+1}-{\mathcal{A}}({\mathbf{X}}^{t+1})\|+(1+2v_{1})\mathcal{Z}^{t}$ $\displaystyle\overset{\text{\char 176}}{\leq}$ $\displaystyle\chi\mathcal{Y}^{t+1}+\mathcal{Z}^{t+1}+(1+2v_{1})\mathcal{Z}^{t},$ $$

where step ① uses the definition of ${\mathbf{d}}_{{\mathbf{y}}}$ in Lemma [E.1](#A5.Thmtheorem1); step ② uses the triangle inequality, and $\beta^{t-1}\leq\beta^{t}$; step ③ uses the triangle inequality; step ④ uses the fact that $\nabla h_{\mu^{t}}({\mathbf{y}})$ is $\tfrac{1}{\mu^{t}}$-smooth for all ${\mathbf{y}}$, and Equality ([68](#A5.E68)); step ⑤ uses the choice $\mu^{t}=\tfrac{\chi}{\beta^{t}}$, and $\tfrac{1}{\beta^{t}}({\mathbf{z}}^{t+1}-{\mathbf{z}}^{t})=\sigma({\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1})$.

(d) We bound the term $\tfrac{1}{\beta^{t}}\|{\mathbf{d}}_{{\mathbf{z}}}\|_{\mathsf{F}}$. We have:

$$ $\displaystyle\tfrac{1}{\beta^{t}}\|{\mathbf{d}}_{{\mathbf{z}}}\|=\tfrac{1}{\beta^{t}}\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|\leq\tfrac{1}{\beta^{0}}\mathcal{Z}^{t}.$ $$

∎

###### Lemma E.3 .

For OADMM-RR, we define $\{{\mathbf{d}}_{{\mathbf{X}}},{\mathbf{d}}_{{\mathbf{X}}^{-}},{\mathbf{d}}_{{\mathbf{y}}},{\mathbf{d}}_{{\mathbf{z}}}\}$ as in Lemma [E.1](#A5.Thmtheorem1). It holds that:

$$ $\displaystyle\textstyle\tfrac{1}{\beta^{t}}\|{\mathbf{d}}_{{\mathbf{X}}}\|_{\mathsf{F}}\leq$ $\displaystyle~{}\textstyle r_{1}\mathcal{X}^{t}+r_{2}\mathcal{Z}^{t}+r_{3}\mathcal{X}^{t-1}+r_{4}\mathcal{Z}^{t-1},$ $\displaystyle\textstyle\tfrac{1}{\beta^{t}}\|{\mathbf{d}}_{{\mathbf{X}}^{-}}\|_{\mathsf{F}}\leq$ $\displaystyle~{}\textstyle r_{5}\mathcal{X}^{t},$ $\displaystyle\textstyle\tfrac{1}{\beta^{t}}\|{\mathbf{d}}_{{\mathbf{y}}}\|\leq$ $\displaystyle~{}\textstyle r_{6}\mathcal{Y}^{t}+r_{7}\mathcal{Z}^{t+1}+r_{8}\mathcal{Z}^{t},$ $\displaystyle\textstyle\tfrac{1}{\beta^{t}}\|{\mathbf{d}}_{{\mathbf{z}}}\|\leq$ $\displaystyle~{}\textstyle r_{9}\mathcal{Z}^{t}.$ $$

Here, $\mathcal{X}^{t}\triangleq\|\tfrac{1}{\beta^{t}}{\mathbf{D}}_{1/2}\|_{\mathsf{F}}$, $\mathcal{Y}^{t}\triangleq\|{\mathbf{y}}^{t}-{\mathbf{y}}^{t-1}\|$, $\mathcal{Z}^{t}\triangleq\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|$.
Also, we define: $r_{1}=2$, $r_{2}=v_{1}C_{A}$, $r_{3}=0$, $r_{4}=0$, $r_{5}=0$.
Notably, the parameters $\{r_{6},r_{7},r_{8},r_{9}\}$ are defined in Lemma [E.2](#A5.Thmtheorem2)

###### Proof.

We define $\mathbf{G}^{t}\triangleq\nabla f({\mathbf{X}}^{t})-\nabla g({\mathbf{X}}^{t})+{\rm{mat}}([\nabla A({\mathbf{X}}^{t})]^{\mathsf{T}}({\mathbf{z}}^{t}+\beta^{t}({\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t})))$.

We define $\dot{\mathcal{L}}({\mathbf{X}})\triangleq L({\mathbf{X}},{\mathbf{y}}^{t},{\mathbf{z}}^{t};\beta^{t},\mu^{t})$, we have: $\nabla\dot{\mathcal{L}}({\mathbf{X}}^{t})=\mathbf{G}^{t}$.

Second, for all ${\mathbf{X}}^{t}\in\mathcal{M}$, we obtain:

$$ $\displaystyle\textstyle\|\partial I_{\mathcal{M}}({\mathbf{X}}^{t})+\nabla\dot{\mathcal{L}}({\mathbf{X}}^{t})\|_{\mathsf{F}}$ (70) $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\textstyle\|\nabla\dot{\mathcal{L}}({\mathbf{X}}^{t})-{\mathbf{X}}^{t}[\nabla\dot{\mathcal{L}}({\mathbf{X}}^{t})]^{\mathsf{T}}{\mathbf{X}}^{t}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle\textstyle\|\mathbf{G}^{t}-{\mathbf{X}}^{t}[\mathbf{G}^{t}]^{\mathsf{T}}{\mathbf{X}}^{t}\|_{\mathsf{F}}=\|{\mathbf{D}}^{t}_{1}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\textstyle 2\cdot\|{\mathbf{D}}_{1/2}\|_{\mathsf{F}}=2\mathcal{X}^{t}\beta^{t}$ $$

where step ① uses Lemma [2.12](#S2.Thmtheorem12); step ② uses the definitions of $\{\mathbf{G}^{t},\mathbf{D}_{\rho}^{t}\}$ as in Algorithm [1](#alg1); step ③ uses Claim (d) of Lemma [2.11](#S2.Thmtheorem11) that:$\|{\mathbf{D}}_{1}\|_{\mathsf{F}}\leq\max(1,1/\rho)\|{\mathbf{D}}_{\rho}\|_{\mathsf{F}}$.

(a) We bound the term $\tfrac{1}{\beta^{t}}\|{\mathbf{d}}_{{\mathbf{X}}}\|_{\mathsf{F}}$. We have:

$$ $\displaystyle\tfrac{1}{\beta^{t}}\|{\mathbf{d}}_{{\mathbf{X}}}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\textstyle\tfrac{1}{\beta^{t}}\|\partial I_{\mathcal{M}}({\mathbf{X}}^{t})+\nabla\dot{\mathcal{L}}({\mathbf{X}}^{t})+v_{1}\tfrac{(\beta^{t-1})^{2}}{\beta^{t}}\cdot{\rm{mat}}([\nabla{\mathcal{A}}({\mathbf{X}}^{t})]^{\mathsf{T}}({\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}))\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\textstyle\tfrac{1}{\beta^{t}}\|\partial I_{\mathcal{M}}({\mathbf{X}}^{t})+\nabla\dot{\mathcal{L}}({\mathbf{X}}^{t})\|_{\mathsf{F}}+v_{1}\|[\nabla{\mathcal{A}}({\mathbf{X}}^{t})]^{\mathsf{T}}({\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t})\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\textstyle 2\mathcal{X}^{t}+v_{1}C_{A}\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|$ $\displaystyle\overset{}{=}$ $\displaystyle\textstyle r_{1}\mathcal{X}^{t}+r_{2}\mathcal{Z}^{t},$ $$

where step ① uses ${\mathbf{d}}_{{\mathbf{X}}}=\partial I_{\mathcal{M}}({\mathbf{X}}^{t})+\nabla f({\mathbf{X}}^{t})-\nabla g({\mathbf{X}}^{t})+{\rm{mat}}([\nabla{\mathcal{A}}({\mathbf{X}}^{t})]^{\mathsf{T}}{\mathbf{z}}^{t})+\{\beta^{t}+2v_{1}\tfrac{(\beta^{t-1})^{2}}{\beta^{t}}\}\cdot{\rm{mat}}([\nabla{\mathcal{A}}({\mathbf{X}}^{t})]^{\mathsf{T}}({\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}))$ with the choice $\alpha_{x}=0$ and $v_{2}=0$ for OADMM-RR; step ② uses the triangle inequality and $\beta^{t-1}\leq\beta^{t}$; step ② uses Inequality ([70](#A5.E70)), and the $C_{A}$-Lipschitz continuity of ${\mathcal{A}}({\mathbf{X}})$.

(b) We bound the term $\tfrac{1}{\beta^{t}}\|{\mathbf{d}}_{{\mathbf{X}}^{-}}\|_{\mathsf{F}}$. Given $v_{2}=0$, we conclude that $\tfrac{1}{\beta^{t}}\|{\mathbf{d}}_{{\mathbf{X}}^{-}}\|_{\mathsf{F}}=0$.

(c) We bound the terms $\tfrac{1}{\beta^{t}}\|{\mathbf{d}}_{{\mathbf{y}}}\|_{\mathsf{F}}$ and $\tfrac{1}{\beta^{t}}\|{\mathbf{d}}_{{\mathbf{z}}}\|_{\mathsf{F}}$. Considering that the same strategies for updating $\{{\mathbf{y}}^{t},{\mathbf{z}}^{t}\}$ are employed, their bounds in OADMM-RR are identical to those in OADMM-ER.

∎

Now, we proceed to prove the main result of this lemma.

###### Lemma E.4 .

(Subgradient Bounds for both OADMM-EP and OADMM-RR ) We define ${\mathbbm{w}}\triangleq\{{\mathbf{X}},{\mathbf{X}}^{-},{\mathbf{y}},{\mathbf{z}}\}$ and ${\mathbbm{u}}\triangleq\{\beta,\beta^{-},\mu,t\}$. We let ${\mathbbm{w}}^{t}\triangleq\{{\mathbf{X}}^{t},{\mathbf{X}}^{t-1},{\mathbf{y}}^{t},{\mathbf{z}}^{t}\}$ and ${\mathbbm{u}}^{t}\triangleq\{\beta^{t},\beta^{t-1},\mu^{t},t\}$. We have: ${\rm{dist}}(\mathbf{0},\partial\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t}))\leq\beta^{t}K(e_{t+1}+e_{t}+e_{t-1})$, where

$$ $\displaystyle e_{t}\triangleq\left\{\begin{array}[]{ll}\|{\mathbf{y}}^{t}-{\mathbf{y}}^{t-1}\|_{2}+\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|_{2}+\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}},&\hbox{{\sf OADMM-EP};}\\ \|{\mathbf{y}}^{t}-{\mathbf{y}}^{t-1}\|_{2}+\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|_{2}+\|\tfrac{1}{\beta^{t}}{\mathbf{D}}^{t-1}_{1/2}\|_{\mathsf{F}},&\hbox{{\sf OADMM-RR}.}\end{array}\right.$ (73) $$

Here, $K\triangleq\max(\sum_{i=1}^{9}r_{i})$, where $\{r_{i}\}_{i=1}^{9}$ is defined in Lemma [E.2](#A5.Thmtheorem2) or Lemma [E.3](#A5.Thmtheorem3).
Additionally, ${\rm{dist}}^{2}(\mathbf{0},\partial\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t}))\triangleq{\rm{dist}}^{2}(\mathbf{0},\partial_{{\mathbf{X}}}\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t}))+{\rm{dist}}^{2}(\mathbf{0},\partial_{{\mathbf{X}}^{-}}\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t}))+{\rm{dist}}^{2}(\mathbf{0},\partial_{{\mathbf{y}}}\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t}))+{\rm{dist}}^{2}(\mathbf{0},\partial_{{\mathbf{z}}}\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t}))$.

###### Proof.

Using the denotations, we have: ${\rm{dist}}(\mathbf{0},\partial_{{\mathbf{X}}}\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t}))=\|{\mathbf{d}}_{{\mathbf{X}}}\|_{\mathsf{F}}$, ${\rm{dist}}(\mathbf{0},\partial_{{\mathbf{X}}^{-}}\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t}))=\|{\mathbf{d}}_{{\mathbf{X}}^{-}}\|_{\mathsf{F}}$, ${\rm{dist}}(\mathbf{0},\partial_{{\mathbf{y}}}\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t}))=\|{\mathbf{d}}_{{\mathbf{y}}}\|$, and ${\rm{dist}}(\mathbf{0},\partial_{{\mathbf{z}}}\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t}))=\|{\mathbf{d}}_{{\mathbf{z}}}\|$.

For both OADMM-EP and OADMM-RR, we have:

$$ $\displaystyle{\rm{dist}}(\mathbf{0},\partial\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t}))$ $\displaystyle=$ $\displaystyle\sqrt{\|{\mathbf{d}}_{{\mathbf{X}}}\|_{\mathsf{F}}^{2}+\|{\mathbf{d}}_{{\mathbf{X}}^{-}}\|_{\mathsf{F}}^{2}+\|{\mathbf{d}}_{{\mathbf{y}}}\|_{\mathsf{F}}^{2}+\|{\mathbf{d}}_{{\mathbf{z}}}\|_{\mathsf{F}}^{2}}$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\|{\mathbf{d}}_{{\mathbf{X}}}\|_{\mathsf{F}}+\|{\mathbf{d}}_{{\mathbf{X}}^{-}}\|_{\mathsf{F}}+\|{\mathbf{d}}_{{\mathbf{y}}}\|_{\mathsf{F}}+\|{\mathbf{d}}_{{\mathbf{z}}}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle r_{1}\mathcal{X}^{t}+r_{2}\mathcal{Z}^{t}+r_{3}\mathcal{X}^{t-1}+r_{4}\mathcal{Z}^{t-1}+r_{5}\mathcal{X}^{t}+r_{6}\mathcal{Y}^{t+1}+r_{7}\mathcal{Z}^{t+1}+r_{8}\mathcal{Z}^{t}+r_{9}\mathcal{Z}^{t}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle K_{1}\{\underbrace{\mathcal{X}^{t+1}+\mathcal{Y}^{t+1}+\mathcal{Z}^{t+1}}_{\triangleq e_{t+1}}+\underbrace{\mathcal{X}^{t}+\mathcal{Y}^{t}+\mathcal{Z}^{t}}_{\triangleq e_{t+1}}+\underbrace{\mathcal{X}^{t-1}+\mathcal{Y}^{t-1}+\mathcal{Z}^{t-1}}_{\triangleq e_{t-1}}\},$ $$

where step ① uses $\sqrt{a+b}\leq\sqrt{a}+\sqrt{b}$ for all $a\geq 0$ and $b\geq 0$; step ② uses Lemma [E.2](#A5.Thmtheorem2); step ③ uses the definition of $K$.

∎

###### Lemma E.1 .

###### Proof.

###### Lemma E.2 .

###### Proof.

###### Lemma E.3 .

###### Proof.

###### Lemma E.4 .

###### Proof.

### E.2 Proof of Lemma 5.4

###### Proof.

We have:

$$ $\displaystyle\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\|{\sf{Retr}}_{{\mathbf{X}}^{t}}(\eta^{t}{\mathbf{D}}^{t}_{\rho})-{\mathbf{X}}^{t}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle k_{1}\|\eta^{t}{\mathbf{D}}^{t}_{\rho}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\max(2\rho,1)\|{\mathbf{D}}^{t}_{1/2}\|_{\mathsf{F}}\cdot k_{1}\eta^{t}$ $\displaystyle\overset{\text{\char 175}}{=}$ $\displaystyle\max(2\rho,1)\|{\mathbf{D}}^{t}_{1/2}\|_{\mathsf{F}}\cdot k_{1}\tfrac{\beta^{0}}{\beta^{t}}b^{t}\gamma^{j}$ $\displaystyle\overset{\text{\char 176}}{\leq}$ $\displaystyle\tfrac{1}{\beta^{t}}\|{\mathbf{D}}^{t}_{1/2}\|_{\mathsf{F}}\cdot\underbrace{k_{1}\beta^{0}\overline{b}\overline{\gamma}\max(2\rho,1)}_{\triangleq c_{0}},$ $$

where step ① uses the update rule of ${\mathbf{X}}^{t+1}$; step ② uses Lemma [2.9](#S2.Thmtheorem9); step ③ uses Claim (c) of Lemma [2.11](#S2.Thmtheorem11); step ④ uses the definition of $\eta^{t}\triangleq\tfrac{\beta^{0}}{\beta^{t}}b^{t}\gamma^{j}$; step ⑤ uses $b^{t}\leq\overline{b}$, and the fact that $\gamma^{j}\leq\overline{\gamma}$.

∎

###### Proof.

### E.3 Proof of Theorem 5.5

###### Proof.

We define

$$ $\displaystyle e_{t}\triangleq\left\{\begin{array}[]{ll}\|{\mathbf{y}}^{t}-{\mathbf{y}}^{t-1}\|+\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|+\|{\mathbf{X}}^{t}-{\mathbf{X}}^{t-1}\|_{\mathsf{F}},&\hbox{{\sf OADMM-EP};}\\ \|{\mathbf{y}}^{t}-{\mathbf{y}}^{t-1}\|+\|{\mathcal{A}}({\mathbf{X}}^{t})-{\mathbf{y}}^{t}\|+\|\tfrac{1}{\beta^{t}}{\mathbf{D}}^{t-1}_{1/2}\|_{\mathsf{F}},&\hbox{{\sf OADMM-RR}.}\end{array}\right.$ (76) $$

Firstly, using Assumption [5.1](#S5.Thmtheorem1), we have:

$$ $\displaystyle\varphi^{\prime}(\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t})-\Theta({\mathbbm{w}}^{\infty};{\mathbbm{u}}^{\infty}))\cdot{\rm{dist}}(\mathbf{0},\partial\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t}))\geq 1.$ (77) $$

Secondly, given the desingularization function $\varphi(\cdot)$ is concave, for any $a\in\mathbb{R}$ and $b\in\mathbb{R}$, we have: $\varphi(b)+(a-b)\varphi^{\prime}(a)\leq\varphi(a)$. Applying the inequality above with $a=\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t})-\Theta({\mathbbm{w}}^{\infty};{\mathbbm{u}}^{\infty})$ and $b=\Theta({\mathbbm{w}}^{t+1};{\mathbbm{u}}^{t+1})-\Theta({\mathbbm{w}}^{\infty};{\mathbbm{u}}^{\infty})$, we have:

$$ $\displaystyle\textstyle(\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t})-\Theta({\mathbbm{w}}^{t+1};{\mathbbm{u}}^{t+1}))\cdot\varphi^{\prime}(\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t})-\Theta({\mathbbm{w}}^{\infty};{\mathbbm{u}}^{\infty}))$ (78) $\displaystyle\leq$ $\displaystyle\textstyle\underbrace{\varphi(\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t})-\Theta({\mathbbm{w}}^{\infty};{\mathbbm{u}}^{\infty}))}_{\triangleq\varphi_{t}}-\underbrace{\varphi(\Theta({\mathbbm{w}}^{t+1};{\mathbbm{u}}^{t+1})-\Theta({\mathbbm{w}}^{\infty};{\mathbbm{u}}^{\infty}))}_{\triangleq\varphi_{t+1}}.$ $$

Third, we derive the following inequalities for OADMM-EP:

$$ $\displaystyle\min(\varepsilon_{z},\varepsilon_{y},\varepsilon_{x})\beta^{t}\{\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|_{2}^{2}+\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}+\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}^{2}\}$ (79) $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\varepsilon_{z}\beta^{t}\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|_{2}^{2}+\varepsilon_{y}\beta^{t}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}+\varepsilon_{x}\ell(\beta^{t})\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}^{2}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\Theta^{t}-\Theta^{t+1}=\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t})-\Theta({\mathbbm{w}}^{t+1};{\mathbbm{u}}^{t+1})$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\textstyle(\varphi_{t}-\varphi_{t+1})\cdot\tfrac{1}{\varphi^{\prime}(\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t})-\Theta({\mathbbm{w}}^{\infty};{\mathbbm{u}}^{\infty})))}$ $\displaystyle\overset{\text{\char 175}}{\leq}$ $\displaystyle\textstyle(\varphi_{t}-\varphi_{t+1})\cdot{\rm{dist}}(\mathbf{0},\partial\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t}))$ $\displaystyle\overset{\text{\char 176}}{\leq}$ $\displaystyle\textstyle(\varphi_{t}-\varphi_{t+1})\cdot K_{1}\beta^{t}(e_{t+1}+e_{t}+e_{t-1}),$ $$

where step ① uses $\ell(\beta^{t})\geq\beta^{t}\underline{\rm{L}}$; step ② uses Lemma [4.5](#S4.Thmtheorem5); step ③ uses Inequality ([78](#A5.E78)); step ④ uses Inequality ([77](#A5.E77)); step ⑤ uses Lemma [5.3](#S5.Thmtheorem3). We further derive the following inequalities:

$$ $\displaystyle\textstyle(e_{t+1})^{2}$ $\displaystyle\triangleq$ $\displaystyle(\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|_{2}+\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}+\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}})^{2}$ (80) $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\textstyle 3\cdot\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|_{2}^{2}+\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}+\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}^{2}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\textstyle\tfrac{3K_{1}}{\min(\varepsilon_{z},\varepsilon_{y},\varepsilon_{x})}\cdot(e_{t+1}+e_{t}+e_{t-1})\cdot(\varphi_{t}-\varphi_{t+1}),$ $$

where step ① uses the norm inequality that $(a+b+c)^{2}\leq 3(a^{2}+b^{2}+c^{2})$ for any $a,b,c\in\mathbb{R}$; step ② uses Inequality ([79](#A5.E79)).

Fourth, we derive the following inequalities for OADMM-RR:

$$ $\displaystyle\min(\varepsilon_{z},\varepsilon_{y},\varepsilon_{x})\beta^{t}\{\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|_{2}^{2}+\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}+\|\tfrac{1}{\beta}{\mathbf{D}}^{t}_{1/2}\|_{\mathsf{F}}^{2}\}$ (81) $\displaystyle\overset{}{\leq}$ $\displaystyle\varepsilon_{z}\beta^{t}\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|_{2}^{2}+\varepsilon_{y}\beta^{t}\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}+\tfrac{\varepsilon_{x}}{\beta}\|{\mathbf{D}}^{t}_{1/2}\|_{\mathsf{F}}^{2}$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\Theta^{t}-\Theta^{t+1}=\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t})-\Theta({\mathbbm{w}}^{t+1};{\mathbbm{u}}^{t+1})$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\textstyle(\varphi_{t}-\varphi_{t+1})\cdot\tfrac{1}{\varphi^{\prime}(\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t})-\Theta({\mathbbm{w}}^{\infty};{\mathbbm{u}}^{\infty})))}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\textstyle(\varphi_{t}-\varphi_{t+1})\cdot{\rm{dist}}(\mathbf{0},\partial\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t}))$ $\displaystyle\overset{\text{\char 175}}{\leq}$ $\displaystyle\textstyle(\varphi_{t}-\varphi_{t+1})\cdot K_{1}\beta^{t}(e_{t+1}+e_{t}+e_{t-1}),$ $$

where step ① uses Lemma [4.8](#S4.Thmtheorem8); step ② uses Inequality ([78](#A5.E78)); step ③ uses Inequality ([77](#A5.E77)); step ④ uses Lemma [5.3](#S5.Thmtheorem3). We further derive the following inequalities:

$$ $\displaystyle\textstyle(e_{t+1})^{2}$ $\displaystyle\triangleq$ $\displaystyle(\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|+\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|+\|\tfrac{1}{\beta}{\mathbf{D}}^{t}_{1/2}\|_{\mathsf{F}})^{2}$ (82) $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\textstyle 3\cdot\|{\mathcal{A}}({\mathbf{X}}^{t+1})-{\mathbf{y}}^{t+1}\|_{2}^{2}+\|{\mathbf{y}}^{t+1}-{\mathbf{y}}^{t}\|_{2}^{2}+\|\tfrac{1}{\beta}{\mathbf{D}}^{t}_{1/2}\|_{\mathsf{F}}^{2}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\textstyle\tfrac{3K_{1}}{\min(\varepsilon_{z},\varepsilon_{y},\varepsilon_{x})}\cdot(\varphi_{t}-\varphi_{t+1})\cdot(e_{t+1}+e_{t}+e_{t-1}),$ $$

where step ① uses the norm inequality that $(a+b+c)^{2}\leq 3(a^{2}+b^{2}+c^{2})$ for any $a,b,c\in\mathbb{R}$; step ② uses Inequality ([81](#A5.E81)).

Finally, considering Inequalities ([80](#A5.E80)) and ([82](#A5.E82)), we establish the following unified inequality applicable to both OADMM-EP and OADMM-RR:

$$ $\displaystyle e_{t+1}^{2}\leq(e_{t+1}+e_{t}+e_{t-1})\cdot\underbrace{\tfrac{3K_{1}}{\min(\varepsilon_{z},\varepsilon_{y},\varepsilon_{x})}}_{\triangleq K_{2}}\cdot(\varphi_{t}-\varphi_{t+1}).$ (83) $$

Considering Inequality ([83](#A5.E83)) and applying Lemma [A.8](#A1.Thmtheorem8) with $p_{t}\triangleq K_{2}\cdot\varphi_{t}$, for all $t\geq i$, we have:

$$ $\displaystyle\textstyle\sum_{i=t}^{\infty}e_{i+1}\leq e_{t}+e_{t-1}+7K_{2}\varphi_{t}.$ (84) $$

Letting $i=1$, we have: $\sum_{t=1}^{\infty}e_{t+1}\leq\textstyle e_{1}+e_{0}+7K_{2}\varphi_{1}\triangleq C_{{\sf e}}$.

∎

###### Proof.

### E.4 Proof of Theorem 5.6

Initially, we prove a lemma demonstrating that the convergence of $d^{t}$ is sufficient to establish the convergence of $\|{\mathbf{X}}^{t}-{\mathbf{X}}^{\infty}\|_{\mathsf{F}}$.

###### Lemma E.5 .

We define $d_{t}\triangleq\sum_{i=t}^{\infty}e_{i+1}$. For both OADMM-EP and OADMM-RR, we have: $\|{\mathbf{X}}^{t}-{\mathbf{X}}^{\infty}\|_{\mathsf{F}}\leq\max(c_{0},1)d_{t}$, where $c_{0}$ is a constant defined in Lemma [5.4](#S5.Thmtheorem4).

###### Proof.

For OADMM-EP, we have for all $t\geq 1$:

$$ $\displaystyle\textstyle\|{\mathbf{X}}^{t}-{\mathbf{X}}^{\infty}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\textstyle\sum_{i=t}^{\infty}\|{\mathbf{X}}^{i}-{\mathbf{X}}^{i+1}\|_{\mathsf{F}}$ $\displaystyle\leq$ $\displaystyle\textstyle\sum_{i=t}^{\infty}\{\|{\mathbf{X}}^{i+1}-{\mathbf{X}}^{i}\|_{\mathsf{F}}+\|{\mathbf{y}}^{i+1}-{\mathbf{y}}^{i}\|+\|{\mathcal{A}}({\mathbf{X}}^{i+1})-{\mathbf{y}}^{i+1}\|\}$ $\displaystyle=$ $\displaystyle\textstyle\sum_{i=t}^{\infty}e_{i+1}\triangleq d_{t},$ $$

where step ① use the triangle inequality.

For OADMM-RR, we have for all $t\geq 1$:

$$ $\displaystyle\textstyle\|{\mathbf{X}}^{t}-{\mathbf{X}}^{\infty}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\textstyle\sum_{i=t}^{\infty}\|{\mathbf{X}}^{i}-{\mathbf{X}}^{i+1}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\textstyle c_{0}\sum_{i=t}^{\infty}\|\tfrac{1}{\beta^{t}}{\mathbf{D}}_{1/2}^{i}\|_{\mathsf{F}}$ $\displaystyle\overset{}{\leq}$ $\displaystyle\textstyle c_{0}\sum_{i=t}^{\infty}\|{\mathbf{y}}^{i+1}-{\mathbf{y}}^{i}\|+\|{\mathcal{A}}({\mathbf{X}}^{i+1})-{\mathbf{y}}^{i+1}\|+\|\tfrac{1}{\beta^{i}}{\mathbf{D}}^{i}_{1/2}\|_{\mathsf{F}}$ $\displaystyle=$ $\displaystyle\textstyle c_{0}\sum_{i=t}^{\infty}e_{i+1}=c_{0}d_{t},$ $$

where step ① use the triangle inequality; step ② uses the Lemma [5.4](#S5.Thmtheorem4) that $\|{\mathbf{X}}^{t+1}-{\mathbf{X}}^{t}\|_{\mathsf{F}}\leq c_{0}\|\tfrac{1}{\beta^{t}}{\mathbf{D}}_{1/2}^{t}\|_{\mathsf{F}}$.
∎

Now, we proceed to prove the theorem.

###### Proof.

We define $d_{t}\triangleq\sum_{i=t}^{\infty}e_{i+1}$.

We define $\varphi_{t}\triangleq\varphi(s^{t})$, where $s^{t}\triangleq\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t})-\Theta({\mathbbm{w}}^{\infty};{\mathbbm{u}}^{\infty})$.

Using the definition of $d_{t}$, we derive:

$$ $\displaystyle\textstyle d_{t+1}$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\textstyle d_{t}\triangleq\sum_{i=t}^{\infty}e_{i+1}$ (85) $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\textstyle e_{t+1}+e_{t}+e_{t-1}+7K_{2}\varphi_{t}$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle\textstyle e_{t+1}+e_{t}+e_{t-1}+7K_{2}\tilde{c}\cdot\{[\varphi(s^{t})]^{\sigma}\}^{\frac{1-\tilde{\sigma}}{\tilde{\sigma}}}$ $\displaystyle\overset{\text{\char 174}}{=}$ $\displaystyle\textstyle e_{t+1}+e_{t}+e_{t-1}+7K_{2}\tilde{c}\cdot\{\tilde{c}(1-\tilde{\sigma})\cdot\frac{1}{\varphi^{\prime}(s^{t})}\}^{\frac{1-\tilde{\sigma}}{\tilde{\sigma}}}$ $\displaystyle\overset{\text{\char 175}}{\leq}$ $\displaystyle\textstyle e_{t+1}+e_{t}+e_{t-1}+7K_{2}\tilde{c}\cdot\{\tilde{c}(1-\tilde{\sigma})\cdot{\rm{dist}}(\mathbf{0},\partial\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t}))\}^{\frac{1-\tilde{\sigma}}{\tilde{\sigma}}}$ $\displaystyle\overset{\text{\char 176}}{\leq}$ $\displaystyle\textstyle e_{t+1}+e_{t}+e_{t-1}+7K_{2}\tilde{c}\cdot\{\tilde{c}(1-\tilde{\sigma})\cdot\beta^{t}K_{2}(e_{t+1}+e_{t}+e_{t-1})\}^{\frac{1-\tilde{\sigma}}{\tilde{\sigma}}}$ $\displaystyle\overset{\text{\char 177}}{=}$ $\displaystyle\textstyle d_{t-2}-d_{t+1}+7K_{2}\tilde{c}\cdot\{\tilde{c}(1-\tilde{\sigma})\cdot\beta^{t}K_{2}(d_{t-2}-d_{t+1})\})^{\frac{1-\tilde{\sigma}}{\tilde{\sigma}}}$ $\displaystyle\overset{}{=}$ $\displaystyle\textstyle d_{t-2}-d_{t+1}+\underbrace{7K_{2}\tilde{c}\cdot[\tilde{c}(1-\tilde{\sigma})K_{2}]^{\frac{1-\tilde{\sigma}}{\tilde{\sigma}}}}_{\triangleq K_{3}}\cdot(\beta^{t}(d_{t-2}-d_{t+1}))^{\frac{1-\tilde{\sigma}}{\tilde{\sigma}}},$ $$

where step ① uses Inequality ([84](#A5.E84)); step ② uses the definitions that $\varphi_{t}\triangleq\varphi(s^{t})$, $s^{t}\triangleq\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t})-\Theta({\mathbbm{w}}^{\infty};{\mathbbm{u}}^{\infty})$, and $\varphi(s)=\tilde{c}s^{1-\tilde{\sigma}}$; step ③ uses $\varphi^{\prime}(s)=\tilde{c}(1-\tilde{\sigma})(s)^{-\tilde{\sigma}}$; step ④ uses Assumption [77](#A5.E77) that $1\leq{\rm{dist}}(\mathbf{0},\partial\Theta({\mathbbm{w}}^{t};{\mathbbm{u}}^{t}))\cdot\varphi^{\prime}(s^{t})$; step ⑤ uses Lemma [5.3](#S5.Thmtheorem3) for both OADMM-EP and OADMM-RR; step ⑥ uses the fact that $e_{t}=d_{t-1}-d_{t+1}$, which implies:

$$ $\displaystyle e_{t+1}+e_{t}+e_{t-1}=(d_{t}-d_{t+1})+(d_{t-1}-d_{t})+(d_{t-2}-d_{t-1})=d_{t-2}-d_{t+1}.$ $$

We consider two cases for Inequality ([85](#A5.E85)).

(a) $\tilde{\sigma}\in(\frac{1}{4},\frac{1}{2}]$. We define $u\triangleq\frac{p(1-\tilde{\sigma})}{\tilde{\sigma}}\in(\tfrac{1}{3},1)\in(0,1)$, where $p=\tfrac{1}{3}$ is a fixed constant. We define $\tau\triangleq\frac{2\tilde{\sigma}-1}{1-\tilde{\sigma}}\in(0,\infty)$, and $\tilde{\beta}^{t}\triangleq K_{3}(\beta^{t})^{\frac{1-\tilde{\sigma}}{\tilde{\sigma}}}$. We define $t^{\prime}\triangleq\{i\,|\,d_{i-2}-d_{i}\leq 1\}$. For all $t\geq t^{\prime}$, we have from Inequality ([85](#A5.E85)):

$$ $\displaystyle d_{t+1}$ $\displaystyle\leq$ $\displaystyle d_{t-2}-d_{t+1}+K_{3}(\beta^{t})^{\frac{1-\tilde{\sigma}}{\tilde{\sigma}}}(d_{t-2}-d_{t+1})^{\frac{1-\tilde{\sigma}}{\tilde{\sigma}}}$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle d_{t-2}-d_{t+1}+\tilde{\beta}^{t}(d_{t-2}-d_{t+1})^{\frac{1-\tilde{\sigma}}{\tilde{\sigma}}}$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle d_{t-2}-d_{t+1}+\tilde{\beta}^{t}(d_{t-2}-d_{t+1})^{\tfrac{1}{1+\tau}}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle d_{t-2}-d_{t+1}+\tilde{\beta}^{t}(d_{t-2}-d_{t+1}),$ $$

where step ① uses the definition of $\tilde{\beta}^{t}$; step ② uses $\tau+1=\frac{2\tilde{\sigma}-1}{1-\tilde{\sigma}}+1=\tfrac{\tilde{\sigma}}{1-\tilde{\sigma}}$; step ② uses the fact that $x^{\nu}\leq x$ if $\nu=\tfrac{1}{1+\tau}\geq 0$ for all $x\in[0,1]$. We further obtain:

$$ $\displaystyle\tfrac{d_{t+1}}{d_{t-2}}\leq\tfrac{1+\tilde{\beta}^{t}}{2+\tilde{\beta}^{t}}.$ $$

Additionally, we have:

$$ $\displaystyle\textstyle\sum_{i=1}^{T}({1}/{\tilde{\beta}^{i}})\overset{\text{\char 172}}{=}\textstyle\mathcal{O}(\sum_{i=1}^{T}\{(i^{p})^{\frac{(\tilde{\sigma}-1)}{\tilde{\sigma}}}\})\overset{\text{\char 173}}{=}\textstyle\mathcal{O}(\sum_{i=1}^{T}i^{-u})\overset{\text{\char 174}}{\geq}\textstyle\mathcal{O}(T^{1-u}),$ $$

where step ① uses $\tilde{\beta}^{t}\triangleq K_{3}(\beta^{t})^{\frac{1-\tilde{\sigma}}{\tilde{\sigma}}}$ and $\beta^{t}\triangleq\beta^{0}(1+\xi t^{p})=\mathcal{O}(t^{p})$; step ② uses the definition of $u$ that $u\triangleq\frac{(1-\tilde{\sigma})p}{\tilde{\sigma}}\in(0,1)$; step ③ uses Lemma [A.7](#A1.Thmtheorem7) that: $\sum_{i=1}^{T}i^{-u}\geq(1-u)T^{1-u}=\mathcal{O}(T^{1-u})$ for all $u\in(0,1)$.

Applying Lemma Lemma [A.10](#A1.Thmtheorem10) with $a=1-u$, we have:

$$ $\displaystyle d_{T}\leq\mathcal{O}\left(\tfrac{1}{\exp(T^{1-u})}\right).$ $$

(b) $\tilde{\sigma}\in(\frac{1}{2},1)$. We define $w\triangleq\frac{1-\tilde{\sigma}}{\tilde{\sigma}}\in(0,1)$. We define $\tau\triangleq\frac{2\tilde{\sigma}-1}{1-\tilde{\sigma}}\in(0,\infty)$, and $\tilde{\beta}^{t}=K_{4}^{\tau+1}\beta^{t}$.

Notably, we have: $d_{t-2}-d_{t}\leq R\triangleq d_{0}$ for all $t\geq 2$. For all $t\geq 2$, we have from Inequality ([85](#A5.E85)):

$$ $\displaystyle d_{t}$ $\displaystyle\leq$ $\displaystyle d_{t-2}-d_{t+1}+K_{3}(\beta^{t})^{\frac{1-\tilde{\sigma}}{\tilde{\sigma}}}(d_{t-2}-d_{t+1})^{\frac{1-\tilde{\sigma}}{\tilde{\sigma}}}$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle K_{3}\{\beta^{t}(d_{t-2}-d_{t+1})\}^{w}+d_{t-2}-d_{t+1}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle K_{3}\{\beta^{t}(d_{t-2}-d_{t+1})\}^{w}+(d_{t-2}-d_{t+1})^{w}\cdot R^{1-w}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle K_{3}\{\beta^{t}(d_{t-2}-d_{t+1})\}^{w}+(d_{t-2}-d_{t+1})^{w}\cdot R^{1-w}\cdot(\tfrac{\beta^{t}}{\beta^{0}})^{w}$ $\displaystyle\overset{}{=}$ $\displaystyle\{\beta^{t}(d_{t-2}-d_{t+1})\}^{w}\cdot\underbrace{(K_{3}+R^{1-w}\cdot(\tfrac{1}{\beta^{0}})^{w})}_{\triangleq K_{4}},$ $$

where step ① uses the the definition of $w$; step ② uses the fact that $\max_{x\in(0,R)}\frac{x}{x^{w}}=R^{1-w}$ if $w\in(0,1)$; step ③ uses $\beta^{0}\leq\beta^{t}$ and $w\in(0,1)$. We further obtain:

$$ $\displaystyle\underbrace{d_{t}^{1/w}}_{=d_{t}^{\tau+1}}\leq(d_{t-2}-d_{t+1})\cdot\underbrace{\beta^{t}K_{4}^{1/w}}_{\triangleq\tilde{\beta}^{t}}.$ $$

Additionally, we have:

$$ $\displaystyle\textstyle\sum_{i=1}^{T}({1}/{\tilde{\beta}^{i}})\overset{\text{\char 172}}{=}\textstyle\mathcal{O}(\sum_{i=1}^{T}({1}/{{\beta}^{i}}))\overset{\text{\char 173}}{=}\textstyle\mathcal{O}(\sum_{i=1}^{T}i^{-p})\overset{\text{\char 174}}{\geq}\textstyle\mathcal{O}(T^{1-p}),$ $$

where step ① uses $\tilde{\beta}^{t}\triangleq K_{4}^{\tau+1}\beta^{t}$; step ② uses $\beta^{t}\triangleq\beta^{0}(1+\xi t^{p})=\mathcal{O}(t^{p})$; step ③ uses Lemma [A.7](#A1.Thmtheorem7) that: $\sum_{i=1}^{T}i^{-u}\geq(1-p)T^{1-u}=\mathcal{O}(T^{1-p})$ for all $p\in(0,1)$.

Applying Lemma Lemma [A.11](#A1.Thmtheorem11) with $a=1-p$, we have:

$$ $\displaystyle d_{T}\leq\mathcal{O}(1/(T^{(1-p)/\tau})).$ $$

∎

###### Lemma E.5 .

###### Proof.

###### Proof.

## Appendix F Applications and Numerical Experiments

In this section, we assess the effectiveness of the proposed algorithm OADMM on the sparse PCA problem by comparing it against existing non-convex, non-smooth optimization algorithms.

$\blacktriangleright$ Application to Sparse PCA. Sparse PCA is a method to produce modified principal components with sparse loadings, which helps reduce model complexity and increase model interpretation []. It can be formulated as:

$$ $\displaystyle\min_{{\mathbf{X}}\in\mathbb{R}^{n\times r}}\tfrac{1}{2m^{\prime}}\|{\mathbf{X}}{\mathbf{X}}^{\mathsf{T}}{\mathbf{D}}-{\mathbf{D}}\|_{\mathsf{F}}^{2}+\rho^{\prime}(\|{\mathbf{X}}\|_{1}-\|{\mathbf{X}}\|_{[k]}),$ $$

where ${\mathbf{D}}\in\mathbb{R}^{n\times m^{\prime}}$ is the data matrix, $m^{\prime}$ is the number of data points, and $\|{\mathbf{X}}\|_{[k]}$ is the $\ell_{1}$ norm the the $k$ largest (in magnitude) elements of the matrix ${\mathbf{X}}$. Here, we consider the DC $\ell_{1}$-largest-$k$ function [] to induce sparsity in the solution. One advantage of this model is that when $\rho^{\prime}$ is sufficient large, we have $\|{\mathbf{X}}\|_{1}\approx\|{\mathbf{X}}\|_{[k]}$, leading to a $k$-sparsity solution ${\mathbf{X}}$.

### F.1 Datasets

In our experiments, we utilize six datasets comprising both randomly generated and publicly available real-world data. These datasets are structured as data matrices $\mathbf{D}\in\mathbb{R}^{m^{\prime}\times d^{\prime}}$. They are denoted as follows: ‘CnnCaltech-$m^{\prime}$-$d^{\prime}$’, ‘TDT2-$m^{\prime}$-$d^{\prime}$’, ‘sector-$m^{\prime}$-$d^{\prime}$’, ‘mnist-$m^{\prime}$-$d^{\prime}$’, ‘randn-$m^{\prime}$-$d^{\prime}$’, and ‘dct-$m^{\prime}$-$d^{\prime}$’, where ${{\rm{randn}}(m,n)}$ generates a standard Gaussian random matrix of size $m\times n$, and $dct(m,n)$ produces a matrix sampled from the discrete cosine transform. The construction of $\mathbf{D}\in\mathbb{R}^{m^{\prime}\times d^{\prime}}$ involves randomly selecting $m^{\prime}$ examples and $d^{\prime}$ dimensions from the original real-world dataset, sourced from [http://www.cad.zju.edu.cn/home/dengcai/Data/TextData.html](http://www.cad.zju.edu.cn/home/dengcai/Data/TextData.html) and [https://www.csie.ntu.edu.tw/~cjlin/libsvm/](https://www.csie.ntu.edu.tw/~cjlin/libsvm/). Subsequently, we normalize each column of $\mathbf{D}$ to possess a unit norm and center the data by subtracting the mean, denoted as $\mathbf{D}\Leftarrow\mathbf{D}-\mathbf{1}\mathbf{1}^{\mathsf{T}}\mathbf{D}$.

$\blacktriangleright$ Experimental Settings. All methods are implemented in MATLAB on an Intel 2.6 GHz CPU with 64 GB RAM. For all retraction-based methods, we only consider the following retraction [, ]: ${\sf{Retr}}_{{\mathbf{X}}}(\mathbf{\Delta})=({\mathbf{X}}+\mathbf{\Delta})({\mathbf{I}}_{r}+\mathbf{\Delta}^{\mathsf{T}}\mathbf{\Delta})^{-1/2}$ with ${\mathbf{X}}\in\mathcal{M}$ and $\mathbf{\Delta}\in{\rm{T}}_{{\mathbf{X}}}\mathcal{M}$. For OADMM, we set $p=1/3$, $\xi=0.9$, $\theta=1.1$, $\sigma=1.5$, $\alpha_{x}=\tfrac{\theta-1}{2(\theta+1)(\xi+2)}$, $\alpha_{y}=0$, $\beta^{0}=100$, $\delta=10^{-12}$, $\rho=1/2$, and $\gamma=0.1$. Given that $W_{h}=0$ in this case, we select $\chi>\sqrt{4\sigma^{2}c_{0}/(2-\sigma)^{2}}$, where $c_{0}=1+\tfrac{\xi}{\sigma^{2}}+\tfrac{\xi}{2\sigma^{2}}$, in accordance with Assumption [3.3](#S3.Thmtheorem3). For simplicity, we do not use the Barzilai-Borwein strategy and only consider a fixed constant $b^{t}=\tfrac{1}{\beta^{0}}$ for all $t$. For ROADMM, we fix the penalty to a constant $\beta=10^{4}$. All algorithms begin with the common initial solution ${\mathbf{x}}^{0}$ drawn from a standard normal distribution. We provide our code in the supplemental material.

$\blacktriangleright$ Experiment Results. We report the objective values for different methods with varying parameters $\rho^{\prime}$ and $k^{\prime}$. The experimental results presented in Figure [1](#A6.F1), [2](#A6.F2), and [3](#A6.F3) reveal the following insights: (i) Sub-Grad is generally less efficient compared to the other methods. (ii) SPGM-EP and SPGM-RR, which rely on a variable smoothing strategy, exhibit slower performance than the multiplier-based variable splitting method. This observation aligns with the commonly accepted notion that primal-dual methods are generally more robust and faster than primal-only methods. (iii) The proposed OADMM-EP and OADMM-RR demonstrate similar results and generally achieve lower objective function values than the other methods.

Figure: (a) w1a-2477-300
Refer to caption: /html/2405.15129/assets/x1.png

Figure: (a) w1a-2477-300
Refer to caption: /html/2405.15129/assets/x9.png

Figure: (a) w1a-2477-300
Refer to caption: /html/2405.15129/assets/x17.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: - The paper discusses the convergence rate of the proposed method, though this might be somewhat standard, as it has been addressed in prior work on nonconvex and nonsmooth ADMM. Moreover, based on the K\L\ framework, when the parameter $\tilde \sigma \in (0, 1/2]$, the convergence rate is linear. Why, then, is the range $(1/4, 1/2]$ in this paper?

-  The authors claim that they can use a larger dual step size, $\sigma \in [1,2)$; however, they do not clarify how this value is chosen. The same issue arises for the parameter $\theta$.

- Although the authors note that OADMM-RR uses the BB step size, only a simple constant step size $b^t \equiv 1$ is employed in the numerical experiments.

- Several parameters in the proposed algorithm require careful tuning. For instance, the authors select $\xi$ from a range---are the results in Figures 1 and 2 based on the optimal $\xi$?

- The performance of OADMM-RR and RADMM-II is very close. Could RADMM-II achieve better results with careful selection of $\beta^t$?
soundness: 2
strengths: The complexity of the ADMM proposed in this paper is superior to that of the existing Riemannian ADMM. Additionally, this paper addresses a more general formulation.
summary: This paper proposes an ADMM-type algorithm to solve a class of nonconvex nonsmooth composite problems under orthogonality constraints. The iteration complexity of the proposed OADMM surpasses that of the ADMM by Li et al. (2022) and aligns with the best-known complexity found in studies such as Beck and Rosset (2023). Numerical results are provided to evaluate the efficiency of the proposed method.
weaknesses: Overall, I find the topic and results intriguing. However, the writing style could be improved for a conference paper. The authors attempt to incorporate extrapolation or adaptive step sizes in the proposed algorithms, yet these techniques are not implemented in the numerical experiments.  If they had focused simply on ADMM and make some comments, the paper might be more accessible.

Moreover, the proposed ADMM is similar to the existing Riemannian ADMM and applies the adaptive technique from Beck and Rosset (2023) to update the smoothing parameter, $\mu$. The authors should emphasize this point in the paper and clarify the key factor contributing to the improved complexity.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Is the approach extendable to general submanifolds? The current approach seems to be applicable only to compact manifolds. For instance, in the primal update, it is challenging to control boundedness, which implies that the lower boundedness of the potential function may not be guaranteed. Could you comment on this?
soundness: 3
strengths: The overall complexity conclusions presented in the paper are reasonable and align with similar algorithms proposed in RADMM (Li et al., 2022) and ManIAL (Deng et al., 2024). This paper can be seen as an accelerated version of RADMM, incorporating the Barzilai-Borwein step size and extrapolation for improved performance.
summary: This paper studies nonconvex nonsmooth problems with orthogonality constraints and proposes a Riemannian primal-dual algorithm called OADMM, which comes with global convergence guarantees and sequential convergence under KL assumption.
weaknesses: The paper is written in a way that compiles all the technical material with complex notations, making it difficult for readers to follow. It would be beneficial if the author could summarize the results more effectively and provide insights and discussions. For instance, highlighting what specifically contributes to the complexity improvement compared to RADMM would enhance clarity. Additionally, I have several detailed concerns:

1. The setting of the paper requires $h$ to be proximal-friendly. In Table 1, when comparing this method to those that do not require proximal operations (e.g., Li et al., 2021), it would be helpful to make a clear statement regarding this distinction before presenting the complexity comparison. This is important because, without proximal information, classical smoothing techniques typically incur an additional worst-case complexity of $\mathcal{O}(\epsilon^{-1})$, which should not be ignored.

2. The author claims that the proposed OADMM can achieve the oracle complexity of $\mathcal{O}(\epsilon^{-3})$, but it is unclear what kind of explicit oracle is being used. If it relies on the classical first-order oracle, some additional costs of the proposed algorithm should be discussed. For example, the subproblem of OADMM-EP restricts it to the manifold (even though some manifolds have closed-form solutions). Additionally, the line search step in OADMM-RR can be very costly, and computing its exact cost is challenging.

3. In the numerical experiments section, the current function $f$ in the given sparse PCA example does not appear to satisfy the $L_f$-smooth assumption. It would be better to do a further computation combining with the properties of the orthogonal constraints to make a equivalent formulation satisfying assumptions fully supported by the theoretical analysis.

4. The paper could be more mathematically rigorous in several places. For example: - Line 038: The author seems to intend to refer to $\mathcal{A}$ rather than $\mathcal{A}(X)$ as not being a mapping. - Line 041: $\mathcal{f}(X)$ is incorrectly referred to as a function, which occurs repeatedly throughout the paper.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Here’s a refined version of each point to improve clarity and readability:

1. Definition 2.7: Since the subgradient is a set, how is Crit calculated?
2. Definition 2.7:  The defined stationarity differs from that in the existing literature because the problem is more complex. Please validate this notion of stationarity with established concepts for problem (1).
3. Algorithm 1: Only one iteration is performed for each \( X \) update. Is the BB step size truly beneficial, given that the gradients $G^{t-1}$ and $G^t$ are computed based on different objectives?
4. Algorithm 1: In classical ADMM, the penalty parameter is typically fixed. Does a constant $\beta^t$ ensure the same convergence guarantee?
5. The connection between Crit and $e^t$ is unclear. From Lemmas 4.7 and 4.12, it appears that the average of $e^t$ converges. However, the definition of $e^t$ involves the optimality of subproblems, such as $x^t - x^{t-1}$ and $y^t - y^{t-1}$. Careful elaboration on $e^t$ and the stationarity of the original problem is needed.
6. Assumption 5.1: How are the KL assumptions on $L$ connected to problem (1)? Currently, it seems standard to show the rate under the KL of $\Theta$.
7. Line 453-454: There are mismatches in the descriptions.
8. The RADMM algorithm cannot handle nonconvex nonsmooth terms in the objectives. How is it implemented?
9. SPGM-EP: It is unclear how these two algorithms are implemented. Is $\|x\|\_1 - \|x\|\_{[k]}$ weakly convex, and is its Moreau envelope smoothing used?
10. Line 471-478: The description of RADMM is missing.
11. Abstract: What does "O" in OADMM stand for?
12. Line 132: "OADMM-EP" should be "OADMM-RR." Are these abbreviations necessary here?
soundness: 2
strengths: 1. The problem class considered seems general and encompasses many application problems with orthogonality constraints.
2. The proposed ADMM methods incorporate several practical strategies, such as prox-linear updates and the Barzilai-Borwein step size.
summary: The paper proposes an ADMM method for solving a class of nonsmooth composite optimization problems with orthogonality constraints. The authors analyze the convergence rate and present results from numerical experiments.
weaknesses: Here’s a refined version of each point for improved clarity and grammar:
1. The definition of the stationary point (i.e., Crit) is not rigorous, and its validation is missing. This makes some difficulty in understanding the complexity results.
2. The algorithm includes many parameters. How can the choices of penalty and smoothing parameters be justified?
3. The presentation could be improved. For example:  
   a. Is there any intuition behind why a one-step update on the $X$ variable ensures convergence, especially given the nonconvexity of the $X$-subproblem?  
   b. Why is the smoothing parameter $\mu$ important?
4. The numerical results appear a bit preliminary, and it would be beneficial to include more applications for comparison.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. Could the authors point out the core technical difficulties in this paper?
2. Could the authors compare OADMM with the ManPG (by linearizing the concave part) in the experiments?
3. Note that the sparse PCA is not weakly convex, it is not rigorious to use the Riemannian subgradient method in the experiment. That is partly why subgradient methods do not work. See Line 473.
4. The applications are rather limited. The authors only provide the sparse PCA example. Could the authors find other applications with the linear operator A not being the identity operator?
5. There is some mistake in Line 602 in the reference part.
6. I do not understand why (a) in Lemma 2.2 holds. Could the authors explain it?
7. The paper claims that the iteration complexity improves from $O(\epsilon^{-4})$ to $O(\epsilon^{-3})$  due to the decreasing step size in lines 142-143. However, in lines 255–259, the paper also claims that the over-relaxation step size $\sigma$ for the dual variable and the Nesterov extrapolation parameter $\alpha$ accelerate the convergence. This is somewhat confusing. My understanding is that the complexity might remain the same if we set $\alpha = 0$, $\sigma = 1$, and $\theta > 1$ (in this case, OADMM would reduce to the classical linearized proximal ADMM). In other words, $\alpha$ and $\sigma$ primarily enhance numerical performance rather than complexity. Could the authors clarify this point to make the mechanism behind the complexity improvement more explicit?
8. The paper does not include a convergence rate analysis for cases where the KL exponent $\tilde{\sigma} < 1/4$. Could the authors provide an explanation for this omission?
9. The description of SPGM as a penalty method (lines 100–103) seems inconsistent with the approach in the original paper, where the problem is solved by taking a gradient step with respect to the smoothed problem $\min_{x \in M} f(x) + h_\mu(x)$ . Are these descriptions equivalent? Clarification on this equivalence would be helpful.
10. It seems that $\mathcal{A}$ could be a smooth mapping with a bounded operator norm and Lipschitz continuous gradient to ensure that the function $S$ is smooth. Is the assumption that $\mathcal{A}$ is a linear mapping is due to practical computation considerations?
soundness: 3
strengths: 1. The authors propose a new algorithm OADMM.
2. The OADMM algorithm proposed in the paper demonstrates an iteration complexity of $O(\epsilon^{-3})$, which aligns with the best-known complexity result for this type of problem.
3. Additionally, the paper establishes the convergence of the entire sequence and derives a convergence rate under the KL assumption.
4. Numerical results show the fast convergence.
summary: This paper addresses a class of structured, nonconvex, nonsmooth optimization problems with orthogonality constraints. The authors propose OADMM to solve these problems by utilizing efficient proximal linearized strategies. Under mild assumptions, OADMM is shown to converge to a critical point with an ergodic rate of $O(1/\epsilon^3)$, and exhibits either super-exponential or polynomial convergence depending on the setting, based on the Kurdyka-Lojasiewicz (KL) inequality. Numerical experiments indicate that OADMM achieves state-of-the-art performance.
weaknesses: 1. This paper is not well-written. A lot of technical results are not well organized. I find it hard to get the novel/essential technical lemmas.
2. The authors study optimization problems of a very general form. However, they only perform experiments on sparse PCA (the linear operator A=I).
3. Line 133 and 461: "(iii) SPGM-EP" should be corrected to "RR."
4. The paper suggests that a Nesterov extrapolation strategy with parameter $\alpha$ can accelerate the convergence of primal variables. It would be beneficial if the authors could provide theoretical or numerical evidence to substantiate this claim. Although this is not critical given the paper's primary focus on theoretical convergence analysis, such verification could further enhance the contribution.
5. In the convergence rate analysis, the paper imposes a smoothness assumption on the function $g$. Given this assumption, it seems possible to absorb $g$ into the smooth function $f$, potentially simplifying the analysis.

Meta-review: This paper considers ADMM for nonsmooth composite optimization with orthogonality constraints. The authors claim that they obtained improved complexity result comparing with existing work. However, all reviewers found that the paper is not well-written. There are technical concepts not defined or not clearly explained, making the paper hard to read and the results hard to verify. The authors did not provide a rebuttal to clarify the issues raised.

Decision: Reject