A Block Coordinate Descent Method for Nonsmooth Composite Optimization under Orthogonality Constraints

## Abstract

Abstract Nonsmooth composite optimization with orthogonality constraints is crucial in statistical learning and data science, but it presents challenges due to its nonsmooth objective and computationally expensive, non-convex constraints. In this paper, we propose a new approach called OBCD , which leverages Block Coordinate Descent (BCD) to address these challenges. OBCD is a feasible method with a small computational footprint. In each iteration, it updates k 𝑘 k italic_k rows of the solution matrix, where k ≥ 2 𝑘 2 k\geq 2 italic_k ≥ 2 , while globally solving a small nonsmooth optimization problem under orthogonality constraints. We prove that OBCD converges to block- k 𝑘 k italic_k stationary points, which offer stronger optimality than standard critical points. Notably, OBCD is the first greedy descent method with monotonicity for this problem class. Under the Kurdyka-Lojasiewicz (KL) inequality, we establish strong limit-point convergence. We also extend OBCD with breakpoint searching methods for subproblem solving and greedy strategies for working set selection. Comprehensive experiments demonstrate the superior performance of our approach across various tasks.

## 1 Introduction

We consider the following nonsmooth composite optimization problem under orthogonality constraints (‘$\triangleq$’ means define):

$$ $\displaystyle\min_{\mathbf{X}\in\mathbb{R}^{n\times r}}~{}{F}(\mathbf{X}) \triangleq f(\mathbf{X})+h(\mathbf{X}),\,s.t.\,\mathbf{X}^{\mathsf{T}}\mathbf{ X}=\mathbf{I}_{r}.$ (1) $$

Here, $n\geq r$ and $\mathbf{I}_{r}$ is a $r\times r$ identity matrix. We do not assume convexity of $f(\mathbf{X})$ and $h(\mathbf{X})$. For brevity, the orthogonality constraints $\mathbf{X}^{\mathsf{T}}\mathbf{X}=\mathbf{I}_{r}$ in Problem (1) is rewritten as $\mathbf{X}\in\mathrm{St}(n,r)\triangleq\{\mathbf{X}\in\mathbb{R}^{n\times r}~{
}|~{}\mathbf{X}^{\mathsf{T}}\mathbf{X}=\mathbf{I}_{r}\}$, where $\mathcal{M}\triangleq\mathrm{St}(n,r)$ is the Stiefel manifold in the literature Edelman et al. (); Absil et al. (); Wen & Yin (); Hu et al. (). We impose the following assumptions on Problem (1) throughout this paper. ($\mathbb{A}$sm-i) For any $\mathbf{X}$ and $\mathbf{X}^{+}$, where $\mathbf{X}$ and $\mathbf{X}^{+}$ only differ at most by $k$ rows with $k\geq 2$, we assume $f:\mathbb{R}^{n\times r}\mapsto\mathbb{R}$ is continuously differentiable for a matrix $\mathbf{H}\in\mathbb{R}^{nr\times nr}$ with $\mathbf{H}\succeq\mathbf{0}$ that:

$$ $\displaystyle f(\mathbf{X}^{+})\leq\mathcal{Q}(\mathbf{X}^{+};\mathbf{X}) \triangleq f(\mathbf{X})+\langle\mathbf{X}^{+}-\mathbf{X},\nabla f(\mathbf{X}) \rangle+\tfrac{1}{2}\|\mathbf{X}^{+}-\mathbf{X}\|_{\mathbf{H}}^{2},$ (2) $$

where $\|\mathbf{H}\|\leq L_{f}$ for some constant $L_{f}>0$ and $\|\mathbf{X}\|_{\mathbf{H}}^{2}\triangleq\mathrm{vec}(\mathbf{X})^{\mathsf{T}}
\mathbf{H}\mathrm{vec}(\mathbf{X})$ (^1^11Given any symmetric matrices $\mathbf{C}\in\mathbb{R}^{n\times n}$ and $\mathbf{D}\in\mathbb{R}^{r\times r}$, we let $\mathbf{H}=\mathbf{D}\otimes\mathbf{C}$. The function $f(\mathbf{X})=\tfrac{1}{2}{\rm{tr}}(\mathbf{X}^{\mathsf{T}}\mathbf{C}\mathbf{X }\mathbf{D})=\tfrac{1}{2}\|\mathbf{X}\|^{2}_{\mathbf{H}}$ satisfies (2) with equality, as $f(\mathbf{X}^{+})=\mathcal{Q}(\mathbf{X}^{+};\mathbf{X})$ holds for all $\mathbf{X}$ and $\mathbf{X}^{+}$.). ($\mathbb{A}$sm-ii) The function $h(\mathbf{X}):\mathbb{R}^{n\times r}\mapsto\mathbb{R}$ is closed, proper, and lower semicontinuous, and potentially non-smooth. Additionally, it is coordinate-wise separable, such that $h(\mathbf{X})=\sum_{i,j}h(\mathbf{X}_{ij})$. Typical examples of $h(\mathbf{X})$ include the $\ell_{p}$ norm function $h(\mathbf{X})=\|\mathbf{X}\|_{p}$ with $p\in\{0,1\}$, and the indicator function for non-negativity constraints $h(\mathbf{X})=\mathcal{I}_{\geq 0}(\mathbf{X})$. ($\mathbb{A}$sm-iii) The following small-sized subproblem can be solved exactly and efficiently: $\min_{\mathbf{V}\in\mathrm{St}(k,k)}~{}\mathcal{P}(\mathbf{V})\triangleq\tfrac
{1}{2}\|\mathbf{V}\|_{\mathbf{Q}^{\prime}}^{2}+\langle\mathbf{V},\mathbf{P}
\rangle+h(\mathbf{V}\mathbf{Z})$ for any given $\mathbf{Z}\in\mathbb{R}^{k\times r}$, $\mathbf{P}\in\mathbb{R}^{k\times k}$, and $\textstyle\mathbf{Q}^{\prime}\in\mathbb{R}^{k^{2}\times k^{2}}$. Here, we employ a notational simplification by defining $h(\mathbf{V}\mathbf{Z})\triangleq\sum_{i,j}h([\mathbf{V}\mathbf{Z}]_{ij})$, given the coordinate-wise separability of the function $h(\cdot)$.

Problem (1) is an optimization framework that plays a crucial role in a variety of statistical learning and data science models, such as sparse Principal Component Analysis (PCA) Journée et al. (); Shalit & Chechik (), nonnegative PCA Zass & Shashua (); Qian et al. (), deep neural networks Cogswell et al. (); Cho & Lee (); Xie et al. (); Bansal et al. (); Massart & Abrol (); Huang & Gao (), electronic structure calculation Zhang et al. (); Liu et al. (), Fourier transforms approximation Frerix & Bruna (), phase synchronization Liu et al. (), orthogonal nonnegative matrix factorization Jiang et al. (), $K$-indicators clustering Jiang et al. (), and dictionary learning Zhai et al. ().

### 1.1 Related Work

We now present some related algorithms in the literature.

$\blacktriangleright$ Minimizing Smooth Functions under Orthogonality Constraints. One difficulty in solving Problem (1) arises from the nonconvexity of the orthogonality constraints. Existing methods for handling this issue can be divided into three classes. (i) Geodesic-like methods Abrudan et al. (); Edelman et al. (); Absil et al. (); Jiang & Dai (). Since calculating geodesics involves solving ordinary differential equations, which may cause computational complexity, geodesic-like methods iteratively compute the geodesic logarithm using simple linear algebra calculations. The work of Wen & Yin () develops a simple and efficient constraint preserving update scheme and achieves low computation complexity per iteration. They combine the feasible update scheme with the Barzilai-Borwein (BB) nonmonotonic line search for optimization with orthogonality constraints. (ii) Projection-like methods Absil et al. (); Golub & Van Loan (). These methods preserve the orthogonality constraints by projection. They decrease the objective value using its current Euclidean gradient direction or Riemannian tangent direction, followed by an orthogonal projection operation. This can be calculated by polar decomposition or approximated by QR factorization. (iii) Multiplier correction methods Gao et al. (; ); Xiao et al. (). Since the Lagrangian multiplier associated with the orthogonality constraint is symmetric and has an explicit closed-form expression at the first-order optimality condition, multiplier correction methods update the multiplier after achieving sufficient reduction in the objective function. This leads to efficient first-order feasible or infeasible approaches.

$\blacktriangleright$ Minimizing Nonmooth Functions under Orthogonality Constraints. Another difficulty of solving Problem (1) comes from the nonsmoothness of the objective function. Existing methods for addressing this problem can be classified into three categories. (i) Subgradient methods Hwang et al. (); Li et al. (). Subgradient methods are analogous to gradient descent methods. Most of the aforementioned geodesic-like and projection-like strategies can be incorporated into the subgradient methods. However, the step size in subgradient methods needs to be diminishing to guarantee convergence. (ii) Proximal gradient methods Chen et al. (). They solve a strongly convex minimization problem over the tangent space using a semi-smooth Newton method to find a descent direction. Subsequently, they maintain the orthogonality constraint through a retraction operation. (iii) Operator splitting methods Lai & Osher (); Chen et al. (); Zhang et al. (). Operator splitting methods introduce linear constraints and decompose the original problem into simpler subproblems, which can be solved separately and exactly. Alternating Direction Methods of Multipliers (ADMM) He & Yuan () and Smoothing Penalty Methods (SPM) Chen () represent two prominent variants of operator splitting methods.

$\blacktriangleright$ Block Coordinate Descent Methods. (Block) coordinate descent is a classical and powerful algorithm that solves optimization problems by iteratively performing minimization along (block) coordinate directions Tseng & Yun (); Xu & Yin (). The BCD methods have recently gained attention in solving nonconvex optimization problems, including sparse optimization Yuan (), $k$-means clustering Nie et al. (), structured nonconvex minimization Yuan (), recurrent neural network Massart & Abrol (), and multi-layer convolutional networks Bibi et al. (); Zeng et al. (). BCD methods have also been used in Shalit & Chechik (); Massart & Abrol () for solving optimization problems with orthogonal group constraints. However, their column-wise BCD methods are limited only to solve smooth minimization problems with $k=2$ and $r=n$ (Refer to Section 4.2 in Shalit & Chechik ()). Our row-wise BCD methods can solve general nonsmooth problems with $k\geq 2$ and $r\leq n$. The work of Gao et al. () proposes a parallelizable column-wise BCD scheme for solving the subproblems of their proximal linearized augmented Lagrangian algorithm. Impressive parallel scalability in a parallel environment of their algorithm is demonstrated. We stress that our row-wise BCD methods differ from the two column-wise counterparts, and no existing BCD methods can address Problem (1) when $h(\mathbf{X})\neq 0$.

$\blacktriangleright$ Summary. Existing solutions have one or more of the following limitations: (i) They rely on full gradient information, incurring high computational costs per iteration. (ii) They cannot handle general nonsmooth composite problems. (iii) They lack descent properties, even worse, they are infeasible methods, achieving solution feasibility only at the limit point. (iv) They often lack rigorous convergence guarantees. (v) They only establish weak optimality at critical points. $\bigstar$ To our knowledge, this represents the first application of BCD methods to solve nonsmooth composite optimization problems under orthogonality constraints, demonstrating strong optimality and convergence guarantees.

### 1.2 Contributions

This paper makes the following contributions. (i) Algorithmically: We propose a Block Coordinate Descent (BCD) algorithm tailored for nonsmooth composite optimization under orthogonality constraints (Section 2). (ii) Theoretically: We provide comprehensive optimality and convergence analyses of our methods (Sections 3 and 4). (iii) Side Contributions: We introduce breakpoint searching methods for solving subproblems when $k=2$ (Section 5), and present two working set selection greedy strategies to improve the computational efficiency of our methods (Section D in the Appendix). (iv) Empirically: Extensive experiments demonstrate that our methods surpass existing solutions in terms of accuracy and/or efficiency (Section 6).

## 2 The Proposed OBCD Algorithm

In this section, we introduce OBCD, a Block Coordinate Descent algorithm for solving general nonsmooth composite problems under Orthogonality constraints, as defined in Problem 1.

We start by presenting a new update scheme designed to maintain the orthogonality constraint.

$\blacktriangleright$ A New Constraint-Preserving Update Scheme. For any partition of the index vector $[1,2,...,n]$ into $[\texttt{{B}},\texttt{{B}}^{c}]$ with $\texttt{{B}}\in\mathbb{N}^{k}$, $\texttt{{B}}^{c}\in\mathbb{N}^{n-k}$, we define $\mathrm{U}_{\texttt{{B}}}\in\mathbb{R}^{n\times k}$ and $\mathrm{U}_{\texttt{{B}}^{c}}\in\mathbb{R}^{n\times(n-k)}$ as: ${\textstyle(\mathrm{U}_{\texttt{{B}}})_{ji}=\left\{\begin{array}[]{ll}1,&\hbox
{$\texttt{{B}}_{i}=j$;}\\
0,&\hbox{else.}\end{array}\right.,(\mathrm{U}_{\texttt{{B}}^{c}})_{ji}=\left\{
\begin{array}[]{ll}1,&\hbox{$\texttt{{B}}^{c}_{i}=j$;}\\
0,&\hbox{else.}\end{array}\right.}$. Therefore, we have the following variable splitting for any $\mathbf{X}\in\mathbb{R}^{n\times r}$: $\mathbf{X}=\mathbf{I}_{n}\mathbf{X}=(\mathrm{U}_{\texttt{{B}}}\mathrm{U}_{
\texttt{{B}}}^{\mathsf{T}}+\mathrm{U}_{\texttt{{B}}^{c}}\mathrm{U}_{\texttt{{B
}}^{c}}^{\mathsf{T}})\mathbf{X}=\mathrm{U}_{\texttt{{B}}}\mathbf{X}(\texttt{{B
}},:)+\mathrm{U}_{\texttt{{B}}^{c}}\mathbf{X}(\texttt{{B}}^{c},:)$, where $\mathbf{X}(\texttt{{B}},:)=\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}\in
\mathbb{R}^{k\times r}$ and $\mathbf{X}(\texttt{{B}}^{c},:)=\mathrm{U}_{\texttt{{B}}^{c}}^{\mathsf{T}}
\mathbf{X}\in\mathbb{R}^{(n-k)\times r}$.

In each iteration $t$, the indices $\{1,2,...,n\}$ of the rows of decision variable $\mathbf{X}\in\mathrm{St}(n,r)$ are separated to two sets B and $\texttt{{B}}^{c}$, where B is the working set with $|\texttt{{B}}|=k$ and $\texttt{{B}}^{c}=\{1,2,...,n\}\setminus\texttt{{B}}$. To simplify notation, we use B instead of $\texttt{{B}}^{t}$, as $t$ can be inferred from the context. We only update $k$ rows of the variable $\mathbf{X}$ via $\mathbf{X}^{t+1}(\texttt{{B}},:)\Leftarrow\mathbf{V}\mathbf{X}^{t}(\texttt{{B}
},:)$ for some appropriate matrix $\mathbf{V}\in\mathbb{R}^{k\times k}$. The following equivalent expressions hold:

$$ $\displaystyle\mathbf{X}^{t+1}(\texttt{{B}},:)=\mathbf{V}\mathbf{X}^{t}(\texttt {{B}},:)$ $\displaystyle\Leftrightarrow$ $\displaystyle\mathbf{X}^{t+1}=(\mathrm{U}_{\texttt{{B}}}\mathbf{V}\mathrm{U}_{ \texttt{{B}}}^{\mathsf{T}}+\mathrm{U}_{\texttt{{B}}^{c}}\mathrm{U}_{\texttt{{B }}^{c}}^{\mathsf{T}})\mathbf{X}^{t}$ (3) $\displaystyle\Leftrightarrow$ $\displaystyle\mathbf{X}^{t+1}=\mathbf{X}^{t}+\mathrm{U}_{\texttt{{B}}}(\mathbf {V}-\mathbf{I}_{k})\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}^{t}.$ (4) $$

We consider the following minimization procedure to iteratively solve Problem (1):

$$ $\displaystyle\min_{\mathbf{V}}\,F(\mathcal{X}_{\texttt{{B}}}^{t}(\mathbf{V})), \,s.t.\,\mathcal{X}_{\texttt{{B}}}^{t}(\mathbf{V})\in\mathrm{St}(n,r),~{}\text {where}~{}\mathcal{X}_{\texttt{{B}}}^{t}(\mathbf{V})\triangleq\mathbf{X}^{t}+ \mathrm{U}_{\texttt{{B}}}(\mathbf{V}-\mathbf{I}_{k})\mathrm{U}_{\texttt{{B}}}^ {\mathsf{T}}\mathbf{X}^{t}.$ (5) $$

The following lemma shows that the orthogonality constraint for $\mathbf{X}^{+}=\mathbf{X}+\mathrm{U}_{\texttt{{B}}}(\mathbf{V}-\mathbf{I}_{k})
\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}$ can be preserved by choosing suitable $\mathbf{V}$ and $\mathbf{X}$.

###### Lemma 2.1 .

(Proof in Appendix E.1) We let $\texttt{{B}}\in\{\mathcal{B}_{i}\}$${}_{i=1}^{\mathrm{C}_{n}^{k}}$, where the set $\{\mathcal{B}_{1},\mathcal{B}_{2},...,\mathcal{B}_{\mathrm{C}_{n}^{k}}\}$ denotes all possible combinations of the index vectors choosing $k$ items from $n$ without repetition. We let $\mathbf{V}\in\mathrm{St}(k,k)$. We define $\mathbf{X}^{+}\triangleq\mathcal{X}_{\texttt{{B}}}(\mathbf{V})\triangleq
\mathbf{X}+\mathrm{U}_{\texttt{{B}}}(\mathbf{V}-\mathbf{I}_{k})\mathrm{U}_{
\texttt{{B}}}^{\mathsf{T}}\mathbf{X}$. (a) For any $\mathbf{X}\in\mathbb{R}^{n\times r}$, we have $[\mathbf{X}^{+}]^{\mathsf{T}}\mathbf{X}^{+}=\mathbf{X}^{\mathsf{T}}\mathbf{X}$. (b) If $\mathbf{X}\in\mathrm{St}(n,r)$, then $\mathbf{X}^{+}\in\mathrm{St}(n,r)$.

###### Remark 2.2 .

When $\mathbf{X}$ is $\epsilon$-orthogonal that $\|\mathbf{X}^{\mathsf{T}}\mathbf{X}-\mathbf{I}_{k}\|_{\mathsf{F}}\leq\epsilon$ with $\epsilon\geq 0$, Lemma 2.1 implies that $\mathbf{X}^{+}$ is also $\epsilon$-orthogonal with $\|[\mathbf{X}^{+}]^{\mathsf{T}}\mathbf{X}^{+}-\mathbf{I}_{k}\|_{\mathsf{F}}\leq\epsilon$.

Thanks to Lemma 2.1, we can now explore the following alternative formulation for Problem (5).

$$ $\displaystyle\bar{\mathbf{V}}^{t}\in\arg\min_{\mathbf{V}}\,F(\mathcal{X}_{ \texttt{{B}}}^{t}(\mathbf{V})),\,s.t.\,\mathbf{V}\in\mathrm{St}(k,k).$ (6) $$

Then the solution matrix is updated via: $\mathbf{X}^{t+1}=\mathcal{X}_{\texttt{{B}}}^{t}(\bar{\mathbf{V}}^{t})$.

The following lemma offers important properties that relate to the update rule $\mathbf{X}^{+}=\mathbf{X}+\mathrm{U}_{\texttt{{B}}}(\mathbf{V}-\mathbf{I}_{k})
\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}$.

###### Lemma 2.3 .

(Proof in Appendix E.2) We define $\mathbf{X}^{+}=\mathbf{X}+\mathrm{U}_{\texttt{{B}}}(\mathbf{V}-\mathbf{I}_{k})
\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}$. For any $\mathbf{X}\in\mathrm{St}(n,r)$, $\mathbf{V}\in\mathrm{St}(k,k)$, $\texttt{{B}}\in\{\mathcal{B}_{i}\}$${}_{i=1}^{\mathrm{C}_{n}^{k}}$, and symmetric matrix $\mathbf{H}\in\mathbb{R}^{nr\times nr}$, we have: (a) $\tfrac{1}{2}\|\mathbf{X}^{+}-\mathbf{X}\|_{\mathbf{H}}^{2}=\tfrac{1}{2}\|
\mathbf{V}-\mathbf{I}_{k}\|_{\underline{\mathbf{Q}}}^{2}$, where $\underline{\mathbf{Q}}\triangleq(\mathbf{Z}^{\mathsf{T}}\otimes\mathrm{U}_{
\texttt{{B}}})^{\mathsf{T}}\mathbf{H}(\mathbf{Z}^{\mathsf{T}}\otimes\mathrm{U}
_{\texttt{{B}}})$, and $\mathbf{Z}\triangleq\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}\in\mathbb
{R}^{k\times r}$. (b) $\tfrac{1}{2}\|\mathbf{X}^{+}-\mathbf{X}\|_{\mathsf{F}}^{2}=\langle\mathbf{I}_{
k}-\mathbf{V},\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}\mathbf{X}^{
\mathsf{T}}\mathrm{U}_{\texttt{{B}}}\rangle$. (c) $\tfrac{1}{2}\|\mathbf{X}^{+}-\mathbf{X}\|_{\mathsf{F}}^{2}\leq\tfrac{1}{2}\|
\mathbf{V}-\mathbf{I}_{k}\|_{\mathsf{F}}^{2}=\langle\mathbf{I}_{k},\mathbf{I}_
{k}-\mathbf{V}\rangle$.

$\blacktriangleright$ The Main Algorithm. The proposed algorithm OBCD is an iterative procedure that sequentially minimizes the objective function along block coordinate directions within a sub-manifold of $\mathcal{M}$.

Starting with an initial feasible solution, OBCD iteratively determines a working set $\texttt{{B}}^{t}$ using specific strategies. It then solves the small-sized subproblem in Problem (6) through successive majorization minimization. This method iteratively constructs a surrogate function that majorizes the objective function, driving it to decrease as expected Mairal (); Razaviyayn et al. (), and it has proven effective for minimizing complex functions.

Figure: Algorithm 1 OBCD, The Proposed Block Coordinate Descent Algorithm for Problem (1).

We now demonstrate how to derive the majorization function for $F(\mathcal{X}_{\texttt{{B}}}^{t}(\mathbf{V}))$ in Problem (6). Initially, for any $\mathbf{X}^{t}\in\mathrm{St}(n,r)$ and $\mathbf{V}\in\mathrm{St}(k,k)$, we establish following inequalities: $f(\mathcal{X}_{\texttt{{B}}}^{t}(\mathbf{V}))-f(\mathbf{X}^{t})\overset{\text{
\char 172}}{\leq}\langle\mathcal{X}_{\texttt{{B}}}^{t}(\mathbf{V})-\mathbf{X}^
{t},\nabla f(\mathbf{X}^{t})\rangle+\tfrac{1}{2}\|\mathcal{X}_{\texttt{{B}}}^{
t}(\mathbf{V})-\mathbf{X}^{t}\|^{2}_{\mathbf{H}}\overset{\text{\char 173}}{=}
\langle\mathrm{U}_{\texttt{{B}}}(\mathbf{V}-\mathbf{I}_{k})\mathrm{U}_{\texttt
{{B}}}^{\mathsf{T}}\mathbf{X}^{t},\nabla f(\mathbf{X}^{t})\rangle+\tfrac{1}{2}
\|\mathbf{V}-\mathbf{I}_{k}\|^{2}_{\underline{\mathbf{Q}}}\overset{\text{\char
1
74}}{\leq}\langle\mathbf{V}-\mathbf{I}_{k},[\nabla f(\mathbf{X}^{t})(\mathbf{X
}^{t})^{\mathsf{T}}]_{\texttt{{B}}\texttt{{B}}}\rangle+\tfrac{1}{2}\|\mathbf{V
}-\mathbf{I}_{k}\|^{2}_{\mathbf{Q}+\alpha\mathbf{I}}$, where step ① uses Inequality (2); step ② uses Claim (a) of Lemma 2.3; step ③ uses $\alpha>0$ and $\underline{\mathbf{Q}}\preceq\mathbf{Q}$, which can be ensured by choosing $\mathbf{Q}$ using one of the following methods:

$$ $\displaystyle\mathbf{Q}=~{}$ $\displaystyle\underline{\mathbf{Q}}\triangleq(\mathbf{Z}^{\mathsf{T}}\otimes \mathrm{U}_{\texttt{{B}}})^{\mathsf{T}}\mathbf{H}(\mathbf{Z}^{\mathsf{T}} \otimes\mathrm{U}_{\texttt{{B}}}),~{}\text{with}~{}\mathbf{Z}\triangleq\mathbf {U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}^{t},$ (7) $\displaystyle\mathbf{Q}=~{}$ $\displaystyle\varsigma\mathbf{I},\,\text{with}~{}\|\underline{\mathbf{Q}}\| \leq\varsigma\leq L_{f}.$ (8) $$

Then, we construct the following function $\mathcal{K}(\mathbf{V};\mathbf{X}^{t},\texttt{{B}})$ that majorizes $F(\mathcal{X}_{\texttt{{B}}}^{t}(\mathbf{V}))$:

$$ $\displaystyle~{}F(\mathcal{X}_{\texttt{{B}}}^{t}(\mathbf{V}))=f(\mathcal{X}_{ \texttt{{B}}}^{t}(\mathbf{V}))+h(\mathcal{X}_{\texttt{{B}}}^{t}(\mathbf{V}))$ $\displaystyle\leq$ $\displaystyle~{}f(\mathbf{X}^{t})+\langle\mathbf{V}-\mathbf{I}_{k},[\nabla f( \mathbf{X}^{t})(\mathbf{X}^{t})^{\mathsf{T}}]_{\texttt{{B}}\texttt{{B}}} \rangle+\tfrac{1}{2}\|\mathbf{V}-\mathbf{I}_{k}\|_{\mathbf{Q}+\alpha\mathbf{I} }^{2}+h(\mathbf{V}\mathbf{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}^{t})$ $\displaystyle\leq$ $\displaystyle~{}\underbrace{\tfrac{1}{2}\|\mathbf{V}-\mathbf{I}_{k}\|_{\mathbf {Q}+\alpha\mathbf{I}}^{2}+\langle\mathbf{V},[\nabla f(\mathbf{X}^{t})(\mathbf{ X}^{t})^{\mathsf{T}}]_{\texttt{{B}}\texttt{{B}}}\rangle+h(\mathbf{V}\mathbf{U} _{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}^{t})+\ddot{c}}_{\mathcal{K}(\mathbf{V}; \mathbf{X}^{t},\texttt{{B}})},$ (9) $$

where $\ddot{c}=f(\mathbf{X}^{t})+h(\mathrm{U}_{\texttt{{B}}^{c}}^{\mathsf{T}}\mathbf
{X}^{t})-\langle\mathbf{I}_{k},[\nabla f(\mathbf{X}^{t})(\mathbf{X}^{t})^{
\mathsf{T}}]_{\texttt{{B}}\texttt{{B}}}\rangle$ is a constant. Here, we use the coordinate-wise separable property of $h(\cdot)$ as follows: $h(\mathcal{X}_{\texttt{{B}}}^{t}(\mathbf{V}))=h(\mathrm{U}_{\texttt{{B}}^{c}}
\mathrm{U}_{\texttt{{B}}^{c}}^{\mathsf{T}}\mathbf{X}^{t}+\mathbf{U}_{\texttt{{
B}}}\mathbf{V}\mathbf{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}^{t})=h(\mathrm{
U}_{\texttt{{B}}^{c}}^{\mathsf{T}}\mathbf{X}^{t})+h(\mathbf{V}\mathbf{U}_{
\texttt{{B}}}^{\mathsf{T}}\mathbf{X}^{t})$. We minimize the upper bound of the right-hand side of Inequality (2), resulting in the minimization problem that $\bar{\mathbf{V}}^{t}\in\arg\min_{\mathbf{V}\in\mathrm{St}(k,k)}\mathcal{K}(
\mathbf{V};\mathbf{X}^{t},\texttt{{B}})$, which can be efficiently and exactly solved due to our assumption.

Three strategies to find the working set B with $|\texttt{{B}}|=k$ can be considered. (i) Random strategy: B is randomly selected from $\{\mathcal{B}_{1},\mathcal{B}_{2},...,\mathcal{B}_{\mathrm{C}_{n}^{k}}\}$ with equal probability ${1}/{\mathrm{C}_{n}^{k}}$. (ii) Cyclic strategy: $\texttt{{B}}^{t}$ takes all possible combinations in cyclic order, such as $\mathcal{B}_{1}\,\rightarrow\,\mathcal{B}_{2}\,\rightarrow...\rightarrow\,
\mathcal{B}_{\mathrm{C}_{n}^{k}}\rightarrow\,\mathcal{B}_{1}\,\rightarrow...$. (iii) Greedy strategy: We propose two novel greedy strategies to find a good working set. Due to space limitation, we have included them in Appendix D.

The proposed OBCD algorithm is summarized in Algorithm 1. Importantly, OBCD is a partial gradient method with low iterative computational complexity as it only assesses $k$ rows of the Euclidean gradient of $\nabla f(\mathbf{X}^{t})$ and the solution $\mathbf{X}^{t}$ to compute the linear term $\langle[\nabla f(\mathbf{X}^{t})(\mathbf{X}^{t})^{\mathsf{T}}]_{\texttt{{B}}
\texttt{{B}}},\mathbf{V}\rangle=\langle[\nabla f(\mathbf{X}^{t})]_{\texttt{{B}
},:}^{\mathsf{T}}[\mathbf{X}^{t}]_{\texttt{{B}},:},\mathbf{V}\rangle$, as shown in Equation (2).

$\blacktriangleright$ Solving the General OBCD Subproblems. The following lemma outlines key properties of the OBCD subproblems.

###### Lemma 2.4 .

(Proof in Appendix E.3) We define $\mathbf{P}\triangleq[\nabla f(\mathbf{X}^{t})(\mathbf{X}^{t})^{\mathsf{T}}]_{
\texttt{{B}}\texttt{{B}}}-\mathrm{mat}(\mathbf{Q}\mathrm{vec}(\mathbf{I}_{k}))
-\alpha\mathbf{I}_{k}$, and $\mathbf{Z}=\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}^{t}$. We have: (a) The subproblem $\bar{\mathbf{V}}^{t}\in\arg\min_{\mathbf{V}\in\mathrm{St}(k,k)}\mathcal{K}(
\mathbf{V};\mathbf{X}^{t},\texttt{{B}})$ in Algorithm 1 is equivalent to the subsequent problem:

$$ $\displaystyle\min_{\mathbf{V}\in\mathrm{St}(k,k)}\mathcal{P}(\mathbf{V}) \triangleq\tfrac{1}{2}\|\mathbf{V}\|_{\mathbf{Q}}^{2}+\langle\mathbf{V}, \mathbf{P}\rangle+h(\mathbf{V}\mathbf{Z}).$ (10) $$

(b) Assume that Formula (8) is used to choose $\mathbf{Q}$. Problem (10) further reduces to the following problem: $\bar{\mathbf{V}}^{t}\in\arg\min_{\mathbf{V}\in\mathrm{St}(k,k)}\mathcal{P}(
\mathbf{V})\triangleq\langle\mathbf{V},\mathbf{P}\rangle+h(\mathbf{V}\mathbf{Z})$. In particular, when $h(\mathbf{X})\triangleq 0$, we obtain: $\bar{\mathbf{V}}^{t}=-\mathbb{P}_{\mathcal{M}}(\mathbf{P})$. Here, $\mathbb{P}_{\mathcal{M}}(\mathbf{P})$ is the nearest orthogonality matrix to $\mathbf{P}$.

###### Remark 2.5 .

(a) By Claim (b) of Lemma 2.4, when $k>2$, $h(\mathbf{X})=0$, and $\mathbf{Q}$ is chosen to be a diagonal matrix as in Equation (8), the subproblem $\bar{\mathbf{V}}^{t}\in\arg\min_{\mathbf{V}\in\mathrm{St}(k,k)}\mathcal{K}(
\mathbf{V};\mathbf{X}^{t},\texttt{{B}})$ in Algorithm 1 can be solved exactly and efficiently using SVD. (b) For general $k$ and $h(\cdot)$, the subproblem may not be solved globally, but a critical point can still be reached. Although strong optimality may be lost, a critical point (discussed later) for the final configuration $\mathbf{X}^{\infty}$ remains achievable.

$\blacktriangleright$ Smallest Possible Subproblems When $k=2$. We now discuss how to solve the subproblems exactly when $k=2$ and $h(\cdot)\neq 0$. The following lemma reveals an equivalent expression for any $\mathbf{V}\in\mathrm{St}(2,2)$.

###### Lemma 2.6 .

(Proof in Appendix E.4) Any orthogonal matrix $\mathbf{V}\in\mathrm{St}(2,2)$ can be expressed as $\mathbf{V}=\mathbf{V}^{\mathrm{rot}}_{\theta}$ or $\mathbf{V}=\mathbf{V}^{\mathrm{ref}}_{\theta}$ for some $\theta\in\mathbb{R}$, where $\mathbf{V}^{\mathrm{rot}}_{\theta}\triangleq(\begin{smallmatrix}\cos(\theta)&
\sin(\theta)\\
-\sin(\theta)&\cos(\theta)\\
\end{smallmatrix}),~{}\mathbf{V}^{\mathrm{ref}}_{\theta}\triangleq(\begin{
smallmatrix}-\cos(\theta)&\sin(\theta)\\
\sin(\theta)&\cos(\theta)\\
\end{smallmatrix})$. We have $\det(\mathbf{V}^{\mathrm{rot}}_{\theta})=1$ and $\det(\mathbf{V}^{\mathrm{ref}}_{\theta})=-1$ for any $\theta$.

Using Lemma 2.6, we can reformulate Problem (10) as the following one-dimensional problem: $\bar{\theta}\in\arg\min_{\theta}\,\mathcal{P}(\mathbf{V}),\,s.t.\,\mathbf{V}
\in\{\mathbf{V}^{\mathrm{rot}}_{\theta},\mathbf{V}^{\mathrm{ref}}_{\theta}\}$. The optimal solution $\bar{\theta}$ can be identified even if $h(\cdot)\neq 0$ using a novel breakpoint searching method, which is discussed later in Section 5.

###### Remark 2.7 .

(i) $\mathbf{V}^{\mathrm{rot}}_{\theta}$ and $\mathbf{V}^{\mathrm{ref}}_{\theta}$ are called Givens rotation matrix and Jacobi reflection matrix respectively in the literature Sun & Bischof (). Previous research only considered $\{\mathbf{V}^{\mathrm{rot}}_{\theta}\}$ for solving symmetric linear eigenvalue problems Golub & Van Loan () and sparse PCA problems Shalit & Chechik (), while we use $\{\mathbf{V}^{\mathrm{ref}}_{\theta},\,\mathbf{V}^{\mathrm{rot}}_{\theta}\}$ for solving Problem (1). (ii) We show the necessity of using $\{\mathbf{V}^{\mathrm{ref}}_{\theta},\,\mathbf{V}^{\mathrm{rot}}_{\theta}\}$ in the following two examples of $2\times 2$ optimization problems with orthogonality constraints: $\min_{\mathbf{V}\in\mathrm{St}(2,2)}F(\mathbf{V})\triangleq\|\mathbf{V}-
\mathbf{A}\|_{\mathsf{F}}^{2}$, and $\min_{\mathbf{V}\in\mathrm{St}(2,2)}F(\mathbf{V})\triangleq\|\mathbf{V}-
\mathbf{B}\|_{\mathsf{F}}^{2}+5\|\mathbf{V}\|_{1}$, where $\mathbf{A}=(\begin{smallmatrix}1&0\\
-1&-1\end{smallmatrix})$ and $\mathbf{B}=(\begin{smallmatrix}1&0\\
1&2\end{smallmatrix})$. The use of the reflection matrix $\mathbf{V}^{\mathrm{ref}}_{\theta}$ is essential in these examples because it results in lower objective values. See Section C.1 in the Appendix for more details.

###### Lemma 2.1 .

###### Remark 2.2 .

###### Lemma 2.3 .

###### Lemma 2.4 .

###### Remark 2.5 .

###### Lemma 2.6 .

###### Remark 2.7 .

## 3 Optimality Analysis

This section provides some optimality analysis for the proposed algorithm.

$\blacktriangleright$ Basis Representation of Orthogonal Matrices. The following theorem is used to characterize any orthogonal matrix $\mathbf{D}\in\mathrm{St}(n,n)$ and $\mathbf{X}\in\mathrm{St}(n,r)$.

###### Theorem 3.1 .

(Proof in Appendix F.1, Basis Representation of Orthogonal Matrices) Assume $k=2$. For all $i\in[\mathrm{C}_{n}^{k}]$, we define $\mathcal{W}_{i}\triangleq\mathbf{I}_{n}+\mathbf{U}_{\mathcal{B}_{i}}(\mathcal{
V}_{i}-\mathbf{I}_{k})\mathbf{U}_{\mathcal{B}_{i}}^{\mathsf{T}}=\mathbf{U}_{
\mathcal{B}_{i}}\mathcal{V}_{i}\mathbf{U}_{\mathcal{B}_{i}}^{\mathsf{T}}+
\mathbf{U}_{\mathcal{B}^{c}_{i}}\mathbf{U}_{\mathcal{B}^{c}_{i}}^{\mathsf{T}}$, where $\mathcal{V}_{i}\in\mathrm{St}(2,2)$. We have: (a) Any matrix $\mathbf{D}\in\mathrm{St}(n,n)$ can be expressed as $\mathbf{D}=\mathcal{W}_{\mathrm{C}_{n}^{k}}...\mathcal{W}_{2}\mathcal{W}_{1}$ using suitable $\mathcal{W}_{i}$ (which depends on $\mathcal{V}_{i}$). Furthermore, if $\forall i,\,\mathcal{V}_{i}=\mathbf{I}_{2}$, then $\mathbf{D}=\mathbf{I}_{n}$. (b) Any matrix $\mathbf{X}\in\mathrm{St}(n,r)$ can be expressed as $\mathbf{X}=\mathcal{W}_{\mathrm{C}_{n}^{k}}...\mathcal{W}_{2}\mathcal{W}_{1}
\mathbf{X}^{0}$ using suitable $\mathcal{W}_{i}$ and any fixed constant matrix $\mathbf{X}^{0}\in\mathrm{St}(n,r)$.

###### Remark 3.2 .

(i) We use both Givens rotation and Jacobi reflection matrices to compute $\mathbf{D}\in\mathrm{St}(n,n)$. This is necessary since a reflection matrix cannot be represented through a sequence of rotations. (ii) The result in Claim (b) of Theorem 3.1 indicates that the proposed update scheme $\mathbf{X}^{+}\Leftarrow\mathbf{X}+\mathrm{U}_{\texttt{{B}}}(\mathbf{V}-
\mathbf{I}_{k})\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}$ as shown in Formula (4) can reach any orthogonal matrix $\mathbf{X}\in\mathrm{St}(n,r)$ for any starting solution $\mathbf{X}^{0}\in\mathrm{St}(n,r)$.

$\blacktriangleright$ First-Order Optimality Conditions for Problem (1). We provide the first-order optimality condition of Problem (1) Wen & Yin (); Chen et al. (). We use $\partial F(\mathbf{X})$ to denote the limiting subdifferential of $F(\mathbf{X})$ Mordukhovich (); Rockafellar & Wets. (), which is always non-empty since $F(\mathbf{X})$ is closed, proper, and lower semicontinuous. Given $f(\mathbf{X})$ is differentiable, we have $\partial F(\mathbf{X})=\partial(f+h)(\mathbf{X})=\nabla f(\mathbf{X})+\partial
h
(\mathbf{X})$. We extend the definition of limiting subdifferential to introduce $\partial_{\mathcal{M}}F(\mathbf{X})$ as the Riemannian limiting subdifferential of $F(\mathbf{X})$ at $\mathbf{X}$, defined as $\partial_{\mathcal{M}}F(\mathbf{X})\triangleq\partial F(\mathbf{X})\ominus(
\mathbf{X}[\partial F(\mathbf{X})]^{\mathsf{T}}\mathbf{X})$, where $\ominus$ is the element-wise subtraction between sets.

Introducing a Lagrangian multiplier matrix $\mathbf{\Lambda}\in\mathbb{R}^{r\times r}$ for the orthogonality constraint, we define the following Lagrangian function of Problem (1): $\mathcal{L}(\mathbf{X},\mathbf{\Lambda})=F(\mathbf{X})+\tfrac{1}{2}\langle
\mathbf{I}_{r}-\mathbf{X}^{\mathsf{T}}\mathbf{X},\mathbf{\Lambda}\rangle$. Notable, the matrix $\mathbf{\Lambda}$ is symmetric, as $\mathbf{X}^{\mathsf{T}}\mathbf{X}$ is symmetric. We state the following first-order optimality condition.

###### Lemma 3.3 .

Critical Point Wen & Yin (); Chen et al. (). A solution $\check{\mathbf{X}}\in\mathrm{St}(n,r)$ is a critical point of Problem (1) if: $\mathbf{0}\in\partial_{\mathcal{M}}F(\check{\mathbf{X}})\triangleq\partial F(
\check{\mathbf{X}})\ominus(\check{\mathbf{X}}[\partial F(\check{\mathbf{X}})]^
{\mathsf{T}}\check{\mathbf{X}})$, where $(\partial F(\check{\mathbf{X}})\ominus\check{\mathbf{X}}[\partial F(\check{
\mathbf{X}})]^{\mathsf{T}}\check{\mathbf{X}})\triangleq\{\mathbb{E}\,|\,
\mathbb{E}=\mathbf{G}-\check{\mathbf{X}}\mathbf{G}^{\mathsf{T}}\check{\mathbf{
X}},\mathbf{G}\in\partial F(\check{\mathbf{X}})\}$. Furthermore, $\mathbf{\Lambda}\in[\partial F(\check{\mathbf{X}})]^{\mathsf{T}}\check{\mathbf
{X}}$.

###### Remark 3.4 .

The critical point condition in Lemma 3.3 can be equivalently expressed as Absil et al. (); Jiang & Dai (); Liu et al. (): $\mathbf{0}\in\mathbb{P}_{\mathrm{T}_{\check{\mathbf{X}}}\mathcal{M}}(\partial F
(\check{\mathbf{X}}))$. Here, $\mathrm{T}_{\mathbf{X}}\mathcal{M}$ is the tangent space to $\mathcal{M}$ at $\mathbf{X}\in\mathcal{M}$ with $\mathrm{T}_{\mathbf{X}}\mathcal{M}=\{\mathbf{Y}\in\mathbb{R}^{n\times r}\,|\,
\mathbf{X}^{\mathsf{T}}\mathbf{Y}+\mathbf{Y}^{\mathsf{T}}\mathbf{X}=\mathbf{0}\}$.

$\blacktriangleright$ Optimality Conditions for the Subproblems. The Euclidean subdifferential of $\mathcal{K}(\mathbf{V};\mathbf{X}^{t},\texttt{{B}}^{t})$ w.r.t. $\mathbf{V}$ can be computed as follows: $\ddot{\mathbf{G}}(\mathbf{V})\triangleq\ddot{\mathbf{\Delta}}+\mathrm{U}_{
\texttt{{B}}}^{\mathsf{T}}[\nabla f(\mathbf{X}^{t})+\partial h(\mathbf{X}^{t+1
})](\mathbf{X}^{t})^{\mathsf{T}}\mathrm{U}_{\texttt{{B}}}$, where $\ddot{\mathbf{\Delta}}=\mathrm{mat}((\mathbf{Q}+\alpha\mathbf{I}_{k})\mathrm{
vec}(\mathbf{V}-\mathbf{I}_{k}))$, and $\mathbf{X}^{t+1}=\mathbf{X}^{t}+\mathrm{U}_{\texttt{{B}}}(\mathbf{V}-\mathbf{I
}_{k})\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}^{t}$. Using Lemma 3.3, we set the Riemannian subdifferential of $\mathcal{K}(\mathbf{V};\mathbf{X}^{t},\texttt{{B}}^{t})$ w.r.t. $\mathbf{V}$ to zero and obtain the following first-order optimality condition for $\bar{\mathbf{V}}^{t}$: $\mathbf{0}\in\partial_{\mathcal{M}}\mathcal{K}(\bar{\mathbf{V}}^{t};\mathbf{X}
^{t},\texttt{{B}}^{t})\triangleq\ddot{\mathbf{G}}(\bar{\mathbf{V}}^{t})\ominus
\bar{\mathbf{V}}^{t}\ddot{\mathbf{G}}(\bar{\mathbf{V}}^{t})^{\mathsf{T}}\bar{
\mathbf{V}}^{t}$.

$\blacktriangleright$ Optimality Conditions and Their Hierarchy. We introduce the following new optimality condition of block-$k$ stationary points.

###### Definition 3.5 .

(Global) Block-$k$ Stationary Point, abbreviated as BSk-point. Let $\alpha>0$ and $k\geq 2$. A solution $\ddot{\mathbf{X}}\in\mathrm{St}(n,r)$ is called a block-$k$ stationary point if: $\forall\texttt{{B}}\in\{\mathcal{B}_{i}\}_{i=1}^{\mathrm{C}_{n}^{k}},~{}
\mathbf{I}_{k}\in\arg\min_{\mathbf{V}\in\mathrm{St}(k,k)}~{}\mathcal{K}(
\mathbf{V};\ddot{\mathbf{X}},\texttt{{B}})$, where $\mathcal{K}(\cdot;\cdot,\cdot)$ is defined in Equation (2).

Remarks. BSk-point states that if we globally minimize the majorization function $\mathcal{K}(\mathbf{V};\ddot{\mathbf{X}},\texttt{{B}})$, there is no possibility of improving the objective function value for $\mathcal{K}(\mathbf{V};\ddot{\mathbf{X}},\texttt{{B}})$ across all $\texttt{{B}}\in\{\mathcal{B}_{i}\}_{i=1}^{\mathrm{C}_{n}^{k}}$.

The following theorem establishes the relation between BSk-points, standard critical points, and global optimal points.

###### Theorem 3.6 .

(Proof in Appendix F.2) We establish the following relationships: (a) $\{\rm{critical\,points}\,\check{\mathbf{X}}\}\supseteq\{\rm{BS}_{2}\text{-}\rm
{points}\,\ddot{\mathbf{X}}\}$. (b) $\{\rm{BS}_{2}\text{-}\rm{points}\,\ddot{\mathbf{X}}\}\supseteq\{\rm{global\,
optimal\,points}\,\bar{\mathbf{X}}\}$. (c) $\{\rm{BS}_{k}\text{-}\textit{points}\,\ddot{\mathbf{X}}\}\supseteq\{\rm{BS}_{k
+1}\text{-}\textit{points}\,\ddot{\mathbf{X}}\}$, where $k\in\{2,3,\ldots,n-1\}$. (d) Furthermore, the reverse of these three inclusions may not always hold true.

###### Remark 3.7 .

The optimality of $\rm{BS}_{2}\text{-}\rm{points}$ is stronger than that of standard critical points Wen & Yin (); Chen et al. (); Absil et al. ().

###### Theorem 3.1 .

###### Remark 3.2 .

###### Lemma 3.3 .

###### Remark 3.4 .

###### Definition 3.5 .

###### Theorem 3.6 .

###### Remark 3.7 .

## 4 Convergence Analysis

This section presents the convergence analysis for the proposed OBCD algorithm.

We denote any point of the limit point set of OBCD (which is not necessarily a singleton) as $\ddot{\mathbf{X}}$. For the case where a random strategy is used to find the working set, OBCD generates a random output $(\bar{\mathbf{V}}^{t},\mathbf{X}^{t+1})$ with $t=0,1,\ldots,\infty$ which depends on the observed realization of the random variable: $\xi^{t}\triangleq(\texttt{{B}}^{1},\texttt{{B}}^{2},\texttt{{B}}^{3},\ldots,
\texttt{{B}}^{t})$.

### 4.1 Global Convergence

We define the $\epsilon$-BSk-point as follows.

###### Definition 4.1 .

($\epsilon$-BSk-point) Given any constant $\epsilon>0$, a point $\ddot{\mathbf{X}}$ is called an $\epsilon$-BSk-point if: $\tfrac{1}{\mathrm{C}_{n}^{k}}\sum_{i=1}^{\mathrm{C}_{n}^{k}}\text{dist}(
\mathbf{I}_{k},\arg\min_{\mathbf{V}}\mathcal{K}$$(\mathbf{V};\mathbf{X},\mathcal{B}_{i}$$)$$)^{2}\leq\epsilon$, where $\mathcal{K}(\cdot;\cdot,\cdot)$ is defined in Equation (2).

###### Theorem 4.2 .

(Proof in Appendix G.1, Subsequence Convergence) We define $\tilde{c}\triangleq\textstyle\tfrac{2}{\alpha}\cdot(F(\mathbf{X}^{0})-F(\ddot{
\mathbf{X}}))$. We have: (a) The following sufficient decrease condition holds for all $t\geq 0$: $\tfrac{\alpha}{2}\|\mathbf{X}^{t+1}-\mathbf{X}^{t}\|_{\mathsf{F}}^{2}\leq
\tfrac{\alpha}{2}\|\bar{\mathbf{V}}^{t}-\mathbf{I}_{k}\|_{\mathsf{F}}^{2}\leq F
(\mathbf{X}^{t})-F(\mathbf{X}^{t+1})$. (b) If the $\texttt{{B}}^{t}$ is selected from $\{\mathcal{B}_{i}\}$${}_{i=1}^{\mathrm{C}_{n}^{k}}$ randomly and uniformly, OBCD finds an $\epsilon$-BSk-point of Problem (1) in at most $T$ iterations in the sense of expectation, where $T\geq\lceil\tfrac{\tilde{c}}{\epsilon}\rceil$. (c) If the $\texttt{{B}}^{t}$ is selected from $\{\mathcal{B}_{i}\}$${}_{i=1}^{\mathrm{C}_{n}^{k}}$ cyclically, OBCD finds an $\epsilon$-BSk-point of Problem (1) in at most $T$ iterations deterministically, where $T\geq\lceil\tfrac{\tilde{c}}{\epsilon}+\mathrm{C}_{n}^{k}\rceil$.

###### Definition 4.1 .

###### Theorem 4.2 .

### 4.2 Strong Convergence under KL Assumption

We prove that OBCD achieves strong convergence based on a non-convex analysis tool called Kurdyka-Łojasiewicz inequality Attouch et al. (); Bolte et al. (); Liu et al. (). We assume that a random strategy is used to find the working set.

Initially, we make the following additional assumptions.

###### Assumption 4.3 .

The function $F^{\circ}(\mathbf{X})=F(\mathbf{X})+\mathcal{I}_{\mathcal{M}}(\mathbf{X})$ is a KL function.

###### Assumption 4.4 .

There exists positive constants $l_{f}$ and $l_{h}$ that: $\forall\mathbf{X},\|\nabla f(\mathbf{X})\|_{2}\leq l_{f},\|\partial h(\mathbf{
X})\|_{2}\leq l_{h}$.

###### Remark 4.5 .

(i) Semi-algebraic functions are a class of functions that satisfy the KL property. These functions are widely used in applications, and they include real polynomial functions, finite sums and products of semi-algebraic functions, and indicator functions of semi-algebraic sets. (ii) The important class of nonsmooth $\ell_{1}$ norm function $h(\mathbf{X})=\|\mathbf{X}\|_{1}$ Chen et al. (; ) satisfies the second condition of Assumption 4.4.

We present the following useful proposition.

###### Proposition 4.6 .

(Kurdyka-Łojasiewicz Property). For a KL function $F^{\circ}(\mathbf{X})$ with $\mathbf{X}\in\text{dom}~{}F^{\circ}$, there exists $\sigma\in[0,1),~{}\eta\in(0,+\infty]$, a neighborhood $\Upsilon$ of $\ddot{\mathbf{X}}$, and a concave continuous function $\varphi(t)=ct^{1-\sigma}$, $c>0$, $t\in[0,\eta)$ such that for all $\mathbf{X}^{\prime}\in\Upsilon$ and satisfies $F^{\circ}(\mathbf{X}^{\prime})\in(F^{\circ}(\ddot{\mathbf{X}}),F^{\circ}(\ddot
{\mathbf{X}})+\eta)$, the following inequality holds: ${\rm{dist}}(\mathbf{0},\partial F^{\circ}(\mathbf{X}^{\prime}))\varphi^{\prime
}(F^{\circ}(\mathbf{X}^{\prime})-F^{\circ}(\ddot{\mathbf{X}}))\geq 1$.

We present the following useful lemma.

###### Lemma 4.7 .

(Proof in Appendix G.2, Riemannian Subgradient Lower Bound for the Iterates Gap) The Riemannian subdifferential of $\mathcal{K}(\mathbf{V};\mathbf{X}^{t},\texttt{{B}}^{t})$ at the point $\mathbf{V}=\mathbf{I}_{k}$ can be computed as: $\partial_{\mathcal{{M}}}\mathcal{K}(\mathbf{I}_{k};\mathbf{X}^{t},\texttt{{B}}
^{t})=\mathrm{U}_{\texttt{{B}}^{t}}^{\mathsf{T}}(\mathbb{D}\ominus\mathbb{D}^{
\mathsf{T}})\mathrm{U}_{\texttt{{B}}^{t}}$, where $\mathbb{D}=[\nabla f(\mathbf{X}^{t})+\partial h(\mathbf{X}^{t})][\mathbf{X}^{t
}]^{\mathsf{T}}$. It holds that: $\mathbb{E}_{\xi^{t+1}}[{\rm{dist}}(\mathbf{0},\partial_{\mathcal{M}}\mathcal{K
}(\mathbf{I}_{k};\mathbf{X}^{t+1},\texttt{{B}}^{t+1}))]\leq\phi\cdot\mathbb{E}
_{\xi^{t}}[\|\bar{\mathbf{V}}^{t}-\mathbf{I}_{k}\|_{\mathsf{F}}]$, where $\phi\triangleq 4(l_{f}+l_{h}+L_{f})+2\alpha$.

Finally, we establish the following strong limit-point convergence results for OBCD.

###### Theorem 4.8 .

(Proof in Appendix G.3, A Finite Length Property). The sequence $\{\mathbf{X}^{t}\}_{t=0}^{\infty}$ has finite length property that: $\sum_{t=1}^{\infty}\mathbb{E}_{\xi^{t}}[\|\mathbf{X}^{t+1}-\mathbf{X}^{t}\|_{
\mathsf{F}}]\leq C<+\infty$ with $C\triangleq 2\sqrt{k}+\tfrac{4\gamma\phi}{\alpha}\varphi(F(\mathbf{X}^{1})-F(
\ddot{\mathbf{X}}))$. Here, $\gamma\triangleq({\mathrm{C}_{n}^{k}}/{\mathrm{C}_{n-2}^{k-2}})^{1/2}$, $\phi$ is defined in Lemma 4.7, and $\varphi(\cdot)$ is the desingularization function defined in Proposition 4.6.

###### Remark 4.9 .

Utilizing the Kurdyka-Łojasiewicz property, one can establish a finite-length property of OBCD, a result considerably stronger than that of Theorem 4.2. Furthermore, by delving deeper into the KL exponent, one can establish the convergence rate of OBCD Attouch et al. (); Bolte et al. ().

###### Assumption 4.3 .

###### Assumption 4.4 .

###### Remark 4.5 .

###### Proposition 4.6 .

###### Lemma 4.7 .

###### Theorem 4.8 .

###### Remark 4.9 .

## 5 Solving the Subproblem when k = 2 𝑘 2 k=2 italic_k = 2

This section presents a novel Breakpoint Searching Method (BSM) to find the global optimal solution of Problem (10) when $k=2$.

Initially, Problem (10) boils down to the following one-dimensional subproblem: $\min_{\theta}~{}\tfrac{1}{2}\|\mathbf{V}\|_{\mathbf{Q}}^{2}+\langle\mathbf{V},
\mathbf{P}\rangle+h(\mathbf{V}\mathbf{Z}),\,s.t.\,\mathbf{V}\in\{\mathbf{V}^{
\mathrm{rot}}_{\theta},\mathbf{V}^{\mathrm{ref}}_{\theta}\}$, which can be further rewritten as: $\bar{\theta}\in\arg\min_{\theta}~{}\tfrac{1}{2}\mathrm{vec}(\mathbf{V})^{
\mathsf{T}}\mathbf{Q}\mathrm{vec}(\mathbf{V})+\langle\mathbf{V},\mathbf{P}
\rangle+h(\mathbf{V}\mathbf{Z}),~{}s.t.~{}{\mathbf{V}\triangleq(\begin{
smallmatrix}\pm\cos(\theta)&\sin(\theta)\\
\mp\sin(\theta)&\cos(\theta)\\
\end{smallmatrix})}$, where ${\mathbf{Q}}\in\mathbb{R}^{4\times 4}$, $\mathbf{P}\in\mathbb{R}^{2\times 2}$, and $\mathbf{Z}\in\mathbb{R}^{2\times r}$. Given $h(\cdot)$ is coordinate-wise separable, we have the following equivalent optimization problem:

$$ $\displaystyle\begin{split}&\min_{\theta}h\left(\cos(\theta)\mathbf{x}+\sin( \theta)\mathbf{y}\right)+a\cos(\theta)+b\sin(\theta)+c\cos^{2}(\theta)+d\cos( \theta)\sin(\theta)+e\sin^{2}(\theta),\end{split}$ (11) $$

where $a=\mathbf{P}_{22}\pm\mathbf{P}_{11}$, $b=\mathbf{P}_{12}\mp\mathbf{P}_{21}$, $c=0.5(\mathbf{Q}_{11}+\mathbf{Q}_{44})\pm\mathbf{Q}_{14}$, $d=-\mathbf{Q}_{12}\pm\mathbf{Q}_{13}\mp\mathbf{Q}_{24}+\mathbf{Q}_{34}$, $e=0.5(\mathbf{Q}_{22}+\mathbf{Q}_{33})\mp\mathbf{Q}_{23}$, $\mathbf{r}=\pm\mathbf{Z}(1,:)$, $\mathbf{s}=\mathbf{Z}(2,:)$, $\mathbf{p}=\mathbf{Z}(2,:)$, $\mathbf{u}=\mp\mathbf{Z}(1,:)$, $\mathbf{x}\triangleq[\mathbf{r};\mathbf{p}]\in\mathbb{R}^{2r\times 1}$, and $\mathbf{y}\triangleq[\mathbf{s};\mathbf{u}]\in\mathbb{R}^{2r\times 1}$.

Our key strategy is to perform a variable substitution to convert Problem (11) into an equivalent problem that depends on the variable $\tan(\theta)\triangleq t$. The substitution is based on the trigonometric identities that $\cos({\theta})={\pm 1}/{\sqrt{1+\tan^{2}(\theta)}}$ and $\sin({\theta})={\pm\tan(\theta)}/{\sqrt{1+\tan^{2}(\theta)}}$.

The following lemma provides a characterization of the global optimal solution for Problem (11).

###### Lemma 5.1 .

(Proof in Appendix H.1) We define $\breve{F}(\tilde{c},\tilde{s})\triangleq a\tilde{c}+b\tilde{s}+c\tilde{c}^{2}+
d\tilde{c}\tilde{s}+e\tilde{s}^{2}+h\left(\tilde{c}\mathbf{x}+\tilde{s}\mathbf
{y}\right)$, and $w\triangleq c-e$. The optimal solution $\bar{\theta}$ to (11) can be computed as: $[\cos(\bar{\theta}),\sin(\bar{\theta})]\in\arg\min_{[c,s]}\,\breve{F}(c,s),\,s
.t.\,[c,s]\in\{[c_{1},s_{1}],[c_{2},s_{2}],[0,1],[0,-1]\}$, where $c_{1}\triangleq\tfrac{1}{\sqrt{1+(\bar{t}_{+})^{2}}}$, $s_{1}=\tfrac{\bar{t}_{+}}{\sqrt{1+(\bar{t}_{+})^{2}}}$, $c_{2}\triangleq\tfrac{-1}{\sqrt{1+(\bar{t}_{-})^{2}}}$, and $s_{2}\triangleq\tfrac{-\bar{t}_{-}}{\sqrt{1+(\bar{t}_{-})^{2}}}$. Furthermore, $\bar{t}_{+}$ and $\bar{t}_{-}$ are respectively defined as:

$$ $\displaystyle\textstyle\bar{t}_{+}\in\arg\min_{t}\,p(t)\triangleq\tfrac{a+bt}{ \sqrt{1+t^{2}}}+\tfrac{w+dt}{{1+t^{2}}}+h(\tfrac{\mathbf{x}+t\mathbf{y}}{\sqrt {1+t^{2}}}),$ (12) $\displaystyle\textstyle\bar{t}_{-}\in\arg\min_{t}\,\tilde{p}(t)\triangleq \tfrac{-a-bt}{\sqrt{1+t^{2}}}+\tfrac{w+dt}{{1+t^{2}}}+h(\tfrac{-\mathbf{x}-t \mathbf{y}}{\sqrt{1+t^{2}}}).$ (13) $$

We describe our BSM to solve Problem (12); our approach can be naturally extended to tackle Problem (13). BSM first identifies all the possible breakpoints / critical points $\Theta$, and then picks the solution that leads to the lowest value as the optimal solution $\bar{t}$, i.e., $\bar{t}\in\arg\min_{t}\,p(t),~{}s.t.~{}t\in\Theta$.

We assume $\mathbf{y}_{i}\neq 0$. If this is not true and there exists $\mathbf{y}_{i}=0$ for some $i$, then $\{\mathbf{x}_{i},\mathbf{y}_{i}\}$ can be removed since it does not affect the minimizer of the problem.

We now show that how to find the breakpoint set $\Theta$ for $h(\mathbf{x})=\lambda\|\mathbf{x}\|_{0}$, where $\lambda\geq 0$. We also provide additional examples of BSM for other different $h(\mathbf{x})$. Due to space limitation, we have included them in Appendix B.

$\blacktriangleright$ Finding the Breakpoint Set for $h(\mathbf{x})\triangleq\lambda\|\mathbf{x}\|_{0}$

Since the function $h(\mathbf{x})\triangleq\lambda\|\mathbf{x}\|_{0}$ is scale-invariant and symmetric with $\|\pm t\mathbf{x}\|_{0}=\|\mathbf{x}\|_{0}$ for all $t>0$, Problem (12) reduces to the following problem:

$$ $\displaystyle\min_{t}p(t)\triangleq\tfrac{a+bt}{\sqrt{1+t^{2}}}+\tfrac{w+dt}{1 +t^{2}}+\lambda\|\mathbf{x}+t\mathbf{y}\|_{0}.$ (14) $$

Given the limiting subdifferential of the $\ell_{0}$ norm function can be computed as $\partial\|t\|_{0}\in{\{\begin{array}[]{ll}\mathbb{R},&\hbox{$t=0$;}\\
\{0\},&\hbox{else.}\end{array}\}}$ (see Appendix C.5), we consider the following two cases. (i) We assume $(\mathbf{x}+t\mathbf{y})_{i}=0$ for some $i$. Then the solution $\bar{t}$ can be determined using $\bar{t}=\tfrac{\mathbf{x}_{i}}{\mathbf{y}_{i}}$. There are $2r$ breakpoints $\{\tfrac{\mathbf{x}_{1}}{\mathbf{y}_{1}},\tfrac{\mathbf{x}_{2}}{\mathbf{y}_{2}
},...,\tfrac{\mathbf{x}_{2r}}{\mathbf{y}_{2r}}\}$ for this case. (ii) We now assume $(\mathbf{x}+t\mathbf{y})_{i}\neq 0$ for all $i$. Then $\lambda\|\mathbf{x}+t\mathbf{y}\|_{0}=2r\lambda$ becomes a constant. Setting the subgradient of $p(t)$ to zero yields: $0=\nabla p(t)=[b(1+t^{2})-(a+bt)t]\cdot\sqrt{1+t^{2}}\cdot t^{\circ}+[d(1+t^{2
})-(w+dt)(2t)]\cdot t^{\circ}$, where $t^{\circ}=(1+t^{2})^{-2}$. Since $t^{\circ}>0$, we obtain: $d(1+t^{2})-(w+dt)2t=-(b-at)\cdot\sqrt{1+t^{2}}$. Squaring both sides, we obtain the following quartic equation: $c_{4}t^{4}+c_{3}t^{3}+c_{2}t^{2}+c_{1}t+c_{0}=0$ for some suitable $c_{4}$, $c_{3}$, $c_{2}$, $c_{1}$ and $c_{0}$. Solving this equation analytically using Lodovico Ferrari’s method , we obtain all its real roots $\{\bar{t}_{1},\bar{t}_{2},...,\bar{t}_{j}\}$ with $1\leq j\leq 4$. There are at most $4$ breakpoints for this case. Therefore, Problem (14) contains at most $2r+4$ breakpoints $\Theta=\{\tfrac{\mathbf{x}_{1}}{\mathbf{y}_{1}},\tfrac{\mathbf{x}_{2}}{\mathbf
{y}_{2}},...,\tfrac{\mathbf{x}_{2r}}{\mathbf{y}_{2r}},\bar{t}_{1},\bar{t}_{2},
...,\bar{t}_{j}\}$. We emphasize that $(2r+4)$ breakpoints are both necessary and sufficient to find the global optimal solution for the fractional problem in Problem (14). Our algorithm fully accounts for all possible solutions without adding extra computational overhead.

###### Lemma 5.1 .

## 6 Experiments

This section provides numerical comparisons of OBCD
against state-of-the-art methods on both real-world and synthetic data. We describe the application of $L_{0}$ norm-based Sparse PCA (SPCA) in the sequel, while additional applications for nonnegative PCA and $\ell_{1}$ norm-based SPCA can be found in Appendix J.

$\blacktriangleright$  Application to $L_{0}$ Norm-based SPCA. $L_{0}$ norm-based Sparse PCA (SPCA) is a method that uses $\ell_{0}$ norm to produce modified principal components with sparse loadings, which helps reduce model complexity and increase model interpretability d’Aspremont et al. (); Chen et al. (). It can be formulated as:

$$ $\displaystyle\min_{\mathbf{X}\in\mathrm{St}(n,r)}~{}-\tfrac{1}{2}\langle \mathbf{X},\mathbf{C}\mathbf{X}\rangle+\lambda\|\mathbf{X}\|_{0},$ (15) $$

where $\mathbf{C}=\mathbf{A}^{\mathsf{T}}\mathbf{A}\in\mathbb{R}^{n\times n}$ is the covariance of the data matrix $\mathbf{A}\in\mathbb{R}^{m\times n}$ and $\lambda>0$.

**Table 1: Comparisons of relative objective values $(F(\mathbf{X})-F_{\text{min}})$ for $L_{0}$ norm-based SPCA across all the compared methods. The $1^{st}$, $2^{nd}$, and $3^{rd}$ best results are colored with red, green and blue, respectively.**
| data-m-n | $F_{\text{min}}$ | LADMM | SPM | LADMM | SPM | OBCD-R |
| --- | --- | --- | --- | --- | --- | --- |
| (id) | (id) | (rnd) | (rnd) | (id) |  |  |
| $r=20$, $\lambda=10$, time limit=30 |  |  |  |  |  |  |
| w1a-2477-300 | -5.2e+03 | 1.92e+03 | 4.55e+03 | 3.30e+02 | 8.05e+02 | 0.00e+00 |
| TDT2-500-1000 | 2.0e+02 | 3.74e+00 | 3.74e+00 | 1.10e+02 | 2.70e+02 | 0.00e+00 |
| 20News-8000-1000 | 2.0e+02 | 1.66e+00 | 1.66e+00 | 1.73e+03 | 1.40e+02 | 0.00e+00 |
| sector-6412-1000 | 1.6e+02 | 4.17e+01 | 4.17e+01 | 1.09e+02 | 5.95e+01 | 0.00e+00 |
| E2006-2000-1000 | 2.0e+02 | 6.38e-01 | 6.38e-01 | 1.15e+03 | 5.00e+02 | 0.00e+00 |
| MNIST-60000-784 | -3.1e+05 | 2.01e+04 | 3.13e+05 | 0.00e+00 | 2.08e+03 | 6.25e+04 |
| Gisette-3000-1000 | -1.0e+06 | 1.64e+04 | 1.98e+04 | 1.15e+04 | 0.00e+00 | 7.31e+04 |
| CnnCaltech-3000-1000 | -4.7e+02 | 1.05e+03 | 3.20e+02 | 1.45e+03 | 2.66e+02 | 0.00e+00 |
| Cifar-1000-1000 | -1.2e+05 | 0.00e+00 | 8.67e+03 | 1.20e+04 | 6.17e+03 | 1.33e+04 |
| randn-500-1000 | -6.3e+03 | 1.09e+03 | 9.71e+02 | 8.90e+02 | 1.29e+03 | 0.00e+00 |

Figure: (a) w1a-2477-300
Refer to caption: x1.png

$\blacktriangleright$  Data Sets. To generate the data matrix $\mathbf{A}$, we consider 10 publicly available real-world or random data sets: ‘w1a’, ‘TDT2’, ‘20News’, ‘sector’, ‘E2006’, ‘MNIST’, ‘Gisette’, ‘Caltech’, ‘Cifar’, ‘randn’. We randomly select a subset of examples from the original data set. The size of $\mathbf{A}\in\mathbb{R}^{m\times n}$ are are chosen from the following set $(m,n)$$\in$$\{(2477,300)$, $(500,1000)$, $(8000,1000)$, $(6412,1000)$, $(2000,1000)$, $(60000,784)$, $(3000,1000)$, $(1000,1000)$, $(500,1000)\}$.

$\blacktriangleright$  Compared Methods. We compare with two existing operator splitting methods: Linearized ADMM (LADMM) Lai & Osher (); He & Yuan () and Smoothing Penalty Method (SPM) Lai & Osher (); Chen (), initialized differently with random and identity matrices, resulting in four variants: LADMM(id), SPM(id), LADMM(rnd), and SPM(rnd). We use a random strategy to find the working set for OBCD, initializing it with the identity matrix, resulting in OBCD-R(id).

$\blacktriangleright$  Implementations. All methods are implemented in MATLAB on an Intel 2.6 GHz CPU with 32 GB RAM. However, our breakpoint searching procedure is developed in C++ and integrated into the MATLAB environment (^2^22Though we prioritize accuracy over speed, the comparisons remain fair despite using different programming languages. The other methods, based on matrix multiplication and SVD, utilize highly optimized BLAS and LAPACK libraries for the computational platform and compilation architecture.), as it requires inefficient element-wise loops in native MATLAB. Code to reproduce the experiments can be found in the supplemental material.

$\blacktriangleright$  Experiment Settings. We compare objective values $(F(\mathbf{X})-F_{\text{min}})$ for different methods after running for 30 seconds, where $F_{\text{min}}$ represents the smallest objective among all methods. For numerical stability in reporting the objectives, we use the count of elements with absolute values greater than a threshold of $10^{-6}$ instead of the original $\ell_{0}$ norm function $\|\mathbf{X}\|_{0}$. We set $\alpha=10^{-5}$ for OBCD. Full-gradient methods have higher per-iteration complexity but require fewer iterations, while OBCD, as a partial-gradient method, has lower per-iteration costs but needs more iterations. Thus, we compare based on CPU time rather than iteration count.

$\blacktriangleright$  Experiment Results. Table 1 and Figure 1 display accuracy and computational efficiency results for $L_{0}$ norm-based SPCA, yielding the following observations: (i) OBCD-R generally delivers the best performance. (ii) Unlike other methods where objectives fluctuate during iterations, OBCD-R monotonically decreases the objective function while maintaining the orthogonality constraint. This is because OBCD is a greedy descent method for this problem class. (iii) While other methods often get stuck in poor local minima, OBCD-R escapes from such minima and generally finds lower objectives, aligning with our theory that our methods locate stronger stationary points.

## 7 Conclusions

In this paper, we introduced OBCD, a new block coordinate descent method for nonsmooth composite optimization under orthogonality constraints. OBCD operates on $k$ rows of the solution matrix, offering lower computational complexity per iteration for $k\geq 2$. Notably, OBCD is the first greedy descent method with monotonicity for this problem class. We also provide a novel optimality analysis, showing how OBCD exploits problem structure to escape bad local minima and find better stationary points than methods focused on critical points. Under the Kurdyka-Lojasiewicz (KL) inequality, we establish strong limit-point convergence. Additionally, we present two extensions: efficient subproblem solvers for $k=2$ and new greedy strategies for working set selection. Extensive experiments demonstrate that OBCD outperforms existing methods.

## Appendix A Notations, Technical Preliminaries, and Relevant Lemmas

### A.1 Notations

Throughout this paper, $\mathcal{M}\triangleq\mathrm{St}(n,r)$ denotes the Stiefel manifold, which is an embedded submanifold of the Euclidean space $\mathbb{R}^{n\times r}$. Boldfaced lowercase letters denote vectors and uppercase letters denote real-valued matrices. We adopt the Matlab colon notation to denote indices that describe submatrices. For given natual numbers $n$ and $k$, we use $\{\mathcal{B}_{1},\mathcal{B}_{2},...,\mathcal{B}_{\mathrm{C}_{n}^{k}}\}$ to denote all the possible combinations of the index vectors choosing $k$ items from $n$ without repetition, where $\mathrm{C}_{n}^{k}$ is the total number of such combinations and $\mathcal{B}_{i}\in\mathbb{N}^{k},~{}\forall i\in[\mathrm{C}_{n}^{k}]$. For any one-dimensional function $p(t):\mathbb{R}\mapsto\mathbb{R}$, we define: $p(\pm x\mp y)\triangleq\min\{p(x-y),p(-x+y)\}$. We use the following notations in this paper.

- •
$[n]$: $\{1,2,...,n\}$
- •
$\|\mathbf{x}\|$: Euclidean norm: $\|\mathbf{x}\|=\|\mathbf{x}\|_{2}=\sqrt{\langle\mathbf{x},\mathbf{x}\rangle}$
- •
$\mathbf{x}_{i}$: the $i$-th element of vector $\mathbf{x}$
- •
$\mathbf{X}_{i,j}$ or $\mathbf{X}_{ij}$ : the ($i^{\text{th}}$, $j^{\text{th}}$) element of matrix $\mathbf{X}$
- •
$\mathrm{vec}(\mathbf{X})$ : $\mathrm{vec}(\mathbf{X})\in\mathbb{R}^{nr\times 1}$, the vector formed by stacking the column vectors of $\mathbf{X}$
- •
$\mathrm{mat}(\mathbf{x})$ : $\mathrm{mat}(\mathbf{x})\in\mathbb{R}^{n\times r}$, Convert $\mathbf{x}\in\mathbb{R}^{nr\times 1}$ into a matrix with $\mathrm{mat}(\mathrm{vec}(\mathbf{X}))=\mathbf{X}$
- •
$\mathbf{X}^{\mathsf{T}}$ : the transpose of the matrix $\mathbf{X}$
- •
$\mathrm{sign}(t)$ : the signum function, $\mathrm{sign}(t)=1$ if $t\geq 0$ and $\mathrm{sign}(t)=-1$ otherwise
- •
$\text{det}(\mathbf{D})$ : Determinant of a square matrix $\mathbf{D}\in\mathbb{R}^{n\times n}$
- •
$\mathrm{C}_{n}^{k}$ : the number of possible combinations choosing $k$ items from $n$ without repetition
- •
$\mathbf{0}_{n,r}$ : A zero matrix of size $n\times r$; the subscript is omitted sometimes
- •
$\mathbf{I}_{r}$ : $\mathbf{I}_{r}\in\mathbb{R}^{r\times r}$, Identity matrix
- •
$\mathbf{X}\succeq\mathbf{0}(\text{or}~{}\succ\mathbf{0})$ : the Matrix $\mathbf{X}$ is symmetric positive semidefinite (or definite)
- •
${\rm{tr}}(\mathbf{A})$ : Sum of the elements on the main diagonal $\mathbf{X}$: ${\rm{tr}}(\mathbf{A})=\sum_{i}\mathbf{A}_{i,i}$
- •
$\langle\mathbf{X},\mathbf{Y}\rangle$ : Euclidean inner product, i.e., $\langle\mathbf{X},\mathbf{Y}\rangle=\sum_{ij}{\mathbf{X}_{ij}\mathbf{Y}_{ij}}$
- •
$\mathbf{X}\otimes\mathbf{Y}$ : Kronecker product of $\mathbf{X}$ and $\mathbf{Y}$
- •
$\|\mathbf{X}\|$ : Operator/Spectral norm: the largest singular value of $\mathbf{X}$
- •
$\|\mathbf{X}\|_{\mathsf{F}}$ : Frobenius norm: $(\sum_{ij}{\mathbf{X}_{ij}^{2}})^{1/2}$
- •
$\nabla f(\mathbf{X})$ : Euclidean gradient of $f(\mathbf{X})$ at $\mathbf{X}$
- •
$\nabla_{\mathcal{M}}f(\mathbf{X})$ : Riemannian gradient of $f(\mathbf{X})$ at $\mathbf{X}$
- •
$\partial F(\mathbf{X})$ : limiting Euclidean subdifferential of $F(\mathbf{X})$ at $\mathbf{X}$
- •
$\partial_{\mathcal{M}}F(\mathbf{X})$ : limiting Riemannian subdifferential of $F(\mathbf{X})$ at $\mathbf{X}$
- •
$\mathcal{I}_{\Xi}(\mathbf{X})$ : the indicator function of a set $\Xi$ with $\mathcal{I}_{\Xi}(\mathbf{X})=0$ if $\mathbf{X}\in\Xi$ and otherwise $+\infty$
- •
$\mathbb{P}_{\Xi}(\mathbf{Z})$ : Orthogonal projection of $\mathbf{Z}$ with $\mathbb{P}_{\Xi}(\mathbf{Z})=\arg\min_{\mathbf{X}\in\Xi}\|\mathbf{Z}-\mathbf{X
}\|_{\mathsf{F}}^{2}$
- •
$\mathbb{P}_{\mathcal{M}}(\mathbf{Y})$ : Nearest orthogonal matrix of $\mathbf{Y}$ with $\mathbb{P}_{\mathcal{M}}(\mathbf{Y})=\arg\min_{\mathbf{X}^{\mathsf{T}}\mathbf{
X}=\mathbf{I}_{r}}\|\mathbf{X}-\mathbf{Y}\|_{\mathsf{F}}^{2}$
- •
$\text{dist}(\Xi,\Xi^{\prime})$ : the distance
between two sets with $\text{dist}(\Xi,\Xi^{\prime})\triangleq\inf_{\mathbf{X}\in\Xi,\mathbf{X}^{
\prime}\in\Xi^{\prime}}\|\mathbf{X}-\mathbf{X}^{\prime}\|_{\mathsf{F}}$
- •
$\mathcal{I}_{\geq 0}(\mathbf{X})$: indicator function of non-negativity constraint with $\mathcal{I}_{\geq 0}(\mathbf{X})={\{\begin{array}[]{ll}0,&\hbox{$\mathbf{X}
\geq\mathbf{0}$;}\\
\infty,&\hbox{else.}\end{array}\}}$
- •
$\|\mathbf{X}\|_{0}$: the number of non-zero elements in the matrix $\mathbf{X}$
- •
$\|\mathbf{X}\|_{1}$: the absolute sum of the elements in the matrix $\mathbf{X}$ with $\|\mathbf{X}\|_{1}=\sum_{i,j}|\mathbf{X}|_{i,j}$
- •
$\mathbb{A}+\mathbb{B}$, $\mathbb{A}-\mathbb{B}$: standard Minkowski addition and subtraction between sets $\mathbb{A}$ and $\mathbb{B}$
- •
$\mathbb{A}\oplus\mathbb{B}$, $\mathbb{A}\ominus\mathbb{B}$: element-wise addition and subtraction between sets $\mathbb{A}$ and $\mathbb{B}$
- •
$\|\partial F(\mathbf{X})\|_{\mathsf{F}}$: the distance from the origin $\mathbf{0}$ to the boundary of the set $\partial F(\mathbf{X})$ with $\|\partial F(\mathbf{X})\|_{\mathsf{F}}=\inf_{\mathbf{Y}\in\partial F(\mathbf{
X})}\|\mathbf{Y}\|_{\mathsf{F}}={\rm{dist}}(\mathbf{0},\partial F(\mathbf{X}))$

### A.2 Technical Preliminaries

As the function $F(\cdot)$ can be non-convex and non-smooth, we introduce some tools in non-smooth analysis Mordukhovich (); Rockafellar & Wets. (). The domain of any extended real-valued function $F:\mathbb{R}^{n\times r}\rightarrow(-\infty,+\infty]$ is defined as $\text{dom}(F)\triangleq\{\mathbf{X}\in\mathbb{R}^{n\times r}:|F(\mathbf{X})|<+\infty\}$. The Fréchet subdifferential of $F$ at $\mathbf{X}\in\text{dom}(F)$ is defined as $\hat{\partial}{F}(\mathbf{X})\triangleq\{\bm{\xi}\in\mathbb{R}^{n\times r}:
\lim_{\mathbf{Z}\rightarrow\mathbf{X}}\inf_{\mathbf{Z}\neq\mathbf{X}}\tfrac{{F
}(\mathbf{Z})-{F}(\mathbf{X})-\langle\bm{\xi},\mathbf{Z}-\mathbf{X}\rangle}{\|
\mathbf{Z}-\mathbf{X}\|_{\mathsf{F}}}\geq 0\}$, while the limiting subdifferential of ${F}(\mathbf{X})$ at $\mathbf{X}\in\text{dom}({F})$ is denoted as $\partial{F}(\mathbf{X})\triangleq\{\bm{\xi}\in\mathbb{R}^{n}:\exists\mathbf{X}
^{t}\rightarrow\mathbf{X},{F}(\mathbf{X}^{t})\rightarrow{F}(\mathbf{X}),\bm{
\xi}^{t}\in\hat{\partial}{F}(\mathbf{X}^{t})\rightarrow\bm{\xi},\forall t\}$. We denote $\nabla{F}(\mathbf{X})$ as the gradient of $F(\cdot)$ at $\mathbf{X}$ in the Euclidean space. We have the following relation between $\hat{\partial}{F}(\mathbf{X})$, $\partial{F}(\mathbf{X})$, and $\nabla{F}(\mathbf{X})$. (i) It holds that $\hat{\partial}{F}(\mathbf{X})\subseteq\partial{F}(\mathbf{X})$. (ii) If the function $F(\cdot)$ is convex, $\partial{F}(\mathbf{X})$ and $\hat{\partial}{F}(\mathbf{X})$ essentially the classical subdifferential for convex functions, i.e., $\partial{F}(\mathbf{X})=\hat{\partial}{F}(\mathbf{X})=\{\bm{\xi}\in\mathbb{R}^
{n\times r}:F(\mathbf{Z})\geq F(\mathbf{X})+\langle\bm{\xi},\mathbf{Z}-\mathbf
{X}\rangle,\forall\mathbf{Z}\in\mathbb{R}^{n\times r}\}$. (iii) If the function $F(\cdot)$ is differentiable, then $\hat{\partial}{F}(\mathbf{X})=\partial{F}(\mathbf{X})=\{\nabla F(\mathbf{X})\}$.

We need some prerequisite knowledge in optimization with orthogonality constraints Absil et al. (). The nearest orthogonality matrix to an arbitrary matrix $\mathbf{Y}\in\mathbb{R}^{n\times r}$ is given by $\mathbb{P}_{\mathcal{M}}(\mathbf{Y})=\hat{\mathbf{U}}\hat{\mathbf{V}}^{\mathsf
{T}}$, where $\mathbf{Y}=\hat{\mathbf{U}}\mathrm{Diag}(\mathbf{s})\hat{\mathbf{V}}^{\mathsf{
T}}$ is the singular value decomposition of $\mathbf{Y}$. We use $\mathcal{N}_{\mathcal{M}}(\mathbf{X})$ to denote the limiting normal cone to $\mathcal{M}$ at $\mathbf{X}$, leading to $\mathcal{N}_{\mathcal{M}}(\mathbf{X})=\partial\mathcal{I}_{\mathcal{M}}(
\mathbf{X})=\{\mathbf{Z}\in\mathbb{R}^{n\times r}:\langle\mathbf{Z},\mathbf{X}
\rangle\geq\langle\mathbf{Z},\mathbf{Y}\rangle,\,\forall\mathbf{Y}\in\mathcal{
M}\}$. The tangent and norm space to $\mathcal{M}$ at $\mathbf{X}\in\mathcal{M}$ are denoted as $\mathrm{T}_{\mathbf{X}}\mathcal{M}$ and $\mathrm{N}_{\mathbf{X}}\mathcal{M}$, respectively. For a given $\mathbf{X}\in\mathcal{M}$, we let $\mathcal{A}_{\mathbf{X}}(\mathbf{Y})\triangleq\mathbf{X}^{\mathsf{T}}\mathbf{Y
}+\mathbf{Y}^{\mathsf{T}}\mathbf{X}$ for $\mathbf{Y}\in\mathbb{R}^{n\times r}$, and we have $\mathrm{T}_{\mathbf{X}}\mathcal{M}=\{\mathbf{Y}\in\mathbb{R}^{n\times r}|
\mathcal{A}_{X}(\mathbf{Y})=\mathbf{0}\}$ and $\mathrm{N}_{\mathbf{X}}\mathcal{M}=2\mathbf{X}\mathbf{\Lambda}\,|\,\mathbf{
\Lambda}=\mathbf{\Lambda}^{\mathsf{T}},\mathbf{\Lambda}\in\mathbb{R}^{r\times r}\}$. For any non-convex and non-smooth function $F(\mathbf{X})$, we use $\partial_{\mathcal{M}}F(\mathbf{X})$ to denote the limiting Riemannian gradient of $F(\mathbf{X})$ at $\mathbf{X}$, and obtain $\partial_{\mathcal{M}}F(\mathbf{X})=\mathbb{P}_{\mathrm{T}_{\mathbf{X}}
\mathcal{M}}(\partial F(\mathbf{X}))$. We denote $\partial F(\mathbf{X})\ominus\mathbf{X}[\partial F(\mathbf{X})]^{\mathsf{T}}
\mathbf{X}\triangleq\{\mathbb{E}\,|\,\mathbb{E}=\mathbf{G}-\mathbf{X}\mathbf{G
}^{\mathsf{T}}\mathbf{X},\mathbf{G}\in\partial F(\mathbf{X})\}$.

### A.3 Relevant Lemmas

We offer a set of useful lemmas, each of which stands independently of context and specific methodology.

###### Lemma A.1 .

Let $k\geq 2$ and $\mathbf{W}\in\mathbb{R}^{n\times n}$. If $\mathbf{0}_{k,k}=\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{W}\mathrm{U}_{
\texttt{{B}}}$ for all $\texttt{{B}}\in\{\mathcal{B}_{i}\}_{i=1}^{\mathrm{C}_{n}^{k}}$, then $\mathbf{W}=\mathbf{0}$. Here, the set $\{\mathcal{B}_{1},\mathcal{B}_{2},...,\mathcal{B}_{\mathrm{C}_{n}^{k}}\}$ represents all possible combinations of the index vectors choosing $k$ items from $n$ without repetition.

###### Proof.

This result is based on elementary deductions. Notably, the conclusion of this lemma does not necessarily hold if $|\texttt{{B}}|=k=1$. This is because any matrix $\mathbf{W}\in\mathbb{R}^{n\times n}$ with $\mathbf{W}_{ii}=0$ for all $i\in[n]$ satisfies the condition of this lemma but is not necessary a zero matrix.
∎

###### Lemma A.2 .

For any matrices $\mathbf{A}\in\mathbb{R}^{k\times k}$ and $\mathbf{C}\in\mathbb{R}^{k\times k}$, we have: $\|\mathbf{A}-\mathbf{A}^{\mathsf{T}}\|_{\mathsf{F}}\leq 2\|\mathbf{A}-\mathbf{
C}\|_{\mathsf{F}}+\|\mathbf{C}-\mathbf{C}^{\mathsf{T}}\|_{\mathsf{F}}$.

###### Proof.

We derive: $\|\mathbf{A}-\mathbf{A}^{\mathsf{T}}\|_{\mathsf{F}}=\|(\mathbf{A}-\mathbf{C})+
(\mathbf{C}-\mathbf{C}^{\mathsf{T}})+(\mathbf{C}^{\mathsf{T}}-\mathbf{A}^{
\mathsf{T}})\|_{\mathsf{F}}\overset{\text{\char 172}}{\leq}\|\mathbf{A}-
\mathbf{C}\|_{\mathsf{F}}+\|\mathbf{C}-\mathbf{C}^{\mathsf{T}}\|_{\mathsf{F}}+
\|\mathbf{C}^{\mathsf{T}}-\mathbf{A}^{\mathsf{T}}\|_{\mathsf{F}}=2\|\mathbf{A}
-\mathbf{C}\|_{\mathsf{F}}+\|\mathbf{C}-\mathbf{C}^{\mathsf{T}}\|_{\mathsf{F}}$, where step ① uses the triangle inequality.

∎

###### Lemma A.3 .

Let $\tau\in\mathbb{R}$, and $\mathbf{A}\in\mathbb{R}^{2\times 2}$ be any skey-symmetric matrix with $\mathbf{A}^{\mathsf{T}}=-\mathbf{A}$. The matrix $\mathbf{Q}=[(\mathbf{I}_{2}+\tfrac{\tau}{2}\mathbf{A})^{-1}(\mathbf{I}_{k}-
\tfrac{\tau}{2}\mathbf{A})]$ is always a rotation matrix with $\det(\mathbf{Q})=1$.

###### Proof.

Since $\mathbf{A}$ is a two-dimensional matrix, it can be expressed in the form: $\mathbf{A}=(\begin{smallmatrix}0&a\\
-a&0\end{smallmatrix})$ for some $a\in\mathbb{R}$. Letting $b=\tfrac{\tau}{2}a$, we derive:

$$ $\displaystyle\mathbf{Q}=(\mathbf{I}_{2}+\tfrac{\tau}{2}\mathbf{A})^{-1}( \mathbf{I}_{k}-\tfrac{\tau}{2}\mathbf{A})\overset{\text{\char 172}}{=}\left( \begin{smallmatrix}1&b\\ -b&1\\ \end{smallmatrix}\right)^{-1}\left(\begin{smallmatrix}1&-b\\ b&1\\ \end{smallmatrix}\right)\overset{\text{\char 173}}{=}\tfrac{1}{1+b^{2}}\left( \begin{smallmatrix}1&-b\\ b&1\\ \end{smallmatrix}\right)\left(\begin{smallmatrix}1&-b\\ b&1\\ \end{smallmatrix}\right)=\tfrac{1}{1+b^{2}}\left(\begin{smallmatrix}1-b^{2}&-2 b\\ 2b&1-b^{2}\\ \end{smallmatrix}\right),$ $$

where step ① uses $\tfrac{\tau}{2}\mathbf{A}=(\begin{smallmatrix}0&b\\
-b&0\end{smallmatrix})$; step ② uses the fact that $(\begin{smallmatrix}a&b\\
c&d\end{smallmatrix})^{-1}=\tfrac{1}{ad-bc}\left(\begin{smallmatrix}d&-b\\
-c&a\\
\end{smallmatrix}\right)^{-1}$ for all $a,b,c,d\in\mathbb{R}$. We further obtain: $\det(\mathbf{Q})\overset{\text{\char 172}}{=}\tfrac{1-b^{2}}{1+b^{2}}\cdot
\tfrac{1-b^{2}}{1+b^{2}}-\tfrac{2b}{1+b^{2}}\cdot\tfrac{-2b}{1+b^{2}}=\tfrac{(
1-b^{2})^{2}+4b^{2}}{(1+b^{2})^{2}}=\tfrac{(1+b^{2})^{2}}{(1+b^{2})^{2}}=1$, where step ① uses the fact that $\det(\begin{smallmatrix}a&b\\
c&d\end{smallmatrix})=ad-bc$ for all $a,b,c,d\in\mathbb{R}$.

∎

###### Lemma A.4 .

For any $\mathbf{W}\in\mathbb{R}^{n\times n}$, we have $\sum_{i=1}^{\mathrm{C}_{n}^{k}}\|\mathbf{W}(\mathcal{B}_{i},\mathcal{B}_{i})\|
_{\mathsf{F}}^{2}=\mathrm{C}_{n-2}^{k-2}\sum_{i}\sum_{j,j\neq i}\mathbf{W}_{ij
}^{2}+\tfrac{k}{n}\mathrm{C}_{n}^{k}\sum_{i}\mathbf{W}_{ii}^{2}$. Here, the set $\{\mathcal{B}_{1},\mathcal{B}_{2},...,\mathcal{B}_{\mathrm{C}_{n}^{k}}\}$ represents all possible combinations of the index vectors choosing $k$ items from $n$ without repetition.

###### Proof.

For any matrix $\mathbf{W}\in\mathbb{R}^{n\times n}$, we define: $\textstyle\mathbf{w}\triangleq\mathrm{diag}(\mathbf{W})\in\mathbb{R}^{n}$, and $\mathbf{W}^{\prime}\triangleq\mathbf{W}-\mathrm{Diag}(\mathbf{w})$.

We have: $\mathbf{W}=\mathrm{Diag}(\mathbf{w})+\mathbf{W}^{\prime}$, this leads to the following decomposition:

$$ $\displaystyle\textstyle\sum_{i=1}^{\mathrm{C}_{n}^{k}}\|\mathbf{U}_{\mathcal{B }_{i}}^{\mathsf{T}}\mathbf{W}\mathbf{U}_{\mathcal{B}_{i}}\|_{\mathsf{F}}^{2}$ $\displaystyle=$ $\displaystyle\textstyle\sum_{i=1}^{\mathrm{C}_{n}^{k}}\|\mathbf{U}_{\mathcal{B }_{i}}^{\mathsf{T}}(\mathrm{Diag}(\mathbf{w})+\mathbf{W}^{\prime})\mathbf{U}_{ \mathcal{B}_{i}}\|_{\mathsf{F}}^{2}$ (16) $\displaystyle=$ $\displaystyle\textstyle\underbrace{\textstyle\sum_{i=1}^{\mathrm{C}_{n}^{k}}\| \mathbf{U}_{\mathcal{B}_{i}}^{\mathsf{T}}\mathrm{Diag}(\mathbf{w})\mathbf{U}_{ \mathcal{B}_{i}}\|_{\mathsf{F}}^{2}}_{\Gamma_{1}}~{}~{}+~{}~{}\underbrace{ \textstyle\sum_{i=1}^{\mathrm{C}_{n}^{k}}\|\mathbf{U}_{\mathcal{B}_{i}}^{ \mathsf{T}}\mathbf{W}^{\prime}\mathbf{U}_{\mathcal{B}_{i}}\|_{\mathsf{F}}^{2}} _{\Gamma_{2}}.$ $$

We first focus on the term $\Gamma_{1}$. We have:

$$ $\displaystyle\Gamma_{1}=\textstyle\sum_{i=1}^{\mathrm{C}_{n}^{k}}\|\mathbf{U}_ {\mathcal{B}_{i}}^{\mathsf{T}}\mathrm{Diag}(\mathbf{w})\mathbf{U}_{\mathcal{B} _{i}}\|_{\mathsf{F}}^{2}\overset{\text{\char 172}}{=}\textstyle\sum_{i=1}^{ \mathrm{C}_{n}^{k}}\|\mathbf{w}_{\mathcal{B}_{i}}\|_{2}^{2}\overset{\text{ \char 173}}{=}\mathrm{C}_{n}^{k}\cdot\tfrac{k}{n}\cdot\|\mathbf{w}\|_{2}^{2}= \textstyle\mathrm{C}_{n}^{k}\cdot\tfrac{k}{n}\cdot\sum_{i}\mathbf{W}_{ii}^{2},$ (17) $$

where step ① uses the fact that $\|\texttt{{B}}^{\mathsf{T}}\mathrm{Diag}(\mathbf{w})\texttt{{B}}\|_{\mathsf{F}
}^{2}=\|[\mathrm{Diag}(\mathbf{w})]_{\texttt{{B}}\texttt{{B}}}\|_{\mathsf{F}}^
{2}=\|\mathbf{w}_{\texttt{{B}}}\|_{2}^{2}$ for any $\texttt{{B}}\in$ $\{\mathcal{B}_{i}\}$${}_{i=1}^{\mathrm{C}_{n}^{k}}~{}$; step ② uses the observation that $\mathbf{w}_{i}$ appears in the term $\textstyle\sum_{i=1}^{\mathrm{C}_{n}^{k}}\|\mathbf{w}_{\mathcal{B}_{i}}\|_{2}^
{2}$ a total of $(\mathrm{C}_{n}^{k}\cdot\tfrac{k}{n})$ times, which can be deduced using basic induction.

We now focus on the term $\Gamma_{2}$. Noticing that $\mathbf{W}^{\prime}_{ii}=0$ for all $i\in[n]$, we have:

$$ $\displaystyle\Gamma_{2}=\textstyle\sum_{i=1}^{\mathrm{C}_{n}^{k}}\|\mathbf{U}_ {\mathcal{B}_{i}}^{\mathsf{T}}\mathbf{W}^{\prime}\mathbf{U}_{\mathcal{B}_{i}} \|_{\mathsf{F}}^{2}\overset{\text{\char 172}}{=}\textstyle\sum_{i}\sum_{j\neq i }[\mathrm{C}_{n-2}^{k-2}(\mathbf{W}^{\prime}_{ij})^{2}]\overset{\text{\char 17 3}}{=}\textstyle\mathrm{C}_{n-2}^{k-2}\sum_{i}\sum_{j,j\neq i}(\mathbf{W}_{ij} )^{2},$ (18) $$

where step ① uses the fact that the term $\sum_{i=1}^{\mathrm{C}_{n}^{k}}\|\mathbf{U}_{\mathcal{B}_{i}}^{\mathsf{T}}
\mathbf{W}^{\prime}\mathbf{U}_{\mathcal{B}_{i}}\|_{\mathsf{F}}^{2}$ comprises $\mathrm{C}_{n-2}^{k-2}$ distinct patterns, each including $\{i,j\}$ with $i\neq j$; step ② uses $\sum_{i,j\neq i}(\mathbf{W}_{ij})^{2}=\sum_{i,j\neq i}(\mathbf{W}^{\prime}_{ij
})^{2}$.

In view of Equalities (16), (17), and (18), we complete the proof of this lemma.
∎

###### Lemma A.5 .

Assume $\mathbf{Q}\mathbf{R}=\mathbf{X}\in\mathbb{R}^{n\times n}$, where $\mathbf{Q}\in\mathrm{St}(n,n)$ and $\mathbf{R}$ is a lower triangular matrix with $\mathbf{R}_{i,j}=0$ for all $i<j$. If $\mathbf{X}\in\mathrm{St}(n,n)$, then $\mathbf{R}$ is a diagonal matrix with $\mathbf{R}_{i,i}\in\{-1,+1\}$ for all $i\in[n]$.

###### Proof.

We derive: $\mathbf{R}\mathbf{R}^{\mathsf{T}}\overset{\text{\char 172}}{=}(\mathbf{Q}
\mathbf{X})(\mathbf{Q}\mathbf{X})^{\mathsf{T}}=\mathbf{Q}\mathbf{X}\mathbf{X}^
{\mathsf{T}}\mathbf{Q}^{\mathsf{T}}\overset{\text{\char 173}}{=}\mathbf{I}$, where step ① uses $\mathbf{R}=\mathbf{Q}^{\mathsf{T}}\mathbf{X}$; step ② uses $\mathbf{X}\in\mathrm{St}(n,n)$ and $\mathbf{Q}\in\mathrm{St}(n,n)$. First, given $\|\mathbf{R}(1,:)\|=1$ and $\mathbf{R}(1,2:n)=0$, we have $\mathbf{R}_{1,1}\in\{-1,+1\}$. Second, we have $\|\mathbf{R}(2,:)\|=1$ and $\mathbf{R}(1,:)^{\mathsf{T}}\mathbf{R}(:,2)=0$, leading to $\mathbf{R}_{1,2}=0$ and $\mathbf{R}_{2,2}\in\{-1,+1\}$. Finally, using similar recursive strategy, we conclude that $\mathbf{R}$ is a diagonal matrix with $\mathbf{R}_{i,i}\in\{-1,+1\}$ for all $i\in[n]$.
∎

###### Lemma A.6 .

We define $\mathrm{T}_{\mathbf{X}}\mathcal{M}\triangleq\{\mathbf{Y}\in\mathbb{R}^{n\times
r
}\,|\,\mathcal{A}_{X}(\mathbf{Y})=\mathbf{0}\}$ and $\mathcal{A}_{\mathbf{X}}(\mathbf{Y})\triangleq\mathbf{X}^{\mathsf{T}}\mathbf{Y
}+\mathbf{Y}^{\mathsf{T}}\mathbf{X}$.
For any $\mathbf{G}\in\mathbb{R}^{n\times r}$ and $\mathbf{X}\in\mathrm{St}(n,k)$, we have: $(\mathbf{G}-\tfrac{1}{2}\mathbf{X}\mathcal{A}_{\mathbf{X}}(\mathbf{G}))=\arg
\min_{\mathbf{Y}\in\mathrm{T}_{\mathbf{X}}\mathcal{M}}\|\mathbf{Y}-\mathbf{G}
\|^{2}_{\mathsf{F}}$.

###### Proof.

The conclusion of this lemma can be found in Absil et al. (). For completeness, we present a short proof.

Consider the convex problem: $\bar{\mathbf{Y}}=\arg\min_{\mathbf{Y}}\|\mathbf{Y}-\mathbf{G}\|_{\mathsf{F}}^{
2},\,s.t.\,\mathbf{X}^{\mathsf{T}}\mathbf{Y}+\mathbf{Y}^{\mathsf{T}}\mathbf{X}
=\mathbf{0}$. Introducing a multiplier $\mathbf{\Lambda}\in\mathbb{R}^{r\times r}$ for the linear constraints leads to the following Lagrangian function: $\tilde{\mathcal{L}}(\mathbf{Y};\mathbf{\Lambda})=\|\mathbf{Y}-\mathbf{G}\|_{
\mathsf{F}}^{2}+\langle\mathbf{X}^{\mathsf{T}}\mathbf{Y}+\mathbf{Y}^{\mathsf{T
}}\mathbf{X},\mathbf{\Lambda}\rangle$. We derive the subsequent first-order optimality condition: $2(\mathbf{Y}-\mathbf{G})+\mathbf{X}(\mathbf{\Lambda}+\mathbf{\Lambda}^{\mathsf
{T}})=\mathbf{0}$, and $\mathbf{X}^{\mathsf{T}}\mathbf{Y}+\mathbf{Y}^{\mathsf{T}}\mathbf{X}=\mathbf{0}$. Given $\mathbf{\Lambda}$ is symmetric, we have $\mathbf{Y}=\mathbf{G}-\mathbf{X}\mathbf{\Lambda}$. Incorporating this result into $\mathbf{X}^{\mathsf{T}}\mathbf{Y}+\mathbf{Y}^{\mathsf{T}}\mathbf{X}=\mathbf{0}$, we obtain: $\mathbf{X}^{\mathsf{T}}(\mathbf{G}-\mathbf{X}\mathbf{\Lambda})+(\mathbf{G}-
\mathbf{X}\mathbf{\Lambda})^{\mathsf{T}}\mathbf{X}=\mathbf{0}$. Given $\mathbf{X}\in\mathrm{St}(n,r)$, we have $\mathbf{X}^{\mathsf{T}}\mathbf{G}-\mathbf{\Lambda}+\mathbf{G}^{\mathsf{T}}
\mathbf{X}-\mathbf{\Lambda}^{\mathsf{T}}=\mathbf{0}$, leading to: $\mathbf{\Lambda}=\tfrac{1}{2}(\mathbf{X}^{\mathsf{T}}\mathbf{G}+\mathbf{G}^{
\mathsf{T}}\mathbf{X})$. Therefore, the optimal solution $\bar{\mathbf{Y}}$ can be computed as $\bar{\mathbf{Y}}=\mathbf{G}-\mathbf{X}\mathbf{\Lambda}=\mathbf{G}-\tfrac{1}{2}
\mathbf{X}(\mathbf{X}^{\mathsf{T}}\mathbf{G}+\mathbf{G}^{\mathsf{T}}\mathbf{X})$.

∎

###### Lemma A.7 .

Consider the following problem: $\min_{\mathbf{X}}\,F^{\circ}(\mathbf{X})\triangleq F(\mathbf{X})+\mathcal{I}_{
\mathcal{M}}(\mathbf{X})$, where $F(\mathbf{X})$ is defined in Equation (1). For any $\mathbf{X}\in\mathrm{St}(n,r)$, it holds that ${\rm{dist}}(\mathbf{0},\partial F^{\circ}(\mathbf{X}))\leq{\rm{dist}}(\mathbf{
0},\partial_{\mathcal{M}}F(\mathbf{X}))$.

###### Proof.

We let $\mathbf{G}\in\partial F(\mathbf{X})$ and define $\mathcal{A}_{\mathbf{X}}(\mathbf{G})\triangleq\mathbf{X}^{\mathsf{T}}\mathbf{G
}+\mathbf{G}^{\mathsf{T}}\mathbf{X}$.

Recall that the following first-order optimality conditions are equivalent for all $\mathbf{X}\in\mathrm{St}(n,r)$: $\left(\mathbf{0}\in\partial F^{\circ}(\mathbf{X})\right)\Leftrightarrow\left(
\mathbf{0}\in\mathbb{P}_{\mathrm{T}_{\mathbf{X}}\mathcal{M}}(\partial F(
\mathbf{X}))\right)$. Therefore, we derive the following results:

$$ $\displaystyle\textstyle{\rm{dist}}(\mathbf{0},\partial F^{\circ}(\mathbf{X}))$ $\displaystyle=$ $\displaystyle\textstyle\inf_{\mathbf{Y}\in\partial F^{\circ}(\mathbf{X})}\| \mathbf{Y}\|_{\mathsf{F}}=\inf_{\mathbf{Y}\in\mathbb{P}_{(\mathrm{T}_{\mathbf{ X}}\mathcal{M})}(\partial F(\mathbf{X}))}\|\mathbf{Y}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\textstyle\|\mathbb{P}_{(\mathrm{T}_{\mathbf{X}}\mathcal{M})}( \mathbf{G})\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle\textstyle\|\mathbf{G}-\tfrac{1}{2}\mathbf{X}\mathcal{A}_{\mathbf {X}}(\mathbf{G})\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 174}}{=}$ $\displaystyle\textstyle\|\mathbf{G}-\tfrac{1}{2}\mathbf{X}(\mathbf{X}^{\mathsf {T}}\mathbf{G}+\mathbf{G}^{\mathsf{T}}\mathbf{X})\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 175}}{=}$ $\displaystyle\textstyle\|(\mathbf{I}-\tfrac{1}{2}\mathbf{X}\mathbf{X}^{\mathsf {T}})(\mathbf{G}-\mathbf{X}\mathbf{G}^{\mathsf{T}}\mathbf{X})\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 176}}{\leq}$ $\displaystyle\textstyle\|\mathbf{G}-\mathbf{X}\mathbf{G}^{\mathsf{T}}\mathbf{X }\|_{\mathsf{F}},$ $$

where step ① uses $\mathbf{G}\in\partial F(\mathbf{X})$; step ② uses Lemma A.6; step ③ uses the definition of $\mathcal{A}_{\mathbf{X}}(\mathbf{G})$; step ④ uses the identity that $\mathbf{G}-\tfrac{1}{2}\mathbf{X}(\mathbf{X}^{\mathsf{T}}\mathbf{G}+\mathbf{G}
^{\mathsf{T}}\mathbf{X})=(\mathbf{I}-\tfrac{1}{2}\mathbf{X}\mathbf{X}^{\mathsf
{T}})(\mathbf{G}-\mathbf{X}\mathbf{G}^{\mathsf{T}}\mathbf{X})$; step ⑤ uses the norm inequality and fact that the matrix $\mathbf{I}-\tfrac{1}{2}\mathbf{X}\mathbf{X}^{\mathsf{T}}$ only contains eigenvalues that are $\tfrac{1}{2}$ or $1$.

∎

###### Lemma A.8 .

Assume $\cos(\theta)\neq 0$. Any pair of trigonometric functions $(\cos(\theta),\sin(\theta))$ can be represented as follows:

- a)
$\cos({\theta})=\tfrac{1}{\sqrt{1+\tan^{2}(\theta)}}$, and $\sin({\theta})=\tfrac{\tan(\theta)}{\sqrt{1+\tan^{2}(\theta)}}$.
- b)
$\cos({\theta})=\tfrac{-1}{\sqrt{1+\tan^{2}(\theta)}}$, and $\sin({\theta})=\tfrac{-\tan(\theta)}{\sqrt{1+\tan^{2}(\theta)}}$.

###### Proof.

For all values of $\theta$ where $\cos(\theta)\neq 0$, the trigonometric functions $\{\sin(\theta),\cos(\theta),\tan(\theta)\}$ are well-defined. Utilizing the identity $\sin^{2}(\theta)+\cos^{2}(\theta)=1$ and $\tan(\theta)\cos(\theta)=\sin(\theta)$, we derive: $(\tan(\theta)\cdot\cos(\theta))^{2}+\cos^{2}(\theta)=1$. Consequently, we find: $\cos(\theta)=\tfrac{\pm 1}{\sqrt{\tan^{2}(\theta)+1}}$. Finally, we can express $\sin(\theta)$ as $\sin(\theta)=\tan(\theta)\cdot\cos(\theta)=\tfrac{\tan(\theta)}{\sqrt{\tan^{2}
(\theta)+1}}$.

∎

###### Lemma A.9 .

Let $A\in\mathbb{R}$ and $B\in\mathbb{R}$. The minimizer of the following one-dimensional problem:

$$ $\displaystyle\bar{\theta}\in\arg\min_{\theta}h(\theta)=A\cos(\theta)+B\sin(\theta)$ (19) $$

will be achieved at $\bar{\theta}$, where $\cos(\bar{\theta})=-\tfrac{A}{\sqrt{A^{2}+B^{2}}}$, $\sin(\bar{\theta})=-\tfrac{B}{\sqrt{A^{2}+B^{2}}}$, and $h(\bar{\theta})=-\sqrt{A^{2}+B^{2}}$.

###### Proof.

Initially, we consider the special case when $\cos(\theta)=0$ or $A=0$. Problem (19) reduces to:

$$ $\displaystyle\bar{\theta}\in\arg\min_{\theta}h(\theta)=B\sin(\theta).$ $$

Clearly, we have:
$\sin(\bar{\theta})=-\tfrac{B}{|B|}$, $\cos(\bar{\theta})=0$, and $h(\bar{\theta})=-|B|$. The conclusion of this lemma holds.

We now assume that $A\neq 0$ and $\cos(\theta)\neq 0$ for all $\theta$. Using the fact that $\tan(\theta)=\tfrac{\sin(\theta)}{\cos(\theta)}$ and $\cos(\theta)^{2}+\sin(\theta)^{2}=1$, we have the following two cases for $\cos(\theta)$ and $\sin(\theta)$:

- a)
$\cos(\theta)=\tfrac{1}{\sqrt{1+\tan^{2}(\theta)}}$, and $\sin(\theta)=\tfrac{\tan(\theta)}{\sqrt{1+\tan^{2}(\theta)}}$
- b)
$\cos(\theta)=\tfrac{-1}{\sqrt{1+\tan^{2}(\theta)}}$, and $\sin(\theta)=\tfrac{-\tan(\theta)}{\sqrt{1+\tan^{2}(\theta)}}$.

Therefore, Problem (19) reduces to the following equivalent minimization problem:

$$ $\displaystyle\bar{\theta}\in\arg\min_{\theta}\tfrac{\pm A\pm\tan(\theta)B}{ \sqrt{1+\tan^{2}(\theta)}}.$ $$

Using the variable substitution that $\tan(\theta)=t$, we have the following equivalent problem:

$$ $\displaystyle\bar{t}\in\arg\min_{t}h(t)\triangleq\tfrac{\pm(A+Bt)}{\sqrt{1+t^{ 2}}}.$ $$

For any optimal solution $\bar{t}$, we have the following necessary first-order optimality condition:

$$ $\displaystyle 0\in\partial h(\bar{t})=\tfrac{\pm B\sqrt{1+{\bar{t}}^{2}}\mp(A+ B{\bar{t}})\cdot(1+{\bar{t}}^{2})^{-1/2}\bar{t}}{1+{\bar{t}}^{2}}$ $\displaystyle\Rightarrow$ $\displaystyle 0\in\pm B\sqrt{1+{\bar{t}}^{2}}\mp\tfrac{(A+B{\bar{t}}){\bar{t}} }{\sqrt{1+{\bar{t}}^{2}}}\Rightarrow B\sqrt{1+{\bar{t}}^{2}}=\tfrac{(A+B{\bar{ t}}){\bar{t}}}{\sqrt{1+{\bar{t}}^{2}}}\Rightarrow{\bar{t}}=\tfrac{B}{A}$ $$

Therefore, we have: $\bar{t}=\tfrac{B}{A}=\tan(\bar{\theta})$. The optimal solution pair $[\cos(\bar{\theta}),\sin(\bar{\theta})]$ for Problem (19) can be computed as one of the following two cases:

- a)
$\cos(\bar{\theta})=\tfrac{A}{\sqrt{A^{2}+B^{2}}}$, and $\sin(\bar{\theta})=\tfrac{B}{\sqrt{A^{2}+B^{2}}}.$
- b)
$\cos(\bar{\theta})=\tfrac{-A}{\sqrt{A^{2}+B^{2}}}$, and $\sin(\bar{\theta})=\tfrac{-B}{\sqrt{A^{2}+B^{2}}}.$

In view of the original problem $\bar{\theta}=\arg\min_{\theta}h(\theta)=A\cos(\theta)+B\sin(\theta)$, we conclude that $\cos(\bar{\theta})=\tfrac{-A}{\sqrt{A^{2}+B^{2}}}$, and $\sin(\bar{\theta})=\tfrac{-B}{\sqrt{A^{2}+B^{2}}}$.

∎

###### Lemma A.1 .

###### Proof.

###### Lemma A.2 .

###### Proof.

###### Lemma A.3 .

###### Proof.

###### Lemma A.4 .

###### Proof.

###### Lemma A.5 .

###### Proof.

###### Lemma A.6 .

###### Proof.

###### Lemma A.7 .

###### Proof.

###### Lemma A.8 .

###### Proof.

###### Lemma A.9 .

###### Proof.

## Appendix B Additional Examples of the Breakpoint Searching Method

In this section, we provide additional examples of BSM for other different $h(\mathbf{x})$.

$\blacktriangleright$ Finding the Breakpoint Set for $h(\mathbf{x})\triangleq\lambda\|\mathbf{x}\|_{1}$

Since the function $h(\mathbf{x})\triangleq\lambda\|\mathbf{x}\|_{1}$ is symmetric, Problem (12) reduces to the following problem:

$$ $\displaystyle\bar{t}\in\arg\min_{t}p(t)\triangleq\tfrac{a+bt}{\sqrt{1+t^{2}}}+ \tfrac{w+dt}{1+t^{2}}+\tfrac{\lambda\|\mathbf{x}+t\mathbf{y}\|_{1}}{\sqrt{1+t^ {2}}}.$ (20) $$

Setting the subgradient of $p(\cdot)$ to zero yields: $0\in\partial p(t)=t^{\circ}[d(1+t^{2})-(w+dt)2t+(b-at)\cdot\sqrt{1+t^{2}}]+t^{
\circ}\lambda\cdot\sqrt{1+t^{2}}\cdot[\langle\mathrm{sign}(\mathbf{x}+t\mathbf
{y}),\mathbf{y}\rangle(1+t^{2})-\|\mathbf{x}+t\mathbf{y}\|_{1}t]$, where $t^{\circ}=(1+t^{2})^{-2}$. We consider the following two cases. (i) We assume $(\mathbf{x}+t\mathbf{y})_{i}=0$ for some $i$. Then the solution $\bar{t}$ can be determined using $\bar{t}=\tfrac{\mathbf{x}_{i}}{\mathbf{y}_{i}}$. There are $2r$ breakpoints $\{\tfrac{\mathbf{x}_{1}}{\mathbf{y}_{1}},\tfrac{\mathbf{x}_{2}}{\mathbf{y}_{2}
},...,\tfrac{\mathbf{x}_{2r}}{\mathbf{y}_{2r}}\}$ for this case. (ii) We now assume $(\mathbf{x}+t\mathbf{y})_{i}\neq 0$ for all $i$. We define $\mathbf{z}\triangleq\{+\tfrac{\mathbf{x}_{1}}{\mathbf{y}_{1}},-\tfrac{\mathbf{
x}_{1}}{\mathbf{y}_{1}},+\tfrac{\mathbf{x}_{2}}{\mathbf{y}_{2}},-\tfrac{
\mathbf{x}_{2}}{\mathbf{y}_{2}},...,+\tfrac{\mathbf{x}_{2r}}{\mathbf{y}_{2r}},
-\tfrac{\mathbf{x}_{2r}}{\mathbf{y}_{2r}}\}\in\mathbb{R}^{4r\times 1}$, and sort $\mathbf{z}$ in non-descending order. Given $\bar{t}\neq\mathbf{z}_{i}$ for all $i$ in this case, the domain $p(t)$ can be divided into $(4r+1)$ non-overlapping intervals: $(-\infty,\mathbf{z}_{1}),(\mathbf{z}_{1},\mathbf{z}_{2}),...,(\mathbf{z}_{4r},
+\infty)$. In each interval, $\mathrm{sign}(\mathbf{x}+t\mathbf{y})\triangleq\mathbf{o}$ can be determined. Combining with the fact that $t^{\circ}>0$ and $\|\mathbf{x}+t\mathbf{y}\|_{1}=\langle\mathbf{o},\mathbf{x}+t\mathbf{y}\rangle$, the first-order optimality condition reduces to: $0=[d(1+t^{2})-(w+dt)2t+(b-at)\cdot\sqrt{1+t^{2}}]+\lambda\cdot\sqrt{1+t^{2}}
\cdot[\langle\mathbf{o},\mathbf{y}\rangle(1+t^{2})-\langle\mathbf{o},\mathbf{x
}+t\mathbf{y}\rangle t]$, which can be simplified as: $(at-b)\cdot\sqrt{1+t^{2}}-\lambda\cdot\sqrt{1+t^{2}}\cdot[\langle\mathbf{o},
\mathbf{y}-t\mathbf{x}\rangle]=[d(1+t^{2})-(w+dt)2t]$. We square both sides and then solve the quartic equation. We obtain obtain all its real roots $\{\bar{t}_{1},\bar{t}_{2},...,\bar{t}_{j}\}$ with $1\leq j\leq 4$. Therefore, Problem (20) contains at most $2r+(4r+1)\times 4$ breakpoints.

$\blacktriangleright$ Finding the Breakpoint Set for $h(\mathbf{x})\triangleq I_{\geq 0}(\mathbf{x})$

Since the function $h(\mathbf{x})\triangleq\mathcal{I}_{\geq 0}(\mathbf{x})$ is scale-invariant with $h(t\mathbf{x})=h(\mathbf{x})$ forall $t\geq 0$, Problem (12) reduces to the following problem:

$$ $\displaystyle\bar{t}\in\arg\min_{t}p(t)\triangleq\tfrac{a+bt}{\sqrt{1+t^{2}}}+ \tfrac{w+dt}{1+t^{2}},\,s.t.\,\mathbf{x}+t\mathbf{y}\geq\mathbf{0}.$ (21) $$

We define $I\triangleq\{i|\mathbf{y}_{i}>0\}$ and $J\triangleq\{i|\mathbf{y}_{i}<0\}$. It is not difficult to verity that $\{x+t\mathbf{y}\geq 0\}\Leftrightarrow\{-\tfrac{\mathbf{x}_{I}}{\mathbf{y}_{I}
}\leq t,t\leq-\tfrac{\mathbf{x}_{J}}{\mathbf{y}_{J}}\}\Leftrightarrow\{lb
\triangleq\max(-\tfrac{\mathbf{x}_{I}}{\mathbf{y}_{I}})\leq t\leq\min(-\tfrac{
\mathbf{x}_{J}}{\mathbf{y}_{J}})\triangleq ub\}$. When $lb>ub$, we can directly conclude that the problem has no solution for this case. Now we assume $ub\geq lb$ and define $P(t)\triangleq\min(ub,\max(t,lb))$. We omit the bound constraint and set the gradient of $p(t)$ to zero, which yields: $0=\nabla p(t)=[b(1+t^{2})-(a+bt)t]\cdot\sqrt{1+t^{2}}\cdot t^{\circ}+[d(1+t^{2
})-(w+dt)(2t)]\cdot t^{\circ}$, where $t^{\circ}=(1+t^{2})^{-2}$. We obtain all its real roots $\{\bar{t}_{1},\bar{t}_{2},...,\bar{t}_{j}\}$ with $1\leq j\leq 4$ after squaring both sides and solving the quartic equation. Combining with the bound constraints, we conclude that Problem (21) contains at most $(4+2)$ breakpoints $\{P(\bar{t}_{1}),P(\bar{t}_{2}),...,P(\bar{t}_{j}),lb,ub\}$ with $1\leq j\leq 4$.

## Appendix C Additional Discussions

This section encompasses various discussions, covering topics such as: (i) simple examples for the optimality hierarchy, (ii) the computation of the matrix $\mathbf{Q}$, (iii) a complexity comparison with full gradient methods, (iv) generalization to multiple row updates, and (v) the subdifferential of the cardinality function.

Figure: (a) $\min_{\mathbf{V}\in\mathrm{St}(2,2)}F(\mathbf{V})\triangleq\|\mathbf{V}- \mathbf{A}\|_{\mathsf{F}}^{2}$
Refer to caption: x5.png

### C.1 Simple Examples for the Optimality Hierarchy

To demonstrate the strong optimality of $\rm{BS}_{2}\text{-}\rm{points}$ and the advantages of the proposed method, we examine the following simple examples of $2\times 2$ optimization problems mentioned in the paper:

$$ $\displaystyle\textstyle\min_{\mathbf{V}\in\mathrm{St}(2,2)}~{}F(\mathbf{V}) \triangleq\|\mathbf{V}-\mathbf{A}\|_{\mathsf{F}}^{2},~{}\text{with}~{}\mathbf{ A}=(\begin{smallmatrix}1&0\\ -1&-1\end{smallmatrix}).$ (22) $\displaystyle\textstyle\min_{\mathbf{V}\in\mathrm{St}(2,2)}~{}F(\mathbf{V}) \triangleq\|\mathbf{V}-\mathbf{B}\|_{\mathsf{F}}^{2}+5\|\mathbf{V}\|_{1},~{} \text{with}~{}\mathbf{B}=(\begin{smallmatrix}1&0\\ 1&2\end{smallmatrix}).$ (23) $$

Figure 2 shows the geometric visualizations of Problems (22) and (23) using the relation $\min_{\theta}\min(F(\mathbf{V}^{\mathrm{rot}}_{\theta}),F(\mathbf{V}^{\mathrm{
ref}}_{\theta}))=\min_{\mathbf{V}\in\mathrm{St}(2,2)}F(\mathbf{V})$. The two objective functions exhibit periodicity with a period of $2\pi$. Within the interval $[0,2\pi)$, each of them contains one unique BS2-point, while the two respective examples contain 4 and 8 critical points. Therefore, the optimality condition of $\rm{BS}_{2}\text{-}\rm{points}$ might be much stronger than that of critical points.

$\textbf{BS}_{2}\text{-}\textbf{points}$ vs. Critical Point: Algorithm Instance Study. We briefly analyze methods that find critical points of Problem (22), and demonstrate how they may lead to suboptimal results for Problem (22). We illustrate this with the notable feasible method based on the Cayley transformation Wen & Yin (). According to Equation (7) from Wen & Yin (), the update rule is defined as: $\mathbf{X}^{t+1}\Leftarrow\mathbf{Q}\mathbf{X}^{t}$, where $\mathbf{Q}\triangleq[(\mathbf{I}_{2}+\tfrac{\tau}{2}\mathbf{A})^{-1}(\mathbf{I
}_{2}-\tfrac{\tau}{2}\mathbf{A})]$. Here, $\tau\in\mathbb{R}$, and $\mathbf{Q}\in\mathbb{R}^{2\times 2}$ is a suitable skew-symmetric matrix. Lemma A.3 shows that the matrix $\mathbf{Q}$ consistently functions as a rotation matrix. Consequently, if $\mathbf{X}^{0}$ is initialized as a rotation matrix, the resulting solution $\mathbf{X}^{t+1}$ will remain confined to this rotation matrix for all $t$.

### C.2 Computing the Matrix 𝐐 𝐐 \mathbf{Q} bold_Q

Computing the matrix $\mathbf{Q}\in\mathbb{R}^{k^{2}\times k^{2}}$ as in (7) can be a challenging task because it involves the matrix $\mathbf{H}\in\mathbb{R}^{nr\times nr}$. However, in practice, $\mathbf{H}$ often has some special structure that enables fast matrix computation. For example, $\mathbf{H}$ might take a diagonal matrix that is equal to $L\mathbf{I}_{nr}$ for some $L\geq 0$ or has a Kronecker structure where $\mathbf{H}=\mathbf{H}_{1}\otimes\mathbf{H}_{2}$ for some $\mathbf{H}_{1}\in\mathbb{R}^{r\times r}$ and $\mathbf{H}_{2}\in\mathbb{R}^{n\times n}$. The lemmas provided below demonstrate how to compute the matrix $\mathbf{Q}$.

###### Lemma C.1 .

Assume (7) is used to find $\mathbf{Q}$. (a) If $\mathbf{H}=\mathbf{H}_{1}\otimes\mathbf{H}_{2}$, we have: $\mathbf{Q}=\mathbf{Q}_{1}\otimes\mathbf{Q}_{2}$, where $\mathbf{Q}_{1}=\mathbf{Z}\mathbf{H}_{1}\mathbf{Z}^{\mathsf{T}}\in\mathbb{R}^{k
\times k}$ and $\mathbf{Q}_{2}=\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{H}_{2}\mathrm{U}_
{\texttt{{B}}}\in\mathbb{R}^{k\times k}$. (b) If $\mathbf{H}=L\mathbf{I}_{nr}$, we have $\mathbf{Q}=(L\mathbf{Z}\mathbf{Z}^{\mathsf{T}})\otimes\mathbf{I}_{k}$.

###### Proof.

Recall that for any matrices $\bar{\mathbf{A}},\bar{\mathbf{B}},\bar{\mathbf{C}},\bar{\mathbf{D}}$ of suitable dimensions, we have the following equality: $(\bar{\mathbf{A}}\otimes\bar{\mathbf{B}})(\bar{\mathbf{C}}\otimes\bar{\mathbf{
D}})=(\bar{\mathbf{A}}\bar{\mathbf{C}})\otimes(\bar{\mathbf{B}}\bar{\mathbf{D}})$.

(a) If $\mathbf{H}=\mathbf{H}_{1}\otimes\mathbf{H}_{2}$, we derive: $\mathbf{Q}\triangleq(\mathbf{Z}^{\mathsf{T}}\otimes\mathrm{U}_{\texttt{{B}}})^
{\mathsf{T}}\mathbf{H}(\mathbf{Z}^{\mathsf{T}}\otimes\mathrm{U}_{\texttt{{B}}}
)=(\mathbf{Z}^{\mathsf{T}}\otimes\mathrm{U}_{\texttt{{B}}})^{\mathsf{T}}(
\mathbf{H}_{1}\otimes\mathbf{H}_{2})(\mathbf{Z}^{\mathsf{T}}\otimes\mathrm{U}_
{\texttt{{B}}})=(\mathbf{Z}^{\mathsf{T}}\otimes\mathrm{U}_{\texttt{{B}}})^{
\mathsf{T}}[(\mathbf{H}_{1}\mathbf{Z}^{\mathsf{T}})\otimes(\mathbf{H}_{2}
\mathrm{U}_{\texttt{{B}}})]=(\mathbf{Z}\mathbf{H}_{1}\mathbf{Z}^{\mathsf{T}})
\otimes(\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{H}_{2}\mathrm{U}_{
\texttt{{B}}})=\mathbf{Q}_{1}\otimes\mathbf{Q}_{2}$.

(b) If $\mathbf{H}=L\mathbf{I}_{nr}$, we have: $\mathbf{Q}\triangleq(\mathbf{Z}^{\mathsf{T}}\otimes\mathrm{U}_{\texttt{{B}}})^
{\mathsf{T}}\mathbf{H}(\mathbf{Z}^{\mathsf{T}}\otimes\mathrm{U}_{\texttt{{B}}}
)=L(\mathbf{Z}^{\mathsf{T}}\otimes\mathrm{U}_{\texttt{{B}}})^{\mathsf{T}}(
\mathbf{Z}^{\mathsf{T}}\otimes\mathrm{U}_{\texttt{{B}}})=L(\mathbf{Z}\mathbf{Z
}^{\mathsf{T}})\otimes\mathbf{I}_{k}$.

∎

###### Lemma C.2 .

Assume (8) is used to find $\mathbf{Q}$. (a) If $\mathbf{H}=\mathbf{H}_{1}\otimes\mathbf{H}_{2}$, we have $\mathbf{Q}=\|\mathbf{Q}_{1}\|\cdot\|\mathbf{Q}_{2}\|\cdot\mathbf{I}$, where $\mathbf{Q}_{1}$ and $\mathbf{Q}_{2}$ are defined in Lemma C.1. (b) If $\mathbf{H}=L\mathbf{I}_{nr}$, we have $\mathbf{Q}=L\|\mathbf{Z}\|^{2}\cdot\mathbf{I}$.

###### Proof.

(a) Using the results in Claim (a) of Lemma C.1, we have: $(\mathbf{Z}^{\mathsf{T}}\otimes\mathrm{U}_{\texttt{{B}}})^{\mathsf{T}}\mathbf{
H}(\mathbf{Z}^{\mathsf{T}}\otimes\mathrm{U}_{\texttt{{B}}})=\mathbf{Q}_{1}
\otimes\mathbf{Q}_{2}\preceq\|\mathbf{Q}_{1}\|\cdot\|\mathbf{Q}_{2}\|\cdot
\mathbf{I}$.

(b) Using the results in Claim (b) of Lemma C.1, we have: $(\mathbf{Z}^{\mathsf{T}}\otimes\mathrm{U}_{\texttt{{B}}})^{\mathsf{T}}\mathbf{
H}(\mathbf{Z}^{\mathsf{T}}\otimes\mathrm{U}_{\texttt{{B}}})=L\mathbf{Z}\mathbf
{Z}^{\mathsf{T}}\otimes\mathbf{I}_{k}\preceq L\|\mathbf{Z}\|_{2}^{2}\cdot
\mathbf{I}$.

∎

###### Lemma C.1 .

###### Proof.

###### Lemma C.2 .

###### Proof.

### C.3 A Computational Complexity Comparison with Full Gradient Methods

We present a computational complexity comparison with full gradient methods using the linear eigenvalue problem: $\min_{\mathbf{X}}~{}F(\mathbf{X})\triangleq\tfrac{1}{2}\langle\mathbf{X},
\mathbf{C}\mathbf{X}\rangle,~{}s.t.~{}\mathbf{X}^{\mathsf{T}}\mathbf{X}=
\mathbf{I}_{r}$, where $\mathbf{C}\in\mathbb{R}^{n\times n}$ is given.

We first examine full gradient methods such as the Riemannian gradient method Jiang & Dai (); Liu et al. (). The computation of the Riemannian gradient $\nabla_{\mathcal{M}}F(\mathbf{X})=\mathbf{C}\mathbf{X}-\mathbf{X}[\mathbf{C}
\mathbf{X}]^{\mathsf{T}}\mathbf{X}$ requires $\mathcal{O}(n^{2}r)$ time, while the retraction step using SVD, QR, or polar decomposition demands $\mathcal{O}(nr^{2})$. Consequently, the overall complexity for Riemannian gradient method is $N_{1}\times\mathcal{O}(n^{2}r)$, where $N_{1}$ is the number of iterations required for convergence.

We now consider the proposed OBCD method where the matrix $\mathbf{Q}$ is chosen to be a diagonal matrix as in Equality (8). (i) We adopt an incremental update strategy for computing the Euclidean gradient $\nabla F(\mathbf{X})=\mathbf{C}\mathbf{X}$, maintaining the relationship $\mathbf{Y}^{t}=\mathbf{C}\mathbf{X}^{t}$ for all $t$. The initialization $\mathbf{Y}^{0}=\mathbf{C}\mathbf{X}^{0}$ occurs only once. When $\mathbf{X}^{t}$ is updated via a $k$-row change, resulting in $\mathbf{X}^{t+1}=\mathbf{X}^{t}+\mathrm{U}_{\texttt{{B}}}(\mathbf{V}-\mathbf{I
})\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}^{t}$, we efficiently reconstruct $\mathbf{C}\mathbf{X}^{t+1}$ by updating $\mathbf{Y}^{t+1}=\mathbf{Y}^{t}+\mathbf{C}\mathrm{U}_{\texttt{{B}}}(\mathbf{V}
-\mathbf{I})\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}^{t}$ in $\mathcal{O}(nr)$ time. (ii) Computing the matrix $\mathbf{P}$ as shown in (10) involves matrix multiplication between matrices $[\nabla f(\mathbf{X}^{t})]_{\texttt{{B}}:}\in\mathbb{R}^{k\times r}$ and $[[\mathbf{X}^{t}]_{\texttt{{B}}:}]^{\mathsf{T}}\in\mathbb{R}^{r\times k}$, which can be done in $\mathcal{O}(rk^{2})$. (iii) Solving the subproblem using small-size SVD takes $\mathcal{O}(k^{3})$ time. Thus, the total complexity for OBCD is $N_{2}\times\mathcal{O}(nr+rk^{2}+k^{3})$, with $N_{2}$ denoting the number of OBCD iterations.

### C.4 Generalization to Multiple Row Updates

The proposed OBCD algorithm can be generalized to multiple row updates scheme.

Assume that $n$ is an even number, and $k=2$. As mentioned in Lemma 2.4, when (8) is used to find $\mathbf{Q}$, the subproblem $\bar{\mathbf{V}}^{t}\in\arg\min_{\mathbf{V}\in\mathrm{St}(k,k)}\mathcal{K}(
\mathbf{V};\mathbf{X}^{t},\texttt{{B}})$ in Algorithm 1 reduces to:

$$ $\displaystyle\min_{\mathbf{V}\in\mathrm{St}(2,2)}\langle\mathbf{V},(\nabla f( \mathbf{X}^{t})[\mathbf{X}^{t}]^{\mathsf{T}})_{\texttt{{B}}\texttt{{B}}} \rangle+h(\mathbf{V}\mathrm{U}_{\texttt{{B}}}\mathbf{X}^{t}).$ (24) $$

One can independently solve $(n/2)$ subproblems, each formulated as follows:

$\min_{\mathbf{V}\in\mathrm{St}(2,2)}\langle\mathbf{V},(\nabla f(\mathbf{X}^{t}
)[\mathbf{X}^{t}]^{\mathsf{T}})_{\texttt{{B}}\texttt{{B}}}\rangle+h(\mathbf{V}
\mathrm{U}_{\texttt{{B}}}\mathbf{X}^{t})$ with $\texttt{{B}}=[1,2]$.

$\min_{\mathbf{V}\in\mathrm{St}(2,2)}\langle\mathbf{V},(\nabla f(\mathbf{X}^{t}
)[\mathbf{X}^{t}]^{\mathsf{T}})_{\texttt{{B}}\texttt{{B}}}\rangle+h(\mathbf{V}
\mathrm{U}_{\texttt{{B}}}\mathbf{X}^{t})$ with $\texttt{{B}}=[3,4]$.

$\ldots$

$\min_{\mathbf{V}\in\mathrm{St}(2,2)}\langle\mathbf{V},(\nabla f(\mathbf{X}^{t}
)[\mathbf{X}^{t}]^{\mathsf{T}})_{\texttt{{B}}\texttt{{B}}}\rangle+h(\mathbf{V}
\mathrm{U}_{\texttt{{B}}}\mathbf{X}^{t})$ with $\texttt{{B}}=[n-1,n]$.

This approach, known as the Jacobi update in the literature, allows for the parallel update of $n$ rows of the matrix $\mathbf{X}$.

Notably, one can consider $k\triangleq|\texttt{{B}}|>2$ when $h(\cdot)=0$, and the associated subproblems can be solved using SVD.

### C.5 Limiting Subdifferential of the Cardinality Function

We demonstrate how to calculate the limiting subdifferential of the cardinality function $h(\mathbf{X})=\|\mathbf{X}\|_{0}$. Given that $h(\mathbf{X})=\|\mathbf{X}\|_{0}$ is coordinate-wise separable, we focus only on the scalar function $h(x)=|x|_{0}$.

The Fréchet subdifferential of the function $h(x)=|x|_{0}$ at $x\in\text{dom}(h)$ is defined as $\hat{\partial}{h}(x)\triangleq\{\xi\in\mathbb{R}:\lim_{z\rightarrow x}\inf_{z
\neq x}\tfrac{h(z)-h(x)-\langle\xi,z-x\rangle}{|z-x|}\geq 0\}$, while the limiting subdifferential of $h(x)$ at $x\in\text{dom}(h)$ is denoted as $\partial{h}(x)\triangleq\{\xi\in\mathbb{R}:\exists x^{t}\rightarrow x,h(x^{t})
\rightarrow h(x),\xi^{t}\in\hat{\partial}{h}(x^{t})\rightarrow\xi,\forall t\}$. We consider the following two cases. (i) $x\neq 0$. We have: $\hat{\partial}{h}(x)=\{\xi\in\mathbb{R}:\lim_{z\rightarrow x}\inf_{z\neq x}
\tfrac{-\langle\xi,z-x\rangle}{|z-x|}\geq 0\}=\{0\}$. (ii) $x=0$. We have: $\hat{\partial}{h}(x)=\{\xi\in\mathbb{R}:\lim_{z\rightarrow x}\inf_{z\neq x}
\tfrac{|z|_{0}-\langle\xi,z-x\rangle}{|z-x|}\geq 0\}=\{\xi\in\mathbb{R}:\lim_{
z\rightarrow x}\inf_{z\neq x}\tfrac{1-\langle\xi,z\rangle}{|z|}\geq 0\}=
\mathbb{R}$.

We therefore conclude that $[\partial\|\mathbf{X}\|_{0}]_{i,j}\in{\{\begin{array}[]{ll}\mathbb{R},&\hbox{$
\mathbf{X}_{i,j}=0$;}\\
\{0\},&\hbox{else.}\end{array}\}}$ for all $i\in[n]$ and $j\in[r]$.

## Appendix D Greedy Strategies for Working Set Selection

In this section, we introduce two novel greedy strategies designed to identify an effective working set to enhance the practical computational efficiency of ODBC for $k=2$, as shown in Algorithm 2. These methods exclusively utilize the current solution $\mathbf{X}^{t}$ and its associated subgradient $\mathbf{G}^{t}\in\partial F(\mathbf{X}^{t})$. Notably, our subsequent discussion relies on an additional variable matrix denoted as the scoring matrix $\mathbf{S}$.

Our first Working Set Selection (WSS) strategy is based on the maximum Stationarity Violation pair, denoted as WWS-SV. It selects the index $\texttt{{B}}=[\bar{i},\bar{j}]$ that most violates the first-order optimality condition.

Our second working set selection strategy is rooted in the maximum Objective Reduction pair, denoted as WWS-OR. It chooses the index $\texttt{{B}}=[\bar{i},\bar{j}]$ that leads to the maximum objective reduction under certain criteria.

We have the following results for the theoretical properties of WWS-SV and WWS-OR.

###### Lemma D.1 .

(Proof in Appendix I.1, Properties of WSS-SV). Assume that the scoring matrix $\mathbf{S}$ is computed using (25), we have: (a) $\mathbf{X}^{t}\in\mathrm{St}(n,r)$ is a critical point $\Leftrightarrow$ $\mathbf{S}=\mathbf{0}$. (b) $\mathbf{S}=\mathbf{0}$ $\Leftrightarrow$ $\mathbf{S}(\bar{i},\bar{j})=0$.

Figure: Algorithm 2 WSS: Working Set Selection via Greedy Strategies.

###### Theorem D.2 .

(Proof in Appendix I.2, Properties of WSS-OR). Assume that the scoring matrix $\mathbf{S}$ is computed using (26). Assume $h(\mathbf{X})=0$ and Equation (7) is used to choose the matrix $\mathbf{Q}$. We have:

(a) The value of $\mathbf{S}_{ij}$ for any given $[i,j]$ can be computed as $\mathbf{S}_{ij}=\min({w}_{1},{w}_{2})$, where ${w}_{1}\triangleq-c_{1}-\sqrt{c_{1}^{2}+c_{2}^{2}}$, ${w}_{2}\triangleq-c_{1}-\sqrt{c_{3}^{2}+c_{4}^{2}}$, $c_{1}\triangleq\mathbf{T}_{ii}+\mathbf{T}_{jj}$, $c_{2}\triangleq\mathbf{T}_{ij}-\mathbf{T}_{ji}$, $c_{3}\triangleq\mathbf{T}_{jj}-\mathbf{T}_{ii}$ and $c_{4}\triangleq\mathbf{T}_{ij}+\mathbf{T}_{ji}$.

(b) If $\mathbf{X}^{t}$ is not a critical point, it holds that: $\mathbf{S}({\bar{i},\bar{j}})<0$ and $F(\mathbf{X}^{t+1})<F(\mathbf{X}^{t})$.

Remarks. (i) The computational complexity of both WSS-MV and WSS-OR for a given pair $[i,j]$ is $\mathcal{O}(r)$. Therefore, the overall computational complexity for all $\mathrm{C}_{n}^{2}$ pairs is $\mathcal{O}(n^{2}r)$. Such computational complexity could be high when $n$ is large. We consider the following more practical approach for $k=2$ in our experiments. We randomly and uniformly sample $p\triangleq\min(n,200)$ elements from the set $\{\mathcal{B}_{i}\}_{i=1}^{\mathrm{C}_{n}^{2}}$ as $\{\bar{\mathcal{B}}_{i}\}_{i=1}^{p}$, and then we pick the working set using $\texttt{{B}}=[\bar{i},\bar{j}]=\arg\max_{i,j,i\neq j}|\mathbf{S}_{ij}|,\,s.t.
\,[i,j]\in\{\bar{\mathcal{B}}_{i}\}_{i=1}^{p}$. This strategy leads to a significant reduction in computational complexity to $\mathcal{O}(pr)$ when $p\ll\mathrm{C}_{n}^{2}$. (ii) When choosing $k$ coordinates with $k>2$, one can simply pick the top-$k$ nonoverlapping coordinates according $|\mathbf{S}|$ as the working set.

###### Lemma D.1 .

###### Theorem D.2 .

## Appendix E Proof for Section 2

### E.1 Proof for Lemma 2.1

###### Proof.

(a) For any $\mathbf{V}\in\mathbb{R}^{k\times k}$ and $\texttt{{B}}\in\{\mathcal{B}_{i}\}$${}_{i=1}^{\mathrm{C}_{n}^{k}}$, we have:

$$ $\displaystyle[\mathbf{X}^{+}]^{\mathsf{T}}\mathbf{X}^{+}-\mathbf{X}^{\mathsf{T }}\mathbf{X}$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle[\mathbf{X}+\mathrm{U}_{\texttt{{B}}}(\mathbf{V}-\mathbf{I}_{k}) \mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}]^{\mathsf{T}}[\mathbf{X}+ \mathrm{U}_{\texttt{{B}}}(\mathbf{V}-\mathbf{I}_{k})\mathrm{U}_{\texttt{{B}}}^ {\mathsf{T}}\mathbf{X}]-\mathbf{X}^{\mathsf{T}}\mathbf{X}$ $\displaystyle=$ $\displaystyle\mathbf{X}^{\mathsf{T}}\mathrm{U}_{\texttt{{B}}}(\mathbf{V}- \mathbf{I}_{k})\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}+[\mathrm{U}_{ \texttt{{B}}}(\mathbf{V}-\mathbf{I}_{k})\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}} \mathbf{X}]^{\mathsf{T}}\mathbf{X}+[\mathrm{U}_{\texttt{{B}}}(\mathbf{V}- \mathbf{I}_{k})\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}]^{\mathsf{T}}[ \mathrm{U}_{\texttt{{B}}}(\mathbf{V}-\mathbf{I}_{k})\mathrm{U}_{\texttt{{B}}}^ {\mathsf{T}}\mathbf{X}]$ $\displaystyle=$ $\displaystyle\mathbf{X}^{\mathsf{T}}\mathrm{U}_{\texttt{{B}}}\left[(\mathbf{V} -\mathbf{I}_{k}+\mathbf{V}^{\mathsf{T}}-\mathbf{I}_{k})+(\mathbf{V}-\mathbf{I} _{k})^{\mathsf{T}}\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathrm{U}_{\texttt{{B }}}(\mathbf{V}-\mathbf{I}_{k})\right]\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}} \mathbf{X}$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle\mathbf{X}^{\mathsf{T}}\mathrm{U}_{\texttt{{B}}}\left[(\mathbf{V} -\mathbf{I}_{k}+\mathbf{V}^{\mathsf{T}}-\mathbf{I}_{k})+(\mathbf{V}-\mathbf{I} _{k})^{\mathsf{T}}(\mathbf{V}-\mathbf{I}_{k})\right]\mathrm{U}_{\texttt{{B}}}^ {\mathsf{T}}\mathbf{X}$ $\displaystyle=$ $\displaystyle\mathbf{X}^{\mathsf{T}}\mathrm{U}_{\texttt{{B}}}(\mathbf{V}- \mathbf{I}_{k}+\mathbf{V}^{\mathsf{T}}-\mathbf{I}_{k}+\mathbf{V}^{\mathsf{T}} \mathbf{V}-\mathbf{V}^{\mathsf{T}}-\mathbf{V}+\mathbf{I}_{k})\mathrm{U}_{ \texttt{{B}}}^{\mathsf{T}}\mathbf{X}$ $\displaystyle=$ $\displaystyle\mathbf{X}^{\mathsf{T}}\mathrm{U}_{\texttt{{B}}}(-\mathbf{I}_{k}+ \mathbf{V}^{\mathsf{T}}\mathbf{V})\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}} \mathbf{X}$ $\displaystyle\overset{\text{\char 174}}{=}$ $\displaystyle\mathbf{X}^{\mathsf{T}}\mathrm{U}_{\texttt{{B}}}\cdot\mathbf{0} \cdot\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}$ $\displaystyle=$ $\displaystyle\mathbf{0},$ $$

where step ① uses $\mathbf{X}^{+}=\mathbf{X}+\mathrm{U}_{\texttt{{B}}}(\mathbf{V}-\mathbf{I}_{k})
\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}$; step ② uses $\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathrm{U}_{\texttt{{B}}}=\mathbf{I}_{k}$; step ③ uses $\mathbf{V}^{\mathsf{T}}\mathbf{V}=\mathbf{I}_{k}$.

(b) Obvious.

∎

###### Proof.

### E.2 Proof of Lemma 2.3

###### Proof.

We define $\mathbf{X}^{+}\triangleq\mathbf{X}+\mathrm{U}_{\texttt{{B}}}(\mathbf{V}-
\mathbf{I}_{k})\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}$, $\underline{\mathbf{Q}}\triangleq(\mathbf{Z}^{\mathsf{T}}\otimes\mathrm{U}_{
\texttt{{B}}})^{\mathsf{T}}\mathbf{H}(\mathbf{Z}^{\mathsf{T}}\otimes\mathrm{U}
_{\texttt{{B}}})$, and $\mathbf{Z}\triangleq\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}$.
(a) We derive the following results:

$$ $\displaystyle\|\mathbf{X}^{+}-\mathbf{X}\|_{\mathbf{H}}^{2}$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\|\mathrm{U}_{\texttt{{B}}}(\mathbf{V}-\mathbf{I}_{k})\mathbf{Z} \|_{\mathbf{H}}^{2}$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle\mathrm{vec}(\mathrm{U}_{\texttt{{B}}}(\mathbf{V}-\mathbf{I}_{k}) \mathbf{Z})^{\mathsf{T}}\mathbf{H}\mathrm{vec}(\mathrm{U}_{\texttt{{B}}}( \mathbf{V}-\mathbf{I}_{k})\mathbf{Z})$ $\displaystyle\overset{\text{\char 174}}{=}$ $\displaystyle\mathrm{vec}(\mathbf{V}-\mathbf{I}_{k})^{\mathsf{T}}(\mathbf{Z}^{ \mathsf{T}}\otimes\mathrm{U}_{\texttt{{B}}})^{\mathsf{T}}\mathbf{H}(\mathbf{Z} ^{\mathsf{T}}\otimes\mathrm{U}_{\texttt{{B}}})\mathrm{vec}(\mathbf{V}-\mathbf{ I}_{k})$ $\displaystyle\overset{\text{\char 175}}{=}$ $\displaystyle\|\mathbf{V}-\mathbf{I}_{k}\|_{(\mathbf{Z}^{\mathsf{T}}\otimes \mathrm{U}_{\texttt{{B}}})^{\mathsf{T}}\mathbf{H}(\mathbf{Z}^{\mathsf{T}} \otimes\mathrm{U}_{\texttt{{B}}})}^{2}$ $\displaystyle\overset{\text{\char 176}}{=}$ $\displaystyle\|\mathbf{V}-\mathbf{I}_{k}\|_{\underline{\mathbf{Q}}}^{2},$ $$

where step ① uses $\mathbf{X}^{+}\triangleq\mathbf{X}+\mathrm{U}_{\texttt{{B}}}(\mathbf{V}-
\mathbf{I}_{k})\mathbf{Z}$; step ② uses $\|\mathbf{X}\|_{\mathbf{H}}^{2}=\mathrm{vec}(\mathbf{X})^{\mathsf{T}}\mathbf{H
}\mathrm{vec}(\mathbf{X})$; step ③ uses $(\mathbf{Z}^{\mathsf{T}}\otimes\mathbf{R})\mathrm{vec}(\mathbf{U})=\mathrm{vec
}(\mathbf{R}\mathbf{U}\mathbf{Z})$ for all $\mathbf{R}$, $\mathbf{Z}$, and $\mathbf{U}$ of suitable dimensions; step ④ uses $\|\mathbf{X}\|_{\mathbf{H}}^{2}=\mathrm{vec}(\mathbf{X})^{\mathsf{T}}\mathbf{H
}\mathrm{vec}(\mathbf{X})$ again; step ⑤ uses the definition of $\underline{\mathbf{Q}}$.

(b) We derive the following equalities:

$$ $\displaystyle\|\mathbf{X}^{+}-\mathbf{X}\|_{\mathsf{F}}^{2}$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\|\mathrm{U}_{\texttt{{B}}}(\mathbf{V}-\mathbf{I}_{k})\mathbf{Z} \|_{\mathsf{F}}^{2}$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle\|(\mathbf{V}-\mathbf{I}_{k})\mathbf{Z}\|_{\mathsf{F}}^{2}$ $\displaystyle\overset{\text{}}{=}$ $\displaystyle\langle(\mathbf{V}-\mathbf{I}_{k})^{\mathsf{T}}(\mathbf{V}- \mathbf{I}_{k}),\mathbf{Z}\mathbf{Z}^{\mathsf{T}}\rangle$ $\displaystyle\overset{\text{\char 174}}{=}$ $\displaystyle 2\langle\mathbf{I}_{k}-\mathbf{V},\mathbf{Z}\mathbf{Z}^{\mathsf{ T}}\rangle+\langle\mathbf{V}-\mathbf{V}^{\mathsf{T}},\mathbf{Z}\mathbf{Z}^{ \mathsf{T}}\rangle.$ $\displaystyle\overset{\text{\char 175}}{=}$ $\displaystyle 2\langle\mathbf{I}_{k}-\mathbf{V},\mathbf{Z}\mathbf{Z}^{\mathsf{ T}}\rangle+0.$ $$

where step ① uses $\mathbf{X}^{+}\triangleq\mathbf{X}+\mathrm{U}_{\texttt{{B}}}(\mathbf{V}-
\mathbf{I}_{k})\mathbf{Z}$; step ② uses the fact that $\|\mathrm{U}_{\texttt{{B}}}\mathbf{V}\|_{\mathsf{F}}^{2}=\|\mathbf{V}\|_{
\mathsf{F}}^{2}$ for any $\mathbf{V}\in\mathbb{R}^{k\times k}$; step ③ uses

$$ $(\mathbf{V}-\mathbf{I}_{k})^{\mathsf{T}}(\mathbf{V}-\mathbf{I}_{k})=\mathbf{I} _{k}-\mathbf{V}^{\mathsf{T}}-\mathbf{V}+\mathbf{I}_{k}=2(\mathbf{I}_{k}- \mathbf{V})+(\mathbf{V}-\mathbf{V}^{\mathsf{T}});$ $$

step ④ uses the fact that $\langle\mathbf{V},\mathbf{Z}\mathbf{Z}^{\mathsf{T}}\rangle=\langle\mathbf{V}^{
\mathsf{T}},(\mathbf{Z}\mathbf{Z}^{\mathsf{T}})^{\mathsf{T}}\rangle=\langle
\mathbf{V}^{\mathsf{T}},\mathbf{Z}\mathbf{Z}^{\mathsf{T}}\rangle$ which holds true as the matrix $\mathbf{Z}\mathbf{Z}^{\mathsf{T}}$ is symmetric.

(c) We have:

$$ $\displaystyle\|\mathbf{X}^{+}-\mathbf{X}\|_{\mathsf{F}}^{2}$ $\displaystyle=$ $\displaystyle\|\mathrm{U}_{\texttt{{B}}}(\mathbf{V}-\mathbf{I}_{k})\mathrm{U}_ {\texttt{{B}}}^{\mathsf{T}}\mathbf{X}\|^{2}_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\|\mathrm{U}_{\texttt{{B}}}\|_{2}^{2}\cdot\|(\mathbf{V}-\mathbf{I }_{k})\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}\|_{\mathsf{F}}^{2}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle\|\mathrm{U}_{\texttt{{B}}}\|_{2}^{2}\cdot\|\mathbf{V}-\mathbf{I} _{k}\|_{\mathsf{F}}^{2}\cdot\|\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\|_{2}^{2} \cdot\|\mathbf{X}\|_{2}^{2}$ $\displaystyle\overset{\text{\char 174}}{=}$ $\displaystyle\|\mathbf{V}-\mathbf{I}_{k}\|_{\mathsf{F}}^{2}$ $\displaystyle\overset{\text{\char 175}}{=}$ $\displaystyle 2\langle\mathbf{I}_{k}-\mathbf{V},\mathbf{I}_{k}\rangle,$ $$

where step ① and step ② uses the norm inequality that $\|\mathbf{A}\mathbf{X}\|_{\mathsf{F}}\leq\|\mathbf{A}\|_{\mathsf{F}}\cdot\|
\mathbf{X}\|$ for any $\mathbf{A}$ and $\mathbf{X}$; step ③ uses $\|\mathrm{U}_{\texttt{{B}}}\|=\|\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\|=\|
\mathbf{X}\|=1$ for any $\mathbf{X}\in\mathrm{St}(n,r)$; step ④ uses the following equalities for any $\mathbf{V}\in\mathrm{St}(k,k)$:

$$ $\|\mathbf{V}-\mathbf{I}_{k}\|_{\mathsf{F}}^{2}=\|\mathbf{V}\|_{\mathsf{F}}^{2} +\|\mathbf{I}_{k}\|_{\mathsf{F}}^{2}-2\langle\mathbf{I}_{k},\mathbf{V}\rangle= \|\mathbf{I}_{k}\|_{\mathsf{F}}^{2}+\|\mathbf{I}_{k}\|_{\mathsf{F}}^{2}-2 \langle\mathbf{I}_{k},\mathbf{V}\rangle=2\langle\mathbf{I}_{k},\mathbf{I}_{k}- \mathbf{V}\rangle.$ $$

∎

###### Proof.

### E.3 Proof of Lemma 2.4

###### Proof.

We define $\mathcal{K}(\mathbf{V};\mathbf{X}^{t},\texttt{{B}})\triangleq\tfrac{1}{2}\|
\mathbf{V}-\mathbf{I}_{k}\|_{\mathbf{Q}+\alpha\mathbf{I}}^{2}+h(\mathbf{V}
\mathbf{Z})+\langle\mathbf{V},[\nabla f(\mathbf{X}^{t})(\mathbf{X}^{t})^{
\mathsf{T}}]_{\texttt{{B}}\texttt{{B}}}\rangle+\ddot{c}$, where $\mathbf{Z}\triangleq\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}^{t}$ and $\ddot{c}=h(\mathrm{U}_{\texttt{{B}}^{c}}^{\mathsf{T}}\mathbf{X}^{t})+f(\mathbf
{X}^{t})-\langle\mathbf{I}_{k},[\nabla f(\mathbf{X}^{t})(\mathbf{X}^{t})^{
\mathsf{T}}]_{\texttt{{B}}\texttt{{B}}}\rangle$ is a constant.

(a) Using the definition of $\mathcal{K}(\mathbf{V};\mathbf{X}^{t},\texttt{{B}})$, we have the following equalities for all $\mathbf{V}\in\mathrm{St}(k,k)$:

$$ $\displaystyle\textstyle\mathcal{K}(\mathbf{V};\mathbf{X}^{t},\texttt{{B}})- \ddot{c}$ $\displaystyle\triangleq$ $\displaystyle\tfrac{1}{2}\|\mathbf{V}-\mathbf{I}\|_{\mathbf{Q}+\alpha\mathbf{I }}^{2}+\langle\mathbf{V},[\nabla f(\mathbf{X}^{t})(\mathbf{X}^{t})^{\mathsf{T} }]_{\texttt{{B}}\texttt{{B}}}\rangle+h(\mathbf{V}\mathbf{Z})$ $\displaystyle\overset{}{=}$ $\displaystyle\tfrac{1}{2}\|\mathbf{V}-\mathbf{I}\|_{\mathbf{Q}}^{2}+\tfrac{ \alpha}{2}\|\mathbf{V}-\mathbf{I}\|_{\mathsf{F}}^{2}+\langle\mathbf{V},[\nabla f (\mathbf{X}^{t})(\mathbf{X}^{t})^{\mathsf{T}}]_{\texttt{{B}}\texttt{{B}}} \rangle+h(\mathbf{V}\mathbf{Z})$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\tfrac{1}{2}\|\mathbf{V}\|_{\mathbf{Q}}^{2}-\langle\mathbf{V}, \mathrm{mat}(\mathbf{Q}\mathrm{vec}(\mathbf{I}_{k}))\rangle+\tfrac{1}{2}\| \mathbf{I}_{k}\|_{\mathbf{Q}}^{2}+\alpha\langle\mathbf{I},\mathbf{I}-\mathbf{V }\rangle+\langle\mathbf{V},[\nabla f(\mathbf{X}^{t})(\mathbf{X}^{t})^{\mathsf{ T}}]_{\texttt{{B}}\texttt{{B}}}\rangle+h(\mathbf{V}\mathbf{Z})$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle\tfrac{1}{2}\|\mathbf{V}\|_{\mathbf{Q}}^{2}+\langle\mathbf{V}, \underbrace{[\nabla f(\mathbf{X}^{t})(\mathbf{X}^{t})^{\mathsf{T}}]_{\texttt{{ B}}\texttt{{B}}}-\mathrm{mat}(\mathbf{Q}\mathrm{vec}(\mathbf{I}_{k}))-\alpha \mathbf{I}_{k}}_{\triangleq\mathbf{P}}\rangle+h(\mathbf{V}\mathbf{Z})+\tfrac{1 }{2}\|\mathbf{I}_{k}\|_{\mathbf{Q}}^{2},$ $$

where step ① uses Claim (c) of Lemma 2.3 that: $\tfrac{\alpha}{2}\|\mathbf{V}-\mathbf{I}_{k}\|_{\mathsf{F}}^{2}=\alpha\langle
\mathbf{I},\mathbf{I}-\mathbf{V}\rangle$; step ② uses the definition of $\mathbf{P}$.

(b) We consider the case that $\mathbf{Q}$ is chosen to be a diagonal matrix that $\mathbf{Q}=\varsigma\mathbf{I}$, where $\varsigma$ is defined in Equation (8). Using $\mathbf{V}\in\mathrm{St}(k,k)$, the term $\tfrac{1}{2}\|\mathbf{V}\|_{\mathbf{Q}}^{2}$ simplifies to a constant with $\tfrac{1}{2}\|\mathbf{V}\|_{\mathbf{Q}}^{2}=\tfrac{\varsigma}{2}k$. We can deduce from (10):

$$ $\displaystyle\bar{\mathbf{V}}^{t}\in\arg\min_{\mathbf{V}\in\mathrm{St}(k,k)} \mathcal{P}(\mathbf{V})\triangleq\langle\mathbf{V},\mathbf{P}\rangle+h(\mathbf {X}).$ (27) $$

In particular, when $h(\mathbf{X})=0$, Problem (27) becomes the nearest orthogonality matrix problem and can be solved analytically, yielding a closed-form solution that:

$$ $\displaystyle\bar{\mathbf{V}}^{t}\in\arg\min_{\mathbf{V}\in\mathrm{St}(k,k)} \tfrac{1}{2}\|\mathbf{V}-(-\mathbf{P})\|_{\mathsf{F}}^{2}=\mathbb{P}_{\mathcal {M}}(-\mathbf{P})=-\mathbb{P}_{\mathcal{M}}(\mathbf{P})=-\tilde{\mathbf{U}} \tilde{\mathbf{V}}^{\mathsf{T}}.$ $$

Here, $\mathbf{P}=\tilde{\mathbf{U}}\mathrm{Diag}(\mathbf{s})\tilde{\mathbf{V}}^{
\mathsf{T}}$ is the singular value decomposition of $\mathbf{P}$ with $\tilde{\mathbf{U}},\tilde{\mathbf{V}}\in\mathrm{St}(k,k)$, $\mathbf{s}\in\mathbb{R}^{k}$, and $\mathbf{s}\geq\mathbf{0}$.

Notably, the multiplier for the orthogonality constraint $\mathbf{V}^{\mathsf{T}}\mathbf{V}=\mathbf{I}_{k}$ can be computed as: $\mathbf{\Lambda}=-\mathbf{P}^{\mathsf{T}}\bar{\mathbf{V}}^{t}\overset{\text{
\char 172}}{=}-[\tilde{\mathbf{U}}\mathrm{Diag}(\mathbf{s})\tilde{\mathbf{V}}^
{\mathsf{T}}]^{\mathsf{T}}\cdot[-\tilde{\mathbf{U}}\tilde{\mathbf{V}}^{\mathsf
{T}}]=\tilde{\mathbf{V}}\mathrm{Diag}(\mathbf{s})\tilde{\mathbf{U}}^{\mathsf{T
}}\tilde{\mathbf{U}}\tilde{\mathbf{V}}^{\mathsf{T}}\overset{\text{\char 173}}{
=}\tilde{\mathbf{V}}\mathrm{Diag}(\mathbf{s})\tilde{\mathbf{V}}^{\mathsf{T}}
\overset{\text{\char 174}}{\succeq}\mathbf{0}$, where step ① uses $\mathbf{P}=\tilde{\mathbf{U}}\mathrm{Diag}(\mathbf{s})\tilde{\mathbf{V}}^{
\mathsf{T}}$ and $\bar{\mathbf{V}}^{t}=-\tilde{\mathbf{U}}\tilde{\mathbf{V}}^{\mathsf{T}}$; step ② uses $\tilde{\mathbf{U}}^{\mathsf{T}}\tilde{\mathbf{U}}=\mathbf{I}_{k}$; step ③ uses $\mathbf{s}\geq 0$.

∎

###### Proof.

### E.4 Proof of Lemma 2.6

###### Proof.

Any $2\times 2$ matrix takes the form $\mathbf{V}=(\begin{smallmatrix}a&b\\
c&d\end{smallmatrix})$. The orthogonality constraint implies that $\mathbf{V}\in\mathrm{St}(2,2)$ meets the following three equations: $1=a^{2}+b^{2}$, $1=c^{2}+d^{2}$, $0=ac+bd$. Without loss of generality, we let $c=\sin(\theta)$ and $d=\cos(\theta)$ with $\theta\in\mathbb{R}$. Then we obtain either (i) $a=\cos(\theta),b=-\sin(\theta)$ or (ii) $a=-\cos(\theta),b=\sin(\theta)$. Therefore, we have the following Givens rotation matrix $\mathbf{V}^{\mathrm{rot}}_{\theta}$ and Jacobi reflection matrix $\mathbf{V}^{\mathrm{ref}}_{\theta}$:

$$ $\displaystyle\mathbf{V}^{\mathrm{rot}}_{\theta}\triangleq\begin{bmatrix}\cos( \theta)&-\sin(\theta)\\ \sin(\theta)&\cos(\theta)\\ \end{bmatrix},~{}\mathbf{V}^{\mathrm{ref}}_{\theta}\triangleq\begin{bmatrix}- \cos(\theta)&\sin(\theta)\\ \sin(\theta)&\cos(\theta)\\ \end{bmatrix}.$ $$

Note that for any $a,b,c,d\in\mathbb{R}$, we have: $\det(\begin{smallmatrix}a&b\\
c&d\end{smallmatrix})=ad-bc$. Therefore, we obtain: $\det(\mathbf{V}^{\mathrm{rot}}_{\theta})=\cos^{2}(\theta)+\sin^{2}(\theta)=1$ and $\det(\mathbf{V}^{\mathrm{rot}}_{\theta})=-\cos^{2}(\theta)-\sin^{2}(\theta)=-1$ for any $\theta\in\mathbb{R}$.

∎

###### Proof.

## Appendix F Proof for Section 3

### F.1 Proof of Theorem 3.1

###### Proof.

(a) First, recall the classical Givens-QR algorithm, which is detailed in Section 5.2.5 of Golub & Van Loan ()). This algorithm can decompose any matrix $\mathbf{X}\in\mathbb{R}^{n\times n}$ (not necessarily orthogonal) into the form $\mathbf{X}=\mathbf{Q}\mathbf{R}$, where $\mathbf{Q}$ is an orthogonal matrix ($\mathbf{Q}\in\mathrm{St}(n,n)$) and $\mathbf{R}$ is a lower triangular matrix with $\mathbf{R}_{ij}=0$ for all $i<j$, achieved through $\mathrm{C}_{n}^{2}=\tfrac{n(n-1)}{2}$ Givens rotation steps.

Combining the result from Lemma A.5, we can conclude that classical Givens-QR algorithm can decompose any orthogonal matrix into the form $\mathbf{X}=\mathbf{Q}\mathbf{R}$, where $\mathbf{Q}\in\mathrm{St}(n,n)$ and $\mathbf{R}$ is diagonal matrix with $\mathbf{R}_{i,i}\in\{-1,+1\}$ for all $i\in[n]$.

We introduce a modification to the Givens-QR algorithm, resulting in our Jacobi-Givens-QR algorithm as presented in Listing LABEL:list:list. This algorithm can decompose any matrix $\mathbf{X}\in\mathrm{St}(n,n)$ into the form $\mathbf{X}=\mathbf{Q}\mathbf{R}$, where $\mathbf{Q}=\mathbf{X}$ and $\mathbf{R}=\mathbf{I}_{n}$, using a sequence of $\mathrm{C}_{n}^{k}$ Givens rotation or Jacobi reflection steps.

[⬇](data:text/plain;base64,ZnVuY3Rpb24gW1EsUl0gPSBKYWNvYmlHaXZlbnNRUihYKQpuID0gc2l6ZShYLDEpOyBRID0gZXllKG4pOyBSID0gWDsKZm9yIGo9MTpuCiAgIGZvciBpPW46LTE6KGorMSkKICAgICAgQiA9IFtpLTE7aV07IFYgPSBHaXZlbnMoUihpLTEsaiksUihpLGopKTsKICAgICAgUihCLDopID0gVicqUihCLDopOyBRKDosQikgPSBRKDosQikqVjsKICAgICAgaWYgKGk9PWorMSAmJiBSKGosaik8MCkKICAgICAgICAgViA9IFstMSAwOyAwIC0xXTsgJSBvciBWID0gWy0xIDA7IDAgMV07CiAgICAgICAgIFIoQiw6KSA9IFYnKlIoQiw6KTsgUSg6LEIpID0gUSg6LEIpKlY7CiAgICAgIGVuZAogICBlbmQKZW5kCmlmKFIobixuKTwwKQogICAgViA9IFsxIDA7MCAtMV07IFIoQiw6KSA9IFYnKlIoQiw6KTsgUSg6LEIpID0gUSg6LEIpKlY7CmVuZAoKZnVuY3Rpb24gViA9IEdpdmVucyhhLGIpCiUgRmluZCBhIEdpdmVucyByb3RhdGlvbiB0aGF0IFYnKlthO2JdID0gW3I7MF0KaWYgKGI9PTApIAogICAgYyA9IDE7IHMgPSAwOwplbHNlCiAgICBpZiAoYWJzKGIpID4gYWJzKGEpKQogICAgICAgIHRhdSA9IC1hL2I7IHMgPSAxL3NxcnQoMSt0YXVeMik7IGMgPSBzKnRhdTsKICAgIGVsc2UKICAgICAgICB0YXUgPSAtYi9hOyBjID0gMS9zcXJ0KDErdGF1XjIpOyBzID0gYyp0YXU7CiAgICBlbmQKZW5kClYgPSBbYyBzOy1zIGNdOw==)
1function [Q,R] = JacobiGivensQR(X)
2n = size(X,1); Q = eye(n); R = X;
3for j=1:n
4 for i=n:-1:(j+1)
5 B = [i-1;i]; V = Givens(R(i-1,j),R(i,j));
6 R(B,:) = V’*R(B,:); Q(:,B) = Q(:,B)*V;
7 if (i==j+1 && R(j,j)<0)
8 V = [-1 0; 0 -1]; % or V = [-1 0; 0 1];
9 R(B,:) = V’*R(B,:); Q(:,B) = Q(:,B)*V;
10 end
11 end
12end
13if(R(n,n)<0)
14 V = [1 0;0 -1]; R(B,:) = V’*R(B,:); Q(:,B) = Q(:,B)*V;
15end
16
17function V = Givens(a,b)
18% Find a Givens rotation that V’*[a;b] = [r;0]
19if (b==0)
20 c = 1; s = 0;
21else
22 if (abs(b) > abs(a))
23 tau = -a/b; s = 1/sqrt(1+tau^2); c = s*tau;
24 else
25 tau = -b/a; c = 1/sqrt(1+tau^2); s = c*tau;
26 end
27end
28V = [c s;-s c];

Please take note of the following four important points in Listing LABEL:list:list.

- a)
When we remove Lines 7-10 and Lines 13-15 from Listing LABEL:list:list, it essentially reverts to the classical Givens-QR algorithm. Givens-QR operates by selecting an appropriate Givens rotation matrix $\mathbf{V}=[\begin{smallmatrix}\cos(\theta)&\sin(\theta)\\
-\sin(\theta)&\cos(\theta)\end{smallmatrix}]$ with a suitable rotation angle $\theta$ to zero-out the matrix element $\mathbf{R}_{ij}$ systematically from left to right ($j=1\rightarrow n$) and bottom to top ($i=n\rightarrow(j+1)$) within every pair of neighboring columns.
- b)
Lines 7-10 and Lines 13-15 can be viewed as correction steps to ensure that the entries $\mathbf{R}_{j,j}=1$ for all $j={n}$.
- c)
Line 7-10 is executed for $(n-2)$ times. In Line 7-10, when Jacobi-Givens-QR detects a negative entry $\mathbf{R}_{i-1,i-1}$ with $i=j+1$, it applies a rotation matrix $\mathbf{V}\triangleq(\begin{smallmatrix}-1&0\\
0&-1\end{smallmatrix})$ to the two rows $\texttt{{B}}=[i-1,i]$ to ensure that(^3^33Alternatively, one can use the reflection matrix $\mathbf{V}\triangleq(\begin{smallmatrix}-1&0\\ 0&1\end{smallmatrix})$ instead of the rotation matrix $\mathbf{V}\triangleq(\begin{smallmatrix}-1&0\\ 0&-1\end{smallmatrix})$ to ensure that $\mathbf{R}_{i-1,i-1}=1$.) $\mathbf{R}_{i-1,i-1}=1$.
- d)
Line 13-15 is executed only once when $\det(\mathbf{X})=-1$. In such cases, we have $\mathbf{R}_{\texttt{{B}}\texttt{{B}}}=(\begin{smallmatrix}1&0\\
0&-1\end{smallmatrix})$ and $\det(\mathbf{R}_{\texttt{{B}}\texttt{{B}}})=-1$, where $\texttt{{B}}=[n-1,n]$ is the two indices for the final rotation or reflection step. To ensure that the resulting $\mathbf{R}_{\texttt{{B}}\texttt{{B}}}$ is an identify matrix, Jacobi-Givens-QR employs a reflection matrix $\mathbf{V}=(\begin{smallmatrix}1&0\\
0&-1\end{smallmatrix})$, leading to $\mathbf{V}^{\mathsf{T}}\mathbf{R}_{\texttt{{B}}\texttt{{B}}}=\mathbf{I}_{2}$.

Therefore, we establish the conclusion that any orthogonal matrix $\mathbf{X}\in\mathrm{St}(n,n)$ can be expressed as $\mathbf{D}=\mathcal{W}_{\mathrm{C}_{n}^{k}}...\mathcal{W}_{2}\mathcal{W}_{1}$, where $\mathcal{W}_{i}=\mathbf{U}_{\mathcal{B}_{i}}\mathcal{V}_{i}\mathbf{U}_{
\mathcal{B}_{i}}^{\mathsf{T}}+\mathbf{U}_{\mathcal{B}^{c}_{i}}\mathbf{U}_{
\mathcal{B}^{c}_{i}}^{\mathsf{T}}$, and $\mathcal{V}_{i}\in\mathrm{St}(2,2)$ is a suitable matrix associated with $\mathcal{B}_{i}$. Furthermore, if $\forall i,\,\mathcal{V}_{i}=\mathbf{I}_{2}$, we have $\forall i,\,\mathcal{W}_{i}=\mathbf{I}_{n}$, leading to $\mathbf{D}=\mathbf{I}_{n}$. This concludes the proof of the first part of this theorem.

(b) For any given $\mathbf{X}\in\mathrm{St}(n,r)$ and $\mathbf{X}^{0}\in\mathrm{St}(n,r)$, we let:

$$ $\displaystyle\bar{\mathbf{D}}=\mathbb{P}_{\mathrm{St}(n,n)}(\mathbf{X}[\mathbf {X}^{0}]^{\mathsf{T}}),$ (28) $$

where $\mathbb{P}_{\mathrm{St}(n,n)}(\mathbf{Y})$ denotes the nearest orthogonality matrix to the given matrix $\mathbf{Y}$.

Assume that the matrix $\mathbf{X}[\mathbf{X}^{0}]^{\mathsf{T}}$ has the following singular value decomposition:

$$ $\displaystyle\mathbf{X}(\mathbf{X}^{0})^{\mathsf{T}}$ $\displaystyle=$ $\displaystyle\mathbf{U}\mathrm{Diag}(\mathbf{z})\mathbf{V}^{\mathsf{T}},~{} \mathbf{z}\in\{0,1\}^{n},~{}\mathbf{U}\in\mathrm{St}(n,n),~{}\mathbf{V}\in \mathrm{St}(n,n).$ $$

Therefore, we have the following equalities:

$$ $\displaystyle\mathrm{Diag}(\mathbf{z})$ $\displaystyle=$ $\displaystyle\mathbf{U}^{\mathsf{T}}\mathbf{X}[\mathbf{X}^{0}]^{\mathsf{T}} \mathbf{V}.$ (29) $\displaystyle\bar{\mathbf{D}}$ $\displaystyle=$ $\displaystyle\mathbf{U}\mathbf{V}^{\mathsf{T}}\in\mathrm{St}(n,n).$ (30) $$

Furthermore, we derive the following results:

$$ $\displaystyle\mathbf{z}\in\{0,1\}^{n}$ $\displaystyle\Rightarrow$ $\displaystyle\mathrm{Diag}(\mathbf{z})^{\mathsf{T}}=\mathrm{Diag}(\mathbf{z}) \mathrm{Diag}(\mathbf{z})^{\mathsf{T}}$ $\displaystyle\Rightarrow$ $\displaystyle\mathbf{U}[\mathrm{Diag}(\mathbf{z})^{\mathsf{T}}-\mathrm{Diag}( \mathbf{z})\mathrm{Diag}(\mathbf{z})^{\mathsf{T}}]\mathbf{U}^{\mathsf{T}} \mathbf{X}=\mathbf{0}$ $\displaystyle\overset{\text{\char 172}}{\Rightarrow}$ $\displaystyle\mathbf{U}[\mathbf{V}^{\mathsf{T}}\mathbf{X}^{0}\mathbf{X}^{ \mathsf{T}}\mathbf{U}-\mathbf{U}^{\mathsf{T}}\mathbf{X}(\mathbf{X}^{0})^{ \mathsf{T}}\mathbf{V}\mathbf{V}^{\mathsf{T}}\mathbf{X}^{0}\mathbf{X}^{\mathsf{ T}}\mathbf{U}]\mathbf{U}^{\mathsf{T}}\mathbf{X}=\mathbf{0}$ $\displaystyle\overset{\text{ }}{\Rightarrow}$ $\displaystyle\mathbf{U}\mathbf{V}^{\mathsf{T}}\mathbf{X}^{0}\mathbf{X}^{ \mathsf{T}}\mathbf{U}\mathbf{U}^{\mathsf{T}}\mathbf{X}-\mathbf{U}\mathbf{U}^{ \mathsf{T}}\mathbf{X}(\mathbf{X}^{0})^{\mathsf{T}}\mathbf{V}\mathbf{V}^{ \mathsf{T}}\mathbf{X}^{0}\mathbf{X}^{\mathsf{T}}\mathbf{U}\mathbf{U}^{\mathsf{ T}}\mathbf{X}=\mathbf{0}$ $\displaystyle\overset{\text{\char 173}}{\Rightarrow}$ $\displaystyle\mathbf{U}\mathbf{V}^{\mathsf{T}}\mathbf{X}^{0}-\mathbf{X}= \mathbf{0}$ $\displaystyle\overset{\text{\char 174}}{\Rightarrow}$ $\displaystyle\bar{\mathbf{D}}\cdot\mathbf{X}^{0}-\mathbf{\mathbf{X}}=\mathbf{0},$ $$

where step ① uses (29); step ② uses $\mathbf{U}\mathbf{U}^{\mathsf{T}}=\mathbf{I}_{n}$, $\mathbf{V}\mathbf{V}^{\mathsf{T}}=\mathbf{I}_{n}$, $\mathbf{X}^{\mathsf{T}}\mathbf{X}=\mathbf{I}_{r}$, and $[\mathbf{X}^{0}]^{\mathsf{T}}\mathbf{X}^{0}=\mathbf{I}_{r}$; step ③ uses (30). We conclude that, for any given $\mathbf{X}\in\mathrm{St}(n,r)$ and $\mathbf{X}^{0}\in\mathrm{St}(n,r)$, we can always find a matrix $\bar{\mathbf{D}}\in\mathrm{St}(n,n)$ such that $\bar{\mathbf{D}}\mathbf{X}^{0}=\mathbf{X}$.

Since the matrix $\bar{\mathbf{D}}\in\mathrm{St}(n,n)$ can be represented as $\bar{\mathbf{D}}=\mathcal{W}_{\mathrm{C}_{n}^{k}}...\mathcal{W}_{2}\mathcal{W}
_{1}$, where $\mathcal{W}_{i}=\mathbf{U}_{\mathcal{B}_{i}}\mathcal{V}_{i}\mathbf{U}_{
\mathcal{B}_{i}}^{\mathsf{T}}+\mathbf{U}_{\mathcal{B}^{c}_{i}}\mathbf{U}_{
\mathcal{B}^{c}_{i}}^{\mathsf{T}}$ for some suitable $\mathcal{V}_{i}\in\mathrm{St}(2,2)$ (as established in the first part of this theorem), we can conclude that any matrix $\mathbf{X}\in\mathrm{St}(n,r)$ can be expressed as $\mathbf{X}=\bar{\mathbf{D}}\mathbf{X}^{0}=\mathcal{W}_{\mathrm{C}_{n}^{k}}...
\mathcal{W}_{2}\mathcal{W}_{1}\mathbf{X}^{0}$.

∎

###### Proof.

### F.2 Proof for Theorem 3.6

###### Proof.

We use $\bar{\mathbf{X}}$, $\ddot{\mathbf{X}}$, and $\check{\mathbf{X}}$ to denote the global optimal point, BSk-point, and critical point of Problem (1), respectively.

Setting the Riemannian subgradient of $\mathcal{K}(\mathbf{V};\ddot{\mathbf{X}},\texttt{{B}})$ w.r.t. $\mathbf{V}$ to zero, we have $\mathbf{0}\in\partial_{\mathcal{M}}\mathcal{K}(\mathbf{V};\ddot{\mathbf{X}},
\texttt{{B}})=\ddot{\mathbf{G}}(\mathbf{V})\ominus\mathbf{V}[\ddot{\mathbf{G}}
(\mathbf{V})]^{\mathsf{T}}\mathbf{V}$, where $\ddot{\mathbf{G}}(\mathbf{V})=\alpha(\mathbf{V}-\mathbf{I}_{k})+\mathrm{U}_{
\texttt{{B}}}^{\mathsf{T}}[\mathrm{mat}(\mathbf{H}\mathrm{vec}(\mathbf{X}^{+}-
\ddot{\mathbf{X}}))+\nabla f(\ddot{\mathbf{X}})+\partial h(\mathbf{X}^{+})]
\ddot{\mathbf{X}}^{\mathsf{T}}\mathrm{U}_{\texttt{{B}}}$ and $\mathbf{X}^{+}=\ddot{\mathbf{X}}+\mathrm{U}_{\texttt{{B}}}(\mathbf{V}-\mathbf{
I}_{k})\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\ddot{\mathbf{X}}$. Letting $\mathbf{V}=\mathbf{I}_{k}$, we have the following necessary but not sufficient condition for any BSk-point:

$$ $\displaystyle\forall\texttt{{B}}\in\{\mathcal{B}_{i}\}_{i=1}^{\mathrm{C}_{n}^{ k}},~{}\mathbf{0}=\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}(\mathbf{G}\ddot{ \mathbf{X}}^{\mathsf{T}}-\ddot{\mathbf{X}}\mathbf{G}^{\mathsf{T}})\mathrm{U}_{ \texttt{{B}}},~{}\text{with}~{}\mathbf{G}\in\nabla f(\ddot{\mathbf{X}})+ \partial h(\ddot{\mathbf{X}}).$ (31) $$

(a) We now show that $\{\rm{critical\,points}\,\check{\mathbf{X}}\}\supseteq\{\rm{BS}_{k}\text{-}\rm
{points}\,\ddot{\mathbf{X}}\}$ for all $k\geq 2$. We let $\mathbf{G}\in\nabla f(\ddot{\mathbf{X}})+\partial h(\ddot{\mathbf{X}})$. Using Lemma A.1, we have:

$$ $\displaystyle\mathbf{0}_{n,n}=\mathbf{G}\ddot{\mathbf{X}}^{\mathsf{T}}-\ddot{ \mathbf{X}}\mathbf{G}^{\mathsf{T}}$ $\displaystyle\overset{}{\Rightarrow}$ $\displaystyle(\mathbf{0}_{n,n}\cdot\ddot{\mathbf{X}})=(\mathbf{G}\ddot{\mathbf {X}}^{\mathsf{T}}-\ddot{\mathbf{X}}\mathbf{G}^{\mathsf{T}})\ddot{\mathbf{X}}$ $\displaystyle\overset{\text{\char 172}}{\Rightarrow}$ $\displaystyle\mathbf{0}_{n,r}=\mathbf{G}-\ddot{\mathbf{X}}\mathbf{G}^{\mathsf{ T}}\ddot{\mathbf{X}},$ $\displaystyle\Rightarrow$ $\displaystyle\ddot{\mathbf{X}}^{\mathsf{T}}\cdot\mathbf{0}_{n,r}=\ddot{\mathbf {X}}^{\mathsf{T}}(\mathbf{G}-\ddot{\mathbf{X}}\mathbf{G}^{\mathsf{T}}\ddot{ \mathbf{X}})$ $\displaystyle\overset{\text{\char 173}}{\Rightarrow}$ $\displaystyle\mathbf{0}_{r,r}=\ddot{\mathbf{X}}^{\mathsf{T}}\mathbf{G}-\mathbf {G}^{\mathsf{T}}\ddot{\mathbf{X}}$ $\displaystyle\Rightarrow$ $\displaystyle\mathbf{0}_{n,n}=\ddot{\mathbf{X}}(\ddot{\mathbf{X}}^{\mathsf{T}} \mathbf{G}-\mathbf{G}^{\mathsf{T}}\ddot{\mathbf{X}})\ddot{\mathbf{X}}^{\mathsf {T}}$ $\displaystyle\overset{\text{\char 174}}{\Rightarrow}$ $\displaystyle\mathbf{0}_{n,n}=\ddot{\mathbf{X}}\underbrace{\ddot{\mathbf{X}}^{ \mathsf{T}}\mathbf{G}\ddot{\mathbf{X}}^{\mathsf{T}}}_{\triangleq\mathbf{G}^{ \mathsf{T}}}-\underbrace{\ddot{\mathbf{X}}\mathbf{G}^{\mathsf{T}}\ddot{\mathbf {X}}}_{\triangleq\mathbf{G}}\ddot{\mathbf{X}}^{\mathsf{T}},$ $$

where steps ① and ② use $\ddot{\mathbf{X}}^{\mathsf{T}}\ddot{\mathbf{X}}=\mathbf{I}_{r}$; step ③ uses Equality (F.2) that $\mathbf{G}=\ddot{\mathbf{X}}\mathbf{G}^{\mathsf{T}}\ddot{\mathbf{X}}$. We conclude that the necessary condition in Equation (31) is equivalent to the optimality condition of critical points.

(b) We now show that $\{\rm{BS}_{2}\text{-}\rm{points}\,\ddot{\mathbf{X}}\}\supseteq\{\rm{global\,
optimal\,points}\,\bar{\mathbf{X}}\}$. We define $\mathcal{X}^{\star}_{\texttt{{B}}}(\mathbf{V})\triangleq\bar{\mathbf{X}}+
\mathrm{U}_{\texttt{{B}}}(\mathbf{V}-\mathbf{I})\mathrm{U}_{\texttt{{B}}}^{
\mathsf{T}}\bar{\mathbf{X}}$, and $\mathcal{K}(\mathbf{V};\mathbf{X},\texttt{{B}})\triangleq f(\mathbf{X})+
\langle\mathbf{V}-\mathbf{I}_{k},[\nabla f(\mathbf{X})(\mathbf{X})^{\mathsf{T}
}]_{\texttt{{B}}\texttt{{B}}}\rangle+\tfrac{1}{2}\|\mathbf{V}-\mathbf{I}_{k}\|
_{\mathbf{Q}+\alpha\mathbf{I}}^{2}+h(\mathrm{U}_{\texttt{{B}}^{c}}^{\mathsf{T}
}\mathbf{X})+h(\mathbf{V}\mathbf{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X})$. We let $\mathbf{V}_{(i)}\in\mathrm{St}(2,2)$ and $\mathcal{B}_{i}\in\{\mathcal{B}_{i}\}_{i=1}^{\mathrm{C}_{n}^{k}}$. We derive:

$$ $\displaystyle\mathcal{K}(\mathbf{I}_{2};\bar{\mathbf{X}},\mathcal{B}_{i}),\, \forall\mathcal{B}_{i}$ (33) $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\textstyle F(\bar{\mathbf{X}})=h(\bar{\mathbf{X}})+f(\bar{\mathbf {X}})$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle\textstyle h(\mathbf{X})+f(\mathbf{X}),\forall\mathbf{X}\in \mathrm{St}(n,r)$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\textstyle h(\bar{\mathbf{X}}+\mathbf{U}_{\mathcal{B}_{i}}( \mathbf{V}_{(i)}-\mathbf{I})\mathbf{U}_{\mathcal{B}_{i}}^{\mathsf{T}}\bar{ \mathbf{X}})+f(\bar{\mathbf{X}}+\mathrm{U}_{\mathcal{B}_{i}}(\mathbf{V}_{(i)}- \mathbf{I})\mathbf{U}_{\mathcal{B}_{i}}^{\mathsf{T}}\bar{\mathbf{X}}),\, \forall\mathbf{V}_{(i)},~{}\forall\mathcal{B}_{i}$ $\displaystyle\overset{\text{\char 175}}{=}$ $\displaystyle\textstyle h(\mathcal{X}^{\star}_{\mathcal{B}_{i}}(\mathbf{V}_{(i )}))+f(\mathcal{X}^{\star}_{\mathcal{B}_{i}}(\mathbf{V}_{(i)})),\,\forall \mathbf{V}_{(i)},~{}\forall\mathcal{B}_{i}$ $\displaystyle\overset{\text{\char 176}}{=}$ $\displaystyle\textstyle\mathcal{K}(\mathbf{V}_{(i)};\bar{\mathbf{X}},\mathcal{ B}_{i}),\,\forall\mathbf{V}_{(i)},~{}\forall\mathcal{B}_{i}$ $\displaystyle\overset{}{=}$ $\displaystyle\textstyle\min_{\mathbf{V}\in\mathrm{St}(2,2)}\mathcal{K}(\mathbf {V};\bar{\mathbf{X}},\mathcal{B}_{i}),\,\forall\mathcal{B}_{i},$ $$

where step ① uses the definition of $\mathcal{K}(\mathbf{V};\mathbf{X},\texttt{{B}})\triangleq f(\mathbf{X})+
\langle\mathbf{V}-\mathbf{I}_{k},[\nabla f(\mathbf{X})(\mathbf{X})^{\mathsf{T}
}]_{\texttt{{B}}\texttt{{B}}}\rangle+\tfrac{1}{2}\|\mathbf{V}-\mathbf{I}_{k}\|
_{\mathbf{Q}+\alpha\mathbf{I}}^{2}+h(\mathrm{U}_{\texttt{{B}}^{c}}^{\mathsf{T}
}\mathbf{X})+h(\mathbf{V}\mathbf{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X})$; step ② uses the definition of $\bar{\mathbf{X}}$; step ③ uses the basis representation of orthogonal matrices when $k=2$, as shown in Theorem 3.1; step ④ uses the definition of $\mathcal{X}^{\star}_{\texttt{{B}}}(\mathbf{V})$; step ⑤ uses the same strategy as in deriving Inequality (2). This leads to:

$$ $\displaystyle\mathbf{I}_{2}\in\arg\min_{\mathbf{V}\in\mathrm{St}(2,2)}~{} \mathcal{K}(\mathbf{V};\bar{\mathbf{X}},\mathcal{B}_{i}),~{}\forall\mathcal{B} _{i}.$ $$

The inclusion above implies that $\{\rm{BS}_{2}\text{-}\rm{points}\,\ddot{\mathbf{X}}\}\supseteq\{\rm{global\,
optimal\,points}\,\bar{\mathbf{X}}\}$.

(c) We now show that $\{\rm{BS}_{k}\text{-}\textit{points}\,\ddot{\mathbf{X}}\}\supseteq\{\rm{BS}_{k
+1}\text{-}\textit{points}\,\ddot{\mathbf{X}}\}$. It is evident that the subproblem of finding $\rm{BS}_{k}\text{-}\textit{points}$ is encompassed within that of finding $\rm{BS}_{k+1}\text{-}\textit{points}$ stationary point. Thus, we conclude that the optimality of the latter is stronger.

(d) The inclusion $\{\rm{critical\,points}\,\check{\mathbf{X}}\}\subseteq\{\rm{BS}_{k}\text{-}\rm
{points}\,\ddot{\mathbf{X}}\}$ may not always hold true. This can be illustrated through simple examples of $2\times 2$ optimization problems under orthogonality constraints (see Appendix Section C.1 for more details). Lastly, it is also evident that the inclusions $\{\rm{BS}_{2}\text{-}\rm{points}\,\ddot{\mathbf{X}}\}\subseteq\{\rm{global\,
optimal\,points}\,\bar{\mathbf{X}}\}$ and $\{\rm{BS}_{k}\text{-}\textit{points}\,\ddot{\mathbf{X}}\}\subseteq\{\rm{BS}_{k
+1}\text{-}\textit{points}\,\ddot{\mathbf{X}}\}$ may not always hold true.

∎

###### Proof.

## Appendix G Proof for Section 4

### G.1 Proof for Theorem 4.2

###### Proof.

We define $\mathcal{K}(\mathbf{V};\mathbf{X}^{t},\texttt{{B}})\triangleq\tfrac{1}{2}\|
\mathbf{V}-\mathbf{I}_{k}\|_{\mathbf{Q}+\alpha\mathbf{I}}^{2}+h(\mathbf{V}
\mathbf{Z})+\langle\mathbf{V},[\nabla f(\mathbf{X}^{t})(\mathbf{X}^{t})^{
\mathsf{T}}]_{\texttt{{B}}\texttt{{B}}}\rangle+\ddot{c}$, where $\mathbf{Z}\triangleq\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}^{t}$ and $\ddot{c}=h(\mathrm{U}_{\texttt{{B}}^{c}}^{\mathsf{T}}\mathbf{X}^{t})+f(\mathbf
{X}^{t})-\langle\mathbf{I}_{k},[\nabla f(\mathbf{X}^{t})(\mathbf{X}^{t})^{
\mathsf{T}}]_{\texttt{{B}}\texttt{{B}}}\rangle$ is a constant.

We define $\tilde{c}\triangleq\textstyle\tfrac{2}{\alpha}\cdot(F(\mathbf{X}^{0})-F(\ddot{
\mathbf{X}}))$.

(a) First, we have the following equalities:

$$ $\displaystyle h(\mathbf{X}^{t+1})-h(\mathbf{X}^{t})\overset{\text{\char 172}}{ =}~{}$ $\displaystyle h(\mathrm{U}_{\texttt{{B}}}\bar{\mathbf{V}}^{t}\mathrm{U}_{ \texttt{{B}}}^{\mathsf{T}}\mathbf{X}^{t}+\mathrm{U}_{\texttt{{B}}^{c}}\mathrm{ U}_{\texttt{{B}}^{c}}^{\mathsf{T}}\mathbf{X}^{t})-h(\mathrm{U}_{\texttt{{B}}} \mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}^{t}+\mathrm{U}_{\texttt{{B}}^ {c}}\mathrm{U}_{\texttt{{B}}^{c}}^{\mathsf{T}}\mathbf{X}^{t})$ $\displaystyle\overset{\text{\char 173}}{=}~{}$ $\displaystyle h(\mathrm{U}_{\texttt{{B}}}\bar{\mathbf{V}}^{t}\mathrm{U}_{ \texttt{{B}}}^{\mathsf{T}}\mathbf{X}^{t})+h(\mathrm{U}_{\texttt{{B}}^{c}} \mathrm{U}_{\texttt{{B}}^{c}}^{\mathsf{T}}\mathbf{X}^{t})-h(\mathrm{U}_{ \texttt{{B}}}\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}^{t})-h(\mathrm{U }_{\texttt{{B}}^{c}}\mathrm{U}_{\texttt{{B}}^{c}}^{\mathsf{T}}\mathbf{X}^{t})$ $\displaystyle\overset{\text{\char 174}}{=}~{}$ $\displaystyle h(\bar{\mathbf{V}}^{t}\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}} \mathbf{X}^{t})-h(\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}^{t}),$ (34) $$

where step ① uses $\mathbf{X}^{t+1}=\mathrm{U}_{\texttt{{B}}}\mathbf{V}\mathrm{U}_{\texttt{{B}}}^
{\mathsf{T}}\mathbf{X}^{t}+\mathrm{U}_{\texttt{{B}}^{c}}\mathrm{U}_{\texttt{{B
}}^{c}}^{\mathsf{T}}\mathbf{X}^{t}$ as in (3) and $\mathbf{I}=\mathrm{U}_{\texttt{{B}}}\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}+
\mathrm{U}_{\texttt{{B}}^{c}}\mathrm{U}_{\texttt{{B}}^{c}}^{\mathsf{T}}$; step ② and step ③ use the coordinate-wise separable structure of $h(\cdot)$.

Second, since $\bar{\mathbf{V}}^{t}$ is the global optimal solution of the problem $\bar{\mathbf{V}}^{t}\in\arg\min_{\mathbf{V}\in\mathrm{St}(k,k)}\mathcal{K}(
\mathbf{V};\mathbf{X}^{t},\texttt{{B}})$, we have: $\mathcal{K}(\bar{\mathbf{V}}^{t};\mathbf{X}^{t},\texttt{{B}})\leq\mathcal{K}(
\mathbf{V};\mathbf{X}^{t},\texttt{{B}}),\forall\mathbf{V}\in\mathrm{St}(k,k)$. Letting $\mathbf{V}=\mathbf{I}_{k}\in\mathrm{St}(k,k)$, we have: $\mathcal{K}(\bar{\mathbf{V}}^{t};\mathbf{X}^{t},\texttt{{B}})\leq\mathcal{K}(
\mathbf{I}_{k};\mathbf{X}^{t},\texttt{{B}})$, leading to:

$$ $\displaystyle h(\bar{\mathbf{V}}^{t}\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}} \mathbf{X}^{t})+\tfrac{1}{2}\|\bar{\mathbf{V}}^{t}-\mathbf{I}_{k}\|^{2}_{ \mathbf{Q}+\alpha\mathbf{I}}+\langle\bar{\mathbf{V}}^{t}-\mathbf{I},[\nabla f( \mathbf{X}^{t})(\mathbf{X}^{t})^{\mathsf{T}}]_{\texttt{{B}}\texttt{{B}}} \rangle\leq h(\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}^{t}).$ (35) $$

Third, we denote $\mathbf{X}^{t+1}=\mathcal{X}_{\texttt{{B}}}^{t}(\bar{\mathbf{V}}^{t})$ and derive:

$$ $\displaystyle f(\mathbf{X}^{t+1})-f(\mathbf{X}^{t})$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\langle\mathcal{X}_{\texttt{{B}}}^{t}(\bar{\mathbf{V}}^{t})- \mathbf{X}^{t},\nabla f(\mathbf{X}^{t})\rangle+\tfrac{1}{2}\|\mathcal{X}_{ \texttt{{B}}}^{t}(\bar{\mathbf{V}}^{t})-\mathbf{X}^{t}\|^{2}_{\mathbf{H}}$ (36) $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle\langle\mathrm{U}_{\texttt{{B}}}(\bar{\mathbf{V}}^{t}-\mathbf{I}_ {k})\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}^{t},\nabla f(\mathbf{X}^{ t})\rangle+\tfrac{1}{2}\|\bar{\mathbf{V}}^{t}-\mathbf{I}_{k}\|^{2}_{\underline {\mathbf{Q}}}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\langle\bar{\mathbf{V}}^{t}-\mathbf{I}_{k},[\nabla f(\mathbf{X}^{ t})(\mathbf{X}^{t})^{\mathsf{T}}]_{\texttt{{B}}\texttt{{B}}}\rangle+\tfrac{1}{ 2}\|\bar{\mathbf{V}}^{t}-\mathbf{I}_{k}\|^{2}_{\mathbf{Q}},$ $$

where step ① uses Inequality (2); step ② uses Claim (a) of Lemma 2.3; step ③ uses $\mathbf{Q}\succcurlyeq\underline{\mathbf{Q}}$.

Adding (G.1), (35), and (36) together, we obtain the following sufficient decrease condition:

$$ $\displaystyle F(\mathbf{X}^{t+1})-F(\mathbf{X}^{t})\leq-\tfrac{\alpha}{2}\| \bar{\mathbf{V}}^{t}-\mathbf{I}_{k}\|_{\mathsf{F}}^{2}\overset{\text{\char 172 }}{\leq}-\tfrac{\alpha}{2}\|\mathbf{X}^{t+1}-\mathbf{X}^{t}\|_{\mathsf{F}}^{2},$ (37) $$

where step ① uses Claim (c) of Lemma 2.3.

(b) We assume that $\texttt{{B}}^{t}$ is selected from $\{\mathcal{B}_{i}\}$${}_{i=1}^{\mathrm{C}_{n}^{k}}$ randomly and uniformly.

Taking the expectation for Inequality (37), we obtain a lower bound on the expected progress made by each iteration:

$$ $\displaystyle\textstyle\mathbb{E}_{\xi^{t}}[F(\mathbf{X}^{t+1})]-F(\mathbf{X}^ {t})\leq-\mathbb{E}_{\xi^{t}}[\tfrac{\alpha}{2}\|\bar{\mathbf{V}}^{t}-\mathbf{ I}_{k}\|_{\mathsf{F}}^{2}].$ $$

Telescoping the inequality above over $t=0,1,...,T$, we have:

$$ $\displaystyle\textstyle\mathbb{E}_{\xi^{T}}[\tfrac{\alpha}{2}\sum_{t=0}^{T}\| \bar{\mathbf{V}}^{t}-\mathbf{I}_{k}\|_{\mathsf{F}}^{2}]\leq\mathbb{E}_{\xi^{T} }[F(\mathbf{X}^{0})-F(\mathbf{X}^{T+1})]\leq\mathbb{E}_{\xi^{T}}[F(\mathbf{X}^ {0})-F(\ddot{\mathbf{X}})],$ $$

where $\ddot{\mathbf{X}}$ denotes the limit point of Algorithm 1. As a result, there exists an index $\bar{t}$ with $0\leq\bar{t}\leq T$ such that

$$ $\displaystyle\mathbb{E}_{\xi^{T}}[\|\bar{\mathbf{V}}^{\bar{t}}-\mathbf{I}_{k} \|_{\mathsf{F}}^{2}]\leq\tfrac{2}{\alpha(T+1)}[F(\mathbf{X}^{0})-F(\ddot{ \mathbf{X}})]=\tfrac{\tilde{c}}{T+1}.$ (38) $$

Furthermore, for any $t$, $\bar{\mathbf{V}}^{t}$ is the optimal solution of the following minimization problem at $\mathbf{X}^{t}$: $\bar{\mathbf{V}}^{t}\in\arg\min_{\mathbf{V}}\min_{\mathbf{V}}\mathcal{K}(
\mathbf{V};\mathbf{X}^{t},\texttt{{B}}^{t})$. Given $\bar{\mathbf{V}}^{t}$ is a random output matrix depends on the observed realization of the random variable $\texttt{{B}}^{t}$, we directly obtain the following equality:

$$ $\displaystyle\textstyle\tfrac{1}{\mathrm{C}_{n}^{k}}\sum_{i=1}^{\mathrm{C}_{n} ^{k}}\text{dist}(\mathbf{I}_{k},\arg\min_{\mathbf{V}}\mathcal{K}(\mathbf{V}; \mathbf{X}^{t},\mathcal{B}_{i}))^{2}=\mathbb{E}_{\xi^{t}}[\|\bar{\mathbf{V}}^{ t}-\mathbf{I}_{k}\|_{\mathsf{F}}^{2}].$ (39) $$

Combining (38) and (39), we conclude that there exists an index $\bar{t}$ with $\bar{t}\in[0,T]$ such that the associated solution $\mathbf{X}^{\bar{t}}$ qualifies as an $\epsilon$-BSk-point of Problem (1), provided that $T$ is sufficiently large such that $\tfrac{\tilde{c}}{T+1}\leq\epsilon$. We conclude that OBCD finds an $\epsilon$-BSk-point of Problem (1) in at most $T$ iterations deterministically, where $T\geq\lceil\tfrac{\tilde{c}}{\epsilon}-1\rceil$.

(c) We assume that $\texttt{{B}}^{t}$ is selected from $\{\mathcal{B}_{i}\}$${}_{i=1}^{\mathrm{C}_{n}^{k}}$ cyclically, i.e., $\mathcal{B}_{1}\rightarrow\mathcal{B}_{2}\rightarrow\mathcal{B}_{3}\rightarrow
\ldots\rightarrow\mathcal{B}_{\mathrm{C}_{n}^{k}-1}\rightarrow\mathcal{B}_{
\mathrm{C}_{n}^{k}}\rightarrow\mathcal{B}_{1}\rightarrow\mathcal{B}_{2}
\rightarrow\mathcal{B}_{3}\rightarrow\ldots$.

Telescoping Inequality (37) over $t$ from 0 0 to $T$ yields:

$$ $\displaystyle\textstyle\tfrac{\alpha}{2}\sum_{t=0}^{T}||\bar{\mathbf{V}}^{t}- \mathbf{I}_{k}||_{F}^{2}\leq F(\mathbf{X}^{0})-F(\mathbf{X}^{T+1})\leq F( \mathbf{X}^{0})-F(\ddot{\mathbf{X}}),$ (40) $$

For notation simplicity, we define $z\triangleq\mathrm{C}_{n}^{k}$ and $e^{t}\triangleq\|\bar{\mathbf{V}}^{t}-\mathbf{I}_{k}\|_{\mathsf{F}}^{2}$. We have from Inequality (40):

$$ $\displaystyle\textstyle\tilde{c}$ $\displaystyle\triangleq$ $\displaystyle\textstyle(F(\mathbf{X}^{0})-F(\ddot{\mathbf{X}}))\cdot\tfrac{2}{\alpha}$ (41) $\displaystyle\geq$ $\displaystyle\textstyle\sum_{t=0}^{T}e^{t}$ $\displaystyle\overset{}{=}$ $\displaystyle\textstyle e^{0}+\sum_{i=1}^{z}e^{i}+\sum_{i=z+1}^{2z}e^{i}+\sum_ {i=2z+1}^{3z}e^{i}+\ldots$ $\displaystyle\textstyle+\sum_{i=[\lfloor T/z\rfloor-1]z+1}^{\lfloor T/z\rfloor z }e^{i}+\sum_{i=\lfloor T/z\rfloor z}^{T}e^{i}$ $\displaystyle\overset{\text{\char 172}}{\geq}$ $\displaystyle\textstyle\sum_{i=1}^{z}e^{i}+\sum_{i=z+1}^{2z}e^{i}+\sum_{i=2z+1 }^{3z}e^{i}+\ldots+\sum_{i=[\lfloor T/z\rfloor-1]z+1}^{\lfloor T/z\rfloor z}e^ {i}$ $\displaystyle\geq$ $\displaystyle\textstyle(\min_{k=1}^{\lfloor T/z\rfloor}[{\sum_{i=(k-1)z+1}^{kz }}e^{i}])\times\lfloor T/z\rfloor,$ $\displaystyle\overset{\text{\char 173}}{\geq}$ $\displaystyle\textstyle(\min_{k=1}^{\lfloor T/z\rfloor}[{\sum_{i=(k-1)z+1}^{kz }}e^{i}])\times(\tfrac{T-z}{z}),$ $$

where step ① uses $e^{i}\geq 0$ for all $i$, and $T\geq\lfloor T/z\rfloor z$ for all $T\geq 0$; step ② uses $\lfloor T/z\rfloor\geq\tfrac{T}{z}-1$ for all $T>z>0$. Inequality (41) implies that there exists an index $\bar{k}$ with $\bar{k}\in[1,\lfloor T/z\rfloor]$ satisfying

$$ $\displaystyle\textstyle\tfrac{1}{z}{\sum_{i=1+(\bar{k}-1)z}^{\bar{k}z}}e^{i} \leq\tfrac{\tilde{c}}{T-z}.$ (42) $$

Such inequality further implies the associated solution $\mathbf{X}^{\bar{k}z}$ qualifies as an $\epsilon$-BSk-point of Problem (1), provided that $T$ is sufficiently large such that $\tfrac{\tilde{c}}{T-z}\leq\epsilon$ and $T>z$. We conclude that OBCD finds an $\epsilon$-BSk-point of Problem (1) in at most $T$ iterations deterministically, where $T\geq\lceil\tfrac{\tilde{c}}{\epsilon}+z\rceil$.

∎

###### Proof.

### G.2 Proof of Lemma 4.7

###### Proof.

For notation simplicity, we define: $\|\partial F(\mathbf{X})\|_{\mathsf{F}}=\inf_{\mathbf{Y}\in\partial F(\mathbf{
X})}\|\mathbf{Y}\|_{\mathsf{F}}={\rm{dist}}(\mathbf{0},\partial F(\mathbf{X}))$.

We define $\mathbb{A}\ominus\mathbb{B}$ as the element-wise subtraction between sets $\mathbb{A}$ and $\mathbb{B}$.

We let $\mathds{H}^{t+1}\in\partial h(\mathbf{X}^{t+1})$, and define:

$$ $\displaystyle\Omega_{0}$ $\displaystyle\triangleq$ $\displaystyle\mathrm{U}_{\texttt{{B}}^{t}}^{\mathsf{T}}[\nabla f(\mathbf{X}^{t +1})+\mathds{H}^{t+1}][\mathbf{X}^{t+1}]^{\mathsf{T}}\mathrm{U}_{\texttt{{B}}^ {t}}\in\mathbb{R}^{k\times k},$ (43) $\displaystyle\Omega_{1}$ $\displaystyle\triangleq$ $\displaystyle\mathrm{U}_{\texttt{{B}}^{t}}^{\mathsf{T}}[\nabla f(\mathbf{X}^{t +1})+\mathds{H}^{t+1}][\mathbf{X}^{t}]^{\mathsf{T}}\mathrm{U}_{\texttt{{B}}^{t }}\in\mathbb{R}^{k\times k},$ (44) $\displaystyle\Omega_{2}$ $\displaystyle\triangleq$ $\displaystyle\mathrm{U}_{\texttt{{B}}^{t}}^{\mathsf{T}}[\nabla f(\mathbf{X}^{t })-\nabla f(\mathbf{X}^{t+1})][\mathbf{X}^{t}]^{\mathsf{T}}\mathrm{U}_{\texttt {{B}}^{t}}\in\mathbb{R}^{k\times k}.$ (45) $$

First, using the optimality of $\bar{\mathbf{V}}^{t}$ for the subproblem, we have:

$$ $\displaystyle\textbf{0}_{k,k}=\tilde{\mathbf{G}}-\bar{\mathbf{V}}^{t}\tilde{ \mathbf{G}}^{\mathsf{T}}\bar{\mathbf{V}}^{t}$ where $\displaystyle\tilde{\mathbf{G}}=\underbrace{\mathrm{mat}((\mathbf{Q}+\alpha \mathbf{I}_{k})\mathrm{vec}(\bar{\mathbf{V}}^{t}-\mathbf{I}_{k}))}_{\triangleq \Upsilon_{1}}+\underbrace{\mathrm{U}_{\texttt{{B}}^{t}}^{\mathsf{T}}[\nabla f( \mathbf{X}^{t})+\mathds{H}^{t+1}](\mathbf{X}^{t})^{\mathsf{T}}\mathrm{U}_{ \texttt{{B}}^{t}}}_{\triangleq\Upsilon_{2}}.$ $$

Using the relation that $\tilde{\mathbf{G}}=\Upsilon_{1}+\Upsilon_{2}$, we obtain the following results from the above equality:

$$ $\displaystyle\mathbf{0}_{k,k}=(\Upsilon_{1}+\Upsilon_{2})-\bar{\mathbf{V}}^{t} (\Upsilon_{1}+\Upsilon_{2})^{\mathsf{T}}\bar{\mathbf{V}}^{t}$ (46) $\displaystyle\overset{\text{\char 172}}{\Rightarrow}$ $\displaystyle\mathbf{0}_{k,k}=\Upsilon_{1}+\Omega_{1}+\Omega_{2}-\bar{\mathbf{ V}}^{t}(\Upsilon_{1}+\Omega_{1}+\Omega_{2})^{\mathsf{T}}\bar{\mathbf{V}}^{t}$ $\displaystyle\Rightarrow$ $\displaystyle\Omega_{1}=\bar{\mathbf{V}}^{t}(\Upsilon_{1}+\Omega_{1}+\Omega_{2 })^{\mathsf{T}}\bar{\mathbf{V}}^{t}-\Upsilon_{1}-\Omega_{2},$ $$

where step ① uses $\Upsilon_{2}=\Omega_{1}+\Omega_{2}$.

Second, since both $\texttt{{B}}^{t}$ and $\texttt{{B}}^{t+1}$ are randomly and dependently selected from $\{\mathcal{B}_{i}\}$${}_{i=1}^{\mathrm{C}_{n}^{k}}$ with replacement, each with an equal probability of $\tfrac{1}{\mathrm{C}_{n}^{k}}$, for any $\mathbf{\tilde{A}}\in\mathbb{R}^{n\times n}$, we have:

$$ $\displaystyle\textstyle\mathbb{E}_{\texttt{{B}}^{t+1}}[\|\mathrm{U}_{{\texttt{ {B}}}^{t+1}}^{\mathsf{T}}\mathbf{\tilde{A}}\mathrm{U}_{{\texttt{{B}}}^{t+1}}\| _{\mathsf{F}}^{2}=\tfrac{1}{\mathrm{C}_{n}^{k}}\sum_{i=1}^{\mathrm{C}_{n}^{k}} \|\mathbf{U}_{\mathcal{B}_{i}}^{\mathsf{T}}\mathbf{\tilde{A}}\mathbf{U}_{ \mathcal{B}_{i}}\|_{\mathsf{F}}^{2}=\mathbb{E}_{\texttt{{B}}^{t}}\|\mathrm{U}_ {\texttt{{B}}^{t}}^{\mathsf{T}}\mathbf{\tilde{A}}\mathrm{U}_{\texttt{{B}}^{t}} \|_{\mathsf{F}}^{2}.$ (47) $$

Third, we derive the following results:

$$ $\displaystyle\mathbb{E}_{\xi^{t+1}}[{\rm{dist}}(\mathbf{0},\partial_{\mathcal{ M}}\mathcal{K}(\mathbf{I}_{k};\mathbf{X}^{t+1},\texttt{{B}}^{t+1}))]=\mathbb{E }_{\xi^{t+1}}[\|\partial_{\mathcal{M}}\mathcal{K}(\mathbf{I}_{k};\mathbf{X}^{t +1},\texttt{{B}}^{t+1})\|_{\mathsf{F}}]$ (48) $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\mathbb{E}_{\xi^{t+1}}[\|\mathrm{U}_{{\texttt{{B}}}^{t+1}}^{ \mathsf{T}}\{\partial F(\mathbf{X}^{t+1})[\mathbf{X}^{t+1}]^{\mathsf{T}} \ominus\mathbf{X}^{t+1}[\partial F(\mathbf{X}^{t+1})]^{\mathsf{T}}\}\mathrm{U} _{{\texttt{{B}}}^{t+1}}\|_{\mathsf{F}}]$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle\mathbb{E}_{\xi^{t}}[\|\mathrm{U}_{\texttt{{B}}^{t}}^{\mathsf{T}} \{\partial F(\mathbf{X}^{t+1})[\mathbf{X}^{t+1}]^{\mathsf{T}}\ominus\mathbf{X} ^{t+1}[\partial F(\mathbf{X}^{t+1})]^{\mathsf{T}}\}\mathrm{U}_{\texttt{{B}}^{t }}\|_{\mathsf{F}}]$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\mathbb{E}_{\xi^{t}}[\|\Omega_{0}-\Omega_{0}^{\mathsf{T}}\|_{ \mathsf{F}}]$ $\displaystyle\overset{\text{\char 175}}{\leq}$ $\displaystyle 2\mathbb{E}_{\xi^{t}}[\|\Omega_{0}-\Omega_{1}\|_{\mathsf{F}}]+ \mathbb{E}_{\xi^{t}}[\|\Omega_{1}-\Omega_{1}^{\mathsf{T}}\|_{\mathsf{F}}]$ $\displaystyle\overset{\text{\char 176}}{=}$ $\displaystyle 2\mathbb{E}_{\xi^{t}}[\|\Omega_{0}-\Omega_{1}\|_{\mathsf{F}}]+ \mathbb{E}_{\xi^{t}}[\|\bar{\mathbf{V}}^{t}(\Upsilon_{1}+\Omega_{1}+\Omega_{2} )^{\mathsf{T}}\bar{\mathbf{V}}^{t}-\Upsilon_{1}-\Omega_{2}-\Omega_{1}^{\mathsf {T}}\|_{\mathsf{F}}]$ $\displaystyle\overset{\text{\char 177}}{=}$ $\displaystyle 2\mathbb{E}_{\xi^{t}}[\|\Omega_{0}-\Omega_{1}\|_{\mathsf{F}}]+ \mathbb{E}_{\xi^{t}}[\|\bar{\mathbf{V}}^{t}\Upsilon_{1}^{\mathsf{T}}\bar{ \mathbf{V}}^{t}-\Upsilon_{1}\|_{\mathsf{F}}]+\mathbb{E}_{\xi^{t}}[\|\bar{ \mathbf{V}}^{t}\Omega_{1}^{\mathsf{T}}\bar{\mathbf{V}}^{t}-\Omega_{1}^{\mathsf {T}}\|_{\mathsf{F}}]$ $\displaystyle\textstyle+\mathbb{E}_{\xi^{t}}[\|\bar{\mathbf{V}}^{t}\Omega_{2}^ {\mathsf{T}}\bar{\mathbf{V}}^{t}-\Omega_{2}\|_{\mathsf{F}}]~{}~{}~{}~{}$ $$

where step ① uses the definition of $\partial_{\mathcal{M}}\mathcal{K}(\mathbf{V};\mathbf{X}^{t+1},\texttt{{B}}^{t+
1})$ at the point $\mathbf{V}=\mathbf{I}_{k}$; step ② uses Equality (47) with $\mathbf{\tilde{A}}=\partial F(\mathbf{X}^{t+1})(\mathbf{X}^{t+1})^{\mathsf{T}}
\ominus\mathbf{X}^{t+1}(\partial F(\mathbf{X}^{t+1}))^{\mathsf{T}}$; step ③ uses the definition of $\Omega_{0}$ in Equation (43); step ④ uses Lemma A.2; step ⑤ uses Equality (46); step ⑥ uses the triangle inequality.

We now establish individual bounds for each term in Inequality (48). For the first term $2\mathbb{E}_{\xi^{t}}[\|\Omega_{0}-\Omega_{1}\|_{\mathsf{F}}]$ in (48), we have:

$$ $\displaystyle 2\mathbb{E}_{\xi^{t}}[\|\Omega_{0}-\Omega_{1}\|_{\mathsf{F}}]$ (49) $\displaystyle\leq$ $\displaystyle 2\mathbb{E}_{\xi^{t}}[\|\mathrm{U}_{\texttt{{B}}^{t}}^{\mathsf{T }}[\nabla f(\mathbf{X}^{t+1})+\mathds{H}^{t+1}][\mathbf{X}^{t+1}-\mathbf{X}^{t }]^{\mathsf{T}}\mathrm{U}_{\texttt{{B}}^{t}}\|_{\mathsf{F}}]$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle 2\mathbb{E}_{\xi^{t}}[\|\mathrm{U}_{\texttt{{B}}^{t}}^{\mathsf{T }}[\nabla f(\mathbf{X}^{t+1})+\mathds{H}^{t+1}][\mathrm{U}_{\texttt{{B}}}(\bar {\mathbf{V}}^{t}-\mathbf{I}_{k})\mathrm{U}_{\texttt{{B}}^{t}}\mathbf{X}^{t}]^{ \mathsf{T}}\mathrm{U}_{\texttt{{B}}^{t}}\|_{\mathsf{F}}]$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle 2(l_{f}+l_{h})\mathbb{E}_{\xi^{t}}[\|\bar{\mathbf{V}}^{t}- \mathbf{I}_{k}\|_{\mathsf{F}}],$ $$

where step ① uses $\mathbf{X}^{t+1}=\mathbf{X}^{t}+\mathrm{U}_{\texttt{{B}}}(\bar{\mathbf{V}}^{t}
-\mathbf{I}_{k})\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}^{t}$; step ② uses the inequality $\|\mathbf{X}\mathbf{Y}\|_{\mathsf{F}}\leq\|\mathbf{X}\|_{\mathsf{F}}\|\mathbf{
Y}\|_{2}$ for all $\mathbf{X}$ and $\mathbf{Y}$ repeatedly, and the fact that $\forall\mathbf{X},\|\nabla f(\mathbf{X})\|_{2}\leq l_{f},\|\partial h(\mathbf{
X})\|_{2}\leq l_{h}$.

For the second term $\mathbb{E}_{\xi^{t}}[\|\bar{\mathbf{V}}^{t}\Upsilon_{1}^{\mathsf{T}}\bar{
\mathbf{V}}^{t}-\Upsilon_{1}\|_{\mathsf{F}}]$ in (48), we have::

$$ $\displaystyle\mathbb{E}_{\xi^{t}}[\|\bar{\mathbf{V}}^{t}\Upsilon_{1}^{\mathsf{ T}}\bar{\mathbf{V}}^{t}-\Upsilon_{1}\|_{\mathsf{F}}]$ (50) $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\mathbb{E}_{\xi^{t}}[\|\bar{\mathbf{V}}^{t}\Upsilon_{1}^{\mathsf{ T}}\bar{\mathbf{V}}^{t}\|_{\mathsf{F}}]+\mathbb{E}_{\xi^{t}}[\|\Upsilon_{1}\|_ {\mathsf{F}}]$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle 2\mathbb{E}_{\xi^{t}}[\|\Upsilon_{1}\|_{\mathsf{F}}]$ $\displaystyle\overset{\text{\char 174}}{=}$ $\displaystyle 2\mathbb{E}_{\xi^{t}}[\|\mathrm{mat}((\mathbf{Q}+\alpha\mathbf{I }_{k})\mathrm{vec}(\bar{\mathbf{V}}^{t}-\mathbf{I}_{k}))\|_{\mathsf{F}}]$ $\displaystyle\leq$ $\displaystyle 2\|\mathbf{Q}+\alpha\mathbf{I}_{k}\|_{2}\cdot\mathbb{E}_{\xi^{t} }[\|\bar{\mathbf{V}}^{t}-\mathbf{I}_{k})\|_{\mathsf{F}}]$ $\displaystyle\overset{\text{\char 175}}{\leq}$ $\displaystyle 2(L_{f}+\alpha)\cdot\mathbb{E}_{\xi^{t}}[\|\bar{\mathbf{V}}^{t}- \mathbf{I}_{k})\|_{\mathsf{F}}]$ $$

where step ① uses the triangle inequality; step ② uses the inequality $\|\mathbf{X}\mathbf{Y}\|_{\mathsf{F}}\leq\|\mathbf{X}\|_{\mathsf{F}}\|\mathbf{
Y}\|_{2}$ for all $\mathbf{X}$ and $\mathbf{Y}$; step ③ uses the definition of $\Omega_{1}$ in (44); step ④ uses the fact that $\|\mathbf{Q}\|\leq L_{f}$.

For the third term $\mathbb{E}_{\xi^{t}}[\|\bar{\mathbf{V}}^{t}\Omega_{1}^{\mathsf{T}}\bar{\mathbf
{V}}^{t}-\Omega_{1}^{\mathsf{T}}\|_{\mathsf{F}}]$ in (48), we have:

$$ $\displaystyle\mathbb{E}_{\xi^{t}}[\|\bar{\mathbf{V}}^{t}\Omega_{1}^{\mathsf{T} }\bar{\mathbf{V}}^{t}-\Omega_{1}^{\mathsf{T}}\|_{\mathsf{F}}]$ (51) $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\mathbb{E}_{\xi^{t}}[\|\bar{\mathbf{V}}^{t}\Omega_{1}^{\mathsf{T} }(\bar{\mathbf{V}}^{t}-\mathbf{I}_{k})+(\bar{\mathbf{V}}^{t}-\mathbf{I})\Omega _{1}^{\mathsf{T}}\|_{\mathsf{F}}]$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle 2\mathbb{E}_{\xi^{t}}[\|\Omega_{1}\|_{2}\cdot\|(\bar{\mathbf{V}} ^{t}-\mathbf{I}_{k})\|_{\mathsf{F}}]$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle 2\mathbb{E}_{\xi^{t}}[\|\nabla f(\mathbf{X}^{t+1})+\mathds{H}^{t +1}\|_{2}\cdot\|(\bar{\mathbf{V}}^{t}-\mathbf{I}_{k})\|_{\mathsf{F}}]$ $\displaystyle\overset{\text{\char 175}}{\leq}$ $\displaystyle 2(l_{f}+l_{h})\mathbb{E}_{\xi^{t}}[\|(\bar{\mathbf{V}}^{t}- \mathbf{I}_{k})\|_{\mathsf{F}}]$ $$

where step ① uses the fact that $-\bar{\mathbf{V}}^{t}\Omega_{1}^{\mathsf{T}}\mathbf{I}_{k}+\bar{\mathbf{V}}^{t
}\Omega_{1}^{\mathsf{T}}=\mathbf{0}$; step ② uses the norm inequality; step ③ uses the fact that $\|\Omega_{1}\|_{2}=\|\mathrm{U}_{\texttt{{B}}^{t}}^{\mathsf{T}}[\nabla f(
\mathbf{X}^{t+1})+\mathds{H}^{t+1}][\mathbf{X}^{t}]^{\mathsf{T}}\mathrm{U}_{
\texttt{{B}}^{t}}\|_{2}\leq\|\nabla f(\mathbf{X}^{t+1})+\mathds{H}^{t+1}\|_{2}$ which can be derived using the norm inequality; step ④ uses the fact that $\forall\mathbf{X},\|\nabla f(\mathbf{X})\|_{2}\leq l_{f},\|\partial h(\mathbf{
X})\|_{2}\leq l_{h}$.

For the fourth term $\mathbb{E}_{\xi^{t}}[\|\bar{\mathbf{V}}^{t}\Omega_{2}^{\mathsf{T}}\bar{\mathbf
{V}}^{t}-\Omega_{2}\|_{\mathsf{F}}]$ in (48), we have:

$$ $\displaystyle\mathbb{E}_{\xi^{t}}[\|\bar{\mathbf{V}}^{t}\Omega_{2}^{\mathsf{T} }\bar{\mathbf{V}}^{t}-\Omega_{2}\|_{\mathsf{F}}]$ (52) $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\mathbb{E}_{\xi^{t}}[\|\bar{\mathbf{V}}^{t}\Omega_{2}^{\mathsf{T} }\bar{\mathbf{V}}^{t}\|_{\mathsf{F}}]+\mathbb{E}[\|\Omega_{2}\|_{\mathsf{F}}]$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle 2\mathbb{E}_{\xi^{t}}[\|\Omega_{2}\|_{\mathsf{F}}]$ $\displaystyle\overset{\text{\char 174}}{=}$ $\displaystyle 2\mathbb{E}_{\xi^{t}}[\|\mathrm{U}_{\texttt{{B}}^{t}}^{\mathsf{T }}[\nabla f(\mathbf{X}^{t})-\nabla f(\mathbf{X}^{t+1})][\mathbf{X}^{t}]^{ \mathsf{T}}\mathrm{U}_{\texttt{{B}}^{t}}\|_{\mathsf{F}}]$ $\displaystyle\overset{\text{\char 175}}{=}$ $\displaystyle 2\mathbb{E}_{\xi^{t}}[\|\nabla f(\mathbf{X}^{t})-\nabla f( \mathbf{X}^{t+1})\|_{\mathsf{F}}]$ $\displaystyle\overset{\text{\char 176}}{=}$ $\displaystyle 2L_{f}\mathbb{E}_{\xi^{t}}[\|\mathbf{X}^{t}-\mathbf{X}^{t+1}\|_{ \mathsf{F}}]$ $\displaystyle\overset{\text{\char 177}}{=}$ $\displaystyle 2L_{f}\mathbb{E}_{\xi^{t}}[\|\bar{\mathbf{V}}^{t}-\mathbf{I}_{k} \|_{\mathsf{F}}],$ $$

where step ① uses the triangle inequality; step ② uses the norm inequality; step ③ uses the definition of $\Omega_{2}=\mathrm{U}_{\texttt{{B}}^{t}}^{\mathsf{T}}[\nabla f(\mathbf{X}^{t})
-\nabla f(\mathbf{X}^{t+1})][\mathbf{X}^{t}]^{\mathsf{T}}\mathrm{U}_{\texttt{{
B}}^{t}}$ in (45); step ④ uses the norm inequality; step ⑤ uses the fact that $\nabla f(\mathbf{X})$ is $L_{f}$-Lipschitz continuous; step ⑥ uses Claim (c) of Lemma 2.3.

In view of (49), (50), (51), (52), and (48), we have:

$$ $\displaystyle\mathbb{E}_{\xi^{t+1}}[\|\partial_{\mathcal{M}}\mathcal{K}( \mathbf{I}_{k};\mathbf{X}^{t+1},\texttt{{B}}^{t+1})\|_{\mathsf{F}}]$ $\displaystyle\leq$ $\displaystyle(c_{1}+c_{2}+c3+c4)\cdot\mathbb{E}_{\xi^{t}}[\|\bar{\mathbf{V}}^{ t}-\mathbf{I}_{k}\|_{\mathsf{F}}]=\phi\cdot\mathbb{E}_{\xi^{t}}[\|\bar{\mathbf {V}}^{t}-\mathbf{I}_{k}\|_{\mathsf{F}}]$ $$

where $c_{1}=2(l_{f}+l_{h})$, $c_{2}=2(L_{f}+\alpha)$, $c_{3}=2(l_{f}+l_{h})$, and $c_{4}=2L_{f}$.

∎

###### Proof.

### G.3 Proof of Theorem 4.8

###### Proof.

First, we show that ${\rm{dist}}(\mathbf{0},\partial_{\mathcal{M}}F(\mathbf{X}^{t}))\leq\gamma\cdot
\mathbb{E}[{\rm{dist}}(\mathbf{0},\partial_{\mathcal{M}}\mathcal{K}(\mathbf{I}
_{k};\mathbf{X}^{t},\texttt{{B}}^{t}))]$, where $\gamma\triangleq({\mathrm{C}_{n}^{k}}/{\mathrm{C}_{n-2}^{k-2}})^{1/2}$. For all $\mathbf{D}^{t}\triangleq\partial F(\mathbf{X}^{t})[\mathbf{X}^{t}]^{\mathsf{T}
}\ominus\mathbf{X}^{t}[\partial F(\mathbf{X}^{t})]^{\mathsf{T}}$, we obtain:

$$ $\displaystyle\|\mathbf{D}^{t}\|_{\mathsf{F}}^{2}$ $\displaystyle=$ $\displaystyle\textstyle\sum_{i}\sum_{j\neq i}(\mathbf{D}^{t}_{ij})^{2}+\sum_{i }\sum_{j=i}(\mathbf{D}^{t}_{ij})^{2}$ (53) $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\textstyle\sum_{i}\sum_{j\neq i}(\mathbf{D}^{t}_{ij})^{2}$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle\textstyle\tfrac{1}{\mathrm{C}_{n-2}^{k-2}}\sum_{i=1}^{\mathrm{C} _{n}^{k}}\|\mathbf{U}_{\mathcal{B}_{i}}^{\mathsf{T}}\mathbf{D}^{t}\mathbf{U}_{ \mathcal{B}_{i}}\|_{\mathsf{F}}^{2}$ $\displaystyle\overset{\text{\char 174}}{=}$ $\displaystyle\textstyle\tfrac{1}{\mathrm{C}_{n-2}^{k-2}}\cdot\mathrm{C}_{n}^{k }\mathbb{E}_{\texttt{{B}}^{t}}[\|\mathrm{U}_{\texttt{{B}}^{t}}^{\mathsf{T}} \mathbf{D}^{t}\mathrm{U}_{\texttt{{B}}^{t}}\|_{\mathsf{F}}^{2}]$ $\displaystyle\overset{\text{\char 175}}{=}$ $\displaystyle\textstyle\gamma^{2}\mathbb{E}_{\texttt{{B}}^{t}}[\|\mathrm{U}_{ \texttt{{B}}^{t}}^{\mathsf{T}}\mathbf{D}^{t}\mathrm{U}_{\texttt{{B}}^{t}}\|_{ \mathsf{F}}^{2}],$ $$

where step ① uses the fact that $\mathbf{D}^{t}_{ii}=0$ for all $i\in[n]$; step ② uses Claim (a) of this lemma with $\mathbf{D}^{t}_{ii}=0$ for all $i\in[n]$; step ③ uses $\mathbb{E}_{\texttt{{B}}^{t}}[\|\mathrm{U}_{\texttt{{B}}^{t}}^{\mathsf{T}}
\mathbf{W}\mathrm{U}_{\texttt{{B}}^{t}}\|_{\mathsf{F}}^{2}]=\tfrac{1}{\mathrm{
C}_{n}^{k}}\sum_{i=1}^{\mathrm{C}_{n}^{k}}\|\mathbf{U}_{\mathcal{B}_{i}}^{
\mathsf{T}}\mathbf{W}\mathbf{U}_{\mathcal{B}_{i}}\|_{\mathsf{F}}^{2}$ as $\texttt{{B}}^{t}$ are chosen from $\{\mathcal{B}_{i}\}$${}_{i=1}^{\mathrm{C}_{n}^{k}}$ randomly and uniformly; ④ uses the definition of $\gamma$. We further derive:

$$ $\displaystyle\|\partial_{\mathcal{M}}F(\mathbf{X}^{t})\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\|\partial F(\mathbf{X}^{t})\ominus\mathbf{X}^{t}[\partial F( \mathbf{X}^{t})]^{\mathsf{T}}\mathbf{X}^{t}\|_{\mathsf{F}}$ (54) $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle\|\partial F(\mathbf{X}^{t})[\mathbf{X}^{t}]^{\mathsf{T}}\mathbf{ X}^{t}\ominus\mathbf{X}^{t}[\partial F(\mathbf{X}^{t})]^{\mathsf{T}}\mathbf{X} ^{t}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\|\partial F(\mathbf{X}^{t})[\mathbf{X}^{t}]^{\mathsf{T}}\ominus \mathbf{X}^{t}[\partial F(\mathbf{X}^{t})]^{\mathsf{T}}\|_{\mathsf{F}}$ $\displaystyle\overset{\text{\char 175}}{=}$ $\displaystyle\gamma\mathbb{E}_{\texttt{{B}}^{t}}[\|\mathrm{U}_{\texttt{{B}}^{t }}^{\mathsf{T}}\{\partial F(\mathbf{X}^{t})[\mathbf{X}^{t}]^{\mathsf{T}} \ominus\mathbf{X}^{t}[\partial F(\mathbf{X}^{t})]^{\mathsf{T}}\}\mathrm{U}_{ \texttt{{B}}^{t}}\|_{\mathsf{F}}]$ $\displaystyle\overset{\text{\char 176}}{=}$ $\displaystyle\gamma\|\partial_{\mathcal{M}}\mathcal{K}(\mathbf{I}_{k};\mathbf{ X}^{t},\texttt{{B}}^{t})\|_{\mathsf{F}}$ $$

where step ① uses the definition of $\partial_{\mathcal{M}}F(\mathbf{X}^{t}))$; step ② uses $[\mathbf{X}^{t}]^{\mathsf{T}}\mathbf{X}^{t}=\mathbf{I}_{r}$; step ③ uses the inequality that $\|\mathbf{A}\mathbf{X}\|_{\mathsf{F}}^{2}\leq\|\mathbf{A}\|_{\mathsf{F}}^{2}$ for all $\mathbf{X}\in\mathrm{St}(n,r)$; step ④ uses Equality (53); step ⑤ uses the definition of $\partial_{\mathcal{M}}\mathcal{K}(\mathbf{I}_{k};\mathbf{X}^{t},\texttt{{B}}^{
t})$.

Second, given $F^{\circ}(\mathbf{X})\triangleq F(\mathbf{X})+\mathcal{I}_{\mathcal{M}}(
\mathbf{X})$ is a KL function by our assumption, we can conclude, from Proposition 4.6, that:

$$ $\displaystyle\tfrac{1}{\varphi^{\prime}(F^{\circ}(\mathbf{X}^{t})-F^{\circ}( \ddot{\mathbf{X}}))}\leq\text{dist}(0,\partial F^{\circ}(\mathbf{X}^{t})).$ (55) $$

Since $\varphi(\cdot)$ is a concave desingularization function, we have: $\varphi(b)+(a-b)\varphi^{\prime}(a)\leq\varphi(a)$. Applying the inequality above with $a=F(\mathbf{X}^{t})-F(\ddot{\mathbf{X}})$ and $b=F(\mathbf{X}^{t+1})-F(\ddot{\mathbf{X}})$, we have:

$$ $\displaystyle(F(\mathbf{X}^{t})-F(\mathbf{X}^{t+1}))\varphi^{\prime}(F(\mathbf {X}^{t})-F(\ddot{\mathbf{X}}))$ (56) $\displaystyle\leq$ $\displaystyle\mathcal{E}^{t}\triangleq\varphi(F(\mathbf{X}^{t})-F(\ddot{ \mathbf{X}}))-\varphi(F(\mathbf{X}^{t+1})-F(\ddot{\mathbf{X}})).$ $$

Third, we derive the following inequalities:

$$ $\displaystyle\mathbb{E}_{\xi^{t}}[\tfrac{\alpha}{2}\|\bar{\mathbf{V}}^{t}- \mathbf{I}_{k}\|_{\mathsf{F}}^{2}]$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle~{}\mathbb{E}_{\xi^{t}}[F(\mathbf{X}^{k})-F(\mathbf{X}^{k+1})]$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle~{}\mathbb{E}_{\xi^{t}}[\tfrac{\mathcal{E}^{t}}{\varphi^{\prime}( F(\mathbf{X}^{t})-F(\ddot{\mathbf{X}}))}]$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle~{}\mathbb{E}_{\xi^{t}}[\mathcal{E}^{t}\cdot\text{dist}(0, \partial F^{\circ}(\mathbf{X}^{t}))]$ $\displaystyle\overset{\text{\char 175}}{\leq}$ $\displaystyle~{}\mathbb{E}_{\xi^{t}}[\mathcal{E}^{t}\cdot\|\partial_{\mathcal{ M}}F(\mathbf{X}^{t})\|_{\mathsf{F}}]$ $\displaystyle\overset{\text{\char 176}}{\leq}$ $\displaystyle~{}\mathbb{E}_{\xi^{t}}[\mathcal{E}^{t}\gamma\|\partial_{\mathcal {M}}\mathcal{K}(\mathbf{I}_{k};\mathbf{X}^{t},\texttt{{B}}^{t})\|_{\mathsf{F}}]$ $\displaystyle\overset{\text{\char 177}}{\leq}$ $\displaystyle~{}\mathbb{E}_{\xi^{t-1}}[\mathcal{E}^{t}\gamma\phi\|\bar{\mathbf {V}}^{t-1}-\mathbf{I}_{k}\|_{\mathsf{F}}]$ $\displaystyle\overset{\text{\char 178}}{\leq}$ $\displaystyle~{}\tfrac{\alpha^{\prime}}{2}\mathbb{E}_{\xi^{t-1}}[\|\bar{ \mathbf{V}}^{t-1}-\mathbf{I}_{k}\|_{\mathsf{F}}^{2}]+\tfrac{(\mathcal{E}^{t} \gamma\phi)^{2}}{2\alpha^{\prime}},~{}\forall\alpha^{\prime}>0,$ $$

where step ① uses the sufficient decrease condition as shown in Theorem 4.2; step ② uses Inequality (56); step ③ uses Inequality (55); step ④ uses Lemma A.7; step ⑤ uses Inequality (54); step ⑥ uses Lemma 4.7; step ⑦ applies the inequality that $\forall\alpha^{\prime}>0,a,b,ab\leq\tfrac{\alpha^{\prime}a^{2}}{2}+\tfrac{b^{2
}}{2\alpha^{\prime}}$ with $a=\mathbb{E}_{\xi^{t-1}}[\|\bar{\mathbf{V}}^{t-1}-\mathbf{I}_{k}\|_{\mathsf{F}}]$ and $b=\mathcal{E}^{t}\gamma\phi$. Multiplying both sides by 2 and taking the square root yields:

$$ $\displaystyle\sqrt{\alpha}\mathbb{E}_{\xi^{t}}[\|\bar{\mathbf{V}}^{t}-\mathbf{ I}_{k}\|_{\mathsf{F}}]$ $\displaystyle\leq$ $\displaystyle\sqrt{\alpha^{\prime}\mathbb{E}_{\xi^{t-1}}[\|\bar{\mathbf{V}}^{t -1}-\mathbf{I}_{k}\|_{\mathsf{F}}^{2}]+\tfrac{(\mathcal{E}^{t}\gamma\phi)^{2}} {\alpha^{\prime}}},~{}\forall\alpha^{\prime}>0$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle\sqrt{\alpha^{\prime}}\mathbb{E}_{\xi^{t-1}}[\|\bar{\mathbf{V}}^{ t-1}-\mathbf{I}_{k}\|_{\mathsf{F}}]+\tfrac{\mathcal{E}^{t}\gamma\phi}{\sqrt{ \alpha^{\prime}}},~{}\forall\alpha^{\prime}>0,$ $$

where step ① uses the inequality that $\sqrt{a+b}\leq\sqrt{a}+\sqrt{b}$ for all $a\geq 0$ and $b\geq 0$. Summing the inequality above over $i$ from 1 to $t$, we have:

$$ $\displaystyle\textstyle\sqrt{\alpha}\mathbb{E}_{\xi^{t}}[\|\bar{\mathbf{V}}^{t }-\mathbf{I}_{k}\|_{\mathsf{F}}]-\sqrt{\alpha^{\prime}}\mathbb{E}_{\xi^{0}}[\| \bar{\mathbf{V}}^{0}-\mathbf{I}_{k}\|_{\mathsf{F}}]+\sum_{i=1}^{t-1}(\sqrt{ \alpha}-\sqrt{\alpha^{\prime}})\mathbb{E}_{\xi^{i}}[\|\bar{\mathbf{V}}^{i}- \mathbf{I}_{k}\|_{\mathsf{F}}]$ (57) $\displaystyle\leq$ $\displaystyle\textstyle\tfrac{\gamma\phi}{\sqrt{\alpha^{\prime}}}\sum_{i=1}^{t }\mathcal{E}^{i}$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle\textstyle\tfrac{\gamma\phi}{\sqrt{\alpha^{\prime}}}\sum_{i=1}^{t }\varphi(F(\mathbf{X}^{i})-F(\ddot{\mathbf{X}}))-\varphi(F(\mathbf{X}^{i+1})-F (\ddot{\mathbf{X}}))$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle\textstyle\tfrac{\gamma\phi}{\sqrt{\alpha^{\prime}}}[\varphi(F( \mathbf{X}^{1})-F(\ddot{\mathbf{X}}))-\varphi(F(\mathbf{X}^{t+1})-F(\ddot{ \mathbf{X}}))]$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle\textstyle\tfrac{\gamma\phi}{\sqrt{\alpha^{\prime}}}\varphi(F( \mathbf{X}^{1})-F(\ddot{\mathbf{X}})),$ $$

where step ① uses the definition of $\mathcal{E}^{i}$ as provided in (56); step ② uses a basic recursive reduction technique; step ③ uses the positivity of the desingularization function $\varphi(\cdot)$.

Finally, with the choice $\alpha^{\prime}=\tfrac{\alpha}{4}$ for Inequality (57), we have:

$$ $\displaystyle\textstyle\sqrt{\alpha}\mathbb{E}_{\xi^{t}}[\|\bar{\mathbf{V}}^{t }-\mathbf{I}_{k}\|_{\mathsf{F}}]+\tfrac{\sqrt{\alpha}}{2}\sum_{i=1}^{t-1} \mathbb{E}_{\xi^{i}}[\|\bar{\mathbf{V}}^{i}-\mathbf{I}_{k}\|_{\mathsf{F}}]- \tfrac{2\gamma\phi}{\sqrt{\alpha}}\varphi(F(\mathbf{X}^{1})-F(\ddot{\mathbf{X} }))$ (58) $\displaystyle\leq$ $\displaystyle\textstyle\tfrac{\sqrt{\alpha}}{2}\mathbb{E}_{\xi^{0}}[\|\bar{ \mathbf{V}}^{0}-\mathbf{I}_{k}\|_{\mathsf{F}}]\overset{\text{\char 172}}{\leq} \textstyle\sqrt{\alpha k},$ $$

where step ① uses $\tfrac{1}{2}\|\mathbf{V}-\mathbf{I}_{k}\|_{\mathsf{F}}\leq\tfrac{1}{2}\|
\mathbf{V}\|_{\mathsf{F}}+\tfrac{1}{2}\|\mathbf{I}\|_{\mathsf{F}}\leq\sqrt{k}$ for all $\mathbf{V}\in\mathrm{St}(k,k)$. Finally, we obtain from Inequality (58):

$$ $\displaystyle\textstyle\sum_{i=1}^{t}\mathbb{E}_{\xi^{i}}[\|\bar{\mathbf{V}}^{ i}-\mathbf{I}_{k}\|_{\mathsf{F}}]\leq\left(\tfrac{4\gamma\phi}{\alpha}\varphi( F(\mathbf{X}^{1})-F(\ddot{\mathbf{X}}))+2\sqrt{k}\right)\triangleq C.$ $$

Using the fact that $\|\mathbf{X}^{t+1}-\mathbf{X}^{t}\|_{\mathsf{F}}^{2}\leq\|\bar{\mathbf{V}}^{t}
-\mathbf{I}_{k}\|_{\mathsf{F}}^{2}$ for all $t\geq 0$ as shown in Claim (c) of Lemma 2.3, we have:

$$ $\displaystyle\textstyle\sum_{i=1}^{t}\mathbb{E}_{\xi^{i}}[\|\mathbf{X}^{i+1}- \mathbf{X}^{i}\|_{\mathsf{F}}]\leq C.$ $$

∎

###### Proof.

## Appendix H Proof for Section 5

### H.1 Proof of Lemma 5.1

###### Proof.

We define $w\triangleq c-e$. We define $\breve{F}(\tilde{c},\tilde{s})\triangleq a\tilde{c}+b\tilde{s}+c\tilde{c}^{2}+
d\tilde{c}\tilde{s}+e\tilde{s}^{2}+h(\tilde{c}\mathbf{x}+\tilde{s}\mathbf{y})$.

Initially, using $\sin^{2}(\theta)=1-\cos^{2}(\theta)$, we obtain the following problem, which is equivalent to Problem (11):

$$ $\displaystyle\bar{\theta}\in\arg\min_{\theta}a\cos(\theta)+b\sin(\theta)+w\cos ^{2}(\theta)+d\cos(\theta)\sin(\theta)+e+h(\cos(\theta)\mathbf{x}+\sin(\theta) \mathbf{y}).$ (59) $$

We assume $\cos(\theta)\neq 0$. Using Lemma A.8, we consider the two cases for $(\cos(\theta),\sin(\theta))$ in Problem (59).

Case a). $\cos({\theta})=\tfrac{1}{\sqrt{1+\tan^{2}(\theta)}}$, and $\sin({\theta})=\tfrac{\tan(\theta)}{\sqrt{1+\tan^{2}(\theta)}}$. Problem (11) reduces to:

$$ $\displaystyle\bar{\theta}_{+}\in\arg\min_{\theta}\tfrac{a+\tan(\theta)b}{\sqrt {1+\tan^{2}(\theta)}}+\tfrac{w+\tan(\theta)d}{{1+\tan^{2}(\theta)}}+h(\tfrac{ \mathbf{x}+\tan(\theta)\mathbf{y}}{\sqrt{1+\tan^{2}(\theta)}}).$ $$

Defining $t=\tan(\theta)$, we have the following equivalent problem:

$$ $\displaystyle\bar{t}_{+}\in\arg\min_{t}\tfrac{a+bt}{\sqrt{1+t^{2}}}+\tfrac{w+ dt}{{1+t^{2}}}+h(\tfrac{\mathbf{x}+\mathbf{y}t}{\sqrt{1+t^{2}}}).$ $$

Therefore, the optimal solution $\bar{\theta}_{+}$ can be computed as:

$$ $\displaystyle\cos(\bar{\theta}_{+})=\tfrac{1}{\sqrt{1+(\bar{t}_{+})^{2}}},~{} \sin(\bar{\theta}_{+})=\tfrac{\bar{t}_{+}}{\sqrt{1+(\bar{t}_{+})^{2}}}.$ (60) $$

Case b). $\cos({\theta})=\tfrac{-1}{\sqrt{1+\tan(\theta)^{2}}}$, and $\sin({\theta})=\tfrac{-\tan(\theta)}{\sqrt{1+\tan(\theta)^{2}}}$. Problem (11) boils down to:

$$ $\displaystyle\bar{\theta}_{-}\in\arg\min_{\theta}\tfrac{-a-\tan(\theta)b}{ \sqrt{1+\tan(\theta)^{2}}}+\tfrac{w+\tan(\theta)d}{{1+\tan(\theta)^{2}}}+h( \tfrac{-\mathbf{x}-\tan(\theta)\mathbf{y}}{\sqrt{1+\tan(\theta)^{2}}}).$ $$

Defining $t=\tan(\theta)$, we have the following equivalent problem:

$$ $\displaystyle\bar{t}_{-}\in\arg\min_{t}\tfrac{-a-bt}{\sqrt{1+t^{2}}}+\tfrac{w+ dt}{{1+t^{2}}}+h(\tfrac{-\mathbf{x}-\mathbf{y}t}{\sqrt{1+t^{2}}}).$ $$

Therefore, the optimal solution $\bar{\theta}_{-}$ can be computed as:

$$ $\displaystyle\cos(\bar{\theta}_{-})=\tfrac{-1}{\sqrt{1+(\bar{t}_{-})^{2}}},~{} \sin(\bar{\theta}_{-})=\tfrac{-\bar{t}_{-}}{\sqrt{1+(\bar{t}_{-})^{2}}}$ (61) $$

In view of (60) and (61), when $\cos(\theta)\neq 0$, the optimal solution $\bar{\theta}$ for Problem (59) is computed as: $[\cos(\bar{\theta}),\sin(\bar{\theta})]\in\arg\min_{c,s}\,\breve{F}(c,s),\,s.t
.\,[c,s]\in\{[\cos(\bar{\theta}_{+}),\sin(\bar{\theta}_{+})],[\cos(\bar{\theta
}_{-}),\sin(\bar{\theta}_{-})]\}$. Taking into account the case when $\cos(\theta)=0$, the optimal solution $\bar{\theta}$ for Problem (59) is computed as:

$$ $\displaystyle~{}~{}~{}~{}~{}~{}[\cos(\bar{\theta}),\sin(\bar{\theta})]\in\arg \min_{c,s}\,\breve{F}(c,s),$ $\displaystyle s.t.\,[c,s]\in\{[\cos(\bar{\theta}_{+}),\sin(\bar{\theta}_{+})], [\cos(\bar{\theta}_{-}),\sin(\bar{\theta}_{-})],[0,1],[0,-1]\}.$ $$

Notably, $\{\cos(\bar{\theta}),\sin(\bar{\theta})\}$ uniquely determines $\bar{\theta}$. Moreover, since the objective function in Problem (11) solely depends on $\{\cos(\theta),\sin(\theta)\}$, computing the exact values of $\bar{\theta}_{+}$ for (60) and $\bar{\theta}_{-}$ for (61) is unnecessary.

∎

###### Proof.

## Appendix I Proof for Appendix Section D

### I.1 Proof of Lemma D.1

###### Proof.

(a) The proof is similar to that of Theorem 3.6. We omit the proof for brevity.

(b) Note that the matrix $\mathbf{S}$ is an anti-symmetric matrix with $\mathbf{S}=-\mathbf{S}^{\mathsf{T}}$ and $\mathrm{diag}(\mathbf{S})=\mathbf{0}$. By observing $[\bar{i},\bar{j}]=\arg\max_{i\in[n],j\in[n],i\neq j}|\mathbf{S}_{ij}|$, we can conclude that:

$$ $\displaystyle\mathbf{S}(\bar{i},\bar{j})=0~{}~{}\Leftrightarrow~{}~{}\mathbf{S }=\mathbf{0}.$ $$

∎

###### Proof.

### I.2 Proof of Theorem D.2

###### Proof.

We define $\texttt{{B}}=[i,j]$.

We define $c_{1}\triangleq\mathbf{T}_{ii}+\mathbf{T}_{jj}$, $c_{2}\triangleq\mathbf{T}_{ij}-\mathbf{T}_{ji}$, $c_{3}\triangleq\mathbf{T}_{jj}-\mathbf{T}_{ii}$ and $c_{4}\triangleq\mathbf{T}_{ij}+\mathbf{T}_{ji}$.

(a) We now focus on the following optimization problem

$$ $\displaystyle\mathbf{S}_{ij}=\min_{\mathbf{V}\in\mathrm{St}(2,2)}\langle \mathbf{V}-\mathbf{I}_{2},\mathbf{T}_{\texttt{{B}}\texttt{{B}}}\rangle.$ (62) $$

We consider two cases for $\mathbf{V}\in\mathrm{St}(2,2)$.

Case a). When $\mathbf{V}$ is a rotation matrix with $\mathbf{V}=\mathbf{V}^{\mathrm{rot}}_{\theta}=\begin{bmatrix}\cos(\theta)&\sin
(\theta)\\
-\sin(\theta)&\cos(\theta)\\
\end{bmatrix}$ for some suitable $\theta$, we have:

$$ $\displaystyle\min_{\mathbf{V}\in\mathrm{St}(k,k)}\langle\mathbf{V}-\mathbf{I}_ {2},\mathbf{T}_{\texttt{{B}}\texttt{{B}}}\rangle$ (63) $\displaystyle=$ $\displaystyle\min_{\theta}\,\langle\begin{bmatrix}\cos(\theta)-1&\sin(\theta) \\ -\sin(\theta)&\cos(\theta)-1\\ \end{bmatrix},\begin{bmatrix}\mathbf{T}_{ii}&\mathbf{T}_{ij}\\ \mathbf{T}_{ji}&\mathbf{T}_{jj}\\ \end{bmatrix}\rangle.$ $\displaystyle=$ $\displaystyle\min_{\theta}\,\cos(\theta)(\mathbf{T}_{ii}+\mathbf{T}_{jj})+\sin (\theta)(\mathbf{T}_{ij}-\mathbf{T}_{ji})-(\mathbf{T}_{ii}+\mathbf{T}_{jj})$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle-\sqrt{(\mathbf{T}_{ii}+\mathbf{T}_{jj})^{2}+(\mathbf{T}_{ij}- \mathbf{T}_{ji})^{2}}-(\mathbf{T}_{ii}+\mathbf{T}_{jj})$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle-\sqrt{c_{1}^{2}+c_{2}^{2}}-c_{1}.$ $$

where step ① uses Lemma A.9 with $A=\mathbf{T}_{ii}+\mathbf{T}_{jj}$ and $B=\mathbf{T}_{ij}-\mathbf{T}_{ji}$; step ② uses the definition of $c_{1}\triangleq\mathbf{T}_{ii}+\mathbf{T}_{jj}$ and $c_{2}\triangleq\mathbf{T}_{ij}-\mathbf{T}_{ji}$.

Case b). When $\mathbf{V}$ is a reflection matrix with $\mathbf{V}=\mathbf{V}^{\mathrm{ref}}_{\theta}=\begin{bmatrix}-\cos(\theta)&
\sin(\theta)\\
\sin(\theta)&\cos(\theta)\\
\end{bmatrix}$ for some suitable $\theta$, we have:

$$ $\displaystyle\min_{\mathbf{V}\in\mathrm{St}(k,k)}\langle\mathbf{V}-\mathbf{I}_ {k},\mathbf{T}_{\texttt{{B}}\texttt{{B}}}\rangle$ (64) $\displaystyle=$ $\displaystyle\min_{\theta}\,\langle\begin{bmatrix}-\cos(\theta)-1&\sin(\theta) \\ \sin(\theta)&\cos(\theta)-1\\ \end{bmatrix},\begin{bmatrix}\mathbf{T}_{ii}&\mathbf{T}_{ij}\\ \mathbf{T}_{ji}&\mathbf{T}_{jj}\\ \end{bmatrix}\rangle.$ $\displaystyle=$ $\displaystyle\min_{\theta}\,\cos(\theta)(\mathbf{T}_{jj}-\mathbf{T}_{ii})+\sin (\theta)(\mathbf{T}_{ij}+\mathbf{T}_{ji})-(\mathbf{T}_{ii}+\mathbf{T}_{jj})$ $\displaystyle\overset{\text{\char 172}}{=}$ $\displaystyle-\sqrt{(\mathbf{T}_{jj}-\mathbf{T}_{ii})^{2}+(\mathbf{T}_{ij}+ \mathbf{T}_{ji})^{2}}-(\mathbf{T}_{ii}+\mathbf{T}_{jj})$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle-\sqrt{c_{3}^{2}+c_{4}^{2}}-c_{1}.$ $$

where step ① uses Lemma A.9 with $A=\mathbf{T}_{jj}-\mathbf{T}_{ii}$ and $B=\mathbf{T}_{ij}+\mathbf{T}_{ji}$; step ② uses the definition of $c_{3}\triangleq\mathbf{T}_{jj}-\mathbf{T}_{ii}$, $c_{4}\triangleq\mathbf{T}_{ij}+\mathbf{T}_{ji}$, and $c_{1}\triangleq\mathbf{T}_{ii}+\mathbf{T}_{jj}$.

In view of Equations (62), (63), and (64), we have:

$$ $\displaystyle\mathbf{S}_{ij}=\min(-\sqrt{c_{1}^{2}+c_{2}^{2}}-c_{1},-\sqrt{c_{ 3}^{2}+c_{4}^{2}}-c_{1}).$ $$

(b) We note that $h(\mathbf{X})=0$ for all $\mathbf{X}$ based on our assumption. If $\mathbf{X}^{t}$ is not a critical point, then the matrix $\mathbf{G}^{t}[\mathbf{X}^{t}]^{\mathsf{T}}\in\mathbb{R}^{n\times n}$ is not symmetric, and the matrix $\mathbf{T}\triangleq\mathbf{G}^{t}[\mathbf{X}^{t}]^{\mathsf{T}}-L\mathbf{X}^{t
}[\mathbf{X}^{t}]^{\mathsf{T}}-\alpha\mathbf{I}_{n}$ is also not symmetric. There exists $\texttt{{B}}=[i,j]$ with $i\neq j$ such that $\mathbf{T}_{ij}\neq\mathbf{T}_{ji}$, and $c_{2}\triangleq\mathbf{T}_{ij}-\mathbf{T}_{ji}\neq 0$. Consequently, $\mathbf{S}_{ij}=\min(w_{1},w_{2})$ becomes strictly negative, as $w_{1}=-c_{1}-\sqrt{c_{1}^{2}+c_{2}^{2}}<0$. Since the pair $[\bar{i},\bar{j}]\in\arg\min_{i,j}\mathbf{S}(i,j)$ is chosen, we conclude that $\mathbf{S}(\bar{i},\bar{j})<0$.

We now prove that a strict decrease is guaranteed with $F(\mathbf{X}^{t+1})<F(\mathbf{X}^{t})$ for OBCD if $\mathbf{X}^{t}$ is not a critical point. We define $\mathcal{X}^{t}_{\texttt{{B}}}(\mathbf{V})\triangleq\mathbf{X}^{t}+\mathrm{U}_
{\texttt{{B}}}(\mathbf{V}-\mathbf{I}_{k})\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}
}\mathbf{X}^{t}$. Since $\bar{\mathbf{V}}^{t}$ is the global optimal solution of the problem $\bar{\mathbf{V}}^{t}\in\arg\min_{\mathbf{V}\in\mathrm{St}(k,k)}\mathcal{K}(
\mathbf{V};\mathbf{X}^{t},\texttt{{B}})$, we have:

$$ $\displaystyle\tfrac{1}{2}\|\bar{\mathbf{V}}^{t}-\mathbf{I}_{k}\|_{\mathbf{Q}+ \alpha\mathbf{I}}^{2}+\langle\bar{\mathbf{V}}^{t}-\mathbf{I}_{k},[\nabla f( \mathbf{X}^{t})(\mathbf{X}^{t})^{\mathsf{T}}]_{\texttt{{B}}\texttt{{B}}}\rangle$ (65) $\displaystyle\leq$ $\displaystyle\tfrac{1}{2}\|\mathbf{V}-\mathbf{I}_{k}\|_{\mathbf{Q}+\alpha \mathbf{I}}^{2}+\langle\mathbf{V}-\mathbf{I}_{k},[\nabla f(\mathbf{X}^{t})( \mathbf{X}^{t})^{\mathsf{T}}]_{\texttt{{B}}\texttt{{B}}}\rangle,~{}\forall \mathbf{V}\in\mathrm{St}(k,k).$ $$

We derive the following inequalities:

$$ $\displaystyle~{}F(\mathbf{X}^{t+1})-F(\mathbf{X}^{t})=f(\mathcal{X}_{\texttt{{ B}}}^{t}(\mathbf{V}))-f(\mathbf{X}^{t})$ (66) $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle~{}\langle\bar{\mathbf{V}}^{t}-\mathbf{I}_{k},[\nabla f(\mathbf{X }^{t})(\mathbf{X}^{t})^{\mathsf{T}}]_{\texttt{{B}}\texttt{{B}}}+\tfrac{1}{2}\| \bar{\mathbf{V}}^{t}-\mathbf{I}_{k}\|_{\mathbf{Q}+\alpha\mathbf{I}}^{2}$ $\displaystyle\overset{\text{\char 173}}{\leq}$ $\displaystyle~{}\tfrac{1}{2}\|\mathbf{V}-\mathbf{I}_{k}\|_{\mathbf{Q}+\alpha \mathbf{I}}^{2}+\langle\mathbf{V}-\mathbf{I}_{k},[\nabla f(\mathbf{X}^{t})( \mathbf{X}^{t})^{\mathsf{T}}]_{\texttt{{B}}\texttt{{B}}}\rangle,~{}\forall \mathbf{V}\in\mathrm{St}(k,k).$ $\displaystyle\overset{\text{\char 174}}{\leq}$ $\displaystyle~{}\min_{\mathbf{V}\in\mathrm{St}(k,k)}\underbrace{\left(\tfrac{1 }{2}\|\mathcal{X}_{\texttt{{B}}}^{t}(\mathbf{V})-\mathbf{X}^{t}\|_{\mathbf{H}} ^{2}+\tfrac{\alpha}{2}\|\mathbf{V}-\mathbf{I}_{k}\|_{\mathsf{F}}^{2}+\langle \mathbf{V}-\mathbf{I}_{k},[\nabla f(\mathbf{X}^{t})(\mathbf{X}^{t})^{\mathsf{T }}]_{\texttt{{B}}\texttt{{B}}}\rangle\right)}_{\triangleq\Xi(\mathbf{V})},$ $$

where step ① uses Inequality (2); step ② uses Inequality (65); step ③ uses the fact that $\tfrac{1}{2}\|\mathbf{V}-\mathbf{I}_{k}\|_{\mathbf{Q}}^{2}=\tfrac{1}{2}\|
\mathcal{X}_{\texttt{{B}}}^{t}(\mathbf{V})-\mathbf{X}^{t}\|_{\mathbf{H}}^{2}$.

We now prove that the right-hand side of (66) is consistently negative with the following inequalities:

$$ $\displaystyle~{}\min_{\mathbf{V}\in\mathrm{St}(k,k)}\Xi(\mathbf{V})$ $\displaystyle\overset{\text{\char 172}}{\leq}$ $\displaystyle~{}\min_{\mathbf{V}\in\mathrm{St}(k,k)}\tfrac{L_{f}}{2}\|\mathcal {X}_{\texttt{{B}}}^{t}(\mathbf{V})-\mathbf{X}^{t}\|_{\mathsf{F}}^{2}+\langle \mathbf{V}-\mathbf{I}_{k},[\nabla f(\mathbf{X}^{t})(\mathbf{X}^{t})^{\mathsf{T }}]_{\texttt{{B}}\texttt{{B}}}-\alpha\mathbf{I}_{k}\rangle$ $\displaystyle\overset{\text{\char 173}}{=}$ $\displaystyle~{}\min_{\mathbf{V}\in\mathrm{St}(k,k)}L_{f}\langle\mathbf{I}_{k} -\mathbf{V},\mathrm{U}_{\texttt{{B}}}^{\mathsf{T}}\mathbf{X}^{t}[\mathbf{X}^{t }]^{\mathsf{T}}\mathrm{U}_{\texttt{{B}}}\rangle+\langle\mathbf{V}-\mathbf{I}_{ k},[\nabla f(\mathbf{X}^{t})(\mathbf{X}^{t})^{\mathsf{T}}]_{\texttt{{B}} \texttt{{B}}}-\alpha\mathbf{I}_{k}\rangle$ $\displaystyle\overset{\text{\char 174}}{=}$ $\displaystyle~{}\min_{\mathbf{V}\in\mathrm{St}(k,k)}\langle\mathbf{V}-\mathbf{ I}_{k},\mathbf{T}_{\texttt{{B}}\texttt{{B}}}\rangle$ $\displaystyle\overset{\text{\char 175}}{<}$ $\displaystyle~{}0,$ $$

where step ① uses $\tfrac{1}{2}\|\mathcal{X}_{\texttt{{B}}}^{t}(\mathbf{V})-\mathbf{X}^{t}\|_{
\mathbf{H}}^{2}\leq\tfrac{L_{f}}{2}\|\mathcal{X}^{t}_{\texttt{{B}}}(\mathbf{V}
)-\mathbf{X}^{t}\|_{\mathsf{F}}^{2}$ as $\|\mathbf{H}\|\leq L_{f}$, and the identity $\tfrac{\alpha}{2}\|\mathbf{V}-\mathbf{I}_{k}\|_{\mathsf{F}}^{2}=-\langle
\mathbf{V}-\mathbf{I}_{k},\alpha\mathbf{I}_{k}\rangle$, which is due to Claim (c) of Lemma 2.3; step ② uses Claim (b) of Lemma 2.3; step ③ uses the definition of $\mathbf{T}\triangleq(\mathbf{G}^{t}-L_{f}\mathbf{X}^{t})(\mathbf{X}^{t})^{
\mathsf{T}}-\alpha\mathbf{I}_{n}$ in Algorithm 2; step ④ uses Claim (a) of this theorem.

∎

###### Proof.

## Appendix J Additional Experiments

In this section, we present the experimental results of the proposed OBCD algorithm on the three tasks, namely $\ell_{0}$ norm-based SPCA, $\ell_{1}$ norm-based SPCA, and Nonnegative PCA using different working set selection strategies.

### J.1 Applications to ℓ 0 subscript ℓ 0 \ell_{0} roman_ℓ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT norm-based SPCA, Nonnegative PCA, and ℓ 1 subscript ℓ 1 \ell_{1} roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT norm-based SPCA

Since we have introduced $\ell_{0}$ norm-based SPCA in Section 6, we now present nonnegative PCA and $\ell_{1}$ norm-based SPCA.

$\blacktriangleright$  Nonnegative PCA Nonnegative PCA is an extension of PCA that imposes nonnegativity constraints on the principal vector Zass & Shashua (); Qian et al. (). This constraint leads to a nonnegative representation of loading vectors and it helps to capture data locality in feature selection. Nonnegative PCA can formulated as:

$$ $\displaystyle\min_{\mathbf{X}\in\mathrm{St}(n,r)}~{}-\tfrac{1}{2}\langle \mathbf{X},\mathbf{C}\mathbf{X}\rangle,\,s.t.\,\mathbf{X}\geq\mathbf{0},$ $$

where $\mathbf{C}\in\mathbb{R}^{n\times n}$ is the covariance matrix of the data.

$\blacktriangleright$  $L_{1}$ Norm-based SPCA. As the $L_{1}$ norm provides the tightest convex relaxation for the $L_{0}$-norm over the unit ball in the sense of $L_{\infty}$-norm, some researchers replace the non-convex and discontinuous $L_{0}$ norm function as in (15) with a convex but non-smooth function Chen et al. (); Vu et al. (); Lu & Zhang (). This leads to the following optimization problem of $L_{1}$ norm-based SPCA:

$$ $\displaystyle\min_{\mathbf{X}\in\mathrm{St}(n,r)}-\tfrac{1}{2}\langle\mathbf{X },\mathbf{C}\mathbf{X}\rangle+\lambda\|\mathbf{X}\|_{1},$ $$

where $\mathbf{C}\in\mathbb{R}^{n\times n}$ is the covariance matrix of the data, and $\lambda>0$.

### J.2 Experiment Setting

We compare the objective values $(F(\mathbf{X})-F_{\text{min}})$ for different methods after running $t$ seconds with $t$ varying from $20$ to $60$, where the constant $F_{\text{min}}$ denotes the smallest objective of all the methods.

Initializations. We use the same initializations for all methods. (i) For the $\ell_{0}$ and $\ell_{1}$ norm-based SPCA tasks, since the optimal solutions are expected to be sparse, we simply set $\mathbf{X}^{0}\in\mathrm{St}(n,r)$ to an identity matrix with $\mathbf{X}^{0}_{ij}=1$ if $i=j$ and otherwise 0. (ii) For the nonnegative PCA task, we use a random nonnegative orthogonal matrix as $\mathbf{X}^{0}$, which can be generated using the following strategy. We first randomly and uniformly partition the index vector $[1,2,...,n]$ into $r$ nonempty groups $\{\mathcal{G}_{i}\}_{i=1}^{r}$ with $\mathcal{G}_{i}$ being the index vector for the $i$-th group, then we set $\mathbf{X}^{0}(\mathcal{G}_{i},i)=\tfrac{1}{|\mathcal{G}_{i}|}$ for all $i\in[r]$, where $|\mathcal{G}_{i}|$ is the number of elements for the $i$-th group.

Variants of OBCD. We consider three variants of OBCD using different working set selection strategies: (i) OBCD-R that uses a simple random strategy; (ii) OBCD-CV that uses a greedy strategy based on maximum stationarity violation pair, and (iii) OBCD-OR that uses a greedy strategy based on objective reduction violation pair. We only consider $|\texttt{{B}}|=k=2$. In order to solve the subproblem for the $\ell_{0}$ norm-based SPCA, $\ell_{1}$ norm-based SPCA, and nonnegative PCA tasks, we use a breakpoint searching method as presented in Section 5 and Section B.

### J.3 Experiment Results on Three Tasks

$\blacktriangleright$  $\ell_{0}$ Norm-based Sparse PCA

We compare OBCD against two state-of-the-art methods: (i) Linearized Alternating Direction Method of Multiplier (LADMM) Lai & Osher () and (ii) Smoothing Penalty Method (SPM) Lai & Osher (); Chen (). We also initialize OBCD-R with the result of LADMM (or SPM) and ran it for 10 seconds to evaluate its effectiveness in improving the solution, leading to LADMM+OBCD-R (or SPM+OBCD-R). To compute the subgradient $\mathbf{G}^{t}\in\partial F(\mathbf{X}^{t})$ at $\mathbf{X}^{t}$ for Algorithm 2, we choose $\mathbf{G}^{t}=-\mathbf{C}\mathbf{X}^{t}+\mathbf{0}$ as $\mathbf{0}$ is the subgradient of the function $\lambda\|\mathbf{X}\|_{0}$.

Figure 3 shows the convergence curve of the compared methods with $\lambda=100$. Table 2, 3, and 4 show the objective values $(F(\mathbf{X})-F_{\text{min}})$ for different methods with varying $\lambda\in\{1,300,1000\}$. Several conclusions can be drawn. (i) Due to the use of greedy strategy, OBCD-CV and OBCD-OR often lead to faster convergence than OBCD-R for this task. (ii) OBCD-R often greatly improves upon LADMM and SPM; this is because our methods find stronger stationary points than LADMM and SPM. (iii) The proposed methods generally deliver the best performance.

$\blacktriangleright$  Nonnegative PCA

We compare OBCD against two state-of-the-art methods: (i) Linearized Alternating Direction Method of Multiplier (LADMM) He & Yuan (); Lai & Osher () and (ii) Smoothing Penalty Method (SPM). We also initialize OBCD-R with the result of LADMM (or SPM) and ran it for 10 seconds to evaluate its effectiveness in improving the solution, leading to LADMM+OBCD-R (or SPM+OBCD-R). To compute the subgradient $\mathbf{G}^{t}\in\partial F(\mathbf{X}^{t})$ at $\mathbf{X}^{t}$ for Algorithm 2, we choose $\mathbf{G}^{t}=-\mathbf{C}\mathbf{X}^{t}+\mathbf{0}$ as $\mathbf{0}$ is the subgradient of $\mathcal{I}_{\geq 0}(\mathbf{X}^{t})$.

Table 5 shows the comparisons of objective values and the violation of the constraints $(F(\mathbf{X})-F_{\text{min}},\|\min(\mathbf{0},\mathbf{X})\|_{\mathsf{F}}+\|
\mathbf{X}^{\mathsf{T}}\mathbf{X}-\mathbf{I}_{r}\|_{\mathsf{F}})$ for different methods. Two conclusions can be drawn. (i) OBCD-CV and OBCD-OR are not as effective as OBCD-R in this task. This is because the matrix $\mathbf{0}$ may not be a suitable choice for the subgradient for the nonsmooth function $\mathcal{I}_{\geq 0}(\mathbf{X})$. (ii) Feasibility of our methods is achieved with $\|\min(\mathbf{0},\mathbf{X})\|_{\mathsf{F}}+\|\mathbf{X}^{\mathsf{T}}\mathbf{
X}-\mathbf{I}_{r}\|_{\mathsf{F}}\leq 10^{-12}$. This is because OBCD is a feasible method. (iii) The proposed methods generally give the best performance. (iv) OBCD-R often greatly improve upon LADMM and SPM, as our methods find stronger stationary points than LADMM and SPM.

$\blacktriangleright$  $\ell_{1}$ Norm-based Sparse PCA

We compare OBCD against the following state-of-the-art methods: (i) Linearized Alternating Direction Method of Multiplier (LADMM) He & Yuan (); (ii) ADMM Lai & Osher (); (iii) Riemannian Subgradient Method (SubGrad) Li et al. (); (iv) Manifold Proximal Gradient Method (ManPG) Chen et al. (). We also initialize OBCD-R with the result of LADMM (or ManPG) and ran it for 10 seconds to evaluate its effectiveness in improving the solution, leading to LADMM+OBCD-R (or ManPG+OBCD-R). To compute the subgradient $\mathbf{G}^{t}\in\partial F(\mathbf{X}^{t})$ at $\mathbf{X}^{t}$ for Algorithm 2, we choose $\mathbf{G}^{t}=-\mathbf{C}\mathbf{X}^{t}+\lambda\mathrm{sign}(\mathbf{X}^{t})$ as $\mathrm{sign}(\mathbf{X})$ is the subgradient of $\|\mathbf{X}\|_{1}$.

Table 6, 7, and 8 show the comparisons of objective values $(F(\mathbf{X})-F_{\text{min}})$ for different methods with varying $\lambda\in\{1,100,1000\}$. Several conclusions can be drawn. (i) ManPG is generally faster than LADMM, ADMM and SubGrad. This is consistent with the reported results in Chen et al. (). (ii) OBCD-OR outperforms the other methods $\{$LADMM, ADMM, SubGrad, ManPG$\}$ by achieving lower objective values. (iii) OBCD-OR often greatly improve upon the LADMM and SPM.

Figure: (a) ‘w1a’
Refer to caption: x7.png

**Table 2: Comparisons of objective values $(F(\mathbf{X})-F_{\text{min}})$ of $L_{0}$ norm-based SPCA for all the compared methods with $\lambda=1$. The $1^{st}$, $2^{nd}$, and $3^{rd}$ best results are colored with red, green and blue, respectively. If the objective values of ‘LADMM+OBCD-R’ (or ‘SPM+OBCD-R’) are smaller than those of ‘LADMM’ (or ‘SPM’) by a margin of $0.1\times a$, where $a$ represents the objective values of ‘LADMM’ (or ‘ManPG’), they will be marked with ${\color[rgb]{0.9961,0,0}(+)}$.**
| data-m-n | $F_{\text{min}}$ | r | LADMM | SPM | OBCD-R | OBCD-SV | OBCD-OR | LADMM | SPM |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| + OBCD-R | +OBCD-R |  |  |  |  |  |  |  |  |
| $\lambda$ = 1.00, time limit=10 |  |  |  |  |  |  |  |  |  |
| w1a-2477-300 | -6.0e+03 | 20 | 8.70e+02 | 4.98e+03 | 3.67e+01 | 3.39e+02 | 0.00e+00 | 3.45e+02 (+) | 7.40e+01 (+) |
| TDT2-500-1000 | 1.6e+01 | 20 | 3.75e+00 | 3.75e+00 | 1.85e-02 | 2.06e-03 | 0.00e+00 | 1.85e-02 (+) | 1.85e-02 (+) |
| 20News-8000-1000 | 1.8e+01 | 20 | 1.66e+00 | 1.66e+00 | 2.19e-02 | 0.00e+00 | 0.00e+00 | 2.19e-02 (+) | 2.19e-02 (+) |
| sector-6412-1000 | -2.6e+01 | 20 | 4.27e+01 | 4.27e+01 | 1.65e+00 | 0.00e+00 | 2.60e-01 | 1.65e+00 (+) | 1.65e+00 (+) |
| E2006-2000-1000 | 1.9e+01 | 20 | 6.36e-01 | 6.36e-01 | 5.00e-04 | 0.00e+00 | 3.40e-03 | 5.00e-04 (+) | 5.00e-04 (+) |
| MNIST-60000-784 | -3.4e+05 | 20 | 2.40e+04 | 3.41e+05 | 4.09e+04 | 0.00e+00 | 5.63e+03 | 1.46e+04 (+) | 4.09e+04 (+) |
| Gisette-3000-1000 | -1.1e+06 | 20 | 2.39e+04 | 2.28e+04 | 4.49e+04 | 0.00e+00 | 3.18e+04 | 1.47e+04 (+) | 1.38e+04 (+) |
| CnnCaltech-3000-1000 | -2.4e+03 | 20 | 8.26e+01 | 1.16e+03 | 7.44e+01 | 1.35e-01 | 5.04e+02 | 0.00e+00 (+) | 1.07e+01 (+) |
| Cifar-1000-1000 | -1.4e+05 | 20 | 6.99e+02 | 6.56e+01 | 1.04e+04 | 2.23e+03 | 5.39e+03 | 6.56e+02 | 0.00e+00 (+) |
| randn-500-1000 | -1.7e+04 | 20 | 1.90e-03 | 1.18e+03 | 7.60e+03 | 7.01e+03 | 6.49e+03 | 0.00e+00 (+) | 1.17e+03 |
| w1a-2477-300 | -8.6e+03 | 50 | 2.92e+03 | 2.46e+03 | 0.00e+00 | 7.11e+02 | 2.60e+02 | 1.99e+02 (+) | 2.73e+02 (+) |
| TDT2-500-1000 | 4.6e+01 | 50 | 4.28e+00 | 4.28e+00 | 2.06e-02 | 3.43e-03 | 0.00e+00 | 9.60e-03 (+) | 2.06e-02 (+) |
| 20News-8000-1000 | 4.8e+01 | 50 | 8.56e-01 | 8.56e-01 | 5.04e-03 | 8.04e-04 | 0.00e+00 | 5.04e-03 (+) | 5.04e-03 (+) |
| sector-6412-1000 | -1.5e+01 | 50 | 5.99e+01 | 5.99e+01 | 1.18e-02 | 0.00e+00 | 4.60e-02 | 1.18e-02 (+) | 1.18e-02 (+) |
| E2006-2000-1000 | 4.9e+01 | 50 | 6.64e-01 | 6.64e-01 | 2.13e-04 | 0.00e+00 | 2.51e-04 | 2.13e-04 (+) | 2.13e-04 (+) |
| MNIST-60000-784 | -3.5e+05 | 50 | 3.09e+04 | 3.28e+04 | 5.88e+04 | 0.00e+00 | 8.81e+02 | 2.21e+04 (+) | 2.32e+04 (+) |
| Gisette-3000-1000 | -1.1e+06 | 50 | 2.85e+04 | 2.47e+04 | 7.78e+04 | 0.00e+00 | 2.56e+04 | 8.31e+03 (+) | 3.41e+03 (+) |
| CnnCaltech-3000-1000 | -2.9e+03 | 50 | 4.22e+02 | 1.57e+03 | 1.35e+02 | 0.00e+00 | 5.89e+02 | 1.49e+02 (+) | 1.26e+02 (+) |
| Cifar-1000-1000 | -1.4e+05 | 50 | 3.34e+03 | 4.67e+01 | 9.69e+03 | 2.88e+03 | 5.36e+03 | 3.34e+03 | 0.00e+00 (+) |
| randn-500-1000 | -3.8e+04 | 50 | 2.26e-03 | 3.33e+03 | 1.99e+04 | 1.74e+04 | 1.75e+04 | 0.00e+00 (+) | 3.33e+03 |
| w1a-2477-300 | -1.2e+04 | 100 | 3.52e+03 | 3.26e+03 | 0.00e+00 | 4.81e+02 | 4.81e+02 | 7.65e+02 (+) | 3.06e+02 (+) |
| TDT2-500-1000 | 9.5e+01 | 100 | 4.45e+00 | 4.45e+00 | 0.00e+00 | 1.10e-02 | 2.06e-02 | 0.00e+00 (+) | 0.00e+00 (+) |
| 20News-8000-1000 | 9.8e+01 | 100 | 9.25e-01 | 9.25e-01 | 0.00e+00 | 7.54e-04 | 8.98e-04 | 0.00e+00 (+) | 0.00e+00 (+) |
| sector-6412-1000 | 1.6e+01 | 100 | 7.42e+01 | 7.42e+01 | 0.00e+00 | 8.65e-01 | 8.88e-02 | 0.00e+00 (+) | 0.00e+00 (+) |
| E2006-2000-1000 | 9.9e+01 | 100 | 6.71e-01 | 6.71e-01 | 1.05e-05 | 0.00e+00 | 2.33e-04 | 1.05e-05 (+) | 1.05e-05 (+) |
| MNIST-60000-784 | -3.5e+05 | 100 | 1.93e+04 | 2.08e+04 | 4.40e+04 | 0.00e+00 | 5.29e+03 | 7.33e+03 (+) | 8.46e+03 (+) |
| Gisette-3000-1000 | -1.1e+06 | 100 | 4.86e+04 | 7.20e+04 | 8.01e+04 | 0.00e+00 | 1.29e+04 | 1.28e+04 (+) | 3.15e+04 (+) |
| CnnCaltech-3000-1000 | -3.5e+03 | 100 | 9.37e+02 | 1.98e+03 | 1.29e+01 | 1.57e+01 | 5.08e+02 | 5.26e+02 (+) | 0.00e+00 (+) |
| Cifar-1000-1000 | -1.4e+05 | 100 | 2.12e+04 | 1.96e+04 | 7.03e+03 | 0.00e+00 | 3.38e+03 | 2.12e+04 | 1.95e+04 |
| randn-500-1000 | -6.7e+04 | 100 | 2.03e-03 | 3.53e+04 | 3.42e+04 | 3.03e+04 | 2.98e+04 | 0.00e+00 (+) | 3.53e+04 |
| $\lambda$ = 1.00, time limit=20 |  |  |  |  |  |  |  |  |  |
| w1a-2477-300 | -6.0e+03 | 20 | 8.75e+02 | 4.99e+03 | 0.00e+00 | 4.22e+02 | 3.74e+01 | 3.26e+02 (+) | 6.99e+01 (+) |
| TDT2-500-1000 | 1.6e+01 | 20 | 3.75e+00 | 3.75e+00 | 1.03e-02 | 0.00e+00 | 3.43e-03 | 1.10e-02 (+) | 1.10e-02 (+) |
| 20News-8000-1000 | 1.8e+01 | 20 | 1.66e+00 | 1.66e+00 | 7.62e-05 | 0.00e+00 | 0.00e+00 | 1.08e-02 (+) | 1.08e-02 (+) |
| sector-6412-1000 | -2.5e+01 | 20 | 4.25e+01 | 4.25e+01 | 8.84e-01 | 0.00e+00 | 6.83e-02 | 9.95e-01 (+) | 9.95e-01 (+) |
| E2006-2000-1000 | 1.9e+01 | 20 | 6.38e-01 | 6.38e-01 | 0.00e+00 | 1.20e-03 | 4.92e-04 | 1.73e-03 (+) | 1.73e-03 (+) |
| MNIST-60000-784 | -3.4e+05 | 20 | 2.63e+04 | 3.43e+05 | 2.03e+04 | 0.00e+00 | 5.73e+03 | 1.47e+04 (+) | 3.43e+04 (+) |
| Gisette-3000-1000 | -1.1e+06 | 20 | 2.82e+04 | 2.71e+04 | 3.02e+04 | 0.00e+00 | 2.27e+04 | 1.82e+04 (+) | 1.71e+04 (+) |
| CnnCaltech-3000-1000 | -2.5e+03 | 20 | 1.69e+02 | 1.25e+03 | 5.65e+01 | 0.00e+00 | 4.84e+02 | 7.67e+01 (+) | 7.64e+01 (+) |
| Cifar-1000-1000 | -1.4e+05 | 20 | 7.09e+02 | 7.31e+01 | 9.22e+03 | 1.34e+03 | 5.01e+03 | 6.61e+02 | 0.00e+00 (+) |
| randn-500-1000 | -1.7e+04 | 20 | 2.02e-03 | 1.18e+03 | 7.29e+03 | 6.95e+03 | 6.42e+03 | 0.00e+00 (+) | 1.17e+03 |
| w1a-2477-300 | -8.6e+03 | 50 | 2.98e+03 | 2.52e+03 | 0.00e+00 | 5.80e+02 | 7.02e+01 | 2.93e+02 (+) | 3.06e+02 (+) |
| TDT2-500-1000 | 4.6e+01 | 50 | 4.29e+00 | 4.29e+00 | 0.00e+00 | 2.74e-03 | 4.80e-03 | 1.17e-02 (+) | 1.17e-02 (+) |
| 20News-8000-1000 | 4.8e+01 | 50 | 8.55e-01 | 8.55e-01 | 2.79e-04 | 0.00e+00 | 0.00e+00 | 1.76e-03 (+) | 1.76e-03 (+) |
| sector-6412-1000 | -1.5e+01 | 50 | 6.01e+01 | 6.01e+01 | 5.54e-02 | 0.00e+00 | 3.70e-02 | 6.44e-02 (+) | 6.44e-02 (+) |
| E2006-2000-1000 | 4.9e+01 | 50 | 6.64e-01 | 6.64e-01 | 5.06e-05 | 0.00e+00 | 0.00e+00 | 2.55e-04 (+) | 2.55e-04 (+) |
| MNIST-60000-784 | -3.6e+05 | 50 | 3.95e+04 | 4.14e+04 | 4.59e+04 | 0.00e+00 | 6.36e+03 | 3.06e+04 (+) | 3.16e+04 (+) |
| Gisette-3000-1000 | -1.1e+06 | 50 | 3.24e+04 | 2.86e+04 | 7.02e+04 | 0.00e+00 | 2.50e+04 | 1.23e+04 (+) | 7.33e+03 (+) |
| CnnCaltech-3000-1000 | -2.9e+03 | 50 | 3.67e+02 | 1.51e+03 | 0.00e+00 | 2.46e+01 | 4.73e+02 | 9.37e+01 (+) | 7.52e+01 (+) |
| Cifar-1000-1000 | -1.4e+05 | 50 | 3.35e+03 | 4.67e+01 | 8.98e+03 | 1.69e+03 | 5.10e+03 | 3.34e+03 | 0.00e+00 (+) |
| randn-500-1000 | -3.8e+04 | 50 | 2.25e-03 | 3.33e+03 | 1.98e+04 | 1.76e+04 | 1.74e+04 | 0.00e+00 (+) | 3.33e+03 |
| w1a-2477-300 | -1.2e+04 | 100 | 3.54e+03 | 3.28e+03 | 0.00e+00 | 5.14e+02 | 2.95e+02 | 6.96e+02 (+) | 3.23e+02 (+) |
| TDT2-500-1000 | 9.5e+01 | 100 | 4.45e+00 | 4.45e+00 | 0.00e+00 | 0.00e+00 | 2.06e-03 | 3.43e-03 (+) | 3.43e-03 (+) |
| 20News-8000-1000 | 9.8e+01 | 100 | 9.26e-01 | 9.26e-01 | 5.67e-04 | 0.00e+00 | 1.10e-03 | 1.40e-03 (+) | 1.40e-03 (+) |
| sector-6412-1000 | 1.6e+01 | 100 | 7.45e+01 | 7.45e+01 | 0.00e+00 | 1.28e-01 | 2.76e-01 | 2.50e-01 (+) | 2.50e-01 (+) |
| E2006-2000-1000 | 9.9e+01 | 100 | 6.72e-01 | 6.72e-01 | 3.58e-05 | 0.00e+00 | 1.00e-04 | 3.60e-04 (+) | 3.60e-04 (+) |
| MNIST-60000-784 | -3.6e+05 | 100 | 2.70e+04 | 2.85e+04 | 4.69e+04 | 0.00e+00 | 6.40e+03 | 1.51e+04 (+) | 1.61e+04 (+) |
| Gisette-3000-1000 | -1.1e+06 | 100 | 4.51e+04 | 4.27e+04 | 7.71e+04 | 0.00e+00 | 1.38e+04 | 8.70e+03 (+) | 3.50e+03 (+) |
| CnnCaltech-3000-1000 | -3.6e+03 | 100 | 6.82e+02 | 2.01e+03 | 0.00e+00 | 1.14e+01 | 4.44e+02 | 2.79e+02 (+) | 2.52e+01 (+) |
| Cifar-1000-1000 | -1.4e+05 | 100 | 8.07e+03 | 2.79e+03 | 7.05e+03 | 0.00e+00 | 3.69e+03 | 8.06e+03 | 2.73e+03 |
| randn-500-1000 | -6.7e+04 | 100 | 2.04e-03 | 7.83e+03 | 3.41e+04 | 3.01e+04 | 3.02e+04 | 0.00e+00 (+) | 7.83e+03 |
| $\lambda$ = 1.00, time limit=30 |  |  |  |  |  |  |  |  |  |
| w1a-2477-300 | -6.1e+03 | 20 | 8.78e+02 | 4.99e+03 | 0.00e+00 | 4.78e+02 | 1.39e+02 | 3.18e+02 (+) | 5.09e+01 (+) |
| TDT2-500-1000 | 1.6e+01 | 20 | 3.75e+00 | 3.75e+00 | 6.86e-03 | 0.00e+00 | 0.00e+00 | 1.10e-02 (+) | 1.10e-02 (+) |
| 20News-8000-1000 | 1.8e+01 | 20 | 1.66e+00 | 1.66e+00 | 0.00e+00 | 0.00e+00 | 0.00e+00 | 1.08e-02 (+) | 1.08e-02 (+) |
| sector-6412-1000 | -2.6e+01 | 20 | 4.27e+01 | 4.27e+01 | 0.00e+00 | 0.00e+00 | 1.42e-14 | 1.19e+00 (+) | 1.19e+00 (+) |
| E2006-2000-1000 | 1.9e+01 | 20 | 6.38e-01 | 6.38e-01 | 0.00e+00 | 3.36e-04 | 3.36e-04 | 1.73e-03 (+) | 1.73e-03 (+) |
| MNIST-60000-784 | -3.4e+05 | 20 | 2.63e+04 | 3.43e+05 | 1.29e+04 | 0.00e+00 | 2.74e+03 | 1.49e+04 (+) | 3.61e+04 (+) |
| Gisette-3000-1000 | -1.1e+06 | 20 | 2.35e+04 | 2.24e+04 | 2.06e+04 | 0.00e+00 | 1.41e+04 | 1.34e+04 (+) | 1.25e+04 (+) |
| CnnCaltech-3000-1000 | -2.5e+03 | 20 | 1.43e+02 | 1.22e+03 | 0.00e+00 | 1.53e+01 | 3.15e+02 | 5.35e+01 (+) | 4.84e+01 (+) |
| Cifar-1000-1000 | -1.4e+05 | 20 | 7.10e+02 | 7.32e+01 | 8.90e+03 | 1.71e+03 | 4.79e+03 | 6.61e+02 | 0.00e+00 (+) |
| randn-500-1000 | -1.7e+04 | 20 | 1.99e-03 | 1.18e+03 | 7.15e+03 | 6.91e+03 | 6.73e+03 | 0.00e+00 (+) | 1.17e+03 |
| w1a-2477-300 | -8.6e+03 | 50 | 2.99e+03 | 2.53e+03 | 0.00e+00 | 6.30e+02 | 3.24e+01 | 2.98e+02 (+) | 3.56e+02 (+) |
| TDT2-500-1000 | 4.6e+01 | 50 | 4.29e+00 | 4.29e+00 | 0.00e+00 | 3.43e-03 | 3.43e-03 | 1.23e-02 (+) | 1.44e-02 (+) |
| 20News-8000-1000 | 4.8e+01 | 50 | 8.55e-01 | 8.55e-01 | 6.77e-05 | 0.00e+00 | 0.00e+00 | 1.76e-03 (+) | 1.76e-03 (+) |
| sector-6412-1000 | -1.5e+01 | 50 | 6.00e+01 | 6.00e+01 | 3.79e-03 | 0.00e+00 | 1.25e-01 | 4.80e-02 (+) | 4.80e-02 (+) |
| E2006-2000-1000 | 4.9e+01 | 50 | 6.64e-01 | 6.64e-01 | 0.00e+00 | 1.53e-05 | 0.00e+00 | 8.88e-05 (+) | 8.88e-05 (+) |
| MNIST-60000-784 | -3.6e+05 | 50 | 4.09e+04 | 4.28e+04 | 3.76e+04 | 0.00e+00 | 5.22e+03 | 3.19e+04 (+) | 3.31e+04 (+) |
| Gisette-3000-1000 | -1.1e+06 | 50 | 3.45e+04 | 3.07e+04 | 6.56e+04 | 0.00e+00 | 2.24e+04 | 1.43e+04 (+) | 9.52e+03 (+) |
| CnnCaltech-3000-1000 | -3.0e+03 | 50 | 4.87e+02 | 1.63e+03 | 6.71e+01 | 0.00e+00 | 5.21e+02 | 2.17e+02 (+) | 1.93e+02 (+) |
| Cifar-1000-1000 | -1.4e+05 | 50 | 3.35e+03 | 4.65e+01 | 8.48e+03 | 1.56e+03 | 4.86e+03 | 3.35e+03 | 0.00e+00 (+) |
| randn-500-1000 | -3.8e+04 | 50 | 2.26e-03 | 3.33e+03 | 1.97e+04 | 1.77e+04 | 1.73e+04 | 0.00e+00 (+) | 3.33e+03 |
| w1a-2477-300 | -1.2e+04 | 100 | 3.55e+03 | 3.29e+03 | 0.00e+00 | 5.28e+02 | 4.09e+02 | 8.37e+02 (+) | 3.17e+02 (+) |
| TDT2-500-1000 | 9.5e+01 | 100 | 4.45e+00 | 4.45e+00 | 0.00e+00 | 0.00e+00 | 0.00e+00 | 4.80e-03 (+) | 4.80e-03 (+) |
| 20News-8000-1000 | 9.8e+01 | 100 | 9.26e-01 | 9.26e-01 | 9.31e-05 | 0.00e+00 | 1.69e-05 | 1.25e-03 (+) | 1.25e-03 (+) |
| sector-6412-1000 | 1.6e+01 | 100 | 7.45e+01 | 7.45e+01 | 9.80e-03 | 1.50e-03 | 0.00e+00 | 2.72e-01 (+) | 2.72e-01 (+) |
| E2006-2000-1000 | 9.9e+01 | 100 | 6.72e-01 | 6.72e-01 | 3.81e-05 | 0.00e+00 | 3.56e-04 | 4.16e-04 (+) | 4.16e-04 (+) |
| MNIST-60000-784 | -3.6e+05 | 100 | 2.64e+04 | 2.80e+04 | 4.42e+04 | 0.00e+00 | 2.36e+03 | 1.46e+04 (+) | 1.56e+04 (+) |
| Gisette-3000-1000 | -1.1e+06 | 100 | 4.89e+04 | 4.64e+04 | 7.75e+04 | 0.00e+00 | 1.10e+04 | 1.26e+04 (+) | 7.26e+03 (+) |
| CnnCaltech-3000-1000 | -3.6e+03 | 100 | 7.40e+02 | 2.07e+03 | 2.64e+01 | 0.00e+00 | 5.13e+02 | 3.36e+02 (+) | 8.59e+01 (+) |
| Cifar-1000-1000 | -1.4e+05 | 100 | 8.91e+03 | 3.56e+03 | 7.52e+03 | 0.00e+00 | 4.47e+03 | 8.90e+03 | 3.49e+03 |
| randn-500-1000 | -6.7e+04 | 100 | 2.03e-03 | 7.83e+03 | 3.41e+04 | 2.97e+04 | 3.02e+04 | 0.00e+00 (+) | 7.83e+03 |

**Table 3: Comparisons of objective values $(F(\mathbf{X})-F_{\text{min}})$ of $L_{0}$ norm-based SPCA for all the compared methods with $\lambda=300$. The $1^{st}$, $2^{nd}$, and $3^{rd}$ best results are colored with red, green and blue, respectively. If the objective values of ‘LADMM+OBCD-R’ (or ‘SPM+OBCD-R’) are smaller than those of ‘LADMM’ (or ‘SPM’) by a margin of $0.1\times a$, where $a$ represents the objective values of ‘LADMM’ (or ‘ManPG’), they will be marked with ${\color[rgb]{0.9961,0,0}(+)}$.**
| data-m-n | $F_{\text{min}}$ | r | LADMM | SPM | OBCD-R | OBCD-SV | OBCD-OR | LADMM | SPM |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| + OBCD-R | +OBCD-R |  |  |  |  |  |  |  |  |
| $\lambda$ = 300.00, time limit=10 |  |  |  |  |  |  |  |  |  |
| w1a-2477-300 | 1.3e+03 | 20 | 2.35e+03 | 3.90e+03 | 9.09e-13 | 9.09e-13 | 9.09e-13 | 9.09e-13 (+) | 0.00e+00 (+) |
| TDT2-500-1000 | 6.0e+03 | 20 | 1.00e+00 | 6.82e-01 | 1.10e-02 | 0.00e+00 | 0.00e+00 | 2.19e-02 (+) | 2.19e-02 (+) |
| 20News-8000-1000 | 6.0e+03 | 20 | 3.71e-01 | 3.41e-01 | 1.08e-02 | 0.00e+00 | 0.00e+00 | 4.76e-03 (+) | 4.76e-03 (+) |
| sector-6412-1000 | 6.0e+03 | 20 | 6.19e+02 | 3.05e+02 | 9.27e-01 | 3.06e-01 | 0.00e+00 | 3.01e+02 (+) | 3.01e+02 |
| E2006-2000-1000 | 6.0e+03 | 20 | 1.20e+03 | 1.44e-01 | 1.03e-03 | 0.00e+00 | 4.95e-04 | 6.00e+02 (+) | 1.03e-03 (+) |
| MNIST-60000-784 | -1.8e+05 | 20 | 4.01e+04 | 1.82e+05 | 0.00e+00 | 1.73e+04 | 3.70e+04 | 1.46e+04 (+) | 0.00e+00 (+) |
| Gisette-3000-1000 | -7.7e+05 | 20 | 2.29e+04 | 4.28e+05 | 1.03e+05 | 0.00e+00 | 1.30e+04 | 1.08e+04 (+) | 3.13e+05 (+) |
| CnnCaltech-3000-1000 | 5.4e+03 | 20 | 2.59e+03 | 1.42e+04 | 4.63e+00 | 2.62e-01 | 0.00e+00 | 1.15e+03 (+) | 9.73e+03 (+) |
| Cifar-1000-1000 | 2.0e+03 | 20 | 4.15e+04 | 1.10e+03 | 1.81e+00 | 0.00e+00 | 0.00e+00 | 3.91e+04 | 1.81e+00 (+) |
| randn-500-1000 | 1.4e+02 | 20 | 1.62e+04 | 1.51e+04 | 1.69e+01 | 0.00e+00 | 0.00e+00 | 5.09e+03 (+) | 3.93e+03 (+) |
| w1a-2477-300 | 6.8e+03 | 50 | 5.26e+03 | 4.14e+03 | 3.64e-12 | 3.64e-12 | 3.64e-12 | 0.00e+00 (+) | 3.64e-12 (+) |
| TDT2-500-1000 | 1.5e+04 | 50 | 9.01e+02 | 1.17e+00 | 7.54e-03 | 0.00e+00 | 6.86e-04 | 3.00e+02 (+) | 1.23e-02 (+) |
| 20News-8000-1000 | 1.5e+04 | 50 | 2.76e+04 | 3.85e-01 | 1.79e-03 | 0.00e+00 | 5.50e-04 | 1.11e+04 (+) | 4.40e-04 (+) |
| sector-6412-1000 | 1.5e+04 | 50 | 2.12e+03 | 1.51e+03 | 7.02e-02 | 0.00e+00 | 1.33e-01 | 2.10e+03 | 1.20e+03 (+) |
| E2006-2000-1000 | 1.5e+04 | 50 | 2.40e+03 | 1.42e-01 | 0.00e+00 | 1.02e-04 | 2.43e-04 | 2.10e+03 (+) | 1.11e-05 (+) |
| MNIST-60000-784 | -2.1e+05 | 50 | 6.06e+04 | 1.07e+05 | 0.00e+00 | 1.19e+04 | 3.98e+04 | 1.18e+04 (+) | 1.58e+04 (+) |
| Gisette-3000-1000 | -7.9e+05 | 50 | 2.18e+05 | 7.30e+05 | 9.65e+04 | 0.00e+00 | 8.62e+03 | 1.89e+05 (+) | 1.03e+05 (+) |
| CnnCaltech-3000-1000 | 1.4e+04 | 50 | 1.14e+03 | 2.47e+03 | 4.09e-01 | 0.00e+00 | 6.09e-01 | 6.01e+02 (+) | 2.09e+03 (+) |
| Cifar-1000-1000 | 5.1e+03 | 50 | 3.63e+04 | 2.38e+03 | 9.66e+00 | 0.00e+00 | 3.64e-12 | 3.34e+04 | 9.66e+00 (+) |
| randn-500-1000 | 7.6e+02 | 50 | 2.93e+04 | 2.02e+03 | 4.90e+00 | 0.00e+00 | 1.73e+00 | 1.31e+04 (+) | 5.58e+02 (+) |
| w1a-2477-300 | 1.8e+04 | 100 | 5.87e+03 | 6.01e+03 | 1.17e-05 | 3.00e+00 | 1.17e-05 | 1.17e-05 (+) | 0.00e+00 (+) |
| TDT2-500-1000 | 3.0e+04 | 100 | 9.90e+03 | 1.20e+03 | 0.00e+00 | 6.86e-04 | 4.12e-03 | 8.70e+03 (+) | 6.00e+02 (+) |
| 20News-8000-1000 | 3.0e+04 | 100 | 1.20e+04 | 9.01e+02 | 0.00e+00 | 1.02e-03 | 1.80e-03 | 7.20e+03 (+) | 9.00e+02 |
| sector-6412-1000 | 3.0e+04 | 100 | 1.23e+04 | 6.02e+03 | 0.00e+00 | 3.82e-01 | 3.60e-01 | 1.05e+04 (+) | 3.90e+03 (+) |
| E2006-2000-1000 | 3.0e+04 | 100 | 1.17e+04 | 1.31e-01 | 1.51e-04 | 2.44e-04 | 2.79e-04 | 9.90e+03 (+) | 0.00e+00 (+) |
| MNIST-60000-784 | -2.6e+05 | 100 | 1.01e+05 | 2.25e+05 | 0.00e+00 | 1.79e+04 | 4.50e+04 | 1.92e+04 (+) | 3.04e+04 (+) |
| Gisette-3000-1000 | -8.3e+05 | 100 | 6.40e+04 | 7.13e+05 | 9.31e+04 | 0.00e+00 | 1.43e+04 | 1.58e+03 (+) | 9.92e+04 (+) |
| CnnCaltech-3000-1000 | 2.8e+04 | 100 | 1.54e+03 | 4.88e+03 | 0.00e+00 | 9.84e-01 | 5.11e-02 | 6.00e+02 (+) | 3.51e+03 (+) |
| Cifar-1000-1000 | 1.1e+04 | 100 | 3.43e+04 | 3.82e+03 | 1.33e+01 | 0.00e+00 | 2.24e+00 | 2.84e+04 (+) | 1.33e+01 (+) |
| randn-500-1000 | 2.1e+03 | 100 | 3.63e+04 | 2.90e+03 | 1.12e+00 | 0.00e+00 | 1.09e+00 | 2.05e+04 (+) | 2.67e+02 (+) |
| $\lambda$ = 300.00, time limit=20 |  |  |  |  |  |  |  |  |  |
| w1a-2477-300 | 1.3e+03 | 20 | 2.35e+03 | 3.90e+03 | 0.00e+00 | 0.00e+00 | 0.00e+00 | 0.00e+00 (+) | 0.00e+00 (+) |
| TDT2-500-1000 | 6.0e+03 | 20 | 1.00e+00 | 6.82e-01 | 1.03e-02 | 0.00e+00 | 0.00e+00 | 2.19e-02 (+) | 2.19e-02 (+) |
| 20News-8000-1000 | 6.0e+03 | 20 | 3.71e-01 | 3.41e-01 | 7.62e-05 | 0.00e+00 | 0.00e+00 | 4.76e-03 (+) | 4.76e-03 (+) |
| sector-6412-1000 | 6.0e+03 | 20 | 6.20e+02 | 3.06e+02 | 1.08e+00 | 0.00e+00 | 0.00e+00 | 3.01e+02 (+) | 3.01e+02 |
| E2006-2000-1000 | 6.0e+03 | 20 | 1.20e+03 | 1.45e-01 | 0.00e+00 | 1.61e-04 | 3.36e-04 | 6.00e+02 (+) | 1.73e-03 (+) |
| MNIST-60000-784 | -1.8e+05 | 20 | 4.54e+04 | 1.87e+05 | 0.00e+00 | 1.28e+04 | 4.01e+04 | 2.09e+04 (+) | 5.36e+03 (+) |
| Gisette-3000-1000 | -7.7e+05 | 20 | 2.75e+04 | 4.32e+05 | 1.04e+05 | 0.00e+00 | 1.18e+04 | 1.50e+04 (+) | 3.16e+05 (+) |
| CnnCaltech-3000-1000 | 5.4e+03 | 20 | 2.59e+03 | 1.42e+04 | 3.00e+00 | 2.62e-01 | 0.00e+00 | 1.15e+03 (+) | 9.73e+03 (+) |
| Cifar-1000-1000 | 2.0e+03 | 20 | 4.15e+04 | 1.10e+03 | 9.33e-01 | 0.00e+00 | 9.09e-13 | 3.91e+04 | 1.81e+00 (+) |
| randn-500-1000 | 1.4e+02 | 20 | 1.62e+04 | 1.51e+04 | 6.40e+00 | 0.00e+00 | 0.00e+00 | 5.09e+03 (+) | 3.93e+03 (+) |
| w1a-2477-300 | 6.8e+03 | 50 | 5.26e+03 | 4.14e+03 | 1.82e-12 | 1.82e-12 | 1.82e-12 | 0.00e+00 (+) | 1.82e-12 (+) |
| TDT2-500-1000 | 1.5e+04 | 50 | 9.01e+02 | 1.17e+00 | 0.00e+00 | 2.74e-03 | 4.80e-03 | 3.00e+02 (+) | 1.65e-02 (+) |
| 20News-8000-1000 | 1.5e+04 | 50 | 2.76e+04 | 3.86e-01 | 6.77e-04 | 3.98e-04 | 0.00e+00 | 1.11e+04 (+) | 8.13e-04 (+) |
| sector-6412-1000 | 1.5e+04 | 50 | 2.12e+03 | 1.51e+03 | 6.12e-02 | 1.66e-01 | 0.00e+00 | 2.10e+03 | 1.20e+03 (+) |
| E2006-2000-1000 | 1.5e+04 | 50 | 2.40e+03 | 1.42e-01 | 1.61e-04 | 0.00e+00 | 6.30e-05 | 2.10e+03 (+) | 1.95e-04 (+) |
| MNIST-60000-784 | -2.2e+05 | 50 | 6.61e+04 | 1.12e+05 | 0.00e+00 | 8.39e+03 | 4.46e+04 | 1.61e+04 (+) | 2.12e+04 (+) |
| Gisette-3000-1000 | -7.9e+05 | 50 | 2.09e+05 | 7.34e+05 | 9.61e+04 | 0.00e+00 | 8.37e+03 | 1.80e+05 (+) | 1.07e+05 (+) |
| CnnCaltech-3000-1000 | 1.4e+04 | 50 | 1.14e+03 | 2.47e+03 | 3.68e-01 | 0.00e+00 | 3.68e-01 | 6.01e+02 (+) | 2.09e+03 (+) |
| Cifar-1000-1000 | 5.1e+03 | 50 | 3.63e+04 | 2.38e+03 | 3.78e+00 | 1.82e-12 | 0.00e+00 | 3.34e+04 | 9.66e+00 (+) |
| randn-500-1000 | 7.6e+02 | 50 | 2.93e+04 | 2.02e+03 | 2.34e+00 | 0.00e+00 | 1.31e+00 | 1.28e+04 (+) | 5.59e+02 (+) |
| w1a-2477-300 | 1.8e+04 | 100 | 5.87e+03 | 6.01e+03 | 3.48e-05 | 3.48e-05 | 3.48e-05 | 3.48e-05 (+) | 0.00e+00 (+) |
| TDT2-500-1000 | 3.0e+04 | 100 | 9.90e+03 | 1.20e+03 | 0.00e+00 | 6.86e-04 | 2.06e-03 | 8.70e+03 (+) | 6.00e+02 (+) |
| 20News-8000-1000 | 3.0e+04 | 100 | 1.20e+04 | 9.01e+02 | 5.17e-04 | 0.00e+00 | 9.74e-04 | 7.20e+03 (+) | 9.00e+02 |
| sector-6412-1000 | 3.0e+04 | 100 | 1.23e+04 | 6.02e+03 | 2.12e-02 | 0.00e+00 | 3.71e-01 | 1.05e+04 (+) | 3.90e+03 (+) |
| E2006-2000-1000 | 3.0e+04 | 100 | 1.17e+04 | 1.31e-01 | 6.55e-05 | 0.00e+00 | 2.58e-05 | 9.90e+03 (+) | 2.39e-04 (+) |
| MNIST-60000-784 | -2.6e+05 | 100 | 1.08e+05 | 2.33e+05 | 0.00e+00 | 1.92e+04 | 4.84e+04 | 2.68e+04 (+) | 3.88e+04 (+) |
| Gisette-3000-1000 | -8.4e+05 | 100 | 7.16e+04 | 7.20e+05 | 9.50e+04 | 0.00e+00 | 9.73e+03 | 9.05e+03 (+) | 1.05e+05 (+) |
| CnnCaltech-3000-1000 | 2.8e+04 | 100 | 1.54e+03 | 4.88e+03 | 3.33e-01 | 2.94e-01 | 0.00e+00 | 6.01e+02 (+) | 3.52e+03 (+) |
| Cifar-1000-1000 | 1.1e+04 | 100 | 3.43e+04 | 3.82e+03 | 4.20e+00 | 0.00e+00 | 4.58e+00 | 2.85e+04 (+) | 1.77e+01 (+) |
| randn-500-1000 | 2.1e+03 | 100 | 3.63e+04 | 2.91e+03 | 9.40e-01 | 0.00e+00 | 2.20e+00 | 2.05e+04 (+) | 2.72e+02 (+) |
| $\lambda$ = 300.00, time limit=30 |  |  |  |  |  |  |  |  |  |
| w1a-2477-300 | 1.3e+03 | 20 | 2.35e+03 | 3.90e+03 | 0.00e+00 | 0.00e+00 | 0.00e+00 | 0.00e+00 (+) | 0.00e+00 (+) |
| TDT2-500-1000 | 6.0e+03 | 20 | 1.00e+00 | 6.82e-01 | 6.86e-03 | 0.00e+00 | 3.43e-03 | 2.19e-02 (+) | 2.19e-02 (+) |
| 20News-8000-1000 | 6.0e+03 | 20 | 3.71e-01 | 3.41e-01 | 0.00e+00 | 0.00e+00 | 0.00e+00 | 4.76e-03 (+) | 4.76e-03 (+) |
| sector-6412-1000 | 6.0e+03 | 20 | 6.20e+02 | 3.06e+02 | 0.00e+00 | 0.00e+00 | 0.00e+00 | 3.01e+02 (+) | 3.01e+02 |
| E2006-2000-1000 | 6.0e+03 | 20 | 1.20e+03 | 1.45e-01 | 0.00e+00 | 3.36e-04 | 0.00e+00 | 6.00e+02 (+) | 1.73e-03 (+) |
| MNIST-60000-784 | -1.8e+05 | 20 | 4.69e+04 | 1.88e+05 | 0.00e+00 | 2.80e+04 | 4.05e+04 | 2.14e+04 (+) | 6.85e+03 (+) |
| Gisette-3000-1000 | -7.7e+05 | 20 | 2.71e+04 | 4.32e+05 | 1.01e+05 | 0.00e+00 | 9.38e+03 | 1.53e+04 (+) | 3.14e+05 (+) |
| CnnCaltech-3000-1000 | 5.4e+03 | 20 | 2.59e+03 | 1.42e+04 | 1.91e+00 | 0.00e+00 | 2.62e-01 | 1.15e+03 (+) | 9.73e+03 (+) |
| Cifar-1000-1000 | 2.0e+03 | 20 | 4.15e+04 | 1.10e+03 | 1.38e-01 | 0.00e+00 | 0.00e+00 | 3.91e+04 | 1.81e+00 (+) |
| randn-500-1000 | 1.4e+02 | 20 | 1.62e+04 | 1.51e+04 | 9.61e-01 | 0.00e+00 | 1.82e-12 | 5.09e+03 (+) | 3.93e+03 (+) |
| w1a-2477-300 | 6.8e+03 | 50 | 5.26e+03 | 4.14e+03 | 1.82e-12 | 1.82e-12 | 1.82e-12 | 0.00e+00 (+) | 1.82e-12 (+) |
| TDT2-500-1000 | 1.5e+04 | 50 | 9.01e+02 | 1.17e+00 | 0.00e+00 | 0.00e+00 | 6.86e-03 | 3.00e+02 (+) | 1.71e-02 (+) |
| 20News-8000-1000 | 1.5e+04 | 50 | 2.76e+04 | 3.86e-01 | 4.66e-04 | 0.00e+00 | 0.00e+00 | 1.11e+04 (+) | 8.13e-04 (+) |
| sector-6412-1000 | 1.5e+04 | 50 | 2.12e+03 | 1.51e+03 | 2.60e-02 | 0.00e+00 | 1.41e-01 | 2.10e+03 | 1.20e+03 (+) |
| E2006-2000-1000 | 1.5e+04 | 50 | 2.40e+03 | 1.42e-01 | 6.30e-05 | 0.00e+00 | 6.30e-05 | 2.10e+03 (+) | 1.63e-04 (+) |
| MNIST-60000-784 | -2.2e+05 | 50 | 7.01e+04 | 1.16e+05 | 0.00e+00 | 2.20e+04 | 4.82e+04 | 2.15e+04 (+) | 2.62e+04 (+) |
| Gisette-3000-1000 | -8.0e+05 | 50 | 2.17e+05 | 7.41e+05 | 9.95e+04 | 0.00e+00 | 1.21e+04 | 1.89e+05 (+) | 1.13e+05 (+) |
| CnnCaltech-3000-1000 | 1.4e+04 | 50 | 1.14e+03 | 2.47e+03 | 3.54e-01 | 0.00e+00 | 0.00e+00 | 6.01e+02 (+) | 2.09e+03 (+) |
| Cifar-1000-1000 | 5.1e+03 | 50 | 3.63e+04 | 2.38e+03 | 3.65e+00 | 3.64e-12 | 0.00e+00 | 3.34e+04 | 9.66e+00 (+) |
| randn-500-1000 | 7.6e+02 | 50 | 2.93e+04 | 2.02e+03 | 7.15e-01 | 9.00e-01 | 0.00e+00 | 1.28e+04 (+) | 5.59e+02 (+) |
| w1a-2477-300 | 1.8e+04 | 100 | 5.87e+03 | 6.01e+03 | 4.46e-06 | 4.46e-06 | 4.46e-06 | 4.46e-06 (+) | 0.00e+00 (+) |
| TDT2-500-1000 | 3.0e+04 | 100 | 9.90e+03 | 1.20e+03 | 0.00e+00 | 2.06e-03 | 2.74e-03 | 8.70e+03 (+) | 6.00e+02 (+) |
| 20News-8000-1000 | 3.0e+04 | 100 | 1.20e+04 | 9.01e+02 | 2.62e-04 | 0.00e+00 | 9.48e-04 | 7.20e+03 (+) | 9.00e+02 |
| sector-6412-1000 | 3.0e+04 | 100 | 1.23e+04 | 6.02e+03 | 0.00e+00 | 6.26e-02 | 1.73e-01 | 1.05e+04 (+) | 4.20e+03 (+) |
| E2006-2000-1000 | 3.0e+04 | 100 | 1.17e+04 | 1.31e-01 | 0.00e+00 | 1.10e-04 | 2.39e-04 | 9.90e+03 (+) | 2.42e-04 (+) |
| MNIST-60000-784 | -2.6e+05 | 100 | 1.09e+05 | 2.34e+05 | 0.00e+00 | 2.08e+04 | 4.91e+04 | 2.94e+04 (+) | 4.34e+04 (+) |
| Gisette-3000-1000 | -8.4e+05 | 100 | 7.56e+04 | 7.24e+05 | 9.50e+04 | 0.00e+00 | 1.18e+04 | 1.52e+04 (+) | 1.12e+05 (+) |
| CnnCaltech-3000-1000 | 2.8e+04 | 100 | 1.54e+03 | 4.88e+03 | 2.36e-01 | 1.54e-02 | 0.00e+00 | 6.02e+02 (+) | 3.52e+03 (+) |
| Cifar-1000-1000 | 1.1e+04 | 100 | 3.43e+04 | 3.82e+03 | 3.34e+00 | 0.00e+00 | 6.39e-01 | 2.88e+04 (+) | 1.97e+01 (+) |
| randn-500-1000 | 2.1e+03 | 100 | 3.63e+04 | 2.91e+03 | 8.94e-01 | 0.00e+00 | 3.11e+00 | 2.17e+04 (+) | 2.80e+02 (+) |

**Table 4: Comparisons of objective values $(F(\mathbf{X})-F_{\text{min}})$ of $L_{0}$ norm-based SPCA for all the compared methods with $\lambda=1000$. The $1^{st}$, $2^{nd}$, and $3^{rd}$ best results are colored with red, green and blue, respectively. If the objective values of ‘LADMM+OBCD-R’ (or ‘SPM+OBCD-R’) are smaller than those of ‘LADMM’ (or ‘SPM’) by a margin of $0.1\times a$, where $a$ represents the objective values of ‘LADMM’ (or ‘ManPG’), they will be marked with ${\color[rgb]{0.9961,0,0}(+)}$.**
| data-m-n | $F_{\text{min}}$ | r | LADMM | SPM | OBCD-R | OBCD-SV | OBCD-OR | LADMM | SPM |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| + OBCD-R | +OBCD-R |  |  |  |  |  |  |  |  |
| $\lambda$ = 1000.00, time limit=10 |  |  |  |  |  |  |  |  |  |
| w1a-2477-300 | 1.5e+04 | 20 | 2.64e+03 | 3.90e+03 | 0.00e+00 | 0.00e+00 | 0.00e+00 | 0.00e+00 (+) | 0.00e+00 (+) |
| TDT2-500-1000 | 2.0e+04 | 20 | 1.00e+03 | 6.82e-01 | 1.85e-02 | 0.00e+00 | 1.71e-02 | 1.00e+03 | 2.95e-02 (+) |
| 20News-8000-1000 | 2.0e+04 | 20 | 3.00e+03 | 3.00e+03 | 2.19e-02 | 0.00e+00 | 0.00e+00 | 3.00e+03 | 1.00e+03 (+) |
| sector-6412-1000 | 2.0e+04 | 20 | 3.01e+03 | 3.00e+03 | 1.65e+00 | 0.00e+00 | 2.60e-01 | 3.00e+03 | 3.00e+03 |
| E2006-2000-1000 | 2.0e+04 | 20 | 1.00e+03 | 1.15e-01 | 9.69e-04 | 0.00e+00 | 2.75e-04 | 1.00e+03 | 9.69e-04 (+) |
| MNIST-60000-784 | -6.3e+04 | 20 | 9.86e+04 | 8.25e+04 | 0.00e+00 | 7.62e+03 | 1.73e+04 | 3.81e+04 (+) | 2.88e+02 (+) |
| Gisette-3000-1000 | -2.2e+05 | 20 | 6.15e+05 | 2.20e+05 | 1.19e+03 | 0.00e+00 | 1.50e+03 | 5.26e+05 (+) | 1.10e+04 (+) |
| CnnCaltech-3000-1000 | 1.9e+04 | 20 | 1.41e+04 | 3.09e+04 | 4.63e+00 | 0.00e+00 | 0.00e+00 | 7.94e+03 (+) | 1.99e+04 (+) |
| Cifar-1000-1000 | 1.6e+04 | 20 | 1.81e+04 | 1.10e+03 | 3.03e+00 | 0.00e+00 | 0.00e+00 | 1.40e+04 (+) | 3.03e+00 (+) |
| randn-500-1000 | 1.4e+04 | 20 | 2.34e+04 | 5.81e+04 | 1.57e+01 | 0.00e+00 | 1.24e+00 | 1.27e+04 (+) | 1.86e+04 (+) |
| w1a-2477-300 | 4.2e+04 | 50 | 5.09e+03 | 4.79e+03 | 0.00e+00 | 0.00e+00 | 5.00e-01 | 0.00e+00 (+) | 0.00e+00 (+) |
| TDT2-500-1000 | 5.0e+04 | 50 | 9.00e+03 | 4.00e+03 | 4.12e-03 | 1.51e-02 | 0.00e+00 | 7.00e+03 (+) | 4.00e+03 |
| 20News-8000-1000 | 5.0e+04 | 50 | 2.60e+04 | 2.00e+03 | 2.16e-03 | 3.05e-04 | 0.00e+00 | 8.00e+03 (+) | 2.00e+03 |
| sector-6412-1000 | 5.0e+04 | 50 | 7.02e+03 | 3.00e+03 | 0.00e+00 | 7.31e-02 | 7.31e-02 | 5.00e+03 (+) | 3.00e+03 |
| E2006-2000-1000 | 5.0e+04 | 50 | 8.00e+03 | 1.00e-01 | 2.92e-04 | 2.11e-04 | 0.00e+00 | 8.00e+03 | 2.79e-04 (+) |
| MNIST-60000-784 | -8.6e+04 | 50 | 1.01e+05 | 1.92e+05 | 0.00e+00 | 8.98e+03 | 2.03e+04 | 1.81e+02 (+) | 6.62e+04 (+) |
| Gisette-3000-1000 | -2.3e+05 | 50 | 7.24e+05 | 2.08e+05 | 1.16e+04 | 0.00e+00 | 2.81e+03 | 8.56e+05 | 1.01e+04 (+) |
| CnnCaltech-3000-1000 | 4.9e+04 | 50 | 6.46e+03 | 3.23e+04 | 4.09e-01 | 0.00e+00 | 0.00e+00 | 3.97e+03 (+) | 2.39e+04 (+) |
| Cifar-1000-1000 | 4.0e+04 | 50 | 2.64e+04 | 2.38e+03 | 8.20e+00 | 0.00e+00 | 0.00e+00 | 2.40e+04 | 8.20e+00 (+) |
| randn-500-1000 | 3.6e+04 | 50 | 4.59e+04 | 9.36e+03 | 5.69e+00 | 2.43e+00 | 0.00e+00 | 3.45e+04 (+) | 5.89e+03 (+) |
| w1a-2477-300 | 8.8e+04 | 100 | 5.95e+03 | 9.30e+03 | 0.00e+00 | 0.00e+00 | 5.00e-01 | 0.00e+00 (+) | 2.50e+00 (+) |
| TDT2-500-1000 | 1.0e+05 | 100 | 5.00e+03 | 4.00e+03 | 0.00e+00 | 1.37e-03 | 1.92e-02 | 5.00e+03 | 4.00e+03 |
| 20News-8000-1000 | 1.0e+05 | 100 | 1.80e+04 | 1.00e+03 | 0.00e+00 | 1.24e-03 | 2.54e-03 | 1.70e+04 | 1.19e-04 (+) |
| sector-6412-1000 | 1.0e+05 | 100 | 2.60e+04 | 2.40e+04 | 0.00e+00 | 6.34e-01 | 4.86e-01 | 1.90e+04 (+) | 2.40e+04 |
| E2006-2000-1000 | 1.0e+05 | 100 | 5.20e+04 | 1.07e-01 | 2.16e-04 | 0.00e+00 | 4.25e-04 | 3.80e+04 (+) | 1.46e-04 (+) |
| MNIST-60000-784 | -1.2e+05 | 100 | 1.65e+05 | 2.02e+05 | 0.00e+00 | 2.03e+04 | 3.03e+04 | 1.14e+04 (+) | 7.02e+03 (+) |
| Gisette-3000-1000 | -2.4e+05 | 100 | 2.96e+04 | 2.01e+05 | 1.25e+04 | 0.00e+00 | 6.70e+02 | 5.66e+03 (+) | 1.38e+04 (+) |
| CnnCaltech-3000-1000 | 9.8e+04 | 100 | 6.84e+03 | 1.67e+04 | 0.00e+00 | 1.40e+00 | 1.87e+00 | 5.99e+03 (+) | 1.59e+04 |
| Cifar-1000-1000 | 8.1e+04 | 100 | 7.16e+04 | 3.82e+03 | 1.51e+01 | 0.00e+00 | 5.56e+00 | 3.39e+04 (+) | 1.51e+01 (+) |
| randn-500-1000 | 7.2e+04 | 100 | 1.05e+05 | 8.62e+03 | 3.29e+00 | 0.00e+00 | 4.35e+00 | 9.80e+04 | 3.91e+03 (+) |
| $\lambda$ = 1000.00, time limit=20 |  |  |  |  |  |  |  |  |  |
| w1a-2477-300 | 1.5e+04 | 20 | 2.64e+03 | 3.90e+03 | 0.00e+00 | 0.00e+00 | 0.00e+00 | 0.00e+00 (+) | 0.00e+00 (+) |
| TDT2-500-1000 | 2.0e+04 | 20 | 1.00e+03 | 6.82e-01 | 1.03e-02 | 1.51e-02 | 0.00e+00 | 1.00e+03 | 2.19e-02 (+) |
| 20News-8000-1000 | 2.0e+04 | 20 | 3.00e+03 | 3.00e+03 | 7.62e-05 | 0.00e+00 | 0.00e+00 | 3.00e+03 | 1.00e+03 (+) |
| sector-6412-1000 | 2.0e+04 | 20 | 3.01e+03 | 3.00e+03 | 1.08e+00 | 0.00e+00 | 0.00e+00 | 3.00e+03 | 3.00e+03 |
| E2006-2000-1000 | 2.0e+04 | 20 | 1.00e+03 | 1.16e-01 | 0.00e+00 | 6.66e-04 | 4.34e-04 | 1.00e+03 | 1.73e-03 (+) |
| MNIST-60000-784 | -6.6e+04 | 20 | 1.02e+05 | 8.60e+04 | 0.00e+00 | 6.08e+03 | 1.80e+04 | 4.05e+04 (+) | 2.39e+03 (+) |
| Gisette-3000-1000 | -2.2e+05 | 20 | 6.09e+05 | 2.21e+05 | 0.00e+00 | 2.07e+03 | 1.52e+03 | 5.20e+05 (+) | 1.13e+04 (+) |
| CnnCaltech-3000-1000 | 1.9e+04 | 20 | 1.41e+04 | 3.09e+04 | 3.00e+00 | 0.00e+00 | 0.00e+00 | 7.94e+03 (+) | 1.89e+04 (+) |
| Cifar-1000-1000 | 1.6e+04 | 20 | 1.81e+04 | 1.10e+03 | 9.33e-01 | 0.00e+00 | 0.00e+00 | 1.00e+04 (+) | 1.81e+00 (+) |
| randn-500-1000 | 1.4e+04 | 20 | 2.34e+04 | 5.81e+04 | 6.92e+00 | 0.00e+00 | 1.82e-12 | 1.27e+04 (+) | 1.47e+04 (+) |
| w1a-2477-300 | 4.2e+04 | 50 | 5.09e+03 | 4.79e+03 | 0.00e+00 | 0.00e+00 | 0.00e+00 | 0.00e+00 (+) | 0.00e+00 (+) |
| TDT2-500-1000 | 5.0e+04 | 50 | 9.00e+03 | 4.00e+03 | 0.00e+00 | 0.00e+00 | 4.80e-03 | 7.00e+03 (+) | 4.00e+03 |
| 20News-8000-1000 | 5.0e+04 | 50 | 2.60e+04 | 2.00e+03 | 5.76e-04 | 3.73e-04 | 0.00e+00 | 8.00e+03 (+) | 2.00e+03 |
| sector-6412-1000 | 5.0e+04 | 50 | 7.02e+03 | 3.00e+03 | 0.00e+00 | 8.58e-02 | 2.87e-02 | 5.00e+03 (+) | 3.00e+03 |
| E2006-2000-1000 | 5.0e+04 | 50 | 8.00e+03 | 1.00e-01 | 1.25e-04 | 0.00e+00 | 2.81e-04 | 8.00e+03 | 3.17e-04 (+) |
| MNIST-60000-784 | -9.2e+04 | 50 | 1.06e+05 | 1.17e+05 | 0.00e+00 | 1.03e+04 | 1.68e+04 | 6.41e+03 (+) | 1.87e+04 (+) |
| Gisette-3000-1000 | -2.3e+05 | 50 | 5.82e+05 | 2.09e+05 | 9.98e+03 | 0.00e+00 | 2.49e+03 | 4.98e+05 (+) | 1.22e+04 (+) |
| CnnCaltech-3000-1000 | 4.9e+04 | 50 | 6.46e+03 | 1.93e+04 | 3.68e-01 | 0.00e+00 | 0.00e+00 | 3.97e+03 (+) | 1.19e+04 (+) |
| Cifar-1000-1000 | 4.0e+04 | 50 | 2.64e+04 | 2.38e+03 | 3.78e+00 | 0.00e+00 | 7.28e-12 | 2.40e+04 | 9.66e+00 (+) |
| randn-500-1000 | 3.6e+04 | 50 | 4.59e+04 | 9.36e+03 | 2.49e+00 | 5.36e-01 | 0.00e+00 | 3.45e+04 (+) | 5.89e+03 (+) |
| w1a-2477-300 | 8.8e+04 | 100 | 5.95e+03 | 9.30e+03 | 0.00e+00 | 0.00e+00 | 0.00e+00 | 0.00e+00 (+) | 2.50e+00 (+) |
| TDT2-500-1000 | 1.0e+05 | 100 | 5.00e+03 | 4.00e+03 | 6.86e-04 | 2.06e-03 | 0.00e+00 | 5.00e+03 | 4.00e+03 |
| 20News-8000-1000 | 1.0e+05 | 100 | 1.80e+04 | 1.00e+03 | 0.00e+00 | 8.04e-04 | 6.10e-04 | 1.70e+04 | 9.48e-04 (+) |
| sector-6412-1000 | 1.0e+05 | 100 | 2.60e+04 | 2.40e+04 | 0.00e+00 | 3.08e-01 | 4.34e-01 | 1.90e+04 (+) | 2.40e+04 |
| E2006-2000-1000 | 1.0e+05 | 100 | 5.20e+04 | 1.07e-01 | 0.00e+00 | 9.01e-05 | 4.69e-05 | 3.80e+04 (+) | 2.54e-04 (+) |
| MNIST-60000-784 | -1.2e+05 | 100 | 1.69e+05 | 2.05e+05 | 0.00e+00 | 2.34e+04 | 2.16e+04 | 1.53e+04 (+) | 1.09e+04 (+) |
| Gisette-3000-1000 | -2.5e+05 | 100 | 3.28e+04 | 2.04e+05 | 1.26e+04 | 0.00e+00 | 1.35e+03 | 8.83e+03 (+) | 1.66e+04 (+) |
| CnnCaltech-3000-1000 | 9.8e+04 | 100 | 6.84e+03 | 1.67e+04 | 0.00e+00 | 7.36e-02 | 1.20e-01 | 5.99e+03 (+) | 1.59e+04 |
| Cifar-1000-1000 | 8.1e+04 | 100 | 7.16e+04 | 3.82e+03 | 4.20e+00 | 0.00e+00 | 4.40e+00 | 3.29e+04 (+) | 1.77e+01 (+) |
| randn-500-1000 | 7.2e+04 | 100 | 1.05e+05 | 8.63e+03 | 0.00e+00 | 8.01e-01 | 1.94e+00 | 9.80e+04 | 3.91e+03 (+) |
| $\lambda$ = 1000.00, time limit=30 |  |  |  |  |  |  |  |  |  |
| w1a-2477-300 | 1.5e+04 | 20 | 2.64e+03 | 3.90e+03 | 0.00e+00 | 0.00e+00 | 0.00e+00 | 0.00e+00 (+) | 0.00e+00 (+) |
| TDT2-500-1000 | 2.0e+04 | 20 | 1.00e+03 | 6.82e-01 | 1.03e-02 | 0.00e+00 | 0.00e+00 | 1.00e+03 | 2.95e-02 (+) |
| 20News-8000-1000 | 2.0e+04 | 20 | 3.00e+03 | 3.00e+03 | 7.62e-05 | 0.00e+00 | 0.00e+00 | 3.00e+03 | 1.00e+03 (+) |
| sector-6412-1000 | 2.0e+04 | 20 | 3.01e+03 | 3.00e+03 | 8.73e-01 | 0.00e+00 | 0.00e+00 | 3.00e+03 | 3.00e+03 |
| E2006-2000-1000 | 2.0e+04 | 20 | 1.00e+03 | 1.16e-01 | 9.75e-05 | 4.33e-04 | 0.00e+00 | 1.00e+03 | 1.83e-03 (+) |
| MNIST-60000-784 | -6.7e+04 | 20 | 1.03e+05 | 8.70e+04 | 0.00e+00 | 1.17e+04 | 2.34e+04 | 4.14e+04 (+) | 4.76e+03 (+) |
| Gisette-3000-1000 | -2.2e+05 | 20 | 6.09e+05 | 2.22e+05 | 3.02e+02 | 0.00e+00 | 1.81e+03 | 5.21e+05 (+) | 1.25e+04 (+) |
| CnnCaltech-3000-1000 | 1.9e+04 | 20 | 1.41e+04 | 3.09e+04 | 2.22e+00 | 0.00e+00 | 0.00e+00 | 7.94e+03 (+) | 1.99e+04 (+) |
| Cifar-1000-1000 | 1.6e+04 | 20 | 1.81e+04 | 1.10e+03 | 1.38e-01 | 1.82e-12 | 0.00e+00 | 1.40e+04 (+) | 3.03e+00 (+) |
| randn-500-1000 | 1.4e+04 | 20 | 2.34e+04 | 5.81e+04 | 2.17e+00 | 0.00e+00 | 1.82e-12 | 1.37e+04 (+) | 1.86e+04 (+) |
| w1a-2477-300 | 4.2e+04 | 50 | 5.09e+03 | 4.79e+03 | 0.00e+00 | 0.00e+00 | 0.00e+00 | 0.00e+00 (+) | 0.00e+00 (+) |
| TDT2-500-1000 | 5.0e+04 | 50 | 9.00e+03 | 4.00e+03 | 0.00e+00 | 7.54e-03 | 6.86e-04 | 7.00e+03 (+) | 4.00e+03 |
| 20News-8000-1000 | 5.0e+04 | 50 | 2.60e+04 | 2.00e+03 | 5.33e-04 | 6.27e-04 | 0.00e+00 | 9.00e+03 (+) | 2.00e+03 |
| sector-6412-1000 | 5.0e+04 | 50 | 7.02e+03 | 3.00e+03 | 1.83e-02 | 1.00e-01 | 0.00e+00 | 5.00e+03 (+) | 3.00e+03 |
| E2006-2000-1000 | 5.0e+04 | 50 | 8.00e+03 | 1.00e-01 | 9.50e-05 | 2.09e-05 | 0.00e+00 | 8.00e+03 | 3.53e-04 (+) |
| MNIST-60000-784 | -9.2e+04 | 50 | 1.07e+05 | 1.17e+05 | 0.00e+00 | 7.67e+03 | 1.77e+04 | 7.37e+03 (+) | 1.98e+04 (+) |
| Gisette-3000-1000 | -2.3e+05 | 50 | 6.32e+05 | 2.10e+05 | 1.02e+04 | 0.00e+00 | 2.81e+03 | 4.04e+05 (+) | 1.27e+04 (+) |
| CnnCaltech-3000-1000 | 4.9e+04 | 50 | 6.46e+03 | 1.93e+04 | 3.68e-01 | 0.00e+00 | 6.14e-01 | 3.97e+03 (+) | 1.39e+04 (+) |
| Cifar-1000-1000 | 4.0e+04 | 50 | 2.64e+04 | 2.38e+03 | 3.65e+00 | 0.00e+00 | 7.28e-12 | 2.40e+04 | 1.50e+01 (+) |
| randn-500-1000 | 3.6e+04 | 50 | 4.59e+04 | 9.36e+03 | 8.90e-01 | 0.00e+00 | 7.13e-01 | 3.55e+04 (+) | 7.86e+03 (+) |
| w1a-2477-300 | 8.8e+04 | 100 | 5.95e+03 | 9.30e+03 | 0.00e+00 | 0.00e+00 | 0.00e+00 | 0.00e+00 (+) | 9.73e+02 (+) |
| TDT2-500-1000 | 1.0e+05 | 100 | 5.00e+03 | 4.00e+03 | 0.00e+00 | 1.37e-03 | 6.86e-04 | 5.00e+03 | 4.00e+03 |
| 20News-8000-1000 | 1.0e+05 | 100 | 1.80e+04 | 1.00e+03 | 2.46e-04 | 0.00e+00 | 2.54e-05 | 1.70e+04 | 1.00e+03 |
| sector-6412-1000 | 1.0e+05 | 100 | 2.60e+04 | 2.40e+04 | 1.89e-02 | 0.00e+00 | 2.12e-01 | 2.00e+04 (+) | 2.40e+04 |
| E2006-2000-1000 | 1.0e+05 | 100 | 5.20e+04 | 1.07e-01 | 2.37e-05 | 0.00e+00 | 1.58e-04 | 3.90e+04 (+) | 3.69e-04 (+) |
| MNIST-60000-784 | -1.2e+05 | 100 | 1.69e+05 | 2.06e+05 | 0.00e+00 | 1.51e+04 | 2.11e+04 | 1.73e+04 (+) | 1.39e+04 (+) |
| Gisette-3000-1000 | -2.5e+05 | 100 | 3.35e+04 | 2.05e+05 | 1.30e+04 | 0.00e+00 | 2.21e+03 | 9.74e+03 (+) | 1.89e+04 (+) |
| CnnCaltech-3000-1000 | 9.8e+04 | 100 | 6.84e+03 | 1.67e+04 | 1.49e-01 | 0.00e+00 | 1.14e+00 | 5.99e+03 (+) | 1.59e+04 |
| Cifar-1000-1000 | 8.1e+04 | 100 | 7.16e+04 | 3.82e+03 | 3.34e+00 | 0.00e+00 | 0.00e+00 | 3.49e+04 (+) | 1.97e+01 (+) |
| randn-500-1000 | 7.2e+04 | 100 | 1.05e+05 | 8.63e+03 | 1.45e+00 | 0.00e+00 | 4.08e+00 | 1.00e+05 | 3.92e+03 (+) |

**Table 5: Comparisons of objective values and the violation of the constraints $(F(\mathbf{X})-F_{\text{min}},\|\min(\mathbf{0},\mathbf{X})\|_{\mathsf{F}}+\| \mathbf{X}^{\mathsf{T}}\mathbf{X}-\mathbf{I}_{r}\|_{\mathsf{F}})$ for nonnegative PCA for all the compared methods. The $1^{st}$, $2^{nd}$, and $3^{rd}$ best results are colored with red, green and blue, respectively. If the objective values of ‘LADMM+OBCD-R’ (or ‘SPM+OBCD-R’) are smaller than those of ‘LADMM’ (or ‘SPM’) by a margin of $0.1\times a$, where $a$ represents the objective values of ‘LADMM’ (or ‘ManPG’), they will be marked with ${\color[rgb]{0.9961,0,0}(+)}$.**
| data-m-n | $F_{\text{min}}$ | r | LADMM | SPM | OBCD-R | OBCD-SV | OBCD-OR | LADMM | SPM |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| + OBCD-R | +OBCD-R |  |  |  |  |  |  |  |  |
| time limit=10 |  |  |  |  |  |  |  |  |  |
| w1a-2477-300 | -5.2e+03 | 10 | 5.14e+02, 2e-13 | 1.29e+03, 1e-05 | 2.77e+02, 6e-14 | 5.00e+02, 2e-14 | 1.77e+02, 2e-14 | 0.00e+00, 2e-13(+) | 1.93e+02, 1e-05(+) |
| TDT2-500-1000 | -3.5e+00 | 10 | 9.22e-01, 1e-14 | 9.39e-01, 5e-05 | 1.59e+00, 1e-14 | 0.00e+00, 3e-14 | 2.67e-02, 3e-14 | 6.50e-01, 4e-14(+) | 6.56e-01, 5e-05(+) |
| 20News-8000-1000 | -1.5e+00 | 10 | 5.03e-01, 4e-14 | 5.16e-01, 1e-04 | 8.90e-01, 9e-14 | 0.00e+00, 5e-14 | 7.71e-03, 4e-14 | 2.87e-01, 1e-13(+) | 3.46e-01, 1e-04(+) |
| sector-6412-1000 | -3.4e+01 | 10 | 2.03e+00, 2e-08 | 2.28e+00, 3e-04 | 1.52e+01, 7e-14 | 0.00e+00, 5e-14 | 6.37e-02, 5e-14 | 1.03e+00, 2e-08(+) | 1.13e+00, 3e-04(+) |
| E2006-2000-1000 | -6.2e-01 | 10 | 1.18e-01, 4e-15 | 1.20e-01, 6e-05 | 4.19e-01, 2e-14 | 0.00e+00, 1e-13 | 2.68e-05, 6e-14 | 8.37e-02, 4e-14(+) | 1.01e-01, 6e-05(+) |
| MNIST-60000-784 | -2.5e+05 | 10 | 4.23e+04, 9e-10 | 6.51e+04, 5e-04 | 5.98e+02, 7e-15 | 4.60e+03, 2e-15 | 3.83e+03, 1e-15 | 0.00e+00, 9e-10(+) | 2.14e+04, 5e-04(+) |
| Gisette-3000-1000 | -1.0e+06 | 10 | 4.55e+04, 8e-10 | 3.89e+04, 7e-04 | 0.00e+00, 6e-15 | 3.18e+03, 2e-15 | 3.64e+03, 2e-15 | 7.47e+03, 8e-10(+) | 4.05e+03, 7e-04(+) |
| CnnCaltech-3000-1000 | -3.4e+03 | 10 | 9.82e+02, 1e-14 | 7.56e+02, 5e-04 | 0.00e+00, 6e-15 | 8.11e+01, 2e-15 | 1.09e+02, 2e-15 | 4.91e+02, 4e-14(+) | 1.89e+02, 5e-04(+) |
| Cifar-1000-1000 | -1.4e+05 | 10 | 1.63e+04, 2e-09 | 2.27e+04, 6e-04 | 0.00e+00, 5e-15 | 4.17e+02, 2e-15 | 3.73e+02, 2e-15 | 9.84e+03, 2e-09(+) | 4.20e+03, 6e-04(+) |
| randn-500-1000 | -6.8e+03 | 10 | 6.38e+02, 2e-11 | 2.80e+02, 5e-04 | 3.05e+02, 6e-15 | 5.60e+02, 2e-15 | 5.50e+02, 4e-15 | 2.68e+02, 2e-11(+) | 0.00e+00, 5e-04(+) |
| w1a-2477-300 | -6.6e+03 | 20 | 1.94e+03, 1e-12 | 2.62e+03, 4e-05 | 3.13e+02, 5e-14 | 1.85e+02, 2e-14 | 2.96e+02, 7e-15 | 0.00e+00, 1e-12(+) | 4.21e+02, 4e-05(+) |
| TDT2-500-1000 | -3.9e+00 | 20 | 1.04e+00, 2e-08 | 1.05e+00, 3e-04 | 1.65e+00, 8e-15 | 0.00e+00, 2e-14 | 5.94e-02, 1e-14 | 6.78e-01, 2e-08(+) | 7.34e-01, 3e-04(+) |
| 20News-8000-1000 | -1.7e+00 | 20 | 6.52e-01, 3e-08 | 6.68e-01, 7e-05 | 7.88e-01, 5e-14 | 0.00e+00, 5e-14 | 5.28e-02, 3e-14 | 4.56e-01, 3e-08(+) | 5.13e-01, 7e-05(+) |
| sector-6412-1000 | -4.6e+01 | 20 | 4.84e+00, 1e-07 | 4.86e+00, 7e-04 | 2.03e+01, 5e-14 | 0.00e+00, 5e-14 | 3.92e-01, 3e-14 | 2.11e+00, 1e-07(+) | 2.43e+00, 7e-04(+) |
| E2006-2000-1000 | -6.5e-01 | 20 | 1.42e-01, 7e-15 | 1.45e-01, 9e-05 | 3.17e-01, 1e-14 | 0.00e+00, 5e-14 | 5.53e-04, 2e-14 | 1.00e-01, 2e-14(+) | 1.22e-01, 9e-05(+) |
| MNIST-60000-784 | -2.7e+05 | 20 | 6.64e+04, 7e-12 | 2.49e+05, 6e-04 | 0.00e+00, 5e-15 | 2.51e+03, 1e-15 | 3.92e+03, 1e-15 | 4.75e+04, 7e-12(+) | 1.13e+04, 6e-04(+) |
| Gisette-3000-1000 | -1.0e+06 | 20 | 2.22e+05, 1e-08 | 3.51e+04, 1e-03 | 0.00e+00, 3e-15 | 8.37e+03, 1e-15 | 1.09e+04, 7e-16 | 5.96e+04, 1e-08(+) | 8.33e+03, 1e-03(+) |
| CnnCaltech-3000-1000 | -3.6e+03 | 20 | 1.43e+03, 4e-12 | 1.30e+03, 1e-03 | 0.00e+00, 3e-15 | 1.28e+02, 1e-15 | 1.40e+02, 1e-15 | 4.13e+02, 4e-12(+) | 3.26e+02, 1e-03(+) |
| Cifar-1000-1000 | -1.4e+05 | 20 | 2.74e+04, 3e-09 | 5.00e+04, 2e-03 | 0.00e+00, 3e-15 | 6.34e+02, 8e-16 | 5.78e+02, 8e-16 | 1.62e+04, 3e-09(+) | 1.41e+04, 2e-03(+) |
| randn-500-1000 | -1.1e+04 | 20 | 6.96e+02, 2e-10 | 5.08e+02, 1e-03 | 1.92e+02, 4e-15 | 9.19e+02, 9e-16 | 9.03e+02, 1e-15 | 3.62e+02, 2e-10(+) | 0.00e+00, 1e-03(+) |
| w1a-2477-300 | -1.2e+04 | 100 | 6.69e+03, 1e-07 | 8.92e+03, 9e-04 | 7.44e+01, 3e-14 | 1.18e+02, 2e-14 | 0.00e+00, 7e-15 | 1.12e+01, 1e-07(+) | 3.96e+01, 9e-04(+) |
| TDT2-500-1000 | -4.5e+00 | 100 | 1.25e+00, 3e-05 | 1.29e+00, 2e-04 | 1.96e+00, 1e-15 | 0.00e+00, 5e-15 | 2.09e-01, 4e-15 | 1.01e+00, 3e-05(+) | 1.06e+00, 2e-04(+) |
| 20News-8000-1000 | -1.9e+00 | 100 | 5.92e-01, 2e-05 | 6.01e-01, 2e-04 | 9.31e-01, 3e-15 | 0.00e+00, 4e-15 | 1.29e-02, 8e-15 | 4.90e-01, 2e-05(+) | 5.11e-01, 2e-04(+) |
| sector-6412-1000 | -8.5e+01 | 100 | 1.41e+01, 1e-04 | 2.59e+01, 5e-03 | 4.80e+01, 7e-15 | 0.00e+00, 2e-14 | 2.31e+00, 1e-14 | 9.98e+00, 1e-04(+) | 1.03e+01, 5e-03(+) |
| E2006-2000-1000 | -6.9e-01 | 100 | 1.63e-01, 3e-07 | 1.66e-01, 4e-06 | 3.64e-01, 2e-15 | 0.00e+00, 1e-14 | 2.41e-03, 1e-14 | 1.42e-01, 3e-07(+) | 1.42e-01, 4e-06(+) |
| MNIST-60000-784 | -3.3e+05 | 100 | 1.45e+05, 3e-06 | 3.20e+05, 2e-03 | 1.72e+04, 2e-15 | 0.00e+00, 2e-15 | 2.59e+03, 2e-15 | 3.88e+04, 3e-06(+) | 1.74e+03, 2e-03(+) |
| Gisette-3000-1000 | -1.1e+06 | 100 | 8.19e+05, 5e-06 | 5.37e+05, 3e-02 | 0.00e+00, 2e-15 | 2.62e+04, 1e-15 | 2.74e+04, 8e-16 | 5.68e+05, 5e-06(+) | 3.18e+05, 3e-02(+) |
| CnnCaltech-3000-1000 | -4.5e+03 | 100 | 3.28e+03, 8e-05 | 2.02e+03, 2e-02 | 0.00e+00, 2e-15 | 1.33e+02, 1e-15 | 1.74e+02, 1e-15 | 1.91e+03, 8e-05(+) | 5.63e+02, 2e-02(+) |
| Cifar-1000-1000 | -1.4e+05 | 100 | 1.18e+05, 3e-05 | 4.92e+04, 3e-02 | 0.00e+00, 2e-15 | 2.78e+02, 9e-16 | 1.26e+03, 1e-15 | 5.10e+04, 3e-05(+) | 3.77e+04, 3e-02(+) |
| randn-500-1000 | -3.4e+04 | 100 | 4.75e+03, 5e-05 | 1.63e+03, 2e-02 | 2.83e+02, 2e-15 | 6.24e+02, 1e-15 | 0.00e+00, 1e-15 | 2.82e+03, 5e-05(+) | 8.40e+02, 2e-02(+) |
| time limit=30 |  |  |  |  |  |  |  |  |  |
| w1a-2477-300 | -5.2e+03 | 10 | 5.14e+02, 2e-13 | 1.28e+03, 1e-05 | 2.77e+02, 1e-13 | 4.96e+02, 4e-14 | 1.61e+02, 4e-14 | 0.00e+00, 2e-13(+) | 1.45e+02, 1e-05(+) |
| TDT2-500-1000 | -3.5e+00 | 10 | 9.22e-01, 1e-14 | 9.39e-01, 3e-05 | 6.60e-01, 6e-14 | 0.00e+00, 4e-14 | 2.65e-02, 3e-14 | 6.70e-01, 4e-14(+) | 6.98e-01, 3e-05(+) |
| 20News-8000-1000 | -1.6e+00 | 10 | 5.09e-01, 2e-14 | 5.22e-01, 1e-05 | 3.08e-01, 1e-13 | 6.47e-03, 8e-14 | 0.00e+00, 6e-14 | 2.84e-01, 1e-13(+) | 3.25e-01, 1e-05(+) |
| sector-6412-1000 | -3.4e+01 | 10 | 2.04e+00, 1e-11 | 2.29e+00, 1e-04 | 4.46e+00, 1e-13 | 0.00e+00, 9e-14 | 6.42e-02, 6e-14 | 1.09e+00, 1e-11(+) | 1.04e+00, 1e-04(+) |
| E2006-2000-1000 | -6.2e-01 | 10 | 1.18e-01, 3e-15 | 1.20e-01, 2e-05 | 1.15e-01, 6e-14 | 0.00e+00, 2e-13 | 2.68e-05, 6e-14 | 8.32e-02, 3e-14(+) | 9.95e-02, 2e-05(+) |
| MNIST-60000-784 | -2.5e+05 | 10 | 4.47e+04, 6e-10 | 6.46e+04, 3e-04 | 0.00e+00, 3e-14 | 5.73e+02, 1e-14 | 5.11e+02, 4e-15 | 3.82e+03, 6e-10(+) | 2.49e+04, 3e-04(+) |
| Gisette-3000-1000 | -1.0e+06 | 10 | 4.49e+04, 9e-10 | 3.32e+04, 5e-04 | 0.00e+00, 2e-14 | 1.04e+03, 4e-15 | 1.09e+03, 4e-15 | 6.41e+03, 9e-10(+) | 4.03e+03, 5e-04(+) |
| CnnCaltech-3000-1000 | -3.4e+03 | 10 | 1.01e+03, 1e-14 | 7.85e+02, 1e-04 | 0.00e+00, 2e-14 | 1.75e+01, 5e-15 | 6.63e+01, 4e-15 | 4.70e+02, 3e-14(+) | 2.05e+02, 1e-04(+) |
| Cifar-1000-1000 | -1.4e+05 | 10 | 1.61e+04, 2e-09 | 2.09e+04, 3e-04 | 0.00e+00, 2e-14 | 1.48e+02, 5e-15 | 6.74e+01, 4e-15 | 9.37e+03, 2e-09(+) | 3.27e+03, 3e-04(+) |
| randn-500-1000 | -6.8e+03 | 10 | 6.34e+02, 2e-11 | 2.82e+02, 1e-04 | 6.97e+01, 2e-14 | 2.66e+02, 6e-15 | 2.74e+02, 7e-15 | 2.23e+02, 2e-11(+) | 0.00e+00, 1e-04(+) |
| w1a-2477-300 | -6.6e+03 | 20 | 1.95e+03, 1e-12 | 2.63e+03, 2e-05 | 3.10e+02, 1e-13 | 1.65e+02, 3e-14 | 2.73e+02, 2e-14 | 0.00e+00, 1e-12(+) | 3.42e+02, 2e-05(+) |
| TDT2-500-1000 | -3.9e+00 | 20 | 1.04e+00, 8e-14 | 1.06e+00, 6e-05 | 7.34e-01, 4e-14 | 0.00e+00, 3e-14 | 9.44e-03, 3e-14 | 6.71e-01, 1e-13(+) | 7.16e-01, 6e-05(+) |
| 20News-8000-1000 | -1.7e+00 | 20 | 6.52e-01, 1e-13 | 6.69e-01, 1e-05 | 3.75e-01, 1e-13 | 0.00e+00, 5e-14 | 5.20e-02, 5e-14 | 4.58e-01, 2e-13(+) | 5.01e-01, 1e-05(+) |
| sector-6412-1000 | -4.6e+01 | 20 | 4.86e+00, 6e-09 | 4.89e+00, 2e-04 | 7.49e+00, 1e-13 | 0.00e+00, 6e-14 | 4.06e-01, 4e-14 | 2.13e+00, 6e-09(+) | 2.59e+00, 2e-04(+) |
| E2006-2000-1000 | -6.5e-01 | 20 | 1.42e-01, 8e-15 | 1.45e-01, 4e-05 | 2.48e-02, 4e-14 | 0.00e+00, 6e-14 | 5.54e-04, 3e-14 | 1.04e-01, 4e-14(+) | 1.18e-01, 4e-05(+) |
| MNIST-60000-784 | -2.7e+05 | 20 | 7.33e+04, 7e-12 | 2.27e+05, 4e-04 | 0.00e+00, 2e-14 | 2.19e+03, 3e-15 | 3.53e+03, 3e-15 | 5.49e+04, 7e-12(+) | 1.73e+04, 4e-04(+) |
| Gisette-3000-1000 | -1.0e+06 | 20 | 2.11e+05, 8e-09 | 3.36e+04, 8e-04 | 0.00e+00, 9e-15 | 3.21e+03, 2e-15 | 4.36e+03, 2e-15 | 6.19e+04, 8e-09(+) | 1.06e+04, 8e-04(+) |
| CnnCaltech-3000-1000 | -3.7e+03 | 20 | 1.50e+03, 4e-12 | 1.38e+03, 3e-04 | 0.00e+00, 1e-14 | 4.37e+01, 2e-15 | 7.19e+01, 2e-15 | 4.98e+02, 4e-12(+) | 3.24e+02, 3e-04(+) |
| Cifar-1000-1000 | -1.4e+05 | 20 | 2.74e+04, 3e-09 | 5.00e+04, 5e-04 | 0.00e+00, 1e-14 | 3.07e+02, 2e-15 | 3.97e+02, 2e-15 | 1.65e+04, 3e-09(+) | 9.86e+03, 5e-04(+) |
| randn-500-1000 | -1.1e+04 | 20 | 8.52e+02, 2e-11 | 6.83e+02, 3e-04 | 0.00e+00, 1e-14 | 5.34e+02, 3e-15 | 4.87e+02, 2e-15 | 5.21e+02, 2e-11(+) | 1.80e+02, 3e-04(+) |
| w1a-2477-300 | -1.2e+04 | 100 | 6.75e+03, 4e-09 | 8.98e+03, 2e-04 | 1.40e+01, 3e-14 | 1.47e+02, 2e-14 | 0.00e+00, 9e-15 | 4.21e-02, 4e-09(+) | 8.87e+01, 2e-04(+) |
| TDT2-500-1000 | -4.8e+00 | 100 | 1.50e+00, 3e-14 | 1.53e+00, 7e-05 | 1.62e+00, 2e-15 | 0.00e+00, 9e-15 | 1.45e-02, 8e-15 | 1.23e+00, 4e-14(+) | 1.28e+00, 7e-05(+) |
| 20News-8000-1000 | -2.0e+00 | 100 | 7.14e-01, 2e-07 | 7.23e-01, 7e-05 | 3.83e-01, 1e-14 | 0.00e+00, 2e-14 | 6.30e-03, 2e-14 | 5.85e-01, 2e-07(+) | 6.18e-01, 7e-05(+) |
| sector-6412-1000 | -8.5e+01 | 100 | 1.49e+01, 6e-08 | 2.67e+01, 2e-03 | 3.17e+01, 3e-14 | 0.00e+00, 4e-14 | 6.21e-02, 2e-14 | 9.81e+00, 6e-08(+) | 1.10e+01, 2e-03(+) |
| E2006-2000-1000 | -6.9e-01 | 100 | 1.64e-01, 5e-14 | 1.67e-01, 2e-06 | 1.19e-01, 3e-15 | 0.00e+00, 4e-14 | 3.06e-04, 4e-14 | 1.42e-01, 5e-14(+) | 1.45e-01, 2e-06(+) |
| MNIST-60000-784 | -3.5e+05 | 100 | 1.60e+05, 3e-08 | 3.39e+05, 6e-04 | 7.01e+03, 3e-15 | 0.00e+00, 2e-15 | 6.68e+03, 2e-15 | 5.71e+04, 3e-08(+) | 1.92e+04, 6e-04(+) |
| Gisette-3000-1000 | -1.1e+06 | 100 | 7.91e+05, 6e-07 | 5.95e+05, 5e-03 | 0.00e+00, 3e-15 | 2.16e+04, 1e-15 | 2.12e+04, 1e-15 | 5.06e+05, 6e-07(+) | 3.53e+05, 5e-03(+) |
| CnnCaltech-3000-1000 | -4.9e+03 | 100 | 3.62e+03, 7e-07 | 2.44e+03, 3e-03 | 0.00e+00, 3e-15 | 1.78e+02, 1e-15 | 1.61e+02, 2e-15 | 2.09e+03, 7e-07(+) | 8.27e+02, 3e-03(+) |
| Cifar-1000-1000 | -1.5e+05 | 100 | 1.20e+05, 2e-07 | 5.97e+04, 3e-03 | 0.00e+00, 3e-15 | 5.10e+02, 1e-15 | 7.80e+02, 1e-15 | 2.98e+04, 2e-07(+) | 4.22e+04, 3e-03(+) |
| randn-500-1000 | -3.6e+04 | 100 | 6.37e+03, 1e-08 | 3.53e+03, 2e-03 | 0.00e+00, 4e-15 | 1.10e+03, 2e-15 | 3.46e+02, 1e-15 | 4.32e+03, 1e-08(+) | 2.50e+03, 2e-03(+) |
| time limit=60 |  |  |  |  |  |  |  |  |  |
| w1a-2477-300 | -5.2e+03 | 10 | 5.15e+02, 2e-13 | 1.27e+03, 1e-05 | 2.78e+02, 2e-13 | 4.98e+02, 9e-14 | 1.56e+02, 1e-13 | 0.00e+00, 2e-13(+) | 1.46e+02, 1e-05(+) |
| TDT2-500-1000 | -3.5e+00 | 10 | 9.22e-01, 1e-14 | 9.39e-01, 1e-05 | 5.16e-02, 1e-13 | 0.00e+00, 6e-14 | 2.66e-02, 5e-14 | 6.43e-01, 5e-14(+) | 6.49e-01, 1e-05(+) |
| 20News-8000-1000 | -1.6e+00 | 10 | 5.09e-01, 1e-14 | 5.22e-01, 9e-07 | 1.83e-02, 2e-13 | 6.49e-03, 9e-14 | 0.00e+00, 7e-14 | 2.71e-01, 9e-14(+) | 3.09e-01, 9e-07(+) |
| sector-6412-1000 | -3.4e+01 | 10 | 2.04e+00, 1e-11 | 2.29e+00, 5e-05 | 1.46e+00, 2e-13 | 0.00e+00, 2e-13 | 5.85e-02, 1e-13 | 9.61e-01, 1e-11(+) | 1.00e+00, 5e-05(+) |
| E2006-2000-1000 | -6.2e-01 | 10 | 1.18e-01, 3e-15 | 1.20e-01, 8e-06 | 1.67e-03, 7e-14 | 0.00e+00, 2e-13 | 2.67e-05, 6e-14 | 7.55e-02, 4e-14(+) | 9.28e-02, 8e-06(+) |
| MNIST-60000-784 | -2.5e+05 | 10 | 4.52e+04, 3e-11 | 6.19e+04, 2e-04 | 3.11e+02, 9e-14 | 0.00e+00, 2e-14 | 1.25e+02, 1e-14 | 1.41e+03, 3e-11(+) | 2.76e+04, 2e-04(+) |
| Gisette-3000-1000 | -1.0e+06 | 10 | 4.23e+04, 9e-10 | 2.62e+04, 4e-04 | 0.00e+00, 7e-14 | 6.86e+02, 1e-14 | 6.03e+02, 9e-15 | 7.00e+03, 9e-10(+) | 2.26e+03, 4e-04(+) |
| CnnCaltech-3000-1000 | -3.4e+03 | 10 | 1.03e+03, 1e-14 | 8.05e+02, 4e-05 | 0.00e+00, 5e-14 | 1.58e+01, 1e-14 | 5.04e+01, 9e-15 | 4.62e+02, 4e-14(+) | 2.08e+02, 4e-05(+) |
| Cifar-1000-1000 | -1.4e+05 | 10 | 1.56e+04, 2e-09 | 1.79e+04, 2e-04 | 5.10e+01, 6e-14 | 8.57e+01, 1e-14 | 0.00e+00, 1e-14 | 9.32e+03, 2e-09(+) | 2.44e+03, 2e-04(+) |
| randn-500-1000 | -6.8e+03 | 10 | 6.74e+02, 2e-11 | 3.23e+02, 3e-05 | 0.00e+00, 5e-14 | 1.95e+02, 2e-14 | 2.04e+02, 1e-14 | 2.36e+02, 2e-11(+) | 2.77e+01, 3e-05(+) |
| w1a-2477-300 | -6.6e+03 | 20 | 1.94e+03, 1e-12 | 2.62e+03, 1e-05 | 3.08e+02, 2e-13 | 1.60e+02, 7e-14 | 2.67e+02, 4e-14 | 0.00e+00, 1e-12(+) | 3.08e+02, 1e-05(+) |
| TDT2-500-1000 | -3.9e+00 | 20 | 1.05e+00, 1e-14 | 1.06e+00, 1e-05 | 8.31e-02, 7e-14 | 3.93e-03, 4e-14 | 0.00e+00, 4e-14 | 6.53e-01, 4e-14(+) | 7.16e-01, 1e-05(+) |
| 20News-8000-1000 | -1.7e+00 | 20 | 6.53e-01, 2e-14 | 6.69e-01, 6e-06 | 1.13e-01, 1e-13 | 0.00e+00, 7e-14 | 3.17e-02, 6e-14 | 4.35e-01, 8e-14(+) | 4.82e-01, 6e-06(+) |
| sector-6412-1000 | -4.6e+01 | 20 | 4.87e+00, 2e-13 | 4.90e+00, 1e-04 | 2.10e+00, 2e-13 | 0.00e+00, 1e-13 | 4.08e-01, 5e-14 | 2.13e+00, 3e-13(+) | 2.38e+00, 1e-04(+) |
| E2006-2000-1000 | -6.5e-01 | 20 | 1.42e-01, 5e-15 | 1.45e-01, 5e-06 | 2.18e-03, 8e-14 | 0.00e+00, 6e-14 | 5.42e-04, 3e-14 | 1.02e-01, 2e-14(+) | 1.13e-01, 5e-06(+) |
| MNIST-60000-784 | -2.8e+05 | 20 | 7.55e+04, 7e-12 | 1.08e+05, 3e-04 | 8.77e+02, 6e-14 | 0.00e+00, 2e-14 | 1.04e+03, 6e-15 | 5.67e+04, 7e-12(+) | 5.17e+04, 3e-04(+) |
| Gisette-3000-1000 | -1.1e+06 | 20 | 1.90e+05, 8e-09 | 3.23e+04, 7e-04 | 0.00e+00, 3e-14 | 2.06e+03, 5e-15 | 1.59e+03, 4e-15 | 5.05e+04, 8e-09(+) | 8.92e+03, 7e-04(+) |
| CnnCaltech-3000-1000 | -3.7e+03 | 20 | 1.53e+03, 4e-12 | 1.41e+03, 1e-04 | 6.78e+00, 3e-14 | 0.00e+00, 5e-15 | 4.81e+01, 4e-15 | 5.00e+02, 4e-12(+) | 3.26e+02, 1e-04(+) |
| Cifar-1000-1000 | -1.4e+05 | 20 | 2.70e+04, 2e-09 | 4.88e+04, 4e-04 | 0.00e+00, 3e-14 | 1.76e+02, 5e-15 | 2.54e+02, 5e-15 | 1.65e+04, 2e-09(+) | 8.28e+03, 4e-04(+) |
| randn-500-1000 | -1.1e+04 | 20 | 1.07e+03, 2e-11 | 9.02e+02, 9e-05 | 0.00e+00, 3e-14 | 4.35e+02, 6e-15 | 4.49e+02, 5e-15 | 6.91e+02, 2e-11(+) | 3.55e+02, 9e-05(+) |
| w1a-2477-300 | -1.2e+04 | 100 | 6.75e+03, 1e-09 | 8.98e+03, 6e-05 | 1.83e+01, 5e-14 | 1.45e+02, 3e-14 | 0.00e+00, 1e-14 | 5.73e+01, 1e-09(+) | 8.76e+01, 6e-05(+) |
| TDT2-500-1000 | -4.8e+00 | 100 | 1.52e+00, 3e-14 | 1.55e+00, 5e-05 | 4.46e-01, 5e-15 | 0.00e+00, 1e-14 | 1.41e-02, 9e-15 | 1.22e+00, 4e-14(+) | 1.27e+00, 5e-05(+) |
| 20News-8000-1000 | -2.0e+00 | 100 | 7.18e-01, 1e-08 | 7.26e-01, 3e-05 | 1.28e-01, 5e-14 | 0.00e+00, 4e-14 | 1.65e-03, 5e-14 | 5.91e-01, 1e-08(+) | 6.26e-01, 3e-05(+) |
| sector-6412-1000 | -8.6e+01 | 100 | 1.52e+01, 3e-09 | 2.69e+01, 8e-04 | 1.09e+01, 8e-14 | 8.85e-02, 5e-14 | 0.00e+00, 4e-14 | 9.94e+00, 3e-09(+) | 1.13e+01, 8e-04(+) |
| E2006-2000-1000 | -6.9e-01 | 100 | 1.64e-01, 5e-14 | 1.67e-01, 5e-07 | 2.06e-02, 8e-15 | 0.00e+00, 4e-14 | 6.25e-05, 4e-14 | 1.34e-01, 5e-14(+) | 1.43e-01, 5e-07(+) |
| MNIST-60000-784 | -3.6e+05 | 100 | 1.63e+05, 1e-08 | 3.45e+05, 4e-04 | 0.00e+00, 5e-15 | 1.94e+03, 2e-15 | 8.98e+03, 2e-14 | 6.47e+04, 1e-08(+) | 2.92e+04, 4e-04(+) |
| Gisette-3000-1000 | -1.1e+06 | 100 | 6.45e+05, 4e-07 | 5.89e+05, 3e-03 | 0.00e+00, 5e-15 | 1.49e+04, 2e-15 | 1.60e+04, 1e-15 | 3.93e+05, 4e-07(+) | 3.26e+05, 3e-03(+) |
| CnnCaltech-3000-1000 | -5.0e+03 | 100 | 3.74e+03, 4e-08 | 2.57e+03, 8e-04 | 0.00e+00, 5e-15 | 1.85e+02, 2e-15 | 1.77e+02, 2e-15 | 2.19e+03, 4e-08(+) | 9.76e+02, 8e-04(+) |
| Cifar-1000-1000 | -1.5e+05 | 100 | 1.21e+05, 5e-08 | 6.10e+04, 1e-03 | 0.00e+00, 5e-15 | 4.59e+02, 2e-15 | 7.68e+02, 2e-15 | 5.51e+04, 5e-08(+) | 4.07e+04, 1e-03(+) |
| randn-500-1000 | -3.7e+04 | 100 | 7.17e+03, 1e-10 | 4.36e+03, 9e-04 | 0.00e+00, 6e-15 | 1.26e+03, 2e-15 | 4.94e+02, 2e-15 | 5.02e+03, 1e-10(+) | 3.32e+03, 9e-04(+) |

Figure: (a) ‘w1a’
Refer to caption: x17.png

**Table 6: Comparisons of objective values $(F(\mathbf{X})-F_{\text{min}})$ of $L_{1}$ norm-based SPCA for all the compared methods with $\lambda=1$. The $1^{st}$, $2^{nd}$, and $3^{rd}$ best results are colored with red, green and blue, respectively. If the objective values of ‘LADMM+OBCD-R’ (or ‘ManPG+OBCD-R’) are smaller than those of ‘LADMM’ (or ‘ManPG’) by a margin of $0.1\times a$, where $a$ represents the objective values of ‘LADMM’ (or ‘ManPG’), they will be marked with ${\color[rgb]{0.9961,0,0}(+)}$.**
| data-m-n | $F_{\text{min}}$ | r | LADMM | RSubGrad | ADMM | ManPG | OBCD-OR | LADMM | ManPG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| + OBCD-R | +OBCD-R |  |  |  |  |  |  |  |  |
| $\lambda$ = 1.00, time limit=10 |  |  |  |  |  |  |  |  |  |
| w1a-2477-300 | -5.7e+03 | 10 | 2.52e+01 | 4.71e+03 | 4.85e+03 | 4.71e+03 | 0.00e+00 | 1.99e+01(+) | 4.45e+01(+) |
| TDT2-500-1000 | 6.6e+00 | 10 | 3.40e+00 | 3.40e+00 | 1.51e+00 | 3.40e+00 | 0.00e+00 | 3.09e-02(+) | 9.81e-02(+) |
| 20News-8000-1000 | 8.5e+00 | 10 | 1.49e+00 | 1.62e+00 | 1.30e+00 | 1.49e+00 | 0.00e+00 | 3.07e-02(+) | 3.07e-02(+) |
| sector-6412-1000 | -2.2e+01 | 10 | 3.00e+01 | 3.00e+01 | 1.62e+01 | 3.00e+01 | 0.00e+00 | 2.73e-01(+) | 2.73e-01(+) |
| E2006-2000-1000 | 9.4e+00 | 10 | 6.17e-01 | 6.37e-01 | 1.60e-01 | 6.17e-01 | 2.32e-04 | 0.00e+00(+) | 0.00e+00(+) |
| MNIST-60000-784 | -3.1e+05 | 10 | 1.91e+02 | 3.13e+05 | 3.13e+05 | 3.13e+05 | 0.00e+00 | 1.60e+02(+) | 1.23e+04(+) |
| Gisette-3000-1000 | -1.1e+06 | 10 | 2.24e+04 | 4.78e+05 | 7.31e+04 | 3.43e+05 | 0.00e+00 | 7.94e+03(+) | 1.39e+04(+) |
| CnnCaltech-3000-1000 | -4.2e+03 | 10 | 2.87e+01 | 1.50e+00 | 9.33e+02 | 8.62e-03 | 2.18e+01 | 2.47e+01(+) | 0.00e+00(+) |
| Cifar-1000-1000 | -1.4e+05 | 10 | 2.75e+03 | 4.28e+04 | 1.22e+04 | 1.26e+00 | 2.54e+03 | 1.53e+03(+) | 0.00e+00(+) |
| randn-500-1000 | -1.3e+04 | 10 | 2.78e+01 | 1.42e+00 | 2.51e+02 | 1.10e-04 | 3.62e+01 | 2.63e+01(+) | 0.00e+00(+) |
| w1a-2477-300 | -7.1e+03 | 20 | 8.04e+01 | 5.95e+03 | 6.05e+03 | 5.94e+03 | 0.00e+00 | 4.91e+01(+) | 1.42e+02(+) |
| TDT2-500-1000 | 1.6e+01 | 20 | 3.75e+00 | 3.91e+00 | 1.89e+00 | 3.75e+00 | 0.00e+00 | 7.54e-03(+) | 7.54e-03(+) |
| 20News-8000-1000 | 1.8e+01 | 20 | 1.66e+00 | 1.94e+00 | 5.55e+00 | 1.66e+00 | 0.00e+00 | 1.08e-02(+) | 1.08e-02(+) |
| sector-6412-1000 | -2.5e+01 | 20 | 4.25e+01 | 2.97e+01 | 3.04e+01 | 4.25e+01 | 0.00e+00 | 9.39e-01(+) | 9.39e-01(+) |
| E2006-2000-1000 | 1.9e+01 | 20 | 6.37e-01 | 8.10e-01 | 7.62e-01 | 6.37e-01 | 0.00e+00 | 1.06e-03(+) | 1.06e-03(+) |
| MNIST-60000-784 | -3.5e+05 | 20 | 1.26e+02 | 3.54e+05 | 3.54e+05 | 3.54e+05 | 1.45e+03 | 0.00e+00(+) | 4.33e+04(+) |
| Gisette-3000-1000 | -1.1e+06 | 20 | 1.31e+04 | 5.01e+05 | 7.75e+04 | 3.57e+05 | 0.00e+00 | 1.90e+03(+) | 2.23e+04(+) |
| CnnCaltech-3000-1000 | -5.1e+03 | 20 | 6.73e+01 | 5.75e+00 | 1.46e+03 | 7.21e-03 | 3.08e+02 | 6.04e+01(+) | 0.00e+00(+) |
| Cifar-1000-1000 | -1.5e+05 | 20 | 3.31e+03 | 4.95e+04 | 1.37e+04 | 1.56e+02 | 6.46e+03 | 2.66e+03(+) | 0.00e+00(+) |
| randn-500-1000 | -2.6e+04 | 20 | 2.56e+01 | 5.82e+00 | 4.11e+02 | 1.51e-03 | 2.61e+02 | 2.44e+01(+) | 0.00e+00(+) |
| w1a-2477-300 | -1.2e+04 | 100 | 1.01e+03 | 5.73e+02 | 4.31e+03 | 3.97e+03 | 0.00e+00 | 8.59e+02(+) | 8.63e+02(+) |
| TDT2-500-1000 | 9.5e+01 | 100 | 4.44e+00 | 9.22e+00 | 3.90e+00 | 4.44e+00 | 0.00e+00 | 0.00e+00(+) | 0.00e+00(+) |
| 20News-8000-1000 | 9.8e+01 | 100 | 9.25e-01 | 4.15e+00 | 7.41e+01 | 9.25e-01 | 2.50e-03 | 0.00e+00(+) | 0.00e+00(+) |
| sector-6412-1000 | 1.6e+01 | 100 | 7.42e+01 | 6.64e+01 | 6.09e+01 | 7.42e+01 | 1.04e+00 | 2.84e-14(+) | 0.00e+00(+) |
| E2006-2000-1000 | 9.9e+01 | 100 | 6.71e-01 | 4.47e+00 | 3.73e+02 | 6.71e-01 | 5.06e-05 | 0.00e+00(+) | 0.00e+00(+) |
| MNIST-60000-784 | -3.9e+05 | 100 | 2.89e+03 | 1.09e+05 | 1.93e+05 | 2.24e+05 | 1.60e+04 | 0.00e+00(+) | 7.58e+04(+) |
| Gisette-3000-1000 | -1.2e+06 | 100 | 3.60e+04 | 8.49e+05 | 1.13e+05 | 3.11e+05 | 2.43e+04 | 0.00e+00(+) | 5.83e+04(+) |
| CnnCaltech-3000-1000 | -6.7e+03 | 100 | 5.35e+02 | 0.00e+00 | 4.43e+03 | 1.40e+03 | 2.24e+03 | 5.33e+02(+) | 1.36e+03(+) |
| Cifar-1000-1000 | -1.5e+05 | 100 | 1.08e+02 | 7.04e+04 | 1.00e+04 | 2.00e+04 | 7.70e+03 | 0.00e+00(+) | 8.15e+03(+) |
| randn-500-1000 | -1.0e+05 | 100 | 1.14e+02 | 0.00e+00 | 1.50e+03 | 6.03e+01 | 4.76e+04 | 1.14e+02(+) | 6.03e+01 |
| $\lambda$ = 1.00, time limit=30 |  |  |  |  |  |  |  |  |  |
| w1a-2477-300 | -5.7e+03 | 10 | 2.60e+01 | 4.71e+03 | 4.85e+03 | 4.71e+03 | 0.00e+00 | 2.07e+01(+) | 4.53e+01(+) |
| TDT2-500-1000 | 6.6e+00 | 10 | 3.40e+00 | 3.40e+00 | 1.51e+00 | 3.40e+00 | 0.00e+00 | 3.09e-02(+) | 3.09e-02(+) |
| 20News-8000-1000 | 8.5e+00 | 10 | 1.49e+00 | 1.53e+00 | 1.30e+00 | 1.49e+00 | 0.00e+00 | 3.07e-02(+) | 3.07e-02(+) |
| sector-6412-1000 | -2.2e+01 | 10 | 3.00e+01 | 3.00e+01 | 1.62e+01 | 3.00e+01 | 0.00e+00 | 2.73e-01(+) | 2.73e-01(+) |
| E2006-2000-1000 | 9.4e+00 | 10 | 6.17e-01 | 6.23e-01 | 1.60e-01 | 6.17e-01 | 6.54e-04 | 0.00e+00(+) | 0.00e+00(+) |
| MNIST-60000-784 | -3.1e+05 | 10 | 1.97e+02 | 3.13e+05 | 3.13e+05 | 3.13e+05 | 0.00e+00 | 1.65e+02(+) | 1.19e+04(+) |
| Gisette-3000-1000 | -1.1e+06 | 10 | 2.53e+04 | 4.35e+05 | 7.61e+04 | 3.24e+05 | 0.00e+00 | 1.06e+04(+) | 1.59e+04(+) |
| CnnCaltech-3000-1000 | -4.2e+03 | 10 | 2.97e+01 | 1.96e+00 | 9.34e+02 | 7.40e-07 | 1.63e+01 | 2.56e+01(+) | 0.00e+00 |
| Cifar-1000-1000 | -1.4e+05 | 10 | 2.79e+03 | 0.00e+00 | 1.22e+04 | 1.20e+01 | 1.96e+02 | 1.57e+03(+) | 1.18e+01(+) |
| randn-500-1000 | -1.3e+04 | 10 | 2.87e+01 | 1.88e+00 | 2.52e+02 | 6.80e-10 | 9.32e+00 | 2.72e+01(+) | 0.00e+00 |
| w1a-2477-300 | -7.1e+03 | 20 | 8.24e+01 | 5.95e+03 | 6.05e+03 | 5.95e+03 | 0.00e+00 | 5.07e+01(+) | 1.43e+02(+) |
| TDT2-500-1000 | 1.6e+01 | 20 | 3.75e+00 | 3.81e+00 | 1.89e+00 | 3.75e+00 | 0.00e+00 | 8.92e-03(+) | 8.92e-03(+) |
| 20News-8000-1000 | 1.8e+01 | 20 | 1.66e+00 | 1.76e+00 | 5.55e+00 | 1.66e+00 | 0.00e+00 | 1.08e-02(+) | 1.08e-02(+) |
| sector-6412-1000 | -2.5e+01 | 20 | 4.25e+01 | 2.92e+01 | 3.04e+01 | 4.25e+01 | 0.00e+00 | 9.61e-01(+) | 9.61e-01(+) |
| E2006-2000-1000 | 1.9e+01 | 20 | 6.38e-01 | 6.45e-01 | 7.61e-01 | 6.38e-01 | 0.00e+00 | 1.30e-03(+) | 1.30e-03(+) |
| MNIST-60000-784 | -3.5e+05 | 20 | 1.25e+02 | 3.54e+05 | 3.54e+05 | 3.54e+05 | 2.02e+02 | 0.00e+00(+) | 4.33e+04(+) |
| Gisette-3000-1000 | -1.1e+06 | 20 | 2.50e+04 | 4.55e+05 | 8.95e+04 | 3.55e+05 | 0.00e+00 | 1.38e+04(+) | 3.72e+04(+) |
| CnnCaltech-3000-1000 | -5.1e+03 | 20 | 7.47e+01 | 1.17e+01 | 1.47e+03 | 1.07e-05 | 4.52e+01 | 6.77e+01(+) | 0.00e+00(+) |
| Cifar-1000-1000 | -1.5e+05 | 20 | 4.40e+03 | 1.02e+04 | 1.48e+04 | 3.16e-01 | 2.75e+03 | 3.73e+03(+) | 0.00e+00(+) |
| randn-500-1000 | -2.6e+04 | 20 | 2.78e+01 | 6.97e+00 | 4.13e+02 | 5.23e-04 | 7.67e+01 | 2.66e+01(+) | 0.00e+00(+) |
| w1a-2477-300 | -1.2e+04 | 100 | 1.11e+03 | 6.63e+02 | 4.41e+03 | 2.20e+03 | 0.00e+00 | 9.60e+02(+) | 1.14e+03(+) |
| TDT2-500-1000 | 9.5e+01 | 100 | 4.45e+00 | 5.91e+00 | 3.90e+00 | 4.45e+00 | 0.00e+00 | 3.43e-03(+) | 3.43e-03(+) |
| 20News-8000-1000 | 9.8e+01 | 100 | 9.26e-01 | 4.77e+00 | 7.06e+01 | 9.26e-01 | 0.00e+00 | 9.23e-04(+) | 9.23e-04(+) |
| sector-6412-1000 | 1.6e+01 | 100 | 7.45e+01 | 6.46e+01 | 6.12e+01 | 7.45e+01 | 0.00e+00 | 2.65e-01(+) | 2.65e-01(+) |
| E2006-2000-1000 | 9.9e+01 | 100 | 6.72e-01 | 1.79e+00 | 3.67e+02 | 6.72e-01 | 0.00e+00 | 3.03e-04(+) | 3.03e-04(+) |
| MNIST-60000-784 | -4.0e+05 | 100 | 1.38e+04 | 7.60e+04 | 2.04e+05 | 2.09e+05 | 0.00e+00 | 1.10e+04(+) | 9.39e+04(+) |
| Gisette-3000-1000 | -1.2e+06 | 100 | 3.95e+04 | 2.65e+05 | 1.16e+05 | 3.13e+05 | 0.00e+00 | 3.52e+03(+) | 6.23e+04(+) |
| CnnCaltech-3000-1000 | -6.8e+03 | 100 | 5.46e+02 | 0.00e+00 | 4.43e+03 | 4.48e+02 | 1.55e+03 | 5.44e+02(+) | 4.47e+02(+) |
| Cifar-1000-1000 | -1.5e+05 | 100 | 1.08e+02 | 4.95e+04 | 9.99e+03 | 1.13e+04 | 4.78e+03 | 0.00e+00(+) | 9.25e+03(+) |
| randn-500-1000 | -1.0e+05 | 100 | 1.31e+02 | 0.00e+00 | 1.50e+03 | 1.08e+00 | 2.23e+04 | 1.31e+02(+) | 1.08e+00 |
| $\lambda$ = 1.00, time limit=60 |  |  |  |  |  |  |  |  |  |
| w1a-2477-300 | -5.7e+03 | 10 | 2.54e+01 | 4.71e+03 | 4.85e+03 | 4.71e+03 | 0.00e+00 | 2.01e+01(+) | 4.35e+01(+) |
| TDT2-500-1000 | 6.6e+00 | 10 | 3.40e+00 | 3.40e+00 | 1.51e+00 | 3.40e+00 | 0.00e+00 | 3.09e-02(+) | 3.09e-02(+) |
| 20News-8000-1000 | 8.5e+00 | 10 | 1.49e+00 | 1.50e+00 | 1.30e+00 | 1.49e+00 | 0.00e+00 | 3.07e-02(+) | 3.07e-02(+) |
| sector-6412-1000 | -2.2e+01 | 10 | 3.00e+01 | 3.01e+01 | 1.63e+01 | 3.00e+01 | 0.00e+00 | 3.54e-01(+) | 3.54e-01(+) |
| E2006-2000-1000 | 9.4e+00 | 10 | 6.18e-01 | 6.21e-01 | 1.61e-01 | 6.18e-01 | 0.00e+00 | 1.59e-03(+) | 1.59e-03(+) |
| MNIST-60000-784 | -3.1e+05 | 10 | 1.98e+02 | 3.13e+05 | 3.13e+05 | 3.13e+05 | 0.00e+00 | 1.66e+02(+) | 1.15e+04(+) |
| Gisette-3000-1000 | -1.1e+06 | 10 | 2.51e+04 | 3.21e+05 | 7.62e+04 | 4.52e+05 | 0.00e+00 | 1.06e+04(+) | 1.92e+04(+) |
| CnnCaltech-3000-1000 | -4.2e+03 | 10 | 2.97e+01 | 1.77e+00 | 9.34e+02 | 3.44e-08 | 1.19e+01 | 2.56e+01(+) | 0.00e+00 |
| Cifar-1000-1000 | -1.4e+05 | 10 | 2.79e+03 | 0.00e+00 | 1.22e+04 | 4.22e+00 | 6.36e+01 | 1.60e+03(+) | 4.19e+00(+) |
| randn-500-1000 | -1.3e+04 | 10 | 2.87e+01 | 1.76e+00 | 2.52e+02 | 8.33e-09 | 6.36e+00 | 2.72e+01(+) | 0.00e+00 |
| w1a-2477-300 | -7.1e+03 | 20 | 9.14e+01 | 5.96e+03 | 6.06e+03 | 5.95e+03 | 0.00e+00 | 5.89e+01(+) | 1.49e+02(+) |
| TDT2-500-1000 | 1.6e+01 | 20 | 3.75e+00 | 3.76e+00 | 1.89e+00 | 3.75e+00 | 0.00e+00 | 1.10e-02(+) | 1.10e-02(+) |
| 20News-8000-1000 | 1.8e+01 | 20 | 1.66e+00 | 1.70e+00 | 5.55e+00 | 1.66e+00 | 0.00e+00 | 1.08e-02(+) | 1.08e-02(+) |
| sector-6412-1000 | -2.6e+01 | 20 | 4.27e+01 | 2.93e+01 | 3.07e+01 | 4.27e+01 | 0.00e+00 | 1.22e+00(+) | 1.22e+00(+) |
| E2006-2000-1000 | 1.9e+01 | 20 | 6.38e-01 | 6.41e-01 | 7.62e-01 | 6.38e-01 | 0.00e+00 | 1.70e-03(+) | 1.70e-03(+) |
| MNIST-60000-784 | -3.5e+05 | 20 | 3.33e+02 | 3.55e+05 | 3.55e+05 | 3.55e+05 | 0.00e+00 | 2.09e+02(+) | 4.30e+04(+) |
| Gisette-3000-1000 | -1.1e+06 | 20 | 3.01e+04 | 3.92e+05 | 9.47e+04 | 5.14e+05 | 0.00e+00 | 1.86e+04(+) | 3.99e+04(+) |
| CnnCaltech-3000-1000 | -5.1e+03 | 20 | 7.53e+01 | 1.16e+01 | 1.47e+03 | 1.57e-05 | 3.28e+01 | 6.83e+01(+) | 0.00e+00(+) |
| Cifar-1000-1000 | -1.5e+05 | 20 | 4.44e+03 | 0.00e+00 | 1.48e+04 | 2.72e+01 | 1.53e+03 | 3.80e+03(+) | 2.71e+01(+) |
| randn-500-1000 | -2.6e+04 | 20 | 2.85e+01 | 7.30e+00 | 4.14e+02 | 1.32e-07 | 2.91e+01 | 2.74e+01(+) | 0.00e+00 |
| w1a-2477-300 | -1.2e+04 | 100 | 1.16e+03 | 7.00e+02 | 4.46e+03 | 1.55e+03 | 0.00e+00 | 1.01e+03(+) | 9.42e+02(+) |
| TDT2-500-1000 | 9.5e+01 | 100 | 4.45e+00 | 5.14e+00 | 3.90e+00 | 4.45e+00 | 0.00e+00 | 4.80e-03(+) | 4.80e-03(+) |
| 20News-8000-1000 | 9.8e+01 | 100 | 9.26e-01 | 2.71e+00 | 7.06e+01 | 9.26e-01 | 0.00e+00 | 1.20e-03(+) | 1.20e-03(+) |
| sector-6412-1000 | 1.6e+01 | 100 | 7.45e+01 | 6.25e+01 | 6.13e+01 | 7.45e+01 | 0.00e+00 | 2.94e-01(+) | 2.94e-01(+) |
| E2006-2000-1000 | 9.9e+01 | 100 | 6.72e-01 | 1.24e+00 | 3.67e+02 | 6.72e-01 | 0.00e+00 | 4.65e-04(+) | 4.65e-04(+) |
| MNIST-60000-784 | -4.1e+05 | 100 | 2.29e+04 | 6.95e+04 | 2.13e+05 | 2.14e+05 | 0.00e+00 | 2.00e+04(+) | 9.80e+04(+) |
| Gisette-3000-1000 | -1.2e+06 | 100 | 5.38e+04 | 4.49e+05 | 1.30e+05 | 3.25e+05 | 0.00e+00 | 1.79e+04(+) | 7.67e+04(+) |
| CnnCaltech-3000-1000 | -6.8e+03 | 100 | 5.51e+02 | 0.00e+00 | 4.43e+03 | 2.67e+02 | 1.13e+03 | 5.49e+02(+) | 2.67e+02 |
| Cifar-1000-1000 | -1.5e+05 | 100 | 1.08e+02 | 4.01e+04 | 9.99e+03 | 1.99e+03 | 2.42e+03 | 0.00e+00(+) | 1.71e+03(+) |
| randn-500-1000 | -1.0e+05 | 100 | 1.53e+02 | 1.56e+01 | 1.53e+03 | 3.45e-03 | 1.04e+04 | 1.53e+02(+) | 0.00e+00(+) |

**Table 7: Comparisons of objective values $(F(\mathbf{X})-F_{\text{min}})$ of $L_{1}$ norm-based SPCA for all the compared methods with $\lambda=100$. The $1^{st}$, $2^{nd}$, and $3^{rd}$ best results are colored with red, green and blue, respectively. If the objective values of ‘LADMM+OBCD-R’ (or ‘ManPG+OBCD-R’) are smaller than those of ‘LADMM’ (or ‘ManPG’) by a margin of $0.1\times a$, where $a$ represents the objective values of ‘LADMM’ (or ‘ManPG’), they will be marked with ${\color[rgb]{0.9961,0,0}(+)}$.**
| data-m-n | $F_{\text{min}}$ | r | LADMM | RSubGrad | ADMM | ManPG | OBCD-OR | LADMM | ManPG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| + OBCD-R | +OBCD-R |  |  |  |  |  |  |  |  |
| $\lambda$ = 100.00, time limit=10 |  |  |  |  |  |  |  |  |  |
| w1a-2477-300 | -3.0e+03 | 10 | 9.05e+02 | 3.48e+03 | 3.57e+03 | 3.57e+03 | 0.00e+00 | 3.74e+01(+) | 1.36e+01(+) |
| TDT2-500-1000 | 1.0e+03 | 10 | 3.24e+01 | 1.64e+02 | 3.40e+00 | 3.40e+00 | 0.00e+00 | 2.90e+01(+) | 9.81e-02(+) |
| 20News-8000-1000 | 1.0e+03 | 10 | 6.88e+01 | 1.11e+03 | 1.49e+00 | 1.49e+00 | 0.00e+00 | 3.76e+01(+) | 3.07e-02(+) |
| sector-6412-1000 | 9.7e+02 | 10 | 1.55e+01 | 8.08e+02 | 3.00e+01 | 3.00e+01 | 0.00e+00 | 3.21e-01(+) | 3.68e-01(+) |
| E2006-2000-1000 | 1.0e+03 | 10 | 4.77e+01 | 4.69e+02 | 6.18e-01 | 6.18e-01 | 0.00e+00 | 1.01e+01(+) | 1.59e-03(+) |
| MNIST-60000-784 | -3.0e+05 | 10 | 7.53e+02 | 3.00e+05 | 3.00e+05 | 3.00e+05 | 0.00e+00 | 6.65e+02(+) | 7.27e+03(+) |
| Gisette-3000-1000 | -1.1e+06 | 10 | 2.35e+04 | 4.70e+05 | 6.42e+04 | 3.99e+05 | 0.00e+00 | 1.27e+04(+) | 1.43e+04(+) |
| CnnCaltech-3000-1000 | 5.9e+02 | 10 | 2.32e+02 | 1.06e+03 | 2.90e+02 | 2.90e+02 | 0.00e+00 | 7.90e+01(+) | 3.80e+00(+) |
| Cifar-1000-1000 | -1.3e+05 | 10 | 2.93e+02 | 3.78e+04 | 8.94e+03 | 7.45e+03 | 1.90e+03 | 0.00e+00(+) | 5.76e+03(+) |
| randn-500-1000 | -2.0e+03 | 10 | 5.40e+02 | 1.17e+03 | 6.68e+02 | 5.12e+02 | 0.00e+00 | 1.57e+02(+) | 5.72e+00(+) |
| w1a-2477-300 | -3.3e+03 | 20 | 1.72e+03 | 4.45e+03 | 4.45e+03 | 4.45e+03 | 5.59e+01 | 1.07e+02(+) | 0.00e+00(+) |
| TDT2-500-1000 | 2.0e+03 | 20 | 2.71e+02 | 8.34e+02 | 3.75e+00 | 3.75e+00 | 0.00e+00 | 2.09e+02(+) | 1.10e-02(+) |
| 20News-8000-1000 | 2.0e+03 | 20 | 1.59e+02 | 2.90e+03 | 1.66e+00 | 1.66e+00 | 0.00e+00 | 7.94e+01(+) | 1.30e-02(+) |
| sector-6412-1000 | 2.0e+03 | 20 | 9.94e+01 | 2.45e+03 | 4.27e+01 | 4.27e+01 | 0.00e+00 | 4.21e+01(+) | 1.19e+00(+) |
| E2006-2000-1000 | 2.0e+03 | 20 | 1.21e+02 | 1.11e+03 | 6.38e-01 | 6.38e-01 | 0.00e+00 | 8.05e+01(+) | 1.83e-03(+) |
| MNIST-60000-784 | -3.3e+05 | 20 | 3.28e+03 | 3.31e+05 | 3.31e+05 | 3.31e+05 | 0.00e+00 | 2.11e+03(+) | 3.82e+04(+) |
| Gisette-3000-1000 | -1.1e+06 | 20 | 2.30e+04 | 4.72e+05 | 7.61e+04 | 3.72e+05 | 0.00e+00 | 1.27e+04(+) | 3.33e+04(+) |
| CnnCaltech-3000-1000 | 1.4e+03 | 20 | 2.46e+02 | 3.72e+03 | 2.66e+03 | 3.90e+02 | 0.00e+00 | 3.32e+01(+) | 4.63e+00(+) |
| Cifar-1000-1000 | -1.3e+05 | 20 | 3.42e+02 | 5.36e+04 | 1.35e+04 | 1.30e+04 | 1.96e+03 | 0.00e+00(+) | 1.06e+04(+) |
| randn-500-1000 | -3.9e+03 | 20 | 1.25e+03 | 3.69e+03 | 8.33e+02 | 8.19e+02 | 0.00e+00 | 5.13e+02(+) | 1.47e+01(+) |
| w1a-2477-300 | -1.7e+03 | 100 | 5.04e+03 | 4.60e+03 | 7.36e+03 | 5.29e+03 | 1.07e+02 | 5.22e+01(+) | 0.00e+00(+) |
| TDT2-500-1000 | 1.0e+04 | 100 | 1.09e+03 | 2.73e+04 | 4.45e+00 | 4.45e+00 | 1.37e-03 | 8.51e+02(+) | 0.00e+00(+) |
| 20News-8000-1000 | 1.0e+04 | 100 | 8.15e+02 | 5.60e+04 | 9.25e-01 | 9.25e-01 | 3.90e-04 | 6.68e+02(+) | 0.00e+00(+) |
| sector-6412-1000 | 9.9e+03 | 100 | 6.26e+01 | 5.12e+04 | 7.43e+01 | 7.43e+01 | 0.00e+00 | 8.32e+00(+) | 3.75e-02(+) |
| E2006-2000-1000 | 1.0e+04 | 100 | 1.09e+03 | 3.39e+04 | 6.71e-01 | 6.71e-01 | 7.81e-04 | 9.16e+02(+) | 0.00e+00(+) |
| MNIST-60000-784 | -3.4e+05 | 100 | 3.32e+04 | 1.24e+05 | 1.57e+05 | 1.75e+05 | 0.00e+00 | 2.70e+04(+) | 6.24e+04(+) |
| Gisette-3000-1000 | -1.1e+06 | 100 | 3.90e+04 | 5.50e+05 | 1.05e+05 | 3.07e+05 | 9.59e+02 | 0.00e+00(+) | 6.87e+04(+) |
| CnnCaltech-3000-1000 | 7.8e+03 | 100 | 9.37e+02 | 3.93e+04 | 1.81e+03 | 9.92e+02 | 1.88e+00 | 6.63e+01(+) | 0.00e+00(+) |
| Cifar-1000-1000 | -1.2e+05 | 100 | 2.86e+02 | 1.32e+05 | 3.72e+04 | 5.68e+04 | 2.65e+03 | 0.00e+00(+) | 2.62e+04(+) |
| randn-500-1000 | -1.8e+04 | 100 | 3.78e+03 | 7.11e+04 | 2.74e+03 | 2.74e+03 | 2.73e+00 | 1.67e+03(+) | 0.00e+00(+) |
| $\lambda$ = 100.00, time limit=30 |  |  |  |  |  |  |  |  |  |
| w1a-2477-300 | -3.0e+03 | 10 | 9.10e+02 | 3.49e+03 | 3.58e+03 | 3.58e+03 | 0.00e+00 | 4.31e+01(+) | 1.73e+01(+) |
| TDT2-500-1000 | 1.0e+03 | 10 | 3.24e+01 | 7.93e+01 | 3.40e+00 | 3.40e+00 | 0.00e+00 | 2.90e+01(+) | 3.09e-02(+) |
| 20News-8000-1000 | 1.0e+03 | 10 | 6.88e+01 | 3.86e+02 | 1.49e+00 | 1.49e+00 | 0.00e+00 | 3.76e+01(+) | 3.07e-02(+) |
| sector-6412-1000 | 9.7e+02 | 10 | 1.55e+01 | 3.52e+02 | 3.00e+01 | 3.00e+01 | 0.00e+00 | 3.21e-01(+) | 3.68e-01(+) |
| E2006-2000-1000 | 1.0e+03 | 10 | 4.77e+01 | 2.10e+02 | 6.18e-01 | 6.18e-01 | 0.00e+00 | 1.01e+01(+) | 1.59e-03(+) |
| MNIST-60000-784 | -3.0e+05 | 10 | 1.33e+03 | 3.01e+05 | 3.01e+05 | 3.01e+05 | 0.00e+00 | 1.25e+03(+) | 7.85e+03(+) |
| Gisette-3000-1000 | -1.1e+06 | 10 | 2.27e+04 | 4.22e+05 | 6.36e+04 | 2.31e+05 | 0.00e+00 | 1.21e+04(+) | 1.58e+04(+) |
| CnnCaltech-3000-1000 | 5.9e+02 | 10 | 2.32e+02 | 4.55e+02 | 2.90e+02 | 2.90e+02 | 0.00e+00 | 7.90e+01(+) | 3.80e+00(+) |
| Cifar-1000-1000 | -1.3e+05 | 10 | 1.37e+03 | 2.98e+01 | 1.00e+04 | 6.75e+01 | 2.99e+03 | 1.07e+03(+) | 0.00e+00(+) |
| randn-500-1000 | -2.0e+03 | 10 | 5.41e+02 | 4.89e+02 | 6.69e+02 | 5.13e+02 | 0.00e+00 | 1.57e+02(+) | 7.08e+00(+) |
| w1a-2477-300 | -3.3e+03 | 20 | 1.72e+03 | 4.44e+03 | 4.45e+03 | 4.45e+03 | 3.07e+01 | 1.07e+02(+) | 0.00e+00(+) |
| TDT2-500-1000 | 2.0e+03 | 20 | 2.71e+02 | 3.75e+02 | 3.75e+00 | 3.75e+00 | 0.00e+00 | 2.09e+02(+) | 1.10e-02(+) |
| 20News-8000-1000 | 2.0e+03 | 20 | 1.59e+02 | 1.00e+03 | 1.66e+00 | 1.66e+00 | 0.00e+00 | 7.93e+01(+) | 1.08e-02(+) |
| sector-6412-1000 | 2.0e+03 | 20 | 9.94e+01 | 7.47e+02 | 4.27e+01 | 4.27e+01 | 0.00e+00 | 4.21e+01(+) | 1.19e+00(+) |
| E2006-2000-1000 | 2.0e+03 | 20 | 1.21e+02 | 5.19e+02 | 6.38e-01 | 6.38e-01 | 0.00e+00 | 8.05e+01(+) | 1.83e-03(+) |
| MNIST-60000-784 | -3.3e+05 | 20 | 4.26e+03 | 3.32e+05 | 3.32e+05 | 3.32e+05 | 0.00e+00 | 3.09e+03(+) | 3.80e+04(+) |
| Gisette-3000-1000 | -1.1e+06 | 20 | 2.66e+04 | 4.51e+05 | 7.96e+04 | 3.60e+05 | 0.00e+00 | 1.60e+04(+) | 3.52e+04(+) |
| CnnCaltech-3000-1000 | 1.4e+03 | 20 | 2.46e+02 | 1.16e+03 | 2.65e+03 | 3.90e+02 | 0.00e+00 | 3.29e+01(+) | 4.37e+00(+) |
| Cifar-1000-1000 | -1.3e+05 | 20 | 3.42e+02 | 1.24e+04 | 1.35e+04 | 7.90e+03 | 1.94e+03 | 0.00e+00(+) | 7.12e+03(+) |
| randn-500-1000 | -3.9e+03 | 20 | 1.26e+03 | 1.31e+03 | 8.40e+02 | 8.26e+02 | 0.00e+00 | 4.98e+02(+) | 2.17e+01(+) |
| w1a-2477-300 | -1.6e+03 | 100 | 4.99e+03 | 2.58e+03 | 7.31e+03 | 5.24e+03 | 3.09e+01 | 0.00e+00(+) | 3.58e+01(+) |
| TDT2-500-1000 | 1.0e+04 | 100 | 1.08e+03 | 1.13e+04 | 4.45e+00 | 4.45e+00 | 0.00e+00 | 9.69e+02(+) | 4.80e-03(+) |
| 20News-8000-1000 | 1.0e+04 | 100 | 8.15e+02 | 2.28e+04 | 9.26e-01 | 9.26e-01 | 0.00e+00 | 6.68e+02(+) | 1.41e-03(+) |
| sector-6412-1000 | 9.9e+03 | 100 | 6.25e+01 | 2.17e+04 | 7.45e+01 | 7.45e+01 | 0.00e+00 | 8.27e+00(+) | 2.97e-01(+) |
| E2006-2000-1000 | 1.0e+04 | 100 | 1.08e+03 | 1.15e+04 | 6.72e-01 | 6.72e-01 | 0.00e+00 | 9.06e+02(+) | 4.94e-04(+) |
| MNIST-60000-784 | -3.5e+05 | 100 | 4.43e+04 | 8.22e+04 | 1.68e+05 | 1.69e+05 | 0.00e+00 | 3.81e+04(+) | 7.46e+04(+) |
| Gisette-3000-1000 | -1.1e+06 | 100 | 5.71e+04 | 4.81e+05 | 1.23e+05 | 3.23e+05 | 0.00e+00 | 1.81e+04(+) | 8.55e+04(+) |
| CnnCaltech-3000-1000 | 7.8e+03 | 100 | 9.39e+02 | 2.52e+04 | 1.76e+03 | 9.93e+02 | 0.00e+00 | 6.75e+01(+) | 1.29e+00(+) |
| Cifar-1000-1000 | -1.2e+05 | 100 | 2.97e+02 | 8.07e+04 | 3.72e+04 | 5.66e+04 | 2.28e+03 | 0.00e+00(+) | 2.58e+04(+) |
| randn-500-1000 | -1.8e+04 | 100 | 3.80e+03 | 2.57e+04 | 2.75e+03 | 2.75e+03 | 0.00e+00 | 1.64e+03(+) | 1.20e+01(+) |
| $\lambda$ = 100.00, time limit=60 |  |  |  |  |  |  |  |  |  |
| w1a-2477-300 | -3.0e+03 | 10 | 9.08e+02 | 3.49e+03 | 3.58e+03 | 3.58e+03 | 0.00e+00 | 4.10e+01(+) | 1.54e+01(+) |
| TDT2-500-1000 | 1.0e+03 | 10 | 3.23e+01 | 2.86e+01 | 3.40e+00 | 3.40e+00 | 0.00e+00 | 2.90e+01(+) | 3.09e-02(+) |
| 20News-8000-1000 | 1.0e+03 | 10 | 6.88e+01 | 1.03e+02 | 1.49e+00 | 1.49e+00 | 0.00e+00 | 3.76e+01(+) | 3.07e-02(+) |
| sector-6412-1000 | 9.7e+02 | 10 | 1.55e+01 | 1.53e+02 | 3.00e+01 | 3.00e+01 | 0.00e+00 | 3.21e-01(+) | 3.68e-01(+) |
| E2006-2000-1000 | 1.0e+03 | 10 | 4.77e+01 | 8.87e+01 | 6.18e-01 | 6.18e-01 | 0.00e+00 | 1.01e+01(+) | 1.59e-03(+) |
| MNIST-60000-784 | -3.0e+05 | 10 | 1.18e+03 | 3.01e+05 | 3.01e+05 | 3.01e+05 | 0.00e+00 | 1.09e+03(+) | 7.17e+03(+) |
| Gisette-3000-1000 | -1.1e+06 | 10 | 1.28e+04 | 3.01e+05 | 5.40e+04 | 3.00e+05 | 0.00e+00 | 2.18e+03(+) | 6.93e+03(+) |
| CnnCaltech-3000-1000 | 6.0e+02 | 10 | 2.28e+02 | 3.15e+02 | 2.86e+02 | 2.86e+02 | 1.57e+01 | 6.79e+01(+) | 0.00e+00(+) |
| Cifar-1000-1000 | -1.3e+05 | 10 | 1.57e+03 | 4.72e+01 | 1.02e+04 | 3.31e+01 | 3.11e+03 | 1.28e+03(+) | 0.00e+00(+) |
| randn-500-1000 | -2.0e+03 | 10 | 5.41e+02 | 3.71e+02 | 6.69e+02 | 5.13e+02 | 0.00e+00 | 1.58e+02(+) | 7.08e+00(+) |
| w1a-2477-300 | -3.3e+03 | 20 | 1.72e+03 | 4.44e+03 | 4.45e+03 | 4.45e+03 | 3.85e+01 | 1.07e+02(+) | 0.00e+00(+) |
| TDT2-500-1000 | 2.0e+03 | 20 | 2.71e+02 | 7.21e+01 | 3.75e+00 | 3.75e+00 | 0.00e+00 | 2.09e+02(+) | 1.10e-02(+) |
| 20News-8000-1000 | 2.0e+03 | 20 | 1.59e+02 | 4.86e+02 | 1.66e+00 | 1.66e+00 | 0.00e+00 | 7.93e+01(+) | 1.08e-02(+) |
| sector-6412-1000 | 2.0e+03 | 20 | 9.93e+01 | 4.84e+02 | 4.27e+01 | 4.27e+01 | 0.00e+00 | 4.20e+01(+) | 1.19e+00(+) |
| E2006-2000-1000 | 2.0e+03 | 20 | 1.20e+02 | 2.25e+02 | 6.38e-01 | 6.38e-01 | 0.00e+00 | 8.05e+01(+) | 1.70e-03(+) |
| MNIST-60000-784 | -3.3e+05 | 20 | 4.84e+03 | 3.32e+05 | 3.32e+05 | 3.32e+05 | 0.00e+00 | 3.65e+03(+) | 3.83e+04(+) |
| Gisette-3000-1000 | -1.1e+06 | 20 | 2.49e+04 | 3.82e+05 | 7.81e+04 | 3.36e+05 | 0.00e+00 | 1.44e+04(+) | 3.52e+04(+) |
| CnnCaltech-3000-1000 | 1.4e+03 | 20 | 2.46e+02 | 7.54e+02 | 2.65e+03 | 3.90e+02 | 0.00e+00 | 3.32e+01(+) | 4.63e+00(+) |
| Cifar-1000-1000 | -1.3e+05 | 20 | 1.18e+03 | 0.00e+00 | 1.43e+04 | 5.38e+02 | 2.66e+03 | 8.43e+02(+) | 3.75e+02(+) |
| randn-500-1000 | -3.9e+03 | 20 | 1.26e+03 | 9.29e+02 | 8.40e+02 | 8.26e+02 | 0.00e+00 | 5.05e+02(+) | 2.17e+01(+) |
| w1a-2477-300 | -1.6e+03 | 100 | 4.99e+03 | 2.04e+03 | 7.31e+03 | 5.24e+03 | 1.13e+01 | 0.00e+00(+) | 2.47e+01(+) |
| TDT2-500-1000 | 1.0e+04 | 100 | 1.08e+03 | 3.86e+03 | 4.45e+00 | 4.45e+00 | 0.00e+00 | 9.69e+02(+) | 4.80e-03(+) |
| 20News-8000-1000 | 1.0e+04 | 100 | 8.15e+02 | 1.19e+04 | 9.26e-01 | 9.26e-01 | 0.00e+00 | 6.68e+02(+) | 1.43e-03(+) |
| sector-6412-1000 | 9.9e+03 | 100 | 6.25e+01 | 1.09e+04 | 7.45e+01 | 7.45e+01 | 0.00e+00 | 8.27e+00(+) | 2.97e-01(+) |
| E2006-2000-1000 | 1.0e+04 | 100 | 1.08e+03 | 5.83e+03 | 6.72e-01 | 6.72e-01 | 0.00e+00 | 9.06e+02(+) | 4.94e-04(+) |
| MNIST-60000-784 | -3.6e+05 | 100 | 5.10e+04 | 6.43e+04 | 1.75e+05 | 1.62e+05 | 0.00e+00 | 4.48e+04(+) | 7.91e+04(+) |
| Gisette-3000-1000 | -1.1e+06 | 100 | 6.56e+04 | 4.70e+05 | 1.32e+05 | 3.29e+05 | 0.00e+00 | 2.66e+04(+) | 9.34e+04(+) |
| CnnCaltech-3000-1000 | 7.8e+03 | 100 | 9.39e+02 | 1.39e+04 | 1.76e+03 | 9.93e+02 | 0.00e+00 | 6.77e+01(+) | 1.42e+00(+) |
| Cifar-1000-1000 | -1.2e+05 | 100 | 2.86e+02 | 6.01e+04 | 3.72e+04 | 5.64e+04 | 2.23e+03 | 0.00e+00(+) | 2.61e+04(+) |
| randn-500-1000 | -1.8e+04 | 100 | 3.78e+03 | 1.49e+04 | 2.74e+03 | 2.74e+03 | 7.43e+00 | 1.62e+03(+) | 0.00e+00(+) |

**Table 8: Comparisons of objective values $(F(\mathbf{X})-F_{\text{min}})$ of $L_{1}$ norm-based SPCA for all the compared methods with $\lambda=1000$. The $1^{st}$, $2^{nd}$, and $3^{rd}$ best results are colored with red, green and blue, respectively. If the objective values of ‘LADMM+OBCD-R’ (or ‘ManPG+OBCD-R’) are smaller than those of ‘LADMM’ (or ‘ManPG’) by a margin of $0.1\times a$, where $a$ represents the objective values of ‘LADMM’ (or ‘ManPG’), they will be marked with ${\color[rgb]{0.9961,0,0}(+)}$.**
| data-m-n | $F_{\text{min}}$ | r | LADMM | RSubGrad | ADMM | ManPG | OBCD-OR | LADMM | ManPG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| + OBCD-R | +OBCD-R |  |  |  |  |  |  |  |  |
| $\lambda$ = 1000.00, time limit=10 |  |  |  |  |  |  |  |  |  |
| w1a-2477-300 | 7.0e+03 | 10 | 3.03e+02 | 2.81e+03 | 2.59e+03 | 2.59e+03 | 0.00e+00 | 0.00e+00(+) | 0.00e+00(+) |
| TDT2-500-1000 | 1.0e+04 | 10 | 1.24e+03 | 1.39e+04 | 3.40e+00 | 3.40e+00 | 0.00e+00 | 5.21e+02(+) | 3.09e-02(+) |
| 20News-8000-1000 | 1.0e+04 | 10 | 8.97e+02 | 7.55e+04 | 1.49e+00 | 1.49e+00 | 0.00e+00 | 3.20e+02(+) | 3.07e-02(+) |
| sector-6412-1000 | 1.0e+04 | 10 | 2.88e+01 | 7.59e+04 | 3.00e+01 | 3.00e+01 | 0.00e+00 | 1.29e-01(+) | 3.68e-01(+) |
| E2006-2000-1000 | 1.0e+04 | 10 | 3.88e+02 | 3.77e+04 | 6.18e-01 | 6.18e-01 | 0.00e+00 | 3.87e+02(+) | 1.59e-03(+) |
| MNIST-60000-784 | -2.2e+05 | 10 | 1.69e+02 | 2.27e+05 | 2.27e+05 | 2.27e+05 | 1.71e+04 | 0.00e+00(+) | 9.07e+03(+) |
| Gisette-3000-1000 | -1.0e+06 | 10 | 2.26e+04 | 5.38e+05 | 6.58e+04 | 4.70e+05 | 0.00e+00 | 1.61e+04(+) | 4.39e+04(+) |
| CnnCaltech-3000-1000 | 9.6e+03 | 10 | 1.06e+02 | 7.01e+04 | 1.20e+03 | 2.90e+02 | 0.00e+00 | 0.00e+00(+) | 3.80e+00(+) |
| Cifar-1000-1000 | -9.2e+04 | 10 | 1.89e+03 | 1.14e+05 | 0.00e+00 | 1.00e+05 | 9.99e+04 | 1.80e+03(+) | 9.99e+04(+) |
| randn-500-1000 | 7.0e+03 | 10 | 3.66e+02 | 8.30e+04 | 3.14e+04 | 5.11e+02 | 0.00e+00 | 8.04e+00(+) | 4.69e+00(+) |
| w1a-2477-300 | 1.5e+04 | 20 | 5.30e+02 | 5.72e+03 | 3.90e+03 | 3.90e+03 | 0.00e+00 | 0.00e+00(+) | 0.00e+00(+) |
| TDT2-500-1000 | 2.0e+04 | 20 | 1.40e+03 | 7.01e+04 | 3.75e+00 | 3.75e+00 | 0.00e+00 | 9.85e+02(+) | 1.10e-02(+) |
| 20News-8000-1000 | 2.0e+04 | 20 | 1.45e+03 | 2.56e+05 | 1.66e+00 | 1.66e+00 | 0.00e+00 | 1.45e+03(+) | 1.30e-02(+) |
| sector-6412-1000 | 2.0e+04 | 20 | 6.06e+02 | 2.06e+05 | 4.27e+01 | 4.27e+01 | 0.00e+00 | 1.37e+00(+) | 1.19e+00(+) |
| E2006-2000-1000 | 2.0e+04 | 20 | 1.65e+03 | 8.70e+04 | 6.38e-01 | 6.38e-01 | 0.00e+00 | 7.18e+02(+) | 1.83e-03(+) |
| MNIST-60000-784 | -2.2e+05 | 20 | 3.85e+03 | 2.41e+05 | 2.41e+05 | 2.41e+05 | 2.71e+04 | 0.00e+00(+) | 3.00e+04(+) |
| Gisette-3000-1000 | -9.9e+05 | 20 | 1.41e+04 | 6.98e+05 | 1.15e+05 | 3.45e+05 | 0.00e+00 | 2.26e+03(+) | 9.40e+04(+) |
| CnnCaltech-3000-1000 | 1.9e+04 | 20 | 2.01e+02 | 2.05e+05 | 8.77e+02 | 3.90e+02 | 0.00e+00 | 8.32e-01(+) | 4.37e+00(+) |
| Cifar-1000-1000 | -8.1e+04 | 20 | 1.52e+02 | 2.22e+05 | 1.04e+05 | 9.83e+04 | 9.72e+04 | 0.00e+00(+) | 9.72e+04(+) |
| randn-500-1000 | 1.4e+04 | 20 | 4.49e+02 | 1.98e+05 | 3.61e+04 | 8.21e+02 | 0.00e+00 | 1.10e+01(+) | 1.69e+01(+) |
| w1a-2477-300 | 8.8e+04 | 100 | 3.43e+03 | 2.56e+05 | 8.92e+03 | 6.89e+03 | 3.00e+00 | 0.00e+00(+) | 0.00e+00(+) |
| TDT2-500-1000 | 1.0e+05 | 100 | 6.91e+03 | 1.10e+06 | 4.45e+00 | 4.45e+00 | 4.80e-03 | 5.95e+03(+) | 0.00e+00(+) |
| 20News-8000-1000 | 1.0e+05 | 100 | 3.61e+03 | 1.88e+06 | 9.25e-01 | 9.25e-01 | 1.01e-03 | 3.53e+03(+) | 0.00e+00(+) |
| sector-6412-1000 | 1.0e+05 | 100 | 4.16e+03 | 1.79e+06 | 7.42e+01 | 7.42e+01 | 8.04e-01 | 3.46e+03(+) | 0.00e+00(+) |
| E2006-2000-1000 | 1.0e+05 | 100 | 4.18e+03 | 1.19e+06 | 6.71e-01 | 6.71e-01 | 1.50e-03 | 4.18e+03(+) | 0.00e+00(+) |
| MNIST-60000-784 | -1.7e+05 | 100 | 5.23e+04 | 1.05e+06 | 1.02e+05 | 2.72e+05 | 1.05e+04 | 0.00e+00(+) | 5.25e+03(+) |
| Gisette-3000-1000 | -9.9e+05 | 100 | 5.95e+04 | 2.41e+06 | 4.30e+05 | 5.84e+05 | 3.49e+04 | 0.00e+00(+) | 3.06e+05(+) |
| CnnCaltech-3000-1000 | 9.8e+04 | 100 | 7.61e+02 | 1.90e+06 | 1.40e+03 | 9.92e+02 | 3.59e+00 | 0.00e+00(+) | 4.00e-02(+) |
| Cifar-1000-1000 | -2.5e+03 | 100 | 1.28e+03 | 1.94e+06 | 8.69e+04 | 8.69e+04 | 8.31e+04 | 0.00e+00(+) | 8.31e+04(+) |
| randn-500-1000 | 7.2e+04 | 100 | 2.91e+03 | 1.90e+06 | 2.76e+05 | 2.74e+03 | 7.14e+00 | 6.87e+02(+) | 0.00e+00(+) |
| $\lambda$ = 1000.00, time limit=30 |  |  |  |  |  |  |  |  |  |
| w1a-2477-300 | 7.0e+03 | 10 | 3.03e+02 | 2.64e+03 | 2.59e+03 | 2.59e+03 | 0.00e+00 | 0.00e+00(+) | 0.00e+00(+) |
| TDT2-500-1000 | 1.0e+04 | 10 | 1.24e+03 | 5.07e+03 | 3.40e+00 | 3.40e+00 | 0.00e+00 | 5.20e+02(+) | 3.09e-02(+) |
| 20News-8000-1000 | 1.0e+04 | 10 | 8.96e+02 | 3.10e+04 | 1.49e+00 | 1.49e+00 | 0.00e+00 | 3.20e+02(+) | 3.07e-02(+) |
| sector-6412-1000 | 1.0e+04 | 10 | 2.88e+01 | 2.42e+04 | 3.00e+01 | 3.00e+01 | 0.00e+00 | 1.29e-01(+) | 3.68e-01(+) |
| E2006-2000-1000 | 1.0e+04 | 10 | 3.87e+02 | 1.26e+04 | 6.18e-01 | 6.18e-01 | 0.00e+00 | 3.87e+02(+) | 1.59e-03(+) |
| MNIST-60000-784 | -2.2e+05 | 10 | 1.72e+02 | 2.27e+05 | 2.27e+05 | 2.27e+05 | 1.68e+04 | 0.00e+00(+) | 8.98e+03(+) |
| Gisette-3000-1000 | -1.0e+06 | 10 | 2.41e+04 | 4.51e+05 | 6.72e+04 | 2.94e+05 | 0.00e+00 | 1.76e+04(+) | 3.24e+04(+) |
| CnnCaltech-3000-1000 | 9.6e+03 | 10 | 1.06e+02 | 2.33e+04 | 1.19e+03 | 2.90e+02 | 0.00e+00 | 0.00e+00(+) | 3.80e+00(+) |
| Cifar-1000-1000 | -9.2e+04 | 10 | 1.89e+03 | 2.35e+04 | 0.00e+00 | 1.00e+05 | 9.99e+04 | 1.80e+03(+) | 9.99e+04(+) |
| randn-500-1000 | 7.0e+03 | 10 | 3.66e+02 | 2.66e+04 | 3.10e+04 | 5.11e+02 | 0.00e+00 | 5.13e+00(+) | 4.69e+00(+) |
| w1a-2477-300 | 1.5e+04 | 20 | 5.30e+02 | 4.27e+03 | 3.90e+03 | 3.90e+03 | 0.00e+00 | 0.00e+00(+) | 0.00e+00(+) |
| TDT2-500-1000 | 2.0e+04 | 20 | 1.40e+03 | 2.96e+04 | 3.75e+00 | 3.75e+00 | 0.00e+00 | 9.84e+02(+) | 1.10e-02(+) |
| 20News-8000-1000 | 2.0e+04 | 20 | 1.45e+03 | 9.38e+04 | 1.66e+00 | 1.66e+00 | 0.00e+00 | 1.45e+03(+) | 1.30e-02(+) |
| sector-6412-1000 | 2.0e+04 | 20 | 6.05e+02 | 6.35e+04 | 4.27e+01 | 4.27e+01 | 0.00e+00 | 1.37e+00(+) | 1.19e+00(+) |
| E2006-2000-1000 | 2.0e+04 | 20 | 1.65e+03 | 3.21e+04 | 6.38e-01 | 6.38e-01 | 0.00e+00 | 7.18e+02(+) | 1.83e-03(+) |
| MNIST-60000-784 | -2.2e+05 | 20 | 3.90e+03 | 2.41e+05 | 2.41e+05 | 2.41e+05 | 2.42e+04 | 0.00e+00(+) | 2.78e+04(+) |
| Gisette-3000-1000 | -1.0e+06 | 20 | 1.91e+04 | 5.27e+05 | 1.20e+05 | 3.38e+05 | 0.00e+00 | 7.17e+03(+) | 1.00e+05(+) |
| CnnCaltech-3000-1000 | 1.9e+04 | 20 | 2.02e+02 | 6.88e+04 | 8.74e+02 | 3.90e+02 | 0.00e+00 | 1.09e+00(+) | 4.63e+00(+) |
| Cifar-1000-1000 | -8.1e+04 | 20 | 1.56e+02 | 9.10e+04 | 1.04e+05 | 9.83e+04 | 9.72e+04 | 0.00e+00(+) | 9.72e+04(+) |
| randn-500-1000 | 1.4e+04 | 20 | 4.49e+02 | 8.47e+04 | 3.54e+04 | 8.21e+02 | 0.00e+00 | 1.10e+01(+) | 1.69e+01(+) |
| w1a-2477-300 | 8.8e+04 | 100 | 3.43e+03 | 1.06e+05 | 8.92e+03 | 6.89e+03 | 0.00e+00 | 0.00e+00(+) | 0.00e+00(+) |
| TDT2-500-1000 | 1.0e+05 | 100 | 6.78e+03 | 7.04e+05 | 4.45e+00 | 4.45e+00 | 0.00e+00 | 5.55e+03(+) | 2.74e-03(+) |
| 20News-8000-1000 | 1.0e+05 | 100 | 3.61e+03 | 1.38e+06 | 9.26e-01 | 9.26e-01 | 0.00e+00 | 3.53e+03(+) | 1.41e-03(+) |
| sector-6412-1000 | 1.0e+05 | 100 | 4.10e+03 | 1.28e+06 | 7.45e+01 | 7.45e+01 | 0.00e+00 | 3.40e+03(+) | 3.12e-01(+) |
| E2006-2000-1000 | 1.0e+05 | 100 | 4.13e+03 | 7.75e+05 | 6.72e-01 | 6.72e-01 | 0.00e+00 | 4.13e+03(+) | 4.60e-04(+) |
| MNIST-60000-784 | -1.7e+05 | 100 | 5.23e+04 | 6.81e+05 | 1.02e+05 | 2.72e+05 | 3.46e+03 | 0.00e+00(+) | 5.25e+03(+) |
| Gisette-3000-1000 | -9.9e+05 | 100 | 5.98e+04 | 1.91e+06 | 4.30e+05 | 5.83e+05 | 2.23e+04 | 0.00e+00(+) | 3.06e+05(+) |
| CnnCaltech-3000-1000 | 9.8e+04 | 100 | 7.62e+02 | 1.31e+06 | 1.39e+03 | 9.93e+02 | 0.00e+00 | 1.41e+00(+) | 1.23e+00(+) |
| Cifar-1000-1000 | -1.9e+03 | 100 | 6.77e+02 | 1.48e+06 | 8.63e+04 | 8.63e+04 | 8.25e+04 | 0.00e+00(+) | 8.25e+04(+) |
| randn-500-1000 | 7.2e+04 | 100 | 2.91e+03 | 1.35e+06 | 2.22e+05 | 2.74e+03 | 0.00e+00 | 6.90e+02(+) | 3.33e+00(+) |
| $\lambda$ = 1000.00, time limit=60 |  |  |  |  |  |  |  |  |  |
| w1a-2477-300 | 7.0e+03 | 10 | 3.03e+02 | 2.58e+03 | 2.59e+03 | 2.59e+03 | 0.00e+00 | 0.00e+00(+) | 0.00e+00(+) |
| TDT2-500-1000 | 1.0e+04 | 10 | 1.24e+03 | 1.98e+03 | 3.40e+00 | 3.40e+00 | 0.00e+00 | 5.19e+02(+) | 3.09e-02(+) |
| 20News-8000-1000 | 1.0e+04 | 10 | 8.94e+02 | 1.32e+04 | 1.49e+00 | 1.49e+00 | 0.00e+00 | 3.19e+02(+) | 3.07e-02(+) |
| sector-6412-1000 | 1.0e+04 | 10 | 2.88e+01 | 1.18e+04 | 3.00e+01 | 3.00e+01 | 0.00e+00 | 1.29e-01(+) | 3.68e-01(+) |
| E2006-2000-1000 | 1.0e+04 | 10 | 3.87e+02 | 5.62e+03 | 6.18e-01 | 6.18e-01 | 0.00e+00 | 3.87e+02(+) | 1.59e-03(+) |
| MNIST-60000-784 | -2.2e+05 | 10 | 1.71e+02 | 2.27e+05 | 2.27e+05 | 2.27e+05 | 1.16e+04 | 0.00e+00(+) | 8.33e+03(+) |
| Gisette-3000-1000 | -1.0e+06 | 10 | 2.62e+04 | 3.30e+05 | 6.93e+04 | 2.65e+05 | 0.00e+00 | 1.97e+04(+) | 4.01e+04(+) |
| CnnCaltech-3000-1000 | 9.6e+03 | 10 | 1.06e+02 | 1.29e+04 | 1.19e+03 | 2.90e+02 | 0.00e+00 | 0.00e+00(+) | 3.80e+00(+) |
| Cifar-1000-1000 | -9.2e+04 | 10 | 1.89e+03 | 1.30e+04 | 0.00e+00 | 1.00e+05 | 9.99e+04 | 1.80e+03(+) | 9.99e+04(+) |
| randn-500-1000 | 7.0e+03 | 10 | 3.66e+02 | 1.44e+04 | 3.09e+04 | 5.11e+02 | 0.00e+00 | 8.04e+00(+) | 4.69e+00(+) |
| w1a-2477-300 | 1.5e+04 | 20 | 5.30e+02 | 4.03e+03 | 3.90e+03 | 3.90e+03 | 0.00e+00 | 0.00e+00(+) | 0.00e+00(+) |
| TDT2-500-1000 | 2.0e+04 | 20 | 1.40e+03 | 1.19e+04 | 3.75e+00 | 3.75e+00 | 0.00e+00 | 9.82e+02(+) | 1.10e-02(+) |
| 20News-8000-1000 | 2.0e+04 | 20 | 1.45e+03 | 5.00e+04 | 1.66e+00 | 1.66e+00 | 0.00e+00 | 1.45e+03(+) | 1.08e-02(+) |
| sector-6412-1000 | 2.0e+04 | 20 | 6.05e+02 | 3.08e+04 | 4.27e+01 | 4.27e+01 | 0.00e+00 | 1.37e+00(+) | 1.19e+00(+) |
| E2006-2000-1000 | 2.0e+04 | 20 | 1.65e+03 | 1.64e+04 | 6.38e-01 | 6.38e-01 | 0.00e+00 | 7.18e+02(+) | 1.70e-03(+) |
| MNIST-60000-784 | -2.2e+05 | 20 | 3.92e+03 | 2.41e+05 | 2.41e+05 | 2.41e+05 | 1.17e+04 | 0.00e+00(+) | 2.85e+04(+) |
| Gisette-3000-1000 | -1.0e+06 | 20 | 2.03e+04 | 4.31e+05 | 1.22e+05 | 4.61e+05 | 0.00e+00 | 7.84e+03(+) | 1.07e+05(+) |
| CnnCaltech-3000-1000 | 1.9e+04 | 20 | 2.02e+02 | 3.98e+04 | 8.74e+02 | 3.90e+02 | 0.00e+00 | 1.09e+00(+) | 4.63e+00(+) |
| Cifar-1000-1000 | -8.1e+04 | 20 | 1.55e+02 | 4.42e+04 | 1.04e+05 | 9.83e+04 | 9.72e+04 | 0.00e+00(+) | 9.72e+04(+) |
| randn-500-1000 | 1.4e+04 | 20 | 4.49e+02 | 4.65e+04 | 3.54e+04 | 8.21e+02 | 0.00e+00 | 1.10e+01(+) | 1.69e+01(+) |
| w1a-2477-300 | 8.8e+04 | 100 | 3.43e+03 | 4.16e+04 | 8.92e+03 | 6.89e+03 | 0.00e+00 | 0.00e+00(+) | 0.00e+00(+) |
| TDT2-500-1000 | 1.0e+05 | 100 | 6.78e+03 | 4.62e+05 | 4.45e+00 | 4.45e+00 | 0.00e+00 | 5.20e+03(+) | 4.80e-03(+) |
| 20News-8000-1000 | 1.0e+05 | 100 | 3.61e+03 | 9.50e+05 | 9.26e-01 | 9.26e-01 | 0.00e+00 | 3.53e+03(+) | 1.43e-03(+) |
| sector-6412-1000 | 1.0e+05 | 100 | 4.10e+03 | 8.69e+05 | 7.45e+01 | 7.45e+01 | 0.00e+00 | 3.40e+03(+) | 2.97e-01(+) |
| E2006-2000-1000 | 1.0e+05 | 100 | 4.13e+03 | 4.91e+05 | 6.72e-01 | 6.72e-01 | 0.00e+00 | 4.13e+03(+) | 4.94e-04(+) |
| MNIST-60000-784 | -1.8e+05 | 100 | 5.53e+04 | 4.36e+05 | 1.06e+05 | 2.75e+05 | 0.00e+00 | 2.94e+03(+) | 8.25e+03(+) |
| Gisette-3000-1000 | -9.9e+05 | 100 | 5.98e+04 | 1.45e+06 | 4.30e+05 | 6.84e+05 | 1.43e+04 | 0.00e+00(+) | 3.07e+05(+) |
| CnnCaltech-3000-1000 | 9.8e+04 | 100 | 7.63e+02 | 9.73e+05 | 1.39e+03 | 9.93e+02 | 0.00e+00 | 1.58e+00(+) | 1.62e+00(+) |
| Cifar-1000-1000 | -1.9e+03 | 100 | 6.75e+02 | 1.16e+06 | 8.63e+04 | 8.63e+04 | 8.25e+04 | 0.00e+00(+) | 8.25e+04(+) |
| randn-500-1000 | 7.2e+04 | 100 | 2.92e+03 | 9.34e+05 | 2.22e+05 | 2.75e+03 | 0.00e+00 | 6.92e+02(+) | 5.80e+00(+) |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 1
questions: N/A
soundness: 3
strengths: 1. Algorithm 

The most novel part of their algorithm is that, instead of updating the entire orthogonal matrix $X\in \mathbb{R}^{n\times r}$ (here $n\gg 1$), they update only two chosen rows of $X$ by left-multiplying a suitable $2 x 2$ orthonormal matrix $V\in St(2,2)$ to those rows. The action of a large orthogonal matrix restricted on two rows does not need to be orthonormal. However, the authors make this simplifying assumption and shows that this does not loose any generality as far as first-order convergence is concerned. I think this is a nice observation. Practically, this brings down the per-iteration computational complexity to $O(n)$, which could be much less than the cost of computing the Riemannian gradient of some loss functions. The authors give an extensive details on how to solve the sub-problem involving $St(2,2)$ to make this approach executable. This algorithm could be quite practical for large-scale problems on Stiefel manifolds. 

2. Optimality measure 

The authors introduce a new measure of first-order optimality that is in general stronger than the usual (Riemannian) stationary points (block-k stationary points). Roughly speaking, the sub-problem should not be improved from the current parameter by an arbitrary adjustment using the 2 x 2 frame. This measure comes directly from the sub-problem of minimizing a majorizing surrogate over $St(2,2)$, so it depends on the particular surrogate $\mathcal{K}$ chosen. This is a somewhat unpleansent feature of their proposed optimality measure, which should in principle depend only on the information about the objective function at the given parameter (e.g., Riemannian subgradient norm of the objective). Nonetheless, the authors provide some examples (in Appendix C.1) that the number of critical points can be strictly larger than the number of their block-k stationary points. 

3. Iteration complexity 

The authors show that an epsilon-relaxation of their block-k stationarity decays is obtained after $O(\eps^{-1})$ iterations, matching  standard iteration complexity results in nonconvex optimization literature. 

4. Improved iteration complexity under KL condition: 

The authors show finite-length property (in Thm. 4.8)  under additional assumption. 

5. Extensive computation and examples for the problems on Stiefel manifold. 

The authors provide nice lemmas that are based on extensive computations using properties of Stiefel manifold along with numerical examples.
summary: The authors study a the problem of minimizing a particular class nonconvex nonsmooth function $f$ over the space of n x r orthogonal matrices (a Stiefel manifold $St(n,r)$ ). Three standard approaches for such problems are as follows: (1) Use gradient descent-type updates in the ambient Euclidean space but use projection onto $St(n,r)$;  (2) Make gradient descent update within the tangent space and retract onto the manifold, see [2]; and (3) Iteratively construct a majorizing surrogate $\hat{f}$ of the objective $f$ and minimize it within the manifold, see [1,3]. The last approach has been particularly successful for optimization problems on Stiefel manifolds, which is the topic of the current paper, see [1]. Due to the non-convex nature of both the objective and the constraint set, researchers typically aim at obtaining first-order convergence guarantees (asymptotically or non-asymptotically with sub-linear rate), suitably defined. Convergence rate can be improved by imposing additional assumptions such as the KL property on the objective. 

The authors use an approach under the umbrella of block majorization-minimization, which progressively minimizes a majorizing surrogate of the block-restricted objective. There proposed method essentially reduces working with high-dimensional (n) orthonormal vectors to only k-dimensional (k fixed, but k=2 seems the only effective case) orthonormal columns, by subsampling k rows at random or according to a pre-determined schedule. The authors derive non-asymptotic iteration complexity to obtain approximate first-order optimal points and obtain improved convergence rate under the additional KL condition. 

There are many existing works on block majorization-minimization in both the Euclidean and the Riemannian settings for nonconvex constrained problems, so the idea itself does not seem very novel. However, there are some interesting ideas and observations that the authors make in this work specifically for Stiefel manifold problems (I will say more on them in Strengths) that can be quite valuable for high-dimensional problems on Stiefel manifolds. But, unfortunately, the paper's presentation is very poor and it cannot be considered for a publication in top ML conferences such as ICLR. The authors fail to give a cohesive narrative of their work, arguments and evidences are fragmented across the main text and the appendix, numerous typos and undefined quantities, and so on (I will say more in the Weaknesses). I think the paper has a valuable contribution, but it needs a significant rewriting from ground-up. I will need to see the whole revised paper so that these issues are successfully addressed, which definitely goes beyond that is expected to be done during the rebuttal. Therefore, overall I recommend a rejection of the paper. 


[1] Breloy, Kumar, Sun, and Palomar, "Majorization-Minimization on the Stiefel Manifold With Application to Robust Sparse PCA", IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 69, 2021

[2] Yuchen Li, Laura Balzano, Deanna Needell, Hanbaek Lyu, “Convergence and Complexity Guarantee for Inexact First-order Riemannian Optimization Algorithms.” ICML 2024 

[3] Yuchen Li, Laura Balzano, Deanna Needell, Hanbaek Lyu, “Convergence and complexity of block majorization-minimization for constrained block-Riemannian optimization” arXiv:2312.10330 (2023)
weaknesses: 1. Presentation: 

The paper does not read very well. This is partly because of the style of the paper writing, which puts most of texts within proposition, lemma, remarks and there are not much narratives that connects them. There are numerous places that quantities are not properly defined. Important equations are written in line within text, which makes it extremely hard to read. Space constraints granted, there are still well-written, easy-to-read theoretical papers in top ML conferences. The current manuscript looks like it is a math paper crammed into 10 pages without much effort to make it readable.

(1) In the very first paragraph, the authors state assumptions in text. This is extremely hard to read and almost impossible to refer back when reading further into the paper. 

(2) 

2. Assumptions: 

In (2), the authors assume that the objective $f$ admits a quadratic surrogate of very specific form. This looks like some kind of restricted smoothness property but with the inner product depending on an additional kernel $$H$$. The authors should discuss if this is a reasonable assumption (e.g., if $f$ is $L$-smooth in the Euclidean or Riemannian sense, is it satisfied?) Otherwise, it is unclear if the paper's framework applies beyond a handful of objective functions (essentially $|| X - B||_{F}^{2}$ plus some nonsmooth regularization terms) and this makes the paper's scope very narrow. 

3. Optimality measure: 

While the optimality measure based on the authors' notion of block-k stationary points is interesting, it is hard to assess how it compares with the usual optimality measure based on (sub)gradient norms or the ones that directly relaxes the variational inequality defining stationary points. Also, another complication is that this measure involves the chosen surrogate as part of the definition, so it is not clear that if one has small BS_k measure then one has small subgradient norm or other standard stationarity measure. 

Furthermore, the authors does not show asymptotically this proposed measure goes to zero. If this result is given, then using their hierarchy between optimality measures, one can deduce that any limit point of the algorithm is a BS_k-point, which is stationary point with additional potentially desirable properties. 

4. Analysis assumes exact solver for the sub-problems: 

While in Remark 2.5 the authors claim that "For general k and h(·), the subproblem may not be solved globally, but a critical point can still be reached. Although strong optimality may be lost, a critical point (discussed later) for the final configuration X∞ remains achievable." This claim is never justified in the paper. First, even with exact computation of sub-problems, they do not show that the iterates asymptotically converge to the stationary points. Second, to support their claim, the authors extend their analysis for Thm. 4.2 with the first point here under the assumption that the sub-problems are inexactly solved. Justifying this is in fact is a non-trivial problem. See [1] for such a result for block inexact Riemannian coordinate descent. 

[1] Yuchen Li, Laura Balzano, Deanna Needell, Hanbaek Lyu, “Convergence and Complexity Guarantee for Inexact First-order Riemannian Optimization Algorithms.” ICML 2024 
 
5. Analysis under KL:

Thm. 4.8 states that essentially the iterates converge after a finite number of iterations under the additional KL assumption. The authors do not specify the the exponent $\sigma\in [0,1)$ in the desingularization function $\varphi$. This cannot be true for all range of $\sigma$ and it should only hold when $\sigma=0$. See Thm. 2.9 in [1]. 

[1] Xu, Yangyang, and Wotao Yin. "A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion." SIAM Journal on imaging sciences 6.3 (2013): 1758-1789.

5. Minor comments: 

(1) Below (2), the norm on $H$: is it the spectral norm? 
(2) L47: $\lVert X \rVert_{p}$ is not coordinate-wise separable. 
(3) L88: The authors imply that subgradient descent with dimihsing stepwise is a limitation. If the stepsizes diminish at a rate not too fast (e.g., $O(n^{-1/2})$), it still give an optimal iteration complexity. Please specify. 
(4) The authors are recommended to cite the following references and discuss their relevance/differences. 

[1] Breloy, Kumar, Sun, and Palomar, "Majorization-Minimization on the Stiefel Manifold With Application to Robust Sparse PCA", IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 69, 2021
[2] Yuchen Li, Laura Balzano, Deanna Needell, Hanbaek Lyu, “Convergence and Complexity Guarantee for Inexact First-order Riemannian Optimization Algorithms.” ICML 2024 
[3] Yuchen Li, Laura Balzano, Deanna Needell, Hanbaek Lyu, “Convergence and complexity of block majorization-minimization for constrained block-Riemannian optimization” arXiv:2312.10330 (2023)

(5) Summary: The authors state that there are not much known results about block marjoziation-minimization or similar type of algorithms for nonconvex nonsmooth problems. This is simply not true. Please revise this assessment.
(6) In Lemma 2.1, the authors define $X^{+}$ and then specify $X$. They should first say "For any $X$, define $X^{+}=$.." 
(7) In Algorithm 1, $\mathcal{K}$ is not defined nor referred.  
(8) In eq. (6), remove comma. 
(9) The construction of quadratic surrogate in (9) resembles very much the discussion in Breloy et a. [1]. The authors should discuss the relation. 
(10) L215: The authors claim that the so-constructed quadratic surrogate can be exactly minimized over $St(k,k)$. But this is the case only if the nonsmooth part is zero (by SVD) or if $k=2$. This should be clearly stated. 
(11) Table 1: The numbers are too small to read. There is no point of adding this kind of table since it is unreadable anyways. 
(12) Appendix C.3: The authors compared computational complexity of computing the Riemannian gradient directly v.s. incrementally updating the gradient using their k-row approach.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: * What is the impact of different strategies for choosing $B_k$?
* How is the problem in (10) solved when $Q$ is not diagonal?
* In the proof of the convergence Theorem 4.2, in line 1495, how come you can upper bound the inequality with the stationary point of the algorithm?
soundness: 2
strengths: I find the concept of updating only $k$-selected rows in the composite optimization over the Stiefel manifold to be an important and useful problem. Having good algorithms in this setting allows for all kind of additional applications, including distributional and privacy aware settings. The idea of the algorithm is straightforward to understand. The authors also focus on $k=2$ case when the iteration subproblems have closed form solution and connect these with classical notions of Givens rotations and Jacobi reflections.
summary: The paper is concerned with solving a composite optimization problem, whose objective is the sum of a smooth and non-smooth function, over the set of $n\times r$ orthogonal matrices, called the Stiefel manifold. The main innovation of the proposed work is to consider a form of **block-coordinate update**, in which only some, $k\leq n$, rows are updated. 

Having a method that updates only $k$-rows would have great computationally practical advanteges: 1) the gradient of the function does not have to be evaluated on all of the $nr$ entries, 2) only $kr$ entries have to be updated, 3) data can be distributed based on the updated blocks. Moreover, if the update consists of only orthogonal rotation by a $k\times k$ on the selected columns, the update automatically belongs to the feasible set.

The main contributions of the paper are:
* the concept of $k$-row update,
* strong limit point convergence of the method under KL-inequality
* greedy strategies for selecting the $k$ indices which are to be updated
weaknesses: While I enjoyed reading the paper, there are several points that are concerning to me.

**Convoluted presentation of some results**
At some points in the paper, there are basic linear algebra concepts presented in an unusual convoluted way. For example the start of Section 2, lines 136 to 186 basically say, that if we have an update $X^+ = V_B X$, where the matrix $V_B$ is an identity apart from a diagonal block indexed by $B$ where it has an orthogonal $k\times k$ matrix, then the next iterate will satisfy $(X^+)^\top X^+ = X^\top V_B^\top V_B X = X^\top X$ and that the distance will depend on the magnitude of the rotation $V_B$. However, since the authors write the update in an unusual form in eq. (4), this fact is, in my opinion, less apparent, than by using orthogonality of $V_B$.

**Inaccuracies**
Similarly, there is a discussion of using not only Givens rotations, stating the advantage of using Jacobi reflections since it will generate whole Stiefel manifold. This is also not entirely accurate I think. Givens rotations generate the whole SO(n) which is sufficient in solving PCA since there is an inherent ambiguity in the solution up to $\pm1$ sign anyway.

**Other minor typos/writing fixes**
* Another difficulty I had was understanding the notation, for example: $[\cdot]_BB$ appearing for the first time in line 199, becomes slightly clear only later in line 224. Overall, I would also suggest to bring more focus on the computational advantages of the formulation.
* The notation of $\mathcal{M}$ in line 187 should be Stiefel manifold I think.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. In Figure 1, the objective function values of other methods seem unchanged. Could the authors explain the reason and elaborate more about the performance of other methods? Besides, sharper notions of stationarity do not necessarily lead to lower objective function values. It can be better to present some average results based on multiple tests.
2. Could the authors add some discussion on the possible parallel implementation of OBCD? Or explain possible difficulties of parallel implementation. Because this aspect would greatly strengthen the practical popularity of OBCD.
3. From Proposition 4.6, the KL holds locally. To apply the KL property, it is required that the iterates stay in the local region. However, the randomness in the algorithm can make the iterates fail to stay in the local region. Could the authors explain how to overcome this difficulty?
4. Could the authors revise the presentation and flow of this paper to make it more readable?
5. Could the authors add some discussion on how this BCD framework can be generalized to other manifold?
7. There are also other papers that tackle nonsmooth manifold opt using BCD (e.g., https://arxiv.org/pdf/2409.01770). Could the authors add some related references?
soundness: 3
strengths: 1. This paper proposes a very interesting row-wise BCD method for composite optimization over orthogonality constraints, which enables low-cost update.
2. The authors show that their algorithm converges to a BSk-point, which is sharper than usual critical points. This is a great point.
3. The proposed algorithm outperforms other existing methods in various problems.
summary: The paper presents OBC to tackle nonsmooth composite optimization with orthogonality constraints. OBCD updates multiple rows of the matrix variable while globally solving a small nonsmooth optimization problem. The authors demonstrate that OBCD converges to block-k stationary points, providing stronger optimality guarantees than traditional critical points, and is the first greedy descent method in this domain to exhibit monotonicity. They establish strong limit-point convergence under the Kurdyka-Lojasiewicz inequality, solve subproblems with breakpoint searching techniques, and provide greedy strategies to improve computational efficiency. Extensive experiments validate OBCD's superior performance in both accuracy and efficiency compared to existing methods.
weaknesses: 1. The paper is not well-written. The organization requires a thorough revising to make this paper more readable.
2. The Asm-iii in Line 49 is restrictive. As the problem can be hard to solve. This can hinder the practical meaning of the theoretical guarantee. The authors propose to use the majorization surrogate and minimize the majorization surrogate only in k=2. Did the authors considered solving subproblems inexactly?

Meta-review: This paper studies block coordinate descent method (BCD) for nonsmooth composite optimization with orthogonality constraints. The authors proposed a new method that minimizes a majorizing surrogate of the block restricted objective, which is obtained by sampling k rows and formulating the subproblem accordingly. Though there are some interesting ideas, the paper is not well written, making it hard to read and difficult to verify the results and claims.

Decision: Reject