Package: pystmopt
PySTOPIntroductionThe STOP toolbox is designed foroptimization problems on the Stiefel manifold, which could be expressed as
$$
\begin{aligned}
\min_{X \in \mathbb{R}^{n\times p}} ~ &f(X)\
\text{s. t.}~& X^\top X = I_p,
\end{aligned}
$$
where $I_p$ refers to the $p$-th order identity matrix, $X$ is a matrix with $n$ rows and $p$ columns. The feasible set of this optimization problem
$$
\mathcal{S}_{n,p} := \left{X \in \mathbb{R}^{n\times p}: X^\top X = I_p \right},
$$
can be regarded as a Riemannian manifold in $\mathbb{R}^{n\times p}$, and we also call it asStiefel manifold.This document describes the python version of SLPG solver.InstallationThe source code of those solvers can be found fromthe websiteor directly fromthis link. Besides, it supports direct installation frompip:pipinstall--userpystopExampleProblem formulationIn this section, we consider the following nonlinear eigenvalue problem
$$
\min_{X \in \mathcal{S}_{n, p}} ~ \frac{1}{2}\mathrm{tr}(X^\top L X) + \frac{\alpha}{4} \rho^\top L^{\dagger} \rho,
$$
where $\rho = \mathrm{Diag}(XX^\top)$, and $L^{\dagger}$ denotes the pseudo-inverse of the positive definite matrix $L$, i.e. $L^{\dagger}LL^{\dagger} = L^{\dagger}$, $LL^{\dagger}L = L$.  Here we uses $\mathrm{Diag}(M)$ to denote the vector that is composed of diagonal entries of the square matrix $M$, while $\mathrm{diag}(v)$ refers to a diagonal matrix with $v$ to be its diagonal entries. Then the cost function and itsEuclidean gradientcan be expressed as
$$
\begin{aligned}
f(X) ={}& \frac{1}{2}\mathrm{tr}(X^\top L X) + \frac{\alpha}{4} \rho^\top L^{\dagger} \rho,\
\nabla f(X) ={}& LX + \alpha \mathrm{diag}(L^{\dagger}\rho)X.
\end{aligned}
$$In this example, we choose $L$ as a tri-diagonal matrix generated byL = gallery('tridiag',n,-1,2,-1). Noting that $L$ is full-rank, then we can conclude that $L^{\dagger} = L^{-1}$ in this case. We solve this simple optimization problem using solvers in STOP to illustrate the most basic usage of the STOP toolbox. For additional theory, readers are recommended to refer the papers in the about page.# Import packagesimportnumpyasnpimportscipyasspfromscipy.sparseimportdiagsfromscipy.sparse.linalgimportspsolve# Import manifolds and solversfrommanifoldimportStiefelfromSolver.SLPGimportSLPG_smooth# Set parametersn=1000p=10alpha=1M=Stiefel(n,p)# Defining objective functionL=diags(np.array([-1,2,-1]),np.array([1,0,-1]),shape=(n,n)).tocsc()defobj_fun(X):LX=L@Xrho=np.sum(X*X,1)Lrho=spsolve(L,rho)fval=0.5*np.sum(X*LX)+(alpha/4)*np.sum(rho*Lrho)grad=LX+alpha*Lrho[:,np.newaxis]*Xreturnfval,grad# Execute the solverX,out_dict=SLPG_smooth(obj_fun,M)Let us review the code step by step. First, we specify the dimension of the problem and specify the Stiefel manifold. InpySTOPpackage, we need to specify the dimension of the Stiefel manifold before executing the solver. The Stiefel manifold should be specified as theSTOP manifold class, for example,# Set parametersn=1000p=10alpha=1# Specify the Stiefel manifoldM=Stiefel(n,p)Herepystop.manifold.stiefelis a build-in function to specify the Stiefel manifold and hence provides essential tools for the algorithm.Then we generate the data (matrix $L$) for the optimization problem by the following code,L=diags(np.array([-1,2,-1]),np.array([1,0,-1]),shape=(n,n)).tocsc()Here we utilizeSciPy.sparseto create a sparse representation of $L$ . Therefore, in each step  we could use thescipy.sparse.linalg.spsolvefunction to compute .Then we specify the cost function and its gradient in the following function# Defin objective functiondefobj_fun(X):LX=L@Xrho=np.sum(X*X,1)Lrho=spsolve(L,rho)fval=0.5*np.sum(X*LX)+(alpha/4)*np.sum(rho*Lrho)grad=LX+alpha*Lrho[:,np.newaxis]*Xreturnfval,gradCurrently, in STOP toolbox, we require the function return the function value and its gradient simultaneously. Usually, computing the function value and gradient simultaneously is much faster than compute them separately, even when cache techniques are involved. To achieve a better performance, we strongly suggest to compute the function value and gradient in a single function.Then we call a solver to solve the nonlinear eigenvalue problem,# Execute the solverX,out_dict=SLPG_smooth(obj_fun,M)SolversThePySTOPsolver classes provide the solvers for optimization. Once we specify the Stiefel manifold and define the objective function, thePySTOPsolver can be executed byX, out_dict = SLPG_smooth(obj_fun, M)HereXis the final output of the problem, andout_dictis a dictionary that contains the log information.NameCommentCallSLPG_smoothPenalty-free first-order method for smooth problemsSLPG_smooth(...)SLPGPenalty-free first-order method for nonsmooth problemsSLPG(...)SLPG_l21Penalty-free first-order method for $\ell_{2,1}$-norm regularized problemsSLPG_l21(...)