## 4 Optimization for ISMLP  
There are three types of parameters to be estimated in the objective function of ISMLP. In this section, an alternating-direction technology is proposed to seek a feasible solution. More specifically, we keep the other variables fixed to update the current variable until the stop condition meet. **Fix \(\mathcal{Z}\), to solve \(\mathbf{P}\) and \(\mathbf{R}\)**: the sub-problem concerning \(\mathbf{P}\) and \(\mathbf{R}\) can be expressed as the following Riemannian manifold-based optimization problem. \[ \begin{split}\min_{\mathbf{P},\mathbf{R}}\mathcal{F}(\mathbf{P}, \mathbf{R}|\mathbf{X},\mathbf{M}_{0})\\ \text{s.t. }\mathbf{P}\in\text{St}(p,d),\ \mathbf{R}\in \mathcal{S}_{++}^{p}.\end{split}  \tag{13}\]  
The above minimization problem can be solved via the product space of the Stiefel and SPD manifold. According to [39], the Stiefel and SPD manifolds are locally homogeneous spaces, and their product space should also follow the smoothness and differentiability. Therefore, \(\mathcal{M}_{p}=\text{St}(p,d)\times\mathcal{S}_{++}^{p}\) can contain a Riemannian structural. Theorem 4: _The set \(\Bigl{(}\text{St}(p,d)\times\mathcal{S}_{++}^{p}\Bigr{)}\backslash\mathcal{O} (p)\) with the following equivalence relation_  
\[[(\mathbf{P},\mathbf{R})]\sim\Bigl{\{}\Bigl{(}\mathbf{PQ},\mathbf{Q}^{\mathsf{ T}}\mathbf{R}\mathbf{Q}\Bigr{)},\forall\mathbf{Q}\in\mathcal{O}(p)\Bigr{\}} \tag{14}\]  
_and Riemannian metric_  
\[ \begin{split} g_{(\mathbf{P},\mathbf{R})}((\xi_{\mathbf{P}},\xi_ {\mathbf{R}}),(\zeta_{\mathbf{P}},\zeta_{\mathbf{R}}))=2\text{Tr}\Bigl{(}\xi_{ \mathbf{P}}^{\mathsf{T}}\zeta_{\mathbf{P}}\Bigr{)}+\\ \text{Tr}\Bigl{(}\mathbf{R}^{-1}\zeta_{\mathbf{R}}\mathbf{R}^{-1} \zeta_{\mathbf{R}}\Bigr{)}\end{split}  \tag{15}\]  
_forms a Riemannian quotient manifold._Proof. : We first prove that the equivalence relation hold: \(\mathcal{F}(\mathbf{P},\mathbf{R}|\mathbf{X},\mathbf{M}_{0})=\mathcal{F}\big{(} \mathbf{PQ},\mathbf{Q}^{T}\mathbf{RQ}|\mathbf{X},\mathbf{M}_{0}\big{)}\), since the following equation holds:  
\[\mathbf{PQ}\mathbf{Q}^{T}\mathbf{RQ}(\mathbf{PQ})^{T}=\mathbf{PR}\mathbf{P}^{T},\forall\mathbf{Q}\in\mathcal{O}(p). \tag{16}\]  
Therefore, the equivalence relation holds. To prove \(\mathcal{M}_{p}\backslash\mathcal{O}(p)\) is a valid quotient manifold, one can follow the proof in [40]. Lastly, as for the Riemannian metric, interested readers can refer to [37]. To perform the Riemannian gradient descent on \(\mathcal{M}_{p}\), we usually follow the \"projection and retraction\" procedures. More specifically, firstly, one can transform the Euclidean gradient as the Riemannian gradient, and then perform the gradient descent step; then, map the intermediate solution back to the manifold [39]. For the Stiefel manifold, the Riemannian gradient can be computed as:  
\[\xi_{\mathbf{P}}=\frac{\partial\mathcal{F}}{\partial\mathbf{P}}-\frac{1}{2} \mathbf{P}\Bigg{(}\mathbf{P}^{T}\frac{\partial\mathcal{F}}{\partial\mathbf{P }}+\bigg{(}\frac{\partial\mathcal{F}}{\partial\mathbf{P}}\bigg{)}^{T} \mathbf{P}\Bigg{)}. \tag{17}\]  
As for the PSD manifold, its Riemannian gradient has the following form:  
\[\xi_{\mathbf{R}}=\frac{1}{2}\mathbf{R}\Bigg{(}\frac{\partial\mathcal{F}}{ \partial\mathbf{R}}+\bigg{(}\frac{\partial\mathcal{F}}{\partial\mathbf{M}} \bigg{)}^{T}\Bigg{)}\mathbf{R}, \tag{18}\]  
where \(\frac{\partial\mathcal{F}}{\partial\mathbf{P}}\) and \(\frac{\partial\mathcal{F}}{\partial\mathbf{R}}\) denote the Euclidean partial gradient of \(\mathcal{F}\) w.r.t \(\mathbf{P}\) and \(\mathbf{R}\), respectively. As for the quotient manifold \(\mathcal{M}_{p}\), the tangent space at \(\Gamma=(\mathbf{P},\mathbf{R})\) is divided into two complementary parts, namely a horizontal part \(\mathcal{H}_{\Gamma}\mathcal{M}_{p}\) and a vertical one \(\mathcal{V}_{\Gamma}\mathcal{M}_{p}\). Importantly, the tangent space of \(\mathcal{M}_{p}\) (denoted as \(\mathcal{T}_{\Gamma}\mathcal{M}_{p}\)) can be uniquely identified as it horizontal part.