## 4 Optimization for ISMLP
More specifically, for the \(k\)-th proxy vector \(\boldsymbol{z}_{k}\), by taking the derivative of \(\mathcal{G}\) with respect to \(\boldsymbol{z}_{k}\), we can get:  
\[ \begin{split}&\frac{\partial\mathcal{G}}{\partial\boldsymbol{z}_{k }}=\frac{1}{|\mathcal{S}|}\left(\sum\limits_{(x_{i},x_{j})\in\mathcal{S}}\frac {1}{q_{ip}(x_{i})}\frac{\partial\bar{q}_{ip}(x_{i})}{\partial\boldsymbol{z}_{k }}+\sum\limits_{(x_{i},x_{i})\in\mathcal{S}\atop p(x_{i})=x_{i}}\frac{1}{q_{ik }}\frac{\partial\bar{q}_{ik}}{\partial\boldsymbol{z}_{k}}\right)\\ &-\frac{\lambda}{n_{u}}\sum\limits_{i=n_{l}+1}^{n}\left(\sum \limits_{j\
eq k}\left((1+\text{log}q_{il})\frac{\partial q_{il}}{\partial \boldsymbol{z}_{k}}\right)+(1+\text{log}q_{ik})\frac{\partial q_{ik}}{\partial \boldsymbol{z}_{k}}\right)\end{split}  \tag{25}\]  
where for the given similar pair \(\big{(}\boldsymbol{x}_{i},\boldsymbol{x}_{j}\big{)}\), \(q_{ip(x_{i})}\) denotes the probability of \(\boldsymbol{x}_{i}\) choosing \(p\big{(}\boldsymbol{x}_{j}\big{)}\) as the proxy vector computed by Equation (6). \(\frac{\partial\bar{q}_{ik}}{\partial\boldsymbol{z}_{k}}=2(1-q_{ik})\mathbf{M} (\boldsymbol{x}_{i}-\boldsymbol{z}_{k})\), and \(\frac{\partial\bar{q}_{il}}{\partial\boldsymbol{z}_{k}}=\frac{-2q_{il}}{\sum \
olimits_{j=1}^{m}\text{exp}\big{(}-d_{\mathbf{M}}^{2}\big{(}\boldsymbol{x}_{ i},\boldsymbol{z}_{j}\big{)}\big{)}}\mathbf{M}(\boldsymbol{x}_{i}- \boldsymbol{z}_{k})\). By setting \(\frac{\partial\mathcal{G}}{\partial\boldsymbol{z}_{k}}\) to zero, Equation (25) is simply a linear equation:  
\[w\mathbf{M}\boldsymbol{z}_{i}=\boldsymbol{\psi} \tag{26}\]  
where  
\[ \begin{split} w=&\frac{1}{|\mathcal{S}|}\Bigg{(}\sum \limits_{(x_{i},x_{j})\in\mathcal{S}\atop p(x_{i})\
eq x_{k}}\frac{1}{\sum \limits_{j=1}^{m}\text{exp}\big{(}-d_{\mathbf{M}}^{2}\Big{(}x_{i},\boldsymbol{z }_{j}\big{)}\big{)}}-\\ &\sum\limits_{(x_{i},x_{j})\in\mathcal{S}\atop p(x_{i})=x_{k}} \frac{1-q_{ik}}{q_{ik}}\Bigg{)}-\frac{\lambda}{n_{u}}\Bigg{(}\sum\limits_{i=n_ {l}+1}^{n}\left(1+\text{log}q_{ik}\right)\frac{1-q_{ik}}{q_{ik}}\\ &+\sum\limits_{l\
eq k}(1+\text{log}q_{il})\frac{q_{il}}{\sum \limits_{j=1}^{m}\text{exp}\big{(}-d_{\mathbf{M}}^{2}\Big{(}\boldsymbol{x}_{i}, \boldsymbol{z}_{j}\big{)}\big{)}}\Bigg{)},\end{split}  \tag{27}\]and \(\mathbf{\psi}\):  
\[\mathbf{\psi}= \frac{\mathbf{M}}{|\mathcal{S}|}\left(\sum_{ \begin{subarray}{c}(\mathbf{x }_{i},\mathbf{x}_{j})\in\mathcal{S}\\ p(\mathbf{x}_{j})\
eq\mathbf{z}_{k}\end{subarray} }\frac{-\mathbf{x}_{i}}{\sum_{j=1}^{m} \exp\left(-d_{\mathbf{M}}^{2}\left(\mathbf{x}_{i},\mathbf{z}_{j}\right)\right)}+\right.\] \[\sum_{ \begin{subarray}{c}(\mathbf{x}_{i},\mathbf{x}_{j})\in\mathcal{S}\\ p(\mathbf{x}_{j})=\mathbf{z}\end{subarray} }\frac{1-q_{ik}}{q_{ik}}\mathbf{x}_{i}\right)+ \frac{\lambda}{n_{u}}\left(\sum_{i=n_{t}+1}^{n}\left(1+\log q_{ik}\right)\frac{ 1-q_{ik}}{q_{ik}} \tag{28}\] \[-\sum_{i\
eq k}(1+\log q_{il})\frac{q_{il}}{\sum_{j=1}^{m}\exp \left(-d_{\mathbf{M}}^{2}\left(\mathbf{x}_{i},\mathbf{z}_{j}\right)\right)}\right) \mathbf{M}\mathbf{x}_{i},\]  
we can get the closed-form solution of \(\mathbf{z}_{k}\):  
\[\mathbf{z}_{k}=(\mathbf{M}+\eta\mathbf{I}_{d})^{-1}\frac{\mathbf{\psi}}{w} \tag{29}\]  
where \(\eta>0\) is a small positive number to make \(\mathbf{A}\) a positively defined matrix. In the experiment, we empirically set it as \(1\mathrm{e}^{-6}\), which works fine. To sum up, we propose an alternating direction strategy to solve the minimization of Equation (11). The sub-problems concerning \(\mathbf{P}\) and \(\mathbf{R}\) are updated on the product manifold of the Stiefel and SPD manifold via the Riemannian gradient descent algorithm [41]. The sub-problem concerning \(\mathcal{Z}\) has a closed-form solution. The main procedure of ISMLP is documented in Algorithm 1. It should be noted that we utilize a Gaussian Mixture Model (GMM) to initialize the set of proxy vectors which are the means of the corresponding components.