# Extrinsic Bayesian Optimization on Manifolds
## 4 Examples and Applications
### Invariant Subspace Approximation on Grassmann Manifolds  
We investigate two important manifolds, the Stiefel manifolds and the Grassmann manifolds (Grassmannians). The Stiefel manifold is the collection of \(p\) orthonormal frames in \(\mathbb{R}^{n}\), that is, \(Stiefel(p,n)=\{X\in\mathbb{R}^{n*p}:X^{T}X=I_{p}\}\). Moreover, the Grassmann manifold is the space of all the subspaces of a fixed dimension \(p\) whose basis elements are \(p\) orthonormal unit vectors in \(\mathbb{R}^{n}\), which is \(Grass(p,n)=\{span(X):X\in\mathbb{R}^{n*p},X^{T}X=I_{p}\}\). Those two manifolds are closely related. The key difference between a point on the Grassmannian and a point on the Stiefel manifold is that the ordering of the \(p\) orthonormal vectors in  
Figure 3: We compare the eBO method with the gradient descent (GD) method on the sphere by calculating the log error, the L2 distance from the true extrinsic mean. Although the eBO method converges slower than the GD method in the first four steps, by adding those stepping points into the eGP, it achieves better numerical precision with more steps. Figure 2: The iteration steps of the eBO method on the sphere \(5^{2}\). The data points \(x_{1},...,x_{n}\) are plotted as those black points on the same latitude of the sphere. The true extrinsic mean is the south pole \((0,0,-1)\). We start with those random blue points on the sphere. The outputs of iterations in our Algorithm 1 are marked as red points on the sphere, converging to the ground truth. does not matter for the former. In other words, the Grassmannian could be viewed as the quotient space of the Stiefel manifold modulo \(O(p)\), the \(p*p\) orthogonal group. That is, \(Grass(p,n)=Stiefel(p,n)/O(p)\). We consider the matrix approximation problem on the Grassmannian subspace manifold and apply the eBO method to solve it. Given a full rank matrix \(F\in\mathbb{R}^{n*m}\), without loss of generality, we assume \(n<=m\) and \(rank(F)=n\), the goal is to approximate this matrix \(F\) in the subspace \(Grass(p,n)\) with \(p<m\). From any matrix \(X\in Grass(p,n)\), we approximate the original matrix \(F\) from Algorithm 3. ```
Initialize the orthogonal matrix \(X\in Grass(p,n)\);  Initialize the matrix \(W\in\mathbb{R}^{p*m}\) and calculate the column \(W_{j}\in\mathbb{R}^{r*1}\) below; for\(j=1,\dots,m\)do \(W_{j}=(X^{T}X)^{-1}X^{T}F_{j}\), where \(F_{j}\) is the \(j\)th column of \(F\);  Return \(XW\) as the approximation. ```  
**Algorithm 3** Matrix approximation on the Grassmann manifold. Here we consider the approximation error in Frobenius norm where \(W\) depends on \(X\):  
\[X=\arg\min_{X\in Grass(p,n)}\|XW-F\|_{F}, \tag{16}\]  
where the objective function \(f=\|XW-F\|_{F}\). In our simulation, we consider the matrix \(F\in\mathbb{R}^{3*6}\) with \(rank(F)=3\) and \(p=2\). As we know, this matrix approximation problem achieves its minimal when \(X\in span(U_{1},U_{2})\) where \(U\) is the left matrix in the Singular Value Decomposition (SVD) and \(U_{j}\) denotes its \(j\)th column. Let \(\tilde{U}=[U_{1},U_{2}]\) contain the first two columns of the SVD part \(U\). To apply the eBO method, we first sample some matrix following \(X_{i}=\tilde{U}+\frac{i(-1)! }{2}\), \(i=1,...,6\) as initial points. Similar to the sphere case, we have to map the minimizer of the acquisition function back to the manifold via the inverse embedding \(J^{-1}\). In \(Grass(2,3)\), for a given matrix \(X\in\mathbb{R}^{3*2}\), \(J^{-1}\) indicates us to apply the SVD decomposition to \(X^{T}X\), then keep the first two columns. Since the closed form of the loss gradient is untouchable, we compare the BO method with the Nelder-Mead method. As shown in Figure 4, our eBO method converges to the ground truth in a few steps faster than the Nelder-Mead method. Figure 4: We compare the eBO method with the Nelder–Mead method on the matrix approximation problem on the Grassmannian. Since the optimal solution is the whole subspace, we calculated the value of function \(f\) instead of the error.