SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} H \left[ q _ {\phi} (\boldsymbol {Y} \mid \boldsymbol {P}) \right] = \mathbb {E} \left[ - \log q _ {\phi} (\boldsymbol {Y} \mid \boldsymbol {P}) \right] (62) \\ = - \int_ {\mathcal {O} (k)} U _ {\mathcal {O} (k)} (\boldsymbol {Q}) \log U _ {\mathcal {O} (k)} (\boldsymbol {Q}) [ d \boldsymbol {Q} ] (63) \\ = - \int_ {\mathcal {O} (k)} \frac {1}{V _ {\mathcal {O} (k)}} \log \frac {1}{V _ {\mathcal {O} (k)}} [ d Q ] (64) \\ = \frac {\log V _ {\mathcal {O} (k)}}{V _ {\mathcal {O} (k)}} \int_ {\mathcal {O} (k)} [ d Q ] (65) \\ = \frac {\log V _ {\mathcal {O} (k)}}{V _ {\mathcal {O} (k)}}. (66) \\ \end{array}
$$
Expectation of prior distribution We used (9) for the prior distribution  $p_{\theta}(\mathbf{Y})$ . Further, reparameterization was used to enable differentiable Monte Carlo estimation of expectations.
$$
\mathbb {E} _ {q _ {\phi} (\boldsymbol {Y} \mid \boldsymbol {P})} [ p _ {\theta} (\boldsymbol {Y}) ] = \frac {1}{L} \sum_ {l = 1} ^ {L} p _ {\theta} (\boldsymbol {X} \boldsymbol {Q} _ {l}) \quad \text {s . t .} \quad \boldsymbol {X} \sim \boldsymbol {Y}, \boldsymbol {Q} _ {l} \in \mathcal {O} (k), \tag {67}
$$
where it was assumed that  $Q$  is sampled from a uniform distribution, which follows Haar measure on  $\mathcal{O}(k)$ .  $L$  is set  $L = 1$ .
C.3 Implementation Details and Experimental setting
The following sections present more details about the network architectures, training hyperparameters, and experimental conditions for each of the experiments in Section 6.
C.3.1 Artificial Textures
Network Architecture The vector field was constructed with the specific input, intermediate, and output layers described in Section 5.3. The GrCNF architecture is shown on top in Table 3. Layers are denoted as Layer in the table, and were processed from top to bottom. Norm. and Act. denote the normalization and activation functions to be applied immediately after the Layer, and the Norm. and Act. were applied in that order. Further, Out Size denotes the output size after Act. Vec denotes the map of vertically concatenating matrices and converting them into a single vector. Moreover, only row Input denotes the size of the input data, not the input layer (HorP). (9) was used for the loss function.
Hyper-parameters The mean  $M$  and covariances  $\mathbf{U}$  and  $\mathbf{V}$  in the prior distribution on the Grassmann manifold were set to  $M = (1.0, 0.0, 0.0)^{\top}$ ,  $\mathbf{U} = \sigma^{2}\mathbf{I}_{3}$ , and  $\mathbf{V} = \mathbf{I}_{1}$ ,  $\sigma = 0.3$ , respectively. Other hyperparameters used during the training of GrCNF are shown in Table 4.
Implementation We used PyTorch (Paszke et al. (2019)) to implement the model and run the experiments. The CNF is based on the implementation in Chen et al. (2018) and the framework of the RCNF (Mathieu & Nickel (2020)). Thus, the ODE was solved using the explicit and adaptive Runge-Kutta method (Dormand & Prince (1980)) of order 5, and worked by projecting each step onto a manifold (Hairer (2006)). The autograd in (7) was calculated with torch.autograd.grad (Paszke et al. (2017)) in PyTorch. The experimental hardware was built with an Intel Core i7-9700 CPU and a single NVIDIA GTX 1060 GPU with 6 GB of RAM.
The code used in the experiment to generate the data distributions on  $\mathrm{Gr}(1,3)$  is shown in Listing 1. This implementation of the data distributions is based on the codes in Kim et al. (2020) and Grathwohl et al. (2019)<sup>6</sup>.
We used the authors' implementation: https://github.com/rtqichen/torchdiffreq.git.
We used the authors' implementations: https://github.com/ANLGBOY/SoftFlow.git and https://github.com/rtqichen/ffjord.git.
Listing 1: Code for the data distributions.
import numpy as np   
def get_data_batch(batch_size, dist):   
rng = np.random.RandomState()   
if dist == "2spirals":   
n = np.sqrt(np.random.randint(batch_size // 2, 1)) * 540 * (2 * np.pi) / 360   
d1x = -np.cos(n) * n + np.random.randint(batch_size // 2, 1) * 0.1   
d1y = np.sin(n) * n + np.random.randint(batch_size // 2, 1) * 0.1   
x = np.vstack((np.hstack((d1x, d1y)), np.hstack((-d1x, -d1y)))) / 3   
sample_2d = x + np.random.randint(*x.shape) * 0.1   
elif dist == "swissroll": data = sklearn:datasets.make_swiss-roll(n_samples=batch_size, noise=.3)[0] data = data.astype("float32")[:, [0, 2]] sample_2d = data / 5   
elif dist == "2circles": data = sklearn:datasets.make_circles(n_samples=batch_size,  $\begin{array}{rl} & \mathrm{factor} = .5,\mathrm{noise} = 0.05\end{array}$  [0] data = data.astype("float32") sample_2d = data * 3   
elif dist == "2sines": x = (rng.rand(batch_size) - 0.5) * 2 * np.pi u = (rng.binomial(1, 0.5, batch_size) - 0.5) * 2 y = u * np.sin(x) * 2.5 x += np.random.choice(*x.shape) * 0.1 y += np.random.choice(*y.shape) * 0.1 sample_2d = np.stack((x, y), 1)   
elif dist == "target": shapes = np.random.randint(7, size=batch_size) mask = [] for i in range(7): mask.append((shapes == i) * 1.) theta = np.linspace(0, 2 * np.pi, batch_size, endpoint=False) x = (mask[0] + mask[1] + mask[2]) * (rng.rand(batch_size) - 0.5) * 4 +  $-\mathrm{mask}[3] + \mathrm{mask}[4] * 0.0 + \mathrm{mask}[5]) * 2 * \mathrm{np. ones}(\mathrm{batch\_size}) + \backslash \mathrm{mask}[6] * \mathrm{np. cos}(\mathrm{theta})$  y = (mask[3] + mask[4] + mask[5]) * (rng.rand(batch_size) - 0.5) * 4 +  $-\mathrm{mask}[0] + \mathrm{mask}[1] * 0.0 + \mathrm{mask}[2]) * 2 * \mathrm{np. ones}(\mathrm{batch\_size}) + \backslash \mathrm{mask}[6] * \mathrm{np. sin}(\mathrm{theta})$  x += np.random.choice(*x.shape) * 0.1 y += np.random.choice(*y.shape) * 0.1 sample_2d = np.stack((x, y), 1)   
norm = sample_2d / np.max(np.linalgNorm(sample_2d, axis=1)) sample_3d = npcondegree((np.ones((batch_size, 1)), norm), axis=1) return sample_3d / np.linalgNorm(sample_3d, axis=1)[:, np.newaxis]
C.3.2 DW4 and LJ13
Network Architecture As in Appendix C.3.1, the vector field was constructed with the specific input, intermediate, and output layers described in Section 5.3. The GrCNF architecture is shown on the bottom left and right in Table 3. The bottom left and right were used for experiments on the DW4 and LJ13 datasets, respectively. The views presented in the table is the same as in Appendix C.3.1. (9) was used for the loss function. In addition, for architectures in methods other than GrCNF, please refer to Garcia Satorras et al. (2021).
Hyperparameters The mean  $M$  and covariances  $U$  and  $V$  in the prior distribution on the Grassmann manifold were set to  $M = I_{4\times 2}$ ,  $U = \sigma^2 I_4$ , and  $V = \sigma^2 I_2$ ,  $\sigma = 0.3$  for DW4 and  $M = I_{13\times 3}$ ,  $U = \sigma^2 I_{13}$ , and  $V = \sigma^2 I_3$ ,  $\sigma = 0.3$  for LJ13, respectively. Other hyperparameters used during the training of GrCNF are shown in Table 4. In addition, for the hyperparameters in methods other than GrCNF, please refer to Garcia Satorras et al. (2021).
Implementation The experimental hardware was built using a single NVIDIA Quadro RTX 8000 GPU with 48 GB of GDDR6 RAM. The other environments were the same as in Appendix C.3.1.
Table 3: Network architectures for each experiment; (left) for the Simple Texture dataset, (middle) for the DW4 dataset, (right) for the LJ13 dataset.
GrCNF for Textures
Layer	Out Size	Norm./Act.
Input	3×1	-
HorP	3×1	-/Tanh
Vec	3	-/-
CS	64	-/Tanh
CS	64	-/Tanh
CS	1	-/Tanh
Grad	3×1	-
GrCNF for DW4
Layer	Out Size	Norm./Act.
Input	4×2	-
HorP	4×2	-/Tanh
Vec	8	-/-
CS	64	-/Tanh
CS	64	-/Tanh
CS	64	-/Tanh
CS	1	-/Tanh
Grad	4×2	-
GrCNF for LJ13
Layer	Out Size	Norm./Act.
Input	13×3	-
HorP	13×3	-/Tanh
Vec	39	-/-
CS	32	-/Tanh
CS	32	-/Tanh
CS	32	-/Tanh
CS	1	-/Tanh
Grad	13×3	-
Full results In Tables 5 and 6, the same DW4 and LJ13 averaged results from Section 6.2 were reported; however, they included the standard deviations over the three runs.
C.3.3 QM9 Positional
Network Architecture On the QM9 Positional, we addressed the task of generating the molecular  $P$  by estimating the scale parameter  $\sqrt{\Lambda} = \mathrm{diag}\left(\left\{\sqrt{\lambda_i}\right\}_{i=1}^3\right)$ , in addition to the generation of the orthonormal basis matrix  $Y$  with GrCNF. Because the molecular generation task requires a specialized loss function based on variational inference, we used (56), as explained in Appendix C.2. We designed two networks to achieve this. The first is the same GrCNF architecture as in previous experiments, and the second is a scale estimator. Table 7 shows the architectures. The left side of the table shows the GrCNF architecture and the right side shows the scale estimator. The scale estimator estimated one scale parameter from each of the three orthonormal basis vectors  $Y = \left\{\boldsymbol{y}_i \in \mathbb{R}^{19}\right\}_{i=1}^3$ , for a total of three parameters  $\left\{\sqrt{\lambda_i} \in \mathbb{R}\right\}_{i=1}^3$ . With the orthonormal orthogonal basis matrix  $Y$  and the estimated scale parameter  $\sqrt{\Lambda}$ , we generated a point cloud  $P = Y\sqrt{\Lambda}$ . In this study, the overall architecture that generates  $P$  is also named GrCNF.
Hyperparameters The mean  $M$  and covariances  $\mathbf{U}$  and  $\mathbf{V}$  in the prior distribution on the Grassmann manifold were set to  $M = I_{19\times 3}$ ,  $\mathbf{U} = \sigma^2\mathbf{I}_{19}$ , and  $\mathbf{V} = \sigma^2\mathbf{I}_3$ ,  $\sigma = 0.3$ , respectively. In addition, for the hyperparameters in methods other than GrCNF, please refer to Garcia Satorras et al. (2021).
Implementation The experimental hardware was built using a single NVIDIA A100 GPU with 80GB PCIe of GDDR6 RAM.
Table 4: List of hyperparameters used in various experiments. A “-” indicates that the hyperparameter is unused.
Textures	DW4	LJ13	QM9
# of Data	Train	∞	102/103/104/105	10/102/103/104	13,831
Validation	500	1000	1000	2,501
Test	500	1000	1000	1,813
Optimizer	Name	Adam	Adam	Adam	Adam
beta1	0.9	0.9	0.9	0.9
beta2	0.999	0.999	0.999	0.999
Weight Decay	-	1.0e-12	1.0e-12	1.0e-12
Learning Rate	1.0e-3	1.0e-4	1.0e-4	5.0e-4
Schedule	Epoch	72000	1000/300/50/6	500/1000/300/50	160
LR Step with 0.1	20000	-	-	-
Batch Size	500	100	10/100/100/100	128
NeuralODE	Integration Time	Training	Training	Training	0.1
atol	1.0e-5	1.0e-5	1.0e-5	1.0e-5
rtol	1.0e-5	1.0e-5	1.0e-5	1.0e-5
Adjoint	×	✓	✓	✓
Table 5: Negative log-likelihood comparison on the test partition of DW4 dataset for different amounts of training samples; averaged over 3 runs and including standard deviations.
# of Samples	DW4
102	103	104	105
GNF	-2.30 ± 1.59	-7.04 ± 0.64	-7.19 ± 0.99	-7.93 ± 1.10
GNF-att	-2.02 ± 1.34	-4.13 ± 1.20	-5.25 ± 0.89	-6.74 ± 0.89
GNF-att-aug	-3.11 ± 2.15	-4.04 ± 3.40	-6.51 ± 0.49	-9.42 ± 1.15
Simple dynamics	-1.22 ± 0.05	-1.28 ± 0.01	-1.36 ± 0.02	-1.39 ± 0.04
E-NF	-0.54 ± 0.45	-9.89 ± 2.30	-12.15 ± 1.16	-15.29 ± 0.53
GrCNF	-12.53 ± 0.92	-13.74 ± 0.30	-14.09 ± 0.44	-16.07 ± 0.46
Table 6: Negative log-likelihood comparison on the test partition of LJ13 dataset for different amounts of training samples; averaged over 3 runs and including standard deviations.
# Samples	LJ13
10	102	103	104
GNF	6.77 ± 0.39	-0.76 ± 1.12	-4.26 ± 2.76	-12.43 ± 1.21
GNF-att	6.91 ± 0.17	1.40 ± 0.79	-6.81 ± 2.09	-12.05 ± 2.28
GNF-att-aug	2.95 ± 0.55	-6.11 ± 1.12	-13.94 ± 0.95	-15.74 ± 0.58
Simple dynamics	-1.10 ± 2.55	-3.87 ± 0.25	-3.72 ± 0.08	-3.59 ± 0.52
E-NF	-12.86 ± 3.67	-15.75 ± 5.02	-31.51 ± 1.19	-32.83 ± 1.98
GrCNF	-23.64 ± 2.23	-44.24 ± 4.26	-58.02 ± 5.43	-58.71 ± 4.71
GrCNF for QM9 Positional
Layer	Out Size	Norm./Act.
Input	19×3	-
HorP	19×3	-/Tanh
Vec	57	-/-
CS	32	-/Tanh
CS	32	-/Tanh
CS	32	-/Tanh
CS	1	-/Tanh
Grad	19×3	-
Table 7: Network architectures for QM9 Positional. (Left) GrCNF architecture, (Right) scale estimation architecture. The scale estimator estimates one scale parameter from each of the three orthonormal basis vectors  $\mathbf{Y} = \{\mathbf{y}_i\in \mathbb{R}^{19}\}_{i = 1}^3$ , for  $\sqrt{\Lambda} = \mathrm{diag}\left(\{\sqrt{\lambda_i}\}_{i = 1}^3\right)$  with three parameters  $\{\sqrt{\lambda_i}\in \mathbb{R}\}_{i = 1}^3$ . Using the orthonormal orthogonal basis matrix  $\mathbf{Y}$  and the estimated scale parameter  $\sqrt{\Lambda}$ , we generate a point cloud  $\mathbf{P} = \mathbf{Y}\sqrt{\Lambda}$ . SiLU is an activation function proposed in (Ramachandran et al. (2017)) and BatchNorm. is a batch normalization layer in (Ioffe & Szegedy (2015)).
Scale Estimator for QM9 Positional
Layer	Out Size	Norm./Act.
Input	19×3	-
FC	128	BatchNorm./SiLU
FC	256	BatchNorm./SiLU
FC	256	BatchNorm./SiLU
FC	128	BatchNorm./SiLU
FC	3	-/ReLU
D Fundamentals of Concepts Associated with Grassmann Manifold
D.1 Definition of Stiefel Manifold
Definition 2. An (orthogonal or compact) Stiefel manifold  $\mathrm{St}(k, D)$  is defined as the set of orthonormal bases of  $k$ -dimensional subspaces in the Euclidean space  $\mathbb{R}^D$  as in (68).
$$
\operatorname {S t} (k, D) := \left\{\boldsymbol {Y} \in \mathbb {R} ^ {D \times k} \mid \boldsymbol {Y} ^ {\top} \boldsymbol {Y} = \boldsymbol {I} _ {k} \right\}. \tag {68}
$$
For  $\mathbf{Y} \in \mathrm{St}(k, D)$ , the space  $\operatorname{span}(\mathbf{Y})$  spanned by its column vectors is the element of  $\operatorname{Gr}(k, D)$ .
$$
f: \operatorname {S t} (k, D) \rightarrow \operatorname {G r} (k, D): \mathbf {Y} \mapsto \operatorname {s p a n} (\mathbf {Y}). \tag {69}
$$
$\operatorname{St}(k, D)$  is a  $Dk - \frac{k(k + 1)}{2}$ -dimensional compact manifold (Absil et al. (2008)).
D.2 Equivalence Relation
To define the equivalence relation  $\sim$  on a Stiefel manifold  $\operatorname{St}(k, D)$ , we introduce the following two lemmas.
Lemma 1. The necessary and sufficient conditions for  $\operatorname{span}(\mathbf{Y}_1) = \operatorname{span}(\mathbf{Y}_2)$  to hold for  $\mathbf{Y}_1, \mathbf{Y}_2 \in \mathrm{St}(k, D)$  are as follows.
$$
\exists \boldsymbol {Q} \in \mathcal {O} (k) \quad s. t. \quad \boldsymbol {Y} _ {2} = \boldsymbol {Y} _ {1} \boldsymbol {Q}. \tag {70}
$$
Proof.  $\operatorname{span}(\mathbf{Y}_1) = \operatorname{span}(\mathbf{Y}_2) \Leftarrow \mathbf{Y}_2 = \mathbf{Y}_1\mathbf{Q}$ . From the definition,  $\mathbf{Y}_1^\top \mathbf{Y}_1 = \mathbf{Y}_2^\top \mathbf{Y}_2 = \mathbf{I}_k$ , the following is obtained:
$$
\boldsymbol {I} _ {k} = \boldsymbol {Y} _ {1} ^ {\top} \boldsymbol {Y} _ {1} = \boldsymbol {Q} ^ {\top} \boldsymbol {Y} _ {2} ^ {\top} \boldsymbol {Y} _ {2} \boldsymbol {Q} = \boldsymbol {Q} ^ {\top} \boldsymbol {Q}. \tag {71}
$$
Thus, there exists a  $k$ -dimensional orthogonal matrix  $\mathbf{Q} \in \mathcal{O}(k)$ . As the subspace  $\operatorname{span}(\mathbf{Y})$  is invariant to coordinate transformations by orthogonal matrices,  $\operatorname{span}(\mathbf{Y}_1) = \operatorname{span}(\mathbf{Y}_2)$  is true.
$\operatorname{span}(\mathbf{Y}_1) = \operatorname{span}(\mathbf{Y}_2) \Rightarrow \mathbf{Y}_2 = \mathbf{Y}_1 \mathbf{Q}$ . From the assumption, we immediately concluded that  $\mathbf{Y}_2 = \mathbf{Y}_1 \mathbf{Q}$  for  $\mathbf{Q} \in \mathcal{O}(k)$ .
Lemma 2. We define the equivalence relation  $\sim$  on  $\operatorname{St}(k, D)$  to be  $\mathbf{Y}_1 \sim \mathbf{Y}_2$  whenever (70) is satisfied with respect to  $\mathbf{Y}_1, \mathbf{Y}_2 \in \operatorname{St}(k, D)$ . The fact that a binary relation is an equivalence relation  $\sim$  implies that the following three statements hold for  $\forall \mathbf{Y}_1, \mathbf{Y}_2, \mathbf{Y}_3 \in \operatorname{St}(k, D)$ .
Reflexivity  $Y_{1}\sim Y_{1}$
Symmetry  $Y_{1}\sim Y_{2}\Rightarrow Y_{2}\sim Y_{1}$
Transitivity  $Y_{1}\sim Y_{2}\wedge Y_{2}\sim Y_{3}\Rightarrow Y_{1}\sim Y_{3}.$
Proof. With Lemma 1, we can confirm that it is valid as follows:
Reflexivity From  $\mathbf{Y}_1 = \mathbf{Y}_1\mathbf{I}$ ,  $\mathbf{I} \in \mathcal{O}$ , we obtain  $\mathbf{Y}_1 \sim \mathbf{Y}_1$ .
Symmetry As  $\mathbf{Y}_2 = \mathbf{Y}_1\mathbf{Q}$  is obtained from  $\mathbf{Y}_1 \sim \mathbf{Y}_2$ , and  $\mathbf{Y}_2\mathbf{Q}^\top = \mathbf{Y}_1$ ,  $\mathbf{Q}^\top \in \mathcal{O}$  is true, then  $\mathbf{Y}_2 \sim \mathbf{Y}_1$  is obtained.
Transitivity  $\mathbf{Y}_3 = \mathbf{Y}_1\mathbf{Q}_1\mathbf{Q}_2$  with  $\mathbf{Y}_2 = \mathbf{Y}_1\mathbf{Q}_1$  and  $\mathbf{Y}_3 = \mathbf{Y}_2\mathbf{Q}_2$ .  $Q_1Q_2$  is  $(Q_{1}Q_{2})^{\top}(Q_{1}Q_{2}) = Q_2^\top Q_1^\top Q_1Q_2 = Q_2^\top Q_2 = I$ . Moreover, as  $(Q_{1}Q_{2})(Q_{1}Q_{2})^{\top} = Q_{1}Q_{2}Q_{2}^{\top}Q_{1}^{\top} = I$  holds,  $Q_1Q_2 \in \mathcal{O}$  is true. Therefore, we concluded  $Y_1 \sim Y_3$ .
The equivalence class of  $\mathbf{Y} \in \mathrm{St}(k, D)$  is denoted by  $[\mathbf{Y}]$ . In other words,  $[\mathbf{Y}]$  is the set of all elements of  $\mathrm{St}(k, D)$  that are equivalent to  $\mathbf{Y}$ , and the equivalence relation on  $\mathrm{St}(k, D)$  divides  $\mathrm{St}(k, D)$  into equivalence classes with no intersection. Thus,  $\mathbf{Y}$  is then referred to as the representative of the equivalence class  $[\mathbf{Y}]$ . The set of equivalence classes is denoted  $\mathrm{St}(k, D) / \sim$  and is referred to as the
7. Reflexive, symmetric and transitive binary relations. As a consequence of these properties, in a given set, one equivalence relation divides (classifies) the set into equivalence classes. Note that  $R$  is a binary relation in the set  $X$  if for any  $x, y \in X$ , only either  $x$  is related to  $y$  by the relation  $R$ , or  $x$  is not related to  $y$  based on the relation that  $R$  occurs. We write  $x$  is related to  $y$  by relation  $R$  as  $xRy$ .
quotient of  $\operatorname{St}(k, D)$  by the equivalence relation  $\sim$ . In addition,  $\pi \colon \to \operatorname{St}(k, D) \to \operatorname{St}(k, D) / \sim$  is the natural projection that maps  $Y \in \operatorname{St}(k, D)$  onto its equivalence class  $[Y]$ . This projection  $\pi$  is surjection.
D.3 Quotient Manifold
Definition 3. Let  $\overline{\mathcal{M}}$  be a manifold with equivalence relation  $\sim$ . The quotient space  $\overline{\mathcal{M}}/\sim$  with  $\sim$  of  $\overline{\mathcal{M}}$  is the set of all equivalence classes. Thus,  $\overline{\mathcal{M}}/\sim \coloneqq \{\pi(\overline{x}) \mid \overline{x} \in \overline{\mathcal{M}}\}$ , where  $\pi: \overline{\mathcal{M}} \to \overline{\mathcal{M}}/\sim$  is the natural projection and  $\pi(\overline{x}) \coloneqq \{\overline{y} \in \overline{\mathcal{M}} \mid \overline{y} \sim \overline{x}\}$ . Then,  $\overline{\mathcal{M}}$  is referred to as the total space or the total manifold. Moreover,  $\overline{\mathcal{M}}/\sim$  is referred to as a quotient manifold of  $\overline{\mathcal{M}}$  if  $\overline{\mathcal{M}}/\sim$  admits a differentiable structure.
Let  $\mathcal{M} = \overline{\mathcal{M}} / \sim$  be a quotient manifold. Further, suppose that  $\overline{\mathcal{M}}$  is endowed with a Riemann metric  $\overline{g}$ , and let  $\pmb{x} = \pi(\pmb{x})$ . The horizontal space  $T_{\overline{x}}^{\mathrm{h}}\overline{\mathcal{M}}$  is the orthogonal complement of the vertical space  $T_{\overline{x}}^{\mathrm{v}}\overline{\mathcal{M}} := T_{\overline{x}}\pi^{-1}(\pmb{x})$  in the tangent space  $T_{\overline{x}}\overline{\mathcal{M}}$  and is defined as the follows:
$$
T _ {\bar {x}} ^ {\mathrm {h}} \bar {\mathcal {M}} := \left(T _ {\bar {x}} ^ {\mathrm {v}} \bar {\mathcal {M}}\right) ^ {\perp} = \left\{\bar {\boldsymbol {\eta}} _ {\bar {x}} \in T _ {\bar {x}} \bar {\mathcal {M}} \mid \bar {g} _ {\bar {x}} \left(\bar {\boldsymbol {\xi}} _ {\bar {x}}, \bar {\boldsymbol {\eta}} _ {\bar {x}}\right) = 0, ^ {\forall} \bar {\boldsymbol {\xi}} _ {\bar {x}} \in T _ {\bar {x}} ^ {\mathrm {v}} \bar {\mathcal {M}} \right\}. \tag {72}
$$
The horizontal lift  $\overline{\pmb{\xi}}_{\overline{x}}^{\mathrm{h}}\in T_{\overline{x}}^{\mathrm{h}}\overline{\mathcal{M}}$  of the tangent vector  $\pmb {\xi}_x\in T_x\mathcal{M}$  at point  $\overline{x}\in \pi^{-1}(\pmb {x})$  is a tangent vector that is uniquely determined as  $d\pi_{\overline{x}}\left(\overline{\pmb{\xi}}_{\overline{x}}^{\mathrm{h}}\right) = \pmb {\xi}_x$  (Absil et al. (2008)).
D.4 Grassmann Manifold Exploiting the Quotient Structure
D.4.1 Tangent Space on a Stiefel Manifold
We describe the relationship between tangent space  $T_{[\mathbf{Y}]}\mathrm{Gr}(k,D)$  on  $\mathrm{Gr}(k,D)$  and tangent space  $T_{\mathbf{Y}}\mathrm{St}(k,D)$  on  $\mathrm{St}(k,D)$  to relate the tangent vectors of a Grassmann manifold  $\mathrm{Gr}(k,D)$  to the tangent vectors of a Stiefel manifold  $\mathrm{St}(k,D)$  in a matrix representation. We take the derivative on both sides of  $\mathbf{Y}(t)^{\top}\mathbf{Y}(t) = \mathbf{I}_p$  in (68) by  $t$  and solve for  $t = 0$ .
$$
\frac {d}{d t} \left\{\boldsymbol {Y} (t) ^ {\top} \boldsymbol {Y} (t) \right\} = \frac {d}{d t} \boldsymbol {I} _ {p} \tag {73}
$$
$$
\frac {d}{d t} \boldsymbol {Y} (t) ^ {\top} \boldsymbol {Y} (t) + \boldsymbol {Y} (t) ^ {\top} \frac {d}{d t} \boldsymbol {Y} (t) = 0 \tag {74}
$$
$$
\frac {d}{d t} \boldsymbol {Y} (0) ^ {\top} \boldsymbol {Y} (0) + \boldsymbol {Y} (0) ^ {\top} \frac {d}{d t} \boldsymbol {Y} (0) = 0 \tag {75}
$$
$$
\overline {{\boldsymbol {\xi}}} _ {\mathbf {Y}} ^ {\top} \mathbf {Y} + \mathbf {Y} ^ {\top} \overline {{\boldsymbol {\xi}}} _ {\mathbf {Y}} = 0, \tag {76}
$$
where  $\overline{\pmb{\xi}}_{\pmb{Y}} = \frac{d}{dt}\pmb{Y}(0)$  is the tangent vector at  $\mathbf{Y}^{8}$ .
Definition 4. Define the tangent space  $T_{\mathbf{Y}} \mathrm{St}(k, D)$  at  $\mathbf{Y}$  on the Stiefel manifold as follows:
$$
T _ {\boldsymbol {Y}} \operatorname {S t} (k, D) = \left\{\overline {{\boldsymbol {\xi}}} _ {\boldsymbol {Y}} \in \mathbb {R} ^ {D \times k} \mid \overline {{\boldsymbol {\xi}}} _ {\boldsymbol {Y}} ^ {\top} \boldsymbol {Y} + \boldsymbol {Y} ^ {\top} \overline {{\boldsymbol {\xi}}} _ {\boldsymbol {Y}} = \boldsymbol {0} _ {k} \right\}. \tag {77}
$$
Let matrix  $\mathbf{Y}_{\perp} \in \mathbb{R}^{D \times (D - k)}$  be a matrix satisfying the following:
$$
\boldsymbol {Y} _ {\perp} ^ {\top} \boldsymbol {Y} _ {\perp} = \boldsymbol {I} _ {D - k}, \quad \boldsymbol {Y} ^ {\top} \boldsymbol {Y} _ {\perp} = 0, \quad \boldsymbol {Y Y} ^ {\top} + \boldsymbol {Y} _ {\perp} \boldsymbol {Y} _ {\perp} ^ {\top} = \boldsymbol {I} _ {D}. \tag {78}
$$
As  $\left[\begin{array}{cc}\boldsymbol{Y} & \boldsymbol{Y}_{\perp}\end{array}\right]$  is an orthogonal matrix<sup>9</sup>, the column vectors of  $\boldsymbol{Y}$  and  $\boldsymbol{Y}_{\perp}$  form an orthonormal basis in  $\mathbb{R}^D$ . Thus, any  $D \times k$  matrix can be written in terms of the  $\boldsymbol{C} \in \mathbb{R}^{k \times k}$  and  $\boldsymbol{B} \in \mathbb{R}^{(D - k) \times k}$  coefficient matrices as follows:
$$
\boldsymbol {Y} \boldsymbol {C} + \boldsymbol {Y} _ {\perp} \boldsymbol {B}, \tag {79}
$$
The tangent space is defined independently for each point of the manifold; hence, the subscript  $\mathbf{Y}$ , as in  $\overline{\xi}_{\mathbf{Y}}$  is clearly stated to emphasize that it is a tangent vector at  $\mathbf{Y}$ .
$\left[ \begin{array}{ll}\boldsymbol {Y} & \boldsymbol{Y}_{\perp} \end{array} \right]^{-1}\left[ \begin{array}{ll}\boldsymbol {Y} & \boldsymbol{Y}_{\perp} \end{array} \right] = \left[ \begin{array}{ll}\boldsymbol {Y} & \boldsymbol{Y}_{\perp} \end{array} \right]^{\top}\left[ \begin{array}{ll}\boldsymbol {Y} & \boldsymbol{Y}_{\perp} \end{array} \right] = \left[ \begin{array}{ll}\boldsymbol {Y} & \boldsymbol{Y}_{\perp} \end{array} \right]\left[ \begin{array}{ll}\boldsymbol {Y} & \boldsymbol{Y}_{\perp} \end{array} \right]^{\top} = \boldsymbol {I}_D.$
where  $\overline{\pmb{\xi}}_Y = Y\pmb {C} + Y_\perp \pmb{B}$  is inserted. The following equation is obtained.
$$
\begin{array}{l} \bar {\xi} _ {Y} ^ {\top} Y + Y ^ {\top} \bar {\xi} _ {Y} = \left(Y C + Y _ {\perp} B\right) ^ {\top} Y + Y ^ {\top} \left(Y C + Y _ {\perp} B\right) (80) \\ = \boldsymbol {B} ^ {\top} \boldsymbol {Y} _ {\perp} ^ {\top} \boldsymbol {Y} + \boldsymbol {C} ^ {\top} \boldsymbol {Y} ^ {\top} \boldsymbol {Y} + \boldsymbol {Y} ^ {\top} \boldsymbol {Y} \boldsymbol {C} + \boldsymbol {Y} ^ {\top} \boldsymbol {Y} _ {\perp} \boldsymbol {B} (81) \\ = \boldsymbol {B} ^ {\top} \boldsymbol {Y} ^ {\top} \boldsymbol {Y} _ {\perp} + \boldsymbol {C} ^ {\top} + \boldsymbol {C} (82) \\ = \boldsymbol {C} ^ {\top} + \boldsymbol {C} (83) \\ = \mathbf {0} _ {k}. (84) \\ \end{array}
$$
Thus, the following equation is derived.