SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\mathbf{V}_0^\Delta = \mathbf{V}_0^X -\mathbf{V}_0^Y$  , and  $\pmb{\Sigma}^{+}(\mathbf{z})$  denotes the pseudo inverse of  $\pmb {\Sigma}(\mathbf{z}) = \sum_{i = 1}^{m}\mathbf{V}_{i}\mathbf{z}\mathbf{z}^{\top}\mathbf{V}_{i}^{\top}$
Proof. The noise term of the SDEs can be written as  $\sigma(Z_t, t)dW_t$  where  $\sigma: \mathbb{R}^n \times [0, T] \to \mathbb{R}^{n \times n}$  is specified in Eq. (25). Thus, the corollary is a direct consequence of Lemma 5, as  $\sigma(\mathbf{z}, t)\sigma(\mathbf{z}, t)^\top = \sum_{i=1}^{m} \mathbf{V}_i\mathbf{z}\mathbf{z}^\top\mathbf{V}_i^\top$ , see Eq. (26).
Finally, the following proposition (Proposition 7) considers exactly the SDEs discussed in the main manuscript. The process  $X$  is the uninformative prior process that we select in advance and the process  $Y$  is the posterior process that we learn from the data. Thus, Eq. (38) is the KL-term that appears in the ELBO in our variational setting.
Proposition 7 (KL divergence for one-point motions on  $\mathbb{S}^{n - 1}$ ). Let  $\mathbf{K}(t):[0,T]\to \mathfrak{so}(n)$  be a continuous map and let  $\alpha \neq 0$ . If  $X$  and  $Y$  are stochastic processes on  $[0,T]$  with (path) distributions  $\mathbb{P}^X$  and  $\mathbb{P}^Y$ , resp., that solve the following Itô SDEs
$$
\mathrm {d} X _ {t} = - \alpha^ {2} \frac {n - 1}{2} X _ {t} \mathrm {d} t + \alpha \mathrm {d} U _ {t} X _ {t} \quad X _ {0} \sim \mathcal {P} _ {\mathbb {S} ^ {n - 1}} ^ {X _ {0}}, \tag {36}
$$
$$
\mathrm {d} Y _ {t} = \left(\mathbf {K} (t) - \alpha^ {2} \frac {n - 1}{2} \mathbf {I} _ {\mathrm {n}}\right) X _ {t} \mathrm {d} t + \alpha \mathrm {d} U _ {t} Y _ {t} \quad Y _ {0} \sim \mathcal {P} _ {\mathbb {S} ^ {n - 1}} ^ {Y _ {0}}, \tag {37}
$$
where  $\mathcal{P}_{\mathbb{S}^{n - 1}}^{X_0}$  and  $\mathcal{P}_{\mathbb{S}^{n - 1}}^{Y_0}$  are distributions supported on the unit  $n$ -sphere  $\mathbb{S}^{n - 1}$  with finite second moment. Then, the KL divergence between  $\mathbb{P}^Y$  and  $\mathbb{P}^X$  is given by
$$
D _ {\mathrm {K L}} \left(\mathbb {P} ^ {Y} \| \mathbb {P} ^ {X}\right) = D _ {\mathrm {K L}} \left(\mathcal {P} _ {\mathbb {S} ^ {n - 1}} ^ {Y _ {0}} \| \mathcal {P} _ {\mathbb {S} ^ {n - 1}} ^ {X _ {0}}\right) + \frac {1}{2 \alpha^ {2}} \int_ {0} ^ {T} \int_ {\mathbb {S} ^ {n - 1}} q _ {Y _ {t}} (\mathbf {z}) \| \mathbf {K} (t) \mathbf {z} \| ^ {2} \mathrm {d} \mathbf {z} \mathrm {d} t. \tag {38}
$$
Proof. By Lemma 4, we can replace Eq. (36) and Eq. (37) by
$$
\mathrm {d} X _ {t} = - \alpha^ {2} \frac {n - 1}{2} \mathbf {I} _ {\mathrm {n}} \mathrm {d} t + \alpha \left(\mathbf {I} _ {\mathrm {n}} - X _ {t} X _ {t} ^ {\top}\right) \mathrm {d} W _ {t}, \quad X _ {0} \sim \mathcal {P} _ {\mathbb {S} ^ {n - 1}} ^ {X _ {0}}, \tag {39}
$$
$$
\mathrm {d} Y _ {t} = \left(\mathbf {K} (t) - \alpha^ {2} \frac {n - 1}{2} \mathbf {I} _ {\mathrm {n}}\right) \mathrm {d} t + \alpha \left(\mathbf {I} _ {\mathrm {n}} - Y _ {t} Y _ {t} ^ {\top}\right) \mathrm {d} W _ {t}, \quad Y _ {0} \sim \mathcal {P} _ {\mathbb {S} ^ {n - 1}} ^ {\mathrm {Y} _ {0}}, \tag {40}
$$
as the solutions follow the same distribution  $\mathbb{P}^X$  and  $\mathbb{P}^Y$ , respectively.
The proposition is thus a consequence of Lemma 5 for a specific choice of  $f, g$  and  $\sigma$ , i.e.,
$$
\begin{array}{l} D _ {\mathrm {K L}} \left(\mathbb {P} ^ {Y} \| \mathbb {P} ^ {X}\right) \\ = D _ {\mathrm {K L}} \left(\mathcal {P} _ {\mathbb {S} ^ {n - 1}} ^ {Y _ {0}} \| \mathcal {P} _ {\mathbb {S} ^ {n - 1}} ^ {X _ {0}}\right) + \frac {1}{2} \int_ {0} ^ {T} \int_ {\mathbb {R} ^ {n}} q _ {Y _ {t}} [ \mathbf {K} (t) \mathbf {z} ] ^ {\top} \left(\alpha^ {2} \left(\mathbf {I} _ {\mathrm {n}} - \mathbf {z} \mathbf {z} ^ {\top}\right) \left(\mathbf {I} _ {\mathrm {n}} - \mathbf {z} \mathbf {z} ^ {\top}\right) ^ {\top}\right) ^ {+} [ \mathbf {K} (t) \mathbf {z} ] \mathrm {d} \mathbf {z} \mathrm {d} t. \tag {41} \\ \end{array}
$$
To finish the proof, we highlight two points. First, Lemma 4 ensures that the process  $Y$  evolves on the unit  $n$ -sphere  $\mathbb{S}^{n-1}$ , so it suffices to integrate  $q_{Y_t}$  over  $\mathbb{S}^{n-1} \subset \mathbb{R}^n$  instead of over the entire  $\mathbb{R}^n$  as in Eq. (41). Second, the integrand simplifies to
$$
\begin{array}{l} \left[ \mathbf {K} (t) \mathbf {z} \right] ^ {\top} \left(\alpha^ {2} \left(\mathbf {I} _ {\mathrm {n}} - \mathbf {z} \mathbf {z} ^ {\top}\right) \left(\mathbf {I} _ {\mathrm {n}} - \mathbf {z} \mathbf {z} ^ {\top}\right) ^ {\top}\right) ^ {+} [ \mathbf {K} (t) \mathbf {z} ] = \alpha^ {- 2} \left[ \mathbf {K} (t) \mathbf {z} \right] ^ {\top} \left(\mathbf {I} _ {\mathrm {n}} - \mathbf {z} \mathbf {z} ^ {\top}\right) [ \mathbf {K} (t) \mathbf {z} ] (42) \\ = \alpha^ {- 2} \| \mathbf {K} (t) \mathbf {z} \| ^ {2}, (43) \\ \end{array}
$$
where the first equality holds because  $(\mathbf{I}_{\mathrm{n}} - \mathbf{z}\mathbf{z}^{\top})$  is a projection, and the second equality holds because  $\mathbf{z}^{\top}\mathbf{K}(t)\mathbf{z} = 0$  as  $\mathbf{K}(t) \in \mathfrak{g}$  is skew-symmetric.
References (Supplementary material)
[61] E. J. Allen, L. J. S. Allen, A. Arciniega, and P. E. Greenwood. "Construction of equivalent stochastic differential equation models". In: Stochastic Analysis and Applications 26.2 (2008), pp. 274-297.
[4] P. Becker, H. Pandya, G. Gebhardt, C. Zhao, J. Taylor, and G. Neumann. "Recurrent Kalman Networks: Factorized Inference in High-Dimensional Deep Feature Spaces". In: ICML. 2019.
[62] A. M. Bloch, P. E. Crouch, J. E. Marsden, and A. K. Sanyal. "Optimal Control and Geodesics on Quadratic Matrix Lie Groups". In: Foundations of Computational Mathematics 8 (4 2008), pp. 469-500.
[6] R. W. Brockett. "Lie Algebras and Lie Groups in Control Theory". In: Geometric Methods in System Theory. Springer, 1973, pp. 43-82.
[8] N. D. Cao and W. Aziz. "The Power Spherical distribution". In: arXiv abs/2006.04437 (2020).
[9] F. P. Casale, A. V. Dalca, L. Saglietti, J. Listgarten, and N. Fusi. "Gaussian Process Prior Variational Autoencoders". In: NeurIPS. 2018.
[10] R. T. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud. “Neural Ordinary Differential Equations”. In: NeurIPS. 2018.
[12] G. S. Chirikjian. "Stochastic models, information theory, and Lie groups". In: Applied and numerical harmonic analysis. Boston, Basel, and Berlin: Birkhäuser, 2012, pp. 361-388.
[18] H. Fu, C. Li, X. Liu, J. Gao, A. Celikyilmaz, and L. Carin. "Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing". In: NAACL. 2019.
[23] K. Ito. "Stochastic Calculus". In: Lect. Notes Phys. 39 (1975). Ed. by H. Araki, pp. 218-223.
[63] I. Karatzas and S. E. Shreve. Brownian motion and stochastic calculus. 2nd edition. Springer, 1998.
[28] D. P. Kingma and J. Ba. "Adam: A Method for Stochastic Optimization". In: ICLR. 2015.
[64] P. Kloeden and E. Platen. Numerical methods for stochastic differential equations. 1991.
[65] J. M. Lee. Introduction to smooth manifolds. Graduate texts in mathematics. Springer, 2003.
[30] X. Li, T.-K. L. Wong, R. T. Chen, and D. Duvenaud. "Scalable Gradients for Stochastic Differential Equations". In: AISTATS. 2020.
[31] M. Liao. Levy processes in Lie groups. Providence, R.I.: Cambridge University Press, 2004.
[36] G. Marjanovic and V. Solo. "Numerical Methods for Stochastic Differential Equations in Matrix Lie Groups Made Simple". In: IEEE Transactions on Automatic Control 63.12 (2018), pp. 4035-4050.
[40] M. Muniz, M. Ehrhardt, M. Günther, and R. Winkler. “Higher strong order methods for linear Itô SDEs on matrix Lie groups”. In: BIT Numerical Mathematics 62.4 (2022), pp. 1095–1119.
[43] M. Opper. "Variational Inference for Stochastic Differential Equations". In: Annalen der Physik 531.3 (2019), p. 1800233.
[66] D. Revuz and M. Yor. Continuous Martingales and Brownian Motion. 3rd edition. Springer, 1999.
[48] Y. Rubanova, R. T. Chen, and D. Duvenaud. "Latent ODE for Irregularly-Sampled Time Series". In: NeurIPS. 2019.
[67] D. Sanz-Alonso and A. M. Stuart. "Gaussian approximations of small noise diffusions in Kullback-Leibler divergence". In: Communications in Mathematical Sciences 15.7 (2017), pp. 2087-2097.
[68] S. Särkkä and A. Solin. Applied stochastic differential equations. Cambridge University Press, 2019.
[49] M. Schirmer, M. Eltayeb, S. Lessmann, and M. Rudolph. "Modeling Irregular Time Series with Continuous Recurrent Units". In: ICML. 2022.
[50] S. N. Shukla and B. M. Marlin. "Multi-Time Attention Networks for Irregularly Sampled Time Series". In: ICLR. 2021.
[51] I. Silva, G. Moody, D. J. Scott, L. A. Celi, and R. G. Mark. “Predicting in-hospital mortality of ICU patients: The PhysioNet/Computing in cardiology challenge 2012”. In: Computing in Cardiology 39 (2012), pp. 245–248.
[69] T. Sottinen and S. Särkkä. "Application of Girsanov theorem to particle filtering of discretely observed continuous-time non-linear systems". In: Bayesian Analysis 3.3 (2008), pp. 555-584.
[55] D. W. Stroock. “On the growth of stochastic integrals”. In: Zeitschrift für Wahrscheinlichkeits-theorie und Verwandte Gebiete 18.4 (1971), pp. 340–344.
[57] B. Øksendal. Stochastic Differential Equations. Universitext. Springer, 1995.
[60] C. Yildiz, M. Heinonen, and H. Lahdesmäki. "ODE²VAE: Deep generative second order ODEs with Bayesian neural networks". In: NeurIPS. 2019.