SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Gaussian processes and the neural processes family. One important and powerful framework to construct distributions over functional spaces are Gaussian processes (Rasmussen, 2003). Yet, they are restricted in their modelling capacity and when using exact inference they scale poorly with the number of datapoints. These problems can be partially alleviated by using neural processes (Kim et al., 2019; Garnelo et al., 2018b; Garnelo et al., 2018a; Jha et al., 2022; Louizos et al., 2019; Singh et al., 2019), although they also assume a Gaussian likelihood. Recently introduced autoregressive NPs (Bruinsma et al., 2023) alleviate this limitation, but they are disadvantaged by the fact that variables early in the auto-regressive generation only have simple distributions (typically Gaussian). Finally, (Dupont et al., 2022) model weights of implicit neural representation using diffusion models.
Stationary stochastic processes. The most popular Gaussian process kernels (e.g. squared exponential, Matérn) are stationary, that is, they are translation invariant. These lead to invariant Gaussian processes, whose samples when translated have the same distribution as the original ones. This idea can be extended to the entire isometry group of Euclidean spaces (Holderrieth et al., 2021), allowing for modelling higher order tensor fields, such as wind fields or incompressible fluid velocity (Macêdo and Castro, 2010). Later, Azangulov et al. (2022) and Azangulov et al. (2023) extended stationary kernels and Gaussian processes to a large class of non-Euclidean spaces, in particular all compact spaces, and symmetric non compact spaces. In the context of neural processes, (Gordon et al., 2020) introduced CONVCNP so as to encode translation equivariance into the predictive process. They do so by embedding the context into a translation equivariant functional representation which is then decoded with a convolutional neural network. Holderrieth et al. (2021) later extended this idea to construct neural processes that are additionally equivariant w.r.t. rotations or subgroup thereof.
Spatial structure in diffusion models. A variety of approaches have also been proposed to incorporate spatial correlation in the noising process of finite-dimensional diffusion models leveraging the multiscale structure of data (Jing et al., 2022; Guth et al., 2022; Ho et al., 2022; Sahara et al., 2021; Hoogeboom and Salimans, 2022; Rissanen et al., 2022). Our methodology can also be seen as a principled way to modify the forward dynamics in classical denoising diffusion models. Hence, our contribution can be understood in the light of recent advances in generative modelling on soft and cold denoising diffusion models (Daras et al., 2022; Bansal et al., 2022; Hoogeboom and Salimans, 2022). Several recent work explicitly introduced a covariance matrix in the Gaussian noise, either on a choice of kernel (Biloš et al., 2022), based on Discrete Fourier Transform of images (Voleti et al., 2022), or via empirical second order statistics (squared pairwise distances and the squared radius of gyration) for protein modelling (Ingraham et al., 2022). Alternatively, (Guth et al., 2022) introduced correlation on images leveraging a wavelet basis.
Functional diffusion models. Infinite dimensional diffusion models have been investigated in the Euclidean setting in (Kerrigan et al., 2022; Pidstrigach et al., 2023; Lim et al., 2023b; Bond-Taylor and Willcocks, 2023; Hagemann et al., 2023; Franzese et al., 2023; Dutordoir et al., 2022; Phillips et al., 2022). Most of these works are based on an extension of the diffusion models techniques (Song et al., 2021; Ho et al., 2020) to the infinite-dimensional space, leveraging tools from the Cameron-Martin theory such as the Feldman-Hajek theorem (Kerrigan et al., 2022; Pidstrigach et al., 2023) to define infinite-dimensional Gaussian measures and how they interact. We refer to (Da Prato and Zabczyk, 2014) for a thorough introduction to Stochastic Differential Equations in infinite dimension. (Phillips et al., 2022) consider another approach by defining countable diffusion processes in a basis. All these approaches amount to learn a diffusion model with spatial structure. Note that this induced correlation is necessary for the theory of infinite dimensional SDE (Da Prato and Zabczyk, 2014) to be applied but is not necessary to implement diffusion models (Dutordoir et al., 2022). Several approaches have been considered for conditional sampling. (Pidstrigach et al., 2023; Bond-Taylor and Willcocks, 2023) modify the reverse diffusion to introduce a guidance term, while (Dutordoir et al., 2022; Kerrigan et al., 2022) use the replacement method. Finally (Phillips et al., 2022) amortise the score function w.r.t. the conditioning context.
Figure 4: Prior and posterior samples (in blue) from the data process and the GeomNDP model, with context points in red and posterior mean in black.
5.1 1D regression over stationary scalar fields
We evaluate GEOMNDPS on several synthetic 1D regression datasets. We follow the same experimental setup as Bruinsma et al. (2020) which we detail in App. F.1. In short, it contains Gaussian (Squared Exponential (SE), MATERN  $\left(\frac{5}{2}\right)$ , WEAKLY PERIODIC) and non-Gaussian (SAWTOOTH and MIXTURE) sample paths, where MIXTURE is a combination of the other four datasets with equal weight. Fig. 9 shows samples for each of these datasets. The Gaussian datasets are corrupted with observation noise with variance  $\sigma^2 = 0.05^2$ . Table 1 reports the average log-likelihood  $p(y^{*}|x^{*},\mathcal{C})$  across 4096 test samples, where the context set size is uniformly sampled between 1 and 10 and the target has fixed size of 50. All inputs  $x^{c},x^{*}$  are chosen uniformly within their input domain which is [-2, 2] for the training data and 'interpolation' evaluation and [2, 6] for the 'generalisation' evaluation.
We compare the performance of GeomNDP to a GP with the true hyperparameters (when available), a (convolutional) Gaussian NP (Bruinsma et al., 2020), a convolutional NP (Gordon et al., 2020) and a vanilla attention-based NDP (Dutordoir et al., 2022) which we reformulated in the continuous diffusion process framework to allow for log-likelihood evaluations and thus a fair comparison—denoted  $\mathrm{NDP}^*$ . We enforce translation invariance in the score network for GeomNDP by subtracting the centre of mass from the input  $x$ , inducing stationary scalar fields.
On the GP datasets, GNP, CONVNPs and GeomNDP methods are able to fit the conditionals perfectly—matching the log-likelihood of the GP model. GNP's performance degrades on the non-Gaussian datasets as it is restricted by its conditional Gaussian assumption, whilst NDPs methods still performs well as illustrated on Fig. 4. In the bottom rows of Table 1, we assess the models ability to generalise outside of the training input range  $x \in [-2, 2]$ , and evaluate them on a translated grid where context and target points are sampled from [2, 6]. Only convolutional NPs (GNP and CONVNp) and T(1)—GEOMNDP are able to model stationary processes and therefore to perform as well as in the interpolation task. The NDP*, on the contrary, drastically fails at this task.
Table 1: Mean test log-likelihood (TLL)  $(\uparrow)\pm 1$  standard error estimated over 4096 test samples are reported. Statistically significant best non-GP model is in bold. \* stands for a TLL below -10. NP baselines from Bruinsma et al. (2020). T(1)-GeomNDP indicates our proposed method with a translation invariant score.
SE	MATÉRN(5/2)	WEAKLY PER.	SAWTOOTH	MIXTURE
INTERPOLAT.	GP (OPTIMUM)	0.70±0.00	0.31±0.00	-0.32±0.00	-	-
T(1)-GEOMNDP	0.72±0.03	0.32±0.03	-0.38±0.03	3.39±0.04	0.64±0.08
NDP*	0.71±0.03	0.30±0.03	-0.37±0.03	3.39±0.04	0.64±0.08
GNP	0.70±0.01	0.30±0.01	-0.47±0.01	0.42±0.01	0.10±0.02
CONVNP	-0.46±0.01	-0.67±0.01	-1.02±0.01	1.20±0.01	-0.50±0.02
GENERALISAT.	GP (OPTIMUM)	0.70±0.00	0.31±0.00	-0.32±0.00	-	-
T(1)-GEOMNDP	0.70±0.02	0.31±0.02	-0.38±0.03	3.39±0.03	0.62±0.02
NDP*	*	*	*	*	*
GNP	0.69±0.01	0.30±0.01	-0.47±0.01	0.42±0.01	0.10±0.02
CONVNP	-0.46±0.01	-0.67±0.01	-1.02±0.01	1.19±0.01	-0.53±0.02
MODEL	SE	CURL-FREE	DIV-FREE
GP	0.56±0.00	0.66±0.00	0.66±0.00
NDP*	0.55±0.00	0.62±0.01	0.62±0.01
E(2)-GEOMNDP	0.56±0.01	0.65±0.01	0.66±0.01
GP (DIAG.)	-1.56±0.00	-1.47±0.00	-1.47±0.00
T(2)-CONVCNP	-1.71±0.01	-1.77±0.01	-1.76±0.00
E(2)-STEERCNP	-1.61±0.00	-1.57±0.00	-1.57±0.01
Figure 5: Quantitative results for experiments on GP vector fields. Mean predictive log-likelihood  $(\uparrow)$  and confidence interval estimated over 5 random seeds. Left: Comparison with neural processes. Statistically significant results are in bold. Right: Ablation study when varying the number of training data samples.
Non white kernels for limiting process. The NDP methods in the above experiment target the white kernel  $\mathbb{1}(x = x^{\prime})$  in the limiting process. In App. F.1.3, we explore different choices for the limiting kernel, such as SE and periodic kernels with short and long lengthscales, along with several score parameterisations, see App. B.3 for a description of these. We observe that although choosing such kernels gives a head start to the training, it eventually yield slightly worse performance. We attribute this to the additional complexity of learning a non-diagonal covariance. Finally, across all datasets and limiting kernels, we found the preconditioned score  $\mathrm{K}\nabla \log p_t$  to result in the best performance.
Conditional sampling ablation. We employ the SE dataset to investigate various configurations of the conditional sampler as we have access to the ground truth conditional distribution through the GP posterior. In Fig. 11 we compute the Kullback-Leibler divergence between the samples generated by GEOMNDP and the actual conditional distribution across different conditional sampling settings. Our results demonstrate the importance of performing multiple Langevin dynamics steps during the conditional sampling process. Additionally, we observe that the choice of noising scheme for the context values  $y_{c}$  has relatively less impact on the overall outcome.
5.2 Regression over Gaussian process vector fields
We now focus our attention to modelling equivariant vector fields. For this, we create datasets using samples from a two-dimensional zero-mean GP with one of the following E(2)-equivariant kernels: a diagonal Squared-Exponential (SE) kernel, a zero curl (CURL-FREE) kernel and a zero divergence (DIV-FREE) kernel, as described in App. D.1.
We equip our model, GeomNDP, with a  $\mathrm{E}(2)$ -equivariant score architecture, based on steerable CNNs (Thomas et al., 2018; Weiler and Cesa, 2021). We compare to  $\mathrm{NDP}^*$  with a non-equivariant attention-based network (Dutordoir et al., 2022). We also evaluate two neural processes, a translation-equivariant CONVCNP (Gordon et al., 2020) and a  $\mathrm{C4} \ltimes \mathbb{R}^2 \subset \mathrm{E}(2)$ -equivariant STEERCNP (Holderrieth et al., 2021). We also report the performance of the data-generating GP, and the same GP but with diagonal posterior covariance GP (DIAG.). We measure the predictive log-likelihood of the data process samples under the model on a held-out test dataset.
We observe in Fig. 5 (Left), that the CNPs performance is limited by their diagonal predictive covariance assumption, and as such cannot do better than the GP (DIAG.). We also see that although  $\mathrm{NDP^{*}}$  is able to fit well GP posteriors, it does not reach the maximum log-likelihood value attained by the data GP, in contrast to its equivariant counterpart GEOMNDP. To further explore gains brought by the built-in equivariance, we explore the data-efficiency in Fig. 5 (Right), and notice that E(2)-GEOMNDP requires few data samples to fit the data process, since effectively the dimension of the (quotiented) state space is dramatically reduced.
5.3 Global tropical cyclone trajectory prediction
Finally, we assess our model on a task where the domain of the stochastic process is a non-Euclidean manifold. We model the trajectories of cyclones over the earth, modelled as sample paths of the form  $\mathbb{R} \to S^2$  coming from a stochastic process. The data is drawn from the International Best Track Archive for Climate Stewardship (IBTrACS) Project, Version 4 ((Knapp et al., 2010; Knapp et al., 2018)) and preprocessed as per App. F.3, where details on the implementation of the score function, the ODE/SDE solvers used for the sampling, and baseline methods can be found.
Figure 6: Left: 1000 samples from the training data. Right: 1000 samples from the trained model.
(a) Interpolation
Figure 7: Top: Examples of conditional trajectories sampled from the GeomNDP model. Blue: Conditioned sections of the trajectory. Green: The actual trajectory of the cyclone. Red: conditional samples from the model. Purple: closest matching trajectories in the dataset to the conditioning data.
(b) Extrapolation
Fig. 6 shows some cyclone trajectories samples from the data process and from a trained GEOMNDP model. We also demonstrate how such trajectories can be interpolated or extrapolated using the conditional sampling method detailed in Sec. 3.3. Such conditional sample paths are shown in Fig. 7. Additionally, we report in Table 2 the likelihood and MSE for a series of methods. The interpolation task involves conditioning on the first and last  $20\%$  of the cyclone trajectory and predicting intermediary positions. The extrapolation task involves conditioning on the first  $40\%$  of trajectories and predicting future positions. We see that the GPs (modelled as  $f: \mathbb{R} \to \mathbb{R}^2$ , one on latitude/longitude coordinates, the other via a stereographic projection, using a diagonal RBF kernel with hyperparameters fitted with maximum likelihood) fail drastically given the high non-Gaussianity of the data. In the interpolation task, the NDP performs as well as the GEOMNDP, but the additional geometric structure of modelling the outputs living on the sphere appears to significantly help for extrapolation. See App. F.3 for more fine-grained results.
In this work, we have extended diffusion models to model invariant stochastic processes over tensor fields. We did so by (a) constructing a continuous noising process over function spaces which correlate input samples with an equivariant kernel, (b) parameterising the score with an equivariant neural network. We have empirically demonstrated the ability of our introduced model GEOMNDP to fit complex stochastic processes, and by encoding the symmetry of the problem at hand, we show that it is more data efficient and better able to generalise.
We highlight below some current limitations and important research directions. First, evaluating the model is slow as it relies on costly SDE or ODE solvers, as existing diffusion models. Second, targeting a white noise process appears to over-perform other Gaussian processes. In future work, we would like to investigate the practical influence of different kernels. Third, strict invariance may sometimes be too strong, we thus suggest softening it by amortising the score network over extra spatial information available from the problem at hand.
Table 2: Comparative results of different models on the cyclone dataset, comparing test set likelihood, interpolation likelihood and mean squared error (MSE), and extrapolation likelihood and mean squared error. These are estimated over 5 random seeds. We only report likelihoods of models defined w.r.t the uniform measure on  $S^2$ .
MODEL	TEST DATA Likelihood	INTERPOLATION	EXTRAPOLATION
Likelihood	MSE (km)	Likelihood	MSE (km)
GEOMNDP (R → S2)	802±5	535±4	162±6	536±4	496±14
STEREOGRAPHIC GP (R → R2/{0})	393±3	266±3	2619±13	245±2	6587±55
NDP (R → R2)	-	-	166±22	-	769±48
GP (R → R2)	-	-	6852±41	-	8138±87
We are grateful to Paul Rosa for helping with the proof, and to José Miguel Hernández-Lobato for useful discussions. We thank the hydra (Yadan, 2019), jax (Bradbury et al., 2018) and geomstats (Miola et al., 2020) teams, as our library is built on these great libraries. Richard E. Turner and Emile Mathieu are supported by an EPSRC Prosperity Partnership EP/T005386/1 between Microsoft Research and the University of Cambridge. Michael J. Hutchinson is supported by the EPSRC Centre for Doctoral Training in Modern Statistics and Statistical Machine Learning (EP/S023151/1).
I. Azangulov, A. Smolensky, A. Terenin, and V. Borovitskiy. Stationary Kernels and Gaussian Processes on Lie Groups and their Homogeneous Spaces I: the Compact Case, Aug. 2022. DOI: 10.48550/arXiv.2208.14960. Cited on pages 2, 6.
I. Azangulov, A. Smolensky, A. Terenin, and V. Borovitskiy. Stationary Kernels and Gaussian Processes on Lie Groups and their Homogeneous Spaces II: non-compact symmetric spaces, Apr. 2023. DOI: 10.48550/arXiv.2301.13088. Cited on pages 2, 6.
A. Bansal, E. Borgnia, H.-M. Chu, J. S. Li, H. Kazemi, F. Huang, M. Goldblum, J. Geiping, and T. Goldstein. Cold diffusion: inverting arbitrary image transforms without noise. arXiv preprint arXiv:2208.09392, 2022. Cited on page 6.
M. Bilos, K. Rasul, A. Schneider, Y. Nevmyvaka, and S. Gunnemann. Modeling Temporal Data as Continuous Functions with Process Diffusion, 2022. URL: https://openreview.net/forum? id=VmJKUypQ8wR. Cited on page 6.
S. Bond-Taylor and C. G. Willcocks. Infinite-diff: infinite resolution diffusion with subsampled mollified states. arXiv preprint arXiv:2303.18242, 2023. Cited on pages 1, 6.
J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang. JAX: composable transformations of Python+NumPy programs, version 0.2.5, 2018. URL: http://github.com/google/jax. Cited on pages 10, 13.
W. Bruinsma, S. Markou, J. Requeima, A. Y. K. Foong, A. Vaughan, T. Andersson, A. Buonomo, S. Hosking, and R. E. Turner. Autoregressive Conditional Neural Processes. In International Conference on Learning Representations, Feb. 2023. URL: https://openreview.net/forum?id=0AsXFPBfTBh. Cited on page 6.
W. Bruinsma, J. Requeima, A. Y. K. Foong, J. Gordon, and R. E. Turner. Gaussian Neural Processes. In 3rd Symposium on Advances in Approximate Bayesian Inference, Feb. 2020. Cited on pages 7, 14, 16.
P. Cattiaux, G. Conforti, I. Gentil, and C. Léonard. Time reversal of diffusion processes under a finite entropy condition. arXiv preprint arXiv:2104.07708, 2021. Cited on pages 2, 3, 5.
S. Chen, S. Chewi, J. Li, Y. Li, A. Salim, and A. R. Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint arXiv:2209.11215, 2022. Cited on page 11.
T. Cohen. Equivariant convolutional networks. PhD thesis, 2021. Cited on page 4.
G. Da Prato and J. Zabczyk. Stochastic equations in infinite dimensions. Cambridge university press, 2014. Cited on page 6.
A. S. Dalalyan. Theoretical guarantees for approximate sampling from smooth and log-concave densities. J. R. Stat. Soc. Ser. B. Stat. Methodol., 79(3):651-676, 2017. ISSN: 1369-7412. DOI: 10.1111/rssb.12183. URL: https://doi.org/10.1111/rssb.12183. Cited on page 13.
G. Daras, M. Delbracio, H. Talebi, A. G. Dimakis, and P. Milanfar. Soft diffusion: score matching for general corruptions. arXiv preprint arXiv:2209.05442, 2022. Cited on page 6.
1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020. Cited on page 13.
V. Voleti, C. Pal, and A. M. Oberman. Score-based Denoising Diffusion with Non-Isotropic Gaussian Noise Models, 2022. URL: https://openreview.net/forum?id=igC8cJKcbOQ. Cited on page 6.
M. Weiler, P. Forre, E. Verlinde, M. Welling, et al. Coordinate independent convolutional networks: isometry and gauge equivariant convolutions on riemannian manifolds, 2021. Cited on pages 4, 5, 7.
M. Weiler and G. Cesa. General  $\$ 123,456$ -Equivariant Steerable CNNs. arXiv:1911.08251 [cs, eess], Apr. 2021. URL: http://arxiv.org/abs/1911.08251. Cited on pages 8, 17.
M. Weiler, M. Geiger, M. Welling, W. Boomsma, and T. S. Cohen. 3D steerable CNNs: Learning rotationally equivariant features in volumetric data. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL: https://proceedings.neurips.cc/paper/2018/file/488e4104520c6aab692863cc1dba45af-Paper.pdf. Cited on pages 16, 17.
J. Wilson, V. Borovitskiy, A. Terenin, P. Mostowsky, and M. Deisenroth. Efficiently sampling functions from gaussian process posteriors. In ICML, pages 10292-10302. PMLR, 2020. Cited on page 2.
O. Yadan. Hydra - a framework for elegantly configuring complex applications. Github, 2019. Cited on pages 10, 13.
J. Yim, B. L. Trippe, V. De Bortoli, E. Mathieu, A. Doucet, R. Barzilay, and T. Jaakkola. Se (3) diffusion model with application to protein backbone generation. arXiv preprint arXiv:2302.02277, 2023. Cited on pages 4, 7.
A Organisation of appendices
In this supplementary, we first introduce in App. B an Ornstein Uhlenbeck process on function space (via finite marginals) along with several score approximations. Then in App. C, we show how this methodology extend to manifold-valued inputs or outputs. Later in App. D, we derive sufficient conditions for this introduced model to yield a group invariant process. What's more in App. E, we study some conditional sampling schemes. Eventually in App. F, we give a thorough description of experimental settings along with additional empirical results.
B Ornstein Uhlenbeck on function space
B.1 Multivariate Ornstein-Uhlenbeck process
First, we aim to show that we can define a stochastic process on an infinite dimensional function space, by defining the joint finite marginals  $\mathbf{Y}(x)$  as the solution of a multidimensional Ornstein-Uhlenbeck process. In particular, for any set of input  $x = (x_{1},\dots ,x_{k})\in \mathcal{X}^{k}$ , we define the joint marginal as the solution of the following SDE
$$
\mathrm {d} \tilde {\mathbf {Y}} _ {t} (x) = (m (x) - \tilde {\mathbf {Y}} _ {t} (x)) / 2 \beta_ {t} \mathrm {d} t + \sqrt {\beta_ {t} K (x , x)} \mathrm {d} \mathbf {B} _ {t}. \tag {8}
$$
Proposition B.1. (Phillips et al., 2022) We assume we are given a data process  $(\mathbf{Y}_0(x))_{x\in \mathcal{X}}$  and we denote by  $\mathbf{G}\sim \mathrm{GP}(0,k)$  a Gaussian process with zero mean and covariance. Then let's define
$$
\mathbf {Y} _ {t} \triangleq e ^ {- \frac {1}{2} \cdot \int_ {s = 0} ^ {t} \beta_ {s} d s} \mathbf {Y} _ {0} + \left(1 - e ^ {- \frac {1}{2} \cdot \int_ {s = 0} ^ {t} \beta_ {s} d s}\right) m + \left(1 - e ^ {- \int_ {s = 0} ^ {t} \beta_ {s} d s}\right) ^ {1 / 2} \mathbf {G}.
$$
Then  $(\mathbf{Y}_t(x))_{x\in \mathcal{X}}$  is a stochastic process (by virtue of being a linear combination of stochastic processes). We thus have that  $\mathbf{Y}_t\xrightarrow[ t\to 0]{a.s.}\mathbf{Y}_0$  and  $\mathbf{Y}_t\xrightarrow[ t\to \infty]{a.s.}\mathbf{Y}_\infty$  with  $\mathbf{Y}_\infty \sim \mathrm{GP}(m,k)$ , so effectively  $(\mathbf{Y}_t(x))_{t\in \mathbb{R}_+}, x\in \mathcal{X}$  interpolates between the data process and this limiting Gaussian process. Additionally,  $\mathcal{L}(\mathbf{Y}_t|\mathbf{Y}_0 = y_0) = \mathrm{GP}(m_t,K_t)$  with  $m_t = e^{-\frac{1}{2}\cdot \int_{s = 0}^t\beta_sd s}y_0 + \left(1 - e^{-\frac{1}{2}\cdot \int_{s = 0}^t\beta_sd s}\right)m$  and  $\Sigma_t = \left(1 - e^{-\int_{s = 0}^t\beta_sd s}\right)K$ . Furthermore,  $(\mathbf{Y}_t(x))_{t\in \mathbb{R}_+,x\in \mathcal{X}}$  is the solution of the SDE in (8).
Proof. We aim to compute the mean and covariance of the process  $(\mathbf{Y}_t)_{t\geq 0}$  described by the SDE (3). First let's recall the time evolution of the mean and covariance of the solution from a multivariate Ornstein-Uhlenbeck process given by
$$
\mathrm {d} \mathbf {Y} _ {t} = f (\mathbf {Y} _ {t}, t) \mathrm {d} t + L (\mathbf {Y} _ {t}, t) \mathrm {d} \mathbf {B} _ {t}. \tag {9}
$$
We know that the time evolution of the mean and the covariance are given respectively by Särkkä and Solin (2019)
$$
\frac {\mathrm {d} m _ {t}}{\mathrm {d} t} = \operatorname {E} [ f (\mathbf {Y} _ {t}, t) ] \tag {10}
$$