SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {\mathrm {d} \Sigma_ {t}}{\mathrm {d} t} = \mathrm {E} [ f (\mathbf {Y} _ {t}, t) (m _ {t} - \mathbf {Y} _ {t}) ^ {\top} ] + \mathrm {E} [ (m _ {t} - \mathbf {Y} _ {t}) f (\mathbf {Y} _ {t}, t) ^ {\top} ] + \mathrm {E} [ L (\mathbf {Y} _ {t}, t) L (\mathbf {Y} _ {t}, t) ^ {\top} ]. \tag {11}
$$
Plugging in the drift  $f(\mathbf{Y}_t, t) = 1/2 \cdot (m - \mathbf{Y}_t)\beta_t$  and diffusion term  $L(\mathbf{Y}_t, t) = \sqrt{\beta_tK}$  from (3), we get
$$
\frac {\mathrm {d} m _ {t}}{\mathrm {d} t} = 1 / 2 \cdot (m - \mathbf {Y} _ {t}) \beta_ {t} \tag {12}
$$
$$
\frac {\mathrm {d} \Sigma_ {t}}{\mathrm {d} t} = \beta_ {t} [ K - \Sigma_ {t} ]. \tag {13}
$$
Solving these two ODEs we get
$$
m _ {t} = e ^ {- \frac {1}{2} \cdot \int_ {s = 0} ^ {t} \beta_ {s} d s} m _ {0} + \left(1 - e ^ {- \frac {1}{2} \cdot \int_ {s = 0} ^ {t} \beta_ {s} d s}\right) m \tag {14}
$$
$$
\Sigma_ {t} = K + e ^ {- \int_ {s = 0} ^ {t} \beta_ {s} d s} (\Sigma_ {0} - K) \tag {15}
$$
with  $m_0 \triangleq \operatorname{E}[\mathbf{Y}_0]$  and  $\Sigma_0 \triangleq \operatorname{Cov}[\mathbf{Y}_0]$ .
Now let's compute the first two moments of  $(\mathbf{Y}_t(x))_{x\in \mathcal{X}}$ . We have
$$
\begin{array}{l} \operatorname {E} \left[ \mathbf {Y} _ {t} \right] = \operatorname {E} \left[ e ^ {- \frac {1}{2} \cdot \int_ {s = 0} ^ {t} \beta_ {s} d s} \mathbf {Y} _ {0} + \left(1 - e ^ {- \frac {1}{2} \cdot \int_ {s = 0} ^ {t} \beta_ {s} d s}\right) m + \left(1 - e ^ {- \frac {1}{2} \cdot \int_ {s = 0} ^ {t} \beta_ {s} d s}\right) \mathbf {G} \right] (16) \\ = e ^ {- \frac {1}{2} \cdot \int_ {s = 0} ^ {t} \beta_ {s} d s} m _ {0} + \left(1 - e ^ {- \frac {1}{2} \cdot \int_ {s = 0} ^ {t} \beta_ {s} d s}\right) m (17) \\ = m _ {t} (18) \\ \end{array}
$$
$$
\begin{array}{l} \operatorname {C o v} \left[ \mathbf {Y} _ {t} \right] = \operatorname {C o v} \left[ e ^ {- \frac {1}{2} \cdot \int_ {s = 0} ^ {t} \beta_ {s} d s} \mathbf {Y} _ {0} \right] + \operatorname {C o v} \left[ \left(1 - e ^ {- \int_ {s = 0} ^ {t} \beta_ {s} d s}\right) ^ {1 / 2} \mathbf {G} \right] (19) \\ = e ^ {- \int_ {s = 0} ^ {t} \beta_ {s} d s} \Sigma_ {0} + \left(1 - e ^ {- \int_ {s = 0} ^ {t} \beta_ {s} d s}\right) K (20) \\ = K + e ^ {- \int_ {s = 0} ^ {t} \beta_ {s} d s} (\Sigma_ {0} - K) (21) \\ = \Sigma_ {t}. (22) \\ \end{array}
$$
B.2 Conditional score
Hence, condition on  $\mathbf{Y}_0$  the score is the gradient of the log Gaussian characterised by mean  $m_{t|0} = e^{-\frac{1}{2} B(t)}\mathbf{Y}_0$  and  $\Sigma_{t|0} = (1 - e^{-B(t)})\mathrm{K}$  with  $B(t) = \int_0^t\beta (s)ds$  which can be derived from the above marginal mean and covariance with  $m_0 = \mathbf{Y}_0$  and  $\Sigma_0 = 0$ .
$$
\begin{array}{l} \nabla_ {\mathbf {Y} _ {t}} \log p _ {t} (\mathbf {Y} _ {t} | \mathbf {Y} _ {0}) = \nabla_ {\mathbf {Y} _ {t}} \log \mathcal {N} (\mathbf {Y} _ {t} | m _ {t | 0}, \Sigma_ {t | 0}) (23) \\ = \nabla_ {\mathbf {Y} _ {t}} - 1 / 2 \left(\mathbf {Y} _ {t} - m _ {t | 0}\right) ^ {\top} \Sigma_ {t | 0} ^ {- 1} \left(\mathbf {Y} _ {t} - m _ {t | 0}\right) + c (24) \\ = - \Sigma_ {t | 0} ^ {- 1} \left(\mathbf {Y} _ {t} - m _ {t | 0}\right) (25) \\ = - \mathrm {L} _ {t | 0} ^ {- \top} \mathrm {L} _ {t | 0} ^ {- 1} \mathrm {L} _ {t | 0} \epsilon (26) \\ = - \mathrm {L} _ {t | 0} ^ {- \top} \epsilon (27) \\ \end{array}
$$
where  $\mathrm{L}_{t|0}$  denotes the Cholesky decomposition of  $\Sigma_{t|0} = \mathrm{L}_{t|0}\mathrm{L}_{t|0}^{\top}$ , and  $\mathbf{Y}_t = m_{t|0} + \mathrm{L}_{t|0}\epsilon$ .
Then we can plugin our learnt (preconditioned) score into the backward SDE 4 which gives
$$
\left. \mathrm {d} \overline {{\mathbf {Y}}} _ {t} \right| x = \left[ - (m (x) - \overline {{\mathbf {Y}}} _ {t}) / 2 + \mathrm {K} (x, x) \nabla_ {\overline {{\mathbf {Y}}} _ {t}} \log p _ {T - t} (t, x, \overline {{\mathbf {Y}}} _ {t}) \right] \mathrm {d} t + \sqrt {\beta_ {t}} \mathrm {K} (x, x) \beta_ {t} \mathrm {d} \mathbf {B} _ {t} \tag {28}
$$
B.3 Several score parametrisations
In this section, we discuss several parametrisations of the neural network and the objective.
For the sake of versatility, we opt to employ the symbol  $D_{\theta}$  for the network instead of  $s_{\theta}$  as mentioned in the primary text, as it allows us to approximate not only the score but also other quantities from which the score can be derived. In full generality, we use a residual connection, weighted by  $c_{\mathrm{out}}, c_{\mathrm{skip}}: \mathbb{R} \to \mathbb{R}$ , to parameterise the network
$$
D _ {\theta} (t, \boldsymbol {Y} _ {t}) = c _ {\text {s k i p}} (t) \boldsymbol {Y} _ {t} + c _ {\text {o u t}} (t) F _ {\theta} (t, \boldsymbol {Y} _ {t}). \tag {29}
$$
We recall that the input to the network is time  $t$ , and the noised vector  $\mathbf{Y}_t = \boldsymbol{\mu}_{t|0} + \boldsymbol{n}$ , where  $\boldsymbol{\mu}_{t|0} = \mathrm{e}^{-B(t) / 2}\mathbf{Y}_0$  and  $\boldsymbol{n} \sim \mathcal{N}(0, \Sigma_{t|0})$  with  $\Sigma_{t|0} = (1 - \mathrm{e}^{-B(t)})K$ . The gram matrix  $K$  corresponds to  $k(X, X)$  with  $k$  the limiting kernel. We denote by  $\mathrm{L}_{t|0}$  and  $\mathrm{S}$  respectively the Cholesky decomposition of  $\Sigma_{t|0} = \mathrm{L}_{t|0}\mathrm{L}_{t|0}^{\top}$  and  $\mathrm{K} = \mathrm{SS}^{\top}$ .
The denoising score matching loss weighted by  $\Lambda (t)$  is given by
$$
\mathcal {L} (\theta) = \mathbb {E} \left[ \| D _ {\theta} (t, \boldsymbol {Y} _ {t}) - \nabla_ {\boldsymbol {Y} _ {t}} \log p _ {t} \left(\boldsymbol {Y} _ {t} \mid \boldsymbol {Y} _ {0}\right) \| _ {\Lambda (t)} ^ {2} \right] \tag {30}
$$
Table 3: Summary of different score parametrisations as well as the values for  $c_{\mathrm{skip}}$  and  $c_{\mathrm{out}}$  that we found to be optimal, based on the recommendation from Karras et al. (2022, Appendix B.6).
No precond.	Precond. K	Precond. S^T	Predict Y0
cskip	0	0	0	1
cout	(σt|0+10-3)-1	(σt|0+10-3)-1	(σt|0+10-3)-1	1
Loss	||σt|0S^T D_θ + z||_2^2	||σt|0D_θ + Sz||_2^2	||σt|0D_θ + z||_2^2	||D_θ - Y0||_2^2
KV∇ log pt	KDθ	D_θ	SDθ	-Σt|0-1(Yt-e-B(t)/2D_θ)
No preconditioning By reparametrisation, let  $\mathbf{Y}_t = \pmb{\mu}_{t|0} + L_{t|0}\pmb{z}$ , where  $\pmb{z} \sim \mathcal{N}(\mathbf{0},\mathbf{I})$ , the loss from Eq. (30) can be written as
$$
\mathcal {L} (\theta) = \mathbb {E} \left[ \| D _ {\theta} (t, \boldsymbol {Y} _ {t}) + \Sigma_ {t | 0} ^ {- 1} \left(\boldsymbol {Y} _ {t} - \boldsymbol {\mu} _ {t | 0}\right) \| _ {\Lambda (t)} ^ {2} \right] \tag {31}
$$
$$
= \mathbb {E} \left[ \| D _ {\theta} (t, \boldsymbol {Y} _ {t}) + \Sigma_ {t | 0} ^ {- 1} L _ {t | 0} \boldsymbol {z} \| _ {\Lambda (t)} ^ {2} \right] \tag {32}
$$
$$
= \mathbb {E} \left[ \| D _ {\theta} (t, \boldsymbol {Y} _ {t}) + L _ {t | 0} ^ {- \top} \boldsymbol {z} \| _ {\Lambda (t)} ^ {2} \right] \tag {34}
$$
Choosing  $\Lambda (t) = \Sigma_{t|0} = L_{t|0}L_{t|0}^{\top}$  we obtain
$$
\mathcal {L} (\theta) = \mathbb {E} \left[ \| L _ {t | 0} ^ {\top} D _ {\theta} (t, \boldsymbol {Y} _ {t}) + \boldsymbol {z} \| _ {2} ^ {2} \right] \tag {35}
$$
$$
= \mathbb {E} \left[ \| \sigma_ {t | 0} S ^ {\top} D _ {\theta} (t, Y _ {t}) + z \| _ {2} ^ {2} \right]. \tag {36}
$$
Preconditioning by  $K$  Alternatively, one can train the neural network to approximate the preconditioned score  $D_{\theta} \approx \mathbf{K}\nabla_{\mathbf{Y}_{t}}\log p_{t}(\mathbf{Y}_{t}|\mathbf{Y}_{0})$ . The loss, weighted by  $\Lambda = \sigma_{t|0}^{2}\mathbf{I}$ , is then given by
$$
\mathcal {L} (\theta) = \mathbb {E} \left[ \| D _ {\theta} (t, \boldsymbol {Y} _ {t}) + K L _ {t | 0} ^ {- \top} \boldsymbol {z} \| _ {\Lambda (t)} ^ {2} \right] \tag {37}
$$
$$
= \mathbb {E} \left[ \| D _ {\theta} (t, \boldsymbol {Y} _ {t}) + \sigma_ {t | 0} ^ {- 1} S \boldsymbol {z} \| _ {\Lambda (t)} ^ {2} \right] \tag {38}
$$
$$
= \mathbb {E} \left[ \| \sigma_ {t | 0} D _ {\theta} (t, \boldsymbol {Y} _ {t}) + S \boldsymbol {z} \| _ {2} ^ {2} \right]. \tag {39}
$$
Precondition by  $S^{\top}$  A variation of the previous one, is to precondition the score by the transpose Cholesky of the limiting kernel gram matrix, such that  $D_{\theta} \approx S^{\top} \nabla_{\mathbf{Y}_t} \log p_t(\mathbf{Y}_t | \mathbf{Y}_0)$ .
The loss, weighted by  $\Lambda = \sigma_{t|0}^2\mathbf{I}$ , becomes
$$
\mathcal {L} (\theta) = \mathbb {E} \left[ \| D _ {\theta} (t, \boldsymbol {Y} _ {t}) + S ^ {\top} L _ {t | 0} ^ {- \top} \boldsymbol {z} \| _ {\Lambda (t)} ^ {2} \right] \tag {40}
$$
$$
= \mathbb {E} \left[ \| D _ {\theta} (t, \boldsymbol {Y} _ {t}) + \sigma_ {t | 0} ^ {- 1} \boldsymbol {z} \| _ {\Lambda (t)} ^ {2} \right] \tag {41}
$$
$$
= \mathbb {E} \left[ \| \sigma_ {t | 0} D _ {\theta} (t, \boldsymbol {Y} _ {t}) + \boldsymbol {z} \| _ {2} ^ {2} \right]. \tag {42}
$$
Predicting  $\mathbf{Y}_0$  Finally, an alternative strategy is to predict  $\mathbf{Y}_0$  from a noised version  $\mathbf{Y}_t$ . In this case, the loss takes the simple form
$$
\mathcal {L} (\theta) = \mathbb {E} \left[ \left\| D _ {\theta} (t, \boldsymbol {Y} _ {t}) - \boldsymbol {Y} _ {0} \right\| _ {2} ^ {2} \right].
$$
The score can be computed from the network's prediction following
$$
\nabla \log p _ {t} \left(\boldsymbol {Y} _ {t} \mid \boldsymbol {Y} _ {0}\right) = - \Sigma_ {t | 0} ^ {- 1} \left(\boldsymbol {Y} _ {t} - \boldsymbol {\mu} _ {t | 0}\right) \tag {43}
$$
$$
= - \Sigma_ {t | 0} ^ {- 1} \left(\boldsymbol {Y} _ {t} - \mathrm {e} ^ {- B (t) / 2} \boldsymbol {Y} _ {0}\right) \tag {44}
$$
$$
\approx - \Sigma_ {t | 0} ^ {- 1} \left(\boldsymbol {Y} _ {t} - \mathrm {e} ^ {- B (t) / 2} D _ {\theta} (t, \boldsymbol {Y} _ {t})\right) \tag {45}
$$
Table 3 summarises the different options for parametrising the score as well as the values for  $c_{\mathrm{skip}}$  and  $c_{\mathrm{out}}$  that we found to be optimal, based on the recommendation from Karras et al. (2022, Appendix B.6). In practice, we found the precondition by  $K$  parametrisation to produce the best results, but we refer to App. F.1.3 for a more in-depth ablation study.
B.4 Exact (marginal) score in Gaussian setting
Interpolating between Gaussian processes  $GP(m_0, \Sigma_0)$  and  $GP(m, \mathrm{K})$
$$
\begin{array}{l} \mathrm {K} \nabla_ {\mathbf {Y} _ {t}} \log p _ {t} (\mathbf {Y} _ {t}) = - \mathrm {K} \Sigma_ {t} ^ {- 1} (\mathbf {Y} _ {t} - m _ {t}) (47) \\ = - \mathrm {K} \left[ \mathrm {K} + e ^ {- \int_ {s = 0} ^ {t} \beta_ {s} d s} \left(\Sigma_ {0} - \mathrm {K}\right) \right] ^ {- 1} \left(\mathbf {Y} _ {t} - m _ {t}\right) (48) \\ = - \mathrm {K} \left(\mathrm {L} _ {t} L _ {t} ^ {\top}\right) ^ {- 1} \left(\mathbf {Y} _ {t} - m _ {t}\right) (49) \\ = - \mathrm {K L} _ {t} ^ {\top - 1} \mathrm {L} _ {t} ^ {- 1} \left(\mathbf {Y} _ {t} - m _ {t}\right) (50) \\ \end{array}
$$
with  $\Sigma_{t} = \mathrm{K} + e^{-\int_{s=0}^{t} \beta_{s} ds} (\Sigma_{0} - \mathrm{K}) = \mathrm{L}_{t} \mathrm{L}_{t}^{\top}$  obtained via Cholesky decomposition.
B.5 Langevin dynamics
Under mild assumptions on  $\nabla \log p_{T - t}$  (Durmus and Moulines, 2016) the following SDE
$$
\mathrm {d} \mathbf {Y} _ {s} = \frac {1}{2} \mathrm {K} \nabla \log p _ {T - t} (\mathbf {Y} _ {s}) \mathrm {d} s + \sqrt {\mathrm {K}} \mathrm {d} \mathbf {B} _ {s} \tag {52}
$$
admits a solution  $(\mathbf{Y}_s)_{s\geq 0}$  whose law  $\mathcal{L}(\mathbf{Y}_s)$  converges with geometric rate to  $p_{T - t}$  for any invertible matrix K.
B.6 Likelihood evaluation
Similarly to Song et al. (2021), we can derive a deterministic process which has the same marginal density as the SDE (3), which is given by the following Ordinary Differential Equation (ODE)—referred as the probability flow ODE
$$
\mathrm {d} \binom {\mathbf {Y} _ {t} (x)} {\log p _ {t} (\mathbf {Y} _ {t} (x))} = \binom {\frac {1}{2} \left\{m (x) - \mathbf {Y} _ {t} (x) - \mathrm {K} (x, x) \nabla \log p _ {t} (\mathbf {Y} _ {t} (x)) \right\} \beta_ {t}} {- \frac {1}{2} \operatorname {d i v} \left\{m (x) - \mathbf {Y} _ {t} (x) - \mathrm {K} (x, x) \nabla \log p _ {t} (\mathbf {Y} _ {t} (x)) \right\} \beta_ {t}} \mathrm {d} t. \tag {53}
$$
Once the score network is learnt, we can thus use it in conjunction with an ODE solver to compute the likelihood of the model.
B.7 Discussion consistency
So far we have defined a generative model over functions via its finite marginals  $\bar{\mathbf{Y}}_T^\theta (x)$ . These finite marginals were to arise from a stochastic process if, as per the Kolmogorov extension theorem (Øksendal, 2003), they satisfy exchangeability and consistency conditions. Exchangeability can be satisfied by parametrising the score network such that the score network is equivariant w.r.t permutation, i.e.  $\mathbf{s}_{\theta}(t,\sigma \circ x,\sigma \circ y) = \sigma \circ \mathbf{s}_{\theta}(t,x,y)$  for any  $\sigma \in \Sigma_{n}$ . Additionally, we have that the noising process  $(\mathbf{Y}_t(x))_{x\in \mathcal{X}}$  is trivially consistent for any  $t\in \mathbb{R}_+$  since it is a stochastic process as per Prop. B.1, and consequently so is the (true) time-reversal  $(\bar{\mathbf{Y}}_t(x))_{x\in \mathcal{X}}$ . Yet, when approximating the score  $\mathbf{s}_{\theta}\approx \nabla \log p_t$ , we lose the consistency over the generative process  $\bar{\mathbf{Y}}_t^\theta (x)$  as the constraint on the score network is non trivial to satisfy. This is actually a really strong constraint on the model class, and as soon as one goes beyond linearity (of the posterior w.r.t. the context set), it is non trivial to enforce without directly parameterising a stochastic process, e.g. as Phillips et al. (2022). There thus seems to be a strong trade-off between satisfying consistency, and the model's ability to fit complex process and scale to large datasets.
C Manifold-valued diffusion process
C.1 Manifold-valued inputs
In the main text we dealt with a simplified case of tensor fields where the tensor fields are over Euclidean space. Nevertheless, it is certainly possible to apply our methods to these settings. Significant work has been done on performing convolutions on feature fields on generic manifolds (a superset of tensor fields on generic manifolds), core references being (Cohen, 2021) for the case of homogeneous spaces and (Weiler et al., 2021) for more general Riemannian manifolds. We
recommend these as excellent mathematical introductions to the topic and build on them to describe how to formulate diffusion models over these spaces.
Tensor fields as sections of bundles. Formally the fields we are interested in modelling are sections  $\sigma$  of associated tensor bundles of the principle  $G$ -bundle on a manifold  $M$ . We shall denote such a bundle  $BM$  and the space of sections  $\Gamma(BM)$ . The goal, therefore, is to model random elements from this space of sections. For a clear understanding of this definition, please see Weiler et al. (2021, pages 73-95) for an introduction suitable to ML audiences. Prior work looking at this setting is (Hutchinson et al., 2021) where they construct Gaussian Processes over tensor fields on manifolds.
Stochastic processes on spaces of sections. Given we can see sections as maps  $\sigma : M \to BM$ , where an element in  $BM$  is a tuple  $(m, b)$ ,  $m$  in the base manifold and  $b$  in the typical fibre, alongside the condition that the composition of the projection  $\mathrm{proj}_i : (m, b) \mapsto m$  with the section is the identity,  $\mathrm{proj}_i \circ \sigma = \mathrm{Id}$  it is clear we can see distribution over sections as stochastic processes with index set the manifold  $M$ , and output space a point in the bundle  $BM$ , with the projection condition satisfied. The projection onto finite marginals, i.e., a finite set of points in the manifold, is defined as  $\pi_{m_1, \ldots, m_n}(\sigma) = (\sigma(m_1), \ldots, \sigma(m_n))$ .
Noising process. To define a noising process over these marginals, we can use Gaussian Processes defined in (Hutchinson et al., 2021) over the tensor fields. The convergence results of Phillips et al. (2022) hold still, and so using these Gaussian Processes as noising processes on the marginals also defines a noising process on the whole section.
Reverse process. The results of (Cattiaux et al., 2021) are extremely general and continue to hold in this case of SDEs on the space of sections. Note we don't actually need this to be the case, we can just work with the reverse process on the marginals themselves, which are much simpler objects. It is good to know that it is a valid process on full sections though should one want to try and parameterise a score function on the whole section akin to some other infinite-dimension diffusion models.
Score function. The last thing to do therefore is parameterise the score function on the marginals. If we were trying to parameterise the score function over the whole section at once (akin to a number of other works on infinite dimension diffusions), this could present some problems in enforcing the smoothness of the score function. As we only deal with the score function on a finite set of marginals, however, we need not deal with this issue and this presents a distinct advantage in simplicity for our approach. All we need to do is pick a way of numerically representing points on the manifold and b) pick a basis for the tangent space of each point on the manifold. This lets us represent elements from the tangent space numerically, and therefore also elements from tensor space at each point numerically as well. This done, we can feed these to a neural network to learn to output a numerical representation of the score on the same basis at each point.
C.2 Manifold-valued outputs
In the setting, where one aim to model a stochastic process with manifold codomain  $\mathbf{Y}_t(x) = (\mathbf{Y}_t(x_1),\dots ,\mathbf{Y}_t(x_n))\in \mathcal{M}^n$ , things are less trivial as manifolds do not have a vector space structure which is necessary to define Gaussian processes. Fortunately, We can still target a known distribution marginally independently on each marginal, since this is well defined, and as such revert to the Riemannian diffusion models introduced in De Bortoli et al. (2021) with  $n$  independent Langevin noising processes
$$
\mathrm {d} \mathbf {Y} _ {t} (x _ {k}) = - \frac {1}{2} \nabla U (\mathbf {Y} _ {t} (x _ {k})) \beta_ {t} \mathrm {d} t + \sqrt {\beta_ {t}} \mathrm {d} \mathbf {B} _ {t} ^ {\mathcal {M}}. \tag {54}
$$
are applied to each marginal. Hence in the limit  $t \to \infty$ ,  $\mathbf{Y}_t(x)$  has density (assuming it exists) which factors as  $dp / d\mathrm{Vol}_{\mathcal{M}}((y(x_1), \dots, y(x_n))) \propto \prod_k e^{-U(y(x_n)))}$ . For compact manifolds, we can target the uniform distribution by setting  $U(x) = 0$ . The reverse time process will have correlation between different marginals, and so the score function still needs to be a function of all the points in the marginal of interest.
D Invariant neural diffusion processes
D.1  $\operatorname{E}(n)$ -equivariant kernels
A kernel  $k: \mathbb{R}^d \times \mathbb{R}^d \to \mathbb{R}^{d \times d}$  is equivariant if it satisfies the following constraints: (a)  $k$  is stationary, that is if for all  $x, x' \in \mathbb{R}^n$
$$
k \left(x, x ^ {\prime}\right) = k \left(x - x ^ {\prime}\right) \triangleq \tilde {k} \left(x - x ^ {\prime}\right) \tag {55}
$$
and if (b) it satisfies the angular constraint for any  $h \in H$
$$
k \left(h x, h x ^ {\prime}\right) = \rho (h) k \left(x, x ^ {\prime}\right) \rho (h) ^ {\top}. \tag {56}
$$
A trivial example of such an equivariant kernel is the diagonal kernel  $k(x, x') = k_0(x, x') \mathrm{I}$  (Holderrieth et al., 2021), with  $k_0$  stationary. This kernel can be understood having  $d$  independent Gaussian process uni-dimensional output, that is, there is no inter-dimensional correlation.
Less trivial examples, are the  $\mathrm{E}(n)$  equivariant kernels proposed in Macêdo and Castro (2010). Namely curl-free and divergence-free kernels, allowing for instance to model electric or magnetic fields. Formally we have  $k_{\mathrm{curl}} = k_0A$  and  $k_{\mathrm{div}} = k_0B$  with  $k_0$  stationary, e.g. squared exponential kernel  $k_0(x,x') = \sigma^2\exp \left(\frac{\|x - x'\|^2}{2l^2}\right)$ , and  $A$  and  $B$  given by
$$
A \left(x, x ^ {\prime}\right) = I - \frac {\left(x - x ^ {\prime}\right) \left(x - x ^ {\prime}\right) ^ {\top}}{l ^ {2}} \tag {57}
$$
$$
B (x, x ^ {\prime}) = \frac {(x - x ^ {\prime}) (x - x ^ {\prime}) ^ {\top}}{l ^ {2}} + \left(n - 1 - \frac {\| x - x ^ {\prime} \| ^ {2}}{l ^ {2}}\right) I. \tag {58}
$$
See Holderrieth et al. (Appendix C, 2021) for a proof.
D.2 Proof of Prop. 3.2
Below we give two proofs for the group invariance of the generative process, one via the probability flow ODE and one directly via Fokker-Planck.
Proof. Reverse ODE. The reverse probability flow associated with the forward SDE (3) with approximate score  $\mathbf{s}_{\theta}(t,\cdot)\approx \nabla \log p_t$  is given by
$$
\mathrm {d} \bar {\mathbf {Y}} _ {t} | x = \frac {1}{2} \left[ - m (x) + \bar {\mathbf {Y}} _ {t} + \mathrm {K} (x, x) \mathbf {s} _ {\theta} (T - t, x, \bar {\mathbf {Y}} _ {t}) \right] \mathrm {d} t \tag {59}
$$
$$
\triangleq b _ {\mathrm {O D E}} (t, x, \bar {\mathbf {Y}} _ {t}) \mathrm {d} t \tag {60}
$$
This ODE induces a flow  $\phi_t^b: X^n \times Y^n \to \mathrm{TY}^n$  for a given integration time  $t$ , which is said to be  $G$ -equivariant if the vector field is  $G$ -equivariant itself, i.e.  $b(t, g \cdot x, \rho(g)\bar{\mathbf{Y}}_t) = \rho(g)b(t, x, \bar{\mathbf{Y}}_t)$ . We have that for any  $g \in G$
$$
b _ {\mathrm {O D E}} (t, g \cdot x, \rho (g) \bar {\mathbf {Y}} _ {t}) = \frac {1}{2} [ - m (g \cdot x) + \rho (g) \mathbf {Y} _ {t} + \mathrm {K} (g \cdot x, g \cdot x) \mathbf {s} _ {\theta} (t, g \cdot x, \rho (g) \bar {\mathbf {Y}} _ {t}) ] \tag {61}
$$
$$
\stackrel {(1)} {=} \frac {1}{2} \left[ - \rho (g) m (x) + \rho (g) \mathbf {Y} _ {t} + \rho (g) \mathrm {K} (x, x) \rho (g) ^ {\top} \mathbf {s} _ {\theta} (t, g \cdot x, \rho (g) \bar {\mathbf {Y}} _ {t}) \right] \tag {62}
$$
$$
\stackrel {(2)} {=} \frac {1}{2} \left[ - \rho (g) m (x) + \rho (g) \mathbf {Y} _ {t} + \rho (g) K (x, x) \rho (g) ^ {\top} \rho (g) \mathbf {s} _ {\theta} (t, x, \bar {\mathbf {Y}} _ {t}) \right] \tag {63}
$$
$$
\stackrel {(3)} {=} \frac {1}{2} \rho (g) [ - m (x) + \mathbf {Y} _ {t} + \mathrm {K} (x, x) \mathbf {s} _ {\theta} (t, x, \bar {\mathbf {Y}} _ {t}) ] \tag {64}
$$
$$
= \rho (g) b _ {\mathrm {O D E}} (t, x, \bar {\mathbf {Y}} _ {t}) \tag {65}
$$
with (1) from the  $G$ -invariant prior GP conditions on  $m$  and  $k$ , (2) assuming that the score network is  $G$ -equivariant and (3) assuming that  $\rho(g) \in O(n)$ . To prove the opposite direction, we can simply follow these computations backwards. Finally, we know that with a  $G$ -invariant probability measure  $p_{\mathrm{ref}}$  and  $G$ -equivariant map  $\phi$ , the pushforward probability measure  $p_{\mathrm{ref}}^{-1} \circ \phi$  is also  $G$ -invariant (Köhler et al., 2020; Papamakarios et al., 2019). Assuming a  $G$ -invariant prior GP, and a  $G$ -equivariant score network, we thus have that the generative model from Sec. 3 defines marginals that are  $G$ -invariant.
Proof. Reverse SDE. The reverse SDE associated with the forward SDE (3) with approximate score  $\mathbf{s}_{\theta}(t,\cdot)\approx \nabla \log p_t$  is given by
$$
\mathrm {d} \bar {\mathbf {Y}} _ {t} | x = \left[ - (m (x) - \bar {\mathbf {Y}} _ {t}) / 2 + \mathrm {K} (x, x) \mathbf {s} _ {\theta} (T - t, x, \bar {\mathbf {Y}} _ {t}) \right] \mathrm {d} t + \sqrt {\beta_ {t} \mathrm {K} (x , x)} \mathrm {d} \mathbf {B} _ {t} \tag {66}
$$
$$
\triangleq b _ {\mathrm {S D E}} (t, x, \bar {\mathbf {Y}} _ {t}) \mathrm {d} t + \Sigma^ {1 / 2} (t, x) \mathrm {d} \mathbf {B} _ {t}. \tag {67}
$$
As for the probability flow drift  $b_{\mathrm{ODE}}$ , we have that  $b_{\mathrm{SDE}}$  is similarly  $G$ -equivariant, that is  $b_{\mathrm{SDE}}(t,g\cdot x,\rho (g)\bar{\mathbf{Y}}_t) = \rho (g)b_{\mathrm{SDE}}(t,x,\bar{\mathbf{Y}}_t)$  for any  $g\in G$ . Additionally, we have that diffusion matrix is also  $G$ -equivariant as for any  $g\in G$  we have  $\Sigma (t,g\cdot x) = \beta_t\mathrm{K}(g\cdot x,g\cdot x) = \beta_t\rho (g)\mathrm{K}(x,x)\rho (g)^\top = \rho (g)\Sigma (t,x)\rho (g)^\top$  since  $\mathrm{K}$  is the gram matrix of an  $G$ -equivariant kernel  $k$ .