SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} k _ {\mathrm {c o s}, \mathrm {I d}, \Phi} (\pmb {\theta} _ {i}, \pmb {\theta} _ {j}) = \mathbb {E} _ {\mathbf {x}} \big [ \cos (\pmb {w} _ {i} ^ {\top} \mathbf {x} + b _ {i}) \cos (\pmb {w} _ {j} ^ {\top} \mathbf {x} + b _ {j}) \big ], \quad \mathbf {x} \sim \mathcal {N} (\mathbf {0}, \pmb {I}) \\ = \frac {1}{2} \mathbb {E} _ {\mathbf {x}} \left[ \cos \left(\left(\boldsymbol {w} _ {i} - \boldsymbol {w} _ {j}\right) ^ {\top} \mathbf {x} + \left(b _ {i} - b _ {j}\right)\right) + \cos \left(\left(\boldsymbol {w} _ {i} + \boldsymbol {w} _ {j}\right) ^ {\top} \mathbf {x} + \left(b _ {i} + b _ {j}\right)\right) \right] \\ = \frac {1}{2} \cos | b _ {i} - b _ {j} | \exp \left(- \frac {1}{2} \| \boldsymbol {w} _ {i} - \boldsymbol {w} _ {j} \| ^ {2}\right) + \cos | b _ {i} + b _ {j} | \exp \left(- \frac {1}{2} \| \boldsymbol {w} _ {i} + \boldsymbol {w} _ {j} \| ^ {2}\right). \\ \end{array}
$$
Proof. This is immediate from the expected value of a product of two correlated Gaussians,  $\pmb{w}_i^\top \pmb{x} + b_i$  and  $\pmb{w}_j^\top \pmb{x} + b_j$ .
Kernel 5. The kernel  $k_{\mathrm{Snake_a}(\cdot) - \frac{1}{2a},\mathrm{Id},\Phi}(\pmb{\theta}_i,\pmb{\theta}_j)$  is equal to
$$
\begin{array}{l} \frac {1}{4 a ^ {2}} k _ {\cos , \operatorname {I d}, \Phi} (2 a \boldsymbol {\theta} _ {i}, 2 a \boldsymbol {\theta} _ {j}) + \boldsymbol {w} _ {j} ^ {\top} \boldsymbol {w} _ {j} \left(\sin (2 a b _ {j}) e ^ {- 2 a ^ {2} \| \boldsymbol {w} _ {j} \| ^ {2}} + \sin (2 a b _ {i}) e ^ {- 2 a ^ {2} \| \boldsymbol {w} _ {i} \| ^ {2}}\right) \\ - \frac {b _ {i}}{2 a} \cos (2 a b _ {j}) e ^ {- 2 a ^ {2} \| \boldsymbol {w} _ {j} \| ^ {2}} - \frac {b _ {j}}{2 a} \cos (2 a b _ {i}) e ^ {- 2 a ^ {2} \| \boldsymbol {w} _ {i} \| ^ {2}} + k _ {\mathrm {I d , I d ,} \Phi} (\boldsymbol {\theta} _ {i} ^ {(1)}, \boldsymbol {\theta} _ {j} ^ {(1)}). \\ \end{array}
$$
Proof. Choosing  $\sigma = \mathrm{Snake}_{\mathrm{a}}(\cdot) - \frac{1}{2a}$  in (4), and expanding the resulting quadratic,
$$
\begin{array}{l} k _ {\mathrm {S n a k e} _ {\mathrm {a}} (\cdot) - \frac {1}{2 a}, \mathrm {I d}} (\pmb {\theta} _ {i}, \pmb {\theta} _ {j}) \\ = \underbrace {\frac {1}{4 a ^ {2}} \mathbb {E} _ {\mathbf {x}} \left[ \cos \left(2 a \left(\boldsymbol {w} _ {i} ^ {\top} \mathbf {x} + b _ {i}\right)\right) \cos \left(2 a \left(\boldsymbol {w} _ {j} ^ {\top} \mathbf {x} + b _ {j}\right)\right) \right]} _ {\text {K e r n e l 2}} - \underbrace {\frac {1}{2 a} \mathbb {E} _ {\mathbf {x}} \left[ \left(\boldsymbol {w} _ {i} ^ {\top} \mathbf {x} + b _ {i}\right) \cos \left(2 a \left(\boldsymbol {w} _ {j} ^ {\top} \mathbf {x} + b _ {j}\right)\right) \right]} _ {\triangleq A} - \\ \underbrace {\frac {1}{2 a} \mathbb {E} _ {\mathbf {x}} \left[ \cos \left(2 a \left(\boldsymbol {w} _ {i} ^ {\top} \mathbf {x} + b _ {i}\right)\right) \left(\boldsymbol {w} _ {j} ^ {\top} \mathbf {x} + b _ {j}\right) \right]} _ {\triangleq B} + \underbrace {\mathbb {E} _ {\mathbf {x}} \left[ \left(\boldsymbol {w} _ {i} ^ {\top} \mathbf {x} + b _ {i}\right) \left(\boldsymbol {w} _ {j} ^ {\top} \mathbf {x} + b _ {j}\right) \right]} _ {\text {K e r n e l 4}}. \tag {15} \\ \end{array}
$$
We now evaluate  $A$  and  $B$ . The terms  $A$  and  $B$  obey a symmetry, so it suffices to evaluate term  $A$ . Term  $A$  can be evaluated using Stein's lemma,
$$
\begin{array}{l} A = \frac{1}{2a}\mathbb{E}\bigl[(z_{1} + b_{i})\cos (z_{2} + 2ab_{j})\bigr],\quad (z_{1},z_{2})^{\top}\sim \mathcal{N}\Bigg(\left( \begin{array}{c}0\\ 0 \end{array} \right),\left( \begin{array}{cc}\boldsymbol{w}_{i}^{\top}\boldsymbol{w}_{i} & 2a\boldsymbol{w}_{i}^{\top}\boldsymbol{w}_{j}\\ 2a\boldsymbol{w}_{j}^{\top}\boldsymbol{w}_{i} & 4a^{2}\boldsymbol{w}_{j}^{\top}\boldsymbol{w}_{j} \end{array} \right)\Bigg) \\ = \frac {1}{2 a} \mathbb {E} \left[ z _ {1} \cos \left(z _ {2} + 2 a b _ {j}\right) \right] + \frac {b _ {i}}{2 a} \mathbb {E} \left[ \cos \left(z _ {2} + 2 a b _ {j}\right) \right] \\ = - \boldsymbol {w} _ {i} ^ {\top} \boldsymbol {w} _ {j} \mathbb {E} \big [ \sin (z _ {2} + 2 a b _ {j}) \big ] + \frac {b _ {i}}{2 a} \mathbb {E} \big [ \cos (z _ {2} + 2 a b _ {j}) \big ] \\ = - \boldsymbol {w} _ {i} ^ {\top} \boldsymbol {w} _ {j} \sin (2 a b _ {j}) \exp (- 2 a ^ {2} \| \boldsymbol {w} _ {j} \| ^ {2}) + \frac {b _ {i}}{2 a} \cos (2 a b _ {j}) \exp (- 2 a ^ {2} \| \boldsymbol {w} _ {j} \| ^ {2}). \\ \end{array}
$$
Assembling all the known individual terms in (15),
$$
\begin{array}{l} k _ {\mathrm {S n a k e} _ {\mathrm {a}} (\cdot) - \frac {1}{2 \alpha}, \mathrm {I d}} (\boldsymbol {\theta} _ {i}, \boldsymbol {\theta} _ {j}) \\ = \frac {1}{4 a ^ {2}} k _ {\cos , \operatorname {I d}} \left(2 a \boldsymbol {\theta} _ {i}, 2 a \boldsymbol {\theta} _ {j}\right) + \boldsymbol {w} _ {i} ^ {\top} \boldsymbol {w} _ {j} \left(\sin \left(2 a b _ {j}\right) \exp \left(- 2 a ^ {2} \| \boldsymbol {w} _ {j} \| ^ {2}\right) + \sin \left(2 a b _ {i}\right) \exp \left(- 2 a ^ {2} \| \boldsymbol {w} _ {i} \| ^ {2}\right)\right) \\ - \frac {b _ {i}}{2 a} \cos (2 a b _ {j}) \exp (- 2 a ^ {2} \| \boldsymbol {w} _ {j} \| ^ {2}) - \frac {b _ {j}}{2 a} \cos (2 a b _ {i}) \exp (- 2 a ^ {2} \| \boldsymbol {w} _ {i} \| ^ {2}) + k _ {\mathrm {I d}, \mathrm {I d}} (\boldsymbol {\theta} _ {i} ^ {(1)}, \boldsymbol {\theta} _ {j} ^ {(1)}). \\ \end{array}
$$
Kernel 6. The kernel  $k_{\mathrm{Snake_a,Id},\Phi}(\pmb {\theta}_i,\pmb {\theta}_j)$  is equal to
$$
\begin{array}{l} \frac {1}{2 a} \left(b _ {i} - \frac {1}{2 a} \cos (2 a b _ {i}) \exp (- 2 a ^ {2} \| \boldsymbol {w} _ {i} \| ^ {2}) + b _ {j} - \frac {1}{2 a} \cos (2 a b _ {j}) \exp (- 2 a ^ {2} \| \boldsymbol {w} _ {j} \| ^ {2})\right) \\ + k _ {\text {S n a k e} _ {\mathrm {a}} (\cdot) - \frac {1}{2 a}, \text {I d ,} \Phi} (\boldsymbol {\theta} _ {i}, \boldsymbol {\theta} _ {j}) + \frac {1}{4 a ^ {2}}. \\ \end{array}
$$
Proof. Choose  $\sigma = \mathrm{Snake}_{\mathrm{a}}$  and note that  $\mathrm{Snake}_{\mathrm{a}}(\cdot) = \left(\mathrm{Snake}_{\mathrm{a}}(\cdot) - \frac{1}{2a}\right) + \frac{1}{2a}$ . The Kernel 5 corresponds with the case  $\left(\mathrm{Snake}_{\mathrm{a}}(\cdot) - \frac{1}{2a}\right)$ , so we are left with three additional terms. These terms may be evaluated directly,
$$
\begin{array}{l} k _ {\text {S n a k e} _ {\mathrm {a}}, \mathrm {I d}} (\boldsymbol {\theta} _ {i}, \boldsymbol {\theta} _ {j}) \\ = k _ {\mathrm {S n a k e} _ {\mathrm {a}} (\cdot) - \frac {1}{2 a}, \mathrm {I d}} \left(\boldsymbol {\theta} _ {i}, \boldsymbol {\theta} _ {j}\right) + \frac {1}{4 a ^ {2}} + \\ \frac {1}{2 a} \left(b _ {i} - \frac {1}{2 a} \cos (2 a b _ {i}) \exp (- 2 a ^ {2} \| \boldsymbol {w} _ {i} \| ^ {2}) + b _ {j} - \frac {1}{2 a} \cos (2 a b _ {j}) \exp (- 2 a ^ {2} \| \boldsymbol {w} _ {j} \| ^ {2})\right). \\ \end{array}
$$
C Examples which generalise standard exponential family models
In this section, we will study examples of the SNEFY model which use activation function  $\sigma(u) = \exp(u/2)$  (or equivalently, up to scaling,  $\sigma(u) = \exp(u)$ ) and as such correspond to a notion of exponential family mixture models allowing negative weights, as discussed in Section 3.2. These examples have tractable kernels whenever the corresponding exponential family has a tractable normalising constant and we can write the kernels directly using Proposition 1.
SNEFY Von Mises-Fisher mixtures. The VMF distribution is a helpful way of defining the notion of a Gaussian distribution to the sphere. The following kernel may be used to define a VMF distribution. Alternatively, it may be viewed as a way of constructing a distribution supported on  $\mathbb{R}^d$  with sufficient statistics which are projected onto the sphere.
Kernel 3. Define  $\operatorname{proj}_{\mathbb{S}^{d-1}}(\pmb{x}) \triangleq \pmb{x} / \| \pmb{x} \|$  to be the projection onto the unit sphere. Then
$$
k _ {\exp , p r o j _ {\mathbb {S} d - 1}, \Phi} (\boldsymbol {\theta} _ {i}, \boldsymbol {\theta} _ {j}) = \exp (b _ {i} + b _ {j}) \frac {\Gamma (d / 2) 2 ^ {d / 2 - 1} I _ {d / 2 - 1} \left(\| \boldsymbol {w} _ {i} + \boldsymbol {w} _ {j} \|\right)}{\| \boldsymbol {w} _ {i} + \boldsymbol {w} _ {j} \| ^ {d / 2 - 1}},
$$
Proof. If  $\mathbf{x} \sim \mathcal{N}(0, I)$ , then  $\mathbf{x} / \| \mathbf{x} \|$  is uniformly distributed on the sphere. From the normalizing constant of the von Mises-Fisher distribution, from Proposition 1, it then follows that
$$
\begin{array}{l} k _ {\exp , \operatorname {p r o j} _ {\mathfrak {S} ^ {d - 1}}, \Phi} (\boldsymbol {\theta} _ {i}, \boldsymbol {\theta} _ {j}) = \mathbb {E} _ {\mathbf {x}} \left[ \exp \left(\boldsymbol {w} _ {i} ^ {\top} \mathbf {x} / \| \mathbf {x} \| + b _ {i} + \boldsymbol {w} _ {j} ^ {\top} \mathbf {x} / \| \mathbf {x} \| + b _ {j}\right) \right], \quad \mathbf {x} \sim \mathcal {N} (\mathbf {0}, \boldsymbol {I}) \\ = \exp (b _ {i} + b _ {j}) \int_ {\mathbb {S} ^ {d - 1}} \exp \left((\boldsymbol {w} _ {i} + \boldsymbol {w} _ {j}) ^ {\top} \boldsymbol {x}\right) d \boldsymbol {x} \frac {\Gamma (d / 2)}{2 \pi^ {d / 2}} \\ = \frac {\exp \left(b _ {i} + b _ {j}\right) \Gamma (d / 2)}{2 \pi^ {d / 2}} \int_ {\mathbb {S} ^ {d - 1}} \exp \left(\left\| \boldsymbol {w} _ {i} + \boldsymbol {w} _ {j} \right\| \boldsymbol {a} ^ {\top} \boldsymbol {x}\right) d \boldsymbol {x}, \quad \text {w h e r e} \boldsymbol {a} \text {i s a u n i t v e c t o r} \\ = \frac {\exp \left(b _ {i} + b _ {j}\right) \Gamma (d / 2)}{2 \pi^ {d / 2}} \frac {(2 \pi) ^ {d / 2} I _ {d / 2 - 1} \left(\left\| \boldsymbol {w} _ {i} + \boldsymbol {w} _ {j} \right\|\right)}{\left\| \boldsymbol {w} _ {i} + \boldsymbol {w} _ {j} \right\| ^ {d / 2 - 1}} \\ = \exp (b _ {i} + b _ {j}) \frac {\Gamma (d / 2) 2 ^ {d / 2 - 1} I _ {d / 2 - 1} \left(\| \boldsymbol {w} _ {i} + \boldsymbol {w} _ {j} \|\right)}{\| \boldsymbol {w} _ {i} + \boldsymbol {w} _ {j} \| ^ {d / 2 - 1}}, \\ \end{array}
$$
where  $I_{p}$  is the modified Bessel function of the first kind of order  $p$ . In the special case of  $p = 1/2$ , we have  $I_{1/2}(z) = \sqrt{\frac{2}{\pi z}} \sinh(z) = \left( \exp(z) - \exp(-z) \right) \sqrt{\frac{1}{2\pi z}}$ . This implies that when  $d = 3$ , since  $\Gamma(3/2) = \frac{\sqrt{\pi}}{2}$ ,
$$
k _ {\exp , \operatorname {p r o j} _ {\mathfrak {S} 2}, \Phi} (\boldsymbol {\theta} _ {i}, \boldsymbol {\theta} _ {j}) = \exp (b _ {i} + b _ {j}) \frac {\left(e ^ {\| \boldsymbol {w} _ {i} + \boldsymbol {w} _ {j} \|} - e ^ {- \| \boldsymbol {w} _ {i} + \boldsymbol {w} _ {j} \|}\right)}{2 \| \boldsymbol {w} _ {i} + \boldsymbol {w} _ {j} \|}.
$$
Note that  $k_{\mathrm{exp,proj}_{\mathbb{S}^{d - 1}},\Phi}(\pmb {\theta}_i,\pmb {\theta}_j) = k_{\mathrm{exp,Id},\nu}(\pmb {\theta}_i,\pmb {\theta}_j)$ , where  $\nu$  is the uniform measure on the sphere  $\mathbb{S}^{d - 1}$ , because if  $\mathbf{x}$  is Gaussian then  $\mathbf{x} / \| \mathbf{x}\|$  is uniform on the sphere. This allows one to construct SNEFY $_{\mathrm{exp,Id},\nu}$  distributions, which are certain "mixtures" of VMF distributions with weights  $\boldsymbol {V}^{\top}\boldsymbol {V}$ .
SNEFY Gaussian mixtures, fixed variance. We may similarly construct kernels corresponding to "mixtures" of Gaussian distributions. The case here corresponds to a case of known fixed variance parameter.
Kernel 7.  $k_{\mathrm{exp,Id,}\Phi}(\pmb {\theta}_i,\pmb {\theta}_j) = \exp (b_i + b_j)\exp \left(\frac{1}{2}\| \pmb {w}_i + \pmb {w}_j\| ^2\right).$
Proof. This is a consequence of the moment generating function of the multivariate Gaussian distribution. More concretely, by Proposition 1,
$$
\begin{array}{l} k _ {\exp , \operatorname {I d}, \Phi} (\boldsymbol {\theta} _ {i}, \boldsymbol {\theta} _ {j}) = \mathbb {E} _ {\mathbf {x}} \left[ \exp \left(\boldsymbol {w} _ {i} ^ {\top} \mathbf {x} + b _ {i} + \boldsymbol {w} _ {j} ^ {\top} \mathbf {x} + b _ {j}\right) \right], \quad \mathbf {x} \sim \mathcal {N} (\mathbf {0}, \boldsymbol {I}) \\ = \exp (b _ {i} + b _ {j}) \mathbb {E} _ {\mathbf {x}} \left[ \exp \left(\left(\boldsymbol {w} _ {i} + \boldsymbol {w} _ {j}\right) ^ {\top} \mathbf {x}\right) \right] \\ = \exp (b _ {i} + b _ {j}) \exp \left(\frac {1}{2} \| \boldsymbol {w} _ {i} + \boldsymbol {w} _ {j} \| ^ {2}\right). \\ \end{array}
$$
SNEFY Poisson mixtures. Most of our examples deal with continuous distributions but in fact SNEFY can readily be used for discrete distribution modelling. This is particularly helpful when the support is large or infinite, for which computing normalising constants can naively be challenging even in the discrete setting. Let  $\mathbb{X} = \{0,1,2,\ldots\}$ ,  $t(x) = x$ , and the base measure  $\mu(dx) = 1/x!\nu(dx)$ , where  $\nu$  is the counting measure. A SNEFY model for a probability mass function which is a mixture of Poisson distributions allowing negative weights is given by
$$
\begin{array}{l} p (x; \boldsymbol {V}, \boldsymbol {w}) = \frac {1}{\operatorname {T r} (\boldsymbol {V} ^ {\top} \boldsymbol {V} \boldsymbol {K} _ {\Theta})} \frac {1}{x !} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {n} \boldsymbol {v} _ {., i} ^ {\top} \boldsymbol {v} _ {., j} \exp \left(\left(w _ {i} + w _ {j}\right) x\right) \\ = \frac {1}{\operatorname {T r} \left(\boldsymbol {V} ^ {\top} \boldsymbol {V} \boldsymbol {K} _ {\boldsymbol {\Theta}}\right)} \frac {1}{x !} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {n} \boldsymbol {v} _ {., i} ^ {\top} \boldsymbol {v} _ {., j} (\lambda_ {i} \lambda_ {j}) ^ {x}, \quad x = 0, 1, 2, \dots . \\ \end{array}
$$
following the usual mean parametrisation  $\lambda_{i} = e^{w_{i}}$ , so the individual mixture components have rates which are geometric means of  $(\lambda_i,\lambda_j)$  pairs.
Kernel 8. Choose the base measure  $\mu(dx) = (x!)^{-1} \nu(dx)$ , where  $\nu$  is the counting measure. We have
$$
k _ {\exp , \mathrm {I d}, (x!) ^ {- 1} \nu} (\pmb {\theta} _ {i}, \pmb {\theta} _ {j}) = \exp \left(b _ {i} + b _ {j}\right) \exp \big (\exp (w _ {i} + w _ {j}) \big).
$$
Proof. This is again direct from Proposition 1. In detail,
$$
\begin{array}{l} k _ {\exp , \operatorname {I d}, (x!) ^ {- 1} \nu} (\boldsymbol {\theta} _ {i}, \boldsymbol {\theta} _ {j}) = \sum_ {x = 0} ^ {\infty} \frac {1}{x !} \exp \left(w _ {i} x + w _ {j} x + b _ {i} + b _ {j}\right) \\ = \exp \left(b _ {i} + b _ {j}\right) \sum_ {x = 0} ^ {\infty} \frac {1}{x !} \exp \left(w _ {i} x + w _ {j} x\right) \\ \end{array}
$$
The second factor involving the sum is the partition function of the Poisson distribution in canonical form, which is  $\exp \bigl (\exp (w_i + w_j)\bigr)$ .
The usual mean parameterisation of the Poisson distribution is through a rate parameter  $\lambda_{i} = \exp (w_{i})$ , which would lead to the kernel representation
$$
k _ {\exp , \operatorname {I d}, (x!) ^ {- 1} \nu} (\boldsymbol {\theta} _ {i}, \boldsymbol {\theta} _ {j}) = \exp \left(b _ {i} + b _ {j}\right) \exp \left(\lambda_ {i} \lambda_ {j}\right).
$$
SNEFY Gaussian mixtures, unknown variance (Squared radial basis function network). We now discuss an intriguing connection between the Gaussian distribution and squared RBF networks. This connection is made possible through our machinery of SNEFY distributions. Let  $\mathbb{X} = \mathbb{R}^d$ ,  $t(\boldsymbol{x}) = (x_1, \ldots, x_d, x_1^2, \ldots, x_d^2)$  (i.e.  $D = 2d$ ) and suppose  $\mu(dx) = dx$  is Lebesgue measure. Choose  $\sigma(\cdot) = \exp(\cdot / 2)$  and consider the  $r$ -th output of our network  $\boldsymbol{f}: \mathbb{R}^D \to \mathbb{R}^m$
$$
f _ {r} (\boldsymbol {t} (\boldsymbol {x}); \boldsymbol {V}, \boldsymbol {\Theta}) = \sum_ {i = 1} ^ {n} v _ {r i} \exp \left(\frac {1}{2} \left(\sum_ {\ell = 1} ^ {d} w _ {i \ell} x _ {\ell} + \sum_ {\ell = 1} ^ {d} \tilde {w} _ {i \ell} x _ {\ell} ^ {2}\right)\right),
$$
where we denoted  $\tilde{w}_{i\ell} = w_{i,d + \ell}$ . In this case, we require that  $\tilde{w}_{e\ell} < 0$  for the model to be (square) integrable. Reparametrising  $\sigma_{i\ell}^2 = -\frac{1}{2\tilde{w}_{i\ell}}$  and  $\mu_{i\ell} = -\frac{w_{i\ell}}{2\tilde{w}_{i\ell}}$  and absorbing the factor  $\exp \left(-\sum_{\ell = 1}^{d}\frac{\mu_{i\ell}^2}{4\sigma_{i\ell}^2}\right)$  into readout parameters  $\mathbf{V}$ , gives
$$
f _ {r} (\pmb {t} (\pmb {x}); \pmb {V}, \pmb {\Theta}) = \sum_ {i = 1} ^ {n} v _ {r i} \exp \left(- \sum_ {\ell = 1} ^ {d} \frac {(x _ {\ell} - \mu_ {i \ell}) ^ {2}}{4 \sigma_ {i \ell} ^ {2}}\right).
$$
Thus, we have recovered a classical radial basis function (RBF) network [52]. These models are well known to have universal approximation properties [36]. For the most commonly used form of the RBF network, we can restrict the parameters  $\sigma_{i\ell}^{2} = \sigma_{i}^{2}$  to be the same across the dimensions, giving
$$
f _ {r} \left(\boldsymbol {t} (\boldsymbol {x}); \boldsymbol {V}, \boldsymbol {\Theta}\right) = \sum_ {i = 1} ^ {n} v _ {r i} \exp \left(- \frac {\left\| \boldsymbol {x} - \boldsymbol {\mu} _ {i} \right\| ^ {2}}{4 \sigma_ {i} ^ {2}}\right), \quad \boldsymbol {x} \in \mathbb {R} ^ {d},
$$
with location parameters  $\pmb{\mu}_i\in \mathbb{R}^d$  and the scale parameters  $\sigma_i^2 >0$ . Note the unusual factor of 4 in front of  $\sigma_i^2$  - this ensures that our model in fact reduces to the usual parametrisation of multivariate normal densities, since we will be modelling the density using the squared norm of  $\pmb{f}$ .
Since  $\mu$  is the Lebesgue measure, SNEFY gives us a density model with respect to the Lebesgue measure as
$$
p (\pmb {x}; \pmb {V}, \pmb {\Theta}) = \frac {1}{\mathrm {T r} (\pmb {V} ^ {\top} \pmb {V} \pmb {K} _ {\Theta})} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {n} \pmb {v} _ {., i} ^ {\top} \pmb {v} _ {., j} \exp \left(- \frac {\| \pmb {x} - \pmb {\mu} _ {i} \| ^ {2}}{4 \sigma_ {i} ^ {2}}\right) \exp \left(- \frac {\| \pmb {x} - \pmb {\mu} _ {j} \| ^ {2}}{4 \sigma_ {j} ^ {2}}\right).
$$
If  $n = 1$ , we recover simply a multivariate normal density  $\mathcal{N}(\pmb{\mu}_1, \sigma_1^2 I)$ . The above model is essentially the same as the one in [42], despite being derived in a very different way.
Kernel 9. Let  $t^{(2)}(\pmb{x}) = (x_1, \dots, x_d, x_1^2, \dots, x_d^2)$  so that  $D = 2d$  be the sufficient statistic. Partition  $\pmb{W} = [\pmb{W}_{[:1:d]}, \tilde{\pmb{W}}]$  and suppose  $\tilde{\pmb{W}} < 0$  element-wise. Choose  $\mu(dx) = dx$  to be the Lebesgue measure. Then
$$
k _ {\exp , \pmb {t} ^ {(2)}, d \pmb {x}} (\pmb {\theta} _ {i}, \pmb {\theta} _ {j}) = \pi^ {d / 2} \exp (b _ {i} + b _ {j}) \prod_ {l = 1} ^ {d} \exp \Big (- \frac {(w _ {i l} + w _ {j l}) ^ {2}}{4 (\tilde {w} _ {i l} + \tilde {w} _ {j l})} \Big) \frac {1}{\sqrt {- (\tilde {w} _ {i l} + \tilde {w} _ {j l})}}
$$
Proof. As with the kernels above, this follows from Proposition 1, since
$$
\begin{array}{l} k _ {\exp , \boldsymbol {t} ^ {(2)}, d \boldsymbol {x}} (\boldsymbol {\theta} _ {i}, \boldsymbol {\theta} _ {j}) = \int_ {\mathbb {R} ^ {d}} \exp \left(\boldsymbol {w} _ {i, 1: d} ^ {\top} \boldsymbol {x} + \tilde {\boldsymbol {w}} _ {i} ^ {\top} \boldsymbol {x} ^ {2} + b _ {1} + \boldsymbol {w} _ {j, 1: d} ^ {\top} \boldsymbol {x} + \tilde {\boldsymbol {w}} _ {j} ^ {\top} \boldsymbol {x} ^ {2} + b _ {j}\right) d \boldsymbol {x} \\ = (2 \pi) ^ {d / 2} \exp (b _ {i} + b _ {j}) \prod_ {l = 1} ^ {d} \exp \left(- \frac {(w _ {i l} + w _ {j l}) ^ {2}}{4 (\tilde {w} _ {i l} + \tilde {w} _ {j l})}\right) \frac {1}{\sqrt {- 2 (\tilde {w} _ {i l} + \tilde {w} _ {j l})}}. \\ \end{array}
$$
While Proposition 1 gives us an expression for the kernel matrix  $K_{\Theta}$  in terms of natural parameters, we can also express it directly in terms of parameters  $\pmb{\mu}_i$ ,  $\sigma_i^2$ . In particular,
$$
\begin{array}{l} \left[ \boldsymbol {K} _ {\Theta} \right] _ {i j} = \exp (b _ {i} + b _ {j}) \int_ {\mathbb {R} ^ {d}} \exp \left(- \frac {\| \boldsymbol {x} - \boldsymbol {\mu} _ {i} \| ^ {2}}{4 \sigma_ {i} ^ {2}}\right) \exp \left(- \frac {\| \boldsymbol {x} - \boldsymbol {\mu} _ {j} \| ^ {2}}{4 \sigma_ {j} ^ {2}}\right) d \boldsymbol {x} \\ = \exp (b _ {i} + b _ {j}) \left(\frac {4 \pi \sigma_ {i} ^ {2} \sigma_ {j} ^ {2}}{\sigma_ {i} ^ {2} + \sigma_ {j} ^ {2}}\right) ^ {d / 2} \exp \left(- \frac {\| \boldsymbol {\mu} _ {i} - \boldsymbol {\mu} _ {j} \| ^ {2}}{4 (\sigma_ {i} ^ {2} + \sigma_ {j} ^ {2})}\right). \\ \end{array}
$$
We briefly state two more cases without an extended discussion.
SNEFY Gamma mixtures.
Kernel 10. Let  $\mathbb{X} = (0,\infty)$ ,  $t(x) = (\log x, -x)$  and  $\sigma = \exp$ . Partition  $\boldsymbol{W} = [W_{[:,1:d]}, \tilde{W}]$  and suppose  $\boldsymbol{W}_{[:,1:d]} > -1$  and  $\tilde{\boldsymbol{W}} > 0$  element-wise. Choose  $\mu(dx) = dx$  to be the Lebesgue measure. Then
$$
k _ {\exp , \boldsymbol {t}, d \boldsymbol {x}} (\boldsymbol {\theta} _ {i}, \boldsymbol {\theta} _ {j}) = \exp (b _ {i} + b _ {j}) \frac {\Gamma (w _ {i 1} + w _ {j 1} + 1)}{(w _ {i 2} + w _ {j 2}) ^ {w _ {i 1} + w _ {j 1} + 1}}.
$$
SNEFY Dirichlet mixtures.
Kernel 11. Let  $\mathbb{X} = \Delta^{D-1}$ ,  $a$  ( $D-1$ -simplex of probability distributions, i.e.  $\pmb{x} \in [0,1]^D$ ),  $\sum_{i=1}^{D} x_i = 1$ . Let  $\sigma = \exp$ . Let  $\pmb{t}(\pmb{x}) = (\log x_1, \dots, \log x_D)$ . Choose  $\mu(dx) = dx$  to be the Lebesgue measure. Suppose  $\pmb{W} > -1$  elementwise. Then
$$
k _ {\exp , \boldsymbol {t}, d \boldsymbol {x}} (\boldsymbol {\theta} _ {i}, \boldsymbol {\theta} _ {j}) = \exp (b _ {i} + b _ {j}) \frac {\prod_ {d = 1} ^ {D} \Gamma (w _ {i d} + w _ {j d} + 1)}{\Gamma \left(D + \sum_ {d = 1} ^ {D} w _ {i d} + w _ {j d}\right)}.
$$
D Marginalisation in the case  $\sigma = \exp (\cdot /2)$
Let  $M$  be a positive semi-definite  $n \times n$  matrix. We will make use of the following SNEFY parametrisation
$$
P (d \boldsymbol {x}; \boldsymbol {M}, \boldsymbol {\Theta}) = \frac {1}{\mathrm {z} (\boldsymbol {M} , \boldsymbol {\Theta})} \sigma (\boldsymbol {W} \boldsymbol {t} (\boldsymbol {x}) + \boldsymbol {b}) ^ {\top} \boldsymbol {M} \sigma (\boldsymbol {W} \boldsymbol {t} (\boldsymbol {x}) + \boldsymbol {b}) \mu (d \boldsymbol {x}). \tag {16}
$$
Since we can always write  $M = V^{\top}V$ , for an  $m \times n$  matrix  $V$ ,  $m \leq n$ , we have
$$
\sigma \left(\boldsymbol {W} \boldsymbol {t} (\boldsymbol {x}) + \boldsymbol {b}\right) ^ {\top} \boldsymbol {M} \sigma \left(W \boldsymbol {t} (\boldsymbol {x}) + \boldsymbol {b}\right) = \| \boldsymbol {V} \sigma \left(\boldsymbol {W} \boldsymbol {t} (\boldsymbol {x}) + \boldsymbol {b}\right) \| ^ {2} = \sum_ {i = 1} ^ {m} \left(\sum_ {j = 1} ^ {n} v _ {i j} \sigma \left(\boldsymbol {w} _ {j} ^ {\top} \boldsymbol {t} (\boldsymbol {x}) + b _ {j}\right)\right) ^ {2},
$$
which is, as in the parametrisation given in the main text, simply the squared Euclidean norm of a multi-output neural network, with  $\mathbf{V}$  corresponding to the weights of the second layer. If we denote by  $v_{\cdot j}$  the  $j$ -th column of  $\mathbf{V}$ , the normalizing constant is given by
$$
\sum_ {j = 1} ^ {n} \sum_ {l = 1} ^ {n} \boldsymbol {v} _ {. j} ^ {\top} \boldsymbol {v}. _ {l} k _ {\sigma , \boldsymbol {t}, \mu} (\boldsymbol {\theta} _ {j}, \boldsymbol {\theta} _ {l}) = \sum_ {j = 1} ^ {n} \sum_ {l = 1} ^ {n} m _ {j l} k _ {\sigma , \boldsymbol {t}, \mu} (\boldsymbol {\theta} _ {j}, \boldsymbol {\theta} _ {l}) = \operatorname {T r} (\boldsymbol {M} \boldsymbol {K} _ {\Theta})
$$
where as before
$$
k _ {\sigma , \boldsymbol {t}, \mu} (\boldsymbol {\theta} _ {i}, \boldsymbol {\theta} _ {j}) = \int \sigma (\boldsymbol {w} _ {i} ^ {\top} \boldsymbol {t} (\boldsymbol {x}) + b _ {i}) \sigma (\boldsymbol {w} _ {j} ^ {\top} \boldsymbol {t} (\boldsymbol {x}) + b _ {j}) \mu (d \boldsymbol {x}).
$$
Now if we let  $\sigma = \exp (\cdot /2)$  we obtain a family which is also closed under marginalisation (in addition to conditioning). The Proposition below generalises Proposition 1 of [42], which considers the special case of the Gaussian PSD mixtures.