SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Future work We see a number of further promising future research directions. First, as we detail in Appendix F, choosing  $\sigma(\cdot) = \exp(i\cdot)$  and identity sufficient statistics results in a kernel  $k_{\mathrm{exp}(i\cdot),\mathrm{Id},\mu}$  which is the Fourier transform of a nonnegative measure. By Bochner's theorem, the kernel is guaranteed to be (real or complex-valued) shift-invariant. The kernel matrix is Hermitian PSD (so that the normalising constant is positive and nonnegative), and we may also allow (but do not require) the readout parameters  $\mathbf{V}$  to be complex. We note that the same result would be obtained if one used a mixture of real-valued cos and sin activations with shared parameters (see Remark 3). Second, an alternative deep model to our deep conditional feature extractor might be to use a SNEFY model as a base measure  $\mu$  for another SNEFY model; this might be repeated  $L$  times. This leads to  $\mathcal{O}(n^{2L})$  integration terms for the normalising constant. The individual terms are tractable in certain cases, for example when  $\sigma$  is exponential or trigonometric. Third, when modelling discrete distributions with trigonometric activations, the NNK can be expressed in terms of convergent Fourier series (see Appendix G) Finally, our integration technique can be applied to other settings. For example, we may build a Poisson point process intensity function using a squared neural network and compute the intensity function in closed-form, offering a model that scales quadratically in the number of neurons  $\mathcal{O}(n^2)$  instead of comparable models which scale cubically in the number of datapoints  $\mathcal{O}(N^3)$  [9, 51].
Russell and Cheng Soon would like to acknowledge the support of the Machine Learning and Artificial Intelligence Future Science Platform, CSIRO. The authors would like to thank Jia Liu for early discussions about the idea.
[1] Francis Bach. Breaking the curse of dimensionality with convex neural networks. Journal of Machine Learning Research, 18(19):1-53, 2017.
[2] R Beck, C-A Lin, EEO Ishida, F Gieseke, RS de Souza, MV Costa-Duarte, MW Hattab, and A Krone-Martins. On the realistic validation of photometric redshifts. Monthly Notices of the Royal Astronomical Society, 468(4):4323-4339, 2017.
[3] Edgar A Bernal. Training deep normalizing flow models in highly incomplete data scenarios with prior regularization. arXiv preprint arXiv:2104.01482, 2021.
[4] Youngmin Cho and Lawrence K. Saul. Kernel methods for deep learning. In Advances in Neural Information Processing Systems, pages 342-350, 2009.
[5] Michael Collins, Sanjoy Dasgupta, and Robert E Schapire. A generalization of principal components analysis to the exponential family. Advances in neural information processing systems, 14, 2001.
[6] Marc Peter Deisenroth, A Aldo Faisal, and Cheng Soon Ong. Mathematics for machine learning. Cambridge University Press, 2020.
[7] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In International Conference on Learning Representations, 2017.
[8] Bradley Efron and Trevor Hastie. Computer Age Statistical Inference: Algorithms, Evidence, and Data Science, volume 6. Cambridge University Press, 2021.
[9] Seth Flaxman, Yee Whye Teh, and Dino Sejdinovic. Poisson intensity estimation with reproducing kernels. In Artificial Intelligence and Statistics, pages 270-279. PMLR, 2017.
[10] Thorsten Glusenkamp. Unifying supervised learning and vaes-automating statistical inference in (astro-) particle physics with amortized conditional normalizing flows. arXiv e-prints, 2020.
[11] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, volume 27, 2014.
[12] Insu Han, Amir Zandieh, Jaehoon Lee, Roman Novak, Lechao Xiao, and Amin Karbasi. Fast neural kernel embeddings for general activations. In Advances in Neural Information Processing Systems, volume 35, pages 35657-35671, 2022.
[13] Insu Han, Amir Zandieh, Jaehoon Lee, Roman Novak, Lechao Xiao, and Amin Karbasi. Fast neural kernel embeddings for general activations. Advances in neural information processing systems, 2022.
[14] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units. arXiv preprint arXiv:1606.08415, 2016.
[15] Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800, 2002.
[16] Aapo Hyvarinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
[17] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in neural information processing systems, pages 8571-8580, 2018.
[18] Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114, 2013.
[19] Thomas Joseph Lawrence. Point pattern analysis on a sphere. Master's thesis, The University of Western Australia, 2018.
[20] Nicolas Le Roux and Yoshua Bengio. Continuous neural networks. In Artificial Intelligence and Statistics, pages 404-411, 2007.
[21] Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
[22] Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as Gaussian processes. In International Conference on Learning Representations, 2018.
[23] Lorenzo Lo Conte, Stefan Mengel, Nicolas Gillis, and Antonio Vergari. Negative mixture models via squaring: Representation and learning. In The 6th Workshop on Tractable Probabilistic Modeling, 2023.
[24] David JC MacKay. Introduction to Gaussian processes, 1998.
[25] Ulysse Marteau-Ferey, Francis Bach, and Alessandro Rudi. Non-parametric models for nonnegative functions. Advances in neural information processing systems, 33:12816-12826, 2020.
[26] Ulysse Marteau-Ferey, Francis Bach, and Alessandro Rudi. Sampling from arbitrary functions via psd models. In International Conference on Artificial Intelligence and Statistics, pages 2823-2861. PMLR, 2022.
[27] P. McCullagh and J.A. Nelder. Generalized Linear Models, Second Edition. Monographs on Statistics and Applied Probability Series. Chapman & Hall, 1989.
[28] Lassi Meronen, Christabella Irwanto, and Arno Solin. Stationary activations for uncertainty calibration in deep learning. Advances in Neural Information Processing Systems, 33:2338-2350, 2020.
[29] Lassi Meronen, Martin Trapp, and Arno Solin. Periodic activation functions induce stationarity. Advances in Neural Information Processing Systems, 34:1673-1685, 2021.
[30] Charlie Nash and Conor Durkan. Autoregressive energy machines. In International Conference on Machine Learning, pages 1735-1744, 2019.
[31] Radford M Neal. Bayesian learning for neural networks. PhD thesis, University of Toronto, 1995.
[32] Frank Nielsen and Vincent Garcia. Statistical exponential families: A digest with flash cards. arXiv preprint arXiv:0911.4863, 2009.
[33] Roman Novak, Lechao Xiao, Jaehoon Lee, Yasaman Bahri, Greg Yang, Jiri Hron, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Bayesian deep convolutional networks with many channels are Gaussian processes. In The International Conference on Learning Representations, 2019.
[34] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. The Journal of Machine Learning Research, 22(1):2617-2680, 2021.
[35] Georgios Papamakarios. Neural density estimation and likelihood-free inference. PhD thesis, University of Edinburgh, 2019.
[36] Jooyoung Park and Irwin W Sandberg. Universal approximation using radial-basis-function networks. Neural computation, 3(2):246-257, 1991.
where  $k_{\sigma ,t,\mu}$  is as defined in (5), and  $\pmb{K}_{\Theta}$  is the PSD matrix whose ith entry is  $k_{\sigma ,t,\mu}(\pmb {\theta}_i,\pmb {\theta}_j)$
Proof. Let  $\widetilde{\pmb{K}_{\Theta}} (\pmb {x})$  be the PSD matrix whose  $ij$ th entry is  $\sigma (\pmb {w}_i^\top \pmb {t}(\pmb {x}) + b_i)\sigma (\pmb {w}_j^\top \pmb {t}(\pmb {x}) + b_j)^\top$ . The squared norm of the neural network evaluation is given by  $\operatorname{Tr}\left(\boldsymbol {V}^{\top}\boldsymbol {V}\widetilde{\boldsymbol{K}_{\Theta}} (\boldsymbol {x})\right)$ , since
$$
\begin{array}{l} \left\| \boldsymbol {V} \sigma (\boldsymbol {W} \boldsymbol {t} (\boldsymbol {x}) + \boldsymbol {b}) \right\| _ {2} ^ {2} = \sum_ {i = 1} ^ {m} \sum_ {j _ {1} = 1} ^ {n} \sum_ {j _ {2} = 1} ^ {n} v _ {i j _ {1}} v _ {i j _ {2}} \sigma \left(\boldsymbol {w} _ {j _ {1}} ^ {\top} \boldsymbol {t} (x) + b _ {j _ {1}}\right) \sigma \left(\boldsymbol {w} _ {j _ {2}} ^ {\top} \boldsymbol {t} (x) + b _ {j _ {2}}\right) \\ = \sum_ {j _ {1} = 1} ^ {n} \sum_ {j _ {2} = 1} ^ {n} \boldsymbol {v} _ {., j _ {1}} ^ {\top} \boldsymbol {v} _ {., j _ {2}} \sigma \left(\boldsymbol {w} _ {j _ {1}} ^ {\top} \boldsymbol {t} (x) + b _ {j _ {1}}\right) \sigma \left(\boldsymbol {w} _ {j _ {2}} ^ {\top} \boldsymbol {t} (x) + b _ {j _ {2}}\right) \\ = \left\langle \boldsymbol {V} ^ {\top} \boldsymbol {V}, \widetilde {\boldsymbol {K} _ {\Theta}} (\boldsymbol {x}) \right\rangle_ {F} \\ = \operatorname {T r} \left(\boldsymbol {V} ^ {\top} \boldsymbol {V} \widetilde {\boldsymbol {K} _ {\Theta}} (\boldsymbol {x})\right), \\ \end{array}
$$
where  $\boldsymbol{v}_{\cdot,j_1}$  denotes the  $j_1$ th column of  $\mathbf{V}$  and  $\langle \cdot, \cdot \rangle_F$  denotes the Frobenius inner product. Therefore using the definition (2) directly and the linearity of the Frobenius inner product, the normalising constant is
$$
\begin{array}{l} \mathrm {z} (\boldsymbol {V}, \boldsymbol {\Theta}) = \int_ {\mathbb {X}} \left\| \boldsymbol {V} \sigma (\boldsymbol {W} \boldsymbol {t} (\boldsymbol {x}) + \boldsymbol {b}) \right\| _ {2} ^ {2} \mu (d \boldsymbol {x}) \\ = \int_ {\mathbb {X}} \operatorname {T r} \left(\boldsymbol {V} ^ {\top} \boldsymbol {V} \widetilde {\boldsymbol {K} _ {\Theta}} (\boldsymbol {x})\right) \mu (d \boldsymbol {x}) \\ = \operatorname {T r} \left(\boldsymbol {V} ^ {\top} \boldsymbol {V} \int_ {\mathbb {X}} \widetilde {\boldsymbol {K} _ {\boldsymbol {\Theta}}} (\boldsymbol {x}) \mu (d \boldsymbol {x})\right) \\ = \operatorname {T r} \left(\boldsymbol {V} ^ {\top} \boldsymbol {V} \boldsymbol {K} _ {\Theta}\right). \tag {13} \\ \end{array}
$$
Proof. The result follows by noticing that the logarithmic derivative property holds,  $\sum_{i=1}^{n} \frac{\partial \Psi}{\partial w_i} = \frac{1}{z} \sum_{i=1}^{n} \frac{\partial z}{\partial w_i}$ , and that by writing
$$
\mathrm {z} = \sum_ {i, j} \boldsymbol {v} _ {., i} ^ {\top} \boldsymbol {v} _ {., j} \int \exp \left(\frac {1}{2} \boldsymbol {w} _ {i} ^ {\top} \boldsymbol {t} (\boldsymbol {x})\right) \exp \left(\frac {1}{2} \boldsymbol {w} _ {j} ^ {\top} \boldsymbol {t} (\boldsymbol {x})\right) \mu (d \boldsymbol {x}),
$$
$$
\begin{array}{l} \frac {\partial \mathbf {z}}{\partial \boldsymbol {w} _ {i}} = \| \boldsymbol {v} _ {., i} \| ^ {2} \int \boldsymbol {t} (\boldsymbol {x}) \exp \left(\boldsymbol {w} _ {i} ^ {\top} \boldsymbol {t} (\boldsymbol {x})\right) \mu (d \boldsymbol {x}) \\ + \boldsymbol {v} _ {., i} ^ {\top} \sum_ {j \neq i} \boldsymbol {v} _ {., j} \int \boldsymbol {t} (\boldsymbol {x}) \exp \left(\frac {1}{2} \boldsymbol {w} _ {i} ^ {\top} \boldsymbol {t} (\boldsymbol {x})\right) \exp \left(\frac {1}{2} \boldsymbol {w} _ {j} ^ {\top} \boldsymbol {t} (\boldsymbol {x})\right) \mu (d \boldsymbol {x}) \\ { = } { \pmb { v } _ { . , i } ^ { \top } \sum _ { j } \pmb { v } _ { . , j } \int \pmb { t } ( \pmb { x } ) \exp \left( \frac { 1 } { 2 } \pmb { w } _ { i } ^ { \top } \pmb { t } ( \pmb { x } ) \right) \exp \left( \frac { 1 } { 2 } \pmb { w } _ { j } ^ { \top } \pmb { t } ( \pmb { x } ) \right) \mu ( d \pmb { x } ) . } \\ \end{array}
$$
Now summing across  $i$  gives  $\sum_{i=1}^{n} \frac{\partial \Psi}{\partial \boldsymbol{w}_i} = \int \boldsymbol{t}(\boldsymbol{x}) P(d\boldsymbol{x})$  as required.
To obtain the second result, we apply the product rule to find
$$
\sum_ {i, j} \frac {\partial^ {2} \Psi}{\partial \pmb {w} _ {i} \pmb {w} _ {j} ^ {\top}} = \frac {1}{\mathrm {z}} \sum_ {i, j} \frac {\partial^ {2} \mathrm {z}}{\partial \pmb {w} _ {i} \pmb {w} _ {j} ^ {\top}} - \left(\frac {1}{\mathrm {z}} \sum_ {i} \frac {\partial \mathrm {z}}{\partial \pmb {w} _ {i}}\right) \left(\frac {1}{\mathrm {z}} \sum_ {j} \frac {\partial \mathrm {z}}{\partial \pmb {w} _ {j} ^ {\top}}\right)
$$
$$
\frac {\partial^ {2} \mathbf {\Omega} _ {\mathrm {Z}}}{\partial \pmb {w} _ {i} \pmb {w} _ {j} ^ {\top}} = \left\{ \begin{array}{l l} \frac {1}{2} \pmb {v} _ {., i} ^ {\top} \pmb {v} _ {., j} \int \pmb {t} (\pmb {x}) \pmb {t} (\pmb {x}) ^ {\top} \exp \left(\frac {1}{2} \pmb {w} _ {i} ^ {\top} \pmb {t} (\pmb {x})\right) \exp \left(\frac {1}{2} \pmb {w} _ {j} ^ {\top} \pmb {t} (\pmb {x})\right) \mu (d \pmb {x}), & \mathrm {i f} i \neq j, \\ \| \pmb {v} _ {., i} \| ^ {2} \int \pmb {t} (\pmb {x}) \pmb {t} (\pmb {x}) ^ {\top} \exp \left(\pmb {w} _ {i} ^ {\top} \pmb {t} (\pmb {x})\right) \mu (d \pmb {x}) \\ + \frac {1}{2} \pmb {v} _ {., i} ^ {\top} \sum_ {r \neq i} \pmb {v} _ {., j} \int \pmb {t} (\pmb {x}) \pmb {t} (\pmb {x}) ^ {\top} \exp \left(\frac {1}{2} \pmb {w} _ {i} ^ {\top} \pmb {t} (\pmb {x})\right) \exp \left(\frac {1}{2} \pmb {w} r ^ {\top} \pmb {t} (\pmb {x})\right) \mu (d \pmb {x}), & \mathrm {i f} i = j. \end{array} \right.
$$
$$
\begin{array}{l} \frac {1}{z} \sum_ {i, j} \frac {\partial^ {2} z}{\partial \boldsymbol {w} _ {i} \boldsymbol {w} _ {j} ^ {\top}} = \frac {1}{z} \sum_ {i = 1} ^ {n} \left(\frac {\partial^ {2} z}{\partial \boldsymbol {w} _ {i} \boldsymbol {w} _ {i} ^ {\top}} + \sum_ {j \neq i} \frac {\partial^ {2} z}{\partial \boldsymbol {w} _ {i} \boldsymbol {w} _ {j} ^ {\top}}\right) \\ { = } { \frac { 1 } { \mathrm { z } } \sum _ { i , j } \pmb { v } _ { i } ^ { \top } \pmb { v } _ { j } \int \pmb { t } ( \pmb { x } ) \pmb { t } ( \pmb { x } ) ^ { \top } \exp \left( \frac { 1 } { 2 } \pmb { w } _ { i } ^ { \top } \pmb { t } ( \pmb { x } ) \right) \exp \left( \frac { 1 } { 2 } \pmb { w } _ { j } ^ { \top } \pmb { t } ( \pmb { x } ) \right) \mu ( d \pmb { x } ) } \\ = \int \boldsymbol {t} (\boldsymbol {x}) \boldsymbol {t} (\boldsymbol {x}) ^ {\top} P (d \boldsymbol {x}), \\ \end{array}
$$
Remark 2. Given the above relationship between the log-normalising constant and the expectation of the sufficient statistic, we can also ask whether the maximum likelihood estimation of the mean parameters  $\mathbb{E}[t(x)]$  proceeds in the same way as in the exponential family case. The answer is positive but with two caveats. First, the log-likelihood need not be concave in  $W$ , and may have many local optima or stationary points. Second, unlike the exponential family distribution, the SNEFY distribution is not determined by its mean parameters, so the MLE estimation of the mean parameters may not constitute a meaningful task in SNEFY modelling (unless we are in the case where precisely the expectation of  $t(x)$  under the SNEFY model is of interest).
Corollary 1. Given a dataset  $\{\mathbf{x}_{\ell}\}_{\ell = 1}^{N}$ , and a SNEFY model with  $\sigma(u) = \exp(\frac{1}{2} u)$ , assume that all rows of the maximum likelihood estimator of  $\mathbf{W}$  are in the interior of the natural parameter space of the corresponding exponential family. Denote the mean parameter as  $\mathbf{m} = \mathbb{E}[t(\mathbf{x})]$ . Then the maximum likelihood estimate of  $\mathbf{m}$  is  $\hat{\mathbf{m}} = \frac{1}{N} \sum_{\ell = 1}^{N} \mathbf{t}(\mathbf{x}_{\ell})$ .
Proof. Since MLE is achieved at a stationary point of the log-likelihood, the proof follows by writing the log likelihood as
$$
\sum_ {\ell = 1} ^ {N} \log p (\boldsymbol {x} _ {\ell}; \boldsymbol {V}, \boldsymbol {\Theta}) = \operatorname {c o n s t} + \sum_ {\ell = 1} ^ {N} \log \| \boldsymbol {f} (\boldsymbol {t} (\boldsymbol {x} _ {\ell}); \boldsymbol {V}, \boldsymbol {\Theta}) \| _ {2} ^ {2} - N \Psi
$$
and concluding that at the MLE  $\{\pmb{w}_i^*\}_{i=1}^n$  for  $\pmb{W}$ , we must have
$$
\sum_ {\ell = 1} ^ {N} \frac {\partial \log \| \boldsymbol {f} (\boldsymbol {t} (\boldsymbol {x} _ {\ell}) ; \boldsymbol {V} , \boldsymbol {\Theta}) \| _ {2} ^ {2}}{\partial \boldsymbol {w} _ {i} ^ {*}} = N \frac {\partial \Psi}{\partial \boldsymbol {w} _ {i} ^ {*}}, \quad i = 1, \dots , n. \tag {14}
$$
$$
\sum_ {i = 1} ^ {n} \frac {\partial \log \| \boldsymbol {f} (\boldsymbol {t} (\boldsymbol {x} _ {\ell}) ; \boldsymbol {V} , \boldsymbol {\Theta}) \| _ {2} ^ {2}}{\partial \boldsymbol {w} _ {i}} = \boldsymbol {t} (\boldsymbol {x} _ {\ell}),
$$
so the result follows by summing (14) over  $i$  and dividing by  $N$ . Note that maximum likelihood estimates are invariant to transformations, even if the transformation is not bijective. So if  $\{\pmb{w}_i^*\}_{i=1}^n$  is an MLE, we may construct a mapping from  $\{\pmb{w}_i^*\}_{i=1}^n$  to a corresponding MLE  $\hat{\pmb{m}}$  for the mean parameter.
Proof. The joint distribution of  $\mathbf{x}$  satisfies
$$
P (d \boldsymbol {x}; \boldsymbol {V}, \boldsymbol {\Theta}) \propto \left\| \boldsymbol {V} \sigma \big (\boldsymbol {W} _ {1} \boldsymbol {t} _ {1} (\boldsymbol {x} _ {1}) + \boldsymbol {W} _ {2} \boldsymbol {t} _ {2} (\boldsymbol {x} _ {2}) + \boldsymbol {b} \big) \right\| _ {2} ^ {2} \mu (d \boldsymbol {x}).
$$
Therefore, the distribution of  $\mathbf{x}_1$  conditionally on  $\mathbf{x}_2 = \mathbf{x}_2$ , which is obtained by dividing the joint distribution by the marginal distribution of  $\mathbf{x}_2$  (which is independent of  $\mathbf{x}_1$ ), satisfies
$$
P _ {1} \left(d \boldsymbol {x} _ {1} \mid \boldsymbol {x} _ {2}; \boldsymbol {V}, \left(\boldsymbol {W} _ {1}, \boldsymbol {W} _ {2} \boldsymbol {t} _ {2} (\boldsymbol {x} _ {2}) + \boldsymbol {b}\right)\right) \propto \left(\boldsymbol {V} \sigma \left(\boldsymbol {W} _ {1} \boldsymbol {t} _ {1} (\boldsymbol {x} _ {1}) + \boldsymbol {W} _ {2} \boldsymbol {t} _ {2} (\boldsymbol {x} _ {2}) + \boldsymbol {b}\right)\right) ^ {2} \mu_ {1} (d \boldsymbol {x} _ {1}).
$$
That is, the term  $W_{2}t_{2}(x_{2}) + b$  is viewed as a constant bias term when the expression on the right hand side is an unnormalised measure with respect to the variable  $x_{1}$ .
Theorem 2. Let  $\mathbf{x} = (\mathbf{x}_1, \mathbf{x}_2)$  be jointly SNEFY $_{\mathbb{X}_1 \times \mathbb{X}_2}$ ,  $t, \sigma, \mu$  with parameters  $\mathbf{V}$  and  $\Theta = ([W_1, W_2], \mathbf{b})$ . Assume that  $\mu(dx) = \mu_1(dx_1)\mu_2(dx_2)$  and  $\mathbf{t}(\mathbf{x}) = (\mathbf{t}_1(\mathbf{x}_1), \mathbf{t}_2(\mathbf{x}_2))$ . Then the marginal distribution of  $\mathbf{x}_1$  is
$$
P _ {1} (d \pmb {x} _ {1}) = \frac {\mathrm {T r} (\pmb {V} ^ {\top} \pmb {V} \widetilde {\cal {C}} _ {\Theta} (\pmb {x} _ {1}))}{\mathrm {z} (\pmb {V} , \Theta)} \mu_ {1} (d \pmb {x} _ {1}),
$$
where  $\widetilde{C} (\pmb {x}_1)_{ij} = k_{\sigma ,\pmb {t}_2,\mu_2}\Bigl ((\pmb{w}_{2i},\pmb{w}_{1i}^\top \pmb {t}_1(\pmb {x}_1) + b_i),(\pmb{w}_{2j},\pmb{w}_{1j}^\top \pmb {t}_1(\pmb {x}_1) + b_j)\Bigr).$
Proof. The marginal distribution of the random variable  $\mathbf{x}_1$  is obtained by marginalising out the joint distribution with respect to  $\mathbf{x}_2$ ,
$$
P_{1}(d\boldsymbol{x}_{1}) = \frac{1}{\mathrm{z}(\boldsymbol{V},\boldsymbol{\Theta})}\underbrace{\left(\int_{\mathbb{X}_{2}}\left\| \boldsymbol{V}\sigma\big(\boldsymbol{W}_{1}\boldsymbol{t}_{1}(\boldsymbol{x}_{1}) + \boldsymbol{W}_{2}\boldsymbol{t}_{2}(\boldsymbol{x}_{2}) + \boldsymbol{b}\big)\right\|_{2}^{2}\mu_{2}(d\boldsymbol{x}_{2})\right)}_{\triangleq_{\mathbb{Z}_{2}}}\mu_{1}(d\boldsymbol{x}_{1}).
$$
The integral  $\mathbf{z}_2$  takes a similar form to  $\mathbf{z}(\mathbf{V},\Theta)$
$$
\mathrm {z} _ {2} = \operatorname {T r} \left(\boldsymbol {V} ^ {\top} \boldsymbol {V} \widetilde {\boldsymbol {C}} _ {\boldsymbol {\Theta}} (\boldsymbol {x} _ {1})\right), \quad \text {w h e r e}
$$
$$
\begin{array}{l} \widetilde {C} _ {i j} \left(\boldsymbol {x} _ {1}\right) = \int_ {\mathbb {X} _ {2}} \sigma \left(\boldsymbol {w} _ {1 i} ^ {\top} \boldsymbol {t} _ {1} \left(\boldsymbol {x} _ {1}\right) + \boldsymbol {w} _ {2 i} ^ {\top} \boldsymbol {t} _ {2} \left(\boldsymbol {x} _ {2}\right) + b _ {i}\right) \sigma \left(\boldsymbol {w} _ {1 j} ^ {\top} \boldsymbol {t} _ {1} \left(\boldsymbol {x} _ {1}\right) + \boldsymbol {w} _ {2 j} ^ {\top} \boldsymbol {t} _ {2} \left(\boldsymbol {x} _ {2}\right) + b _ {j}\right) \mu_ {2} \left(d \boldsymbol {x} _ {2}\right) \\ = k _ {\sigma , \boldsymbol {t} _ {2}, \mu_ {2}} \left(\left(\boldsymbol {w} _ {2 i}, \boldsymbol {w} _ {1 i} ^ {\top} \boldsymbol {t} _ {1} (\boldsymbol {x} _ {1}) + b _ {i}\right), \left(\boldsymbol {w} _ {2 j}, \boldsymbol {w} _ {1 j} ^ {\top} \boldsymbol {t} _ {1} (\boldsymbol {x} _ {1}) + b _ {j}\right)\right). \\ \end{array}
$$
B Derivation of neural network kernels
Kernel 1.  $k_{\sigma, \mathrm{Id}, \Phi_{C, m}}(\theta_i, \theta_j) = k_{\sigma, \mathrm{Id}, \Phi}(\mathcal{T}\theta_i, \mathcal{T}\theta_j)$ , where  $\mathcal{T}\Theta = (\boldsymbol{W}\boldsymbol{A}, \boldsymbol{b} + \boldsymbol{W}\boldsymbol{m})$ ,  $\mathcal{T}\theta_i = (\boldsymbol{w}_i^\top \boldsymbol{A}, b_i + \boldsymbol{w}_i^\top \boldsymbol{m})$  and  $\boldsymbol{A}$  is a matrix factor such that covariance  $\boldsymbol{C} = \boldsymbol{A}\boldsymbol{A}^\top$ .
Proof. The NNK may be expressed as an expectation with respect to a Gaussian random variable  $\mathbf{x}$  with mean  $m$  and covariance matrix  $C$ . It holds that  $\mathbf{x} \stackrel{d}{=} \mathbf{A}\mathbf{z} + m$ , where  $\mathbf{z}$  is a zero-mean independent standard Gaussian random vector, so the kernel may be expressed in terms of an expectation over  $\mathbf{z}$  instead. More concretely,
$$
\begin{array}{l} k _ {\sigma , \operatorname {I d}, \Phi_ {C, m}} (\boldsymbol {\theta} _ {i}, \boldsymbol {\theta} _ {j}) = \mathbb {E} _ {\mathbf {x}} \left[ \sigma \left(\boldsymbol {w} _ {i} ^ {\top} \mathbf {x} + b _ {i}\right) \sigma \left(\boldsymbol {w} _ {j} ^ {\top} \mathbf {x} + b _ {j}\right) \right] \\ = \mathbb {E} _ {\mathbf {z}} \left[ \sigma \left(\boldsymbol {w} _ {i} ^ {\top} \boldsymbol {A} \mathbf {z} + \boldsymbol {w} _ {i} ^ {\top} \boldsymbol {m} + b _ {i}\right) \sigma \left(\boldsymbol {w} _ {j} ^ {\top} \boldsymbol {A} \boldsymbol {z} + \boldsymbol {w} _ {j} ^ {\top} \boldsymbol {m} + b _ {j}\right) \right] \\ = k _ {\sigma , \operatorname {I d}, \Phi} (\mathcal {T} \boldsymbol {\theta} _ {i}, \mathcal {T} \boldsymbol {\theta} _ {j}). \\ \end{array}
$$
Kernel 2.  $k_{\mathrm{cos,Id,}\Phi}(\pmb{\theta}_i,\pmb{\theta}_j) = \frac{\cos|b_i - b_j|}{2}\exp \left(\frac{-\|\pmb{w}_j - \pmb{w}_j\|^2}{2}\right) + \frac{\cos|b_i + b_j|}{2}\exp \left(\frac{-\|\pmb{w}_j + \pmb{w}_j\|^2}{2}\right).$
Proof. First observe that the expected value of the cosine of a Gaussian random variable can be evaluated by equating the real and imaginary components of the characteristic function of a Gaussian random variable and the expected value of Euler's form. That is, if  $z$  is Gaussian with mean  $m$  and variance  $v^2$ ,
$$
\begin{array}{l} \mathbb {E} e ^ {i \mathbf {z}} = \mathbb {E} [ \cos (\mathbf {z}) ] + i \mathbb {E} [ \sin (\mathbf {z}) ] = e ^ {i m - \frac {1}{2} v ^ {2}} \\ = (\cos \mu + i \sin m) e ^ {- \frac {1}{2} v ^ {2}} \\ \Longrightarrow \mathbb {E} [ \cos (z) ] = \cos (m) e ^ {- \frac {1}{2} v ^ {2}}. \\ \end{array}
$$
With this identity at hand, we proceed by direct evaluation of (4).