SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Russell Tsuchida* Data61-CSIRO
Cheng Soon Ong  
Data61-CSIRO & Australian National University
Dino Sejdinovic  
School of CMS & AIML, The University of Adelaide
Flexible models for probability distributions are an essential ingredient in many machine learning tasks. We develop and investigate a new class of probability distributions, which we call a Squared Neural Family (SNEFY), formed by squaring the 2-norm of a neural network and normalising it with respect to a base measure. Following the reasoning similar to the well established connections between infinitely wide neural networks and Gaussian processes, we show that SNEFYs admit closed form normalising constants in many cases of interest, thereby resulting in flexible yet fully tractable density models. SNEFYs strictly generalise classical exponential families, are closed under conditioning, and have tractable marginal distributions. Their utility is illustrated on a variety of density estimation, conditional density estimation, and density estimation with missing data tasks.
Probabilistic modelling lies at the heart of machine learning. In both traditional and contemporary settings, ensuring the probability model is appropriately normalised (or otherwise bypassing the need to compute normalising constants) is of central interest for maximum likelihood estimation and related statistical inference procedures. Tractable normalising constants allow the use of probability models in a variety of applications, such as anomaly detection, denoising and generative modelling.
Traditional statistical approaches [27, 5] rely on mathematically convenient models such as exponential families [50]. Such models can often be normalised in closed form, but are often only suitable for relatively low-dimensional and simple data. Early approaches in deep learning use neural networks as energy functions inside Gibbs distributions [21, Equation 2]. Such distributions typically have very intractable normalising constants, and so either surrogate losses for the negative log likelihood involving the energy function are used or MCMC, score matching [16], variational or sampling methods [15, 30] are used to approximate the normalising constant. See [35, §2.6.1, §2.2] for more energy-based and other methods.
Modern computer technology and power allows us more flexible models [8, page 68], by partially or completely relaxing the requirement for mathematical tractability. Contemporary high-dimensional modelling, such as generative image models, rely primarily on neural network models [18, 11, 35, 34]. For example, normalising flows [40] use constrained neural network layers to transform random variables from base measures with tractable Jacobian determinants associated with appropriately constrained but flexible pushforward maps. Similarly to normalising flows, we use neural networks to define expressive densities, but we model the densities directly without using constrained transformations of random variables. Moreover, our approach can be readily applied in conjunction with normalising flows and other deep learning devices such as deep feature extractors.
*Software available at https://github.com/RussellTsuchida/snefy.
Support X	Base measure μ	Sufficient statistic t(x)	Activation function σ	b ≠ 0	Kernel kσ,t,μ
Any setting mirroring a previously derived NNGPK, e.g.
Rd	ΦC,0	Id(x)	erf	X	[52]
(·)p+, p ∈ N	X	[4]
LReLU	X	[48]
GELU [14]	X	[47]
cos	X	[37]
Any tractable exponential family [50, 32] setting, e.g.
Rd	ΦC,m	x/||x||2	exp	✓	Kernel 3
Sd-1	Uniform	Id(x)	✓
Rd	ΦC,m	✓	Kernel 7
{0,1,2,...}	(x!)-1ν	✓	Kernel 8
New tractable integration settings, e.g.
Rd	ΦC,m	Id(x)	cos	✓	Kernel 2
[0,1]d	Uniform	Φ-1(x)
Rd	ΦC,m	Id(x)	Snakea[55, 39]	✓	Kernel 6
Table 1: Examples of settings admitting a closed-form for the normalising constant  $\mathbf{z}(\mathbf{V},\Theta)$  (2) by leveraging a closed-form NNK  $k_{\sigma ,t,\mu}$  (4). In each case,  $\mathrm{z}(V,\Theta) = \mathrm{Tr}(V^{\top}VK_{\Theta})$ , where the entries of the matrix  $K_{\Theta}$  are described according to the NNK  $k_{\sigma ,t,\mu}$  in Identity 1.  $\Phi_{C,m}$  denotes the CDF of a multivariate normal distribution with mean  $m$  and covariance matrix  $C$  and  $\nu$  denotes counting measure. Rows with citations have been considered previously in the context of NNGPKs, but not as normalising constants and not with a reversal of the role of input and parameter. Note the cases where  $b\neq 0$ ; this setting is not considered by others, because when the role of parameters and data is in the usual setting,  $b = 0$  covers a sufficiently general setting. Noticing that SNEFYs strictly generalise exponential family mixture models (see § 3.2), fixing  $\sigma (\cdot) = \exp (\cdot /2)$  and a given base measure  $\mu$  and sufficient statistic  $t$  for which the exponential family log-partition function is known also leads to tractable normalising constants. More known closed-form kernels as well as approximate kernels that can be adapted to our setting are given in [12].
Contributions. Let  $(\Omega, \mathcal{F}, \mu)$  denote a measure space with  $\Omega \subseteq \mathbb{R}^d$ , sigma algebra  $\mathcal{F}$ , and nonnegative measure  $\mu$ . Let  $V \in \mathbb{R}^{m \times n}$  be the readout parameters of a neural network and let  $W \in \mathbb{R}^{n \times D}$  and  $b \in \mathbb{R}^n$  be the weights and biases of a hidden layer of a neural network  $f: \mathbb{R}^D \to \mathbb{R}^m$  with activation function  $\sigma$ . For some support  $\mathbb{X} \in \mathcal{F}$ , define a probability distribution  $P$  (and corresponding probability density  $p$  with respect to base measure  $\mu$ ) to be proportional to squared 2-norm of the evaluation of the neural network  $f$ ,
$$
P (d \boldsymbol {x}; \boldsymbol {V}, \boldsymbol {\Theta}) \triangleq \frac {\mu (d \boldsymbol {x})}{\mathrm {z} (\boldsymbol {V} , \boldsymbol {\Theta})} \| \boldsymbol {f} (\boldsymbol {t} (\boldsymbol {x}); \boldsymbol {V}, \boldsymbol {\Theta}) \| _ {2} ^ {2}, \quad \boldsymbol {f} (\boldsymbol {t}; \boldsymbol {V}, \boldsymbol {\Theta}) = V \sigma (\boldsymbol {W} t + \boldsymbol {b}), \boldsymbol {\Theta} = (\boldsymbol {W}, \boldsymbol {b}), \tag {1}
$$
whenever the normalising constant  $\mathbf{z}(\boldsymbol{V},\boldsymbol{\Theta}) \triangleq \int_{\mathbb{X}}\| \boldsymbol{f}(\boldsymbol{t}(\boldsymbol{x});\boldsymbol{V},\boldsymbol{\Theta})\|_2^2\mu(\boldsymbol{d}\boldsymbol{x})$  is finite and non-zero. Here we call  $\mu$  the base measure,  $\boldsymbol{t}:\mathbb{X}\to \mathbb{R}^{D}$  the sufficient statistic and  $\sigma$  the activation function. We will call the corresponding family of probability distributions, parametrised by  $(\boldsymbol{V},\boldsymbol{\Theta})$ , a squared neural family (SNEFY) on  $\mathbb{X}$ , and denote it by SNEFYx,t,σ,μ. SNEFYs are new flexible probability distribution models that strike a good balance between mathematical tractability, expressivity and computational efficiency. When a random vector  $\mathbf{x}$  follows a SNEFY distribution indexed by parameters  $(\boldsymbol{V},\boldsymbol{\Theta})$ , we write  $\mathbf{x} \sim \mathrm{SNEFY}_{\mathbb{X},t,\sigma,\mu}(\boldsymbol{V},\boldsymbol{\Theta})$  or simply  $\mathbf{x} \sim P(\cdot;\boldsymbol{V},\boldsymbol{\Theta})$ , where there is no ambiguity.
Our main technical challenge is in exactly computing the normalising constant,  $\mathbf{z}(\mathbf{V},\Theta)$ , where
$$
\mathrm {z} (\boldsymbol {V}, \boldsymbol {\Theta}) \triangleq \int_ {\mathbb {X}} \| \boldsymbol {f} (\boldsymbol {t} (\boldsymbol {x}); \boldsymbol {V}, \boldsymbol {\Theta}) \| _ {2} ^ {2} \mu (d \boldsymbol {x}), \quad \boldsymbol {f} (\boldsymbol {t}; \boldsymbol {V}, \boldsymbol {\Theta}) = \boldsymbol {V} \sigma (\boldsymbol {W} \boldsymbol {t} + \boldsymbol {b}), \quad \boldsymbol {\Theta} = (\boldsymbol {W}, \boldsymbol {b}). \tag {2}
$$
The normalising constants we consider are special cases in the sense that they apply to specific (but commonly appearing in applications) choices of activation function  $\sigma$ , sufficient statistic  $t$  and base measure  $\mu$  over support  $\mathbb{X}$ . See Table 1. Our analysis both exploits and informs a connection with
2We later verify that  $t$  is indeed a sufficient statistic, see (7). Note  $\pmb{t}$  maps from  $d$  to  $D$  dimensions.
so-called neural network Gaussian process kernels (NNGPKs) [31] in a generalised form, which we refer to as neural network kernels (NNKs).
We discuss some important theoretical properties of SNEFY such as exact normalising constant calculation, marginal distributions, conditional distributions, and connections with other probability models. We then consider a deep learning setting, where SNEFYs can either be used as base distributions in (non-volume preserving) normalising flows [46], or may describe flexible conditional density models with deep learning feature extractors. We demonstrate SNEFY on a variety of datasets.
Notation We use lower case non-bold (like  $a$ ) to denote scalars, lower case bold to denote vectors (like  $a$ ) and upper case bold to denote matrices (like  $A$ ). Random variables (scalar or vector) are additionally typeset in sans-serif (like  $a$  and  $a$ ). The special zero vector  $(0,\ldots,0)$  and identity matrix elements are  $0$  and  $I$ . We use subscripts to extract (groups of) indices, so that for example,  $\boldsymbol{w}_i$  is the  $i$ th row of the matrix  $\boldsymbol{W}$  (as a column vector),  $\boldsymbol{v}_{\cdot,i}$  is the  $i$ th column of the matrix  $\boldsymbol{V}$ , and  $b_i$  is the  $i$ th element of the vector  $\boldsymbol{b}$ . We use  $\Theta = (\boldsymbol{W},\boldsymbol{b}) \in \mathbb{R}^{n\times(D+1)}$  to denote the concatenated hidden layer weights and biases. Correspondingly, we write  $\theta_i = (\boldsymbol{w}_i,b_i) \in \mathbb{R}^{D+1}$  for the  $i$ th row of  $\Theta$ . We will use a number of special functions.  $\Phi_{C,m}$  denotes the cumulative distribution function (CDF) of a Gaussian random vector with mean  $m$  and covariance matrix  $\boldsymbol{C}$ . We also use a shorthand  $\Phi_C = \Phi_{C,0}$  and  $\Phi = \Phi_{I,0}$ .
Single hidden layer neural networks We consider a feedforward neural network  $\pmb{f} : \mathbb{R}^{D} \to \mathbb{R}^{m}$ ,
$$
\boldsymbol {f} (\boldsymbol {t}; \boldsymbol {V}, \boldsymbol {\Theta}) = \boldsymbol {V} \sigma (\boldsymbol {W} \boldsymbol {t} + \boldsymbol {b}) \tag {3}
$$
with activation function  $\sigma$ , hidden weights  $\pmb{W} \in \mathbb{R}^{n \times D}$  and biases  $\pmb{b} \in \mathbb{R}^n$  and readout parameters  $\pmb{V} \in \mathbb{R}^{m \times n}$ . Here  $\sigma$  is applied element-wise to its vector inputs, returning a vector of the same shape.
Neural network kernels In certain theories of deep learning, one often encounters a bivariate Gaussian integral called the neural network Gaussian process kernel NNGPK. The NNGPK first arose as the covariance function of a well-behaved single layer neural network with random weights [31]. In the limit as the width of the network grows to infinity, the neural network (3) with suitably well-behaved (say, independent Gaussian) random weights converges to a zero-mean Gaussian process, so that the NNGPK characterises the law of the neural network predictions. These limiting models can be used as functional priors in a classical Gaussian process sense [53].
In our setting, the positive semidefinite (PSD) NNGPK appears in an entirely novel context, where the role of the hidden weights and biases  $\theta_{i} = (\pmb{w}_{i},b_{i})$  and the data  $\pmb{x}$  is reversed. Instead of marginalising out the parameters and evaluating at the data, we marginalise out the data and evaluate at the parameters. The NNGPK  $k_{\sigma ,\mathrm{Id},\Phi_I}$  admits a representation of the form
$$
k _ {\sigma , \operatorname {I d}, \Phi_ {I}} \left(\boldsymbol {\theta} _ {i}, \boldsymbol {\theta} _ {j}\right) \triangleq \mathbb {E} _ {\mathbf {x}} \left[ \sigma \left(\boldsymbol {w} _ {i} ^ {\top} \mathbf {x} + b _ {i}\right) \sigma \left(\boldsymbol {w} _ {j} ^ {\top} \mathbf {x} + b _ {j}\right) \right], \quad \mathbf {x} \sim \mathcal {N} (\mathbf {0}, I). \tag {4}
$$
We do not discuss in detail how it is constructed in earlier works [31, 22, 33, 17], where usually  $b_{i} = b_{j} = 0^{3}$ , but not always[49]. When  $b_{i} = b_{j} = 0$ , closed-form expressions for the NNGPK are available for different choices of  $\sigma$  and  $\pmb{t}$  [52, 20, 4, 48, 37, 47, 28, 13]. However, the setting of  $\pmb{b} \neq \mathbf{0}$  is important in our context (as we show in §2.2) and presents additional analytical challenges.
We will require a more general notion of an NNGPK which we call a neural network kernel (NNK). We introduce a function  $t$  which may be thought of as a warping function applied to the input data. Such warping is common in kernels and covariance functions and can be used to induce desirable analytical and practical properties [37, 29, 24, §5.4.3]. We also integrate with respect to more general measures  $\mu$  instead of the standard Gaussian CDF,  $\Phi$ . We define the NNK to be
$$
k _ {\sigma , \boldsymbol {t}, \mu} \left(\boldsymbol {\theta} _ {i}, \boldsymbol {\theta} _ {j}\right) \triangleq \int_ {\mathbb {X}} \sigma \left(\boldsymbol {w} _ {i} ^ {\top} \boldsymbol {t} (\boldsymbol {x}) + b _ {i}\right) \sigma \left(\boldsymbol {w} _ {j} ^ {\top} \boldsymbol {t} (\boldsymbol {x}) + b _ {j}\right) \mu (d \boldsymbol {x}). \tag {5}
$$
<sup>3</sup>After accounting for the reversal of parameters and data.
2 Closed form squared neural families
2.1 Normalising constants
Observe from (2) that by swapping the order of integration and multiplication by  $V$ , the normalising constant is quadratic in elements of  $V$ . The coefficients of the quadratic depend on  $\Theta = (W, b)$ . We now characterise these coefficients of the quadratic in terms of the NNK evaluated at rows  $\theta_i, \theta_j$  of  $\Theta$ , which are totally independent of  $V$ . The proof of the following is given in Appendix A.
Identity 1. The integral (2) admits a representation of the form
$$
\mathrm {z} (\boldsymbol {V}, \boldsymbol {\Theta}) = \operatorname {T r} \left(\boldsymbol {V} ^ {\top} \boldsymbol {V} \boldsymbol {K} _ {\boldsymbol {\Theta}}\right) \tag {6}
$$
where  $k_{\sigma, \mathbf{t}, \mu}$  is as defined in (5), and  $\pmb{K}_{\Theta}$  is the PSD matrix whose  $i$ th entry is  $k_{\sigma, \mathbf{t}, \mu}(\pmb{\theta}_i, \pmb{\theta}_j)$ .
By Identity 1, the normalised measure (1) then admits the explicit representation
$$
P (d \pmb {x}; \pmb {V}, \pmb {\Theta}) = \frac {\mathrm {T r} (\pmb {V} ^ {\top} \pmb {V} \widetilde {\pmb {K} _ {\Theta}} (\pmb {x}))}{\mathrm {T r} (\pmb {V} ^ {\top} \pmb {V} \pmb {K} _ {\Theta})} \mu (d \pmb {x}) = \frac {\mathrm {v e c} (\pmb {V} ^ {\top} \pmb {V}) ^ {\top} \mathrm {v e c} (\widetilde {\pmb {K} _ {\Theta}} (\pmb {x}))}{\mathrm {v e c} (\pmb {V} ^ {\top} \pmb {V}) ^ {\top} \mathrm {v e c} (\pmb {K} _ {\Theta})} \mu (d \pmb {x}),
$$
where  $\widetilde{K_{\Theta}} (\pmb {x})$  is the PSD matrix whose  $ij$ th entry is  $\sigma (\pmb {w}_i^\top \pmb {t}(\pmb {x}) + b_i)\sigma (\pmb {w}_j^\top \pmb {t}(\pmb {x}) + b_j)^\top$ . We used the cyclic property of the trace, writing the numerator as  $\operatorname {Tr}\left(\sigma^{\top}\boldsymbol{V}^{\top}\boldsymbol {V}\sigma\right) = \operatorname {Tr}\left(\boldsymbol {V}\sigma \sigma^{\top}\boldsymbol{V}^{\top}\right) =$ $\operatorname {Tr}\left(V^{\top}V\sigma \sigma^{\top}\right)$ . We emphasise again that the role of the data  $\pmb {t}(\pmb {x})$  and the hidden weights and biases  $\Theta$  in the NNK  $k_{\sigma ,t,\mu}$  are reversed compared with how they have appeared in previous settings. We may compute evaluations of  $k_{\sigma ,t,\mu}$  in closed form for special cases of  $(\sigma ,t,\mu)$  using various identities in  $\mathcal{O}(d)$ , where  $d$  is the dimensionality of the domain of integration, as we soon detail in § 2.2. Combined with the trace inner product, this leads to a total cost of computing  $\mathrm{z}(\pmb {V},\Theta)$  of  $\mathcal{O}(m^2 n + dn^2)$ , where  $n$  and  $m$  are respectively the number of neurons in the first and second layers.
Remark 1 (Alternative parameterisations). SNEFY models depend on readout parameters  $V$  only through the direction of  $\operatorname{vec}(V^\top V)$  and not on its norm or sign. For example, one can always find another parameterisation of readout parameters that results in the same probability distribution but has a normalising constant of 1. Furthermore, noticing that  $V$  only appears as a PSD matrix  $M \triangleq V^\top V$  of rank at most  $\min(m, n)$ , one may alternatively parameterise a SNEFY by  $(M, \Theta)$ .
2.2 Neural network kernels
In § 2.1, we reduced computation of the integral (2) to computation of a quadratic involving evaluations of the NNK (5). Several closed-forms are known for different settings of  $\sigma$ , all with  $t = \mathrm{Id}$  and  $b = 0$ . The motivation behind derivation of existing known results is from the perspective of inference in infinitely wide Bayesian neural networks [31] or to derive certain integrals involved in computing predictors of infinitely wide neural networks trained using gradient descent [17]. Here we describe some new settings that have not been investigated previously that are useful for the new setting of SNEFY. Recall from (5), that our kernel,  $k_{\sigma, t, \mu}$ , is parametrised by activation function  $\sigma$ , warping function (sufficient statistic)  $t$ , and base measure  $\mu$ . All derivations are given in Appendix B.
The first kernel describes how we may express the kernels with arbitrary Gaussian base measures  $\Phi_{C,m}$  in terms of the kernels with isotropic Gaussian base measures  $\Phi$ . This means that it suffices to consider isotropic Gaussian base measures in place of arbitrary Gaussian base measures.
Kernel 1.  $k_{\sigma, \mathrm{Id}, \Phi_C, m}(\theta_i, \theta_j) = k_{\sigma, \mathrm{Id}, \Phi}(\mathcal{T}\theta_i, \mathcal{T}\theta_j)$ , where  $\mathcal{T}\Theta = (W\mathbf{A}, \mathbf{b} + W\mathbf{m})$ ,  $\mathcal{T}\theta_i = (\boldsymbol{w}_i^\top \mathbf{A}, b_i + \boldsymbol{w}_i^\top \mathbf{m})$  and  $\mathbf{A}$  is a matrix factor such that covariance  $\mathbf{C} = \mathbf{A}\mathbf{A}^\top$ .
This kernel can also be used to describe kernels corresponding with Gaussian mixture model base measures. The second kernel we describe is a minor extension to the case  $b \neq 0$  of a previously considered kernel [37].
Kernel 2.  $k_{\mathrm{cos,Id,}\Phi}(\pmb {\theta}_i,\pmb {\theta}_j) = \frac{\cos|b_i - b_j|}{2}\exp \left(\frac{-\|\pmb{w}_j - \pmb{w}_j\|^2}{2}\right) + \frac{\cos|b_i + b_j|}{2}\exp \left(\frac{-\|\pmb{w}_j + \pmb{w}_j\|^2}{2}\right).$
A similar result and derivation holds for the case of  $k_{\sin, \mathrm{Id}, \Phi}$ , which we do not reproduce here. We now mention a case that shares a connection with exponential families (see § 3.2 for a detailed description of this connection). The following and more  $\sigma = \exp$  cases are derived in Appendix C.
Kernel 3. Define  $\operatorname{proj}_{\mathbb{S}^{d-1}}(\boldsymbol{x}) \triangleq \boldsymbol{x} / \| \boldsymbol{x} \|$  to be the projection onto the unit sphere. Then
$$
k _ {\exp , p r o j _ {\mathbb {S} ^ {d - 1}}, \Phi} (\boldsymbol {\theta} _ {i}, \boldsymbol {\theta} _ {j}) = \exp (b _ {i} + b _ {j}) \frac {\Gamma (d / 2) 2 ^ {d / 2 - 1} I _ {d / 2 - 1} \left(\| \boldsymbol {w} _ {i} + \boldsymbol {w} _ {j} \|\right)}{\| \boldsymbol {w} _ {i} + \boldsymbol {w} _ {j} \| ^ {d / 2 - 1}},
$$
where  $I_{p}$  is the modified Bessel function of the first kind of order  $p$ . In the special case  $d = 3$ , we have the closed-form  $k_{\exp,proj_{\mathbb{S}^2},\Phi}(\pmb{\theta}_i,\pmb{\theta}_j) = \exp(b_i + b_j)\frac{(e^{||\pmb{w}_i + \pmb{w}_j||} - e^{-||\pmb{w}_i + \pmb{w}_j||})}{2||\pmb{w}_i + \pmb{w}_j||}$ .
We end this section with a new analysis of the Snake  $a$  activation function, given by
$$
\operatorname {S n a k e} _ {a} (z) = z + \frac {1}{a} \sin^ {2} (a z) = z - \frac {1}{2 a} \cos (2 a z) + \frac {1}{2 a}.
$$
The  $\mathrm{Snake}_a$  function [55] is a neural network activation function that can resemble the ReLU on an interval for special choices of  $a$ , is easy to differentiate, and as we see shortly, admits certain attractive analytical tractability. We note that a similar activation function has been found using reinforcement learning to search for good activation functions [39, Table 1 and 2, row 3], up to an offset and hyperparameter  $a = 1$ . The required kernel is expressed in terms of the linear kernel (Kernel 4) and the kernel corresponding with the activation function of [39], i.e. snake without the offset,  $\mathrm{Snake}_a(\cdot) - \frac{1}{2a}$  (Kernel 5). We first describe the linear kernel.
Kernel 4.  $k_{\mathrm{Id},\mathrm{Id},\Phi}(\pmb {\theta}_i,\pmb {\theta}_j) = \pmb {w}_i^\top \pmb {w}_j + b_ib_j.$
We now derive the kernel corresponding with  $\mathrm{Snake}_{\mathrm{a}}$  activation functions up to an offset.
Kernel 5. The kernel  $k_{\mathrm{Snake_a}(\cdot) - \frac{1}{2a},\mathrm{Id},\Phi}(\pmb {\theta}_i,\pmb {\theta}_j)$  is equal to
$$
\begin{array}{l} \frac {1}{4 a ^ {2}} k _ {\cos , \operatorname {I d}, \Phi} (2 a \boldsymbol {\theta} _ {i}, 2 a \boldsymbol {\theta} _ {j}) + \boldsymbol {w} _ {j} ^ {\top} \boldsymbol {w} _ {j} \left(\sin (2 a b _ {j}) e ^ {- 2 a ^ {2} \| \boldsymbol {w} _ {j} \| ^ {2}} + \sin (2 a b _ {i}) e ^ {- 2 a ^ {2} \| \boldsymbol {w} _ {i} \| ^ {2}}\right) \\ - \frac {b _ {i}}{2 a} \cos (2 a b _ {j}) e ^ {- 2 a ^ {2} \| \boldsymbol {w} _ {j} \| ^ {2}} - \frac {b _ {j}}{2 a} \cos (2 a b _ {i}) e ^ {- 2 a ^ {2} \| \boldsymbol {w} _ {i} \| ^ {2}} + k _ {\mathrm {I d}, \mathrm {I d}, \Phi} (\boldsymbol {\theta} _ {i} ^ {(1)}, \boldsymbol {\theta} _ {j} ^ {(1)}). \\ \end{array}
$$
The kernel corresponding with  $\mathrm{Snake_a}$  activations is then stated in terms of Kernel 4 and 5.
Kernel 6. The kernel  $k_{\mathrm{Snake}_{\mathrm{a}}, \mathrm{Id}, \Phi}(\pmb{\theta}_i, \pmb{\theta}_j)$  is equal to
$$
\begin{array}{l} \frac {1}{2 a} \left(b _ {i} - \frac {1}{2 a} \cos (2 a b _ {i}) \exp (- 2 a ^ {2} \| \boldsymbol {w} _ {i} \| ^ {2}) + b _ {j} - \frac {1}{2 a} \cos (2 a b _ {j}) \exp (- 2 a ^ {2} \| \boldsymbol {w} _ {j} \| ^ {2})\right) \\ + k _ {\mathrm {S n a k e} _ {\mathrm {a}} (\cdot) - \frac {1}{2 a}, \mathrm {I d}, \Phi} (\boldsymbol {\theta} _ {i}, \boldsymbol {\theta} _ {j}) + \frac {1}{4 a ^ {2}}. \\ \end{array}
$$
3 Properties of squared neural families
3.1 Fisher-Neyman factorisation and sufficient statistics
If the base measure  $\mu$  is absolutely continuous with respect to some measure  $\nu$ , and  $\frac{d\mu}{d\nu}:\Omega \to [0,\infty)$  is the Radon-Nikodym derivative, then the SNEFY admits a probability density function  $p(\cdot |\mathbf{V},\Theta)$  with respect to  $\nu$ ,
$$
p (\boldsymbol {x} \mid \boldsymbol {V}, \boldsymbol {\Theta}) = \underbrace {\frac {d \mu}{d \nu} (\boldsymbol {x})} _ {\text {I n d e p e n d e n t o f} \boldsymbol {V}, \boldsymbol {\Theta}} \times \underbrace {\frac {\left\| \boldsymbol {f} \left(\boldsymbol {t} (\boldsymbol {x}) ; \boldsymbol {V} , \boldsymbol {\Theta}\right) \right\| _ {2} ^ {2}}{\mathrm {z} (\boldsymbol {V} , \boldsymbol {\Theta})}} _ {\text {D e p e n d s o n} \boldsymbol {x} \text {o n l y t h r o u g h} \boldsymbol {t} (\boldsymbol {x})}. \tag {7}
$$
The Fisher-Neyman theorem (for example, see Theorem 6.14 of [6]) says that the existence of such a factorisation is equivalent to the fact that  $\pmb{t}$  is a sufficient statistic for the parameters  $V, \Theta$ .
3.2 Connections with exponential families
In this section we will use the activation  $\sigma(u) = \exp(u/2)$ . We note that we can absorb the bias terms  $b$  into the  $V$  parameters<sup>4</sup> and obtain as a special case the following family of distributions