SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\inf  _ {F _ {\theta} ^ {\mathrm {S C}}} \left\| H - F _ {\theta} ^ {\mathrm {S C}} \right\| \geq \inf  _ {F \in E} \| H - F \| = \| (1 - \Pi) H \| > 0. \tag {20}
$$
First, we give a statement of the universal approximation theorem as in [33].
Theorem C.1. (Universal Approximation Theorem) The continuous transfer function  $\phi$  is not polynomial if and only if for every  $k\in \mathbb{N},m\in \mathbb{N}$ , compact set  $U\subset \mathbb{R}^k$ ,  $f\in C(U,\mathbb{R}^{m})$ , and  $\varepsilon >0$ , there exists  $n\in \mathbb{N},W_2\in \mathbb{R}^{n\times k},b\in \mathbb{R}^n,W_1\in \mathbb{R}^{m\times n}$  such that
$$
\sup  _ {x \in U} \| f (x) - g (x) \| <   \varepsilon \tag {21}
$$
where  $g(x) = W_1\phi(W_2x + b)$ . Or equivalently,  $\Sigma_n = \text{span} \left\{ \phi(\mathbf{w} \cdot \mathbf{x} + b) : \mathbf{w} \in \mathbb{R}^k, b \in \mathbb{R} \right\}$  is dense in  $C(\mathbb{R}^k)$  if and only if  $\phi$  is not polynomial.
The theorem says that for any transfer function that is not polynomial (tanh for example), there is a multi-layer neural network with the transfer function that is able to approximate any continuous function to arbitrary precision on a closed and bounded subset of  $\mathbb{R}^m$ . The proof of Theorem 3 proceeds in three steps:
1. Choose a large enough compact support  $U$  for  $s_{\theta}(\mathbf{x})$  such that
$$
\int_ {\mathbb {R} ^ {m} \backslash U} p (\mathbf {x}) \| \nabla \log p (\mathbf {x}) - s _ {\theta} (\mathbf {x}) \| ^ {2} d \mathbf {x} = \int_ {\mathbb {R} ^ {m} \backslash U} p (\mathbf {x}) \| \nabla \log p (\mathbf {x}) \| ^ {2} d \mathbf {x} <   \frac {\varepsilon}{2} \tag {22}
$$
2. Choose a large enough  $n$  and appropriate  $\theta$  such that the universal approximation theorem can be applied to show that
$$
\int_ {U} p (\mathbf {x}) \left\| \nabla \log p (\mathbf {x}) - s _ {\theta} (\mathbf {x}) \right\| ^ {2} d \mathbf {x} <   \frac {\varepsilon}{2} \tag {23}
$$
3. Combine equation (22) and (23) so that we have
$$
\mathbb {E} _ {\mathbf {x} \sim p (\mathbf {x})} \left[ \| \nabla \log p (\mathbf {x}) - s _ {\theta} (\mathbf {x}) \| ^ {2} \right] = \int_ {\mathbb {R} ^ {m}} p (\mathbf {x}) \| \nabla \log p (\mathbf {x}) - s _ {\theta} (\mathbf {x}) \| ^ {2} d \mathbf {x} <   \varepsilon \tag {24}
$$
We begin with two lemmas that will be helpful in the proof of the theorem. Both of them are standard results<sup>3</sup>, but we include here for completeness. Let the upper incomplete Gamma function be defined by  $\Gamma(s, C) = \int_{C}^{\infty} r^{s-1} e^{-r} dr$ .
Lemma C.2. If  $s$  is a positive integer, then  $\Gamma(s, C) = (s - 1)!e^{-C}\sum_{i=0}^{s-1}\frac{C^i}{i!}$ .
Proof. This can be proved by induction. For  $s = 1$
$$
\Gamma (1, C) = \int_ {C} ^ {\infty} e ^ {- r} d r = e ^ {- C} \tag {25}
$$
Therefore the conclusion holds. For  $s > 1$ , we assume that for  $s = k$ , the conclusion holds. Then for  $s = k + 1$ , Through integration by parts, we have
$$
\Gamma (k + 1, C) = s \Gamma (k, C) + C ^ {k} e ^ {- C} \tag {26}
$$
$$
\begin{array}{l} \Gamma (k + 1, C) = k (k - 1)! e ^ {- C} \sum_ {i = 0} ^ {k - 1} \frac {C ^ {i}}{i !} + C ^ {k} e ^ {- C} \\ = k! e ^ {- C} \left(\frac {C ^ {k}}{k !} + \sum_ {i = 0} ^ {k - 1} \frac {C ^ {i}}{i !}\right) \tag {27} \\ = k! e ^ {- C} \sum_ {i = 0} ^ {k} \frac {C ^ {i}}{i !} \\ \end{array}
$$
Therefore the conclusion holds for  $s = k + 1$  as well. By induction, it holds for all positive integer  $s$ .
Lemma C.3.  $\Gamma (s,C)\to 0$  as  $C\rightarrow \infty$
Proof. When  $C \to \infty$ ,  $\Gamma(s, C) < \int_{C}^{\infty} r^{\lceil s - 1 \rceil} e^{-r} dr$ . Therefore W.L.O.G., we can assume that  $s - 1$  is a positive integer. By lemma C.2,
$$
\lim  _ {C \rightarrow \infty} \Gamma (s, C) = (s - 1)! \sum_ {i = 0} ^ {s - 1} \lim  _ {C \rightarrow \infty} \frac {e ^ {- C} C ^ {i}}{i !} = 0 \tag {28}
$$
since the exponential term  $e^{-C}$  is beyond all orders.
Now we give a formal proof of Theorem 3.
Theorem. Suppose that we are given a probability distribution with continuously differentiable density function  $p(\mathbf{x}) : \mathbb{R}^m \to \mathbb{R}^+$  and score function  $\nabla \log p(\mathbf{x})$  for which there exist constants  $M_1, M_2, a, k > 0$  such that
$$
p (\mathbf {x}) <   M _ {1} e ^ {- a \| \mathbf {x} \|} \tag {29}
$$
$$
\left\| \nabla \log p (\mathbf {x}) \right\| ^ {2} <   M _ {2} \| \mathbf {x} \| ^ {k} \tag {30}
$$
when  $\| \mathbf{x}\| >L$  for large enough  $L$ . Then for any  $\varepsilon >0$ , there exists a recurrent neural network whose firing-rate dynamics are given by (11), whose recurrent weights, output weights and the diffusion coefficient are given by  $W_{\mathrm{rec}}\in \mathbb{R}^{n\times n}$  of rank  $m$ ,  $W_{\mathrm{out}}\in \mathbb{R}^{m\times n}$ , and  $\sigma \in \mathbb{R}^{n\times m}$  respectively, such that, for a large enough  $n$ , the score of the stationary distribution of the output units  $s_\theta (\mathbf{x})$  satisfies  $\mathbb{E}_{\mathbf{x}\sim p(\mathbf{x})}[\| \nabla \log p(\mathbf{x}) - s_\theta (\mathbf{x})\|^2 ] < \varepsilon$ .
3see https://dlmf.nist.gov/8.11
Step 1 First of all, we would like to find a compact set  $U_{C} = \{\mathbf{x}|\| \mathbf{x}\|_{2}\leq C\}$  such that  $\int_{\mathbb{R}^m\setminus U_C}\| \nabla \log p(\mathbf{x}) - s_\theta (\mathbf{x})\| ^2 p(\mathbf{x})d\mathbf{x} < \frac{\varepsilon}{2}$ . We can upper bound the integral by the following calculation using spherical coordinates when  $C > L$ .
$$
\begin{array}{l} \int_ {\mathbb {R} ^ {m} \backslash U _ {C}} \| \nabla \log p (\mathbf {x}) - s _ {\theta} (\mathbf {x}) \| ^ {2} p (\mathbf {x}) d \mathbf {x} = \int_ {\mathbb {R} ^ {m} \backslash U _ {C}} \| \nabla \log p (\mathbf {x}) \| ^ {2} p (\mathbf {x}) d \mathbf {x} (31) \\ <   \int_ {\mathbb {R} ^ {m} \backslash U _ {C}} M _ {1} M _ {2} e ^ {- a \| \mathbf {x} \|} \| \mathbf {x} \| ^ {k} d \mathbf {x} (32) \\ = \frac {2 M _ {1} M _ {2} \pi^ {m / 2}}{\Gamma (\frac {m}{2})} \int_ {C} ^ {\infty} r ^ {m + k - 1} e ^ {- a r} d r (33) \\ = \frac {2 M _ {1} M _ {2} \pi^ {m / 2}}{a ^ {m + k} \Gamma (\frac {m}{2})} \int_ {a C} ^ {\infty} u ^ {m + k - 1} e ^ {- u} d u (34) \\ = \frac {2 M _ {1} M _ {2} \pi^ {m / 2}}{a ^ {m + k} \Gamma (\frac {m}{2})} \Gamma (m + k, a C) (35) \\ \end{array}
$$
Note that we assume  $s_{\theta}(\mathbf{x}) = \mathbf{0}$  outside of  $U_C$ . By lemma C.3, there exists a large enough constant  $\widetilde{C}$  such that  $\Gamma(m + k, a\widetilde{C}) < \frac{a^{m + k}\Gamma(\frac{m}{2})\varepsilon}{4M_1M_2\pi^{m/2}}$ . Therefore
$$
\int_ {\mathbb {R} ^ {m} \backslash U _ {\tilde {C}}} \| \nabla \log p (\mathbf {x}) - s _ {\theta} (\mathbf {x}) \| ^ {2} p (\mathbf {x}) d \mathbf {x} <   \frac {2 M _ {1} M _ {2} \pi^ {m / 2}}{a ^ {m + k} \Gamma (\frac {m}{2})} \cdot \frac {a ^ {m + k} \Gamma (\frac {m}{2}) \varepsilon}{4 M _ {1} M _ {2} \pi^ {m / 2}} = \frac {\varepsilon}{2} \tag {36}
$$
Step 2 Next we write down the specific form of  $s_{\theta}(\mathbf{x})$ , the score of the stationary distribution of  $\mathbf{x}$  for the reservoir-sampler dynamics (11)
$$
s _ {\theta} (\mathbf {x}) = 2 \left(W _ {\text {o u t}} \sigma \sigma^ {T} W _ {\text {o u t}} ^ {T}\right) ^ {- 1} (- \alpha \mathbf {x} + \alpha W _ {\text {o u t}} \phi (\widetilde {W} _ {\text {r e c}} \mathbf {x} + I)). \tag {37}
$$
Since  $\sigma$  is learnable, we can define  $\sigma = W_{\mathrm{out}}^T (W_{\mathrm{out}}W_{\mathrm{out}}^T)^{-1}$  such that  $W_{\mathrm{out}}\sigma \sigma^T W_{\mathrm{out}}^T$  is an identity matrix 5:
$$
\begin{array}{l} W _ {\text {o u t}} \sigma \sigma^ {T} W _ {\text {o u t}} ^ {T} = W _ {\text {o u t}} W _ {\text {o u t}} ^ {T} \left(W _ {\text {o u t}} W _ {\text {o u t}} ^ {T}\right) ^ {- 1} \left(W _ {\text {o u t}} W _ {\text {o u t}} ^ {T}\right) ^ {- 1} W _ {\text {o u t}} W _ {\text {o u t}} ^ {T} \tag {38} \\ = \mathbf {I} \\ \end{array}
$$
As  $p$  is continuously differentiable, by Theorem C.1, there exists  $W_{\mathrm{out}}$ ,  $\widetilde{W}_{\mathrm{rec}}$  such that
$$
\sup  _ {\mathbf {x} \in U _ {\widetilde {C}}} \left\| W _ {\text {o u t}} \phi (\widetilde {W} _ {\text {r e c}} \mathbf {x} + I) - \left(\mathbf {x} + \frac {\nabla \log p (\mathbf {x})}{2 \alpha}\right) \right\| ^ {2} <   \frac {\varepsilon}{8 \alpha^ {2}} \tag {39}
$$
Therefore with those choices of  $W_{\mathrm{out}}, \widetilde{W}_{\mathrm{rec}}$  and  $\sigma$ , we have
$$
\begin{array}{l} \int_ {U _ {\tilde {C}}} \| \nabla \log p (\mathbf {x}) - s _ {\theta} (\mathbf {x}) \| ^ {2} p (\mathbf {x}) d \mathbf {x} <   \sup _ {\mathbf {x} \in U _ {\tilde {C}}} \| \nabla \log p (\mathbf {x}) - s _ {\theta} (\mathbf {x}) \| ^ {2} \\ = \sup _ {\mathbf {x} \in U _ {\tilde {C}}} \left\| \nabla \log p (\mathbf {x}) - 2 (- \alpha \mathbf {x} + \alpha W _ {\mathrm {o u t}} \phi (\widetilde {W} _ {\mathrm {r e c}} \mathbf {x} + I)) \right\| ^ {2} \\ = 4 \alpha^ {2} \sup  _ {\mathbf {x} \in U _ {\widetilde {C}}} \left\| \frac {\nabla \log p (\mathbf {x})}{2 \alpha} + \mathbf {x} - W _ {\text {o u t}} \phi \left(\widetilde {W} _ {\text {r e c}} \mathbf {x} + I\right) \right\| ^ {2} \\ <   4 \alpha^ {2} \cdot \frac {\varepsilon}{8 \alpha^ {2}} \\ = \frac {\varepsilon}{2} \tag {40} \\ \end{array}
$$
4Equality (31) is the only place where this assumption is used. The equality also holds if  $s_{\theta}(\mathbf{x})$  grows at a similar rate to  $\nabla \log p$ .
A technical subtlety is that  $W_{\mathrm{out}}W_{\mathrm{out}}^{T}$  is not necessarily invertible. However, since every transformation considered here is continuous, we could add a small perturbation to  $W_{\mathrm{out}}$  such that  $W_{\mathrm{out}}W_{\mathrm{out}}^{T}$  is invertible, and our result still holds.
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {x} \sim p (\mathbf {x})} [ \| \nabla \log p (\mathbf {x}) - s _ {\theta} (\mathbf {x}) \| ^ {2} ] = \left(\int_ {\mathbb {R} ^ {m} \backslash U _ {\tilde {C}}} + \int_ {U _ {\tilde {C}}}\right) p (\mathbf {x}) \| \nabla \log p (\mathbf {x}) - s _ {\theta} (\mathbf {x}) \| ^ {2} d \mathbf {x} \\ <   \frac {\varepsilon}{2} + \frac {\varepsilon}{2} = \varepsilon \\ \end{array}
$$
D Pseudocode for training the reservoir-sampler network
Algorithm 1: Training RSN
Input: Training samples  $\{\mathbf{x}_i\}$ , pseudo recurrent weights  $\widetilde{W}_{\mathrm{rec}}$ , output weights  $W_{\mathrm{out}}$ , start noise level  $\sigma_1$ , noise decay factor  $C$ , number of noise level  $N$
Output: trained recurrent weights  $W_{\mathrm{rec}}$ , trained output weights  $W_{\mathrm{out}}$ , trained diffusion coefficient  $\sigma$
for  $i = 1,\dots ,N$  do
while not done do
Sample  $\mathbf{x}$  from the training set
Perturb the samples with Gaussian noise  $\widetilde{\mathbf{x}} = \mathbf{x}_j + \varepsilon, \varepsilon \sim \mathcal{N}(0, \sigma_i^2\mathcal{I})$
Compute the gradient of the score-matching loss
$$
\frac {1}{2} \nabla_ {\widetilde {W} _ {\mathrm {r e c}}, W _ {\mathrm {o u t}}} \left\| 2 \alpha \left(W _ {\mathrm {o u t}} \phi \left(\widetilde {W} _ {\mathrm {r e c}} \mathbf {x} + I\right) - \mathbf {x}\right) - \frac {\mathbf {x} - \widetilde {\mathbf {x}}}{\sigma_ {i} ^ {2}} \right\| ^ {2}
$$
Update  $\tilde{W}_{\mathrm{rec}}$ $W_{\mathrm{out}}$  with gradient descent
$$
\sigma_ {i + 1} = C \sigma_ {i}
$$
$$
W _ {\text {r e c}} \leftarrow \widetilde {W} _ {\text {r e c}} W _ {\text {o u t}}
$$
$$
\sigma \leftarrow W _ {\mathrm {o u t}} ^ {T} (W _ {\mathrm {o u t}} W _ {\mathrm {o u t}} ^ {T}) ^ {- 1}
$$
$$
\text {r e t u r n} \tilde {W} _ {\text {r e c}}, W _ {\text {o u t}}
$$
E Accelerated sampling with reservoir-sampler network
To construct an irreversible dynamics that accelerates sampling, we let the divergence-free (DF) field  $\mathbf{G}$  be of the form  $J\nabla p$ , where  $J = -J^{T}$  can be any skew-symmetric matrix that is either learned through training or prescribed. Then from equation (6) we see that the drift term  $F_{\theta}$  satisfies
$$
F _ {\theta} (\mathbf {v}) = \Sigma \nabla \log p (\mathbf {v}) + J \nabla \log p (\mathbf {v}) = (\Sigma + J) \nabla \log p (\mathbf {v}). \tag {41}
$$
Therefore the corresponding score matching problem is  $\alpha (\Sigma +J)^{-1}(W_{out}\phi (\tilde{W}_{rec}x + I) - x)\approx \nabla \log p(x)$ . With a slight modification, the pseudo code for an accelerated RSN training algorithm is given in algorithm 2. We repeat the experiment in Section 4.1 with the accelerated RSN. The results are shown in Figure 5. We see that the accelerated RSN is able to sample from the distribution faster than the RSN.
F.1 Failure cases for sampler-only networks with hyperbolic tangent nonlinearity
Figure 6 shows that with tanh nonlinearity, sampler-only network is able to sample from the distribution we consider in Section 4.1. However, this is because the score function of the distribution considered can be spanned by just two basis functions equipped by the drift term of the sampler-only network. Therefore, our theory predicts that the sampler-only network will not be able to approximate score functions that are not exactly spanned by these two basis functions. Indeed, if we consider a slightly different Gaussian mixture distribution,  $p_{\mathrm{data}}(x) = \frac{1}{2} (\mathcal{N}(-1,0.0625) + \mathcal{N}(1,0.25))$ , then the sampler-only network will not be able to recover the score function as shown in Figure 7, while RSN still succeeds in recovering the score function.
Input: Training samples  $\{\mathbf{x}_i\}$ , pseudo recurrent weights  $\widetilde{W}_{\mathrm{rec}}$ , output weights  $W_{\mathrm{out}}$ , pseudo diffusion coefficient  $\tilde{\sigma}$ , skew-symmetric matrix  $J$ , start noise level  $\sigma_1$ , noise decay factor  $C$ , number of noise level  $N$
Algorithm 2: Training accelerated RSN
for  $i = 1,\dots ,N$  do while not done do Sample  $\mathbf{x}$  from the training set Perturb the samples with Gaussian noise  $\widetilde{\mathbf{x}} = \mathbf{x}_j + \varepsilon ,\varepsilon \sim \mathcal{N}(0,\sigma_i^2\mathcal{I})$  Compute the gradient of the score-matching loss  $\begin{array}{r}\frac{1}{2}\nabla_{\widetilde{W}_{\mathrm{rec}},W_{\mathrm{out}},\tilde{\sigma},J}\left\| \alpha (\frac{1}{2}\tilde{\sigma}\tilde{\sigma}^T +J)^{-1}(W_{\mathrm{out}}\phi (\widetilde{W}_{\mathrm{rec}}\mathbf{x} + I) - \mathbf{x}) - \frac{\mathbf{x} - \widetilde{\mathbf{x}}}{\sigma_i^2}\right\| ^2 \end{array}$  Update  $\widetilde{W}_{\mathrm{rec}},W_{\mathrm{out}},\tilde{\sigma}$  with gradient descent Update  $J$  with gradient descent (Optional)  $\sigma_{i + 1} = C\sigma_{i}$
$W_{\mathrm{rec}}\gets \widetilde{W}_{\mathrm{rec}}W_{\mathrm{out}}$ $\sigma \leftarrow W_{\mathrm{out}}^{T}(W_{\mathrm{out}}W_{\mathrm{out}}^{T})^{-1}\tilde{\sigma}$
return  $W_{\mathrm{rec}}$ ,  $W_{\mathrm{out}}$
$x_{1}$
Figure 5: Sampling speed of accelerated RSN We repeat the experiment described in Section 4.1 where we try to sample from a 2-D Laplacian mixture distribution. The figure shows the Wasserstein distance between the marginalized distributions of generated samples and true samples for both dimensions. We see that compared to vanilla RSN, accelerated RSN is able to sample from the stationary distribution with faster relaxation time.
$x_{2}$
Figure 6: Sampler-only networks with hyperbolic tangent nonlinearity are able to sample from a 1-D Gaussian mixture distribution with a symmetric score function.. See Figure 2 for an explanation of the abbreviations. Here we use tanh nonlinearity for sampler-only networks (SO-SC and SO-FR). All 3 types of networks are able to match the score function and sample from the distribution.
Figure 7: Sampler-only network with hyperbolic tangent nonlinearity fails to sample from a 1-D Gaussian mixture distribution with a non-symmetric score function. See Figure 2 for an explanation of the abbreviations. Here we use tanh nonlinearity for sampler-only networks (SO-SC and SO-FR). Only RSN is able to match the score function and sample from the distribution.
F.2 Sampling using autoencoder
We show that we can replace the fixed linear PCA projection used in Section 4.2 with pretrained nonlinear autoencoder. We test the algorithm on both the MNIST dataset (Figure 8) and the CIFAR-10 dataset (Figure 9).
Figure 8: Samples produced by our reservoir sampler network when trained to fit the latent distribution produced by a non-linear convolution autoencoder from MNIST. The distribution was 32-dimensional. The reservoir sampler network had 500 reservoir neurons.
G Related works
In this section, we examine the neural dynamics considered in [17, 1, 38, 19] and discuss how our work is related.
Dong et al. [17] uses Hamiltonian Langevin dynamics to accelerate sampling in continuous attractor networks. The Hamiltonian dynamics below are considered in the paper (cf. eq (19) and (20)):
$$
\left\{ \begin{array}{l l} \tau_ {s} \frac {d \mathbf {s}}{d t} & = \boldsymbol {\alpha} \mathbf {y} \\ \tau_ {z} \frac {d \mathbf {y}}{d t} & = - \beta \boldsymbol {\alpha} ^ {- 1} \mathbf {y} + \Lambda (\mathbf {s} ^ {\mathbf {o}} - \mathbf {s}) + \sqrt {\tau_ {z}} \boldsymbol {\sigma} _ {\mathbf {y}} \boldsymbol {\xi} _ {t}. \end{array} \right. \tag {42}
$$
The overdamped Langevin dynamics that samples from the same marginal stationary distribution over  $s$  as (42):
$$
\frac {d \mathbf {s}}{d t} = \Lambda \left(\mathbf {s} ^ {\mathbf {o}} - \mathbf {s}\right) + \sqrt {2} \boldsymbol {\xi} _ {t}, \tag {43}
$$
where  $\mathbf{s}^{\mathrm{o}}$  is an observation generated by a latent feature  $\mathbf{s}$  according to a Gaussian distribution, and the linear term  $\Lambda (\mathbf{s}^{\mathrm{o}} - \mathbf{s})$  comes from the external input. So the network dynamics in Dong et al. [17] themselves do not generate samples without the external input, and the study does not clarify how to supply the network with nonlinear score functions.
CIFAR 10 Images
Reconstructed Images
Samples
Figure 9: Samples produced by our reservoir sampler network when trained to fit the latent distribution produced by a non-linear convolution autoencoder from CIFAR-10. We show both the original images and the reconstructed images to evaluate the sampling quality. The distribution was 128-dimensional. The reservoir sampler network had 20000 reservoir neurons.
Aitchison and Lengyel [1] have shown that Hamiltonian Monte Carlo (HMC) naturally maps onto the dynamics of excitatory-inhibitory neural networks. However, the neural dynamics considered are entirely linear in terms of the network variables (cf. eq (23) and (24)). Therefore, the dynamics are again equivalent to the synaptic current dynamics with an identity transfer function in Section 3.2. The same applies to the analysis in Masset et al. [38], which also only considers linear neural dynamics. By Proposition 2 and the decomposition discussed in Kwon et al. [31], these dynamics can only sample from probability distributions whose score functions are linear, i.e. Gaussians.
The work by Echeveste et al. [19] provides a unifying model for several dynamical phenomena in sensory cortices by training a nonlinear recurrent excitatory-inhibitory neural circuit model to perform sampling-based probabilistic inference. The synaptic neural dynamics below (cf. equation (8)) is used, where the nonlinear transfer function is supralinear, i.e.  $\phi (u) = k\lceil u\rceil^n$
$$
\tau_ {i} \frac {d u _ {i}}{d t} = - u _ {i} + \sum_ {j = 1} ^ {N} w _ {i j} \phi \left(u _ {j}\right) + h _ {i} (t) + \eta_ {i} (t). \tag {44}
$$
As discussed in Section 3.2, while they are richer than linear dynamics, dynamics of this form still have limited ability to sample from complex probability distributions. Moreover, we note that the training criterion considered in Echeveste et al. [19] only matches the mean and the variance of the stationary distribution. These points are in contrast to our work here, where we show that the reservoir-sampler network, which extends the dynamics (44), can be trained via alternate methods to sample from complex probability distributions by matching their score functions.
H Details of numerical experiments
All experiments were run on one NVIDIA Quadro RTX 6000 GPU. We always use the model saved at the last training iteration.
H.1 Training details for Section 4.1
For the 1-D bimodal Gaussian distribution experiment (Figure 2), we used 1000 reservoir neurons (only in the reservoir sampler network) and 1 sampler neuron. We trained all 3 networks (SO-FR, SO-SC, RS-FR) for 79000 iterations with the Adam optimizer. The learning rate was 0.0001, and the batch size was 128. We started by adding Gaussian noise with standard deviation (std)  $\sigma_{1} = 1$  to the training samples. We decreased the std every 7900 iterations by a constant factor, i.e.  $\sigma_{i+1} / \sigma_{i} = C$ , such that  $\sigma_{10} = .01$ . When simulating the stochastic neural dynamics, we used a step size of  $10^{-4}$ , and ran 10000 steps.
For the mixture of 2-D Laplace distributions experiment (Figure 3), we used 1000 reservoir neurons (only in the reservoir sampler network) and 2 sampler neurons. We trained the RS-FR network for 62,500 iterations with the Adam optimizer. The learning rate was 0.0001, and the batch size was 128. We started by adding Gaussian noise with std  $\sigma_{1} = 1$  to the training samples. We decreased the std every 6250 iterations by a constant factor, i.e.  $\sigma_{i + 1} / \sigma_i = C$ , such that  $\sigma_{10} = .01$ . When simulating the stochastic neural dynamics, we used a step size of  $10^{-4}$ , and ran 20000 steps.
H.2 Training details for Section 4.2
For the MNIST experiment (Figure 2), we used 20000 reservoir neurons (only in the reservoir sampler network) and 300 sampler neurons. We trained all 3 networks (SO-FR, SO-SC, RS-FR) for 1000 epochs with the Adam optimizer. The learning rate was 0.001, and the batch size was 64. We started by adding Gaussian noise with std  $\sigma_{1} = 1$  to the training samples. We decreased the std every 100 epochs by a constant factor, i.e.  $\sigma_{i + 1} / \sigma_i = C$ , such that  $\sigma_{10} = .01$ . When simulating the stochastic neural dynamics, we used a step size of  $10^{-6}$ , and ran 30000 steps.
I Statement of code availability
All code is available on Github