SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\tau_ {v} \frac {\mathrm {d} \mathbf {v}}{\mathrm {d} t} = - \frac {m \mathbf {v}}{2} + m \sqrt {2 \tau_ {v}} \xi , \tag {15}
$$
where  $m > 0$  controls the adaptation strength. The auxiliary variable  $\mathbf{v}$  can keep the network moving while reaching the local minima, which essentially serves as the momentum term (Fig.3B). When there is no adaptation  $(m = 0)$ , the above dynamic will degenerate to the Langevin sampling (LS) described in Eq.(3). And the stationary distribution of the above dynamic remains to be  $p_{\theta}(\mathbf{z}|\mathbf{x})$  (See SI for proof). By substituting the joint distribution of exponential family Eq.(7) into the above dynamic, we can obtain the SLD inference dynamic,
$$
\tau_ {z} \frac {\mathrm {d} \mathbf {x} _ {l}}{\mathrm {d} t} = f ^ {\prime} (\mathbf {x} _ {l}) \theta_ {l - 1} ^ {T} \varepsilon_ {l - 1} + \phi^ {\prime} (\mathbf {x} _ {l}) \boldsymbol {\eta} _ {l} + g ^ {\prime} (\mathbf {x} _ {l}) - \mathbf {v} _ {l} + \sqrt {2 \tau_ {z}} \xi_ {l}, \tag {16}
$$
$$
\tau_ {v} \frac {\mathrm {d} \mathbf {v} _ {l}}{\mathrm {d} t} = - \frac {m \mathbf {v} _ {l}}{2} + m \sqrt {2 \tau_ {v}} \xi_ {l}. \tag {17}
$$
Here we exemplify the neural adaptation with spike frequency adaptation (SFA) [38]. In the neural system, the adaptation current  $\mathbf{v}_l$  accumulates the noise term  $\xi_l$  coming from the ion concentrations, release of neural transmitters, activation/inactivation of ion channels and so on, which is described by Eq.(17). The adaptation current induces suppression on neurons  $\mathbf{x}_l$ , acting as momentum variables to accelerate the sampling process (Eq.(16)).
We conduct further investigation to elucidate the precise mechanism by which noisy adaptation facilitates the acceleration of the sampling process in the HEE model. Considering that the energy function  $-\ln p_{\theta}(\mathbf{x}_{1:L}|\mathbf{x})$  is non-convex, the sampling process can be divided into two parts. In the first part, the network needs to find a local minima and samples near the local minima, which takes a certain amount of time called recurrence time  $\mathcal{T}_{rec}$  (Fig.3C). In the second part, the network needs to leave the local minima and find a new one, which takes a certain amount of time called the escape time  $\mathcal{T}_{esc}$ . Typically, we have  $\mathcal{T}_{rec} \ll \mathcal{T}_{esc}$ . The total time to get stationary distribution can be approximated by  $\mathcal{T} = \mathcal{T}_{esc} + \mathcal{T}_{rec}$ . It can be proved that [39] the recurrence time is bounded by  $\mathcal{T}_{rec} = \mathcal{O}(1 / \lambda_1(H_J))$ .  $\lambda_1(H_J)$  is the smallest eigenvalue of the matrix  $H_J$ ,
$$
H _ {J} = \left( \begin{array}{c c} H / \tau_ {z} & I / \tau_ {z} \\ 0 & m I / \left(2 \tau_ {v}\right) \end{array} \right) \tag {18}
$$
where  $H$  is the Hessian matrix of the energy function  $-\ln p_{\theta}(\mathbf{x}_{1:L}|\mathbf{x}_0)$ . The smallest eigenvalue of  $H_J$  is calculated as  $\lambda_1(H_J) = \min \{\lambda_1(H) / \tau_z, m / (2\tau_v)\}$ . Thus, in the case  $m > 2\lambda_1(H)\tau_v / \tau_z$ , the SLD can reduce the recurrence time  $\mathcal{T}_{rec}$  to accelerate the sampling process. And the escape
Figure 4: Evaluation on 2D synthetic datasets and FashionMNIST: a mixture of four Gaussian distribution (first line), a mixture of four banana-shaped distribution (second line), pinwheel-shaped distribution (third line).
time is bounded by  $\mathcal{T}_{\text{esc}} = \mathcal{O}\left(\sqrt{1 / \operatorname*{det}(H_J)}\right)$ . The determinant of  $H_{J}$  is calculated as  $\operatorname*{det}(H_J) = \operatorname*{det}(H)(m\tau /2\tau_v)^{\sum n_l}$ . Thus,  $\operatorname*{det}(H_J)$  is monotonically increasing with  $m$ , indicating that the larger  $m$  is, the shorter the time it takes to escape from the local minima.
The analysis presented above demonstrates that the convergence speed of the inference dynamic is determined by the values of  $\lambda_1(H)$  and  $\operatorname{det}(H)$ . This valuable insight can be leveraged to guide the design of the network architecture, enabling the creation of more efficient and effective models. Specifically, with a fixed total number of neurons  $\sum_{l} n_{l}$ , increasing the number of layers  $L$  results in a deeper network, while decreasing the number of layers results in a wider network. Practically, we show that  $\operatorname{det}(H)$  will decrease with layers  $L$  while  $\lambda_1$  will increase (Fig.3D), which indicates that there is a trade-off between the recurrence time  $\mathcal{T}_{rec}$  and the escaping time  $\mathcal{T}_{esc}$  with different layers  $L$  (See SI for detailed setting and analysis).
In this section, we firstly validate the capability of HEE model for approaching complex distribution by examining the quality of generation. Then, we show that our model demonstrates similarity in the representation of natural images to the biological visual system. And adaptation can induce oscillatory behavior and transient overshoots in neurons during the inference phase.
5.1 Generation
Firstly, we conducted experiments using three variations of the HEE model, each with different  $\phi(x)$  functions and sampling methods (Tabel 1), to evaluate their capabilities. The experiments were performed on both 2D synthetic datasets and the FashionMINST. We use the fully connected architecture, i.e.,  $\theta_l$  has no zero elements. The results (second and third column) show that HEE with linear statistic (HEE-L) struggles to capture the complex distribution. Moreover, we theoretically prove that HEE-L can only approach unimodal distributions (See SI for detailed proof). For HEE-NL, some modes are missing while using the joint generation. And when the spacing between modes is large, there is an issue of non-uniformity among different modes. And the marginal generation converges much faster than the joint generation. In FashionMNIST, it takes less time for marginal method to get the generation of high-quality images.
Then, we employ the HEE-NL-A with layers  $L = 10$  on the CIFAR10 unconditional. The sparse connection is employed as a method to mimic the receptive field behavior found in biological systems. We quantitatively evaluate image quality of HEE-NL-A with Inception score [40] and FID score [41] in
Model	φ(x)	Sampling method
HEE-L	x	LS
HEE-NL	sigmoid(x)	LS
HEE-NL-A	sigmoid(x)	SLD
Tabel 2. Overall, we achieve a performance comparable with the previous EBMs. And the generation quality of the marginal method is better than joint method, which agrees with the previous results [42].
We further use the HEE-NL-A trained on CIFAR10 to explore the relationship between the latent features and semantic information, including orientation, color and category.
Orientation: Simple cells [43] and complex cells [44] are the most prominent and widely observed neurons in the biological visual system that exhibit tuning to orientation. They are found in the primary visual cortex (V1) of numerous animal species [45, 46]. We present the model with gabor images of different orientations commonly used in experiments (Fig.6A) and compute the mean and variance of the neural responses for each neuron. Then, we use a Gaussian curve with bandwidth limited from  $20^{\circ}$  to  $90^{\circ}$  and a two-modes Gaussian curve to fit the simple cell and complex cell, respectively (Fig.6A&B). We find that the proportion of simple cells and complex cells remains relatively consistent across each layer and both decrease with increasing layers in our model (Fig.6C).
End stopping: In the HEE model, interneurons essentially represent the error term in the PCNs. We have observed the phenomenon of 'end stopping' in interneurons (Fig.6D), which aligns with the end-stopping behavior observed in error neurons in the PCNs [21].
Color: Recent study shows that there is a hierarchical representation for chromatic processing across the ventral pathway of macaque [47]. We present our model using reshaped natural images [48] and employ principal component analysis to demonstrate that the middle layer's neural representation's most informative dimensions carry chromatic information (Fig.6E).
Category: Visual object recognition is believed to be solved by the brain hierarchically [49]. A recent study [50] demonstrate that the inferotemporal cortex, situated in the deeper layer of the visual pathway, is capable of constructing a linear map of the object space. For each layer in our model, we employed a linear support vector machine (SVM) to classify the ten labels of the CIFAR10. The SVM was trained using the average neural responses as features. Fig.6F illustrates the projection of the features in the last layer onto the SVM weights corresponding to the "cat" and "dog" labels. Furthermore, we observe that the classification accuracy improves as we move up the layers of our model (Fig.6G), which is consistent with findings in both biological visual systems [50] and the artificial neural networks [51].
Phenomena: Oscillations [52] and transients [53] are two kinds of spatial-temporal dynamic features in neural systems, which play a crucial role during the sampling process [13]. Here, we show that by adjusting the adaptation strength  $m$ , the oscillation frequency of the HEE model can span within the
Table 1: Table of different HEE models.
Model	IS	FID
HEE-NL-A (Joint)	5.95	43.21
EBM (single) [27]	6.02	40.58
HEE-NL-A (Marginal)	6.47	37.05
MEG (Generator) [42]	6.49	35.02
EBM (10 ensemble)	6.78	38.20
MEG (MCMC)	7.31	33.18
Table 2: Table of Inception and FID scores.
Figure 5: Marginal generation of HEE-NL-A on CIFAR10.
Figure 6: (A)(B) The red dots show the average firing rate of two neurons in  $\mathbf{x}_1$  with different gabor-like stimulus. Different tuning curves are used to fit simple cells and complex cells. (C) The proportion of simple cells and complex cells in each layer. (D) We show horizontal bars of varying lengths to the HEE. The red dots show the average firing rate of a neuron in  $\epsilon_1$  whose corresponding neuron in  $\mathbf{x}_1$  is a simple cell preferring 0 degree. (E) The images are plotted at the location corresponding to the projection of their average neural response in layer 5 onto the first two principal components. Red images are located in the upper left corner, while blue-green images are located in the lower right corner. (F) The true label of cat, dog, and other categories should be respectively located in the second, fourth, and third quadrants. (G) The classification accuracy of the SVM in each layer. (H) We sampled and statistically analyzed the distribution of the highest firing rate frequencies of all neurons during the inference phase in the first  $100\tau_z$  for different values of  $m$ . The gamma band is centered around the dashed line. (I) We sampled and statistically assessed the maximum change in firing rates of all neurons during the inference phase in the first  $100\tau_z$  for different values of  $m$ . We refer to the mean of the maximum change values as the 'average step size'. We utilize the average step size of the neurons during the sampling process as an indicator of transients.
range of  $20 - 80\mathrm{Hz}$  (gamma band), which is widely observed in visual systems [54] (Fig.6H). And stimulus-onset transients of the firing rate can also be enhanced by the adaptation (Fig.6I).
The present study investigates the sampling-based inference and learning dynamic within the framework of an intrinsic generative model. We introduce the HEE model as a neural implementation that utilizes neural dynamics and Hebbian learning. Additionally, we demonstrate that the inclusion of
neural adaptation can significantly accelerate the sampling process and give rise to various dynamic phenomena throughout the network. In this section, we will discuss several related theories and models.
Probabilistic Population Code (PPC) [55, 56] is another theory that explains how the brain performs Bayesian inference, in which neural responses are interpreted as the parameters of the probability distributions. We adopted the idea [57] that PPC theory incorporates two generative models. In the framework of PPC, the experimenter presents the subject with observation  $\mathbf{x}$  (gabor image) based on the semantic information  $\mathbf{s}$  (orientation), which actually defines an external generative model from  $\mathbf{s}$  to  $\mathbf{x}$ . And the subjects hold an intrinsic generative model with latent variable  $\mathbf{z}$  represented by neural response to interpret the observation  $\mathbf{x}$ . PPC theory integrates two generative models into a single generative model, in which the neural response  $\mathbf{z}$  is regarded as the observation generated from the semantic information  $\mathbf{s}$ . We propose that the learning dynamic occurs exclusively within the intrinsic generative model, as the brain is not aware of the external generative model.
Energy-based models (EBMs) [20, 58] When EBMs were initially proposed [20], they had latent variables corresponding to neurons. Later, to enhance the model's expressive power, hierarchical structures were introduced [59, 60]. Such EBMs could typically ensure local learning in space. As artificial neural networks have become increasingly powerful, it has been observed that for generative tasks, there's no need to explicitly introduce neurons as latent variables within EBMs. Instead, one can directly employ a neural network to represent the energy [27]. Training such EBMs often involves utilizing BP. The distinction between these EBMs and traditional EBMs is akin to the difference between dynamic systems and recurrent neural networks.
Regardless of whether it's the traditional EBMs or the new type of EBMs, both involve the challenge of estimating the partition function. This difficulty arises from the fact that as the depth of the energy function increases, the total sample space required multiplies the space for each layer. In the case of HEE, we allocate the partition function across each layer. As a result, the total sample space required is the sum of the spaces for each layer. This significantly reduces the required sample space.
Predictive coding networks (PCNs) [21, 22] The interneurons in the HEE model serve a similar role to the prediction error in PCNs. And our theoretical analysis shows that the predictions in PCNs essentially represent the decomposed log-partition function. And PCNs don't stress the sampling-based inference, which requires them to approximate the energy function using variation inference by delta function. Sampling-based inference can assist the network in exploring the posterior probability space, leading to a more accurate estimation of the energy function. Additionally, it can account for the observed neural variability in experiments.
Diffusion models (DDPMs) [61, 62] The HEE model and DDPMs share the same joint distribution and both exhibit a hierarchical Markov structure, which may contribute to the HEE model's strong expressive potential. The marginal generation is also called latent space MCMC [63, 42], which is similar to the generation process of DDPMs. While DDPMs unfolds the Markov chain over time, the HEE model unfolds it between layers of neurons. However, in order to reach a better performance, DDPMs use a fixed diffusion process as the inference dynamic, which may not be adopted by our brain since the latent variables in our brain carry semantic information (such as simple cells [43]).
Speed up sampling [17, 38] In previous work, inhibition neurons were used to serve as momentum terms to accelerate sampling [17]. However, this approach required a one-to-one correspondence between inhibition neurons and excitatory neurons. In our approach, we consider the adaptive properties inherent in each neuron itself to serve as momentum, naturally resolving the one-to-one correspondence issue. Furthermore, our consideration extends to sampling in a non-convex energy space, which differs from the prior focus solely on convex space convergence properties [38].
I'd like to express my gratitude to Tianqiu Zhang and Chaoming Wang for their assistance in configuring the experimental environment. I also want to thank BrainPy [64] for their support throughout this work. Special thanks to Yumeng Cao for reviewing the article's grammar and expression. This work was supported by Science and Technology Innovation 2030-Brain Science and Brain-inspired Intelligence Project (No. 2021ZD0200204).
[63] Yoshua Bengio, Grégoire Mesnil, Yann Dauphin, and Salah Rifai. Better mixing via deep representations. In International conference on machine learning, pages 552-560. PMLR, 2013.
[64] Chaoming Wang, Xiaoyu Chen, Tianqiu Zhang, and Si Wu. Brainpy: a flexible, integrative, efficient, and extensible framework towards general-purpose brain dynamics programming. bioRxiv, pages 2022–10, 2022.