SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Gehua Ma
College of Computer Science and Technology
gehuama@icloud.com
Runhao Jiang
RhJiang@zju.edu.cn
Rui Yan
Zhejiang University of Technology
ryan@zjut.edu.cn
Huajin Tang*
Developing computational models of neural response is crucial for understanding sensory processing and neural computations. Current state-of-the-art neural network methods use temporal filters to handle temporal dependencies, resulting in an unrealistic and inflexible processing paradigm. Meanwhile, these methods target trial-averaged firing rates and fail to capture important features in spike trains. This work presents the temporal conditioning spiking latent variable models (TeCoS-LVM) to simulate the neural response to natural visual stimuli. We use spiking neurons to produce spike outputs that directly match the recorded trains. This approach helps to avoid losing information embedded in the original spike trains. We exclude the temporal dimension from the model parameter space and introduce a temporal conditioning operation to allow the model to adaptively explore and exploit temporal dependencies in stimuli sequences in a natural paradigm. We show that TeCoS-LVM models can produce more realistic spike activities and accurately fit spike statistics than powerful alternatives. Additionally, learned TeCoS-LVM models can generalize well to longer time scales. Overall, while remaining computationally tractable, our model effectively captures key features of neural coding systems. It thus provides a useful tool for building accurate predictive computational accounts for various sensory perception circuits.
Building precise computational models of neural response to natural visual stimuli is a fundamental scientific problem in sensory neuroscience. These models can offer insights into neural circuit computations, reveal new mechanisms, and validate theoretical predictions [1, 2, 3, 4, 5, 6]. However, constructing such models is challenging due to the complex nonlinear processes involved in neural coding, such as synaptic transmission and spiking dynamics. For modeling retinal responses, early attempts using linear-nonlinear (LN) models and generalized linear models (GLMs) were successful with simple data such as white noise [2, 7] but fell short with more complex stimuli like natural visual scenes [8, 9]. Artificial neural networks (ANNs), which are powerful function approximators and loosely resemble biological neurons and architectures [10, 11], have shown promise in modeling
*Correspondence: htang@zju.edu.cn
the visual stimuli coding process through various ANN-based methods [9, 12, 13, 14, 15]. Recent research has demonstrated that complex neural activity can be well represented in a low-dimensional space [16, 17]. This has led to growing interest in a type of neural network known as latent variable models (LVMs). LVMs strike a balance between model accuracy and representation space simplicity, enabling accurate neural coding modeling and interpretation of neural activity in a low-dimensional space [18, 19, 20, 21, 22, 23, 24]. Although the current state-of-the-art neural network models for visual neural coding have produced decent results and provided some exciting insights, they have two major limitations:
- Most of these works focus on simulating the firing rates of real neurons. This is a decent choice (following the classic Poisson LN/GLM models) for artificial neuron networks, which are essentially real-valued processing-based. However, as a trial-averaged spike statistic, firing rates only characterize some aspects of the original spike train [25]. As a result, directly using firing rates as the target may result in losing information embedded in the original spike trains [26, 27].
- Existing models mostly employ fixed-length temporal filters. For example, the CNN approach [9] concatenates stimuli within a fixed duration as input and processes these inputs using temporal filters. Therefore, they cannot process long stimuli sequences like a real neural circuit. Instead, they must slice the long sequences into fixed-length short segments and process them separately, thus losing biological realism (see also Fig. 2A). Secondly, in simulation, the learned models can only take inputs of the same length as during the training phase, thus limiting their flexibility.
Although these two points are important for a realistic computational model, an approach that addresses these limitations is still lacking. This work introduces the TeCoS-LVM (temporal conditioning spiking LVM) models of the neural responses to natural visual stimuli. We employ spiking neurons to allow the model to aim directly at producing realistic spike trains. This avoids spike train information loss that might occur when targeting spike statistics. To address the second limitation, we completely exclude the temporal dimension from the parameter space and introduce a temporal conditioning operation to handle the temporal dependencies. Inspired by the information compression in biological coding systems, we formalize TeCoS-LVM models within the information bottleneck framework of LVMs and introduce a general learning method for them. Evaluations on real neural recordings demonstrate that TeCoS-LVM models accurately fit spike statistics and produce realistic spike activities. Further simulations show that TeCoS-LVM models learned on short sequences can generalize well to longer time scales and exhibit memory mechanisms similar to those in biological cognitive circuits.
Leaky integrate-and-fire spiking neuron We adopt the Leaky Integrate-and-Fire (LIF) neuron model in this work, which briefly describes the sub-threshold membrane potential dynamics as  $\tau_{m}\frac{\mathrm{d}u}{\mathrm{d}t} = -(u - u_{\mathrm{reset}}) + RI(t)$ , where  $u_{t}$  denotes membrane potential,  $R$ ,  $\tau_{m}$  are membrane resistance, time constant, and  $I$  is the input current.  $\nu_{\mathrm{th}}$ ,  $u_{\mathrm{reset}}$  denote the firing threshold, resting potential, respectively. In practice, it leads to the following discrete-time computational form [28, 29, 30]. The sub-threshold dynamic, firing, and resetting are written as  $u_{t} = \tau u_{t-1} + I_{t}$ ;  $o_{t}$  = Heaviside  $(u_{t} - \nu_{\mathrm{th}})$ ;  $u_{t} = u_{\mathrm{reset}}$  if  $o_{t} = 1$ , where  $o_{t}$  is the spike output,  $I_{t}$  is the input current, usually transformed by a parameterized mapping, and  $\tau$  is the membrane time constant. In this discrete form, the membrane constant  $\tau_{m}$  is combined with timestep dt for simplification, as in all our experiments, the length of simulation timestep dt is fixed. To introduce a simple model of neuronal spiking and refractoriness, we assume  $\nu_{\mathrm{th}} = 1$ ,  $\tau = 0.5$  is a fixed constant, and  $u_{\mathrm{reset}} = 0$  for all spiking neurons throughout this research.
Variational information bottleneck The information bottleneck (IB) principle [31] offers an appealing framework to formulate LVMs. In short, a desired LVM should have latent representations that are maximally expressive regarding its target while being maximally compressive about its input. Let us consider a latent variable model  $\theta$  with input  $\mathbf{x}$ , target  $\mathbf{y}$ , and latent representation  $\mathbf{z}$  defined by a parametric encoder  $q(\mathbf{z}|\mathbf{x};\theta)$ . Let  $I(\mathbf{z},\mathbf{y};\theta)$  be the mutual information between the  $\mathbf{z}$  and  $\mathbf{y}$ , and  $I(\mathbf{z},\mathbf{x};\theta)$  be the mutual information between  $\mathbf{z}$  and  $\mathbf{x}$ , IB suggests an objective
$$
\max  _ {\theta} I (\mathbf {z}, \mathbf {y}; \theta) \text {s . t .} I (\mathbf {z}, \mathbf {x}; \theta) <   I _ {c}, \tag {1}
$$
where  $I_{c}$  is the information constraint. With the introduction of a Lagrange multiplier  $\beta$  [31], the IB objective is equivalent to
$$
\max  _ {\theta} [ I (\mathbf {z}, \mathbf {y}; \theta) - \beta I (\mathbf {z}, \mathbf {x}; \theta) ]. \tag {2}
$$
Figure 1: Overview of our approach. A. Graphical illustration of TeCoS-LVM models of retinal neural response to natural visual scenes. Our model can directly generate neural response sequences in real-time, thus faithfully simulating the real neural computation process (see also Fig. 2A). B. Full graphical illustration of the computational operations. C. Separate illustrations of the prior, encoder inference, decoder generation, and hidden state update operations. With the introduction of the hidden state (acts as the sensory memory), our prior, encoder and decoder are linked to the entire stimuli sequence  $\mathbf{x}_{1:t}$  rather than just the current stimulus  $\mathbf{x}_t$  (see also Appendix 3.1). Hence, although we completely exclude the time dimension from the model parameter space, TeCoS-LVM can still adaptively explore and exploit temporal information for predictive (generative) modeling.
Deep variational IB [32, 33] leverages variational inference to construct a lower bound on the IB objective in (2). By assuming the factorization  $p(\mathbf{x}, \mathbf{y}, \mathbf{z}) = p(\mathbf{z}|\mathbf{x})p(\mathbf{y}|\mathbf{x})p(\mathbf{x})$ , we have an equivalent objective that comprises one predictive term and one compressive term (refer to Appendix D) as follows,
$$
\min  _ {\boldsymbol {\theta}} \underbrace {\mathbb {E} _ {q (\mathbf {z} | \mathbf {x} ; \boldsymbol {\theta})} [ - \log p (\mathbf {y} | \mathbf {z} ; \boldsymbol {\theta}) ]} _ {\mathcal {L} ^ {\text {p r e d :}} \text {e n c o u r a g i n g p r e d i c t i v e p o w e r}} + \beta \cdot \underbrace {\mathrm {K L} [ q (\mathbf {z} | \mathbf {x} ; \boldsymbol {\theta}) \| p (\mathbf {z}) ]} _ {\mathcal {L} ^ {\text {c o m p :}} \text {e n c o u r a g i n g c o m p r e s s i o n}}, \tag {3}
$$
where  $\mathrm{KL}[Q\| P]$  is the Kullback-Leibler divergence between two distributions, and  $p(\mathbf{y}|\mathbf{z};\theta)$  is a parameterized decoder.
3.1 Temporal Conditioning Spiking Latent Variable Model
Basic formulation (see also Appendix D) We denote a sequence of visual stimuli as  $\mathbf{x} = (\mathbf{x}_1, \dots, \mathbf{x}_t, \dots, \mathbf{x}_T) \in \mathbb{R}^{T \times \dim[\mathbf{x}_t]}$ ,  $\dim[\mathbf{x}_t]$  stands for the dimension of  $\mathbf{x}_t$ . At each timestep  $t$ , one high-dimensional visual stimulus  $\mathbf{x}_t$  is received. We want to predict the corresponding neural population response  $\mathbf{y}_t \in \{0, 1\}^{\dim[\mathbf{y}_t]}$ , where  $\dim[\mathbf{y}_t]$  denotes the number of retinal ganglion cells (RGCs). This is implemented by an LVM which first compresses the visual stimuli into a low-dimensional latent representation  $\mathbf{z}_t \in \mathbb{R}^{\dim[\mathbf{z}_t]}$ , and then decodes the neural population response from it. Biological neural coding can effectively compress observed stimuli while retaining the informative contents [34, 35, 36, 37]. Therefore, we further encourage this LVM to construct a latent space in which  $\mathbf{z}$  have maximal predictive power regarding response  $\mathbf{y}$  while being maximally compressive about input stimuli  $\mathbf{x}$  [38]. As a result, our target to model the neural coding process of visual stimuli turns to an optimization problem of minimizing the loss function (3) presented in the variational IB framework.
However, assuming independence along the temporal dimension becomes inappropriate due to the complex temporal dependencies in stimuli-neural response modeling [39, 40, 41]. To address this, we introduce a hidden state into the prior, encoder, and decoder of the latent variable model. This hidden state maintains earlier stimuli information [42, 43], allowing the entire inference-generation
process to be conditioned on the entire stimuli sequence  $\mathbf{x}_{1:t}$  rather than just  $\mathbf{x}_t$  (Fig. 5). To enable adaptive exploitation and accumulation of temporal dependencies within the stimuli sequence, we employ a recurrent network to update the hidden state [44, 45, 46, 47], formally,
$$
\mathbf {h} _ {t} = f _ {\mathrm {R N N}} \left(\mathbf {x} _ {t}, \mathbf {h} _ {t - 1}\right). \tag {4}
$$
The hidden state here serves as the sensory memory [48], akin to the Tolman-Eichenbaum Machine [49], which employs an attractor network to store and retrieve memories.
Note that, as we focus on high-dimensional visual stimuli here, we use a spiking convolutional feature extractor to reduce the dimensionality of the stimuli. It is shared in the hidden state update and the temporal conditioning encoder inference process (as illustrated in Fig. 1). To make our notation simpler and easier to understand during reading, we do not explicitly denote the feature extractor in our formulations (stimuli features for de facto computation, but still marked as  $\mathbf{x}$ ).
Temporal conditioning prior With the introduction of the hidden state, the prior over latent variable is no longer a standard isotropic Gaussian. Instead, it becomes a parameterized conditional distribution that depends on the hidden state, which carries information about previous stimuli. Formally, the latent variable follows the distribution  $p(\mathbf{z}_t|\mathbf{x}_{1:t - 1})$ , given by
$$
p \left(\mathbf {z} _ {t} \mid \mathbf {x} _ {1: t - 1}\right) = \mathcal {N} \left(\mathbf {z} _ {t}; \underbrace {\phi^ {\text {p r i o r}} \left(\mathbf {h} _ {t - 1}\right)} _ {\text {M e a n} \mu}, \operatorname {d i a g} \left(\underbrace {\phi^ {\text {p r i o r}} \left(\mathbf {h} _ {t - 1}\right)} _ {\text {V a r i a n c e} \sigma^ {2}}\right)\right), \tag {5}
$$
where  $\phi^{\mathrm{prior}}$  denotes the parametric model to compute the means and variances, a spiking MLP. In particular, the real-valued means and variations are calculated through the linear readout synapses of spiking neurons.
Temporal conditioning encoder In a similar manner, the encoder is not only a function of  $\mathbf{x}_t$ , but also a function of  $\mathbf{h}_{t - 1}$ . By Eq. 4, the hidden state  $\mathbf{h}_{t - 1}$  is a function of  $\mathbf{x}_{1:t - 1}$ . Hence, the temporal conditioning encoder defines the distribution  $q(\mathbf{z}_t|\mathbf{x}_{1:t})$ . We can write
$$
q \left(\mathbf {z} _ {t} \mid \mathbf {x} _ {1: t}\right) = \mathcal {N} \left(\mathbf {z} _ {t}; \psi^ {\text {e n c}} \left(\mathbf {x} _ {t}, \mathbf {h} _ {t - 1}\right), \operatorname {d i a g} \left(\psi^ {\text {e n c}} \left(\mathbf {x} _ {t}, \mathbf {h} _ {t - 1}\right)\right)\right), \tag {6}
$$
where  $\psi^{\mathrm{enc}}$  is the parameterized model for computing the variational posterior distribution, which is also a spiking MLP.
Temporal conditioning decoder The decoder generates the neural population responses  $\mathbf{y}_t$  only using the latent representation  $\mathbf{z}_t$  and hidden state  $\mathbf{h}_{t - 1}$  as inputs. As the hidden state is a function of previous stimuli, the decoder defines the distribution  $p(\mathbf{y}_t|\mathbf{z}_t,\mathbf{x}_{1:t - 1};\psi^{\mathrm{dec}})$ , where  $\psi^{\mathrm{dec}}$  stands for the decoder parameters. Since we use spiking neurons, the decoder will directly output spike trains that simulate the recorded neural population responses.
Model learning The TeCoS-LVM models are optimized to minimize a loss function (Eq. 3, see also Appendix D) that has the form:
$$
\mathcal {L} = \mathcal {L} ^ {\text {p r e d}} + \beta \mathcal {L} ^ {\text {c o m p}}, \tag {7}
$$
and all synaptic weights are optimized jointly during the learning. To elucidate the loss function, we first examine the compressive term. As we adopt Gaussian distributions in our temporal conditioning prior and encoder, the compressive term composed of KL divergences can be calculated analytically. In particular, we have,
$$
\mathcal {L} ^ {\text {c o m p}} = \frac {1}{T} \sum_ {t = 1} ^ {T} \mathrm {K L} [ q (\mathbf {z} _ {t} | \mathbf {x} _ {1: t}; \psi^ {\text {e n c}}) \| p (\mathbf {z} _ {t} | \mathbf {x} _ {1: t - 1}; \phi^ {\text {p r i o r}}) ]. \tag {8}
$$
On the other hand, since our model directly outputs spike trains, we adopt the Maximum Mean Discrepancy (MMD) as the predictive loss term [26, 50]. Also, we use the first-order postsynaptic potential (PSP) kernel that can effectively depict the temporal dependencies in spike train data [51, 52]. Denoting the predicted, recorded spike trains as  $\hat{\mathbf{y}}$ ,  $\mathbf{y}$ , respectively, we can write the PSP kernel MMD predictive loss as
$$
\mathcal {L} ^ {\text {p r e d}} = \frac {1}{T} \sum_ {t = 1} ^ {T} \sum_ {\tau = 1} ^ {t} \left\| \operatorname {P S P} \left(\hat {\mathbf {y}} _ {1: \tau}\right) - \operatorname {P S P} \left(\mathbf {y} _ {1: \tau}\right) \right\| ^ {2}, \tag {9}
$$
where  $\mathrm{PSP}(\mathbf{y}_{1:\tau}) = (1 - \frac{1}{\tau_s})\mathrm{PSP}(\mathbf{y}_{1:\tau -1}) + \frac{1}{\tau_s}\mathbf{y}_\tau$ , and we set the time constant  $\tau_{s} = 2$
3.2 TeCoS-LVM Models
We shall consider two types of TeCoS-LVM models, TeCoS-LVM and TeCoS-LVM Noisy in this work. Specifically:
- In TeCoS-LVM, all spiking neurons are LIF neurons.
- In TeCoS-LVM Noisy, all spiking neurons are Noisy LIF spiking neurons ([53], refer to Appendix C for details). Using Noisy LIF neuron allows TeCoS-LVM to have neuron-level stochasticity, which is considered a crucial component in biological neural computation [54, 55, 56].
4.1 Dataset and Baselines
We perform evaluations and analyses on real neural recordings from RGCs of dark-adapted axolotl salamander retinas [57]. The dataset contains spike responses of two retinas on two movies (see also Appendix E). We partitioned all records into stimuli-response sample pairs of 1 second (30 time bins) and down-sampled the frames to 90 pixel  $\times$  90 pixel. This results in four datasets, each split into non-overlapping train/test  $(50\% / 50\%)$  parts. We shall refer to these four datasets as follows for brevity. Movie 1 Retina 1: records of 38 RGCs on movie 1 ("salamander movie"), 75 repetitions. Movie 1 Retina 2: records of 49 RGCs on movie 1, 30 repetitions. Movie 2 Retina 1: records of 38 RGCs on movie 2 ("wildlife movie"), 107 repetitions. Movie 2 Retina 2: records of 49 RGCs on movie 2, 42 repetitions.
CNN model We use the state-of-the-art CNN model [9, 15] for comparison. In the CNN model, the network's predicted outputs are the average maximum likelihood estimation of retina responses in firing rates based on spatiotemporal (frames are concatenated on the channel dimension) input (see also Appendix G).
IB-Disjoint IB-Disjoint [21] is a high-performance model that performs similarly to the IB-GP model that uses a Gaussian Process prior [21]. This model employs an isotropic Gaussian as the prior over latents. It is similar to the vanilla variational IB, where the latent variables are assumed to be independent along the temporal dimension.
4.2 Metrics and Features for Evaluations and Visualizations
Pearson correlation coefficient This metric evaluates the model performance by calculating the Pearson correlation coefficient between the recorded and predicted firing rates [9, 14, 15]. The higher the value, the better the performance. Refer to Appendix F for more details.
Spike train dissimilarity This metric assesses the model performance by computing the dissimilarity between recorded and predicted spike trains. A lower value indicates better model performance. Here we use the MMD with a first-order PSP kernel [51, 52] to measure the spike train dissimilarity [27, 50, 58] (see also Appendix F).
Spike autocorrelogram The spike autocorrelogram is computed by counting the number of spikes that occur around each spike within a predefined time window [14, 9]. The resulting trace is then normalized to its maximum value (which occurs at the origin of the time axis by construction), and the maximum value is set to zero for better visualization. Refer to Appendix F for more details.
TeCoS-LVM hyper-parameters were fixed to be the same on all four datasets. We set the latent variable dimension to 32 and the hidden state dimension to 64 by default (see also Appendix A). We used the Adam optimizer  $(\beta_{1} = 0.9, \beta_{2} = 0.999)$  with a cosine-decay learning rate scheduler [59], starting at a rate of 0.0003. The mini-batch size was set to 64, and the models were trained for 64 epochs. We used the same architectures to implement TeCoS-LVM models. The factor  $\beta$  in the loss function (7) is set to 0.01 by default to balance information compression and predictive power [32, 60]. More experimental details are presented in Appendix G.
At test time, the test samples have the same length as the training samples (30 bins, 1 second) by default. However, the learned model can handle longer test samples as our model excludes the time dimension from its parameter space. This will be discussed further in the Results sub-section. We tested TeCoS-LVM models with a "warmup period" of 0.5 seconds (15 bins), during which the
model's predictions will be discarded (see also Fig. 4A). This was done because our hidden state was zero-initialized, making the early predictions less accurate.
Figure 2: TeCoS-LVM models well fit the spike statistics of real neural activities. The error bars (SD) were computed across multiple random seeds. A, Graphical illustrations of the processing flows of baselines and TeCoS-LVM models. B, Heatmaps of the real (Data) and predicted firing rates on Movie 2 Retina 2 test data. The TeCoS-LVM Noisy model precisely reproduces the firing rate patterns of real neural data. C, Histograms of firing rate Pearson correlation coefficients on test data of four datasets. D, Autocorrelograms acquired on test data of four datasets. While the baselines fail, the TeCoS-LVM models accurately capture the spike autocorrelations. E, Recorded and predicted activities of two representative neurons responding to repeated trials of a randomly selected test segment of Movie 1 Retina 1 data.
4.4.1 TeCoS-LVM Models Accurately Fit Real Spike Activities and Statistics
As shown in Fig. 2B, 2C, TeCoS-LVM models can effectively fit the recorded firing rates. Especially, TeCoS-LVM Noisy can reproduce real firing rates more accurately than baselines. Furthermore, we verified the potential information loss caused by using trial-averaged statistics as the optimization target. While the firing rate target may allow models to learn coarse-grained features, it does not necessarily yield optimal capturing of fine-grained features such as spike autocorrelations. As shown by the autocorrelograms in Fig. 2D, the baselines failed to capture the spike autocorrelation
Figure 3: TeCoS-LVM models synthesize realistic neural activities and generalize well to larger time scales. A. Examples of the predicted spike trains on a Movie 1 Retina 2 test data clip. The spike trains generated by the TeCoS-LVM models are closer to the recorded spike trains than those of the baselines. B. Histograms of spike train dissimilarities on test data. The dissimilarities acquired using TeCoS-LVM models are significantly lower than those of baselines. C. TeCoS-LVM models learn general temporal dependencies. We ran TeCoS-LVM models trained using 1-second sequences on longer test sequences and observed that extending the length of the test sequence only leads to minor performance drops.
feature accurately. By contrast, TeCoS-LVM models reproduced the spike autocorrelations more precisely. The TeCoS-LVM models also outperformed other models in synthesizing more realistic spike activities (Fig. 3A&B). Our results suggest that TeCoS-LVM models can accurately fit real spike activities and statistics. In particular, the TeCoS-LVM Noisy model significantly outperforms baselines on all metrics.
We also noticed that including neuronal noise is important for modeling neural activities. Because of the absence of neuron-level randomness, the TeCoS-LVM model failed to reproduce trial-to-trial variability (Fig. 2E) in real neurons. Consequently, as shown in Fig. 2B, TeCoS-LVM's firing rate predictions (obtained by averaging over multiple runs) are less smooth than others, thus negatively impacting the firing rate correlations. That explains why the TeCoS-LVM achieved lower spike train dissimilarities on some data (Movie 1 Retina 2, Movie 2 Retina 2 in Fig. 3B) but still lads behind other methods in terms of the firing rate metric (Fig. 2C).