SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Evaluation using more spike distances The TeCoS-LVM models are specifically optimized to minimize the discrepancy in the PSP kernel space, making them naturally advantageous in comparing PSP-MMD-based spike train dissimilarity. To ensure a more fair assessment of the spike train prediction quality, we have considered several different spike train distances. Specifically, we computed van Rossum, Victor-Purpura, and SPIKE distances (refer to Appendix F for more details) between the predicted spike trains with the recorded ones. We observed consistent and significant improvements in the TeCoS-LVM models over two state-of-the-art baselines, as shown by results in Table 1. We also noticed that when considering these spike train distances, the performance of TeCoS-LVM Noisy is slightly affected due to the noise-perturbed neuronal dynamics, which accords with results in Fig. 3B. However, the overall performance of TeCoS-LVM Noisy surpasses the baselines by a large margin and, moreover, effectively reproduces the variability in neural processing.
4.4.2 Train Short, Test Long: Learned TeCoS-LVM Models Generalize to Longer Time Scales
TeCoS-LVM models completely exclude the time dimension from parameter space and, therefore, can handle input stimuli sequences of any duration without being limited to the length of training data (1 second here). We evaluated the temporal scalability of TeCoS-LVM models by running them on
longer test data sequences. The firing rate correlation coefficients and spike train dissimilarities were calculated to measure the performance changes. TeCoS-LVM models consistently produce accurate coding results at different time scales. Results in Fig. 3C show that increasing the length of the test sequence only brings slight performance drops, as evidenced by the minor decrease in the firing rate correlations and little increase in the spike train dissimilarities. This demonstrates that TeCoS-LVM models learn general (multi-time-scales) temporal dependencies from short-sequence training.
Table 1: Evaluation results using SPIKE, Victor-Purpura, and van Rossum spike train distances, results reported here are averaged across multiple trials.
DISTANCE TYPE	SPIKE	Victor-Purpura	van Rossum	SPIKE	V.-P.	van Rossum
DATA
MODEL	MOVIE1 RETINA1	MOVIE1 RETINA2
TeCoS-LVM Noisy (This work)	0.155	14.024	238.614	0.116	21.599	425.871
TeCoS-LVM (This work)	0.124	12.835	127.346	0.111	18.182	150.445
McIntosh NeurIPS-16	0.207	19.601	376.822	0.220	39.168	2672.211
Rahmani NeurIPS-22	0.224	21.916	394.020	0.219	39.075	2276.706
DATA
MODEL	MOVIE2 RETINA1	MOVIE2 RETINA2
TeCoS-LVM Noisy (This work)	0.162	14.412	553.510	0.153	28.441	1135.805
TeCoS-LVM (This work)	0.128	12.693	308.784	0.123	22.666	574.298
McIntosh NeurIPS-16	0.212	22.713	1650.823	0.221	39.261	2638.964
Rahmani NeurIPS-22	0.204	22.271	1615.934	0.221	38.378	2244.981
Figure 4: Visualizations of hidden state and latent space dynamics. A. The evolution of  $\|\mathbf{h}_t\|$  over time after initialization. B. SSIM curve of neighboring stimuli and cosine distance curves of adjacent hidden state vectors of TeCoS-LVM models. C. Scatterplots of the Pearson correlation coefficients between hidden state cosine distance and stimuli SSIM. D. Model performances under different weighting factor  $\beta$ s, obtained on Movie 1 Retina 2 test data. E. Visualizations of  $\mathbf{z}_t$  dynamics in TeCoS-LVM Noisy models learned using different levels of  $\beta$ . F. Model performances with and without TeCo (shorthand of temporal conditioning operation). We increased the number of parameters for models without TeCo to be approximately on par with the standard models.
4.4.3 Hidden State Analyses: Exploring Memory Mechanisms in TeCoS-LVMs
Interestingly, TeCoS-LVM models' predictions are less accurate for that short period after initialization, likely due to the model's sensory memory requiring time to accumulate. We observed that it takes at most 0.5 seconds for the L2 norm of the hidden state to reach the average value of consecutive runs on a long sequence (Fig. 4A). As a result, we set a warmup period of 0.5 seconds for all our tests. We further investigated the memory mechanism in TeCoS-LVM models, which is implemented by the hidden state update. To this end, we quantified the variation of stimuli by computing the Structural Similarity (SSIM) of adjacent stimuli  $(\mathbf{x}_t$  and  $\mathbf{x}_{t - 1})$ . If the change in stimuli at a certain moment is drastic, then the SSIM at this moment is low. For the sensory system, the stimulus it receives at that time is quite novel or surprising [61]. We also measured the memory update magnitude by calculating the cosine distance of adjacent hidden state vectors  $(\mathbf{h}_t$  and  $\mathbf{h}_{t - 1}$ ). As shown in Fig. 4B,
C, the two have a strong negative correlation. This suggests that the TeCoS-LVM models perform more significant memory updates (indicated by a large cosine distance) when facing highly dynamic stimuli (low SSIM); only minor updates are performed when stimuli are relatively stable. This is consistent with previous sensory neuroscience findings suggesting that the sensory circuits focus more on unpredictable or surprising events. When repeatedly exposed to an initially novel stimulus, the neural processing becomes less active [62, 63, 61, 64].
4.4.4 Latent Space Dynamic Analyses
We next explored the latent variables under different weighting factor  $\beta$  settings. Setting a proper  $\beta$  value can lead to richer latent variable dynamics. It is more advantageous to show how they relate to factors like anatomy and behavior through correlation analysis, thereby interpreting the inferred latent variables. The compressive loss term forces the latent variable to act like a minimal sufficient statistic of stimuli for predicting neural responses. Therefore, tiny  $\beta$  values correspond to weak regularization provided by the compressive term. In this case, the latent representation learns to be more deterministic (Fig. 4E-Left). This prevents the model from benefiting from the regularization brought by the compressive loss term, resulting in poorer performance compared to moderate  $\beta$  values (Fig. 4D). When using a large  $\beta$ , we observed that the latent variables seem to have lost some temporal information in the stimuli (Fig. 4E-Right). And, if  $\beta$  is further increased, the model cannot obtain enough stimuli information to predict the responses, resulting in a sharp decline in performance (Fig. 4D).
4.4.5 Functional Implication Analyses by In-silico Ablating
Temporal conditioning operation We then evaluated the performance of TeCoS-LVM models without temporal conditioning operations, which is a reduced variant with an isotropic Gaussian prior (see also Appendix Fig. 5). Results in Fig. 4F show that the TeCo operation significantly improves model performance by exploiting the temporal dependencies. This suggests that visual coding requires the integration and manipulation of temporally dispersed information from continuous streams of stimuli.
Spiking hidden neurons We conducted experiments to investigate the effect of employing spiking neurons in modeling neural activity to natural stimuli (see also Appendix H). Specifically, we replaced all spiking hidden neurons in the TeCoS-LVM models with LIF-Rate neurons[65] while retaining only the output spiking neurons so that the training pipeline remains unchanged. The LIF-Rate neurons maintain the neuronal recurrence by using the same membrane potential update as LIF and Noisy LIF neurons. This allows us to directly assess the effect of using spiking neurons by ablating them. Our experimental results show that the use of spiking neurons can significantly enhance the performance of computational models. As can be seen from Table 2, all performance indicators have significantly decreased after replacing the spiking hidden units with rate ones. Previous studies have pointed out that natural stimuli have a sparse latent structure [66]; therefore, using spiking hidden units may better fit the latent structure of natural stimuli by utilizing sparse spike representation. Also, previous research [67] indicated that the coding method of SNNs can lead to highly competitive results, which is consistent with the conclusion of this part of our experiment.
Table 2: Ablation results of using spiking hidden neurons. An  $\uparrow$  indicates that the higher the value, the better, while a  $\downarrow$  suggests the opposite. Results reported are averaged across multiple trials.
Spiking hidden units	CC (↑)	Spike Train Dissim. (↓)	SPIKE (↓)	Victor-Purpura (↓)	van Rossum (↓)
Mov1 Ret1	TeCoS-LVM	Yes	0.579	371.057	0.124	12.835	127.346
No	0.254	850.418	0.259	45.117	3416.557
TeCoS-LVM Noisy	Yes	0.728	354.989	0.155	14.024	238.614
No	0.653	370.099	0.167	14.805	291.706
Mov2 Ret2	TeCoS-LVM	Yes	0.616	1003.489	0.123	22.666	574.298
No	0.471	1273.267	0.180	35.080	1890.910
TeCoS-LVM Noisy	Yes	0.822	1021.384	0.153	28.441	1135.805
No	0.748	1078.830	0.159	29.249	1144.087
This work builds on previous research on modeling neural responses to visual scenes. Early attempts included Linear-nonlinear (LN) [68, 69] and Generalized Linear Models (GLMs) [2, 7]. However,
these models have limited capabilities and fail when faced with more complex stimuli [8, 9]. A promising way is to leverage powerful neural networks. One attractive advantage of this approach is that it may eliminate the need to specify spike statistics explicitly. McIntosh et al. [9] proposed a convolutional neural network (CNN) approach that outperformed LN and GLM baselines by a large margin. Some researchers have also investigated CNN variants with a recurrent layer [9, 15]. Batty et al. [12] proposed a hybrid model that combines GLMs and recurrent neural networks (RNNs) to separate spatial and temporal processing components in neural coding. Our model also has specified structures that deal with the time dimension, but the temporal and spatial processing are tightly integrated. Generative adversarial networks (GANs) have also been used to synthesize realistic neural activities [14], but this method cannot be used to predict neural responses given the stimuli. Mahuas et al. [70] introduced a novel two-step strategy to improve the basic GLMs. Bellec et al. [71] introduced a model with spiking response model (SRM) neurons and proposed a sample and measure criteria in addition to the traditional Poisson likelihood objective. More recently, Rahmani et al. [21] introduced the Gaussian Process prior and variational information bottleneck to LVMs for modeling retinal responses. Compared to biological coding systems, these methods have two aspects of authenticity missing (Fig. 2A). Firstly, most of these methods target trial-averaged firing rates. Secondly, they cannot directly process long stimuli sequences in a natural paradigm but must divide them into segments of pre-defined lengths to be processed separately.
Latent variable models LVMs are popular tools for investigating and modeling neural response [18, 19, 72]. Some studies suggest that low-dimensional latent factors can effectively represent high-dimensional neural activities [16, 17]. In recent works, LVMs demonstrated promising results in uncovering the low-dimensional structure underlying complex neural activities in the motor region [20, 23, 22]. Also, LVMs have shown promise in visual neural coding modeling [18, 19, 21].
Conditioning Conditioning is an effective technique to integrate information from multiple sources and realize adaptive information gating. Typically, conditioning methods rely on external information sources like labels [73] or outputs from other models [74, 75]. Some studies have demonstrated that direct conditioning on previous states can also significantly improve performance, especially in a sequential processing regime [49, 76]; these approaches are often termed self-conditioning or temporal conditioning. Notably, the temporal conditioning design exposed here is loosely inspired by the Tolman-Eichenbaum Machine [49], where the conditional operations ensure that the model profits from a sequential learning paradigm.
This work presents the TeCoS-LVM (temporal conditioning spiking LVM) models. Our approach is formalized within the information bottleneck framework of latent variable models, inspired by the efficient coding theory [34, 38]. TeCoS-LVM models can directly produce neural response sequences in real-time rather than repeatedly producing single-step predictions, thus faithfully simulating the biological coding process. Using the retinal response to natural scenes as an example, we showed that TeCoS-LVM models effectively fit spike statistics, spike features, and recorded spike trains. In particular, TeCoS-LVM models that incorporate Noisy LIF neurons significantly exceed high-performance baselines. Also, learned TeCoS-LVM models generalize well to longer time scales, indicating that they can learn general temporal features from short-sequence training. Overall, TeCoS-LVM models effectively capture key features of biological coding systems while remaining computationally tractable.
Outlook Although we use visual coding as an example to demonstrate the impressive performance of TeCoS-LVM models in this article, the models exposed here can be easily applied to sensory data of other modalities. As such, TeCoS-LVM demonstrates a promising framework for building computational accounts for various sensory neural circuits and will enable richer models of complex neural computations in the brain.
Limitations As an accurate model that relates stimulus-driven responses, from a neuroconnectionsim perspective [77], the TeCoS-LVM model aims to provide neuroscientific insights and understandings at the computational level. Compared to previous endeavors, TeCoS-LVM takes spike-based neural computation and natural processing paradigms into account. Nonetheless, while the TeCoS-LVM model makes predictions at the level of retinal cells, its fundamental principles are at a computational level, so it is only partially biophysically realistic.
This work is supported by the National Key Research and Development Program of China under Grant 2020AAA0105900 and the National Natural Science Foundation of China under Grant 62236007, Grant U2030204.
The authors are grateful for the generous support from Professor Arno Onken from the University of Edinburgh. The authors would also like to acknowledge anonymous reviewers and chairs of NeurIPS 2023 for providing insightful comments to help improve this work.
[1] Tim Gollisch and Markus Meister. Eye smarter than scientists believed: neural computations in circuits of the retina. Neuron, 65(2):150-164, 2010.
[2] Jonathan W Pillow, Jonathon Shlens, Liam Paninski, Alexander Sher, Alan M Litke, EJ Chichilnisky, and Eero P Simoncelli. Spatio-temporal correlations and visual signalling in a complete neuronal population. Nature, 454(7207):995-999, 2008.
[3] David B Kastner and Stephen A Baccus. Coordinated dynamic encoding in the retina using opposing forms of plasticity. Nature neuroscience, 14(10):1317-1322, 2011.
[4] Joseph J Atick and A Norman Redlich. Towards a theory of early visual processing. Neural computation, 2(3):308-320, 1990.
[5] Bence P Ölveczky, Stephen A Baccus, and Markus Meister. Segregation of object and background motion in the retina. Nature, 423(6938):401-408, 2003.
[6] Eric Y. Wang, Paul G. Fahey, Kayla Ponder, Zhuokun Ding, Andersen Chang, Taliah Muhammad, Saumil Patel, Zhiwei Ding, Dat Tran, Jiakun Fu, Stelios Papadopoulos, Katrin Franke, Alexander S. Ecker, Jacob Reimer, Xaq Pitkow, Fabian H. Sinz, and Andreas S. Tolias. Towards a foundation model of the mouse visual cortex. bioRxiv, 2023. doi: 10.1101/2023.03.21.533548.
[7] Jonathan W Pillow, Liam Paninski, Valerie J Uzzell, Eero P Simoncelli, and EJ Chichilnisky. Prediction and decoding of retinal ganglion cell responses with a probabilistic spiking model. Journal of Neuroscience, 25(47):11003-11013, 2005.
[8] Alexander Heitman, Nora Brackbill, Martin Greschner, Alexander Sher, Alan M Litke, and EJ Chichilnisky. Testing pseudo-linear models of responses to natural scenes in primate retina. BioRxiv, 2016.
[9] Lane McIntosh, Niru Maheswaranathan, Aran Nayebi, Surya Ganguli, and Stephen Baccus. Deep learning models of the retinal response to natural scenes. In NeurIPS, 2016.
[10] Nikolaus Kriegeskorte. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annual review of vision science, 1:417-446, 2015.
[11] Guangyu Robert Yang and Xiao-Jing Wang. Artificial neural networks for neuroscientists: a primer. Neuron, 107(6):1048-1070, 2020.
[12] Eleanor Batty, Josh Merel, Nora Brackbill, Alexander Heitman, Alexander Sher, Alan Litke, E.J. Chichilnisky, and Liam Paninski. Multilayer recurrent network models of primate retinal ganglion cell responses. In ICLR, 2017.
[13] Fabian Sinz, Alexander S Ecker, Paul Fahey, Edgar Walker, Erick Cobos, Emmanouil Froudarakis, Dimitri Yatsenko, Zachary Pitkow, Jacob Reimer, and Andreas Tolias. Stimulus domain transfer in recurrent models for large scale cortical population prediction on video. In Advances in neural information processing systems, volume 31, 2018.
[14] Manuel Molano-Mazon, Arno Onken, Eugenio Piasini*, and Stefano Panzeri*. Synthesizing realistic neural population activity patterns using generative adversarial networks. In *ICLR*, 2018.
Appendix of Temporal Conditioning Spiking Latent Variable Models of the Neural Response to Natural Visual Scenes
A Hidden State and Latent Space Experiments
Figure 5: Graphical illustration of the temporal conditioning operation (TeCo). After completely excluding the temporal dimension from the model parameter space, we introduced the temporal conditioning operation to handle the temporal information. In particular, this operation enables memory-dependent processing as in biological coding circuits.
Figure 6: Performances under different hidden state and latent space dimension settings on Movie 2 Retina 2 data. For hidden state experiments, the latent space dimension is set to 32. And for latent space experiments, the hidden state dimension is 64. The error bars (SD) were calculated via various random seeds.
To further study the sensitivity to the choice of hyperparameters, we evaluated the performance of TeCoS-LVM models under different hidden state and latent space dimensionality settings. The results, as shown in Appendix Fig. 6A, indicate that increasing the latent space dimension improved performance. Still, further increases (larger than 32) had little effect when the latent space dimension was large. This also suggests that the number of latent factors for the visual stimuli coding task we considered is not very large. On the other hand, increasing the hidden state dimension enhances sensory memory capacity and benefits temporal conditioning operations. As shown in Appendix Fig. 6B, the performance does not increase significantly when hidden state dimensionality increases to a certain extent (around 64). In particular, although the spike train dissimilarity decreases, the firing rate correlation score almost no longer increases. This is consistent with our observation in the main text, namely, a lower spike train dissimilarity does not always indicate a higher firing rate correlation score (refer to Fig. 2,3). The results in Appendix Fig. 6B also indicate that the proposed temporal conditioning mechanism can effectively utilize sensory memory and achieve good results even when the hidden state dimension is limited.
B Surrogate Gradient Learning in Spiking Networks
In conventional SNN Surrogate Gradient Learning (SGL, pseudo derivative, derivative approximation) [29], the derivative of the firing function  $\frac{\partial o}{\partial u}$  is replaced by a smooth function (pseudo derivative
function) SG to mesh with the backpropagation scheme [78]. This ad-hoc technique in SNN research is popular, particularly for large-scale networks. It allows for compatibility with popular automatic differentiation packages such as PyTorch and TensorFlow, simplifying the implementation of SNNs. This surrogate gradient function can be a triangular, rectangular (gate), sigmoidal, or ERF function [78]. In SGL, the gradient  $g_{l}$  w.r.t. synaptic weights of layer  $l$  is calculated by
$$
\operatorname {S G L}: \hat {g} _ {l} = \sum_ {m} \nabla_ {\theta_ {t}} u _ {t} ^ {l, m} \underbrace {\operatorname {S G} \left(u _ {t} ^ {l , m} - v _ {\text {t h}}\right)} _ {\text {S u r g o r a t e t h e e x a c t d e r i v a t i v e} \partial o _ {t} ^ {l, m} / \partial u _ {t} ^ {l, m}} \nabla_ {o _ {t} ^ {l, m}} \mathcal {L} _ {t}, \tag {10}
$$
where  $l, m$  denotes neuron  $m$  in layer  $l$ , and  $\mathcal{L}_t$  is the instant loss value.
C Leveraging Noisy Spiking Neural Models
Here, we use the implementation in [53] to leverage the power of noisy spiking neural models. Spiking neurons with noisy neuronal dynamics have been extensively studied in prior literature [25]. Recent research of Ma et al. [53] extended them to larger networks by providing a general formularization and demonstrating their computational advantages theoretically and empirically. The Noisy LIF presented here is based on previous works that use diffusive approximation [79, 80, 25], where the sub-threshold dynamic is described by the Ornstein-Uhlenbeck process:
$$
\tau_ {m} \frac {\mathrm {d} u}{\mathrm {d} t} = - (u - u _ {\text {r e s e t}}) + R I (t) + \xi (t), \text {e q .} \mathrm {d} u = - (u - u _ {\text {r e s e t}}) \frac {\mathrm {d} t}{\tau_ {m}} + R I (t) \frac {\mathrm {d} t}{\tau_ {m}} + \sigma \mathrm {d} W _ {t}, \tag {11}
$$
the white noise  $\xi$  is a stochastic process,  $\sigma$  is the amplitude of the noise and  $\mathrm{d}W_{t}$  are the increments of the Wiener process in  $\mathrm{d}t$  [25]. As  $\sigma \mathrm{d}W_{t}$  are random variables drawn from a zero-mean Gaussian, this formulation is directly applicable to discrete-time simulations. Specifically, using the Euler-Maruyama method, we get a Gaussian noise term added on the right-hand side of the noise-free LIF dynamic. Without loss of generality, we extend the additive noise term in the discrete form to general continuous noise [81], the sub-threshold dynamic of Noisy LIF can be represented as:
$$
\text {N o i s y L I F} u _ {t} = \tau u _ {t - 1} + I _ {t} + \epsilon , \tag {12}
$$
where  $I_{t}$  is the input, the noise  $\epsilon$  is independently drawn from a known distribution and satisfies  $\mathbb{E}[\epsilon] = 0$  and  $p(\epsilon) = p(-\epsilon)$ . The constant  $\tau$  here combines the simulation timestep length and the real membrane decay  $\tau_{m}$ , which is a simplification when the timestep we cope with is fixed. This work considers the Gaussian noise  $\epsilon \sim \mathcal{N}(0,0.2^{2})$ .
The membrane potentials and spike outputs become random variables due to random noise injection. Leveraging noise as a medium, we naturally obtain the firing probability distribution of Noisy LIF based on the threshold firing mechanism [53]:
$$
\mathbb {P} [ \text {f i r i n g a t t i m e} t ] = \mathbb {P} \underbrace {\left[ u _ {t} + \epsilon > v _ {\text {t h}} \right]} _ {\text {T h r e s h o l d - b a s e d f i r i n g}} = \underbrace {\mathbb {P} \left[ \epsilon <   u _ {t} - v _ {\text {t h}} \right] \triangleq F _ {\epsilon} (u _ {t} - v _ {\text {t h}})} _ {\text {C u m u l a t i v e D i s t r i b u t i o n F u n c t i o n d e f i n i t i o n}},
$$
where  $F$  denotes the cumulative distribution function. Therefore, we have that,
$$
o _ {t} = \left\{ \begin{array}{l l} 1, \text {w i t h p r o b a b i l i t y} & F _ {\epsilon} \left(u _ {t} - v _ {\mathrm {t h}}\right), \\ 0, \text {w i t h p r o b a b i l i t y} & \left(1 - F _ {\epsilon} \left(u _ {t} - v _ {\mathrm {t h}}\right)\right). \end{array} \right. \tag {13}
$$
The expressions above exemplify how noise acts as a resource for computation [82]. Thereby, we can formulate the firing process of Noisy LIF as [53]
$$
\text {N o i s y L I F p r o b a b i l i s t i c f i r g :} o _ {t} \sim \text {B e r n o u l l i} \left(F _ {\epsilon} \left(u _ {t} - v _ {\mathrm {t h}}\right)\right), \tag {14}
$$