Temporal Conditioning Spiking Latent Variable Models of the Neural Response to Natural Visual Scenes
Authors: Gehua Ma
Runhao Jiang
Rui Yan
Huajin Tang
Venue: NeurIPS 2023 poster (2023)
Keywords: neuroscience
neural coding
sensory neuroscience
visual coding
SNN
spiking neural networks
generative model
latent variable model
cognitive computational neuroscience
computational neuroscience

Abstract: Developing computational models of neural response is crucial for understanding sensory processing and neural computations. Current state-of-the-art neural network methods use temporal filters to handle temporal dependencies, resulting in an **unrealistic and inflexible processing paradigm**. Meanwhile, these methods target **trial-averaged firing rates** and fail to capture important features in spike trains. This work presents the temporal conditioning spiking latent variable models (***TeCoS-LVM***) to simulate the neural response to natural visual stimuli. We use spiking neurons to produce spike outputs that directly match the recorded trains. This approach helps to avoid losing information embedded in the original spike trains. We exclude the temporal dimension from the model parameter space and introduce a temporal conditioning operation to allow the model to adaptively explore and exploit temporal dependencies in stimuli sequences in a **natural paradigm**. We show that TeCoS-LVM models can produce more realistic spike activities and accurately fit spike statistics than powerful alternatives. Additionally, learned TeCoS-LVM models can generalize well to longer time scales. Overall, while remaining computationally tractable, our model effectively captures key features of neural coding systems. It thus provides a useful tool for building accurate predictive computational accounts for various sensory perception circuits.

Temporal Conditioning Spiking Latent Variable Models of the Neural Response to Natural Visual Scenes
Gehua Ma
Runhao Jiang
College of Computer Science and Technology
College of Computer Science and Technology
Zhejiang University
Zhejiang University
g gehuama@icloud. com
RhJi ang@zju. edu. cn
Rui Yan
Huajin Tang"
College of Computer Science and Technology College of Computer Science and Technology Zhejiang University
Zhejiang University of Technology ryan@zjut. edu. cn
Abstract
Developing computational models of neural response is crucial for understand- ing sensory processing and neural computations. Current state-of-the-art neural network methods use temporal filters to handle temporal dependencies, resulting in an unrealistic and inflexible processing paradigm. Meanwhile, these methods target trial-averaged firing rates and fail to capture important features in spike trains. This work presents the temporal conditioning spiking latent variable models (TeCoS-LVM) to simulate the neural response to natural visual stimuli. We use spiking neurons to produce spike outputs that directly match the recorded trains. This approach helps to avoid losing information embedded in the original spike trains. We exclude the temporal dimension from the model parameter space and in- troduce temporal conditioning operation to allow the model adaptively explore and exploit temporal dependencies stimuli sequences We show that TeCoS-LVM models can produce more realistic spike activities and accurately fit spike statistics than powerful alternatives. Additionally, learned TeCoS-LVM models can generalize well longer time scales. Overall, while remaining computationally tractable, our model effectively captures key features neural coding systems. thus provides useful tool for building accurate predictive computational accounts for various sensory perception circuits.
natural paradigm.
Introduction
Building precise computational models of neural response to natural visual stimuli is fundamental scientific problem in sensory neuroscience. These models can offer insights into neural circuit
computations, reveal new mechanisms, and validate theoretical predictions 22 4 5 6). However, constructing such models is challenging due to the complex nonlinear processes involved in neural coding, such as synaptic transmission and spiking dynamics. For modeling retinal responses, early attempts using linear-nonlinear (LN) models and generalized linear models (GLMs) were successful with simple data such white noise 2 7 but fell short with more complex stimuli like natural visual scenes (8] (. Artificial neural networks (ANNs), which are powerful function approximators and loosely resemble biological neurons and architectures I0 have shown promise in modeling
"Corresoondene:: htang@zju. eew.eeejuuuuu.cc
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

the visual stimuli coding process through various ANN -Nseb methods 12 13 14 [5]. Recent research has demonstrated that complex neural activity can be well represented in low-dimensional space 16 17 This has led to growing interest in a type of neural network known as latent variable models Ms. LVMs strike balance between model accuracy and representation space simplicity, enabling accurate neural coding modeling and interpretation of neural activity in low-dimensional space [18 19 20
21 22 23 24). Although the current state-of-the-art neural network models for
visual neural coding have produced decent results and provided some exciting insights, they have two major limitations:
Most of these works focus on simulating the firing rates of real neurons. This is a decent choice (following the classic Poisson LN/GLM models) for artificial neuron networks, which are essen- tially real-valued processing- -aaeesding-baed.. However, as a riiaa-averaged spike statistic, firing rates only characterize some aspects of the original spike train [25] As result, directly using firing rates as the target may result in losing information embedded in the original spike trains [26] (7].
Existing models mostly employ fixed-length temporal filters. For example, the CNN approach [9] concatenates stimuli within a fixed duration as input and processes these inputs using temporal filters. Therefore, they cannot process long stimuli sequences like a real neural circuit. Instead, they must slice the long sequences into fixed-length short segments and process them separately, thus losing biological realism (see also Fig. 2 Secondly, in simulation, the learned models can only take inputs of the same length as during the training phase, thus limiting their flexibility.
Although these two points are important for a realistic computational model, an approach that addresses these limitations is still lacking. This work introduces the TeCoS-LVM (temporal condi- tioning spiking LVM) models of the neural responses to natural visual stimuli. We employ spiking neurons allow the model to aim directly at producing realistic spike trains. This avoids spike train information loss that might occur when targeting spike statistics. To address the second limitation, we completely exclude the temporal dimension from the parameter space and introduce a temporal conditioning operation to handle the temporal dependencies. Inspired by the information compression in biological coding systems, we formalize TeCoS-LVM models within the information bottleneck framework of LVMs and introduce general learning method for them. Evaluations on real neural recordings demonstrate that TeCoS-LVM models accurately fit spike statistics and produce realistic spike activities. Further simulations show that TeCoS-LVM models learned on short sequences can generalize well to longer time scales and exhibit memory mechanisms similar to those in biological cognitive circuits.
2
Preliminaries
Leaky integrate-and-five spiking neuron We adopt the Leaky Integrate-and-Fing (LIF) neuron model in this work, which briefly describes the sub-threshold membrane potential dynamics as Tm ă - RI(I), where u, denotes membrane potential, R, Tm are membrane resistance, time constant, and / is the input current. Vih. freset denote the firing threshold, resting potential, respectively. In practice, it leads to the following discrete-time computational form 28 29 30 The sub-threshe -hrreshldd dynamic. firing, and resetting are written as u; = TU,-1 + L1; Hcaviside(4, Vun) Ifeeee if o, where o, is the spike output, 1, is the input current, usually transformed by a parameterized mapping. and r is the membrane time constant. In this discrete form, the membrane constant Tm is combined with timestep di for simplification, as in all our experiments, the length of simulation timestep di is fixed. To introduce a simple model of neuronal spiking and refractoriness, we assume = 1, 0.5 a fixed constant, and Mreset for all spiking neurons throughout this research.
Variational information bottleneck The information bottleneck (IB) principle 33 offers an appealing framework to formulate LVMs. In short, desired LVM should have latent representations that are maximally expressive regarding its target while being maximally compressive about its input. Let us consider a latent variable model 0 with input x, target y. and latent representation z defined by a parametric encoder q(z]x; 0). Let I(2, y; f) be the mutual information between the z and y, and I(z, x; 0) be the mutual information between and x, IB suggests an objective
max I(z. y: 0) s.t. I(z, x) 0) lc. 0
where 4. is the information constraint. With the introduction of a Lagrange multiplier / [1, the IB objective is equivalent to
max[I(z, y; 0) -BI(z, x: 8)).
(2)
2

---

RNN
8
Visual Stimull 1-T
Spiking Neuron Network Encoder
Spiking Neuron Network Decoder
Easun Extraction
Latent Variable
Spike Train Dissiilanity
Recorded Vier
Spike train sini
time
Viruall RoCs
Sme
B
Visual Samulius
Latent Variable
Neural Population Response
Condltooing Prior
Temporal
Inforence
Generation
Update
x
prior
prior
prior
prior
h
h
Hidden State (Seneory Memory)
Q(x,1x,.d)
ply, 12 x11--)
h function of X1:t
Divergance
Figure Overview of our approach. A. Graphical illustration of TeCoS-LVM models of retinal neural response to natural visual scenes. Our model can directly generate neural response sequences in real-time, thus faithfully simulating the real neural computation process (see also Fig. (A). B. Full graphical illustration of the computational operations. C. Separate illustrations of the prior, encoder inference, decoder generation, and hidden state update operations. With the introduction of the hidden state (acts as the sensory memory), our prior, encoder and decoder are linked to the entire stimuli sequence Xı;; rather than just the current stimulus x, (see also Appendix 33T.. Hence, although we completely exclude the time dimension from the model parameter space, TeCoS-LVM can still adaptively explore and exploit temporal information for predictive (generative) modeling.
Deep variational IB 32 33 leverages variational inference to construct a lower bound the IB objective in (2). By assuming the factorization p(x.y. y, z) = p(xx)))()))))(x), we have an equivalent objective that comprises one predictive term and compressive term (refer to Appendix D as follows,
min Bgtax:00| log p(ylz; 0)) +B
KL [g(l/x: 0)lppz))
g encouraging predictive power
grorp: encouraging compeessoo
where KL[Q|IP] is the Kullback-Leibler divergence between two distributions, and p(y|z: 0) is a parameterized decoder.
3 Methodologies
3.1 Temporal Conditioning Spiking Latent Variable Model
Basic formulation (see also Appendix D We denote a sequence of visual stimuli as (X).-..,,, ........ .x) [xdim(xx) dim[x,] stands for the dimension of x, At each timestep f, one high- dimensional visual stimulus is received. We want to predict the corresponding neural population response y; E (0, 1) dimily, where dim[y.] denotes the number of retinal ganglion cells (RGCs). This is implemented by an LVM which first compresses the visual stimuli into a low-dimensional latent representation z, E didi(e)) and then decodes the neural population response from it. Biological neural coding effectively compress observed stimuli while retaining the informative contents 34 33 36 37. Therefore, we further encourage this LVM to construct a latent space in which z have maximal predictive power regarding response y while being maximally compressive about input stimuli x (38] As a result, our target to model the neural coding process of visual stimuli turns to an optimization problem of minimizing the loss function presented in the variational IB framework.
However, assuming independence along the temporal dimension becomes inappropriate due to complex temporal dependencies in stimuli-neural response modeling [39
40 41 To address this, we introduce a hidden state into the prior, encoder, and decoder of the latent variable model. This
hidden state maintains earlier stimuli information 42 43), allowing the entire inference-generation

---

process to be conditioned on the entire stimuli sequence X1n rather than just (Fig. 5 To enable adaptive exploitation and accumulation of temporal dependencies within the stimuli sequence, we employ recurrent network to update the hidden state [4
45 46 47 formally,
h = FRNN(X, h
(4)
The hidden state here serves as the sensory memory 48 akin to the Tolman-Fiichenbamm Machine 49). which employs attractor network to store and retrieve memories.
Note that, as we focus on nigh-diimnsiooal visual stimuli here, we use a spiking convolutional feature extractor to reduce the dimensionality of the stimuli. It is shared in the hidden state update and the temporal conditioning encoder inference process (as illustrated in Fig. D. To make our notation simpler and easier to understand during reading, our formulations (stimuli features for de facto computation, but still marked as x).
do not explicitly denote the feature extractor in
Temporal conditioning prior With the introduction of the hidden state, the prior over latent variable no longer a standard isotropic Gaussian. Instead, it becomes a parameterized conditional distribution that depends on the hidden state, which carries information about previous stimuli. Formally, the latent variable follows the distribution p(zz(X)=-1), given by
p(z,Xta-1)== = N(zi: ( grrior diag( gatior
Meun p
Variance
where ø denotes the parametric model to compute the means and variances, spiking MLP. In particular, the real-valued means and variations are calculated through the linear readout synapses of spiking neurons.
Temporal conditioning encoder In a similar manner, the encoder is not only a function of x, but also function of h By Eq. 4 the hidden state h,-1 is function of X1:-1- Hence, the temporal conditioning encoder defines the distribution ((z/)x:)) We can write
g(z/\11)) = N(z
"/"""""", "/ 1,-1), diag(u"" h-1))).
where t/ is the parameterized model for computing the variational posterior distribution, which is also a spiking MLP.
Temporal conditioning decoder The decoder generates the neural population responses only using the latent representation z, and hidden state h as inputs. As the hidden state is function of previous stimuli, the decoder defines the distribution Plyılz,,Xır-1 gile), where y stands for the decoder parameters. Since we use spiking neurons, the decoder will directly output spike trains that simulate the recorded neural population responses.
Model learning The TeCoS-LVM models are optimized to minimize loss function (Eq. also Appendix D) that has the form:
f + comm.
and all synaptic weights are optimized jointly during the learning. To elucidate the loss function, we first examine the compressive term. As we adopt Gaussian distributions in our temporal conditioning prior and encoder, the compressive term composed of KL divergences can be calculated analytically. In particular, we have,
T comp Σ
(8)
On the other hand, since our model directly outputs spike trains, we adopt the Maximum Mean Discrepancy (MMD) as the predictive loss term [26] 50]. Also, use the first-order postsynaptic potential (PSP) kernel that can effectively depict the temporal dependencies spike train data [5]] 52 Denoting the predicted, recorded spike trains as &. y, respectively, we can write the PSP kernel MMD predictive loss as
pred
PSP()).:> PSP(y ΣΣ
T
where PSP(Y = (1 1)PPPP().--) Hy.. and set the time constant T, = 2.

---

3.2 TeCoS-LVM Models
We shall consider two types of TeCoS-LVM models, TeCoS-LVM and TeCoS-LVM Noisy in this work. Specifically:
In TeCoS-LVM, all spiking neurons are LIF neurons.
In TeCoS-LVM Noisy, all spiking neurons Noisy LIF spiking neurons (153) refer to Appendix
for details). Using Noisy LIF neuron allows TeCoS-LVM, to have neuron-levdl stochasticity,
which is considered crucial component in biological neural computation [54 55 56
Experiments
4.1 Dataset and Baselines
We perform evaluations and analyses on real neural recordings from RGCs of dark-adapted axolotl salamander retinas [57]]. The dataset contains spike responses of two retinas on two movies (see also Appendix ED. We partitioned all records into response sample pairs of second (30 time bins) and down-sampled the frames to 90 pixelx90 pixel. This results in four datasets, each split into non-overlapping train/test (50%%50%) parts. We shall refer to these four datasets as follows for brevity. Movie Retina 1: records of 38 RGCs movie 1 ("salamander movie"), 75 repetitions. Movie Retina2 2: records of 49 RGCs on movie 1, 30 repetitions. Movie Retina records of 38 RGCs on movie 2 ("wildlife movie"), 107 repetitions. Movie 2 Retina 2: records of 49 RGCs on movie 2, 42 repetitions.
CNN model We the fott--of-thearr CNN model 9 13 for comparison. In the CNN model, the network's s predicted outputs are the average maximum likelihood estimation retina responses in firing rates based on spatiotemporal (frames are concatenated on the channel dimension) input (see also Appendix G
IB-Disjoint IB-Disjoint (2 high-performane model that performs similarly to the IB-GP model that uses Gaussian Process prior (21]. This model employs an isotropic Gaussian the prior over latents. is similar the vanilla variational IB, where the latent variables are assumed to be independent along the temporal dimension.
4.2 Metries and Features for Evaluations and Visualizations
Pearson correlation coefficient This metric evaluates the model performance by calculating the Pearson correlation coefficient between the recorded and predicted firing rates [9] 14. 15 The higher the value, the better the performance. Refer Appendix[F]te for more details.
Spike train dissimilarity This metric assesses the model performance by computing the dissimilar- ity between recorded and predicted spike trains. lower value indicates better model performance. Here we use the MMD with a first-order PSP kernel [51, 52 to measure the spike train dissimilarity (see also Appendix F.
27 50 58)
Spike autocorrelogram The spike autocorrelogram is computed by counting the number of spikes that occur around each spike within a predefined time window [4. 9. The resulting trace is then normalized to its maximum value (which occurs at the origin of the time axis by construction), and the maximum value is set to zero for better visualization. Refer to Appendix F for more details.
4.3 Experimental Details
TeCoS-LVM hyper-parameters were fixed to be the same all four datasets. We set the latent variable dimension to 32 and the hidden state dimension to 64 by default (see also Appendix (A). We used the Adam optimizer (B) 0.9,82 0.999) with cosine-decay learning rate scheduler [59], starting at a rate of 0.0003. The mini-bact batih size was set to 64, and the models were trained for 64 epochs. We used the same architectures to implement TeCoS-LVM models. The factor in the loss function
7 is set to 0.01 by default to balance information compression and predictive power 32 60]. More experimental details are presented in Appendix G
At test time, the test samples have the same length as the training samples (30 bins, second) by default. However, the learned model can handle longer test samples as our model excludes the time dimension from its parameter space. This will be discussed further in the Results sub-section. We tested TeCoS-LVM models with "warmup period" of 0.5 seconds (15 bins), during which the
5

---

model's predictions will be discarded (see also Fig. AA). This was done because our hidden state was vero-initialized, making the early predictions less accurate.
long stimuli sequance
Swaght
TECOS-DVM (-0.21M paramo)
(-0 peramn)
split
TECOS-LVM Nokey (-0.1M paramss)
IB-Disjoint (-O 450 params)
Baseline Bellene
sequences
predict (eeparately)
single
concet
Pearson Corelation
0.76
0.9
0.0
0.91
firing
lenglhh fanut pre-diined the model learnin phoce
0.56-
Bio coding aircult-
long simull
long stimuli
TeCoB-LVA
king stimuli sequenies
0.36
0.3
0.2
0.3
sequeences
predict
D
Retina
Retina
sequaance
0.16
0.20
fers seg, lengthh" Beentble cam any lengih
0.12
D.
B
0.12
Movie
0.08
Data
0.00
LAM
TECOS. TECOS-
LWM LVM Noey
-0.0
6:00
20
033
0.20
Namalladd numttrr tumter ne bes
0.16-
0.12
Movie
ONN
0.04
1B- Digjoint
0.00
0.00
-0.30
0.50
Example neuron
Example neuron
Brng siee
farggrat (he)
20
Data
Data
TECOS-LVM Noiey
TECOS-LVM Noisy
TECOS-LVM
TECOS-LVM
Figure 2: TcCoS-LVM models well fit the spike statistics real neural activities. The error bars (SD) were computed across multiple random seeds. A. Graphical illustrations of the processing flows of baselines and LVM models. B. Heatmaps of the real (Data) and predicted firing rates on Movie 2 Retina 2 test data. The TeCoS-LVE Noisy model precisely reproduces the firing rate patterns of real neural data. C. Histograms of firing rate Pearson correlation coefficients on test data of four datasets. D. Autocorrelograms acquired the TeCoS-LVM models accurately capture the spike autocorrelations. E. Recorded and predicted activities of representative neurons responding repeated trials of randomly selected test. segment of Movie Retina I data.
data of four datasets. While the baselines fail,
4.4 Results
4.4.1
TeCoS-LVM Models Accurately Fit Real Spike Activities and Statistics
As shown in Fig. pC. TeCoS-LVM models can effectively fit the recorded firing rates. Especially, TeCoS-LVM Noisy can reproduce real firing rates more accurately than baselines. Furthermore, we verified the potential information loss caused by using trial-averaged statistics as the optimization target. While the firing rate target may allow models to learn coarse-grained features, it does not necessarily yield optimal capturing of fine-grained features such as spike autocorrelations. As shown by the autocorrelograms in Fig. 2D, the basclines failed to capture the spike autocorrelation

---

0.9
B
TECOS LVM Nolyy
Dispoin
Dala
Mov1 Ret1
Mov1 Re12
Mov2 Ret1
Mov2
Spite train dissin
TE-COS LVM Nakey
1060
400
C
TeCo6-LVM
TeCoE-LVM Notay
TECOS. LVM
Mov1 Ret1
Ret2
Mov2 Ret1
Mov2 Ret2
Spoko toun disimm Plarson Constation
0.72
0.88
0.00
0.84
0.54
ONN
1.G.7
400
Dispoint
840
340
800
560
985
1
10
3
1367 10
135.7 10
seg, length
Figure 3: TeCoS-LVM models synthesize realistic neural activities and generalize well to larger time scales. A. Examples of the predicted spike trains on a Movie 1 Retina 2 test data clip. The spike trains generated by the TeCoS-LVM, models are closer to the recorded spike trains than those of the baselines. B. Histograms spike train dissimilarities on test data. The dissimilarities acquired using TeCoS-LVM models are significantly lower than those of baselines. C. TeCoS-LVV models learn general temporal dependencies. We ran TeCoS-LVM models trained using 1-second sequences on longer test sequences and observed that extending the length of the test sequence only leads to minor performance drops.
feature accurately. By contrast, TeCoS-LVM models reproduced the spike autocorrelations more precisely. The TeCoS-LVM models also outperformed other models in synthesizing more realistic spike activities (Fig. BA&BB. Our results suggest that TeCoS-LVM, models accurately fit real spike activitics and statistics. In particular, the TeCoS-LVM Noisy model significantly outperforms baselines all metrics.
We also noticed that including neuronal noise is important for modeling neural activities. Because of the absence of neuron-level randomness, the TeCoS-LVV model failed to reproduce trial-to-trial variability (Fig. E) in real neurons. Consequently, as shown in Fig. _B, TeCos-LVM's firing rate predictions (obtained by averaging over multiple runs) are less smooth than others, thus negatively impacting the firing rate correlations. That explains why the TeCoS-LVM achieved lower spike train dissimilarities some data (Movie 1 Retina 2, Movie 2 Retina in Fig. 3) but still lads behind other methods in terms of the firing rate metric (Fig. (C).
Evaluation using more spike distances The TeCoS-LVM models are specifically optimized to minimize the discrepancy the PSP kernel space, making them naturally advantageous in comparing DSP-MMD-aasee spike train dissimilarity. To ensure a more fair assessment of the spike train prediction quality, we have considered several different spike train distances. Specifically, we computed van Rossum, Victor-Purpura, and SPIKE distances (refer to Appendix F for more details) between the predicted spike trains with the recorded ones. We observed consistent and significant improvements in the TeCoS-LVM models over two state-of-the-an basclines, as shown by results in Table [ We also noticed that when considering these spike train distances, the performance of TeCoS-LVM Noisy is slightly affected due to the worse-perturbed neuronal dynamics, which accords with results in Fig. BB. However, the overall performance of TeCoS-LVM Noisy surpasses the baselines by large margin and, moreover, effectively reproduces the variability in neural processing.
4.4.2 Train Short, Test Long: Learned TeCoS-LVM Models Generalize to Longer Time Scales
TeCoS-LVM models completely exclude the time dimension from parameter space and, therefore, can handle input stimuli sequences of any duration without being limited to the length of training data (I second here). We evaluated the temporal scalability of TeCoS-LVM models by running them
7

---

longer test data sequences. The firing rate correlation coefficients and spike train dissimilarities were calculated measure the performance changes. TcCoS-LVM models consistently produce accurate coding results at different time scales. Results in Fig. 3 show that increasing the length of the test sequence only brings slight performance drops, as evidenced by the minor decrease in the firing rate correlations and little increase in the spike train dissimilarities. This demonstrates that TeCoS-LVM models learn general (multi-time-seor simelss temporal dependencies from seort-sequence training.
Table 1:
Evaluation results using SPIKE, fictor-Purpura, and van Rossum spike train distances, results reported here are averaged across multiple trials.
DISTANCE TYPE
SPIKE
Victor-Purpura van Rossum
SPIKE
V.-.
van Rossum
MODEL.
DATA
MovIEl RETINAl
MOVIE RETINA2
TeCoS-LVM Noisy (This work)
0.155
14.024
238.614
0.116
21.599
425.871
TeCoS-LVX (This work)
0.124
12.835
127.346
0.111
18.182
150,445
McIntosh NeurlPS- 16
0.207
19.601
376.822
0.220
39.168
2672.211
Rahmani NeurlPS-22 22
0.224
21,916
394.020
0.219
39.075
2276.706
MODEI.
DATA
MOVIE2 RETINAl
MOVIE2 RETINA2
TeCOS-LVM Noisy (This work)
0.162
14,412
553.510
0.153
28. 18.411
1135,805
TeCoS-IVV (This work)
0.128
12.693
308.
0.123
22.666
574,298
McIntosh SeurlPS-Ig 16
0.212
22.713
1650.823
0.221
39.261
2638.965
Rahmani NeurlPS-22
0.204
22.271
1615.934
0.221
38.378
2244.981
A
reallime hidden nor hiddann norn on long sequence
D
TECO9-UM
TECOBLVM
TRCOS4 Voley
TECDSAVMM Nolky
1b,
Poemaa Comeattion
Paunson Cunsltion
Spiia tran dasim.
1200
Hamepp p0700
16
0.7
500
1.00
0.20
0.54
500
0.5
Atarcaer TenO
REDostac TECO
MMedeed
TEGO
TECOS-VM
0.25
0
OR Bar
0.00
0.18
TECOS-LYM
E
decressed compressvv regutaization wide into bettlenec
nammy Info botteneck
Incressed term Lntizaattio
0.00
tiny 1=11-4
mocerate 1=1e2
large 10114
Time
moan
875
150
6:0
133
TECOS-VV
Naay
varience
15
Doenedat dal.
07 077:99
0,000
10
0.7-
0.1
model output
0.1
0.5
0.8
0.5
05
08
Figure 4: Visualizations hidden state and latent space dynamics. A. The evolution of ||h,]l over time after initialization. B. SSIM curve of neighboring stimuli and cosine distance curves of adjacent hidden state vectors of TcCoS-LVM models. C. Scatterplots of the Pearson correlation coefficients between hidden state cosine distance and stimuli SSIM. D. Model performances under different weighting factor fa, obtained on Movie Retina 2 test data. E. Visualizations of z, dynamics in TeCoS-LVM Noisy models learned using different levels of B. F. Model performances with and without TeCo (shorthand of temporal conditioning operation). We increased the number of parameters for models without TeCo to be approximately on par with the standard models.
4.4.3 Hidden State Analyses: Exploring Memory Mechanisms in TeCoS-LVMS
Interestingly, TeCoS-LVM, models' predictions are less accurate for that short period after initializa- tion, likely due to the model's 's sensory memory requiring time to accumulate. We observed that it takes at most 0.5 seconds for the L2 norm of the hidden state reach the average value of consecutive runs on long sequence (Fig. ()) As result, we set a warmup period of 0.5 seconds for all our tests. We further investigated the memory mechanism in TeCoS-LVM models, which implemented by the hidden state update. To this end, we quantified the variation of stimuli by computing the Structural Similarity (SSIM) of adjacent stimuli (x, and x,-)). If the change stimuli at a certain moment is drastic, then the SSIM at this moment is low. For the sensory system, the stimulus it receives that time is quite novel or surprising [T]. We also measured the memory update magnitude by calculating the cosine distance of adjacent hidden state vectors (h, and h-1)). As shown in Fig. 4B.

---

C, the two have strong negative correlation. This suggests that the TeCoS-LVM models perform more significant memory updates (indicated by a large cosine distance) when facing highly dynamic stimuli (low SSIM); only minor updates are performed when stimuli are relatively stable. This is consistent with previous sensory neuroscience findings suggesting that the sensory circuits focus more on unpredictable or surprising events. When repeatedly exposed to an initially novel stimulus, the neural processing becomes less active [62 63 64].
4.4.4 Latent Space Dynamic Analyses
We next explored the latent variables under different weighting factor / settings. Setting a proper value can lead to richer latent variable dynamics. is more advantageous to show how they relate to factors like anatomy and behavior through correlation analysis, thereby interpreting the inferred latent variables. The compressive loss term forces the latent variable to act like a minimal sufficient statistic of stimuli for predicting neural responses. Therefore, tiny / values correspond weak regularization provided by the compressive term. In this case, the latent representation learns to be more deterministic (Fig. [E-Left) This prevents the model from benefiting from the regularization brought by the compressive loss term, resulting in poorer performance compared to moderate B values (Fig. 4D). When using large B, we observed that the latent variables seem to have lost some temporal information in the stimuli (Fig. -iRRggh) And, if ß is further increased, the model cannot obtain enough stimuli information predict the responses. resulting in a sharp decline in performance (Fig. 4D).
4.4.5 Functional Implication Analyses by In-silico Ablating
Temporal conditioning operation We then evaluated the performance of TeCoS- models without temporal conditioning operations, which is a reduced variant with isotropic Gaussian prior (see also Appendix Fig. 5). Results in Fig. 4 show that the TeCo operation significantly improves model performance by exploiting the temporal dependencies. This suggests that visual coding requires the integration and manipulation of temporally dispersed information from continuous streams of stimuli.
Spiking hidden neurons We conducted experiments to investigate the effect of employing spiking neurons in modeling neural activity to natural stimuli (see also Appendix HD. Specifically, we replaced all spiking hidden neurons in the TeCoS-LVM models with LIF-Rate neurons [65] while retaining only the output spiking neurons that the training pipeline remains unchanged. The LIF-Rate neurons maintain the neuronal recurrency by using the same membrane potential update as LIF and Noisy LIF neurons. This allows us to directly assess the effect of using spiking neurons by ablating them. Our experimental results show that the use of spiking neurons can significantly enhance the performance of computational models. As can be seen from Table all performance indicators have significantly decreased after replacing the spiking hidden units with rate ones. Previous studies have pointed out that natural stimuli have sparse latent structure [66]: therefore, using spiking hidden units may better fit the latent structure of natural stimuli by utilizing sparse spike representation. Also, previous research (67] indicated that the coding method of SNNs can lead to highly competitive results, which is consistent with the conclusion of this part of our experiment.
Table Ablation results of using spiking hidden neurons. An indicates that the higher the value, the better, while J suggests the opposite. Results reported are averaged across multiple trials.
Spiking
CC (1)
Spike Train
SPIKE
Victor Vietor-Purpura
van Rossum
hidden units
Dissim. ())
(L)
(L)
())
Mov1 Real
TeCoS-LVM
Yes
No
0.579
0.254
371.057 850.418
0.259
0.124
45.117
12.835
127.346 3416.557
TeCoS- Noisy
Yes
0.728
354.989
0.155
14.024
238.614
No
0.653
370.099
0.167
14.805
291.706
Mov2 Rar2
TeCoS-LVM
Yes
0.616
1003.489
0.123
22.666
574.298
No
0.471
1273.267
0.180
35.080
1890.910
TeCoS-LVM Noisy
Yes
No
0.822
0.748
1078.830
1021 384
0.159
0.153
29.249
28 28.441
1135.80 805
1144.087
5 Related works
This work builds on previous research on modeling neural responses to visual scenes. Early attempts included Linear-nonlinear (LN) [68 69) and Generalized Linear Models (GLMs) 7.77. However,

---

these models have limited capabilitiees and fail when faced with more complex stimuli g 9]. A promising way is leverage powerful neural networks. One attractive advantage of this approach is that it may eliminate the need to specify spike statistics explicitly. McIntosh et al. [9] proposed convolutional neural network (CNN) approach that outprroormdd IN and GL.M baselines by large margin. Some researchers have also investigated CNN variants with a recurrent layer [9 Batty et al. [12 proposed a hybrid model that combines GLMs and recurrent neural networks (RNNs) to separate spatial and temporal processing components in neural coding, Our model also has specified structures that deal with the time dimension, but the temporal and spatial processing are tightly integrated. Generative adversarial networks (GANs) have also been used to synthesize realistic neural activities [14]. but this method cannot be used to predict neural responses given the stimuli. Mahuas et al. 770 introduced novel two-step strategy to improve the basic GLMs. Bellec et al. 71 introduced a model with spiking response model (SRM) neurons and proposed a sample and measure criteria in addition to the traditional Poisson likelihood objective. More recently, Rahmani et al. [2 introduced the Gaussian Process prior and variational information bottleneck to LVMs for modeling retinal responses. Compared biological coding systems, these methods have two aspects of authenticity missing (Fig. [ Firstly, most of these methods target trial-averaged firing rates. Secondly, they cannot directly process long stimuli sequences in a natural paradigm but must divide them into segments of pre-defined lengths to be processed separately.
Latent variable models LVMs are popular tools for investigating and modeling neural response 11 19 72). Some studies suggest that -iw-dimensional latent factors can effectively represent nigh-dimensional neural activities [16 [7). In recent works, LVMs demonstrated promising results in uncovering the low-dimensional structure underlying complex neural activities in the motor region
20 23 22].
Also, LVMs have shown promise in visual neural coding modeling [8] 19 21.
Conditioning Conditioning is an effective technique to integrate information from multiple sources and realize adaptive information gating. Typically, conditioning methods rely on external information sources like labels 73] or outputs from other models 774 75). Some studies have demonstrated that direct conditioning on previous states can also significantly improve performance, especially in a sequential processing regime [49 76]; these approaches are often termed self-conditioning or temporal conditioning. Notably, the temporal conditioning design exposed here is loosely inspired Machine [49], where the conditional operations ensure that the model profits from sequential learning paradigm.
by the
6 Discussion
This work presents the TeCoS-LVM (temporal conditioning spiking LVM) models. Our approach is formalized within the information bottleneck framework of latent variable models, inspired by the efficient coding theory 34 38 TeCoS-LVM models can directly produce neural response sequences in real-time rather than repeatedly producing single-step predictions, thus faithfully simulating the biological coding process. Using the retinal response to natural scenes as an example, we showed that TeCoS-LVM models effectively fit spike statisties, spike features, and recorded spike trains. In particular, TeCOS-LVM models that incorporate Noisy LIF neurons significantly exceed high- performance baselines. Also, learned TeCoS-LVM models generalize well to longer time scales, indicating that they can learn general temporal features from short-sequence training. Overall, TeCoS-LVM models effectively capture key features of biological coding systems while remaining computationally tractable.
Outlook Although we use visual coding as an example to demonstrate the impressive performance of TeCoS-LVM models in this article, the models exposed here can be easily applied to sensory data of other modalities. As such, TcCoS-LVM demonstrates a promising framework for building computational accounts for various sensory neural circuits and will enable richer models of complex neural computations in the brain.
Limitations As an accurate model that relates stimulus-driven responses, from a neuroconnec- tionsim perspective (77). the TeCoS-LVAs model aims to provide neuroscientific insights and un- derstandings at the computational level. Compared to previous endeavors, TeCoS-LVM takes spike-based neural computation and natural processing paradigms into account. Nonetheless, while the TeCoS-LVV model makes predictions at the level of retinal cells, its fundamental principles are at computational level, so it is only partially biophysically realistic.
10

---

Acknowledgement
This work is supported by the National Key Research and Development Program of China under Grant 2020AAA0105900 and the National Natural Science Foundation of China under Grant 62236007,
Grant U2030204.
The authors are grateful for the generous support from Professor Arno Onken from the University of Edinburgh. The authors would also like acknowledge anonymous reviewers and chairs of NeurIPS 2023 for providing insightful comments to help improve this work.
References
[1] Tim Gollisch and Markus Meister. Eye smarter than scientists believed: neural computations in circuits of the retina. Neuron, 150-144 2010.
[2] Jonathan W Pillow, Jonathon Shlens. Liam Paninski, Alexander Sher, Alan M Litke, EJ Chichilnisky, and Eero P Simoncelli. Spatio-temporal correlations and visual signalling in complete neuronal population. Nature, 454(7207):9955999) 2008
[3] David B Kastner and Stephen Baccus. Coordinated dynamic encoding in the retina using opposing forms of plasticity. Nature neuroscience, 14(10):1317-132) 2011.
[4] Joseph J Atick and A Norman Redlich. Towards theory of early visual processing. Neural computation, 2(3):308-2 -00000 1990.
[5] Bence P Ölveczky, Stephen A Baccus, and Markus Meister. Segregation of object and back- ground motion the retina. Nature, 423(6938):401-4005 2003.
[6] Eric Y. Wang, Paul G. Fahey, Kayla Ponder, Zhuokun Ding, Andersen Chang, Taliah Muham- mad, Saumil Patel, Zhiwei Ding, Dat Tran, Jiakun Fu, Stelios Papadopoulos, Katrin Franke, Alexander S. Ecker, Jacob Reimer, Xaq Pitkow, Fabian H. Sinz, and Andreas S. Tolias. Towards a foundation model of the mouse visual cortex. bioRxiv, 2023. doi: 00.1101/2023.33.2..3
[7] Jonathan W Pillow, Liam Paninski, Valerie J Uzzell, Eero P Simoncelli, and EJ Chichilnisky. Prediction and decoding retinal ganglion cell responses with a probabilistic spiking model. Journal of Neuroscience, 25(47): (0047)):1103-1100000 11013, 2005.
[8] Alexander Heitman, Nora Brackbill, Martin Greschner, Alexander Sher, Alan Litke, and EJ Chichilnisky. Testing pseudo-lineat models of responses to natural scenes in primate retina. BioRxiv, 2016.
[9] Lane McIntosh, Niru Maheswaranathan, Aran Nayebi, Surya Ganguli, and Stephen Baccus. Deep learning models of the retinal response to natural scenes. NeurIPS, 2016.
[10] Nikolaus Kriegeskorte. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annual review of vision science, 1:417-446, 2015.
[11] Guangyu Robert Yang and Xiao-Jing Wang. Artificial neural networks for neuroscientists: a primer. Neuron, 10486):044811077 2020.
[12] Eleanor Batty, Josh Merel, Nora Brackbill, Alexander Heitman, Alexander Sher, Alan Litke, E.J. Chichilnisky, and Liam Paninski. Multilayer recurrent network models of primate retinal ganglion cell responses. In ICLR, 2017.
[13] Fabian Sinz, Alexander S Ecker, Paul Fahey, Edgar Walker, Erick Cobos, Emmanouil Froudarakis, Dimitri Yatsenko, Zachary Pitkow, Jacob Reimer, and Andreas Tolias. Stim- ulus domain transfer in recurrent models for large scale cortical population prediction video. In Advances in neural information processing systems, volume 31, 2018.
[14] Manuel Molano-Mazon, Arno Onken, Eugenio Piasini*, and Stefano Panzeri*. Synthesizing realistic neural population activity patterns using generative adversarial networks. ICLR, 2018.
11

---

[15] Yajing Zheng, Shanshan Jia, Zhaofei Yu, Jian K Liu, and Tiejun Huang. Unraveling neural coding of dynamic natural visual scenes via convolutional recurrent neural networks. Patters, 2(10): 2021.
[16] Patrick T Sadtler, Kristin M Quick, Matthew D Golub, Steven M Chase, Stephen I Ryu, Elizabeth C Syyle-Kabbra, Byron M Yu, and Aaron P Batista. Neural constraints on learning. Nature, 5 512(7515):423 426,, 2014.
[17] Gamaleldin F Elsayed and John P Cunningham. Structure in neural population recordings: an expected byproduct of simpler phenomena? Nature neuroscience, 20(9):131--111 2017.
[18] Jakob H Macke, Philipp Berens, Alexander S Ecker, Andreas S Tolias, and Matthias Bethge. Generating spike trains with specified correlation coefficients. Neural computation, 21(2): 397-423, 2009.
[19] Dmitry R Lyamzin, Jakob H Macke, and Nicholas A Lesica Modeling population spike trains with specified vare-narying spike rates, trial-to-trial variability, and pairwise signal and noise correlations. Frontiers computational neuroscience, 4: 144, 2010.
[20] Chethan Pandarinath, Daniel J O' Shea, Jasmine Collins, Rafal Jozefowicz, Sergey D Stavisky, Jonathan C Kao, Eric M Trautmann, Matthew Kaufman, Stephen Ryu, Leigh Hochberg, al. Inferring single-trial neural population dynamics using sequential auto-encoders. Nature methods, 15(10):805-815 2018.
[21] Babak Rahmani, Demetri Psaltis, and Christophe Moser. Natural image synthesis for the retina with variational information bottleneck representation. In NewrlPS, volume 35, pages 6034-6046, 2022.
[22] Ran Liu, Mehdi Azabou, Max Dabagia, Chi-Heng Lin, Mohammad Gheshlaghi Azar, Keith Hengen, Michal Valko, and Eva Dyer. Drop, swap, and generate: A self-supervised approach for generating neural activity. In NeurIPS, volume 34, pages 10587-10599, 2021.
[23] Ding Zhou and Xue-Xin Wei. Learning identifiable and interpretable latent models of high- dimensional neural activity using pi-vae. In NeurIPS, volume 33, pages 7234-7247, 2020.
[24] Siwei Wang, Benjamin Hoshal, Elizabeth A de Laittre, Olivier Marre, Michael Berry, and Stephanie Palmer. Learning low-dimensional generalizable natural features from retina using a u-net. In NeurIPS, 2022.
[25] Wulfram Gerstner, Werner M Kistler, Richard Naud, and Liam Paninski. Neuronal dynamics: From single neurons to networks and models of cognition. Cambridge University Press, 2014.
[26] Memming Park, Sohan Seth, Murali Rao, and Jose C Principe. Strictly positive-definite spike train kernels for
divergences. Neural computation, 24(8):2223-22500 2012.
[27] I1 Memming Park, Sohan Seth, Antonio RC Paiva, Lin Li, and Jose C Principe. Kernel methods on spike train space for neuroscience: tutorial. IEEE Signal Processing Magazine, 30(4): 149-160, 2013.
[28] Yujie Wu, Lei Deng, Guoqi Li, and et al. Spatio-temporal backpropagation for training high- performance spiking neural networks. Frontiers in Neuroscience, 12:1-12, 2018
[29] Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):11-61 2019.
[30] Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Leg- enstein, and Wolfgang Maass. A solution to the learning dilemma for recurrent networks of spiking neurons. Nature communications, 11(1):3625, 2020.
[31] Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 1999.
[32] Alexander Alemi, Ian Fischer, Joshua Dillon, and Kevin Murphy. Deep variational information bottleneck. In ICLR, 2017.
12

---

[33] Matthew Chalk, Olivier Marre, and Gasper Tkacik. Relevant sparse codes with variational information bottleneck. In NeurIPS, 2016.
[34] Alexander Borst and Frédéric Theunissen. Information theory and neural coding. Nature neuroscience, 2(11):947-95)) 1999.
[35] Markus Meister and Michael Berry. The neural code of the retina. Neuron, 22(3):435 450, 1999.
[36] Andreas Nieder and Earl K Miller. Coding of cognitive magnitude: Compressed scaling of numerical information in the primate prefrontal cortex. Neuron, 37(1):149-13) 157, 2003.
[37] Juan A Gallego, Matthew G Perich, Lee E Miller, and Sara A Solla. Neural manifolds for the control of movement. Neuron, 94(5):978-984 2017.
[38] Matthew Chalk, Olivier Marre, and Gašper Tkačik. Toward a unified theory of efficient, predictive, and sparse coding. Proceedings of the National Academy of Sciences, 115(1): 186-191, 2018. doi: (0.107//paa..771111111
[39] Andreas K Engel, Peter König, Andreas K Kreiter, Thomas B Schillen, and Wolf Singer. Temporal coding in the visual cortex: new vistas on integration in the nervous system. Trends in neurosciences, 15(6):218-226 1992.
[40] Pamela Reinagel and R Clay Reid. Temporal coding of visual information in the thalamus. Journal of neuroscience, 20(14)55992-5400, 2000.
[41] Michele Rucci, Ehud Ahissar, and David Burr. Temporal coding of visual space. Trends in cognitive sciences, 22(10):883 895, 2018.
[42] Nicolas Boulanger-Lecaaddowski Yoshua Bengio, and Pascal Vincent. Modeling temporal dependencies in high-dimensional sequences: application to polyphonic music generation and transcription. In ICML, pages 1881-1888, 2012.
[43] Justin Bayer and Christian Osendorfer. Learning stochastic recurrent networks. In NeurIPS 2014 Workshop on Advances Variational Inference, 2014.
[44] Otto Fabius and Joost Van Amersfoort. Variational recurrent auto-encoders. arXiv preprint arXiv:1412.6587, 2014.
[45] Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In NeurlPS, volume 28, 2015.
[46] Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Rezende, and Daan Wierstra. Draw: A recurrent neural network for image generation. In JCML, pages 1462-1471. PMLR, 2015.
[47] Aston Zhang, Zachary C Lipton, Mu Li, and Alexander Smola. Dive into deep learning. arXiv preprint arXiv:2106.1 1342, 2021.
[48] George Sperling. The information available in brief visual presentations. Psychological monographs: General and applied, 74(11):1, 1960.
[49] James CR Whittington, Timothy H Muller, Shirley Mark, Guifen Chen, Caswell Barry, Neil Burgess, and Timothy EJ Behrens. The tolman-cichenbaun machine: unifying space and relational memory through generalization in the hippocampal formation. Cell, 18395): 1263, 2020.
[50] Diego Arribas, Yuan Zhao, and Il Memming Park. Rescuing neural spike train models from bad mle. NeurIPS, volume 33, pages 2293-2303, 2020.
[51] Friedemann Zenke and Surya Ganguli. Superspike: Supervised learning in multilayer spiking neural networks. Neural computation, 306)):514-1541, 2018.
[52] Wenrui Zhang and Peng Li. Temporal spike sequence learning via backpropagation for deep spiking neural networks. In NeurIPS, volume 33, pages 12022-12033, 2020.
13

---

[53] Gehua Ma, Rui Yan, and Huajin Tang. Exploiting noise resource for computation and learning in spiking neural networks. Patterns, 2023. doi: doi.org/10. 1tt///001116//// patto//.paterr.223....0 100833.
[54] Richard Kempter, Wulfram Gerstner, Leo Van Hemmen, and Hermann Wagner. Extracting os- cillations: Neuronal coincidence detection with noisy periodic spike input. Neural computation, 1089):187722077, 1998.
[55] Richard Stein, E Roderich Gossen, and Kelvin E. Jones. Neuronal variability: noise or part of the signal? Nature Reviews Neuroscience, 6(5):389-37,, 2005.
[56] Aldo Faisal, Luc PJ Selen, and Daniel M Wolpert. Noise in the nervous system. Nature Reviews Neuroscience, 94):2922-303, 2008.
[57] Arno Onken, Jian K Liu, PP Chamanthi R Karunasekara, Ioannis Delis, Tim Gollisch, and Stefano Panzeri. Using matrix and tensor factorizations for the single-trial analysis of population spike trains. PLoS computational biology, 12(1 1):e 189, 2016.
[58] Hiromichi Kamata, Yusuke Mukuta, and Tatsuya Harada. Fully spiking variational autoencoder. In AAAI, volume 36, pages 7059-7066, 2022.
[59] Iyya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In ICLR, 2017.
[60] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta- VAE: Learning basic visual concepts with a constrained variational framework. In ICLR, 2017.
[61] Laurent Itti and Pierre Baldi. Bayesian surprise attracts human attention. In Y. Weiss,
B. Schölkopf, and Platt, editors, NeurlPS, volume 18, 2005.
[62] Stelios M Smirnakis, Michael J Berry, David K Warland, William Bialek, and Markus Meister. Adaptation of retinal processing to image contrast and spatial scale. Nature, 386(6620);69-755 1997.
[63] Solange P Brown and Richard H Masland. Spatial scale and cellular substrate of contrast adaptation by retinal ganglion cells. Nature neuroscience, 4(1):44-51 51, 2001.
[64] Hadar Levi-Aharoni, Oren Shriki, and Naftali Tishby. Surprise response as probe for com- pressed memory states. PLOS Computational Biology, 16(2):1-21, 02 2020.
[65] Chloe N. Winston, Dana Mastrovito, Eric Shea-Browe and Stefan Mihalas. Heterogeneity in Neuronal Dynamies Is Learned by Gradient Descent for Temporal Processing Tasks. Neural Computation, 3554):555-592, 03 2023. doi: 1162///co_a_000
[66] Aapo Hyvärinen, Jarmo Hurri, and Patrick Hoyer. Natural image statistics: A probabilistic approach to early computational vision., volume 39. Springer Science & Business Media, 2009.
[67] Alan Jeffares, Qinghai Guo, Pontus Stenetorp. and Timoleon Moraitis. Spike-inspired rank coding for fast and accurate recurrent neural networks. In ICLR, 2022.
[68] EJ Chichilnisky. A simple white noise analysis of neuronal light responses. Network: computa- tion neural systems, 12(2):199, 2001.
[69] Justin Keat, Pamela Reinagel, R Clay Reid, and Markus Meister. Predicting every spike: model for the responses of visual neurons. Neuron, 30(3):803-817, 2001.
[70] Gabriel Mahuas, Giulio Isacchini, Olivier Marre, Ulisse Ferrari, and Thierry Mora. A new inference approach for training shallow and deep generalized linear models of noisy interacting neurons. In Advances neural information processing systems, volume 33, pages 5070-5080, 2020.
14

---

[71] Guillaume Bellec, Shuqi Wang, Alireza Modirshanechi, Johanni Brea, and Wulfram Gerstner. Fitting summary statisties of neural data with a differentiable spiking network simulator. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan, editors, NeurIPS, pages 18552-18563, 2021.
[72] Cole Hurwitz, Akash Srivastava, Kai Xu, Justin Jude, Matthew Perich, Lee Miller, and Matthias Hennig. Targeted neural dynamical modeling. In NeurlPS, volume 34, pages 2021.
[73] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, pages 8162-8171. PMLR, 2021.
[74] Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1 1-37))::33333300 20,-332,022 2022.
[75] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
[76] Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. In ICLR, 2023.
[77] Adrien Doerig, Rowan P Sommers, Katja Seeliger, Blake Richards, Jenann Ismael, Grace W Lindsay, Konrad P Kording, Talia Konkle, Marcel AJ Van Gerven, Nikolaus Kriegeskorte, et al. The neuroconnectionist research programme. Nature Reviews Neuroscience, pages 1-20, 2023.
[78] Friedemann Zenke and Tim P Vogels. The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. Neural Computation, 3394):899-925, 2021.
[79] Hans E Plesser and Wulfram Gerstner. Noise in integrate-and fire neurons: from stochastic input to escape rates. Neural Computation, 12(2):367-388 2000.
[80] Anthony N Burkitt. A review of the integrate-and and---ndd-i fire neuron model: I. homogeneous synaptic input. Biological Cybernetics, 95(1):1-19, 2006.
[81] O. E. Somndorff-Nielsee and Neil Shephard. Non Non-gaussian and some of their uses in financial economics. Journal of the Royal Statistical Society: Series (Statistical Methodology), 6372)1167- 2001.
models
[82] Wolfgang Maass. Noise as resource for computation and learning in networks of spiking neurons. Proceedings of the IEEE, 102(5):860-88) 2014.
[83]
Wolfgang Maass. On the computational power of noisy spiking neurons. In NeurlPS, 1995.
[84] Diederik Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
[85] Markus Meister, Jerome Pine, and Denis A Baylor. Multi-neurooall signals from the retina: acquisition and analysis. Journal of neuroscience methods, 51(1):95-106, 1994.
[86] Daniel Bölinger and Tim Gollisch. Closed-loop measurements of iso-response stimuli reveal dynamic nonlinear stimulus integration in the retina. Neuron, 73(2):333-344 2012.
[87] Mark CW van Rossum. A novel spike distance. Neural computation, 13(4):751-763, 2001.
[88] Jonathan D Victor and Keith P Purpura. Metric-space analysis of spike trains: theory, algorithms and application. Network: computation in neural systems, 8(2):127-164, 1997.
[89] Thomas Kreuz, Daniel Chicharro, Conor Houghton, Ralph G Andrzejak, and Florian Mormann. Monitoring spike train synchrony. Journal of neurophysiology, 1457--114771442, 2013.
15

---

Appendix of Temporal Conditioning Spiking Latent Variable Models of the Neural Response Natural Visual Scenes
A Hidden State and Latent Space Experiments
Temporal Conditioning (TeCo)
w/o TeCo
Adaptive accumulate utllize xxt
Loss temporal information
L-2
1-2
x-1
t-1
x
X
x
pnior
Z
prior 2-1
t-1
prior
ZE
Temporal connection
Spatial connection
Y1
t-1
Y
Figure 5: Graphical illustration of the temporal conditioning operation (TeCo). After completely excluding the temporal dimension from the model parameter space, we introduced the temporal conditioning operation to handle the temporal information. particular, this operation enables memory-dependent processing as in biological coding circuits.
A
TECOS-LVM
B
TECOS-LVM
TECOS-LVM Noisy
TECOS-LVM Noisy
Pearson Correlation
Spike train dissim.
1050
CNN bosolin: 1548 30
Pearson Correlation
Spike train dissim.
1100
CNN baselin: 1548.00
0.78
0.78
1050
CNN beeotine
1000
CNN baseline
0.68
0.68
1000
0.58
950
0.58
950
8
128
129
8y
8
latent variable dim.
hidden state dim.
Figure 6: Performances under different hidden state and latent space dimension settings on Movie Retina data. For hidden state experiments, the latent space dimension is set to 32. And for latent space experiments, the hidden state dimension is 64. The error bars (SD) were calculated via various random seeds.
To further study the sensitivity the choice of hyperparameters, evaluated the performance of TeCoS-LVM models under different hidden state and latent space dimensionality settings. The results, as shown in Appendix Fig. (6A, indicate that increasing the latent space dimension improved performance. Still. further increases (larger than 32) had little effect when the latent space dimension was large. This also suggests that the number latent factors for the visual stimuli coding task considered is not very large. On the other hand, increasing the hidden state dimension enhances sensory memory capacity and benefits temporal conditioning operations. As shown Appendix Fig. 6B, the performance does not increase significantly when hidden state dimensionality increases to certain extent (around 64). particular, although the spike train dissimilarity decreases, firing rate correlation score almost no longer increases. This is consistent with observation in the main text, namely, a lower spike train dissimilarity does not always indicate a higher firing rate correlation score (refer to Fig. 2 The results in Appendix Fig. 6B also indicate that the proposed temporal conditioning mechanism can effectively utilize sensory memory and achieve good results even when the hidden state dimension is limited.
B Surrogate Gradient Learning in Spiking Networks
conventional SNN Surrogate Gradient Learning (SGL., pseudo derivative, derivative approximation) 29]. the derivative of the firing function do/Ou is replaced by smooth function (pseudo derivative
16

---

function) SG to mesh with the backpropagation scheme 778. This ad-hoc technique in SNN research popular, particularly for large-scale networks. allows for compatibility with popular automatic differentiation packages such as Py PyTorch and TensorFlow, simplifying the implementation of SNNs. This surrogate gradient function can be a triangular, rectangular (gate), sigmoidal, or ERF function 78 In SGL., the gradient gi w.r.t. synaptic weights of layer / is calculated by
SGL: &1 =
SG - Vin
V Votn
(10)
Surrogate the exact derivative del" 10u)
where 1, m denotes neuron in in layer I, and L, is the instant loss value.
C Leveraging Noisy Spiking Neural Models
Here, we use the implementation in 53 to leverage the power of noisy spiking neural models. Spiking neurons with noisy neuronal dynamics have been extensively studied in prior literature extended them to larger networks by providing a general formularization and demonstrating their computational advantages theoretically and empirically. The
[25] Recent research of Ma et al. [
Noisy LIF presented here is based on previous works that use diffusive approximation 77 80 25 where the sub-threshold dynamic is described by the
process:
du dr
di
RI(I) di Tm
= -(u Mreset) + RI(1) + g(O, eq. du -(u Mreset)
+ odw,
(11)
Im
the white noise is a stochastic process, o is the amplitude of the noise and dW, are the increments of the Wiener process in di [25). As odWW are random variables drawn from zero-mean Gaussian, this formulation is directly applicable to discrete-time simulations. Specifically, using the Euler- Maruyama method, we get Gaussian noise term added on the right-hand side of the noise-free LIF dynamic. Without loss of generality, we extend the additive noise term in the discrete form to general continuous noise [81), the sub-threshold dynamic of Noisy LIF can be represented as:
Noisy LIF sub-threshold dynamic:
Tu, + 1, +
(12)
where I, is the input, the noise e is independently drawn from a known distribution and satisfies E[e] = and p(e) p(-e). The constant here combines the simulation timestep length and the real membrane decay Tm. which is a simplification when the timestep we cope with is fixed. This work considers the Gaussian noise e N(O, 0.2?).
The membrane potentials and spike outputs become random variables due to random noise injection. Leveraging noise as a medium, we naturally obtain the firing probability distribution of Noisy LIF based on the threshold firing mechanism [53]
P[frring time r] [M, +4++ Val
Ple 14 Vtnl F. Van)
Threshold-based firng
Cumulative Distribution Function definition
where F denotes the cumulative distribution function. Therefore, we have that,
1, with probability
F,(u; - Vih).
= 0. with probability
(13)
(1 F.
The expressions above exemplify how noise acts as resource for computation 82 Thereby, we can formulate the firing process of Noisy LIF as (53)
Noisy LIF probabilistic firing: Bernouli (F.(4, -va)),
(14)
Specifically, it relates previous literature noise escape models, in which the difference Vah governs the neuron firing probabilities [83 79 In addition, Noisy LIF employs the same resetting mechanism the LIF model.
C.1 Noise-Driven Learning in Networks of Noisy LIF Neurons
The Driven Learning (NDL) rule 53 networks of Noisy LIF neurons theoretically sound general form of Surrogate Gradient Learning. In particular, the gradient w.r.t to synaptic
17

---

we assume a factorization of the joint distribution as weights in layer / is computed by
Post-synaptic factor
NDL: &1
Σ IM1
Vo 1m
F' am -
6 C
(15)
Pro-synaptic factor
Global learning signal
where the superscript 1, denotes neuron in layer I, L' is the loss value. Here, the post-synaptic factor in NDL. calculated by the probability distribution function of the postsynaptic neuron's membrane potential noise.
As shown in Equation 13] NDL is well-compatible with the backpropagation computation paradigm wrap the inference and learning of Noisy
in trending libraries like PyTorch. Therefore, LIF neurons into a module. By replacing the original LIF neuron module (with Surrogate Gradient Learning) with the Noisy LIF module (with NDL), we can easily implement noisy spiking neural networks arbitrary architectures in a plug-and salug-and-plee manner. An example can be found at https //gi trithhu...u
D Derivation of the Optimization Objective of TeCoS-LVM Models
Basic formulation - from an efficient coding (38] perspective We denote sequence visual stimuli as x = (x))=1--7, where X, E R ddim(k], dim x,] stands for the dimension of x, Similarly, we denote neural population response (target) as (y,) {0, 1)7xxin(x]), where dim[y,] denotes the number of retinal ganglion cells (RGCs). At each timestep r, one high-dimensional visual stimulus x, received, and we want to predict the neural population response y,- This is implemented by LVM which first compresses the visual stimuli into low-dimensional latent representation then decodes the neural population response from it. Inspired by the information compression feature in the neural coding process [34
[Rdimiz) and
36 37 we further encourage this LVM to construct latent space in which have maximal predictive power regarding y while being maximally compressive about Therefore, our target to model the neural coding of visual stimuli turns into an optimization problem within the IB framework. Note that stimulus from other modalities can be processed in a similar vein, but here use the visual case example.
35
Following previous IB literature follows,
(16) namely, we assume a Markov chain y + x + z, which implies p(z)x.y) = p(z)x), indicating that latent representation cannot directly depend the target response y. Recall that, according the IB principle [31], our objective has the form
p(x, y, 2) p(z/x p)p())))p()) = p(xxs))(y)x)p(x),
IB objective: max[ I(z,y)
-BI(z, x) J.
(17)
Predictive term Compresiive term
The predictive term encourages predictive power, while the compressive term enforces information
compression. And it is equivalent to minimizing a loss function - I(z,, -l((z x).
Let us examine the predictive term l(z. y) first. The mutual information between z and y given by p(y.z)
I(z. y) dydzp(y, z) log
p(y)p(z)
(18)
p(ylz)
dydzp(y, log p(y)
According to the assumed Markov chain (16). the likelihood p(ylz) is defined by
dxp(ylx)p(x(2)> dxp(ylx) p(z)x)p(x)
p(ylz) dxp(xx
(19)
p(z)
and is approximated by variational decoder plylz; in our case. Given the fact that KL(p(ylz)lp((yzz de) 2 O, we can write
p(ylz) f dyp(ylz) p(ylz; sddee)
20
(20)
=f (og S
dyp(y'z) log plylz; loyy.va
18

---

Therefore,
f dydzp(y. Zo p(ylz; val))
y) 2
p(y)
(21)
ʃ dydzply, 2) log p(y plylz, (doe) Hly).
Since the target information entropy H(y) is independent of the optimization procedure of the parametric model, it can be ignored. Thus, max I(z,y) max f dydzp(y, z) log p(yiz; wb ). By Eq.
16 ply. z) pxp(x) therefore.
max [(z. y) = max S dxddddxp()p(yy))paaaa p(ylz; "
(22)
We now consider the compressive term Bl(z, x) the IB objective (17). and temporally discard the constant factor B. The mutual information between input stimuli and latent representation is given by
f dxdzp(x, z) log
p(z|x)
1(z, x) =
p(z)
(23)
f dzp(z) log p(z).
dxdzp(x, z) log p(z)x)
Let p(z; gariar) be variational approximation to the marginal p(z), because KL(p(z)llp(z; )rrior) 2 0, we have that
f dzp(z) log p(z) z f dzp(z) log p(z; sprio)..
(24)
With Eq.
23 in tow and using parametric encoder q(zix; ) have the following upper bound:
I(z,x) s S [xxzp(x))(Zzxx "n log g(z)x; )
(25)
p(z; dprior)
By Eqs. 22 25 we have lower bound for the IB objective as follows,
f
dxddydkp(x)p((xxx/lllx// a log p(ylz; y) (l(2,)))
(26)
BB dxdzp(x)g(zxx nh log gz sTn))
p(z; Φrrive)
Using 0 to denote all the parameters (gptior yn sdee and other learnable parameters, like those of the feature extractor) of the model following the main text, have that
max[/(z,y: -1(1,, x; 0)) min L.
(27)
@
where
L= f
(xchodep(x)ptyssti(llxss y log p(ylz: dee)
gi cocouraging predictive power
(28)
B f
dxdzp(x)g(z)x; the) log g(z|x; yenc)
p(z; dpprioe
fomm; encouraging compression
As for now, we have the formulation presented Eq. in the main text. We proceed to derive the exact loss function for TeCoS-LVM model learning. Following previous literature 132 we can approximate the data distribution p(x, y) using the empirical data distribution Σ 5(x- (( where ò is the dirac delta function. Hence, we have that
L R
Byte, (appaa log pZ 16 dec) +B (l ye [lp(,, ((riie))
T
comp
(29)
pred
+B
7
Predictive (total): Dee
Compressive term (total); L Tomp
19

---

As we adopt Gaussian distributions in our temporal conditioning prior and encoder, we can analytically compute the compressive loss Leomp composed of Kullback-Leibler divergences.
We then turn to the predictive term in our objective (17 As our model directly produces simulated spike trains, compute the spike train dissimilarity between the prediction f1:, and the real record Y1:r to assess the predictive power of our model directly. This dissimilarity is used as the predictive loss term each timestep. In particular, we employ the Maximum Mean Discrepancy (MMD) to measure the distance between spike trains. This approach has been proven suitable for spike trains in previous literature [26 [5], following ref. [50), we use postsynaptic potential (PSP) function kernel for MMD. We employ the first-order synaptic model as the PSP function to capture the temporal dependencies in spike train data effectively [5]]. The PSP kernel shall use is given by
Σ PSP( where PSP(y1:)) -PSPy)S
Kpsp ( 1: V1:1
Ts Yr.
(30)
T,
here T, is a synaptic time constant set to by default. We can write the (squared) PSP kernel MMD between the empirical data distribution and the predictive distribution as
Σ B By (PSP()::)) 1.)] By IP))(y
MMD[p (1.)). p(y):))
(31)
In practice, approximate the spike train dissimilarity, which is measured by the squared PSP kernel MMD in Eq. 31 by - PSP(y..)) [50]. Therefore, the predictive loss term is given by
prred Σ (PSP( PSP(Y
(32)
Together with the compressive term Soomp KLI P (Z,)). by Eq. 29
calculate the loss function and optimize TcCoS-LVM models. To allow direct backpropagation through a single sample of the stochastic latent representation, use the reparameterization trick as described in
E
Data Description
Movie
("salamander movie")
Movie ("wildlife movie")
Figure 7: Example frames from Movie and Movie 2 in the data we used.
We perform evaluations and analyses on real neural recordings from RGCs of dark-adapted axolotl salamander retinas. The original dataset (57] contains the spike neural responses (collected using multi-electrode arrays [85] (6) of two retinas on two movies. Movie 1 contains natural scenes of salamanders swimming in the water. Movie 2 contains complex natural scenes of a tiger on a prey hunt. Both movies were roughly 60 long and were discretized into bins of 33 ms. All movie frames were converted to grayscale with resolution of 360 pixelx360 pixel at 7.5 fmx7.55 7. pm per pixel, covering 2700 μmx27000 um area the retina. For retina 1, we have 75 repetitions for movie and 107 repetitions for movie 2. For retina 2, we have 30 and 42 repetitions for movie 1 and movie 2, respectively. Some example frames shown in Appendix Fig.
20

---

F Metrics, Features used in Evaluations and Visualizations
F1 Evaluation Metrics
Pearson correlation coefficient (Pearson CC, CC) This metric evaluates the model performance by calculating the Pearson correlation coefficient between the recorded and predicted firing rates 14 [1]]. The higher the value, the better the performance. For spike-output TeCoS-LVM models, the firing rates are calculated using 20 repeated trials.
Spike train dissimilarity (Spike train dissim.) This metric assesses the model performance by computing the dissimilarity between recorded and predicted spike trains. A lower value indicates better model performance. We use the MMD with first-order PSP kernel [51 [2 to measure
the spike train dissimilarity [27 50 58 The first-order PSP function is given by PSP(y):))
(1 (SP(y))))) H.y,, where T, is synaptic constant and is set to 2. Given recorded spike
train F1:7 and predicted spike train 91.7. this metric is calculated by Σ IIPSP(y):)) - PSP( Because of the variability of neural activities, we randomly selected ten (trials) recorded spike trains and used their average value in our evaluations.
van Rossum distance (van Rossum) This spike train distance was introduced in ref. (87) where the discrete spike trains are convolved by an exponential kernel Heavisidet exp(-1/TR) here we use Tи 10. The final scores are computed by averaging results calculated using ten recorded spike trains.
Victor- Pector-Purpura distance (V.-P.) This spike train distance 88 measures the dissimilarity between two spike trains by summing up the minimum cost of transforming one spike train into the other by insertion, deletion, and shifting operations. We use the average results from ten trials the final metric.
SPIKE distance (SPIKE) The SPIKE distance 89 is time-scale independent metric for quanti- fying the dissimilarity between spike trains. Its value is bounded in the interval [0. 1]. and zero is obtained only for perfectly identical trains.
F.2
Spike Feature
Spike autocorrelogram The spike autocorrelogram is computed by counting the number of spikes that occur around each spike within a predefined time window [4 _. The resulting trace is then normalized to its maximum value (which occurs the origin of the time axis by construction). In the main text, the maximum value set to zero for better visualization and comparison.
G Experimental Details
G.1 Experimental Platform
The models are implemented using Python and PyTorch. Our experiments were conducted on workstation with an Intel-10400, one NVIDIA 3090, and 64 GB RAM.
G.2 Implementation Details
TeCoS-LVM models For T models, all hyper-parameters on all datasets are fixed to be the same. We set the latent variable dimension to 32 and the hidden state dimension to 64 by default. We used the Adam optimizer (B; = 0.9,83 = 0.999) with a cosine-decay learning rate of 0.0003 with mini-batch size 64. Training of these models is carried out for 64 epochs. We used the same architectures to implement all the TeCoS-LVM models (see Table 33.. For LIF neuron TeCoS-LVM models (denoted as TeCoS-LVin gradient (see also Eq. 10) SGERF(x) = Y exp(-x²). For Noisy LIF neuron TeCoS-LVM models
used surrogate gradient learning (SGL) with ERF surrogate
(denoted as TeCoS-LVM Noisy), we used the Gaussian noise N(e; 0, 0.22) and the corresponding Noise- Driven Learning (a theoretically well-defined general form of SGL), which is described in
21

---

Appendix C Eq. 15 Since random latent variables are involved in our model, we also employed the reparameterization trick for efficient training.
Baselines We followed the settings in their original implementations for the CNN 9 IB-Disjoint 21] models. Some of these settings leverage the prior statistical structure information of the firing rate, thus improving the performance of these models (9. In particular, the CNN model is trained with Gaussian noise injection, L-2 norm regularization (0.001) over the model parameters, and LI norm regularization (0.001) over the predicted activations [9 15]. The IB-Disjoint model optimized with B = 0.01, which has proven to lead better predictive power 2 We also use the reparamcterization trick for efficient training for the IB- IB-Disjoint model. We use constant learning rate 0.001, a mini-batch size 64, and a default Adam optimizer for these two models. Following their original implementations, network architectures of these models are listed in Appendix Table
15 and
used the early stopping technique in training. The
Table 3: List of network architectures (functional models) our experiments. conv for the convolu- tional layer, fc for the fulle-connected layer, GRU for the gated recurrent unit layer.
MODEL NAME
DESCRIPTION
Noisy
(feature extractor) 16conv25-32comvl11-2 (real-valued RNN) GRU64
(LIF/Noisy LIF spiking neurons,
input channel=1);
(encoder) fc64-fc64
TeCoS-LVM Rate (LIF-Rate neurons input channel=l);
(encoder mean) fc32
(encoder std) fc32
TeCoS-LVM Noisy Rate (Noisy LIF-Rate neurons, input channel=1).
(prior) fc64-fc64
(prior mean) fc32
(prior std) fc32
(decoder) FcGGGS
CNN
$2conv25--atchNormm
(ReLU neurons, input channel=T))
fc#RGC>-Btchhorm
ParametricSoftPllu
IB-Disjoint
(ReLU neurons, input channel=T)
fc64-BatchNorm
(encoder)
(encoder mean) (encoder std) fc32-Battee BatchNorm (decoder) -etthhhNooo--fGGGG BatchNomm ParametricSoftills
Symbol descriptions (parameter type parameter): channel number conv kernel size; fc channel number; GRU
hidden state dimension.
H Details of Ablation Experiments
H.1 The effect of using spiking neurons
In this part, we constructed two variants (TeCoS-LVM Rate and TeCoS-LVM Noisy Rate) by replacing all the hidden spiking neurons in the TeCoS-LVM model with neurons that exhibit the same internal recurrence but provide non-spiking output. In other words, in these two variants, the activation of our hidden neurons transitioned from discrete spiking Heaviside functions continuous functions. Specifically, we adopted the rate-output neuron model named GLIFR introduced in ref. (65). In particular, the modified LIF-Rate neuron model we used here still uses the membrane update rules of LIF (and Noisy LIF). The output activation function of the LIF-Rate model is described by sigmoid (=="te) where the parameter 0,, controls the smoothness of the membrane voltage-spike relationship. In doing so, the internal representation constructed in a real-valued space rather than in sparse spike space as TeCoS-LVM models with all LIF and Noisy LIF neurons.
22


Reviews:
[{"review_id": "InMf1eRYVs", "reviewer": "Reviewer_j9bD", "summary": "This paper proposes a model, which is composed of spiking neural networks and a conventional recurrent network, to reproduce neuronal responses, i.e., retinal ganglion cells. The author claims that the novelty of this model is incorporating the spiking network, which makes the model directly outputs spikes and is more biologically plausible. The author did simulations to show the model outperforms alternatives in reproducing retinal ganglion cells' activities.", "questions": "- Conceptually, I don't understand why the model with spiking network encoder and decoder could render better results. I am really interested in some deep, theoretical discussions on this issue. I could understand if the output layer is a spiking network, it would fit the spiking neuronal responses better. Nonetheless, I don't understand why using spiking net as intermediate layers could improve the performance. Can the author provide an alternative result to replace the spiking intermediate layers by rate-based neurons and compare the performance of the two versions?\n\n- Another conceptual question. From the neural pathway point of view, the neural circuits from the retina to the visual thalamus, i.e., LGN, is equivalent to dimensionality reduction because the number of neurons decreases with the neural pathway. And then from LGN to the visual cortex the neuron number will expand a lot. In this sense, the visual pathway inside the retina, and the one from the retina to LGN looks like an __encoder__, while the pathway from LGN to the visual cortex is like a __decoder__. If my statement was true, the retina is just an encoder with less number of neurons in the output layer (retinal ganglion cells) compared with the input layer (photoreceptor). Hence, to reproduce retinal ganglion cells' responses, what is the advance of considering the proposed model architecture (Fig. 1B) with an encoder (dimensionality reduction) and a decoder (dimensionality expansion?", "limitations": "See my comments in Questions.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "### Originality\nIn my understanding, the paper has two novelties: 1) the model incorporates spiking neural networks as encoder and decoder; 2) the model uses an RNN to provide temporal conditioning which is more flexible than previous models whose kernels have fixed temporal durations. \n\n### Quality \nThe writing has clear explanations of the probabilistic framework, but it is quite unclear how the probabilistic framework is connected with underlying spiking networks (see my comments below).", "weaknesses": "### Major\nI understand the variational information bottleneck framework, and the text of this part is clearly written. However, it is unclear the details of the spiking networks and how they are incorporated into the probabilistic framework. \n1) Specifically, how the distributions' parameters $\\phi$ and $\\psi$ in Eqs. 6 and 7 are related to spiking networks? PS: I only see line 151 saying $\\phi$ is a linear readout of spiking neurons' responses.  \n2) What are the structures of spiking networks in the encoder and decoder? Are the pure feedforward spiking network, or recurrent spiking network?\n3) During the model training, did the authors only train the readout weight from the spiking network? or also train the synaptic weights inside the spiking network? \n\nWithout providing these details, I am not confident in judging the novelty of this paper. I am happy to see the authors' rebuttal to explain this.\n\n### Minor\n- Line 177, “Sec. C” and Line 188, “Sec. E”: I don’t know where these sections are. In the supplementary?\n- Eq. 3: does this equation miss a term of $\\beta I_c$ in the Lagrangian multiplier?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear reviewer, thank you very much for your detailed review and positive comments, which have greatly encouraged us. Our response is as follows.**\n\n> I understand the variational information bottleneck framework, and the text of this part is clearly written. However, it is unclear the details of the spiking networks and how they are incorporated into the probabilistic framework.\n>\n> 1. Specifically, how the distributions' parameters $\\phi$ and $\\psi$ in Eqs. 6 and 7 are related to spiking networks? \n> 2. What are the structures of spiking networks in the encoder and decoder? Are the pure feedforward [...]\n> 3. During the model training, did the authors only train the readout weight from the spiking network [...]\n\n**Re:**   \n\n1. For the temporal conditioning prior $\\phi\\^{prior}$ , we use a spiking network, which takes the hidden state $\\mathbf{h}\\_t$ as input, and outputs the distribution means and variances using linear readout synapses. The temporal conditioning encoder $\\psi\\^{enc}$ is a spiking network that receives the current stimulus $\\mathbf{x}\\_t$ and hidden $\\mathbf{h}\\_{t-1}$ as inputs, and produce mean and variance using linear readout synapses. The temporal conditioning decoder $\\psi\\^{dec}$ takes the latent $\\mathbf{z}\\_t$ and the hidden $\\mathbf{h}\\_{t-1}$ as inputs, and produces spike activities (predictions).  \n\n2. The spiking networks (for parameterizing those $\\phi$ and $\\psi$s) mentioned above are all feed-forward; they are spiking MLPs. Since we introduced temporal conditioning operations using hidden states to handle temporal dependencies, we followed the feed-forward network structure, as in previous works [1,2]. We found that such a simple feed-forward structure can already produce very competitive performance. Adding more recurrent connections in the model may bring some improvement, but it also increases the complexity and the number of parameters of the model, which is not conducive to further analysis. Therefore, in this work, we only consider using a feed-forward structure.\n3. During the model training, we optimize all synaptic weights inside the model, including those of the spiking networks, and the RNN, which maintains the hidden $\\mathbf{h}$.  \n\n[1] Deep learning models of the retinal response to natural scenes. *NIPS*, 2016.\n\n[2] Interpreting the retinal neural code for natural scenes ... *Neuron*, 2023.\n\n> Line 177, “Sec. C” and Line 188, “Sec. E”: I don’t know where these sections are. In the supplementary? \n>\n> Eq. 3: does this equation miss a term of $\\beta I_c$ in the Lagrangian multiplier?\n\n**Re:**  Yes, they are in the Appendix. We'd like to revise our presentation in revision to make it clearer.  The $I_c$ term is not missing in equation 3, because we are maximizing an equivalent objective [1]. \n\n[1] Deep Variational IB. *ICLR*, 2016.\n\n> Conceptually, I don't understand why the model with spiking network encoder and decoder could render better results [...]\n\n**Re:** We have conducted additional experiments following your comments (***please refer to \"Author Rebuttal by Authors\" panel above for Table 1,2***). We additionally consider *Victor-Purpura*, *van-Rossum*, and *SPIKE* distances as metrics. The results show that our variants can also achieve competitive performance than baselines, but lag behind standard tecos-lvm models. \n\nSince our main network is feed-forward, using spiking neurons can enhance the capture of the spatiotemporal features embedded in the stimuli stream (which is essentially spatiotemporal) through the neuronal temporal dynamics (ANN neurons don't have that), thus producing better predictions. Also, a hallmark of natural stimuli is their sparse latent structure [1,2]. By using spiking neurons in the intermediate layer, the model can take advantage of the sparse representation (compared with those of ANN neurons) to better fit the latent structure of natural stimuli, resulting in improved performances.\n\n[1] *Natural image statistics...* Springer, 2009.\n\n[2] Toward a uniﬁed theory of efﬁcient, predictive... *PNAS*, 2018\n\n> Another conceptual question. From the neural pathway point of view, the neural circuits from the retina to the visual thalamus, i.e., LGN, is equivalent to dimensionality reduction because the number of neurons decreases with the neural pathway. And then from LGN to the visual cortex the neuron number will expand a lot. In this sense, the visual pathway inside the retina, and the one from the retina to LGN looks like an **encoder**, while the pathway from LGN to the visual cortex is like a **decoder**. [...] Hence, to reproduce retinal ganglion cells' responses, what is the advance of considering the proposed model architecture (Fig. 1B) with an encoder (dimensionality reduction) and a decoder (dimensionality expansion?\n\n**Re:**  Considering the proposed architecture allows us to simultaneously obtain latent dynamics for future interpretation/analyses [1-3] and achieve better modeling performance. \n\nThe encoder-decoder we use here is not solely for dimensionality expansion/reduction (Fig.1A is only schematic), but rather to simulate a two-stage efficient neural coding process [4] of inferring latent codes from stimuli and then decoding neural responses from latent codes. And, the latent dimension in our model is adjustable, e.g., if we choose a large latent dimension (larger than the output RGC dim.), then the whole process becomes dimension reduction.  So the entire TECOS-LVM model is performing the encoding process (stimuli to the retina), which is, functionally, an encoder as you mentioned. We would like to supplement the relevant content in the revised manuscript and improve the graphics in Fig. 1 to address your concern.\n\n[1] Inferring single-trial neural population dynamics... *Nat. Methods*, 2018.\n\n[2] Targeted neural dynamical modeling. *NIPS*, 2021.\n\n[3] Drop, swap, and generate... *NIPS*, 2021.\n\n[4] Toward a uniﬁed theory of efﬁcient, predictive... *PNAS*, 2018\n\n"}, {"review_id": "1nC9u9ihXZ", "reviewer": "Reviewer_qZ33", "summary": "This paper proposes a spiking latent variable model of neural response to natural visual stimuli. The model is trained to directly predict spike trains instead of trial-averaged firing rates, and designed to work, at test time, on sequences that are longer than sequences seen during training.", "questions": "- The model can be applied to sensory data of other modalities, how sensitive is the model to the choice of hyperparameters? For example, the latent dimension or the hidden dimension? \n- Why was the warmup period similar for all four datasets (0.5s), one would expect to observe differences due to different dynamics of the stimuli.", "limitations": "As stated, the proposed model is only partially realistic.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The authors identify two critical limitations of biophysically-realistic computational models. They propose a model that effectively address these limitations. Directly predicting spike trains using spiking neural networks, appears to be a natural solution.\n2. The empirical experiments suggest that this results in a more accurate prediction of neural responses.", "weaknesses": "1. A main claim of the paper is that the model generalizes to longer time scales unseen during training. It is unclear whether the model is actually leveraging long-range interaction in its prediction. The results do not demonstrate that the memory of the model is holding long-range history from the sequence. It is unclear whether a model with a sliding window (of 1s) would perform the same or worse.\n2. While the authors do a great job at analyzing different aspects of the model including the impact of hyperparameters on the learned latent space, it remains unclear how this more bio-realistic model provides novel insights into sensory processing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear reviewer, many thanks for your detailed review and comments! Our response is as follows.**\n\n> A main claim of the paper is that the model generalizes to longer time scales [...]\n\n**Re:**  We would like to point out that our model only uses single-step stimulus input for prediction and leverages temporal dependencies by holding a memory in its predictions (as mentioned by Reviewer *QFPa*, *qcrE*). Also, the IB-Disjoint model has an LVM structure and uses a 1s sliding window, similar to the model you described. According to experimental results in Fig.2 and 3 in the paper, our model outperform IB-Disjoint in all evaluations. \n\nAnd to further address your concern, we implemented a 1s sliding window variant (*SlidWindVariant*, and *SlidWindVariant Noisy*) of our model following your description, and ran tests on Mov1Ret1 data (Table below). The results show that these sliding window variants performed worse in all metrics, reflecting the advantages of using a temporal conditioning structure to handle these temporal interactions.\n\n| metric\\model       | TeCoSLVM | TeCoSLVMNoisy | SlidWindVariant | SlidWindVariant Noisy | CNN    | IBDisj. |\n| ------------------ | -------- | ------------- | --------------- | --------------------- | ------ | ------- |\n| firing rate CC     | 0.579    | 0.727         | 0.567           | 0.681                 | 0.690  | 0.653   |\n| VictorPurpuraDist. | 12.84    | 14.02         | 18.62           | 19.93                 | 19.60  | 21.92   |\n| vanRossumDist.     | 127.35   | 238.61        | 307.78          | 343.02                | 376.82 | 394.02  |\n| SPIKE Dist.        | 0.124    | 0.155         | 0.197           | 0.210                 | 0.207  | 0.224   |\n\n> While the authors do a great job at analyzing different aspects of the model including [...], it remains unclear how this more bio-realistic model provides novel insights into sensory processing.\n\n**Re:**  To name some concrete sensory processing (encoding of natural stimuli) studies that could benefit from our model. Fitting neural response to natural stimuli more accurately: works like [1] could benefit from our model to identify microcircuits that mediate/gate diverse encoding properties in sensory processing. Intrinsic/latent dynamics, specific circuit mechanism characterization:  our model provides an efficient approach to decompose the computations in sensory processing into the latents (a generative model that enables a low-dimension representation [2]), which could allow neuroscientists to generate new hypotheses, or test their hypotheses about how interneurons with diverse response properties are combined to perform sensory processing [2-4] and provide neurally-grounded understandings of sensory computations therein.  \n\n[1] Multiplexed computations... *Nat. Comm.*, 2017\n\n[2] Stimulus-and goal-oriented frameworks... *Nat. Neurosci.*, 2019. \n\n[3] Toward a unified theory of efficient... *PNAS*, 2018. \n\n[4] Interpreting the retinal neural... *Neuron*, 2023.\n\n> The model can be applied to sensory data of other modalities, how sensitive is the model to the choice of hyperparameters? For example, the latent dimension or the hidden dimension?\n\n**Re:** Our results show that the model is quite robust to the hyperparameters you mentioned and can achieve good performance even when the hidden state/latent space dimension is small.\n\nWe evaluated the performance of TeCoS-LVM models under different hidden state and latent space dimensionality settings.  The results, shown in Appendix Fig. 6A, indicate that increasing the latent space dimension improved performance. Still, further increases ($>32$)  had little effect when the latent space dimension was large. This also suggests that the number of latent factors for the visual stimuli coding task we currently consider is not very large. On the other hand, increasing the hidden state dimension enhances sensory memory capacity and benefits temporal conditioning operations. As shown in Appendix Fig. 6B, the performance does not increase significantly when hidden state dimensionality increases to a certain extent (about $64$). The results in Appendix Fig. 6B also indicate that the proposed temporal conditioning mechanism can effectively utilize sensory memory and achieve good results even when the hidden state dimension is small (memory capacity is small).\n\n> Why was the warmup period similar for all four datasets (0.5s), one would expect to observe differences due to different dynamics of the stimuli.\n\n**Re:**  Thanks for your meticulous review. In fact, the setting of this warmup period is currently, more empirical. This means, *we cannot use it as an \"indicator\" to reflect the different dynamics of different data*. The length of this period is more empirical, as it is difficult to precisely choose the most \"suitable\" warmup period length for different data. In other words, currently, we lack a reliable quantitative approach to accurately measure which warmup length is exactly \"optimal\" or most \"correct\". \n\n\n"}, {"review_id": "lpKOecclpY", "reviewer": "Reviewer_QFPa", "summary": "The authors model retinal ganglion cell responses to natural stimuli using a spiking latent variable model. They employ the (variational) Information Bottleneck (IB) method to compress the visual representation, similar to last year’s NeurIPS paper by Rahamni et al. However, this work differs in using binary responses (discretizes spike trains) instead of count responses, an input channel size of 1 instead of T, and temporal conditioning (RNN) instead of GP prior to handle temporal dependencies. They compare their results to Rahamni et al’s IB method and a CNN-based architecture on 4 salamander datasets. They show that those baselines fail to capture the spike autocorrelation, and a noisy version of their LVM model can also reproduce real firing rates more accurately.", "questions": "- Eq (1) 3rd eq with $u_t$ on both sides, yielding $u_t=u_{reset}/o_t$, is odd.\n\n- line 103: Isn't $\\tau_m$ is the membrane constant, not $\\tau$? It seems $\\tau=e^{-\\Delta t/\\tau_m}$ with bin size $\\Delta t$.\n\n- Why do you compare to IB-disjoint, not IB-GP? The authors of [19] favored IB-GP (temporal prior over latents) over IB-disjoint (no temporal prior). \n\n- Does IB-GP need fewer latents that are more interpretable latents than yours?\n\n- Line 290: Where is the Fig 5 you refer to?\n\n- There are various measures of spike train dissimilarity in the literature, e.g. Victor-Purpura distance, Van Rossum distance, SPIKE-distance. You use the MMD in results and use an objective function (Eq 10) that specifically optimizes for this particular measure, thus obtaining good results. Do your conclusions hold for the other commonly used measures of spike train synchrony as well? \n", "limitations": "The authors have discussed limitations of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- good performance\n- scales well to long time series\n- modeling single trial spike trains, not just trial-averaged firing rates", "weaknesses": "- Stimuli are converted to spikes, then to real valued signals (LVM), then back to spikes (ganglion cells). This does not align with visual processing in the retina: photoreceptors exhibit graded responses, as do bipolar cells (typically), whereas ganglion cells spike.\n- The authors stress the importance of taking temporal dependencies into account, but instead of comparison to IB-GP they compare to IB-disjoint. Whereas the former has a temporal prior over latents the latter does not.\n- A comparison to a method that also predicts spike trains not firing rates is lacking. Such a method could be competitive with regard to the considered spike train dissimilarity and autocorrelation measures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear reviewer,** \n\n**Thank you very much for your thorough review and positive comments, which have greatly encouraged us! Our response is as follows.**\n\n> Stimuli are converted to spikes, then to real valued signals (LVM), then back to spikes (ganglion cells). This does not align with visual processing in the retina: photoreceptors exhibit graded responses, as do bipolar cells (typically), whereas ganglion cells spike.\n\n**Re:** Yes, the computational model reflects some, but not all, architectural, computational, and anatomical motifs of neural circuit formations [1-4].  The proposed dynamic spiking model aims to reproduce the neural activities by simulating the underlying general neural coding processes (as also mentioned in Reviewer qcrE); in doing so, it ignores some known neuronal biophysical motifs. And in this manuscript, we specialize our model to a natural stimuli case (as a paradigmatic example) for evaluations and investigations. We would like to add related discussions in the revised version of the manuscript to address your concern.\n\n[1] Stimulus-and goal-oriented frameworks ... *Nat. Neurosci.*, 2019.\n\n[2] The Tolman-Eichenbaum machine ... *Cell*, 2020.\n\n[3] Fitting summary statistics of neural data ... *NIPS*, 2021.\n\n[4] Mesoscopic modeling of hidden spiking neurons. *NIPS*, 2022.\n\n> Eq-1-3rd eq with $u_t$ on both sides, yielding $u_t=u_{reset}/o_t$ , is odd.\n\n**Re:**  Thanks for the correction. We will revise this expression in our revision to make it clearer. The original intention here is to indicate the resetting in the iterative calculation form we use, i.e., reset to $u\\_{reset}$ after the neuron emits a spike.\n\n> line103: Isn't $\\tau_m$ is the membrane constant, not $\\tau$? It seems $\\tau=\\exp(-\\Delta t / \\tau_m)$ with bin size $\\Delta t$.\n\n**Re:** Thanks for pointing out this unclear part. In our experiments, we actually handle data that is sampled using a fixed timestep. Thus, following previous work (e.g. [1]), when implementing spiking models in our case, we replace the genuine time constant with a fixed time constant which \"incorporates\"  $\\Delta t$. That is, in our implementation in this case, the model weight is associated with the simulation timestep (although strictly, the model weights should be irrelevant to $\\Delta t$). We will improve the presentations in our revision to make it clearer. \n\n[1] Brain-inspired global-local learning incorporated ... *Nat. Comm.*, 2022.\n\n> * The authors stress the importance of taking temporal dependencies into account, but instead of comparison to IBGP they compare to IB-disjoint. Whereas the former has a temporal prior over latents the latter does not. Why do you compare to IB-disjoint, not IBGP? The authors of [19] favored IBGP ...\n>\n> * Does IBGP need fewer latents that are more interpretable latents than yours?\n\n**Re:** \n\n* This is mainly because, in evaluations on our data, we found that the performances of IBDisjoint were better. E.g., on Mov1Ret1, the Pearson CC *(IBDisjoint vs IBGP)* 0.65vs 0.61. Actually, this phenomenon happens sometimes (\"…IB-Disjoint obtains superior performance than IB-GP…\" from Appendix 4 in [19]). Another possibility is that our re-implementations (as [19] has not yet released them) lack some technical details which are omitted in the paper [19].  Besides, IBGP has additional parameters (those of the Cauchy kernel) that need to be adjusted, this makes its hyper-parameter tuning process quite tricky when the data is changed. \n* In the main experiments in the IB paper, only nine neurons were actually used, so the latent dimension was small. Our tests show that our methods can actually, use a smaller latent dimension to produce more accurate predictions. For example, on Mov2Ret2, the firing rate Pearson CC of TeCoSLVM-Noisy with *latent-dim=8* was *0.81*, while those of the IB model with *latent-dim=32* was *0.66*. Thus, our method can use fewer latents, which is more advantageous from the interpretable aspect.\n\n> Line 290:Where is Fig5\n\n**Re:** Yes, in Appendix (it was re-uploaded as *Fig1* in the PDF file in \"*Author Rebuttal by Authors*\" panel). We will like to revise our manuscript to fix the unclear ref. you mentioned.  \n\n> A comparison to a method that also predicts spike trains not firing rates is lacking...\n\n**Re:** To address your concern, we have conducted additional experiments using two variants of our model, following your comments (***Please refer to \"Author Rebuttal by Authors\" panel at the top of that page for TABLE 1, 2***). We noticed that, these variants also have certain advantages compared to the baseline, which also demonstrates the efficiency of our models.\n\n> There are various measures of spike train distances, e.g. Victor Purpura, van Rossum, SPIKE dist...\n\n**Re:** We have conducted additional evaluations on two benchmarks (Mov1Ret1, Mov2Ret2) following your comment and present them in tables as follows. According to these results (***Please refer to \"Author Rebuttal by Authors\" TABLE 1, and TABLE 2***), on these spike train distance metrics, our series of methods (tecos/tecos-noisy and two variants) maintain a stable advantage over the baselines. \n\n\n"}, {"review_id": "vfNdR1OI7y", "reviewer": "Reviewer_qcrE", "summary": "The authors propose a new model called temporal conditioning spiking latent variable models (TeCoSLVM) that uses spiking neurons to simulate neural response to visual stimuli. They claim that this approach helps to preserve information in spike trains and adapt to temporal dependencies. In experiments with retinal data, they demonstrate that the TeCoS-LVM models produce more realistic spike activities, fit spike statistics accurately, and generalize well to longer time scales.", "questions": "I integrated the questions in the weaknesses above. ", "limitations": "Limitations are discussed in one short paragraph at the end. It could be a bit more extensive. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Models that accurately predict spiking activity are still an open problem. The authors make an interesting contribution to this by defining a latent variable model that is trained with an information bottleneck objective. The work is, as far as I can tell, original. A few links to existing work are missing, which I mention below. The paper is mostly well and clearly written, and has an overall good quality. I have a few questions for the authors, but if they are answered satisfactorily, the paper makes a significant contribution to the development of dynamic spiking models. ", "weaknesses": "I have a few questions regarding the paper and a few hints for relevant literature. \n---\nDecoder: \n* I couldn't find information on the actual form of the decoder in the main paper. In particular, how is $\\psi_{dec}$ computed? \n* How are the latent states integrated into the LIF neurons? \n* What is also not entirely clear to me is whether the model has access to real spike trains from previous time steps or not. \n* From the figure it doesn't look like it, but since the decoder is not clear to me I cannot be sure. It would be nice if the authors could clarify. This is particularly relevant to make sure the comparison to the CNN is fair.  \n\nExperiments: \n* How are the spiking rates computed?\n* What loss function is used to train the CNN? Poisson loss? Would it be possible to also train the spiking model with Poisson loss (i.e. do you have the rate)? The reason I am asking that is that Fig 3B shows that your TECOS models perform better in terms of spike train dissimilarity. However they were trained on that. If the CNN was trained on a different objective, the comparison is a bit unfair. \n\nFig3: \n* How do you generate spikes from the CNN in Fig 3A. \n\n---\nRegarding dynamic models of neural activity, these two paper might be relevant (in particular regarding the point of predicting variable length sequences):\n- Fabian H. Sinz, Alexander S. Ecker, Paul G. Fahey, Edgar Y. Walker, Erick Cobos, Emmanouil Froudarakis, Dimitri Yatsenko, Xaq Pitkow, Jacob Reimer, Andreas S. Tolias Stimulus domain transfer in recurrent models for large scale cortical population prediction on video\n- Eric Y Wang, Paul G. Fahey, Kayla Ponder, Zhuokun Ding, Andersen Change, Taliah Muhammad, Saumil Patel, Zhiwei Ding, Dat T. Tran, Jiakun Fu, Stelios Papadopoulos, Katrin Franke, Alexander S. Ecker, Jacob Reimer, Xaq Pitkow, Fabian H. Sinz, Andreas S. Tolias Towards a Foundation Model of the Mouse Visual Cortex\n\nRegarding spiking neurons, this paper might be relevant: \n- Ramesh, Poornima, Atayi, Mohamad, Macke, Jakob H: Adversarial training of neural encoding models on population spike trains", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear reviewer,** \n\n**Thanks very much for your detailed review and positive comments, which have greatly encouraged us! Our response is as follows.**\n\n> Decoder:\n>\n> 1. I couldn't find information on the actual form of the decoder in the main paper. In particular, how is $\\psi_{dec}$  computed\n> 2. How are the latent states integrated into the LIF neurons\n> 3. What is also not entirely clear to me is whether the model has access to real spike trains from previous time steps or not\n> 4. From the figure it doesn't look like it, but since the decoder is not clear to me I cannot be sure. It would be nice if the authors could clarify. This is particularly relevant to make sure the comparison to the CNN is fair\n\n**Re:**  Thanks for your comment! We will improve relevant descriptions in our revised manuscript to make them clearer.\n\n1. The temporal conditioning decoder $\\psi\\^{dec}$ is a two-layer spiking MLP that takes the current latent $\\mathbf{z}\\_t$ and hidden (which acts as a sensory memory) $\\mathbf{h}\\_{t-1}$ as inputs, and outputs neural response predictions.\n2. Currently, in our implementation, we feed the latent states as the electric current input into the spiking neuron.\n3. & 4.  The model will not access real spike trains (and stimuli) of previous timesteps (as described in Figure 1. C in the manuscript). Take time $t$ as an example, our model only takes the current stimulus $\\mathbf{x}\\_t$, and the sensory memory (hidden) $\\mathbf{h}\\_{t-1}$. This means, compared to the CNN model, our model receives fewer inputs (CNN receives a batch of stimuli [1,2], while we only have a single timestep input). Therefore, regarding the amount of input information received for producing a single-step prediction, our comparison is actually more favorable to CNN, but our model still performs better.\n\n[1] Deep learning models of the retinal response to natural scenes. *NeurIPS*, 2016.\n\n[2] Interpreting the retinal neural code for natural scenes ... *Neuron*, 2023.\n\n> Experiments:\n>\n> 1. How are the spiking rates computed\n> 2. * What loss function is used to train the CNN? Poisson loss? Would it be possible to also train the spiking model with Poisson loss (i.e. do you have the rate)\n>    * The reason I am asking that is that Fig 3B shows that your TECOS models perform better in terms of spike train dissimilarity. However they were trained on that. If the CNN was trained on a different objective, the comparison is a bit unfair.\n\n**Re:** \n\n1. TeCoS models are spike-output, so in our tests, the firing rates are calculated using 20 repeated trials. \n2. * Yes, the CNN model is trained using the Poisson loss [1]. Since our models directly output spike trains, it is not possible to *directly* use Poisson loss for them (additional conversions are required to achieve this, but the necessity of doing so is limited). \n   * Regarding the spike train distance comparison, we have conducted additional experiments that use different spike train distances to make to comparison more convincing (***Please refer to \"Author Rebuttal by Authors\" at the top of this page for TABLE 1 and 2.*** ). According to these results in Table 1, and Table 2, on these spike train distance metrics, our series of methods (tecos/tecos-noisy and our two variants) maintain a stable superiority over the baselines.  \n\n[1] Deep learning models of the retinal response to natural scenes. *NeurIPS*, 2016.\n\n> Fig3: How do you generate spikes from the CNN\n\n**Re:** Following previous works [1,2], the spikes are generated by sampling from the predicted poisson distributions. \n\n[1] Deep learning models of the retinal response to natural scenes. *NeurIPS*, 2016.\n\n[2] Interpreting the retinal neural code for natural scenes... *Neuron*, 2023.\n\n> Regarding dynamic models of neural activity, these two papers might be relevant (in particular regarding the point of predicting variable length sequences):\n>\n> * Stimulus domain transfer in recurrent models for large scale ... *NeurIPS*, 2018.\n>\n> * Towards a Foundation Model of the Mouse Visual Cortex. *bioRxiv*, 2023\n>\n> Regarding spiking neurons, this paper might be relevant:\n>\n> * Adversarial training of neural encoding models ... *NeurIPS Workshop on NeuroAI*, 2019. \n\n**Re:** Thank you for pointing out the relevant literature!  We are very happy to add these relevant discussions to our revised manuscript to address your concern. We find that these excellent works (Sinz 2018, Wang 2023) leverage a hybrid \"core\" model (feedforward+recurrent) to shift along the temporal dimension. This is closely related to the temporal conditioning structure we used in this work. Regarding the related literature (Poornima 2019) you mentioned, we noticed that the part using adversarial training for optimization to capture the deterministic and stochastic components in neural population activities is very relevant to our motivation for using noisy LIF neurons. "}]

Author rebuttal:
We sincerely thank all *Reviewers* for their thorough reviews and *Chairs* for their efforts. We have responded to each point in your comments separately. *Due to word count limitations, we had to shorten some of your reviews when quoting them.* Please find our replies in your corresponding panels. 

-----

Below, we refer to the two variants (from Rev. **j9bD** and **QFPa**) of the proposed tecos-lvm as the model ***"variant"*** (which has the same structure as tecos-lvm, only uses LIF output neurons, other hidden neurons are real-valued), and the model ***"variant Noisy"*** ( same as the ***variant***, but uses noisy LIF output neurons). Results are means computed across repeats. Also, we added Victor Purpura, van Rossum, and SPIKE distances (the lower the better) as evaluation metrics (from Rev. **QFPa** and **qcrE**).



***TABLE 1:*** Comparison with variants on Mov1Ret1. CC stands for Pearson Correlation Coefficients (the higher the better). The TeCoS-LVM, and TeCoS-LVM Noisy models are the main models introduced in our manuscript.

| metric \ model | TeCoSLVM | TeCoSLVM Noisy | variant | variantNoisy | CNN    | IBDisjoint |
| -------------- | -------- | -------------- | ------- | ------------ | ------ | ---------- |
| firing rate CC | 0.579    | 0.727          | 0.468   | 0.720        | 0.690  | 0.653      |
| Victor Purpura | 12.84    | 14.02          | 14.58   | 13.11        | 19.60  | 21.92      |
| van Rossum     | 127.35   | 238.61         | 179.12  | 245.45       | 376.82 | 394.02     |
| SPIKE          | 0.124    | 0.155          | 0.144   | 0.136        | 0.207  | 0.224      |

***TABLE 2:*** Comparison with variants on Mov2Ret2. CC stands for Pearson Correlation Coefficients (the higher the better). The TeCoS-LVM, and TeCoS-LVM Noisy models are the main models introduced in our manuscript.

| metric \ model | TeCoSLVM | TeCoSLVM Noisy | variant | variantNoisy | CNN     | IBDisjoint |
| -------------- | -------- | -------------- | ------- | ------------ | ------- | ---------- |
| firing rate CC | 0.616    | 0.822          | 0.586   | 0.798        | 0.708   | 0.656      |
| Victor Purpura | 22.67    | 28.44          | 23.71   | 28.49        | 39.26   | 38.38      |
| van Rossum     | 574.30   | 1135.81        | 682.51  | 1205.68      | 2638.96 | 2244.98    |
| SPIKE          | 0.123    | 0.153          | 0.133   | 0.148        | 0.221   | 0.221      |

----


Meta-review: The reviewers were unanimous in their appraisal of this paper's contributions as above the bar for acceptance to NeurIPS.  I congratulate the authors on their detailed rebuttals, which were critical in leading some reviewers to raise their scores.  I'm pleased to report that this paper has been accepted to NIPS.  Congratulations!  Please revise the manuscript to address all reviewer comments and questions.
Decision: Accept (poster)