Long-Range Feedback Spiking Network Captures Dynamic and Static Representations of the Visual Cortex under Movie Stimuli
Authors: Liwei Huang
Zhengyu Ma
Liutao Yu
Huihui Zhou
Yonghong Tian
Venue: NeurIPS 2024 poster (2024)
Keywords: Brain-Inspired Computational Model
Spiking Neural Network
Mouse Visual Cortex
Primary area: neuroscience_and_cognitive_science

Abstract: Deep neural networks (DNNs) are widely used models for investigating biological visual representations. However, existing DNNs are mostly designed to analyze neural responses to static images, relying on feedforward structures and lacking physiological neuronal mechanisms. There is limited insight into how the visual cortex represents natural movie stimuli that contain context-rich information. To address these problems, this work proposes the long-range feedback spiking network (LoRaFB-SNet), which mimics top-down connections between cortical regions and incorporates spike information processing mechanisms inherent to biological neurons. Taking into account the temporal dependence of representations under movie stimuli, we present Time-Series Representational Similarity Analysis (TSRSA) to measure the similarity between model representations and visual cortical representations of mice. LoRaFB-SNet exhibits the highest level of representational similarity, outperforming other well-known and leading alternatives across various experimental paradigms, especially when representing long movie stimuli. We further conduct experiments to quantify how temporal structures (dynamic information) and static textures (static information) of the movie stimuli influence representational similarity, suggesting that our model benefits from long-range feedback to encode context-dependent representations just like the brain. Altogether, LoRaFB-SNet is highly competent in capturing both dynamic and static representations of the mouse visual cortex and contributes to the understanding of movie processing mechanisms of the visual system. Our codes are available at https://github.com/Grasshlw/SNN-Neural-Similarity-Movie.

Long-Range Feedback Spiking Network Captures Dynamic and Static Representations of the Visual Cortex under Movie Stimuli
Liwei Huangh.? Zhengyu Ma Liutao Yu?, Huihui Zhou², Yonghong Tian 1,2,3x
School of Computer Science, Peking University, China
Peng Cheng Laboratory. China Schooo of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, China huanglw20@stu. www.00tttuuuu.e..ee.. pku.edu.cm cn,
(mazhy, yult, zhouhh]@per en, yhtian@pku. edu. edu. en
Abstract
Deep neural networks (DNNs) are widely used models for investigating biological visual representations. However, existing DNNs are mostly designed to analyze neural responses to static images, relying feedforward structures and lacking physiological neuronal mechanisms. There limited insight into how the sual cortex represents natural movie stimuli that contain context-rich information. To address these problems, this work proposes the long-range feedback spiking network LLRRaB--SNel which mimics top-down connections between cortical regions and incorporates spike information processing mechanisms inherent to biological neurons. Taking into account the temporal dependence of representa- tions under movie stimuli, we present Time-Series Representational Similarity Analysis (TSRSA) measure the similarity between model representations and visual cortical representations of mice. LoRaFB-SNer exhibits the highest level representational similarity, outperforming other well-known and leading alterna- tives across various experimental paradigms, especially when representing long movie stimuli. We further conduct experiments quantify how temporal structures (dynamic information) and static textures (static information) of the movie stimuli influence representational similarity, suggesting that our model benefits from long- range feedback to encode context-dependes Altogether, LoRaFB- --ee is highly competent in capturing both dynamic and static representations of the mouse visual cortex and contributes to the understanding of movie processing mechanisms of the visual system. Our codes are available wwwp::/hw.....aaao.oooaao//aeaaa
representations just like the brain.
Introduction
Understanding how the biological visual cortex processes information under natural stimuli with computational models is critical scientific goal in visual neuroscience. In this realm, deep neural networks have emerged as the predominant tools [28 32 57 surpassing traditional models, due to their profound success in matching neural representations (58 hierarchies
6 40), revealing functional 53 47 and explaining functionally specialized processing mechanisms
5
4
19 46
[1, [1] of the biological visual cortex. Despite these advancements, research has mainly focused on static image stimuli, leaving gap in the understanding of neural responses to dynamic, context- rich movie stimuli. This oversight is particularly critical given that the visual system receives
CCoresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

predominantly dynamic information and integrates the information in both spatial [23 24] and temporal (20] dimensions. To address this challenge, there need for models with enhanced biological plausibility, capable of encoding the varied types of information inherent in movie stimuli, in order to deepen our comprehension of the processing mechanisms of the visual cortex.
While bottom-up (feedforward) connections dominate visual processing [1 10), many studies have emphasized the crucial role of top-down (feedback) and lateral connections, which are widespread providing diverse coding mechanisms and augmenting temporal This has inspired significant strides in incorporating recur- rent structures into computational models, effectively enhancing their ability to emulate brain- like neural representations and biological behavioral patterns [39] 35| 30 (33. Meanwhile, spiking neural networks (SNNs) [37 with brain-like neuronal computational mechanisms have been developed as more biologically plausible models 21,17 26 3] Using deep SNNs to model the visual cortex has yielded preliminary success [22 60). Nonetheless, attempts to combine these biologically plausible
in the visual cortex 18]
12
48
representation protocols
16
52].
structures and mechanisms are lacking.
In this work, we introduce the long-range feedback spiking network (LoRaFB-SNet) to capture both dynamic and static representations of the mouse visual cortex under movie stimuli. To demonstrate the effectiveness of the long-rang feedback and the spike mechanism in explaining the information processing mechanisms of the visual system, we design a series of experiments for analyses based on representational similarity (Figure D). The main contributions are as follows.
To mimic top-down connections between cortical regions and to utilize the spike mechanism with dynamic properties, we construct novel deep spiking network with long-range feedback connections, significantly improving the biological plausibility of the model.
Considering time-dependent sequences of spikes, we propose Time-Series Representational Similarity Analysis (TSRSA) to measure representational similarity between models and the mouse visual cortex.
For neural representations of the mouse visual cortex under movie stimuli, LoRaFB-SNe trained on the UCF101 dataset significantly outperforms other outstanding alternatives in all experiments, demonstrating the critical role of the structures and mechanisms of our model and the training task.
By varying temporal structures or static textures of movie stimuli fed to models, we quantify the effects of dynamic and static information representational similarity. The results show that LoRaFB-SS -Seet processes movie stimuli to form context-dependent representations in a brain-like like manner, providing deeper insight into the movie processing mechanisms of the visual cortex.
Overall, our proposed novel model achieves the highest neural similarity under movie stimuli and better captures brain-like dynamic and static representations, shedding light on the movie coding strategy of the mouse visual cortex.
2 Related Work
As deep neural networks have attracted widespread interest computational models in visual neuroscience, incorporating more biologically plausible structures and mechanisms has become a major avenue to advance the study of neural representations. We summarize some prominent work.
The deep recurrent network models Efforts construct reasonable recurrent structures [39 35 30) have yielded models good at fitting neural representations and revealing neural dynamics. Recurrent cells from automated search have aided models in predicting the dynamics of neural activity [39]. Notably, CORnet with manually designed architectures has matched the hierarchy of the visual cortex, earning recognition as a leading model this community (35] However, most work has focused on studying neural representations under static stimuli, limited aspect of stimuli,
Besides, while some studies have explored neural responses to movie stimuli 49 only exploited localized lateral connections.
29
they have
The deep spiking network models networks to perform simple temporal tasks 31
Early studies have applied shallow, even single-layer spiking
59 44 and to investigate biological properties 36 50 (6). Some recurrent spiking models have not only emphasized the criticality of homeostatic

---

TSRSA
Representation RSV Matrix
RSV Representation Matrix
VISp VISI VISrl
Correlation
Long- Long-Range Feedback
VISal
V1Spm
Representational Similarity
Connection
VISam
frame i-k
Frame
frame
k
1: original movie
2: shuffled frame
original movie
3: noise image
The overview of our experiments. Six visual cortical regions of the mouse and the long- range feedback spiking network receive the same original movie stimuli to generate the representation matrices. TSRSA is applied to two representation matrices measure representational similarity. In addition, the network receives two modified versions of the movie stimuli (one with broken temporal structures and the other with varied static textures), while the visual cortex still receives the original movie. These two additional experiments are used to quantify the effects dynamic (temporal) and static (textural) information representational similarity. See Section for details.
Figure 1:
regulation in biological neurons (36), but also explained the dynamic regime in the brain associated
with cognition and working memory [56). Recently, deep spiking networks have begun to be used to analyze neural representations, demonstrating enhanced representational similarity [22 and more accurate prediction of the temporal-dynamic trajectories of cortical activity [60] various neural
datasets compared to traditional network counterparts. However, these spiking models are pure feedforward networks and are confined to the study of static stimuli.
3 Methods
3.1
Long-Range Feedback Spiking Network
3.1.1 Architecture
We develop LoRaFB-SNee guided by two principles of biological plausibility. First, we use the LIF neuron (see Appendix A) as the basic unit of our network, which models the membrane potential dynamics of biological neurons (55) and encodes information through spike sequences like the visual cortex. Second, we design the long-range feedback structure mimic tross-regional top-down connections that are widespread the mouse visual cortex (Figure AA). This long-range recurrence complementary spiking neurons with self-accumulation for representing temporal information.
As feedforward connections are dominant in the visual cortex, the backbone LoRaFB-SN is still feedforward, with the recurrent module embedded to introduce long-range feedback (Figure 2 In particular, the construction of the recurrent module is described as follows.
The recurrent module consists of three components: a feedforwarr module, a long-range feedback module, and a fusion module. The feedforward module submodule of the backbone network. consisting of a stack of convolution, pooling, batch normalization, and spiking neurons, which plays major role in abstracting spatial features from visual stimuli and encoding the visual content. This module receives the fused features of the outputs from the long-range feedback module and the previous stage. The long-range feedback module is composed of depthwise transposed convolution, batch normalization, and spiking neurons. On the one hand, depthwise transposed convolution effectively reduces the number of network parameters and upsamples the feature map to match the inputs. On the other hand, some work has shown that such structure might mimic parallel information processing streams in mouse cortical regions and improve representational similarity 22 The fusion module first concatenates the inputs of the current module (the outputs of the

---

Cniput
Dutput
Augoredde
V.lsam
Suhmodule
Racabone
Stage
DW
Auiin
SN
BN
VISI
VI5p
BN
SN
Feedforward
Stage
PW
longonaage firtbark
CONCAT
Fredback
Injut
Figure 2: A. The schematic of six visual cortical regions in the mouse. For brevity, we show parts of the cross-regional feedforward and feedback connections reported from physiological research. B. The schematic of LoRaFB-SNet with the embedded recurrent module. See Section 3. for details.
previous stage) and the outputs of the long-range feedback module in the channel dimension, and then integrates the feedforward and feedback information through pointwise convolution, batch normalization, and spiking neurons. The recurrent module can be formulated as:
R SN(BN(DW(LL.))), A (CONNPW(CONNCTTOO"" R))),,
(2)
0 FA))..
where SN is spiking neurons, BN is batch normalization, DW depthwise transposed convolution, PW is pointwise convolution, CONCAT is channel-wise concatenation. O denotes the outputs of stage at time step I. Similarly, R. and A denote the outputs of the long-range feedbuck and fusion modules respectively. Ft denotes all operations in the feedforward module (Appendix
3.1.2 Pre-Training and Representation Extraction
We pre-train LoRaFB-SNet the UCFI01 dataset and the ImageNet dataset using SpikingJelly [12]
Specifically, for training the video action recognition task on UCF101, each sample (a video clip) contains 16 frames, and one frame is the input at each time step (the simulating time steps T' 16). For training the object recognition task on ImageNet, each sample (an image) is input to networks times (the simulating time steps T 4). See Appendix C for details.
After pre- pre-training the networks, we feed them with the same movie stimuli used in the neural dataset and obtain features from all selected layers. For networks trained UCF101, the entire movie is continuously and uninterruptedly fed into networks in the form of one frame per time step. For networks trained on ImageNet, all frames in the movie considered as independent images and fed into networks separately. Each movie frame is input 4 times, which is consistent with training.
3.2 Representational Similarity Metric
To assess representational similarity between models and the mouse visual cortex at the population level under temporal sequential stimuli, two main problems need to addressed. First, the metric not only analyze static properties of representations, but also preserve temporal relationships of time-series representations to facilitate the analysis of dynamic properties. Second, neurons recorded from the visual cortex are far fewer than units in network layer, making difficult to directly compare representations between the two systems.
We present Time-Series Representational Similarity Analysis (TSRSA) based on Representational
Similarity Analysis (RSA) 34
which has been widely used for the comparison of neural The original RSA focuses on the similarity between neural representations corresponding to each pair of independent stimuli, whereas TSRSA quantifies the similarity between representations corresponding sequential stimuli, taking into account temporal
representations 30
38
46

---

sequential relationships. We detail the implementation of TSRSA as follows. First, we acquire representation matrices R (2........ ......,,T)) ........T
E RN from each layer of networks and each cortical region, where N is the number of units/neurons and T is the number of movie frames. The columns are arranged in chronological order, i.e. F, represents population responses to the movie frame f. Second, we use the Pearson correlation coefficient to compute the similarity between each given column rt and all subsequent columns, yielding the representational similarity vector S, (St1, St2,.... ................................ Stps. ...). The element Stp is Corr (rr, Pt+p), where 0 < p < - t. We then concatenate all vectors to obtain the complete representational similarity vectors Smodel for a network layer and Scortex for a cortical region, which extract both static features and temporal relationships of neural representations. Finally, we compute the Spearman rank correlation coefficient between S mmodel and S cortex to quantify the similarity. Using this metric, we perform layer-by-layee measurement for a network, evaluating all selected layers to visual cortical regions. Notably, when obtaining similarity vectors, we choose the Pearson correlation coefficient for computational efficiency, since both model features and neural data are very high-dimensional. On the other hand, the Spearman rank correlation coefficient is chosen quantify the similarity between two visual systems due to its ability to better capture nonlinear relations.
3.3 Quantifying Effects of Dynamic and Static Information on Representational Similarity
To analyze how visual models process diverse types of information in movic stimuli, we modify temporal structures (dynamic) and static textures (static) of the original movie and obtain variant dynamic or static representations of networks. Mcanwhile, cortical representations are maintained since the movie presented to mice is unchanged. By measuring the similarity between the modified network outputs and the unaltered cortical representations, static information on representational similarity and attempt to glimpse movie processing mechanisms of the visual cortex from the encoding properties of our model. The methods for modifying temporal structures and static textures are as follows.
quantify the effects of dynamic and
Dynamic information We disrupt the frame order of the movie and feed the shuffled movie into networks. producing network representations that differ from the original due distinct dynamic sequential information. To obtain frame order with different levels of alteration while avoiding extreme chaos (e.g., moving the first frame to the last), we divide the entire movie into multiple windows with the same number of frames and randomly shuffle the frames only within each window. We conduct 10 sets of experiments with different window sizes. Each set comprises 10 trials to provide enough statistical power. We calculate the level of chaos for every trial, which is defined as I 1-r, where r is the Spearman rank correlation coefficient between the disrupted frame order and the original order. Since the movie presented to mice is invariant, we rearrange the network representation matrix to the original order to ensure that matches the order of the mouse representation matrix when conducting TSRSA. In this way, we maintain the correspondence between static representations of two systems, while isolating changes in dynamic representations of networks. This allows us to focus on evaluating the effects of dynamic information.
Static information We randomly select a certain proportion of movie frames and replace them with Gaussian noise images whose static textures are completely different from both network training scenes and biological experiment stimuli. We then feed the new movie into networks to obtain variant static representations. To minimize dense local replacement and preserve as much dynamic information as possible, the movie divided into equal-sized windows and only one frame in each window is replaced. We similarly run 10 experimental sets with different window sizes, each consisting of 10 trials. The ratio of replacement is the inverse of the number of frames per window. Replacing movie frames results in change in static representations of networks, while the overall frame order remains the same as the original. Admittedly, changing static information will inevitably change temporal structures as well. We attenuate this influence by distributing noise images as sporadically as possible and emphasize how static information affects representational similarity.
4 Experiments
4.1
Neural Dataset
We conduct analyses using subset of the Allen Brain Observatory Visual Coding dataset 9 This dataset, recorded by Neuropixel probes, consists of neural spikes with high temporal resolution
48
5

---

A
ImageNet
LoRaFB Stee
CORnet
2p-CCC
SEW-ResNet
SEW-RosNet ImageeNet
Moviel
Mevic2
Moviel
Movie2
TSRSA Seare
TSRS Store
13330
0.5000
0L3483
0.2937
0. 0.1720
0.5300
0.48555
N. 2302
0.0939
0.3700
0:27177
0.25
D
Moviel (30)
Model
Moskd
1.
LARAFR-SNN
1,5
Coltect
TSRSA Soare
Rechet-2-CCC
1.4
0.33
26
10
Movir2
1.3
1,2
1.0
Laltali-SNNat COOlad Lattali-NNaa RaCNe-p-CCC
1.8 40
130
100 138
90
Clip 1lpplann Lengih (s)
Clip Length (s)
Figure 3: A. The TSRSA scores of three models pre-trained on UCF101. B. The TSRSA scores of feedback/feedforwaat spiking networks pre-trained on UCF101/ImageNet. C. The TSRSA score curves of three models pre-trained on UCF101 for different movie clip lengths. We randomly select continuous movie clips of different lengths and plot TSRSA scores between models' and the visual cortex's 's representations corresponding random seeds. D. The ratios of our model's scores to those of the alternative models for different clip lengths. The ratio tends to increase for longer clips.
these clips. The
bar is the standard error 10
from mouse visual cortical regions (VISp, VISI, VISrI, VISal, VISpm, VISam, see Appendix D for details). Each region contains hundreds of recorded neurons to minimize the effects of neuronal variability, facilitating the analysis of neural population representations. The visual stimuli presented to the mice consist of two movies, one for 30s (Movie/), repeated for 20 trials, and the other for 120s (Movie2), repeated for 10 trials. The frame rate of both movies 30Hz. neural responses with the peristimulus time histogram (PSTH), sum the number of spikes in each movie frame and take the average over all trials for each neuron. To focus on neurons that are more responsive to visual input, we excluded those firing less than empirical threshold of 0.5 spikes/second 42 54].
pre-process
As discussed in the study (9] proposing this dataset, the class of neurons responsive to movie stimuli found in all six cortical regions. Therefore, for a given model, we take the maximum scores across layers per region and report the average score over six regions as the model's TSRSA score.
4.2 Models for Comparisons
We select three models for comprehensive comparisons demonstrate the effectiveness of each character in LoRaFB-SNet
CORnet It is one of the most influential recurrent networks [35 for modeling the visual cortex and has been used as benchmark in many studies 77. The prototype was trained on ImageNet and here we pre-train it on UCF101 with the same training procedure as LoRaFB-SNet. The comparison with CORnet, which has recurrent connections but lacks the spike mechanism, aims to show the critical role of spiking neurons in our model.
ResNet-2p-CPP This model emulates the ventral and dorsal pathways of the mouse visual cortex with parallel pathway architectures [ pre-trained on UCF101. While our model incorporates spiking neurons and recurrent connections handle sequential inputs of any length, ResNet-2p-CPC uses

---

Table 1: The neural ceilings and the scores of all models under two movie stimuli. In brackets are the percentages of model scores compared to the neural ceiling.
Neural Ceiling
CORnet
ResNet-2p-CPC
SEW-ResNet
LoRaFB-SNNt
Moviel Movie2
0.801 0.616++0.009
0.506 (61.6%)
0.348 (42.4%)
0.452 (55.0%)
0.520 (63.3%
0.223 (36.2%)
0.172 (27.9%)
0.094 (15.2%)
0.283 (45.9%)
fixed-length filters to process temporal data, allowing comparison between different approaches to dynamic information processing.
SEW-ResNet It pure feedforward spiking network [3] that has shown the best performance in fitting neural representations of the visual cortex [22 60 We pre-train it on both UCF101 and ImageNet. Since the comparison with it focuses on the role of feedback connections, our model adopts identical feedforward structures to its.
4.3 Results
4.3.1
Comparisons of Representational Similarity
We perform representational similarity analyses on the neural dataset under two movie stimuli respectively, and refer to the two cases as Moviel and Movie2.
As shown in Figure AA among the three models pre-trained on UCFIOL, LoRaFB- Beeeet outperforms the other two well-known bio-inspired models. Specifically, our model performs moderately better than CORnet (Moviel: +2.8%; Movie2: +26.8% and significantly better than ResNet-2p-CPC (Moviel: +49.3%; Movie2: +64.4%). To further quantify how similar our model's representations are brain representations, we obtain neural ceilings by randomly splitting the neural data into two halves and computing the TSRSA score [T Our model attains 63.3% and 45.9% of the ceilings and achieves great improvement over other models, which suggests that our model effectively captures neural representations of the brain and is meaningfully closer to the mouse visual cortex. We also report similarity scores of our model to each cortical region (Appendix B and show that our model yields robustness across different regions. In addition to the population representation analysis, we use linear regression fit model representations to temporal profiles of individual biological neurons and compute R²2 as the similarity, the results of which also demonstrate the superiority of our models (Appendix[F)
To emphasize the joint role of long-range feedback connections and pre-training on a video dataset, we compare feedback/feedforwarr spiking networks pre-trained on image/video datasets (Figure 3). For models trained on ImageNet, LoRaFB-SNel and SEW-ResNet achieve comparable TSRSA scores, suggesting that feedback connections do not have a significant effect on models trained on static images. For models trained on UCF101, LoRaFB-SNet performs significantly better than SEW-ResNet (Moviel: +1..2%; Movie2: +201.1%). Besides, when comparing models trained on UCF101 and ImageNet, we find that our model outperforms those trained on ImageNet, while SEW-ResNet instead performs worse. This may be explained by the fact that SEW-ResNer trained on a video dataset not only fails to capture dynamic information effectively, but also is compromised on the ability to represent static information. Taken together, it is the combination of long-range feedback connections and pre-training temporal features and capture representations of the visual cortex under movie stimuli.
video dataset that enables our model to better extract
Notably, from the above results, we find that our model gains more pronounced advantages in similarity when the movie stimuli are longer. To further corroborate this phenomenon, we randomly select movie clips of different lengths from the original movic stimuli and compute TSRSA scores between models' and the visual cortex's representations corresponding to these movie clips. As shown in Figure CC. TSRSA scores show decreasing trend with increasing clip length, suggesting that it more difficult for models to capture brain-like representations when movie stimuli are longer. This result is reasonable since longer movie stimuli increase the diversity of neural response patterns the visual cortex. Nonetheless, our model consistently outperforms the other two models across all clip lengths and shows increasing improvement ratios as movie clips get longer (Figure 3 P). These results suggest that the more biologically plausible structure and mechanism may allow LoRaFB- SNet to efficiently process accumulated visual information longer time scales, just like the brain.
7

---

A
B
NSKS Store
TSRSA Scure
Portaral Serees
9.35
0.35
Drep (M
0:2
LeluFB-SNe ECPIEI
SEN--RacNe DEFIOI
LaRaFI-SNNA Inaepchic
0.35
10
Level of Chans
16
Raiin Rephacement
Figure 4: A. The TSRSA score curves of LoRaFB-SNet and SEW-ResNet trained on UCF101 with different levels of chaos (the main plot) and the drop rate curves of experimental scores compared with the original score (the subplot). The horizontal coordinates both plots are the level of chaos. In the main plot, the dashed horizontal lines indicate the original scores between models and the mouse visual cortex under the original movie. Each large point on the curve indicates the average result of a set of experiments, and each small point indicates the result of one trial in set. The vertical error bar is the 99% confidence interval of the score over 10 trials, while the horizontal error bar is the 99% confidence interval of the level of chaos. In the subplot, the curves show the average drop rate and the average level of chaos over 10 trials for all experimental sets. LoRaFB -Sett shows a large drop in scores while SEW-ResNe shows small drop. B. The TSRSA score curves of LoRaFB-SNN trained on UCF1 OCF1101/Imageet with different ratios of replacement. The elements in the main plot indicate similar content as in A. LoRaFB-Shi similar decreasing trend in scores. C. The TSRSA scores of feedhack/feeddorwaat spiking networks trained on UCFI0I Cceeee// for the neural dataset under natural scene stimuli.
trained UCF101 and ImageNet both exhibit
Specifically, compared to CORnet which also has recurrent connections, LoRaFB-SNet utilizes the spike firing mechanism to form spike-sequential coding and incorporates long-range feedback to enhance the ability to extract temporal features. Compared to ResNet-2p-CPC, LoRaFB-SNG. exploits membrane potential dynamics and recurrent modules to process dynamic information, more flexibly handling movies of different lengths without being limited by the temporal filter size.
Overall, our model, LoRaFB-SNet. outperforms other influential and outstanding alternatives across multiple experimental paradigms, suggesting that biologically plausible long-range feedback connec- tions and spiking neurons significantly contribute better modeling neural representations of the mouse visual cortex, especially when processing long- -ur-duration movies.
4.3.2 Experiments to Analyze the Effects of Dynamic and Static Information
For the experiments of shuffling movie frames to change dynamic information, we compare the results of LoRaFB-SNet and SEw-ResNet trained on UCF101 (Figure 4A) to investigate the importance of brain-like dynamic representations for modeling the visual cortex. As the curves in the main plot show, any alteration from the original frame order results a lower TSRSA score between models and the mouse visual cortex than the original score, and the score decreases as the level of chaos increases. Frame shuffling disrupts the continuity and temporal structures of the movie, leading to changes in dynamic representations of models. Considering that we align representation matrices of models and the visual cortex along the movie frame dimension when applying TSRSA, the decrease in similarity is mostly attributed to variations in dynamic representations, while the effect of static representations is negligible.
Furthermore, by comparing drop rate curves between LoRaFB-SNet and SEW-ResNet, we find out that the drop rate of LoRaFB-SNet increases with the level of chaos and eventually reaches staggering 46.5%. However, the drop rate of SEW-ResNet is consistently lower than that of LoRaFB-SNet and the maximum is even lower than 9%. These results reveal two significant findings. First, our model captures biological dynamic representations very well and is sensitive to disruptions in dynamic information. Second, although SEW-ResNet is also trained on a video dataset, the vast majority of its representations depend on static rather than dynamic information. Therefore, its similarity score is less affected by temporal structure distortions. In addition, we perform the same experiment CORnet that also has recurrent connections (Figure A of Appendix G which

---

Table 2: The TSRSA scores in different cases for ablation studies under Moviel.
Continuous
Discontinuous
ImageNet
No-spike
SEW-ResNet
LoRaFB-SNN
Score
0.524
0.474
0.486
0.498
0.452
0.520
shows a similar result to our model. In conclusion, these results underscore the superior capability of LoRaFB-SNet, due to its long-range feedback connections, to capture temporal relationships and represent dynamic information in more context-dependee manner, which is likely to be the potentially crucial mechanism for processing movie stimuli in the mouse visual cortex.
In addition to the analysis of dynamic representations, the role of static representations in modeling the visual cortex should not be overlooked. In the experiments of replacing frames modify static information, we compare the results of LoRaFB-Shh trained on UCF101 and ImageNet (Figure BB). Similar the results in the experiments of shuffling movie frames, TSRSA scores are mostly lower than the original in the case of replacing noise frames, and the score decreases as the ratio of replacement increases. We also use other types of noise images for replacement, which exhibits similar impact (Figure B of Appendix G.. Obviously, static representations of models change lot due to the totally different static textures between the original movie frames and the noise images, resulting in a decrease in similarity. While scores of both LoRaFB-SNN trained on UCF101 and ImageNet show a similar decreasing trend with the increasing replacement rate, the former is steadily higher than the latter. For the model trained on an image dataset, the movie frames are treated as independent individuals, so model representations completely depend on static information and there is no temporal relationship between representations of two frames. In contrast, the model trained on video dataset encodes both static and dynamic information to form representations. Consequently, although the high replacement ratio also affects static representations of the UCF101-trainet LoRaFBB- SNet, its dynamic representations to original movie frames may moderate the drop in similarity score to some extent.
In addition natural movie stimuli, we measure the representational similarity between models and the mouse visual cortex under static natural scene stimuli (the neural data are also from the Allen Brain Observatory Visual Coding dataset). As shown in Figure 4C and Table o of Appendix G our model also outperforms other models in encoding static information and yields more brain-like representations under static stimuli. Besides, models trained on UCF101 perform better than those trained on ImageNet, even in representing static natural scene stimuli.
In summary, LoRaFB-SNC trained on video dataset is able extract spatio-temporal features simultaneously and represent dynamic and static information of movie stimuli in a way more similar the mouse visual cortex. In particular, the powerful ability encode temporal relationships makes LoRaFB-SNet's representations more context-deped -enexxt-eepeneet,, which may provide new insights into the mechanisms of movie information processing in the visual system.
4.3.3 Ablation Studies for Training Datasets and Model Structures
To figure out whether the static content of UCF101 data or the temporal structure of continuous videos benefits LoRaFB-SNet to capture brain-lie like representationss we build two datasets based on UCF101 to pre-train LoRaFB-SNN and compare their similarity. One dataset consists of continuous videos from UCF101, while the other consists of disordered and discontinuous videos made up of randomly selected frames from UCF101. As shown in Table 2 LoRaFB- videos outperforms that trained on discontinuous videos, suggesting that the temporal structure rather than the static content plays an important rolc. Furthermore, LoRaFB-SNN videos performs even worse than that trained on ImageNet, showing that the static content of UCF101 is not closer to the test movie stimuli than that of ImageNet in terms of data distribution.
trained on continuous
trained on discontinuous
To consolidate the conclusion about the effectiveness of spiking neurons and long-range feedback connections, we perform direct comparisons with two models trained on UCF101, one without spiking neurons but with the same structure as LoRaFB-SNet (No-spike), and the second with the same spiking neurons and feed-forward structure but without feedback connections (SEW-ResNet), The results Table Tsho show that the model lacking either of these characters performs worse than our model. providing further evidence that both characters of our model play critical role.

---

Discussion
In this work, we propose LoRaFB-SNNt (Long-Rang Feedback Spiking Network) to model neural representations of the mouse visual cortex under movie stimuli. Incorporating long-range feedback connections and spiking neurons, LoRaFB-SNN B-ee offers more biologically plausible architectures and processing mechanisms. Tested the mouse neural dataset with TSRSA, LoRaFB-SNet significantly surpasses existing outstanding and influential computational models across multiple experimental paradigms. We extend the analysis to dynamic and static information representations of networks and the visual cortex with two meticulously designed experiments, providing evidence that LoRaFB-SNV more brain-like manner. Specifically, our model
able encode dynamic and static information efficiently processes visual stimuli with long duration and forms context-depenee representations. Overall, LoRaFB-SS
effectively captures dynamic and static representations of the visual cortex and helps to reveal movie processing mechanisms in the visual system.
For spiking neurons in our model, we hypothesize that membrane potential dynamics and spike information encoding are helpful to better capture brain-like representations, which may require further analyses support. Benefiting from feedback connections, our model gains particular advantages in representing dynamic information. Some studies [45] have suggested that the effects of recurrent connections in the visual cortex vary over time, influencing dynamic representations. The specific contributions and interplay of feedforward and feedback connections to the encoding protocol remain be explored. In conclusion, while our model explains some mechanisms of information processing in the visual cortex, more biologically plausible mechanisms, such as local recurrent connections and sophisticated neuronal models, deserve be introduced and studied.
LoRaFB-SNN demonstrates significant efficacy in modeling visual representations of mice. As a biologically plausible spiking network, it holds potential general and promising framework for studying the visual cortex of other species and investigating other sensory modalities, helping to understand more intricate neural computations.
Acknowledgments and Disclosure of Funding
This work is supported by grants from the Beijing Science and Technology Plan: (No. 241 1000044224411, the National Natural Science Foundation of China (No. 62027804, No. 62425101, No. 62332002, No. 62088102 102, No. 62206141, and No. 62236009), and the major key project of the Peng Cheng Laboratory (PC12021A13).
References
[1] Shahab Bakhtiari, Patrick Mincault, Timothy Lillicrap, Christopher Pack, and Blake Richards. The functional specialization of visual cortex emerges from training parallel pathways with self-supervised predictive learning. In Advances in Neural Information Processing Systems 34, pages 25164-25178. 2021.
[2] Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Leg- enstein, and Wolfgang Maass. A solution to the learning dilemma for recurrent networks of spiking neurons. Nature communications, 11(1):3625, 2020.
[3] Romain Brette, Michelle Rudolph, Ted Carnevale, Michael Hines, David Beeman, James M Bower, Markus Diesmann, Abigail Morrison, Philip H Goodman, Frederick C Harris, et al. Simulation of networks of spiking neurons: a review of tools and strategies. Journal of computational neuroscience, 23(3):349-39) 2007.
[4] Santiago A Cadena, George H Denfield, Edgar Y Walker, Leon A Gatys, Andreas S Tolias, Matthias Bethge, and Alexander Ecker. Deep convolutional models improve predictions of macaque vI responses to natural images. PLoS computational biology, 15(4):e100697) 2019,
[5] Charles F Cadieu, Ha Hong, Daniel LK Yamins, Nicolas Pinto, Diego Ardila, Ethan A Solomon, Najib Majaj, and James DiCarlo. Deep neural networks rival the representation of primate cortex for core visual object recognition. PLoS computational biology, 10(12): 2014.
10

---

[6] Le Chang, Bernhard Egger, Thomas Vetter, and Doris Y Tsao. Explaining face representation in the primate brain using different computational models. Current Biology, 31(13):2785-22) 2021.
[7] Colin Conwell, David Mayo, Andrei Barbu, Michael Buice, George Alvarez, and Boris Katz. Neural regression, representational similarity, model zoology & neural taskonomy at scale in rodent visual cortex. In Advances Neural Information Processing Systems 34, pages 5590--6077, 2021.
[8] Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David Cox, and James DiCarlo. Simulating primary visual cortex at the front of cnns improves robustness to image perturbations. Advances in Neural Information Processing Systems 33, pages 13073-13087, 2020.
[9] Saskia EJ de Vries, Jerome Lecoq, Michael Buice, Peter A Groblewski, Gabriel K Ocker, Michael Oliver, David Feng, Nicholas Cain, Peter Ledochowitsch, Daniel Millman, et al. A large-scale standardized physiological survey reveals functional organization of the mouse visual cortex. Nature neuroscience, 23(1):138-151 2020.
[10] James DiCarlo, Davide Zoccolan, and Nicole C Rust. How does the brain solve visual object recognition? Neuron, 73(3)4115-34, 2012.
[11] Katharina Dobs, Julio Martinez, Alexander JE Kell, and Nancy Kanwisher. Brain-like func- tional specialization emerges spontaneously in deep neural networks. Science advances, 3(1a))bb13)113 2022.
[12] Wei Fang, Yanqi Chen, Jianhao Ding, Zhaofei Yu, Timothée Masquelier, Ding Chen, Liwei Huang, Huihui Zhou, Guoqi Li, and Yonghong Tian. Spikingjelly; An open-source machine learning infrastructure platform for spike-based intelligence. Science Advances, 9400) 1480, 2023.
[13] Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. In Advances in Neural Information Processing Systems 34, pages
2021.
[14] Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. Incorporating learnable membrane time constant enhance learning of spiking neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2661-2671, 2021.
[15] Daniel Felleman and David C Van Essen. Distributed hierarchical processing in the primate cerebral cortex. Cerebral cortex (New York, NY: 1991), 1(1):1-47, 1991.
[16] Winrich Freiwald and Doris Y Tsao. Functional compartmenntllzztioo and viewpoint generalization within the macaque face-processing system. Science, 330(6005):8558855 -511,,55551500 2010.
[17] Wulfram Gerstner and Werner Kistler. Spiking neuron models: Single neurons, populations, plasticity. Cambridge university press, 2002.
[18] Charles Gilbert and Wu Li. Top-down influences on visual processing. Nature Reviews Neuroscience, 14(5):350-366 03600 2013.
[19] Umut Güçlü and Marcel AJ van Gerven. Deep neural networks reveal gradient in the complexity of neural representations across the ventral stream. Journal of Neuroscience, 35(27):10005-00014 2015.
[20] Uri Hasson, Funice Yang, Ignacio Vallines, David J Heeger, and Nava Rubin. A hierarchy of temporal receptive windows in human cortex. Journal of Neuroscience, 28(10):2539-2555 2008.
[21] Alan Hodgkin and Andrew Huxley, A quantitative description of membrane current and its application conduction and excitation in nerve. The Journal of physiology, 117(4):500, 1952.
11

---

[22] Liwei Huang, Zhengyu Ma, Liutao Yu, Huihui Zhou, and Yonghong Tian. Deep spiking neural networks with high representation similarity model visual pathways of macaque and mouse. In The Thirty-Seventl AAAI Conference Artificial Intelligence, pages 31-39, 2023.
[23] David Hubel and Torsten Wiesel. Receptive fields of single neurones the cat's striate cortex. The Journal of physiology, 148(3):574, 1959.
[24] David H Hubel and Torsten N Wiesel. Receptive fields and functional architecture of monkey striate cortex. The Journal of physiology, 195(1):115-243, 1968.
[25] Jean-Michel Hupe, Andrew James, Pascal Girard, Stephen G Lomber, Bertram Payne, and Jean Bullier. Feedback connections act on the early part of the responses in monkey visual cortex. Journal of neurophysiology, 85(1):134-144 2001.
[26] Eugene M Izhikevich. Which model to use for cortical spiking neurons? IEEE Transactions on Neural Networks, 15(5):1063-1070 2004.
[27] Kohitij Kar, Jonas Kubilius, Kailyn Schmidt, Elias B Issa, and James J DiCarlo. Evidence that recurrent circuits are critical to the ventral stream's 's execution of core object recognition behavior. Nature neuroscience, 22(6))974-983, 2019.
[28] Seyed-Mahdi Khaligh-Razza and Nikolaus Kriegeskorte. Deep supervised, but not un- supervised, models may explain it cortical representation. PLoS computational biology, 10(11):e 2014.
[29] Meenakshi Khosla, Gia H Ngo, Keith Jamison, Amy Kuceyeski. and Mert R Sabuncu. Cortical response to naturalistic stimuli is largely predictable with deep neural networks. Science Advances, 7(22):eabe7547, 2021.
[30] Tim C Kietzmann, Courtney J Spoerer, Lynn KA Sörensen, Radoslaw M Cichy, Olaf Hauk, and Nikolaus Kriegeskorte. Recurrence is required capture the representational dynamics of the human visual system. Proceedings of the National Academy of Sciences, 116(43)21185422866 2019.
[31] Robert Kim, Yinghao Li, and Terrence J Sejnowski. Simple framework for constructing functional spiking recurrent neural networks. Proceedings of the national academy of sciences, 1 (T(45)):2811-22888 2019.
[32] Nikolaus Kriegeskorte. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annual review of vision science, 1:417-446, 2015.
[33] Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity cnanysis the branches of systems neuroscience. Frontiers in systems neuroscience, 2:4, 2008
[34] Nikolaus Kriegeskorte, Marieke Mur, Douglas A Ruff, Roozbeh Kiani, Jerzy Bodurka, Hossein Esteky, Keiji Tanaka, and Peter A Bandettini. Matching categorical object representations in inferior temporal cortex of man and monkey. Neuron, (116):1126-1141 2008.
[35] Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Rishi Rajalingham, Ha Hong, Najib Majaj. Elias Issa, Pouya Bashivan, Jonathan Prescott-Ron Kailyn Schmidt, et al. Brain-like object recognition with high-performing shallow recurrent anns. Advances in Neural Information Processing Systems 32, pages 12785-12796, 2019.
[36] Zhengyu Ma, Gina G Turrigiano, Ralf Wessel, and Keith B Hengen. Cortical circuit dynamics are homeostatically tuned to criticality in vivo. Neuron, 104(4):555666, 2019.
[37] Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural networks, 10(9): 1997.
[38] Johannes Mehrer, Courtney Spoerer, Nikolaus Kriegeskorte, and Tim C Kietzmann. Individual differences among deep neural network models. Nature communications, 11(1):5725, 2020.
12

---

[39] Aran Nayebi, Daniel Bear. Jonas Kubilius, Kohitij Kar, Surya Ganguli, David Sussillo, James J DiCarlo, and Daniel Yamins. Taskedriven convolutional recurrent models of the visual system. In Advances in Neural Information Processing Systems 31, pages 5295-5306 2018.
[40] Aran Nayebi, Nathan CL Kong, Chengxu Zhuang, Justin L Gardner, Anthony M Norcia, and Daniel LK Yamins. Mouse visual cortex as a limited resource system that self-learns an ecologically-general representation. PLOS Computational Biology, (010)ee0011006 2023.
[41] Emre O Neftei, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient- -based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51-63 2019.
[42] Lucas Pinto, Michael J Goard, Daniel Estandian, Min Xu, Alex C Kwan, Seung- Lee, Thomas C Harrison, Guoping Feng, and Yang Dan. Fast modulation of visual perception by basal forebrain cholinergic neurons. Nature neuroscience,
2013.
[43] Rishi Rajalingham, Aída Piccato, and Mehrdad Jazayeri. Recurrent neural networks with explicit representation of dynamic latent variables can mimic behavioral patterns in a physical inference task. Nature Communications, 13(1):5865, 2022.
[44] Arjun Rao, Philipp Plank, Andreas Wild, and Wolfgang Maass. A long short-term memory for ai applications in spike-based neuromorphic hardware. Nature Machine Intelligence, 4(5):467-479, 2022.
[45] João D Semedo, Anna Jasper, Amin Zandvakili, Aravind Krishna, Amir Aschner, Christian K Machens, Adam Kohn, and Byron M Yu. Feedforward and feedback interactions between visual cortical areas use different population activity patterns. Nature communications, 13(1):1099, 2022.
[46] Jianghong Shi, Eric Shea-Brown, and Michael Buice. Comparison against task driven artificial neural networks reveals functional properties in mouse visual cortex. In Advances in Neural Information Processing Systems 32, pages 5765-5775, 2019.
[47] Jianghong Shi, Bryan Tripp, Eric BieeaB Stefan Mihalas, and Michael A. Buice. Mousenet: biologically constrained convolutional neural network model for the mouse visual cortex. PLOS Computational Biology, 18(9)ee001422)
[48] Joshua Siegle, Xiaoxuan Jia, Séverine Durand, Sam Gale, Corbett Bennett, Nile Graddis, Greggory Heller, Tamina K Ramirez, Hannah Choi, Jennifer A Luviano, et al. Survey of spiking in the mouse visual system reveals functional hierarchy. Nature, 592(7852):86-99, 2021.
[49] Fabian Sinz, Alexander S Ecker, Paul Fahey, Edgar Walker, Erick Cobos, Emmanouil Froudarakis, Dimitri Yatsenko, Zachary Pitkow, Jacob Reimer, and Andreas Tolias. Stim- ulus domain transfer in recurrent models for large scale cortical population prediction on video. In Advances in Neural Information Processing Systems 31, 2018
[50] Laura E Suárez, Blake Richards, Guillaume Lajoie, and Bratislav Misic. Learning function from structure in neuromorphic networks. Nature Machine Intelligence, 3(9):771-788) 2021.
[51] Yasuko Sugase, Shigeru Yamane, Shoogo Ueno, and Kenji Kawano. Global and fine information coded by single neurons in the temporal visual cortex. Nature, 400(6747):869 873, 1999.
[52] Hanlin Tang, Martin Schrimpf, William Lotter, Charlotte Moerman, Ana Paredes, Josue Or- tega Caro, Walter Hardesty, David Cox, and Gabriel Kreiman. Recurrent computations for visual pattern completion. Proceedings of the National Academy of Sciences, 115(35)88835-88 8840, 2018.
[53] Kasper Vinken and Hans Op de Beeck. Using deep neural networks to evaluate object vision tasks in rats. PLOS computational biology, 1703)e1008714 2021.
[54] Brice Williams, Joseph Del Rosario, Tomaso Muzzu, Kayla Peelman, Stefano Coletta, Edyta Bichler, Anderson Speed, Lisa Meyer-Baese, Aman B Saleem, and Bilal Haider. Spatial modulation of dark versus bright stimulus responses in the mouse visual system. Current Biology, 31(18):4172-4177 2021.
13

---

[55] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. In The Thirty- Third AAAI Conference on Artificial Intelligence, pages 131 1311-, 2019.
[56] Xiaohe Xue, Ralf D Wimmer, Michael M Halassa, and Zhe Sage Chen. Spiking recurrent neural networks represent task-relevant neural sequences in rule-dependent computation. Cognitive Computation, pages 1-23, 2022.
[57] Daniel LK Yamins and James DiCarlo. Using goal-driven deep learning models understand sensory cortex. Nature neuroscience, 19(3):356-365, 2016.
[58] Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-optimizzl hierarchical models predict neural responses higher visual cortex. Proceedings of the national academy of sciences, (11(23)88619-88)) 8624, 2014.
[59] Bojian Yin, Federico Corradi, and Sander M Bohté. Accurate and efficient classification with adaptive spiking recurrent neural networks. Nature Machine Intelligence, 3(10):905-913, 2021.
[60] Jie Zhang, Liwei Huang, Zhengyu Ma, and Huihui Zhou. Predicting the cemporal-yyname trajectories of cortical neuronal responses in non-human primates based on deep spiking neural network. Cognitive Neurodynamics, pages 1-12, 2023.
14

---

A Spiking Neuron Model
The spiking neuron model we used in LoRaFB-SNed is the Leaky Integrate-and-Fiig (LIF) model. As mentioned in [3) Vi, X, and S₁ denote the state (membrane voltage), input (current) and output (spike) the spiking neuron model respectively at time step t, and the dynamics of the LIF model can be described as follows:
H, = Vi-1 + [xx (V2-1 Vreset));
(4)
S. = O - Vihrest):
(5)
Vi H. St) + Vteset
(6)
While Vi is the membrane voltage after the trigger of spike, H, is also the membrane voltage, but after charging and before spike firing. 7 the membrane time constant to control the rate of spiking neuron leakage. (xx) is the unit step function, St equals if H, is greater than equal to the threshold voltage Vihresh and 0 otherwise. Meanwhile, V, is reset to Vreset when spike fires. Here, and Vreset O, which are widely used empirical values for the visual
set 2, Vihresh
task training.
Considering that (zx) is non-differentiable at O, we use the inverse tangent function as the surrogate gradient function 41 to approximate the derivative function during propaaapoaat.nnn...
B Detailed Structure of Feedforward Module
The feedforward module is submodule of the backbone network, which is made up of a stack of convolution, pooling, batch normalization, and spiking neurons. We adopt the residual block in SEW-ResNet [[3), which cures the vanishing/explodinn gradient problems of spiking networks. The feedforward modules of all stages in LoRaFB-SNG share this structure but with different hyperparameters (Figure
Input from fusion module
CONV
BN
SN
CONV
BN
SN
Output
long-range feedback module
Figure 5: Detailed structure of the feedforward module. CONV is convolution. BN is batch normalization. SN is spiking neurons. denotes an element-wise operation with two spike features.
C Pre-Training Implementation
In order for networks extract meaningful features from the visual input, we pre-train them with two visual tasks. Specifically, LoRaFB-SMM -Seet SEW-ResNet and CORnet are pre-trained with the video action recognition task on UCF101, and the first two are also pre-trained with the object recognition task on ImageNet. Notably, ResNet-2p-CPP has already been pre-trained on UCF101 in the original paper [ and we directly adopt its open-source parameters. The pre-training implementation is detailed as follows.
Pre-Training on UCF101 In the pre- vre-trannig procedure on UCF101, all video frames are resized to 224 224 and each sample video clip of 16 frames that are continuously fed into networks. Since the input at each time step is video frame, the simulating time steps T of networks are 16. All networks are trained on UCF101 for 100 epochs on 8 GPUs (NVIDIA V100) with a mini-batch size of 32. The optimizer is SGD with momentum of 0.9 and a weight decay of 0.0001. The initial learning rate is 0.1 and we apply línear warm-up for epochs. We decay the learning rate with cosine anncaling, where the maximum number of iterations is equal to the number of epochs.
15

---

Pre- Tre-Training on ImageNet In the pre-training procedure on ImageNet, each image is resized to 224 x 224 and fed into spiking networks 4 times. In other words, for each sample, spiking networks are simulated with time steps 7 4, and the input is the same image at each time step. We train all spiking networks on ImageNet for 320 epochs
GPUs (NVIDIA V100) with a mini-batch size of 32. We also use SGD as the optimizer and set the momentum to 0.9 and the weight decay to O. The initial learning rate is 0.1 with a linear warm-up for epochs. Cosine annealing with the maximum number of iterations equal to the number of epochs is also applied decay the learning rate.
D Supplementary Information of the Neural Dataset
In this work, we use a subset of the Allen Brain Observatory Visual Coding dataset 9 48 recorded from six visual cortical regions of the mouse with Neuropixel probes. The full names and abbrevia- tions of all cortical regions are listed in Table Besides, we present the number of neurons before and after the exclusion of those firing less than 0.5 spikes/s under movie stimuli. The exclusion criteria resulted in the removal of no more than 10% of neurons from each region, suggesting that most neurons are responsive.
Table Detailed information of the neural dataset.
Cortical Region
Abbreviation
Total Neurons
Neurons after Exclusion (Moviel/Movie)
primary visual cortex
VISp
2015
1880/1854
lateromedial area
VISI
933
861/851
rostrolateral area anterolateral area
VISrl
1415
1302/1267
VISal
1553
1445/1420
posteromedial area anteromedial area VISam
VISpm
879
820/807
1506
1394/1366
E
TSRSA Scores of LoRaFB-SNe to Each Cortical Region
We report TSRSA scores of our model to each cortical region of the mouse. The results show that our model achieves stable scores across regions and there is significant difference.
Table 4: The scores LoRaFB-SNC to each cortical region under two movie stimuli.
VISp
VISI
VISrl
VISal
VISpm
VISam
Moviel 0.527
0.501
0.505
0.515
0.530
0.544
Movie2 0.222
0.262
0.296
0.300
0.301
0.315
F Results of Linear Regression for Individual Neurons
We use linear regression to model representations to temporal profiles of individual neurons and report R² as the similarity. The results show that our model consistently performs better than alternative models on this metric (Table 5). In addition, we present some examples of real temporal profiles of biological neurons and the regressed results of model (Figure 6) which also demonstrate the good fitting performance of our model.
Table 5: The similarity scores of all models using the regression- metric.
CORnet
ResNet-2p-CPC
SEW-ResNet
LoRaFB-SA
Moviel
0.4326
0.4167
0.4241
0.4335
Movie2
0.1790
0.1751
0.1720
0.1836
16

---

A
V15P
VISI
VISri
Average Firing
VISal
ViSpm
ViSam
Moviel (30s)
B
VISP
VISI
ViSri
7.5
Firing
VISal
V1Sppm
Movie2 (120s)
Figure 6: The real and regressed temporal profiles of individual biological neurons. We choose one neuron each cortical region as an example.
G Additional Results of Experiments for Dynamic and Static Information
For the experiments of changing dynamic information, CORnet yields similar result to model (Figure 7A), which solidifies our conclusion about the effectiveness of feedback connections. For the experiments of modifying static information, experiments, including noise images from the uniform distribution and static natural images from the Allen Brain Observatory Visual Coding dataset. The results of UCF101 -trinned LoRaFB-SA. B-Seet shown in Figure B, suggesting that various noise images for replacement all have a similar impact representational similarity. Besides, report similarity scores of all models under static natural scenes stimuli Table and show that our model achieves the highest score.
two other types of images for replacing frame
A
B
0.45
ISRSA
TSRSA
0.38
0.25
gausian
Lolta B-SN ECT10E
noise from the uniforms distribation
statlc natural
0.35
10
Level of Chass
Ratio Replneement
Figure 7: A. The TSRSA score curves of LoRaFB-SNet and CORnet trained on UCF101 with different levels of chaos. The elements in the plot indicate similar content as in 4...re/4A.. B. The TSRSA score curves of LoRaFB-SNel trained on UCF101 with different ratios of replacement. There three types of noise images. The elements in the plot indicate similar content in Figure 4B.
Table 6: The similarity scores of all models under static natural scenes stimuli.
CORnet
ResNet-2p-CPC
SEW-ResNet
LoRaFB-SNet
Statie Scene
0.3544
0.3435
0.3935
0.4130

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please see abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please Section
18

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The work does not involve proof of theory.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer:| Yes]
Justification: We detail all the techniques for reproducing the results of our work in Section
g
Guidelines:
The answer NA means that the paper does not include experiments.
19

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The codes are available at https://thhhaacccmmmmmmsss////sd Movie,
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //tpps.cc/publicc///cccc
www.ddd/eeeooeuummmmccoo ssi ssionyolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
20

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Please Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Please see Figure and
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
21

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our work conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work does not involve social impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Our work does not release data or models with a high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
22

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We describe in Section 4.1
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com /dtaaset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work does not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing experiments and research with human subjects.
23

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
24


Reviews:
[{"review_id": "gNt2Jx1IkX", "reviewer": "Reviewer_YZqi", "summary": "To better understand visual processing in the brain, this paper presents a spiking neural network with top-down connections. It follows the trend over the past few years of building deep neural network models to approximate brain architecture and match brain and behavioral data. Simply put, the goal is to have a network model that can perform visual tasks like the visual neural systems in the brain and align well with the brain in terms of representation. With such a model, we can pose many questions that we typically ask about the real brain within these models. Thus, the performance is measured by how well these networks match the brain representations. The model introduced here, LoRaFB-SNet, differs from previous models in two main aspects, as I understand it: it differs from previous DNN models such as CORnet in terms of the spiking units versus traditional DNN units, and also the authors of CORnet only looked at static but not dynamic visual processing; it is different from SEW-ResNet in terms of the top-down connections feature, as SEW-ResNet is a purely feedforward model. Based on the integration of the good features of the previous works, this work conducted good research into the Effects of Dynamic and Static Information, which provides insight for neuroscientists into this model.", "questions": "1. **Importance of Spiking Mechanisms:** The use of spiking mechanisms is a central feature of your model. However, the paper lacks a detailed discussion on how these mechanisms enhance the model's performance compared to non-spiking models like CORnet. Could you elaborate on why spiking mechanisms are critical for your model? How do they improve the representation of dynamic and static visual information? Clarification on this could significantly affect the evaluation of your model's novelty and effectiveness.\n\n2. **Data and Model Robustness:** The results presented utilize data from only two movies provided by the Allen Institute, with one movie showing no significant difference in performance and the other showing very low similarity scores. Can you discuss the expected robustness of your model's performance across additional movies, animals, or visual regions? How can you justify the robustness and significance of your results with such a limited dataset? Would additional data be necessary to strengthen your conclusions, or do you believe the current results are sufficiently convincing?\n\n3. **Dependency on Pretraining Tasks:** How dependent are the model's outcomes on the specific pretraining tasks used? Given the unique properties of spiking networks in processing temporal sequences, does the choice of pretraining data significantly influence the results? Understanding this could help in assessing the model's generalizability and applicability to other datasets or tasks.", "limitations": "As mentioned in the Weaknesses and Questions sections, this review identifies two primary areas of concern:\n\n1.  There is insufficient discussion about how spiking mechanisms enhance the model's performance compared to non-spiking models. Addressing this could significantly clarify the unique contributions of your approach.\n\n2.  The paper does not thoroughly validate the robustness and significance of the results across various datasets, which is crucial for substantiating the model's applicability and effectiveness.\n\nAdditionally, some minor areas that could improve the paper include:\n\n- **Impact of the Pretraining Task:** The influence of the pretraining task on the model’s performance is not clearly articulated. Clarifying this could help understand the adaptability of the model under different conditions.\n\n- **Exploration of Local Recurrent Connections:** There is potential value in exploring the impact of local recurrent connections, which could provide deeper insights into the comprehensive functionalities of the visual cortex.\n\nAddressing these points would significantly enhance the quality of the paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The strength of this work is highlighted by the match of the research question and the approaches the authors took. They first identified the representation related to dynamic visual processing in neuroscience, then drew insights from the literature in neuroscience that the top-down connection is important in this processing in the brain, and then built a model incorporating this feature, as well as considering the spiking mechanisms. The question is indeed an important one in neuroscience and the approaches the authors take and their results show that it is a promising direction. Their thorough knowledge of literature and previous works in both the neuroscience and machine learning community is clear, as they integrate the good and important features from previous works, and make it a comprehensive model, and also their experimental design clearly addresses their research questions.", "weaknesses": "The primary concern I have with this paper revolves around the impact of spiking mechanisms within the model. In neural processing, there is a longstanding debate on the relevance of timing versus firing rate for information processing. Recent evidence in neuroscience literature has indeed highlighted the importance of spike timing as a crucial source of dynamic information in visual systems[1]. However, the paper does not sufficiently discuss how spiking mechanisms influence the model's performance or represent a significant improvement over traditional mechanisms. Although spiking mechanisms are a key feature of the model, the discussion and experimental design focusing on these mechanisms are limited. The only mention of these aspects is in the second paragraph of section 4.3.3 and Table 2, which do not clearly demonstrate the importance of spiking mechanisms or how they contribute distinctly to the model's capabilities. This is a crucial gap, as a more detailed exploration here could significantly enhance the novelty of the work compared to previous models like CORnet, which lacks spiking mechanisms but incorporates recurrent connections.\n\nAdditionally, the comparisons made in the experiments need a more focused analysis on this point. For instance, in Figure 3, panel A shows that in Movie 1, the performance difference between LoRaFB-SNet and CORnet is minimal, despite the latter lacking spiking mechanisms. Panels B and C do not include comparisons with CORnet, and no explanation is provided for Panel D in the main text or the figure descriptions, which might be an oversight. Figure 4 also omits CORnet in the comparisons. A more thorough comparison with CORnet is vital since it too has recurrent connections but does not incorporate spiking mechanisms. Beyond comparisons with CORnet, more effort should be directed towards differentiating LoRaFB-SNet from its non-spiking version, to better illustrate why spiking is necessary and how it significantly impacts both the model and neural systems.\n\nAnother minor point is the paper's focus on region-to-region top-down feedback while seemingly neglecting the potential impact of local recurrent connections, which could also be significant as suggested by recent literature[2].\n\n[1] Quintana, Daniel, Hayley Bounds, Julia Veit, and Hillel Adesnik. \"Balanced bidirectional optogenetics reveals the causal impact of cortical temporal dynamics in sensory perception.\" bioRxiv (2024).\n\n[2] Oldenburg, Ian Antón, William D. Hendricks, Gregory Handy, Kiarash Shamardani, Hayley A. Bounds, Brent Doiron, and Hillel Adesnik. \"The logic of recurrent circuits in the primary visual cortex.\" Nature neuroscience 27, no. 1 (2024): 137-147.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the notable and perceptive comments. We will do our best to address the comments and provide detailed responses point by point, below.\n\n**1. Importance of Spiking Mechanisms.**\n\nIn our work, we design a deep spiking network based on rate coding and pretrain it on large-scale datasets. The training algorithms of temporal coding spiking networks are still not mature [1]. Given that our work is centered around deep neural network models, the rate coding spiking network is a better choice.\n\n**To better demonstrate the importance of spiking mechanisms**, we clarify our existing results (the comparisons with CORnet in Fig. 3A, 3C and 3D) and perform more experiments for the non-spike version of LoRaFB-SNet (termed LoRaFB-CNet) as suggested by the reviewers. The results (see details below) provide strong evidences that spiking mechanisms play a crucial role, allowing our model to better capture neural representations of the visual cortex. Since our spiking network is based on rate coding, we think that the following two properties may contribute. First, our network encodes and transfers information exclusively in the form of spikes, just like the brain. Second, the membrane potential dynamics of spiking neurons help to process dynamic information, which complements the long-range feedback connections well.\n\nThe clarifications and new results about spiking mechanisms:\n\n* LoRaFB-SNet outperforms CORnet and LoRaFB-CNet on both two movies (*Movie1*: **0.5202**, 0.5060, 0.4975; *Movie2*: **0.2827**, 0.2230, 0.2619).\n\n* LoRaFB-SNet consistently outperforms CORnet on different lengths of movie clips (Fig. 3C) and shows the increasing ratio compared to CORnet (Fig. 3D). **The results suggest our model performs significantly better on longer movie clips**, echoing the results in Fig. 3A. Since longer movie stimuli increase the diversity of population neuronal response patterns in the visual cortex [2, 3], it is more difficult for models to capture brain-like representations. Our model has a more pronounced advantage in this case. See discussions in lines 234-243.\n\n* We added the experiments on *Movie2* in Figure 3C for the LoRaFB-CNet. As shown in Table R1, the results support the above conclusions. Besides, LoRaFB-CNet outperforms CORnet for all clip lengths, suggesting that our model structure also benefits information processing in the long movie.\n\n|Model|30s|50s|70s|80s|90s|100s|110s|120s|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|**LoRaFB-SNet**|**0.405**|**0.353**|**0.319**|**0.284**|**0.302**|**0.299**|**0.295**|**0.283**|\n|**LoRaFB-CNet**|0.384|0.332|0.297|0.249|0.270|0.278|0.277|0.262|\n|**CORnet**|0.328|0.275|0.234|0.192|0.194|0.202|0.218|0.223|\n\nTable R1: TSRSA scores with different movie clip lengths on *Movie2*. The standard error is omitted.\n\n* For the experiments of static natural scene stimuli in Fig. 4C, we add the results of CORnet and the LoRaFB-CNet (**0.4130**, 0.3544, 0.3411), which shows that our model also outperforms them in this neural dataset.\n\nThe clarifications and new results about feedback connections:\n\n* Fig. 4A compares the changes in similarity scores between the network with feedback and fully feedforward structures when the dynamic information is disrupted, supporting that the long-range feedback connections allow LoRaFB-SNet to represent dynamic information in a more context-dependent manner. See discussions in lines 272-275. We add the experiments for CORnet and LoRaFB-CNet in **Fig. R1 in the PDF of the \"global rebuttal\"**, which **solidifies our conclusion about the effectiveness of feedback connections**.\n\n* Fig. 4B does not involve comparisons of model structures and mechanisms. Therefore, we don't include CORnet and LoRaFB-CNet here.\n\nWe will add the above results and discussion to our revised manuscript.\n\n**2. Data and Model Robustness.**\n\nAs mentioned above, the similarity score of models suffers from an increase in movie length, and LoRaFB-SNet shows a more pronounced advantage in longer movie experiments. The significant improvement in our model's score compared to other models, coupled with the overall lower scores of other models on *Movie2*, underscores the **robust advantage of our model across different movies**. In fact, when we use randomly selected 30s (the length of *Movie1*) clips from the 120s *Movie2*, LoRaFB-SNet achieves a score of 0.405 (about 80\\% of the score on *Movie1*). The scores of our model in six visual regions also indicate the **robustness across brain regions** (please refer to **response 4 to Reviewer rzHc**).\n\nIn summary, experiments and analysis in a variety of settings effectively demonstrate the performance and robustness of our model. In the future, despite the paucity of publicly available neural datasets under natural movie stimuli, we will try to apply our model to more datasets to solidify our conclusions.\n\n**3. Dependency on Pretraining Tasks.**\n\nWe have preliminarily discussed the influence of pretraining datasets and tasks (Fig. 3B and the first paragraph of Section 4.3.3). In particular, the video recognition task better benefits model's neural similarity. Besides, temporal structures of data contribute more than the static content. Larger datasets and other video tasks may have different impacts, which we will explore in the further work.\n\n**4. Exploration of Local Recurrent Connections.**\n\nWe agree that lateral connections within brain regions are useful for brain-like modeling. However, our work focuses on feedback connections across regions to build bio-inspired models. We will introduce local recurrent connections in the future for exploration.\n\n[1] \"Temporal-coded spiking neural networks with dynamic firing threshold Learning with event-driven backpropagation.\" ICCV 2023.\n\n[2] \"Rapid learning in cortical coding of visual scenes.\" Nature Neuroscience 2007.\n\n[3] \"Representation of visual scenes by local neuronal populations in layer 2/3 of mouse visual cortex.\" Front Neural Circuits 2011.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9d2H4rW3gj", "reviewer": "Reviewer_dwyD", "summary": "The authors in this work proposes a long-range feedback spiking network (LoRaFB-SNet) whose architecture is similar to neuronal and synaptic behavior in the cortical regions of the brain. Furthermore they propose a Time-Series Representational Similarity Analysis framework to measure the similarity between model representations and visual cortical representations of mice. The proposed model  exhibits the highest level of representational similarity (following the analysis framework), outperforming the current baselines.", "questions": "1. Is there a reason for not using VLMs, etc as a baseline. Understanding attention mechanism which is relevant from the perspective of movie-data processing might be significant.\n\n2. In the experiments why were the Neurons firing less than 0.5 spikes/s excluded?", "limitations": "Suggestions are added in the weaknesses section.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper has a strong motivation to analyse the similarity in the dynamic and static representations of movie-based stimuli on neural models and the actual visual cortical representation of mice. The motivation of using feedback connections is also biologically significant. \nTSRSA as an analysis tool to understand the similarity between the representation of neural models and visual cortex representation of an actual biological brain also seems very interesting.", "weaknesses": "1. The model architecture is not particularly novel since past works on SNNs have shown that long-range feedback plays an important role in processing visual information [1]. TSRSA seemed to be one of the novel contributions of the paper. The explanation can be made more mathematical since currently its more of a textual description which can be hard to follow. \n\n2. The paper's baselines (mainly ResNets and RNNs) could benefit from further expansion. A more comprehensive analysis of these representations could be particularly insightful when compared with state-of-the-art vision models such as VLMs, Video-LMs, or SSM-based architectures. Moreover, providing deeper insights into how these representations are processed within the actual visual cortex would not only be beneficial but also enhance understanding within the broader machine learning community.\n\n3/Suggestion: Since this work is more directed at understanding biological implications of how visual cortex represents video-based information and not from leveraging energy-efficiency aspects of SNNs, it might be more relevant to explore bio-plausible learning mechanism (STDP-based), more sophisticated neuronal models such as HH model instead of the simplistic LIF, etc. to develop a more comprehensive analysis.\n\nReference\n1. Xiao, Mingqing, Qingyan Meng, Zongpeng Zhang, Yisen Wang, and Zhouchen Lin. \"Training feedback spiking neural networks by implicit differentiation on the equilibrium state.\" Advances in neural information processing systems 34 (2021): 14516-14528.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the clear and thoughtful comments. We will do our best to address the reviewer's concerns and answer the questions in the following.\n\n**1. The novelty of our model architecture.**\n\nThe work of Xiao, Mingqing focuses on deriving the equilibrium state of a spiking neural network by introducing feedback connections, and in turn deriving the gradient of parameters for training the network without having to consider the forward process. **However, their algorithm has only been validated on networks with shallow and stacked layers on small datasets like CIFAR10**, and it is not yet known whether the equilibrium state is tractable in deeper and more complex network structures (e.g., skipping connections).\n\nIn contrast, we introduce multiple long-range feedback connections into a deeper spiking network and train it with a commonly used algorithm (BPTT with surrogate gradient) on large-scale datasets. This provides a more general structure and training paradigm for exploring feedback spiking networks.\n\n**2. The TSRSA.**\n\nWe thank the reviewer for recognizing TSRSA. As suggested by the reviewer, we provide a more mathematical representation of the method here and will add it in our manuscript.\n\nWe define the representation matrix as $\\mathrm{R}=(\\mathrm{r}\\_1, \\mathrm{r}\\_2, ..., \\mathrm{r}\\_t , ... , \\mathrm{r}\\_T)\\in\\mathbb{R}^{N\\times{T}}$ of a given model layer or a given cortical region, where $\\mathrm{r}\\_t$ represents the population responses of $N$ units/neurons to the movie frame $t$. We first compute the Pearson correlation coefficient between the responses to a given movie frame $t$ and to its subsequent frames to obtain the representational similarity vector $\\mathrm{s}\\_t$. The vector is formulated as:\n\n$\n\\mathrm{s}\\_t=(s\\_{t1}, s\\_{t2}, ..., s\\_{tp}, ...),\n$\n\n$\ns\\_{tp}=Corr\\_{Pearson}(\\mathrm{r}\\_t, \\mathrm{r}\\_{t+p}),\n$\n\nwhere $0<p<T-t$.\n\nSecond, we concatenate all vectors to obtain the complete representational similarity vectors $S\\_{model}$ for a network layer and $S\\_{cortex}$ for a cortical region. Finally, we compute the Spearman rank correlation coefficient between $S\\_{model}$ and $S\\_{cortex}$ as the similarity score.\n\n**3. Is there a reason for not using VLMs, etc as a baseline.**\n\nWe agree that exploring the information processing mechanisms of VLMs with the real visual cortex may lead to new insights for the machine learning and neuroscience communities. **However, our work focuses on constructing biologically plausible deep neural network (DNN) models in terms of the structure and dynamics**, to improve representational similarity to real biological neural responses, providing new insights into information processing mechanisms of movie stimuli in the visual cortex. Therefore, the baselines we chose for comparison are all bio-inspired DNNs, which are widely used in studies of neural representational similarity to the visual cortex.\n\nSOTA vision models such as VLM on visual tasks have not been chosen as baselines in our work due to the following points.\n\n* The core of our work is to build brain-like models, not to achieve better performance on visual tasks. However, VLMs are designed in a performance-oriented manner, without taking biological plausibility (e.g., attention mechanisms) into account [1].\n\n* Previous studies already proved that visual task performance is not positively correlated with neural representation similarity. Instead, higher task performance may lead to poorer brain-like models [2].\n\n* Transformer-based vision models have shown poor similarity (worse than convolutional networks) to real neural responses in the visual cortex [2, 3]. Similarly, transformer-based language models (LLMs) have been questioned for not truly reflecting the core elements of human language processing [4].\n\n**4. More bio-plausible learning mechanisms and more sophisticated neuronal models.**\n\nDespite the higher biological plausibility of the advised learning mechanism and neuronal models, introducing them in deep spiking networks and training them on large datasets suffers from problems such as difficulty in convergence and untrainability. Therefore, we have not considered them in our current work.\n\n**5. In the experiments why were the Neurons firing less than 0.5 spikes/s excluded?**\n\nWhen analyzing neuronal spiking activity, researchers often exclude neurons with firing rates below a certain threshold to focus on neurons that are responsive to visual inputs and to more effectively extract stimulus-related neural representations. An empirical threshold of 0.5 spikes/second is commonly used in many studies [5, 6].\n\n\n[1] Paria Mehrani & John K. Tsotsos. \"Self-attention in vision transformers performs perceptual grouping, not attention.\" Frontiers in Computer Science 2023.\n\n[2] Drew Linsley, et al. \"Performance-optimized deep neural networks are evolving into worse models of inferotemporal visual cortex.\" NeurIPS 2023.\n\n[3] Liwei Huang, et al. \"Deep Spiking Neural Networks with High Representation Similarity Model Visual Pathways of Macaque and Mouse.\" AAAI 2023.\n\n[4] Ebrahim Feghhi, et al. \"What Are Large Language Models Mapping to in the Brain? A Case Against Over-Reliance on Brain Scores.\" arXiv 2024.\n\n[5] Lucas Pinto, et al. \"Fast modulation of visual perception by basal forebrain cholinergic neurons.\" Nature Neuroscience 2013.\n\n[6] Brice Williams, et al. \"Spatial modulation of dark versus bright stimulus responses in the mouse visual system.\" Current Biology 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kux6olt5w4", "reviewer": "Reviewer_rzHc", "summary": "Authors propose a deep spiking network with feedforward and feedback connectivity trained on natural movies and on static images, and compare the similarity of representations in their artificial network to the similarity of representations evaluated in the mouse visual cortex. Using such measure, they find a good fit of the artificial spiking network with the data. Feedback connections and the spiking nature of neural activity are both important to achieve good performance, potentially because they allow the model to extract temporal features and encode information with spiking sequences. The model outperforms state-of-the art alternatives.", "questions": "Authors report that the spiking nature of neural activity is computationally useful to the model as it allows spike-sequential coding and extraction of temporal features. Moreover, spiking seems to allow processing the information more flexibly as it is not limited by specific filer sizes. These results seem as a major insights of the model, but they should be backed up by further evidence. Could authors provide more clear results showing the benefits of spiking? If so, these results should also be better emphasized, e.g. in the abstract. \n\nAuthors do not provide much information about how they trained their network in the main part of the paper. While I appreciated the writing, some more details on the training would help to better assess the soundness of results.\n\nTo evaluate the TSRSA score, authors evaluate such score in every measured cortical region of the mouse brain and average across regions. An alternative could be to instead compare the artificial network with each region and report which region has the activity that is the most similar to the artificial network. Is there a good reason for averaging? Could some regions be better fitted than others?", "limitations": "Limitations are addressed, even though authors could consider discussing the lack of lateral connectivity in their model. Lateral connectivity has a major impact on the neural activity in biological neural networks.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes a new model (building of previous research) that is both outperforming notable alternatives among artificial deep neural networks, as well as it brings plausible insights about the benefits of feedback processing and spiking nature of neural activity on the information processing in biological brains. This work is of interest to the broader audience of NeurIPS.", "weaknesses": "In line 318 authors mention the necessity of having a spiking model and of having feedback connections but do not refer to any specific Figure or Appendix. It would be important to provide evidence for these claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for being supportive of our work and for the constructive comments. We will try our best to address the comments. Below are our detailed responses.\n\n**1. About line 318.**\n\nWe apologize for not refering to the corresponding table. The conclusion comes from the results on the right side of Table 2. We will add a reference to that table in the manuscript.\n\n**2. About spiking mechanisms?**\n\nWe already have some experiments in our manuscript to compare spiking and non-spiking networks to demonstrate the importance and benefits of the spiking mechanism. What's more, according to the suggestion of reviewers, we conduct additional experiments to provide more evidence and to strengthen our conclusions (please refer to **response 1 to Reviewer YZqi**). We will add and emphasize these results in our manuscript.\n\n**3. About the training of our work.**\n\nDue to space constraints, we have put the detailed network training procedure and training parameters in the appendix. Please refer to Appendix C.\n\n**4. Is there a good reason for averaging? Could some regions be better fitted than others?**\n\nWe use the average across brain regions as the similarity score for two reasons. First, neurons responsive to movie stimuli are found in all six cortical regions [1]. Second, the depths of the network layer with the best similarity score to most mouse cortical regions are at the same level [2]. We show the individual similarity scores of our model to each region in Table R1, suggesting that there is no significant difference across regions.\n\n||VISp|VISl|VISrl|VISal|VISpm|VISam|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|\n|*Movie1*|0.5274|0.5007|0.5049|0.5147|0.5295|0.5438|\n|*Movie2*|0.2223|0.2618|0.2955|0.3003|0.3012|0.3153|\n\nTable R1: TSRSA scores of LoRaFB-SNet to six cortical regions, respectively.\n\n[1] Saskia EJ de Vries, et al. \"A large-scale standardized physiological survey reveals functional organization of the mouse visual cortex.\" Nature neuroscience 2020.\n\n[2] Jianghong Shi, et al. \"Comparison against task driven artificial neural networks reveals functional properties in mouse visual cortex.\" NeurIPS 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their valuable time and their thoughtful and constructive comments. We do our best to answer the questions raised by reviewers in each individual rebuttal.

Since some reviewers are concerned about the importance of the spiking mechanism of our model, we clarify our existing results and add some new experiments (please refer to **response 1 to Reviewer YZqi** for details). As a result, we demonstrate the validity of the spiking mechanism and discuss its properties that benefit our model:

* Our network encodes and transfers information exclusively **in the form of spikes**, just like the brain.

* **The membrane potential dynamics** of spiking neurons help to process dynamic information, which complements the long-range feedback connections well.

Meta-review: this work proposes the long-range feedback spiking network (LoRaFB-SNet), which mimics top-down connections between cortical regions and incorporates spike information processing mechanisms inherent to biological neurons. Futhermore, the authors used Time-Series Representational Similarity Analysis (TSRSA) to measure the similarity between model representations and visual cortical representations of mice. These analyses confirmed the critical role of long-range feedback in representing dynamic and static visual information. All reviewers appreciated the novelty of this work and especially the critical role of spiking format. I am inclined to accept this paper.
Decision: Accept (poster)