SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Well-defined visual receptive fields emerge To assess whether the  $\mathrm{CNN + GRU_1}$  model learned meaningful visual receptive fields, we used gradient ascent (see Methods) to find the maximally activating stimulus for each neuron. Receptive fields for the 32 best-predicted neurons are shown in Fig. 4. Interestingly, most of them had well-defined excitatory and inhibitory subregions, often resembling receptive fields of orientation-selective neurons. Most excitatory and inhibitory subregions spanned approximately  $30^{\circ}$  of visual angle (the full width of the frame, 80 pixels, roughly corresponding to  $120^{\circ}$  of visual angle), which is roughly on the same order of magnitude compared to receptive field sizes typically observed in mouse V1, varying from  $10^{\circ}$  to  $30^{\circ}$  [8, 25, 50].
Receptive fields were noticeably different across mice. Whereas Mouse 1 and 3 had visual receptive fields with strongly excitatory subregions, most model neurons for Mouse 2 appeared to be inhibited
Model	History	Mouse 1	Mouse 2	Mouse 3
cc↑	MSE↓	cc↑	MSE↓	cc↑	MSE↓
CNN+GRU1	48 ms	.646 ± .136	.0543	.508 ± .166	.0917	.607 ± .132	.0801
CNN+GRU2	96 ms	.649 ± .139	.0528	.506 ± .174	.0898	.607 ± .133	.0811
CNN+GRU3	144 ms	.653 ± .139	.0528	.528 ± .160	.0843	.604 ± .134	.0790
CNN+GRU4	192 ms	.650 ± .142	.0525	.566 ± .169	.0799	.614 ± .136	.0773
CNN+GRU5	240 ms	.645 ± .144	.0556	.519 ± .177	.0933	.598 ± .134	.0807
CNN+GRU6	288 ms	.654 ± .142	.0526	.549 ± .175	.0823	.598 ± .138	.0798
CNN+GRU7	336 ms	.644 ± .148	.0534	.533 ± .169	.0931	.596 ± .133	.0806
CNN+GRU8	384 ms	.646 ± .146	.0547	.546 ± .179	.0840	.594 ± .141	.0825
Table 3:  $\mathrm{CNN + GRU}_k$  model trained with input from  $k$  timesteps on the full feature set  $(\{\mathcal{B}\cup \mathcal{D}\}_{\times})$ . Best performing networks are indicated in bold. cc: cross-correlation, mean  $\pm$  standard deviation (↑: the higher the better), MSE: mean-squared error (↓: the lower the better).
Figure 4: The maximally activating stimuli learned in  $\mathrm{CNN + GRU_1}$ , generated via gradient ascent. The 32 neurons with the highest cross-correlation (cc) from each mouse are shown, sorted by cc.
by visual signals (same colorbar across panels). In addition, several model neurons lacked pronounced visual receptive fields, indicating that they were more strongly driven by behavioral variables. Even though model fits were repeated with different initial values for the behavioral variables, the resulting visual receptive fields looked qualitatively the same (see Appendix C), thus demonstrating the validity of the generated receptive fields. In addition, even some of the best-predicted neurons lack a pronounced or spatially structured receptive field, implying that these neurons could be primarily driven by behavioral variables.
Analysis of behavioral saliency maps reveals different types of neurons Intrigued by the fact that some neurons lacked pronounced visual receptive fields, we aimed to analyze the influence of behavioral state on the predicted neuronal response by performing a saliency map analysis on the behavioral inputs (see Methods). Since different behavioral variables operate on different input ranges, we first standardized the saliency map activities for each behavioral variable across the model neuron population. Saliency map activities further than 1 standard deviation from the mean were then interpreted as "driving" the neuron, allowing us to categorize each neuron as being driven by one or multiple behavioral variables (Fig. 5).
We first asked which neurons in our model were driven by which behavioral variables (Fig. 5, top). Consistent with [25], we found a large fraction of model neurons driven by eye and head position, and smaller fractions driven by locomotion speed and pupil size. Approximately  $20 - 30\%$  of neurons were not driven by any of these behavioral variables, rendering their responses purely visual.
However, a particular neuron could be driven by multiple behavioral variables. Repeating the above analysis, we found that most model neurons showed mixed selectivity (i.e., responding to different categories of information, such as visual and motor signals, or stimulus and reward signals), with only a minority of cells responding exclusively to a single behavioral variable, (Fig. 5, middle). Adding the interaction terms between behavioral variables (Fig. 5, bottom) did not change the fact that most model V1 neurons encoded combinations of multiple behavioral variables, often relating information about the animal's eye position to head position and locomotor speed.
In this paper, we propose a deep recurrent neural network that achieves state-of-the-art predictions of V1 activity in freely moving mice. We discovered that our model outperforms previous models under these more naturalistic conditions, which could be attributed to the better alignment of this data with the computations performed by the mouse visual system, based on its natural visual environment and
Figure 5: Effect of behavioral variables on model neuron activity, inferred by the saliency analysis. A) Fraction of neurons that are "driven by" (i.e., their saliency map activation is further than 1 standard deviation from the mean) different behavioral variables (similar to [25]). A neuron that responds to (e.g.) both position and speed may be counted twice. Neurons without a strong behavioral drive are categorized as "vision only". B) Fraction of neurons that are uniquely driven by a specific behavioral variable. Still, a large fraction of neurons are driven by multiple behavioral variables. C) Same as B), but split with interaction terms.
behavioral characteristics. Similar to previous models, we found that a simple CNN architecture is sufficient to predict the visual response properties of cells in mouse V1.
In addition, mouse V1 is known to be strongly modulated by signals related to the movement of the animal's eyes, head, and body [21, 22, 25], which are severely restricted in head-fixed preparations. Models trained on head-fixed preparations may thus be limited in their predictive power. In contrast, our model was able to predict V1 activity on a 1-hour continuous data stream, during which the animal freely explored a real-world arena. Our analyses demonstrate the impact of the animal's behavioral state on V1 activity and reveal that most model V1 neurons exhibit mixed selectivity to multiple behavioral variables.
Accurate predictions of mouse V1 activity under natural conditions Our brains did not evolve to view stationary stimuli on a computer screen. However, most research on neural coding in vision has been conducted under head-fixed conditions, which do not mirror natural behavior and thus provide limited insight into visual processing in real-world environments. Some visual functions mediated by the ventral stream, such as identifying faces and objects, resemble this condition, but the real visual environment is constantly shifting due to self-motion, leading to dynamic activities such as navigation or object reaching, typically mediated by the dorsal stream. To truly understand visual perception in natural environments, we need to capture the computational principles when the subjects are in motion.
In this research, we take the initial steps towards this by modeling a novel data type encompassing neural activity coupled with a visual scene captured from a freely moving animal's perspective. This represents a dramatic (but, in our opinion, crucial) shift in the "parameter space" of visual input, from static images projected on a screen to dynamic, real-world visual input.
Surprisingly, visual responses were best predicted with a standard three-layer ("vanilla") CNN (Table 1), as compared to a multitude of more sophisticated models that included autoencoders, variational autoencoders, filter bank models, and pre-trained ResNet and EfficientNet architectures. One possible explanation might be that the neurons in our dataset were selective for other behavioral inputs that we did not have access to, and that the vanilla CNN architecture imposed the fewest assumptions about how visual input contributed to the neural activity. In addition, visual receptive fields for Mouse 2 were noticeably different from the other two mice, exhibiting pronounced inhibitory subregions (Fig. 4, center). This is consistent with the fact that the cortical probes of Mouse 2 were more superficial compared to the other two mice [25], so the recorded neurons may have both different anatomical inputs and different visual responses.
Mixed selectivity of behavioral variables Our experiments demonstrated that the models incorporating behavioral variables and their interactions performed substantially better than the models relying exclusively on visual inputs. Moreover, our saliency map analysis showed that only around  $25\%$  of model neurons could be considered purely visual, with the majority of model neurons driven by multiple behavioral variables.
This widespread mixed selectivity is consistent with previous literature suggesting that V1 neurons may be modulated by a high-dimensional latent representation of several behavioral variables related to the animal's movement, recent experiences, and behavioral goals [20]. It is also consistent with the idea of a basis function representation [51, 52], which allows a population of neurons to conjunctively represent multiple behaviorally relevant variables. Such representations are often employed by higher-order visual areas in primate cortex to implement sensorimotor transformations [17, 51, 53]. It is intriguing to find computational evidence for such a representation as early as V1 in the mouse. Future computational studies should therefore aim to study the mechanisms by which V1 neurons might construct a nonlinear combination of behavioral signals.
Limitations and future work. While our study opens a new perspective on modeling neural activity during natural conditions, there are a few limitations that need to be acknowledged. First, our data was relatively limited (around 50 neurons per animal, for 3 animals). The development of a Sensorium-style standardized dataset [14] for freely-moving mice would significantly benefit future research in this area, enabling more robust comparisons between different modeling approaches. Second, it would be beneficial to integrate other modalities that are known to be encoded in mouse V1 into the model. One such example is reward signals [54], which could provide additional information about the animal's decision-making processes and motivations during exploration.
This work was supported by the National Institute of Neurological Disorders and Stroke of the National Institutes of Health under Award Number R01-NS121919. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
[45] Philip R. L. Parker, Dylan M. Martins, Emmalyn S. P. Leonard, Nathan M. Casey, Shelby L. Sharp, Elliott T. T. Abe, Matthew C. Smear, Jacob L. Yates, Jude F. Mitchell, and Christopher M. Niell. A dynamic sequence of visual processing initiated by gaze shifts, August 2022. Pages: 2022.08.23.504847 Section: New Results.
[46] Pouya Bashivan, Kohitij Kar, and James J. DiCarlo. Neural population control via deep image synthesis. Science, 364(6439):eaav9436, May 2019. Publisher: American Association for the Advancement of Science.
[47] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps, April 2014. arXiv:1312.6034 [cs].
[48] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition, December 2015. arXiv:1512.03385 [cs].
[49] Mingxing Tan and Quoc V. Le. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks, September 2020. arXiv:1905.11946 [cs, stat] version: 5.
[50] Gert Van den Bergh, Bin Zhang, Lutgarde Arckens, and Yuzo M. Chino. Receptive-field properties of V1 and V2 neurons in mice and macaque monkeys. Journal of Comparative Neurology, 518(11):2051-2070, 2010. _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/cne.22321.
[51] Michael Beyeler, Emily L. Rounds, Kristofor D. Carlson, Nikil Dutt, and Jeffrey L. Krichmar. Neural correlates of sparse coding and dimensionality reduction. PLOS Computational Biology, 15(6):e1006908, June 2019.
[52] S. Fusi, E. K. Miller, and M. Rigotti. Why neurons mix: high dimensionality for higher cognition. Curr Opin Neurobiol, 37:66-74, April 2016.
[53] A. Pouget and L. H. Snyder. Computational approaches to sensorimotor transformations. Nat Neurosci, 3 Suppl:1192-8, November 2000.
[54] Marshall G. Shuler and Mark F. Bear. Reward Timing in the Primary Visual Cortex. Science, 311(5767):1606-1609, March 2006. Publisher: American Association for the Advancement of Science.
A Data Acquisition
The full procedure of data collection and data preprocessing is described in detail in Ref. [25], but are briefly described below for the reader's convenience.
Visual scenes from the mouse's perspective, neural activity in V1, and various behavioral variables were simultaneously recorded from three adult mice who were freely exploring an arena with a state-of-the-art head-mounted recording system [25]. The recording system consisted of high-density silicon probes, two miniature cameras and an inertial measurement unit (IMU). Electrophysiology data was acquired at  $30\mathrm{kHz}$  using a  $11\mu \mathrm{m} \times 15\mu \mathrm{m}$  multi-shank linear silicon probe (128 channels) implanted in the center of the left monocular V1. One wide-angled camera (around  $120^{\circ}$ ) was aimed outwards to capture the visual scene available to the right eye of the mouse at  $16\mathrm{ms}$  per frame ("worldcam"). A second camera was aimed at the right eye (illuminated by an infrared-LED) to record a video feed of the right eye at  $30\mathrm{Hz}$ . The IMU acquired three-axis gyroscope and accelerometer data at  $30\mathrm{kHz}$ . In addition, a top-down camera recorded the mouse in the arena at  $60\mathrm{Hz}$ .
During experiments, mice were placed in an arena where they could move around freely for about 1 hour. The arena was approximately  $48\mathrm{cm}$  long  $\times 37\mathrm{cm}$  wide  $\times 30\mathrm{cm}$  high. The gray floor was covered with black-and-white Legos to provide visual contrast. One wall of the arena was a monitor displaying a moving black-and-white spots stimulus, and the other three walls were covered with wallpaper with static stimuli including white noise, black-and-white high-spatial-frequency gratings, and black-and-white low-spatial-frequency gratings. In order to encourage foraging behavior during the recording sessions, small fragments of tortilla chips were sparsely distributed across the arena.
The worldcam video was downsampled to  $60 \times 80$  pixels. DeepLabCut [43] was used to extract extract eye position  $(\theta, \phi)$  and pupil radius  $(\sigma)$ . Pitch  $(\rho)$  and roll  $(\omega)$  of the mouse's head position were extracted from the IMU. Locomotion speed  $(s)$  was estimated from the top-down camera feed using DeepLabCut [43]. Electrophysiology data bandpass-filtered between  $0.01\mathrm{Hz}$  and  $7.5\mathrm{kHz}$ , and spike-sorted with Kilosort 2.5 [44]. Single units were selected using Phy2 (see [45]) and inactive units (mean firing rate  $< 3\mathrm{Hz}$ ) were removed. This yielded 68, 32, and 49 active units for Mouse 1-3, respectively. To prepare the data for machine learning, all data streams were deinterlaced and resampled at  $20.83\mathrm{Hz}$  (48 ms per frame; Fig. 1C).
B Vision-Only Models
B.1 Hyperparameter Tuning
We performed a grid search to find the optimal CNN kernel size (3, 5, 7, 9), number of channels (32, 64, 128, 256, 512; in various combinations), and dropout rate (0, 0.25, 0.5). While other models often rely on kernel size 3 for their CNN, we found these small kernels to lead to worse performance, perhaps due to the mouse's low-resolution vision. Kernel size 7 performed best.
We repeated the grid search for CNNs with different number of convolutional layers. The resulting 3-layer CNN with 0.5 dropout rate outperformed many differently sized networks, such as a 1-layer CNN with 1024 channels (i.e., a shallow but wide network), a 2-layer CNN, or a 4-layer CNN. Choice of learning rates and optimizers had no notable effect on the final performance of the networks.
B.2 Autoencoder
We initially hypothesized that an autoencoder could provide regularization benefits over a "vanilla" CNN, because the reconstruction loss might encourage the model to learn visual features that are useful for decoding. We used an encoder  $\phi$  to map the original frame  $\mathcal{F}$  to a vector  $\nu$  in the latent space, which was present at the bottleneck, while the decoder  $\psi$  then mapped the vector  $\nu$  from the latent space to the output.
$$
\phi : \mathcal {F} \rightarrow \mathcal {V}, \tag {1}
$$
$$
\psi : \mathcal {V} \rightarrow \mathcal {F}, \tag {2}
$$
$$
\phi , \psi = \operatorname {a r g m i n} _ {\phi , \psi} \| \mathcal {F} - (\psi \cdot \phi) \mathcal {F} \| ^ {2}. \tag {3}
$$
After hyperparameter search, we settled on size 256 for the latent space vector, and the weight of the reconstruction loss relative to the Poisson loss was fixed at 0.5. Both the encoder and the decoder were 3-layer CNNs, and their numbers of channels were symmetric. However, after testing a number of autoencoders with different configurations (Table 4), we found that a simple 3-layer CNN outperformed any and all of them.
Kernel size, encoder #channels	Mouse 1	Mouse 2	Mouse 3
cc↑	MSE↓	cc↑	MSE↓	cc↑	MSE↓
3, 16 × 32 × 64	.539 ± .149	.0728	.389 ± .128	.107	.502 ± .129	.0996
5, 16 × 32 × 64	.550 ± .147	.0728	.363 ± .116	.109	.508 ± .135	.0983
7, 16 × 32 × 64	.525 ± .152	.0732	.353 ± .121	.117	.509 ± .131	.0980
9, 16 × 32 × 64	.518 ± .147	.0752	.315 ± .101	.119	.492 ± .135	.0997
3, 32 × 64 × 128	.543 ± .144	.0737	.367 ± .128	.109	.503 ± .131	.100
5, 32 × 64 × 128	.551 ± .149	.0723	.361 ± .109	.119	.514 ± .132	.0984
7, 32 × 64 × 128	.539 ± .145	.0739	.390 ± .118	.109	.492 ± .129	.100
9, 32 × 64 × 128	.510 ± .155	.0758	.331 ± .119	.112	.500 ± .134	.101
3, 64 × 128 × 256	.541 ± .146	.0758	.374 ± .123	.110	.514 ± .127	.0990
5, 64 × 128 × 256	.552 ± .145	.0777	.362 ± .119	.110	.508 ± .134	.104
7, 64 × 128 × 256	.553 ± .134	.0688	.369 ± .104	.111	.530 ± .136	.0992
9, 64 × 128 × 256	.537 ± .146	.0811	.355 ± .109	.119	.500 ± .128	.105
Table 4: Performance of different autoencoders. The numbers of channels in the decoder were symmetric with those of the encoder. Best performing networks are indicated in bold.  $cc$ : cross-correlation, mean  $\pm$  standard deviation across neurons ( $\uparrow$ : the higher the better), MSE: mean-squared error ( $\downarrow$ : the lower the better).
C Visual Receptive Fields
To test whether the recovered visual receptive fields are sensitive to the choice of initial values for the behavioral variables, we repeated our experiments by initializing the behavioral variables with noise sampled from the uniform distribution  $[-1, 1)$ . The values remained the same throughout the process of gradient ascent.
We found that different values of behavioral variables resulted in similar visual receptive fields. A representative example is shown in Fig. C.1. Although some of the absolute values changed, receptive field structure stayed qualitatively the same, with identical excitatory and inhibitory subregions to the ones reported in Fig. 4.
Mouse 1
Figure C.1: The maximally activating stimuli learned in  $\mathrm{CNN + GRU_1}$ , generated via gradient ascent with a fixed and randomly initialized behavioral variable. The 32 neurons with the highest cross-correlation  $(cc)$  from each mouse are shown, sorted by  $cc$ .