Multimodal Deep Learning Model Unveils Behavioral Dynamics of V1 Activity in Freely Moving Mice
Authors: Aiwen Xu
Yuchen Hou
Cris M. Niell
Michael Beyeler
Venue: NeurIPS 2023 poster (2023)
Keywords: neuroscience
cognitive science
multimodal learning
representation learning
network architecture
computational biology
visual perception
TL;DR: We propose a novel multimodal deep-learning model that integrates visual, behavioral, and temporal dynamics and achieves state-of-the-art predictions of V1 activity in freely-moving mice.

Abstract: Despite their immense success as a model of macaque visual cortex, deep convolutional neural networks (CNNs) have struggled to predict activity in visual cortex of the mouse, which is thought to be strongly dependent on the animal’s behavioral state. Furthermore, most computational models focus on predicting neural responses to static images presented under head fixation, which are dramatically different from the dynamic, continuous visual stimuli that arise during movement in the real world. Consequently, it is still unknown how natural visual input and different behavioral variables may integrate over time to generate responses in primary visual cortex (V1). To address this, we introduce a multimodal recurrent neural network that integrates gaze-contingent visual input with behavioral and temporal dynamics to explain V1 activity in freely moving mice. We show that the model achieves state-of-the-art predictions of V1 activity during free exploration and demonstrate the importance of each component in an extensive ablation study. Analyzing our model using maximally activating stimuli and saliency maps, we reveal new insights into cortical function, including the prevalence of mixed selectivity for behavioral variables in mouse V1. In summary, our model offers a comprehensive deep-learning framework for exploring the computational principles underlying V1 neurons in freely-moving animals engaged in natural behavior.

Multimodal Deep Learning Model Unveils Behavioral Dynamics of V1 Activity in Freely Moving Mice
Aiwen Xu
Yuchen Hou
Department of Computer Science University of California, Santa Barbara Santa Barbara, CA 93117 aivnxu@ucsb.ecm
Department of Computer Science University of California, Santa Barbara
Santa Barbara, CA 93117 yuchenhou@ucab.cem edu
Cristopher M. Niell
Department of Biology, Institute of Neuroscience University Oregon
Eugene, OR 97403
cni mmielluuoregon edu
Michael Beyeler
Department of Computer Science
Department of Psychological & Brain Sciences University of California, Santa Barbara
Santa Barbara, CA 93117 mbeyeler@ucsb. edu
Abstract
Despite their immense success model of macaque visual cortex, deep convolu- tional neural networks (CNNs) have struggled to predict activity in visual cortex of the mouse, which is thought to be strongly dependent on the animal's behav- ioral state. Furthermore, most computational models focus on predicting neural responses to static images presented under head fixation, which dramatically different from the dynamic, continuous visual stimuli that arise during movement in the real world. Consequently, it is still unknown how natural visual input and different behavioral variables may integrate over time to generate responses in primary visual cortex (V1). To address this, introduce multimodal recurrent neural network that integrates gaze-contingent visual input with behavioral and temporal dynamics explain VI activity in freely moving mice. We show that the model achieves state-of-the- predictions of V activity during free exploration and demonstrate the importance of each component in an extensive ablation study. Analyzing our model using maximally activating stimuli and saliency maps, we reveal new insights into cortical function, including the prevalence of mixed se- lectivity for behavioral variables in mouse V1. In summary, our model offers a comprehensive deep-learning framework for exploring the computational principles underlying VI neurons in freely-moving animals engaged in natural behavior.
1 Introduction
Computational models have been crucial in providing insight into the underlying mechanisms by which neurons in the visual cortex respond external stimuli. Deep convolutional neural networks (CNNs) have had immense success as predictive models of the primate ventral stream, in cases where
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

the animal was passively viewing stimuli or simply maintaining fixation [1-5]. Despite their success, these CNNs are poor predictors of neural responses in mouse visual cortex [6], which is thought be shallower and more parallel than that of primates [7, 8). According to the best models in the literature [9-14], the mouse visual system is more broadly tuned and operates on relatively low-resolution inputs to support a variety of behaviors [15]. However, these models were limited to predicting neural responses to controlled (and potentially ethologically irrelevant) stimuli that were passively viewed by head-fixed animals.
Movement is a critical element of natural behavior. In the visual system, eye and head movements during locomotion and orienting transform the visual scene in potentially both beneficial (e.g., by providing additional visual cues) and detrimental ways (e.g., by introducing confounds due to self- movement) [16-18]. Movement-related activity is widespread in mouse cortex [19, 20] and prevalent in primary visual cortex (V1) [21, 22]. For instance, V1 neurons of freely moving mice show robust responses to head and eye position [23, 24), which may contribute multiplicative gain to the visual response [25] that cannot be replicated under head fixation. V1 activity may be further modulated by variables that depend on the state of the animal and its behavioral goals [20, 22, 26, 27J. However, how these behavioral variables may integrate to modulate visual responses VI is unknown. Furthermore, a comprehensive predictive model of VI activity in freely moving animals is still lacking.
To address these challenges, we make the following contributions:
We introduce a multimodal recurrent neural network that integrates caze-contingent visual input with behavioral and temporal dynamics to explain V1 activity during natural vision in freely moving mice.
We show that the model achieves state-of-the art predictions of VI activity during free exploration based on visual input and behavior, demonstrating the ability to accurately model neural responses in the dynamic regime of movement through the visual scene.
We uncover new insights into cortical neural coding by analyzing our model with maximally activating stimuli and saliency maps, and demonstrate that mixed selectivity of visual and behavioral variables is prevalent in mouse VI.
2
Related Work
The mouse, as a model organism, offers unparalleled experimental access to the mammalian cerebral cortex [28]. Computational models of mouse V1, including generalized linear models (GLMs) [25, 29] and customized models mimicking the mouse visual hierarchy [30], have been crucial in providing deeper insights into the range of computations performed by visual cortex. More recently, deep CNNs have also been used to model mouse V1 [9-13, 31-34].
Despite their success in predicting neural activity in the macaque visual cortex [35], deep CNNs trained on ImageNet have had limited success in predicting mouse visual cortical activity [6]. This is perhaps not surprising, as most ImageNet stimuli belong to static images of human-rclevant semantic categories and may thus be of low ethological relevance for rodents. More importantly, these deep CNNs may not be the ideal architecture to model mouse visual cortex, which is known to be shallower and more parallel than primate visual cortex [36, 37]. In addition, mice are known to have lower visual acuity than that of primates [7, 8), and much of their visual processing may be devoted to active, movement- based behavior rather than passive analysis of the visual scene [22, 38. 39]. Although the majority of VI neurons is believed encode low-level visual features [40], their activity often strongly modulated by behavioral variables related to eye and head position [23-25], locomotion [18, 21, 22], arousal [27, 41], and the recent history of the animal [26]. Furthermore, mouse V1 is highly interconnected with both cortical and subcortical brain areas, which contrasts with feedforward, hierarchical models of visual processing [22].
A common architectural approach that has proved quite successful is to split the network into different components (first introduced by [11]):
a "core" network, which typically consist of a CNN used to extract convolutional features from the visual stimulus [11, 12, 25, 42], sometimes in combination with recurrent network [11]; a "shifter" network, which mimics gaze shifts by learning (typically affine) transformation from
head- to eye-centered coordinates, either applied to the pixel input [11, 25] or a CNN layer [12]; a "readout" network, which learns mapping from artificial to biological neurons [11, 12, 42).

---

Owing to the difficulty of developing a predictive model of mouse cortex, Willeke et al. [14] recently invited submissions to the Sensorium competition held NeurIPS '22. The competition introduced a benchmark dataset of V1 neural activity recorded from head-fixed mice on a treadmill viewing static images, with simultaneous measurements of running speed, pupil size, and eye position. A baseline model was provided as well, which consisted of a 4-layer CNN core in combination with shifter and readout network [12]. Even though 26 teams submitted 194 different models, the overall improvement to the baseline performance was modest, raising the single trial correlation from .287 to .325 in the Sensorium and from .384 to .453 in the Sensorium+ competition. Architectural innovations (e.g.. Transformers, Normalizing Flows, YOLO, and knowledge distillation), were unable to make an impact, as most improvements were gained from ensemble methods. A promising direction was taken by the winning model, which attempted to learn a latent representation of the "brain state" from the various behavioral variables, inspired by [20]. However, the model utilized the timestamps of the test set estimate recent neuronal actívities, which the other competitors did not have access to.
Taken together, we identified three main limitations of previous work that this study aims to address:
Head-fixed preparations. Most previous models operated on data from animals in head-fixed conditions with static stimuli, which do not mirror natural behavior and thus provide limited insight into visual processing in real-world environments. In contrast, the present work is applied to state-of-thhear neurophysiological recordings of V1 activity in freely moving mice. This represents dramatic shift in the "parameter space" of visual input, from static images to dynamic, real-world visual input. One could imagine that this will make the modeling process more difficult, because the stimulus set is more complex, or easier, because it is more matched to the computational challenge the brain evolved for.
Limited influence of behavioral state. Previous models often limited the influence of behavioral state to eye measurements and treadmill running speed, which were either concatenated with the visual features [14, 41], utilized in the shifter network to determine the gaze-contingent retinal input [11. 14], or used to predict multiplicative gain factor [11].
Missing temporal dynamics. Most previous modeling works ignored the temporal factors that might influence V1 activity and overlooked the dynamic nature of visual processing (but see [1T). We overcome this limitation by utilizing approximately 1 l-hour-long recordings of three mice freely exploring an arena, and model is capable of handling continuous data streams of any length.
3 Methods
-radd-m recording system We had access to data from three adult mice who were freely exploring 48 em long x 37 em wide x 30 cm high arena (Fig. IA), collected with a state-of-the- art recording system [25] that combined high-density silicon probes with miniature gean-mounted cameras (Fig. 1B). One camera was aimed outwards to capture the visual scene from the mouse's 's perspective ("worldcam") ") at 16 ms per frame (downsampled to 60 x 80 pixels). A second camera, aimed at the eye, was used to extract eye position (0, Φ) and pupil radius (α) at 30 Hz using DeepLabCut [43]. Pitch (p) and roll (w) of the mouse's head position were extracted at 30 kHx from the inertial measurement unit (IMU). 0, di 0. and w allowed for the worldcam video to be corrected for eye movements: A 3-layer colynccoeee shifter network (where each linear layer was accompanied by Tanh and BatchNorm) was trained to predict rotation (bounded by +36") and a shift (bounded by 16 pixels horizontally and 112 pixels vertically) based on 0, 6, p. and w to convert each frame to head- to eye-centered coordinates. Locomotion speed (s) was estimated from the top-down camera feed using DeepLabCut [43]. Electrophysiology data was acquired at 30 kHz using a 11 μm X 15 pm multi-shank linear silicon probe ( (128 channels) implanted in the center of the left monocular VI, then bandpass- -indpass-fitered between 0.01 Hz and 7.5 kHz, and spike-sorted with Kilosort 2.5 [44]. Single units were selected using Phy2 (see [45]) and inactive units (mean firing rate < 3 Hz) were removed. This yielded 68, 32, and 49 active units for Mouse 1-3, respectively. To prepare the data for machine learning, all data streams were deinterlaced and resampled at 20.83 Hz (48 ms per frame; Fig. 1C). For more detailed description of the dataset, see Appendix A and Ref. [25).
Model architecture We used a 3-layer CNN (kernel size 7, 128 x 64 x 32 channels) to encode the visual stimulus. Each convolutional layer was followed by a BatchNorm layer, a ReLUU, and Dropout layer (0.5 rate). A fully fully-connected layer transformed the learned visual features into
3

---

A
C
S
64
B
ephys worldcam
IMU
neuron #
32
prattt to 1hth apal buat math detuin with aatc Pooenaa we
eyecam
0
time (s)
9.6
Figure 1: Schematic of the head-mounted recording system for freely moving mice (adapted from [25]). A) Three mice freely explored a 48 cm long 37 wide 30 cm high arena. B) Preparation included a silicon probe for electrophysiological recording in V1 (yellow), miniature cameras for recording the mouse's s eye position and pupil size (0, σ, and σ; magenta), and visual scene (blue), and inertial measurement unit for measuring head orientation (p and w; green). C) Sample data from 9.68 period during free movement showing (from top) visual scene, horizontal and vertical eye position, pupil size, head pitch and roll, locomotor speed, and raster plot of 64 units.
visual feature vector, (Fig. 2, top-right). In a purely visual version of the model, was fed into fully-connected layer, followed by softplus layer, to yield a neuronal response prediction.
To encode behavioral state, constructed an input vector from different sets of behavioral variables:
S: all behavioral variables used the Sensorium+ competition [14], consisting of running speed
(s). pupil size (ar), and its temporal derivative (Ø);
B: all behavioral variables used in [25], consisting of eye position (0, Φ), head position (p. w),
pupil size (o), and running speed (s);
D: the first-order derivatives of the variables in B, namely 0, á. W, à, and
To test for interactions between behavioral variables, these sets could also include the pairwise multiplication of their elements; e.g., Bx {byby A (b;, by) B3. The input vector was then passed through a batch normalization layer and a fully connected layer (subjected to a strong LI norm for feature selection) to produce behavioral vector, b.
We then concatenated the vectors v, and their element- product v C (all calculated for each individual input frame), ), fed them through batch normalization layer, and input them to 1-layer gated recurrent unit (GRU) (hidden size of 512). To incorporate temporal dynamics, we constructed different versions (GRU,) of the model that had access to h previous frames. A fully-connected layer and softplus activation function were applied to yield the neuronal response prediction.
Training and model evaluation Since the visual input depended on the movement of the mouse and the mouse could in very different behavioral states over the length of the recording. the data was highly inhomogencous across time. To deal with the continuous and dynamic nature of the data, we therefore split the 1 h-llong recording into 10 consecutive segments. The first 70% % of each segment were then reserved for training (including an 80-20 validation split) and the remaining 30 % for testing.
Note that it is unlikely for data to "leak" from the train segment into the test segment. While it is possible that the mouse could have been exploring the same part of the arena at different segments of the recording, it was free to move its head, eyes, and body as it saw fit. Thus two duplicate data points could only be produced by the animal exactly duplicating the time courses its eye, head, and body movement in the exact same location of the arena.

---

CNN GRU (Vision and Behavior)
CNN (Vision only)
ONN
FC
GRU
Softpluss
Figure 2: Model architecture diagram. The vision-only network (top-right) was CNN network, predicting the neural activity at time given the visual input at time f (48 ms bins). The full model combined the CNN with behavioral encoder and gated recurrent unit (GRU), predicting the neural activity at time I given the visual and behavioral inputs from time ( to I n.
Models were separately trained on the data from each mouse. Model parameters were optimized with Adam (batch size: 256, CNN learning rate: ,0001, full model: .0002) to minimize the Poisson loss between predicted neuronal response (f) and ground truth (r) N Ef
log fz), where N denotes the number of recorded neurons for each mouse. We used early stopping on the validation set (patience: 5 epochs), which led all models to converge in less than 50 epochs. Due to the large number of syper-paramcters the specific network and training settings were determined using a validation set (see Appendix B).
combination of grid search and manual exploration
To evaluate model performance, we calculated the cross-correlation (cc) between smoothed version (2s boxcar filter) of the predicted and ground-truth response for each recorded neuron [25].
NVIDIA RTX 3090 with 24GB memory. All code, data used to train the models, and weights of the trained model can be found at https: //gi rithub.... www.///wwwivviiiiioooooo.c.cccii
All models were implemented in PyTorch and trained
20tp//222322222 Xu- Xu-Multimodal -aotmoddaaaaa-ee-V-- V1.
Maximally activating stimuli We used gradient ascent [42] to discover the visual stimuli that most strongly activate particular model neuron in our network. The visual input was initialized with noise sampled in (.5, 2). The behavioral variables were initialized to vector of all ones, and updated in the loop with the visual stimuli. We used the Adam optimizer to repeatedly add the gradient of the target neuron's 's activity with respect to imputs. We also applied L2 regularization (weight of .02) and Laplacian regularization (weight of 0.01) [46] on the image. This procedure was repeated 6400 times. The resulting. maximally activating visual stimuli were smoothed with a Butterworth filter (low-pass, .05 cutoff frequency ratio) to reduce the impact of high-frequency noise.
Saliency map We computed a saliency map [47] of the behavioral vector for each neuron to discover which behavioral variables contributed most strongly to each model neuron's activity. We iterated through the test dataset, recorded the gradient of each behavioral input with respect to each neuron's prediction, and then averaged the gradients per neuron to obtain the saliency map.
4 Results
Mouse V1 activity is best predicted with 3-layer CNN To determine the purely visual con- tribution to VI responses, we experimented with a large number vision architectures (see Ap- pendix B). In the end, vanilla 3-layer CNN (kernel size 7, 128 x 64 x 32 channels) yielded
5

---

the best cross-correlation between predicted and ground-truth responses (Table 1), outperforming the best autoencoder architecture (kernel size: 7, encoder: 64 128 x 256 channels, decoder: 256 x 128 x 64 channels), ResNet-18 18 [48] (a 20-layer CNN with the first input channel being re- placed by 1), EfficientNet--bl [49] (a 65-layer CNN with the first input channel being replaced by 1), and the Sensorium baseline [12] (a 4-layer CNN with a readout network). The greatest improvement in cross-correlation was achieved for Mouse
Behavioral variables improve most neuronal predictions Once identified the 3-layer CNN as the best visual encoder, we added the different sets of behavioral variables to the network. To allow for fair comparison with the Sensorium+ baseline [14], we first limited ourselves to S {or à s). but then gradually added more behavioral variables (B) [25] as well the derivatives of these variables (D) and multiplicative pairs (Bx and ()x)).
The results are shown in Table All models were able to outperform the Sensorium+ baseline, and the addition of behavioral variables and their interactions further improved model performance. Note that although the full model used a GRU to combine visual and behavioral features, the input sequence length was always (i.c., GRU,). That being said. it is possible that the GRU learned long-term correlations that the Sensorium+ baseline model did not have access to. Nevertheless, the biggest performance improvements were gained through the addition of behavioral variables related to head and eye position (which are present in B but not in S), their derivatives (D), and multiplicative interactions between these variables ((B U x)
We also wondered whether the prediction only some VI neurons would benefit from the addition of these behavioral variables. To our surprise, the cross-correlation between predicted and ground-truth responses improved for almost all recorded V1 neurons (Fig. 3).
Mouse
Mouse 2
Mouse 3
Model
cc
MSE 4
MSE $
cc f
MSE
CNN
.596 + 134
.0626
.424 + .141
.100
.552 + .138
.0908
Autoencoder
.555 = .14
.0710
.370 + .145
.112
.521
.144
.0974
ResNet-18 [48]
.517 = .159
.0782
.366 + .171
.107
.511
.138
.0944
EfficientNNt-BOO [49]
.542 = .153
.0694
.393 + 165
.103
.510
.127
.0965
Sensorium [12]
.519 = 114
.0754
.381 + 128
119
.97 + .136
.100
Table Best-performing vision models, compared to the Sensorium baseline [12] (see Appendix B
for more). Best- pertorerforming networks are indicated in bold. cc: crosscoorelattion mean + standard deviation across neurons (f: the higher the better), MSE: mean- -quured error (4: the lower the better).
Mouse
Mouse 2
Mouse 3
Feature Set
Cc 1
MSE J
cc f
MSE 4
cc t
MSE
(BUD)x
646 + .366
.0543
.088 + 166
.0917
.607 + 132
.0801
CNN+GRU
BUD
.644 + .111
.0543
.467 + 180
.0966
.599 + .144
.0797
Bx
.639 + .141
.0555
.484 + .155
.0925
.593 + .122
.0812
B
.641 + .136
.0555
.450 + .18
.0962
.583 + .136
.0825
S
.623 + .142
.0551
.498 + .154
.0911
.579 + .135
.0828
Sensorium+ .540 + .138
.0696
.441
.181
.101
.47 - .146
.0975
Table 2: CNN+GRU, model, compared to the Sensorium+ baseline [12]. trained on different behavioral feature sets. S = {a, s): the set of variables used the Sensorium+ competition [14].
= (0, o., w. s): the set of variables from [25]. D (Ó. de W., p. a...; s): the derivatives of B. Ax {a,a, Y (a, a,) E A} denotes the set of all multiplicative pairs. U denotes the union operator. Best performing networks are indicated bold. ce: cross cross-correlation, mean + standard deviation across neurons (f: the higher the better), MSE: mean-squared error (l: the lower the better).

---

Mouse 1
Mouse 2
Mouse 3
1.0
0.8
0.8
.22
0.8
CD vis+beeh
CC vis-beh
CC vis+bsh
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0
0.2
0.4 0.6 0.8
1.0
0
0.2
0.4
0.6 0.8 Cis
1.0
0
0.2
0.4
0.6
0.8
1.0
vis
CCvss
Figure 3: The integration of behavioral variables improved the cross-correlation (cc) for the majority of neurons. Each dot represents a neuron. A dot above the dashed diagonal indicates a higher oc with the inclusion of behavioral variables. Histograms (small insets) illustrate the distribution of the improvement cc across the neuronal population.
Access to longer series of data in time further improves predictive performance After we best model performance, we extended the GRU's temporal dependence by allowing the input to vary from one frame (48 ms) total of eight frames (384 ms), and assessed the model's performance.
identified the full behavioral feature set ((B U D)x) x) the one yielding the
The results are shown in Table 3. The amount of temporal information needed by the model to reach peak predictive performance cross-correlation, 192 ms, 192 ms, and 192 ms in terms of mean-squared error, respectively). This indicates that temporal information is important for predicting dynamic neural activity. However, the dependence on temporal information has a limit, and different neurons in VI might possess different temporal capacities.
similar across mice (288 ms, 192 ms, and 192 ms in terms of
Well-defined visual receptive fields emerge assess whether the CNN+GRU, model learned meaningful visual receptive fields, we used gradient ascent (see Methods) to find the maximally activating stimulus for each neuron. Receptive fields the 32 best-predicted neurons are shown in Fig. 4. Interestingly, most of them had well-defined excitatory and inhibitory subregions, often resembling receptive fields of orientation-selectiee neurons. Most excitatory and inhibitory subre- gions spanned approximately 30" of visual angle (the full width of the frame, 80 pixels, roughly corresponding to 120% visual angle), which is roughly on the same order of magnitude compared to receptive field sizes typically observed in mouse V1, varying from 10 to 30 [8, 25, 50].
Receptive fields were noticeably different across mice. Whereas Mouse and had visual receptive fields with strongly excitatory subregions, most model neurons for Mouse 2 appeared to be inhibited
Mouse
Mouse 2
Mouse 3
Model
History
cc
MSE 1
cc
MSE 1
cc
MSE J
CNN+GRU,
48 ms
.646 +: .33
.0543
.508 + .166
.0917
.607 1 .132
.0801
CNN+GRU2
96 ms
.649 + .139
.0228
.506 + 174
.0898
.607 + .133
.0811
CNN+GRU;
144 ms
.653 .139
.0528
.528 + .16
.0843
.604 + .134
.0790
CNN+GRU,
192 ms
.650 + .142
.0525
.566 .169
.0799
.6144 .136
.0773
CNN+GRUs
240 ms
.645 + .144
.0556
.519 .177
.0933
.598 + .134
.0807
CNN+GRU, 288 ms
.654 .142
.0526
.549 + .175
.0823
.598 = .138
.0798
CNN+GRU?
336 ms
.644 +14
.0534
.533 + 169
.0931
.596 + .13
.0806
CNN+GRU,
384ms
.646 .14
.0547
546
.0840
.594 + .141
.0825
Table 3: CNN+GRU; model trained with input from timesteps the full feature set ((UU))
Best performing networks are indicated in bold. ec: cross-correlation. mean = standard deviation (t: the higher the better), MSE: mean-squared error (J: the lower the better).
7

---

Mouse
Mouse
Mouse 3
mod
COMB 275
COMOTT
cnd
305
modATA
202627
Figure 4: The maximally activating stimuli learned in CNN+GRU,, generated via gradient ascent. The 32 neurons with the highest cross-corred cross-correlatio (cc) from each mouse are shown, sorted by cc.
by visual signals (same colorbar across panels). In addition, several model neurons lacked pronounced visual receptive fields, indicating that they were more strongly driven by behavioral variables. Even though model fits were repeated with different initial values for the behavioral variables, the resulting visual receptive fields looked qualitatively the same (see Appendix C), thus demonstrating the validity of the generated receptive fields. addition, even some of the best best-predicted neurons lack pronounced or spatially structured receptive field, implying that these neurons could be primarily driven by behavioral variables.
Analysis behavioral saliency maps reveals different types of neurons Intrigued by the fact that some neurons lacked pronounced visual receptive fields, we aimed to analyze the influence of behavioral state on the predicted neuronal response by performing saliency map analysis on the behavioral inputs (see Methods). Since different behavioral variables operate on different input ranges, we first standardized the saliency map activities for each behavioral variable across the model neuron population. Saliency map activities further than standard deviation from the mean were then interpreted as "driving" the neuron, allowing us to categorize each neuron as being driven by one or multiple behavioral variables (Fig. 5).
We first asked which neurons our model were driven by which behavioral variables (Fig. 5, top). Consistent with [25], we found a large fraction of model neurons driven by eye and head position, and smaller fractions driven by locomotion speed and pupil size. Approximately 20-30% of neurons were not driven by any of these behavioral variables, rendering their responses purely visual.
However, particular neuron could be driven by multiple behavioral variables. Repeating the above analysis, we found that most model neurons showed mixed selectivity (i.c., responding to different categories of information, such visual and motor signals, stimulus and reward signals), with only a minority of cells responding exclusively to a single behavioral variable, (Fig. 5, middle). Adding the interaction terms between behavioral variables (Fig. 5, bottom) did not change the fact that most model V1 neurons encoded combinations of multiple behavioral variables, often relating information about the animal's eye position head position and locomotor speed.
Discussion
In this paper, we propose a deep recurrent neural network achieves state-of-the-ar predictions of VI activity in freely moving mice. We discovered that our model outperforms previous models under these more naturalistic conditions, which could be attributed the better alignment of this data with the computations performed by the mouse visual system, based on its natural visual environment and

---

Mouse 1
Mouse 2
Mouse 3
position
is driven by
speed
pupil
vision only
is exclusively driven by
position
speed
pupil
multiple
vision only
position
is exclusively driven by
speed
pupil
head x eye
head speed
head pupil
eye speed
eye pupil
multiple
vision only
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
fraction neurons
fraction of neurons
fraction neurons
Figure 5: Effect of behavioral variables model neuron activity, inferred by the saliency analysis. A) Fraction of neurons that are "driven by" (i.c., their saliency map activation further than I standard deviation from the mean) different behavioral variables (similar [25]). A neuron that responds to (e.g.) both position and speed may be counted twice. Neurons without strong behavioral drive are categorized as "vision only". B) Fraction of neurons that are uniquely driven by specific behavioral variable. Still, a large fraction neurons are driven by multiple behavioral variables. C) Same as B), but split with interaction terms.
behavioral characteristics. Similar to previous models, we found that simple CNN architecture is sufficient to predict the visual response properties cells mouse VI.
In addition, mouse VI is known to be strongly modulated by signals related to the movement of the animal's eyes, head, and body [21, 22, 25], which are severely restricted head-fixed preparations. Models trained on head-fixee preparations may thus be limited in their predictive power. In contrast, our model was able to predict V1 activity on a 1-hour continuous data stream, during which the animal freely explored real-world arena. Our analyses demonstrate the impact of the animal's 'ss behavioral state on VI activity and reveal that most model V1 neurons exhibit mixed selectivity to multiple behavioral variables.
Accurate predictions of mouse VI activity under natural conditions Our brains did not evolve to view stationary stimuli on a computer screen. However, most research on neural coding vision has been conducted under fexd-fixed conditions, which do not mirror natural behavior and thus provide limited insight into visual processing in roal-world environments. Some visual functions mediated by the ventral stream, such identifying faces and objects, resemble this condition, but the real visual environment is constantly shifting due to self-motion, leading to dynamic activities such navigation object reaching, typically mediated by the dorsal stream. To truly understand visual perception in natural environments, we need to capture the computational principles when the subjects in motion.
In this research, we take the initial steps towards this by modeling a novel data type encompassing neural activity coupled with visual scene captured from freely moving animal's 's perspective. This represents dramatic (but, our opinion, crucial) shift in the "parameter space" of visual input, from static images projected screen to dynamic, real-world visual input.

---

Surprisingly, visual responses were best predicted with a standard three-layer ("vanilla") CNN (Table 1). as compared multitude of more sophisticated models that included autoencoders, variational autoencoders, filter bank models, and pre-trained ResNet and EfficientNet architectures. One possible explanation might be that the neurons in our dataset were selective for other behavioral inputs that we did not have access to, and that the vanilla CNN architecture imposed the fewest assumptions about how visual input contributed to the neural activity. addition, visual receptive fields for Mouse 2 were noticeably different from the other two mice, exhibiting pronounced inhibitory subregions (Fig. 4, center). This is consistent with the fact that the cortical probes of Mouse were more superficial compared to the other two mice [25], so the recorded neurons may have both different anatomical inputs and different visual responses.
Mixed selectivity of behavioral variables Our experiments demonstrated that the models incor- porating behavioral variables and their interactions performed substantially better than the models relying exclusively visual inputs. Moreover, saliency map analysis showed that only around 25% model neurons could considered purely visual, with the majority of model neurons driven by multiple behavioral variables.
This widespread mixed selectivity consistent with previous literature suggesting that V1 neurons may be modulated by high-dimensions latent representation of several behavioral variables related to the animal's movement, recent experiences, and behavioral goals [20]. is also consistent with the idea of basis function representation [51, 52], which allows population of neurons conjunctively represent multiple behaviorally relevant variables. Such representations are often employed by higher-order visual areas in primate cortex implement sensorimotor transformations [17, 51, 53]. is intriguing to find computational evidence for such a representation as early as V1 in the mouse. Future computational studies should therefore aim study the mechanisms by which V1 neurons might construct a nonlinear combination of behavioral signals.
Limitations and future work. While our study opens activity during natural conditions, there are a few limitations that need to be acknowledged. First, data relatively limited (around 50 neurons per animal, for 3 animals). The development of Sensorium-style standardized dataset [14] for freely-moving mice would significantly benefit future research in this area, enabling more robust comparisons between different modeling approaches. Second, it would be beneficial to integrate other modalities that are known to be encoded in mouse V1 into the model. One such example is reward signals [54], which could provide additional information about the animal's 's decision makinn processes and motivations during exploration.
perspective modeling neural
Acknowledgments
This work was supported by the National Institute of Neurological Disorders and Stroke of the National Institutes of Health under Award Number RO1-NS1219) 21919. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
10

---

References
[1] Daniel L. K. Yamins. Ha Hong, Charles F. Cadieu, Ethan A. Solomon, Darren Seibert, and James J. DiCarlo. Performance-optmmized hierarchical models predict neural responses higher visual cortex. Proceedings of the National Academy of Sciences, 111(23):8619-866) 8624, June 2014, Publisher: Proceedings of the National Academy of Sciences.
[2] Seyed-Mahdi Khaligh-Razavi and Nikolaus Kriegeskorte. Deep Supervised, but Not Unsupervised, Models May Explain IT Cortical Representation. PLOO Computational Biology, (0(1T):=100391 November 2014. Publisher: Public Library of Science.
[3] Umut Güçlüü and Marcel A. J. Gerven. Deep Neural Networks Reveal Gradient in the Complexity of Neural Representations across the Ventral Stream, Journal of Neuroscience, 35(27):10005- 1000000000000,000011110100101100 July 2015. Publisher: Society for Neuroscience Section: Articles.
[4] Santiago A. Cadena, George H. Denfield, Edgar Walker, Leon A. Gatys, Andreas Tolias, Matthias Bethge, and Alexander S. Ecker. Deep convolutional models improve predictions of macaque VI responses to natural images. PLOS Computational Biology, 15(4); 1006897, April 2019. Publisher: Public Library of Science.
[5] William F. Kindel, Elijah D. Christensen, and Joel Zylberberg. Using deep learning to probe the neural code for images in primary visual cortex. Journal of Vision, 19(4):29, April 2019.
[6] Santiago A. Cadena, Fabian H. Sinz, Taliah Muhammad, Emmanouil Froudarakis, Erick Cobos, Edgar Y. Walker, Jake Reimer, Matthias Bethge, Andreas Tolias, and Alexander Ecker. How well do deep neural networks trained on object recognition characterize the mouse visual system? October 2019.
[7] Glen Prusky, Paul R West, and Robert M Douglas. Behavioral assessment of visual acuity in mice and rats. Vision Research, 40( (0)1))2200-2200)) 12209, July 2000.
[8] Cristopher M. Niell and Michael P. Stryker. Highly Selective Receptive Fields in Mouse Visual Cortex. The Journal of Neuroscience, (8(30))7520-7533 July 2008
[9] David Klindt, Alexander S Feker, Thomas Fuler, and Matthias Bethge. Neural system identification for large populations separating what" and " where". In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[10] Alexander S. Ecker, Fabian H. Sinz, Emmanouil Froudarakis, Paul G. Fahey, Santiago A. Cadena, Edgar Y. Walker, Erick Cobos, Jacob Reimer, Andreas Tolias, and Matthias Bethge. A cotation-equivarian convolutional neural network model of primary visual cortex. December 2018.
[11] Fabian Sinz, Alexander S Ecker, Paul Fahey, Edgar Walker, Erick Cobos, Emmanouil Froudarakis, Dimitri Yatsenko, Zachary Pitkow, Jacob Reimer, and Andreas Tolias. Stimulus domain transfer in recurrent models for large scale cortical population prediction on video. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
[12] Konstantin-Klenn Lurz, Mohammad Bashiri, Konstantin Willeke, Akshay Jagadish, Eric Wang. Edgar Y. Walker, Santiago A. Cadena, Taliah Muhammad, Frick Cobos, Andreas S. Tolias, Alexander S. Eeker, and Fabian H. Sinz. Generalization in data-driven models of primary visual cortex. January 2021.
[13] Shahab Bakhtiari, Patrick Mineault, Timothy Lillicrap, Christopher Pack, and Blake Richards. The functional specialization of visual cortex emerges from training parallel pathways with self-supervised predictive learning. In Advances in Neural Information Processing Systems, volume 34, pages 25164- 25178. Curran Associates, Inc., 2021.
[14] Konstantin F. Willeke, Paul G. Fahey, Mohammad Bashiri, Laura Pede, Max F. Burg, Christoph Blessing, Santiago A. Cadena. Zhiwei Ding. Konstantin- Klemmens Lurz, Kayla Ponder, Taliah Muhammad, Saumil S. Patel, Alexander Ecker, Andreas S. Tolias, and Fabian H. Sinz. The Sensorium competition on predicting large-scale mouse primary visual cortex activity, June 2022. (tti:22000006666 [cs. q-bio].
[15] Andrew D. Huberman and Cristopher M. Niell. What can mice tell us about how vision works? Trends in Neumsciences,, 34(9)4664443, Sepeembber 2011.
[16] James J. Gibson. The Ecological Approach to Visual Perception: Classic Edition. Psychology Press, November 2014.
SBSLBQAAQBAI.
[17] M. Beyeler, N. Dutt, and J. L. Krichmar. 3D Visual Response Properties of MSTd Emerge from an Efficient, Sparse Population Code. Journal of Neuroscience, 36(32):8399 415, August 2016.

---

[18] Philip R. L. Parker, Morgan A. Brown, Matthew C. Smear, and Cristopher M. Niell. Seveement-Reltter Signals in Sensory Areas: Roles in Natural Behavior. Trends Neurusciences, 43(8):581-595, August 2020.
[19] Laura Busse, Jessica A. Cardin, M. Eugenia Chiappe, Michael M. Halassa, Matthew J. McGinley. Takayuki Yamashita, and Aman B. Saleem. Sensation during Active Behaviors. Journal of Neuro- science. (7(45):10226-18882 November 2017. Publisher: Society for Neuroscience Section: Symposium and Mini-Symposium.
[20] Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Charu Bai Reddy, Matteo Carandini, and Kenneth D. Harris. Spontaneous behaviors drive multidimensional, brainwide activity. Science, 364(6437xeaax7889) April 2019. Publisher: American Association for the Advancement of Science.
[21] Cristopher M. Niell and Michael P. Stryker. Modulation of Visual Responses by Behavioral State in Mouse Visual Cortex. Neuron, 6544)777, February 2010.
[22] Emmanouil Froudarakis. Paul G. Fahey, Jacob Reimer, Stelios M. Smirnakis, Edward J. Tehovnik, and Andreas S. Tolias. The Visual Cortex in Context. Annual Review of Vision Science, 5(1 5(1):317-339, 2019.
_eprint: https://oi.org// I 46/ammrrex-vissoon-09955177777-1
[23] Ame Meyer, Jasper Poort, John O' Keeefe, Maneesh Sahani, and Jennifer F. Linden. A Camera System Integrates Detailed Behavioral Monitoring with Multichannel Electrophysiology in Freely Moving Mice. Neuron, 100(1) (4660,,,6 October 2018.
[24] Grigori Guitchounts, Javier Masís, Steffen B. E. Wolff, and David Cox. Encoding of 3D Head Orienting Movements in the Primary Visual Cortex. Neuron, 108(3):512-5255 525..4, November 2020.
[25] Philip R. L.. Parker, Elliot T. T. Abe, Emmalyn S. P. Leonard, Dylan M. Martins, and Cristopher M. Niell. Joint coding of visual input and eye/head position in V1 of freely moving mice. Neuron, September 2022.
[26] Laura Busse, Asli Ayaz, Neel T. Dhruv, Steffen Katzner, Aman B. Saleem, Marieke L. Schölvinck, Andrew D. Zaharia, and Matteo Carandini. The Detection of Visual Contrast in the Behaving Mouse. Journal of Neuroscience, 31331)111311-11361 August 2011. Publisher: Society for Neuroscience Section: Articles.
[27] Sylvia Schröder, Nicholas A. Steinmetz, Michael Krumin, Marius Pachitariu, Mattco Rizzi, Leon Lagnado, Kenneth Harris, and Matteo Carandini. Arousal Modulates Retinal Output. Neuron 1070))487-495.e) August 2020.
[28] Cristopher M. Niell and Massimo Scanziani. How Cortical Circuits Implement Cortical Computations: Mouse Visual Cortex as a Model. Annual Review of Neuroscience, 44(1):517-546, 2021. _eprint: https://doi.org/10.
[29] Pieter M. Goltstein, Sandra Reinert, Tobias Bonhocffer, and Mark Hübener. Mouse visual cortex arcas represent perceptual and semantic features of learned visual categories. Nature Neuroscience, 14410 1451, October 2021. Number: 10 Publisher: Nature Publishing Group.
[30] Ján Antolík, Sonja B. Hofer, James A. Bednar, and Thomas D. Mrsic-Flogel. Model Constrained by Visual Hierarchy Improves Prediction of Neural Responses Natural Scenes. PLOS Computational Biology, 1604927, June 2016. Publisher: Public Library of Science
[31] Mohammad Bashiri, Edgar Walker, Konstantin Klemens Lurz, Akshay Jagadish, Taliah Muhammad, Zhiwei Ding. Zhuokun Ding, Andreas Tolias, and Fabian Sinz. A flow-based latent state generative model of neural population responses to natural images. In Advances in Neural Information Processing Systems, volume 34, pages 15801-15815. Curran Associates, Inc., 2021.
[32] Aran Nayebi, Nathan C. L. Kong, Chengxu Zhuang, Justin L. Gardner, Anthony M. Norcia, and Daniel L. K. Yamins. Mouse visual cortex as limited resource system that self-learns ccologically-gomee general representation. PLOS Computational Biology, (010))1001506, October 2023. Publisher. Public Library of Science.
[33] Jianghong Shi, Eric Shea-Brown, and Michael Buice. Comparison Against Task Driven Artificial Neural Networks Reveals Functional Properties in Mouse Visual Cortex. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[34] Jianghong Shi, Bryan Tripp. Eric Brow-B Stefan Mihalas, and Michael A. Buice. MouseNet: A biologically constrained convolutional neural network model for the mouse visual cortex. PLOS Computational Biology, 1010047, September 2022. Publisher: Public Library of Science.
12

---

[35] Galen Pogoncheff, Jacob Granley, and Michael Beyeler. Explaining VI Properties with Biologically Constrained Deep Learning Architecture. In Advances Neural Information Processing Systems. Curran Associates, Inc., November 2023.
[36]
Julie A. Harris, Stefan Mihalas, Karla E. Hirokawa, Jennifer D. Whitesell, Hannah Choi, Amy Bernard, Phillip Bohn, Shiella Caldejon, Linzy Casal, Andrew Cho, Aaron Feiner, David Feng, Nathalie Gaudreault. Charles R. Gerfen, Nile Graddis, Peter A. Groblewski, Alex M. Henry, Anh Ho, Robert Howard, Joseph E. Knox, Leonard Kuan, Xiuli Kuang. Jerome Lecoq. Phil Lesnar, Yaoyao Li, Jennifer Luviano, Stephen McConoughey, Marty T. Mortrud, Maitham Nacemi, Lydia Ng. Seung Wook Oh, Benjamin Ouellette, Elisc Shen, Staci A. Sorensen, Wayne Wakeman, Quanxin Wang, Yun Wang, Ali Williford, John W. Phillips, Allan R. Jones, Christof Koch, and Hongkui Zeng. Hierarchical organization of cortical and thalamic connectivity. Nature, Group.
Novemher 2019. Number: 7781 Publisher: Nature Publishing
[37] Joshua H. Siegle, Xiaoxuan Jia, Séverine Durand, Sam Gale, Corbett Bennett, Nile Graddis, Greggory Heller, Tamina K. Ramirez, Hannah Choi, Jennifer A, Luviano, Peter A. Groblewski, Ruweida Ahmed, Anton Arkhipov, Amy Bernard, Yazan N. Billeh, Dillan Brown, Michael A. Buice. Nicolas Cain, Shiella Caldejon, Linzy Casal, Andrew Cho, Maggie Chvilicek. Timothy C. Cox, Kael Dai, Daniel Denman. Saskia E. J. de Vries, Roald Dietzman, Luke Esposito, Colin Farrell, David Feng. John Galbraith, Marina Garrett, Emily C. Gelfand, Nicole Hancock, Julie A. Harris, Robert Howard, Brian Hu, Ross Hymenn Ramakrishnan Iyer, Erika Jessett, Katelyn Johnson, India Kato, Justin Kiggins, Sophic Lambert, Jerome Lecoq. Peter Ledochowitsch, Jung Hoon Lec, Arielle Leon, Yang Li, Elizabeth Liang. Fuhui Long, Kyla Mace, Jose Melchior, Daniel Millman, Tyler Mollenkopf. Chelsea Nayan. Lydia Ng. Kiet Ngo, Thuyahn Nguyen, Philip R. Nicovich, Kat North, Gabriel Koch Ocker, Doug Ollerenshaw, Michael Oliver, Marius Pachitariu, Jed Perkins, Melissa Reding, David Reid, Miranda Robertson, Kara Ronellenfitch, Sam Seid, Cliff Slaughterbeck, Michelle Stoecklin, David Sullivan, Ben Sutton, Jackie Swapp. Carol Thompson, Kristen Turner, Wayne Wakeman, Jennifer D. Whitesell, Derrie Williams, Ali Williford. Rob Young. Hongkui Zeng. Sarah Naylor, John W. Phillips, R. Clay Reid, Stefan Mihalas, Shawn R. Olsen, and Christof Koch. Survey of spiking in the mouse visual system reveals functional hierarchy. Nature, 592(7752)) April 2021.
Number: 7852 Publisher: Nature Publishing Group.
[38] Angie M Michaiel, Elliott TT Abe, and Cristopher M Niell. Dynamics of gaze control during prey capture in freely moving mice. ele 9ce57458, July 2020. Publisher: el lffe Sciences Publications, Ltd.
[39] Aman B Saleem. Two stream hypothesis of visual processing for navigation in mouse. Current Opinion in Newrobiology, 64:70-78, October 2020.
[40] Cristopher M. Niell. Cell Types, Circuits, and Receptive Fields in the Mouse Visual Cortex. Annual Review of Neuroscience, 38(1):41 431, 2015. _eprint: https://ddii.or/10.. http://...1////nneeeeeeee.e (at-aauuuuu--euuuo00711-----0000
[41] Katrin Franke. Konstantin F. Willeke, Kayla Ponder, Mario Galdamez. Na Zhou, Taliah Muhammad, Saumil Patel, Emmanouil Froudarakis, Jacob Reimer, Fabian H. Sinz, and Andreas S. Tolias. State- dependent pupil dilation rapidly shifts visual feature selectivity. Nature, 610(7930):128- 134. October 2022. Number: 7930 Publisher: Nature Publishing Group.
[42] Edgar Walker, Fabian Sinz, Erick Cobos, Taliah Muhammad, Emmanouil Froudarakis, Paul G. Fahey, Alexander S. Ecker, Jacob Reimer. Xaq Pitkow, and Andreas S. Tolias. Inception loops discover what excites neurons most using deep predictive models. Nature Neuroscience, 2212222000-20 December 2019. Number: 12 Publisher: Nature Publishing Group.
[43] Alexander Mathis, Pranav Mamidanna, Kevin M. Cury, Taiga Abe. Venkatesh N. Murthy, Mackenzie Wey- gandt Mathis, and Matthias Bethge. DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nature Neuroscience, (1()): September 2018. Number: 9 Publisher: Nature Publishing Group.
[44] Nicholas A. Steinmetz, Cagatay Aydin, Anna Lebedeva, Michacl Okun, Marius Pachitariu, Marius Bauza, Maxime Beau, Jai Bhagat, Claudia Böhm, Martijn Broux, Susu Chen, Jennifer Colonell, Richard J. Gardner, Bill Karsh, Fabian Kloosterman, Dimitar Kostadinov, Carolina Mora-Lopez, John O'Callaghan, Junchol Park. Jan Putzeys. Britton Sauerbrei, Rik J. J. van Daal, Abraham Z. Vollan, Shiwei Wang, Marleen Welkenhuysen. Zhiwen Ye, Joshua T. Dudman, Barundeb Dutta, Adam W. Hantman, Kenneth D. Harris, Albert K. Lee, Edvard Moser, John Keefe, Alfonso Renart, Karel Svoboda, Michael Häusser, Sebastian Haesler, Matteo Carandini, and Timothy D. Harris. Neuropixels 2.0: A miniaturized high-density probe for stable, long- term brain recordings. Science, 372(6539).cab44588 April 2021. Publisher: American Association for the Advancement of Science.
13

---

[45] Philip R. L. Parker, Dylan M. Martins, Emmalyn S. P. Leonard, Nathan M. Casey. Shelby L. Sharp, Elliott
T. T. Abe, Matthew C. Smear, Jacob L. Yates, Jude F. Mitchell, and Cristopher M. Niell. A dynamic sequence visual processing initiated by gaze shifts, August 2022. Pages: 2022.08.23.504447 Section: New Results.
[46] Pouya Bashivan, Kohitij Kar, and James DiCarlo. Neural population control via deep image synthesis. Science, 364(6449)) May 2019. Publisher: American Association for the Advancement of Science.
[47] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep Inside Convolutional Networks: Visualis- ing Image Classification Models and Saliency Maps, April 2014. arXiv: 1tt1:1311.6004 [cs].
[48] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition, December 2015. arXiv: 1etr::11200388 [es].
[49] Mingxing Tan and Quoc V. Le. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks, September 2020, arXiv:1905, 1946 [cs, stat] version: 5.
[50] Gert Van den Bergh, Bin Zhang. Lutgarde Arckens, and Yuzo M. Chino. Receptive-field properties of VI and V2 neurons in mice and macaque monkeys. Journal of Comparative Neurology, 518(11):2051-2770 2010. _eprint: https://onlinelibaaryyw www.aaayywwlly.oo..... otwp://rwww.iilc.ooooooo/ooo/ooo wttp/////...00//mmm.o 222311
[51] Michael Beyeler, Emily L. Rounds, Kristofor D. Carlson. Nikil Dutt, and Jeffree L. Krichmar, Neural
correlates of sparse coding and dimensionality reduction. PLOS Computational Biology, 1506908, June 2019.
[52] S. Fusi, E. K. Miller, and M. Rigotti. Why neurons mix: high dimensionality for higher cognition. Curr Opin Neurobiol, 37:66-74 74, April 2016.
[53] A. Pouget and L. H. Snyder. Computational approaches to sensorimotor transformations. Nat Neurosci, Suppl:1192-8 November 2000.
[54] Marshall G. Shuler and Mark Bear. Reward Timing in the Primary Visual Cortex. Science, 311(5767):1006-6609 March 2006. Publisher: American Association for the Advancement Science.
14

---

Appendix
A
Data Acquisition
The full procedure of data collection and data preprocessing is described in detail in Ref. [25], but briefly described below for the reader's convenience.
Visual scenes from the mouse's perspective, neural activity in VI, and various behavioral variables simultaneously recorded from three adult mice who were freely exploring arena with state-of- the-art head-mounted recording system [25]. The recording system consisted high-density silicon probes, miniature cameras and an inertial measurement unit (IMU). Elletrophyssollogy data was acquired at 30 kHz using a 11 um X 15 pm multi-shank linear silicon prohe (128 channels) implanted in the center of the left monocular VI. One wide-angled camera (around 120%) was aimed outwards to capture the visual scene available to the right eye of the mouse at 16 ms per frame second camera was aimed the right eye (illuminated by an infrared-LED) record a video feed of the right eye at 30 Hz. The IMU acquired three-axis gyroscope and accelerometer data at 30 kHz. In addition, a top-down camera recorded the mouse in the arena at 60 Hz.
("worldcam"),
During experiments, mice were placed in an arena where they could move around freely for about 1 hour. The arena approximately 48 em long 37 cm wide 30 em high. The gray floor was covered with black-and-white Legos to provide visual contrast. One wall of the arena was a monitor displaying moving black-and-white spots stimulus, and the other three walls wallpaper with static stimuli including white noise, black-and-whhe and black-and-whit low-spatial-frqquenyy gratings. order encourage foraging behavior during the recording sessions, small fragments of tortilla chips were sparsely distributed across the arena.
covered with gratings,
high-spatial-frequeees
The worldcam video was downsampled to 60 80 pixels. DeepLabCut [43] was used to extract extract eye position (6, Φ) and pupil radius (o)) Pitch (p) and roll (w) of the mouse's 's head position were extracted from the IMU. Locomotion speed (s) estimated from the top-down camera feed using DeepLabCut [43]. Electrophysiology data bandpass-filtes filtered between 0.01 Hz and 7.5 kHz, and spike-sorted with Kilosort 2.5 [44). Single units were selected using Phy2 (see [45]) and inactive units (mean firing rate < 3 Hz) were removed. This yielded 68, 32, and 49 active units for Mouse 1-3, respectively. To prepare the data for machine learning, all data streams were deinterlaced and resampled at 20.83 Hz (48 ms per frame; Fig. IC).
15

---

B Vision-Only Models
B.1 Hyperparameter Tuning
We performed a grid search to find the optimal CNN kernel size (3. 5, 9), number of channels (32, 64, 128. 256, 512; in various combinations), and dropout rate (0, 0.25, 0.5). While other models often rely on kernel size 3 for their CNN, we found these small kernels to lead to worse performance, perhaps due to the mouse's low-resolution vision. Kernel size performed best.
We repeated the grid search for CNNs with different number of convolutional layers. The resulting 3-layer CNN with 0.5 dropout CNN with 1024 channels (i.e., a shallow but wide network), a 2-layer CNN, or a 4-layer CNN. Choice of learning rates and optimizers had notable effect the final performance of the networks.
outperformed many differently sized networks, such as I-layer
B.2 Autoencoder
We initially hypothesized that an autoencoder could provide regularization benefits over a "vanilla" CNN, because the reconstruction loss might encourage the model to learn visual features that useful for decoding. We used an encoder & to map the original frame F a vector V in the latent space, which was present at the bottleneck, while the decoder then mapped the vector from the latent space to the output.
F+V, / V - F,
Φ argmin a. ||F (sb d0.F1
After hyperparameter search, we settled on size 256 for the latent space vector, and the weight of the reconstruction loss relative to the Poisson loss was fixed at 0.5. Both the encoder and the decoder were 3-layer CNNs, and their numbers of channels were symmetric. However, after testing a number of autoencoders with different configurations (Table 4), we found that simple 3-layer CNN outperformed any and all of them.
Mouse
Mouse 2
Mouse 3
Kernel size, enender #channels
f
MSE
t
MSE J.
cc
MSR +
3, 64
.539
.0728
.389 .128
.107
.502
.129
.0996
5. 32 x 64
.550 = .0728
.363 .116
.109
.135
.0983
7, x x 64
.525 = .152
.0732
.353 + .121
.117
.509
.131
.0980
9, 1.66 x 32 x 64
.518 = .147
.0752
.355 + .100
.119
.492
.135
.0997
3,32 64 x 128
.543 - .144
.0737
.367 + .128
.109
.503
.131
.100
5, x 64 x 128
.551 = .149
.0723
.31 = .009
.119
.514 .132
.0984
7.32 x 64 x 128
.539 = .145
.0739
.30 + .188
.009
.492 .129
.00
9,32 x 64 x 128
.510 = .155
.0758
.331 + .19
.112
.500 + .11
.101
3, 64 x 128 x 256
.541 = .146
.0758
.34 .123
.110
.514 .127
.0990
5, 64 x 128 x 256
.552 = .145
.0777
.362 .119
.110
.508 + .31
.101
7, 64 x 128 256
.53 + 134
.6688
.669 + 1044
1.04
.111
.530 + 136
1336
.0992
9. 64 128 x 256
.37 .16
.0811
.355 - ..09
.199
.5000 - .18
.105
Table 4: Performance of different autoencoders. The numbers of channels in the decoder were symmetric with those of the encoder. Best performing networks indicated in bold. ce: cross- correlation, mean standard deviation across neurons (f: the higher the better), MSE: mean-squared error (J: the lower the better).
16

---

C Visual Receptive Fields
To whether the recovered visual receptive fields are sensitive to the choice of initial values for the behavioral variables, we repeated our experiments by initializing the behavioral variables with noise sampled from the uniform distribution [-1,1) 1). The values remained the same throughout the process of gradient ascent.
We found that different values of behavioral variables resulted in similar visual receptive fields. A representative example is shown in Fig. C.I. Although some of the absolute values changed, receptive field structure stayed qualitatively the same, with identical excitatory and inhibitory subregions the ones reported in Fig. 4.
Mouse
Mouse
Mouse 3
43m
2-414
Figure C.1: The maximally activating stimuli learned in CNN+GRU, generated via gradient ascent with a fixed and randomly initialized behavioral variable. The 32 neurons with the highest cross- correlation (cc) from each mouse are shown, sorted by cc.
17


Reviews:
[{"review_id": "rArFmd9Zab", "reviewer": "Reviewer_ZQ1m", "summary": "The authors in this work propose a novel multimodal approach to design encoder models of mouse visual processing. The authors identify the limitations of unnaturalistic (head-fixed) recording, limited behavioral inputs and lack of temporal dynamics in prior recordings and models of mouse visual system and address these with new data recorded from 3 freely-moving mice that they have access to which fixes the above-mentioned issues. The multimodal architecture, a CNN-GRU encoder simultaneously encodes time-locked visual input and elaborate mouse behavioral state (running speed, eye and head position, pupil size, the first derivatives and pairwise products of these quantities). A neural response prediction readout is attached to the output of the encoder. The authors show that a 3-layer CNN is the best predictor of mouse visual responses and that behavioral input consistently outperforms a purely vision-based encoder. Further gradient ascent visualization and saliency-based analyses add interpretability to the above encoder.", "questions": "Please find my suggestions in the weaknesses section of my review.", "limitations": "Limitations have been adequately addressed in this submission.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The proposed work is quite original and sound in the joint-computational modeling of mouse visual and behavioral responses in freely-moving mice. The encoder architecture proposed by the authors is quite interpretable and empirically performs well to predict the neural activity data. The originality in this work in my opinion is from the modeling that the authors propose. The dataset that has been used here seems to not be original to this work but prior work referenced by the authors in the Methods section.\n+ It is (although not surprising) clear from the experimental results that joint-encoding of vision and behavior outperforms predictions from a purely vision based model. The methods and results sections have been written quite well in adequate details on preprocessing techniques, training of the encoder and metrics used to evaluate the quality of visual encoding.\n+ Additional visualization and saliency based analysis performed by the authors adds interpretability to the proposed encoder model as it shows well-defined receptive fields (mostly those model neurons with high correlation to Mouse 1 neurons though) and selectivity to various behavioral attributes.", "weaknesses": "- Unfortunately it seems that the dataset has been published already and is accessed by this submission; the modeling efforts built on top seem like a relatively small contribution in my opinion. Especially since the kind of architecture designed and its hyperparameters aren't justified with a systematic search. Did the authors attempt alternative architectural choices, normalization techniques other than BatchNorm, different training approaches (unsupervised / self-supervised) or other recurrent units that are not included in this submission? Please provide the required justification for the presented architectural choices.\n- Related work section seems to be quite limited and I believe it could be enhanced to further include more information about other relevant computational models of mouse visual activity.\n- Other than Figure 1 and Figure 4, the other figures seem to have poor spatial resolution and need to be improved in quality for better readability.\n- Very little information is included about the neural activity dataset, I believe it would be good to add more details about the kind of visual stimuli used and about the recording setup in the Methods section or in the supplementary information.\n- I could not find mention of whether the authors are open to sharing their models and code, this would be appreciated if the authors want to maximize the reproducibility of their work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their encouraging comments and are pleased that they agree about the originality of the work. We do believe that this work constitutes a significant modeling contribution, as most previous modeling attempts have focused on head-fixed datasets with static stimuli, which may be appropriate for foveating animals, but arguably provide limited insight into visual processing in real-world environments. The extension to freely-moving datasets is not trivial, and the present work demonstrates state-of-the-art performance on a dataset collected with state-of-the-art equipment.\n\n> Please provide the required justification for the presented architectural choices. Did the authors attempt alternative architectural choices, normalization techniques other than BatchNorm, different training approaches (unsupervised / self-supervised) or other recurrent units that are not included in this submission?\n\nWe indeed experimented with several different network architectures (inspired by the results of the Sensorium competition) prior to arriving at the proposed network. This included autoencoders (whose details are described in the appendix), variational autoencoders, pre-trained deep neural networks (ResNet and EfficientNet), filter bank models, and LSTM/RNN networks. The hyperparameter search for the CNN architecture is described in detail in the appendix.\nWe did not attempt normalization techniques other than BatchNorm, because we found BatchNorm to be most effective in our previous experience with multimodal modeling and BatchNorm was standard in the field [11], [12]. We did not try unsupervised or self-supervised learning because our task (predicting the firing rate of each neuron based on visual and behavioral inputs) was supervised in nature and it was not clear how to formalize it so that other training techniques could be applied. We agree this could be a promising avenue for future research.\n\n> Related work section seems to be quite limited and I believe it could be enhanced to further include more information about other relevant computational models of mouse visual activity\n\nWe agree that this section should be expanded to include other relevant computational models of the mouse visual activity. We will make sure of that in the camera-ready version.\n\n> Other than Figure 1 and Figure 4, the other figures seem to have poor spatial resolution and need to be improved in quality for better readability.\n\nThank you for pointing out the image resolution issue. We will ensure the figures are of high spatial resolution in the camera-ready version.\n\n> Very little information is included about the neural activity dataset\n\nWe agree that more information about the neural activity dataset could facilitate the flow of the paper, but we did not go into too much detail because space was limited and because it was reported in detail in the original publication which described the experimental results and the dataset that we accessed [24]. We will make sure to include a more detailed description of the dataset in the appendix in the camera-ready version.\n\n> I could not find mention of whether the authors are open to sharing their models and code\n\nThe reviewer raises an important point, and we are happy to share all pre-trained models and code (line 151)."}, {"review_id": "zGlJJkMcuY", "reviewer": "Reviewer_NhDi", "summary": "The authors introduce a multimodal recurrent neural network to integrates information beyond vision - behavioral and temporal dynamics processed by a separate head to explain V1 activity in mice. This model (vision + behavioral and temporal dynamics) is the state-of-the-art in prediction of V1 activity during free exploration. Further, this model is analyzed using maximal-activating stimuli and saliency maps, to obtain insights into function of mice V1 area.", "questions": "* I am surprised ResNet models do not do as well the smaller CNN. The skip connections in the resnet should have resulted in the model being at least as good as the shallower model. Have you looked into why?\n", "limitations": "NA", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper is well written and the integration of information beyond vision (or any one modality) for neural predictivity is novel, as far as I know. \n\n* The results are strong and convincing with proper ablation studies ", "weaknesses": "* Vision only model seems to have fewer parameters compared to the ones with GRU on top. This makes it hard to say if the improvements come from the extra parameters (unlikely).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their encouraging comments and suggestions. \n\n> Vision only model seems to have fewer parameters compared to the ones with GRU on top. This makes it hard to say if the improvements come from the extra parameters (unlikely).\n\nIt is true that our vision-only models had fewer parameters than our vision-and-behavior models - this was unavoidable as adding behavioral and temporal dynamics increased the input space and thus necessitated more parameters. However, we agree with the reviewer that the performance improvements were not primarily due to the added number of parameters, for several reasons. First, we found that adding more parameters to the vision-only model did not improve performance. For this, we investigated several vision-only alternatives, which are described in the appendix, such as adding additional layers (Appendix A.1), more channels (Appendix A.1), and a decoder architecture  (Appendix A.2 and Table 4). Second, we experimented with both GRU and LSTM architectures, and found that GRU (despite the lower number of extra parameters) drastically outperformed LSTM.\n\n> I am surprised ResNet models do not do as well the smaller CNN. The skip connections in the resnet should have resulted in the model being at least as good as the shallower model. Have you looked into why?\n\nThe reviewer raises a great point. During our early experiments with different ResNet architectures, we noticed that these models were drastically overfitting our dataset. It is possible that this problem could be alleviated with more data, but as is, our dataset is already large by experimental standards in the field. Another possibility is that mouse visual cortex has a shallower architecture than macaque visual cortex (as pointed out in the main text), thus making deeper ResNet architectures less suitable for the task.\n"}, {"review_id": "1rfznzDEz3", "reviewer": "Reviewer_riS4", "summary": "The authors use convolutional neural networks to fit data from neuronal recordings in primary visual cortex (V1) of the mouse while the animal is freely exploring the environment. The model incorporates visual signals but also other behavioral variables (“multimodal” aspect). The proposed model provides better fits to neuronal data than other benchmark models. The neuronal and behavioral data are particularly interesting and can provide interesting insights into neuronal computations when interpreted with mechanistic models.", "questions": "The differences across different neurons (section starting on line 212) is quite interesting, especially if it can be connected in the future to the ongoing efforts to characterize different cell types in mice. Expanding on this section would be useful in terms of neuroscience. What is distinct about each type of neuron, how do their properties relate to their firing rates, variability, receptive field sizes and properties, etc. This is where the paper can go from data fitting into first steps into a mechanistic understanding of V1 properties.", "limitations": "There is no discussion about limitations, of which there are plenty.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The ability to fit neural data in a dynamic fashion is interesting and distinguishes this work from previous work in the field.\n\nNumerically, the proposed model and incorporation of behavioral variables provide better fits to the data than the alternatives tested.\nThe differences across different neurons (section starting on line 212) is quite interesting, especially if it can be connected in the future to the ongoing efforts to characterize different cell types in mice.\n\nThe neuronal and behavioral data are particularly interesting and probably the highlight of the paper. These data can provide interesting insights into neuronal computations when interpreted with mechanistic models and would be very useful for future investigators that are interested in building theories and models of brain function.", "weaknesses": "One could imagine incorporating a lot of different world variables to fit neuronal responses. Here the authors choose a very reasonable set of such variables, including pupil size, head direction, moving speed. What exactly this means in terms of the function of V1 is not clear.\n\nThis kind of neural data fitting has become extremely common in the field. However, it remains unclear what kind of conclusions one can draw from this type of data fitting. \nTake the first sentence of the results. Table 1 does show that the 3-layer CNN is slightly better than other models, as the subtitle indicates. Then what? Does this mean that mouse V1 functions like a 3-layer CNN? Clearly not. There are lots of problems with 3-layer CNNs, including adversarial attacks, lack of robustness to noise, etc. These issues are not tested here. Thus, it is difficult to state what we learn about V1 function from the fact that the 3-layer CNN is slightly better than, say, ResNet-18. It is certainly better in fitting data from this experiment, what this means in the big picture of V1 computations is unclear. \n\nThe next section indicates that behavioral variables improve the data fitting, which I agree with. However, it remains very unclear that V1 neurons would have inputs that directly reflect the behavioral variables incorporated into the model. Perhaps V1 neurons could get indirect inputs that relate to pupil size, head direction, running speed, etc. From a neuroscience perspective, the critical question is to understand the anatomical inputs, the mechanisms underlying these dependences. \n\nThe notion of mixed selectivity alluded to in the discussion has been extensively used in neuroscience and remains rather useless. Neurons show “mixed selectivity” only insofar as the investigators use variables that are not directly related to the neuronal responses. Without mechanistic models, there are lots of representations that might seem as “mixed selectivity”. As a trivial example, consider a neuron that responds to 45 degrees orientation preferentially. But the investigators try to fit the responses in terms of horizontal and vertical and then claim “mixed selectivity!”. There is obviously no such thing, the neuron represents 45 degrees, period . The claims in the paper follow the same logic. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed feedback and constructive comments. We agree that data fitting is not the final answer to understanding the functioning of mouse visual cortex, and we do not presume to be able to provide such an answer. However, we believe that our modeling efforts may provide valuable insights about visual cortex function, potentially leading to mechanistic insights in the future. For instance, previous models of mouse V1 have seen limited success in incorporating behavioral variables in their prediction (e.g., Sensorium+ competition). Rather than concluding that behavioral input may play a secondary role in visual cortex function, our study suggests that the move to freely-moving datasets is paramount to revealing the role of behavioral input to predictions of V1 activity. Moreover, identifying the behavioral variables that modulate processing (and particularly their impact on coding) may guide future experimental efforts to identify their neural basis.\n\n> However, it remains very unclear that V1 neurons would have inputs that directly reflect the behavioral variables incorporated into the model. Perhaps V1 neurons could get indirect inputs that relate to pupil size, head direction, running speed, etc. From a neuroscience perspective, the critical question is to understand the anatomical inputs, the mechanisms underlying these dependences.\n\nWe agree with the reviewer in general. However, our results are indeed aligned with recent neuroscientific work that highlights how locomotion, arousal, and motor signals enter V1 (Fu et al., 2014, doi:10.1016/j.cell.2014.01.050; Leinweber et al., 2017, doi:10.1016/j.neuron.2017.08.036). Froudarakis et al. 2019 (doi:10.1146/annurev-vision-091517-034407) and Parker et al. 2020 (doi:10.1016/j.tins.2020.05.005) provide a thorough review of non-visual inputs to visual cortex. Given that these anatomical inputs exist, the question remains how V1 neurons may integrate these behavioral variables with visual information. To this end, our study provides a computational account of how V1 neurons may combine these multimodal inputs during free exploration.\n\n> The notion of mixed selectivity alluded to in the discussion has been extensively used in neuroscience and remains rather useless.\n\nWe agree with the reviewer that “mixed selectivity” should be clearly defined in order to be meaningful. In the context of this study, mixed selectivity refers to a neuron encoding two different categories of information: e.g., visual and motor signals, or stimulus and reward signals. Combining horizontal and vertical edges would not count as such; this would be a purely visual response. We will make sure to clearly define this term in the camera-ready version. \n\n> The differences across different neurons (section starting on line 212) is quite interesting, especially if it can be connected in the future to the ongoing efforts to characterize different cell types in mice. Expanding on this section would be useful in terms of neuroscience.\n\nWe agree with the reviewer that an important next step is to connect our results to characterizing different cell types in mice from a mechanistic and anatomical point of view. While at present we can speculate about these differences based on our model fits, to thoroughly answer this question, we would require a different set of tools that are beyond the scope of this work.\n"}, {"review_id": "Tpovi8QdYv", "reviewer": "Reviewer_xdTa", "summary": "The authors propose a neural network for predicting activity of V1 neurons in freely moving mice. The data consists of ~hour long electrophysiological recordings of a neuronal population while the mouse is freely exploring the space. Simultaneously with the neural activity, the experimental setup allows the authors to record the visual scenes from the mouse perspective as well as a number of behavioral variables.\n\nThe network consists of two modules: one computes visual features, the other encodes behavioral variables. The outputs of both modules are concatenated and fed into the recurrent unit (GRU) accounting for the temporal dynamics, the output of which is decoded into the predicted neural activity of each neuron in the recorded population.\n\nThe models achieve state-of-the-art performance both among the visual models and among those including behavioral variables. The authors analyze the model by computing MEIs as well as performing saliency analysis to study the effect of individual behavioral variables.", "questions": "- I am surprised to see that your vision-only network quite significantly outperforms the baselines despite having a standard CNN-FC architecture. Why do you think it is the case? And specifically for mouse 2, why do you think your model is able to perform so much better than the baselines?\n\n- Do you think the training data could leak into the test set in the 70-30 splits of continuous segments as you described in the paper? I guess it is not unlikely that a freely moving mouse could be exploring the same part of the arena in some of the train and test segments?\n\n- How did you set the values of the behavioral variables for the visual MEIs computation? It would be very interesting to see how the MEIs change as a function of behavioral variables as the animal explores the space around it.\n\n- Some of the previous work (e.g. [11]) explicitly assumed the existence of localized receptive fields by factorising the readout into sparse spatial masks and a feature vector. But looking at the MEIs you generated, many neurons don't have receptive fields, so such an architecture doesn't seem reasonable. Do you have any ideas why the models with an explicit receptive field assumption perform well nevertheless?", "limitations": "The limitations are adequately addressed in the paper.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I think it is a very good paper.\n\n+ It is clearly written\n+ It discusses an important problem (integration of behavioral variables in predicting mouse V1 activity)\n+ The proposed model achieves a noticeable improvement over the previous state-of-the-art", "weaknesses": "I don't see any significant methodological weaknesses.\n\nI think the main limitation is the small size of the dataset (a few dozen neurons per mouse). For example, the diversity of the MEIs in Fig. 4 both within and across the animals is very interesting, but a fairly small dataset size doesn't allow us to draw any quantitative conclusions about these MEIs beyond visual inspection. It would be really nice to have a calcium dataset of thousands of neurons in a freely moving mouse (similar to the Sensorium one) to address this issue, but it is of course not a critique of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their encouraging comments.\n\n> It would be really nice to have a calcium dataset of thousands of neurons in a freely moving mouse (similar to the Sensorium one)\n\nWe agree with the reviewer that creating a large and standardized neural recording dataset from freely moving mice would be the dream. We outlined this in our future work section. Unfortunately, to date, Calcium imaging in a freely moving animal is not feasible. (Calcium imaging was possible in the Sensorium competition because the animals were head-fixed.) A yield of ~100 units is considered state of the art for freely moving animals. Furthermore, it is not clear whether the fast neuronal dynamics associated with locomotion could be captured with Calcium imaging, which is relatively slow.\n\n> I am surprised to see that your vision-only network quite significantly outperforms the baselines despite having a standard CNN-FC architecture. \n\nWe were also surprised to see that vanilla CNNs were better than the other architectures we experimented with, including autoencoders, variational autoencoders, filter bank models, and pre-trained deep neural networks (ResNet and EfficientNet). In our opinion, the reason might be that the neurons in the mouse visual cortex were selective for other behavioral inputs and the vanilla CNN architecture imposed the fewest assumptions about how the visual input contributed to the neural activity. This is especially true for Mouse 2: we learned that the cortical probes of Mouse 2 were more superficial compared to the other two mice, so the recorded neurons may have both different anatomical inputs and different visual responses. \n\n> Do you think the training data could leak into the test set in the 70-30 splits of continuous segments as you described in the paper? I guess it is not unlikely that a freely moving mouse could be exploring the same part of the arena in some of the train and test segments?\n\nWhile it is possible that the mouse could have been exploring the same part of the arena at different segments of the recording, it was free to move its head, eyes, and body as it saw fit. Thus “data leak” is unlikely as two duplicate data points could only be produced by the animal exactly duplicating the time courses of its eye, head, and body movement in the exact same location of the arena.\n\n> How did you set the values of the behavioral variables for the visual MEIs computation? It would be very interesting to see how the MEIs change as a function of behavioral variables as the animal explores the space around it.\n\nThe behavioral variables were initialized to a vector of all ones, and updated in the loop with the visual MEI. We agree that setting a fixed behavioral variable vector and inspecting the changes in visual MEIs would be an interesting analysis, which we would be happy to provide in the camera-ready version.\n\n> Some of the previous work (e.g. [11]) explicitly assumed the existence of localized receptive fields by factorising the readout into sparse spatial masks and a feature vector. [...] Do you have any ideas why the models with an explicit receptive field assumption perform well nevertheless?\n\nWe agree with the reviewer that enforcing a localized receptive field may hinder good model fits. In contrast to previous work, our analysis used gradient ascent to reveal the maximally exciting visual input. We speculate that previous models may have still worked well because the assumption of a localized receptive field was imposed on the final feature map output (e.g., [11]), which is drastically different (and at a coarser resolution) than the raw visual input. Therefore, the receptive field in those models was not exactly as localized as claimed, which may explain why these models still performed well.\n\n"}]

Author rebuttal:
We thank the reviewers for their constructive comments. We are pleased that reviewers agreed this is a clearly written paper (ZQ1m, xdTa) describing novel work (ZQ1m, NhDi) with strong results (NhDi, riS4, xdTa) that may help build theories and models of brain function (riS4).
Reviewers raised insightful questions about the details of the dataset (ZQ1m, xdTa), the choice of our model architecture and comparison to alternative architectures (ZQ1m, NhDi, xdTa), and the implications for our mechanistic understanding of V1 processing (riS4).

Below we address each of the reviewer’s questions and concerns point by point. In short, the dataset we accessed in this work is the state-of-the-art setup for collecting neural activity data from freely moving mice. We have indeed performed an architecture search to arrive at our novel multimodal network architecture (see Appendix). While our selection of behavioral variables was largely guided by known computational properties of mouse V1, our multimodal model may help unveil the computational principles by which non-visual inputs are integrated in mouse visual cortex. Lastly, we would like to emphasize that we are happy to provide all our pre-trained models and code upon acceptance.


Meta-review: The reviewers agree that it’s a technically solid paper with a novel contribution that advances the field of predictive modeling of neural responses. Some reviewers raised the concern that the paper’s contribution may be “just a data fitting exercise” and questioned whether such “data fitting” is useful. While this (somewhat philosophical) discussion is certainly both interesting and important, it applies to the entire field of predictive modeling and is probably better kept separately from the decision about this individual paper.
Decision: Accept (poster)