Title: Multimodal Deep Learning Model Unveils Behavioral Dynamics of V1 Activity in Freely Moving Mice
Venue: NeurIPS 2023
Accepted: yes
Keywords: neuroscience

cognitive science

multimodal learning

representation learning

network architecture

computational biology

visual perception
Abstract: Despite their immense success as a model of macaque visual cortex, deep convolutional neural networks (CNNs) have struggled to predict activity in visual cortex of the mouse, which is thought to be strongly dependent on the animal’s behavioral state. Furthermore, most computational models focus on predicting neural responses to static images presented under head fixation, which are dramatically different from the dynamic, continuous visual stimuli that arise during movement in the real world. Consequently, it is still unknown how natural visual input and different behavioral variables may integrate over time to generate responses in primary visual cortex (V1). To address this, we introduce a multimodal recurrent neural network that integrates gaze-contingent visual input with behavioral and temporal dynamics to explain V1 activity in freely moving mice. We show that the model achieves state-of-the-art predictions of V1 activity during free exploration and demonstrate the importance of each component in an extensive ablation study. Analyzing our model using maximally activating stimuli and saliency maps, we reveal new insights into cortical function, including the prevalence of mixed selectivity for behavioral variables in mouse V1. In summary, our model offers a comprehensive deep-learning framework for exploring the computational principles underlying V1 neurons in freely-moving animals engaged in natural behavior.

Multimodal Deep Learning Model Unveils Behavioral Dynamics of V1 Activity in Freely Moving Mice

Abstract

Despite their immense success as a model of macaque visual cortex, deep convolutional neural networks (CNNs) have struggled to predict activity in visual cortex of the mouse, which is thought to be strongly dependent on the animal's behavioral state. Furthermore, most computational models focus on predicting neural responses to static images presented under head fixation, which are dramatically different from the dynamic, continuous visual stimuli that arise during movement in the real world. Consequently, it is still unknown how natural visual input and different behavioral variables may integrate over time to generate responses in primary visual cortex (V1). To address this, we introduce a multimodal recurrent neural network that integrates gaze-contingent visual input with behavioral and temporal dynamics to explain V1 activity in freely moving mice. We show that the model achieves state-of-the-art predictions of V1 activity during free exploration and demonstrate the importance of each component in an extensive ablation study. Analyzing our model using maximally activating stimuli and saliency maps, we reveal new insights into cortical function, including the prevalence of mixed selectivity for behavioral variables in mouse V1. In summary, our model offers a comprehensive deep-learning framework for exploring the computational principles underlying V1 neurons in freely-moving animals engaged in natural behavior.

1 Introduction

Computational models have been crucial in providing insight into the underlying mechanisms by which neurons in the visual cortex respond to external stimuli. Deep convolutional neural networks (CNNs) have had immense success as predictive models of the primate ventral stream, in cases where

the animal was passively viewing stimuli or simply maintaining fixation [1-5]. Despite their success, these CNNs are poor predictors of neural responses in mouse visual cortex [6], which is thought to be shallower and more parallel than that of primates [7, 8]. According to the best models in the literature [9-14], the mouse visual system is more broadly tuned and operates on relatively low-resolution inputs to support a variety of behaviors [15]. However, these models were limited to predicting neural responses to controlled (and potentially ethologically irrelevant) stimuli that were passively viewed by head-fixed animals.

Movement is a critical element of natural behavior. In the visual system, eye and head movements during locomotion and orienting transform the visual scene in potentially both beneficial (e.g., by providing additional visual cues) and detrimental ways (e.g., by introducing confounds due to selfmovement) [16-18]. Movement-related activity is widespread in mouse cortex [19, 20] and prevalent in primary visual cortex (V1) [21, 22]. For instance, V1 neurons of freely moving mice show robust responses to head and eye position [23, 24], which may contribute a multiplicative gain to the visual response [25] that cannot be replicated under head fixation. V1 activity may be further modulated by variables that depend on the state of the animal and its behavioral goals [20, 22, 26, 27]. However, how these behavioral variables may integrate to modulate visual responses in V1 is unknown. Furthermore, a comprehensive predictive model of V1 activity in freely moving animals is still lacking.

To address these challenges, we make the following contributions:

We introduce a multimodal recurrent neural network that integrates gaze-contingent visual input with behavioral and temporal dynamics to explain V1 activity during natural vision in freely moving mice.

We show that the model achieves state-of-the-art predictions of V1 activity during free exploration based on visual input and behavior, demonstrating the ability to accurately model neural responses in the dynamic regime of movement through the visual scene.

We uncover new insights into cortical neural coding by analyzing our model with maximally activating stimuli and saliency maps, and demonstrate that mixed selectivity of visual and behavioral variables is prevalent in mouse V1.

2 Related Work

The mouse, as a model organism, offers unparalleled experimental access to the mammalian cerebral cortex [28]. Computational models of mouse V1, including generalized linear models (GLMs) [25, 29] and customized models mimicking the mouse visual hierarchy [30], have been crucial in providing deeper insights into the range of computations performed by visual cortex. More recently, deep CNNs have also been used to model mouse V1 [9-13, 31-34].

Despite their success in predicting neural activity in the macaque visual cortex [35], deep CNNs trained on ImageNet have had limited success in predicting mouse visual cortical activity [6]. This is perhaps not surprising, as most ImageNet stimuli belong to static images of human-relevant semantic categories and may thus be of low ethological relevance for rodents. More importantly, these deep CNNs may not be the ideal architecture to model mouse visual cortex, which is known to be shallower and more parallel than primate visual cortex [36, 37]. In addition, mice are known to have lower visual acuity than that of primates [7, 8], and much of their visual processing may be devoted to active, movement-based behavior rather than passive analysis of the visual scene [22, 38, 39]. Although the majority of V1 neurons is believed to encode low-level visual features [40], their activity is often strongly modulated by behavioral variables related to eye and head position [23-25], locomotion [18, 21, 22], arousal [27, 41], and the recent history of the animal [26]. Furthermore, mouse V1 is highly interconnected with both cortical and subcortical brain areas, which contrasts with feedforward, hierarchical models of visual processing [22].

Acommon architectural approach that has proved quite successful is to split the network into different components (first introduced by [11]):

a 'core' network, which typically consist of a CNN used to extract convolutional features from the visual stimulus [11, 12, 25, 42], sometimes in combination with a recurrent network [11];

a 'shifter' network, which mimics gaze shifts by learning a (typically affine) transformation from head- to eye-centered coordinates, either applied to the pixel input [11, 25] or a CNN layer [12];

a 'readout' network, which learns a mapping from artificial to biological neurons [11, 12, 42].

Owing to the difficulty of developing a predictive model of mouse cortex, Willeke et al. [14] recently invited submissions to the Sensorium competition held at NeurIPS '22. The competition introduced a benchmark dataset of V1 neural activity recorded from head-fixed mice on a treadmill viewing static images, with simultaneous measurements of running speed, pupil size, and eye position. A baseline model was provided as well, which consisted of a 4-layer CNN core in combination with a shifter and readout network [12]. Even though 26 teams submitted 194 different models, the overall improvement to the baseline performance was modest, raising the single trial correlation from . 287 to . 325 in the Sensorium and from . 384 to . 453 in the Sensorium+ competition. Architectural innovations (e.g., Transformers, Normalizing Flows, YOLO, and knowledge distillation), were unable to make an impact, as most improvements were gained from ensemble methods. A promising direction was taken by the winning model, which attempted to learn a latent representation of the 'brain state' from the various behavioral variables, inspired by [20]. However, the model utilized the timestamps of the test set to estimate recent neuronal activities, which the other competitors did not have access to.

Taken together, we identified three main limitations of previous work that this study aims to address:

Head-fixed preparations. Most previous models operated on data from animals in head-fixed conditions with static stimuli, which do not mirror natural behavior and thus provide limited insight into visual processing in real-world environments. In contrast, the present work is applied to state-of-the-art neurophysiological recordings of V1 activity in freely moving mice. This represents a dramatic shift in the 'parameter space" of visual input, from static images to dynamic, real-world visual input. One could imagine that this will make the modeling process more difficult, because the stimulus set is more complex, or easier, because it is more matched to the computational challenge the brain evolved for.

Limited influence of behavioral state. Previous models often limited the influence of behavioral state to eye measurements and treadmill running speed, which were either concatenated with the visual features [14, 41], utilized in the shifter network to determine the gaze-contingent retinal input [11, 14], or used to predict a multiplicative gain factor [11].

Missing temporal dynamics. Most previous modeling works ignored the temporal factors that might influence V1 activity and overlooked the dynamic nature of visual processing (but see [11]). We overcome this limitation by utilizing approximately 1-hour-long recordings of three mice freely exploring an arena, and our model is capable of handling continuous data streams of any length.

3 Methods

Head-mounted recording system We had access to data from three adult mice who were freely exploring 48 cm long × 37 cm wide × 30 cm high arena (Fig. 1A), collected with a state-of-theart recording system [25] that combined high-density silicon probes with miniature head-mounted cameras (Fig. 1B). One camera was aimed outwards to capture the visual scene from the mouse's perspective ('worldcam') at 16 ms per frame (downsampled to 60 × 80 pixels). A second camera, aimed at the eye, was used to extract eye position ( θ , ϕ ) and pupil radius ( σ ) at 30 Hz using DeepLabCut [43]. Pitch ( ρ ) and roll ( ω ) of the mouse's head position were extracted at 30 kHz from the inertial measurement unit (IMU). θ , ϕ , ρ , and ω allowed for the worldcam video to be corrected for eye movements: A 3-layer fully-connected shifter network (where each linear layer was accompanied by Tanh and BatchNorm) was trained to predict a rotation (bounded by ± 36 ◦ ) and a shift (bounded by ± 16 pixels horizontally and ± 12 pixels vertically) based on θ , ϕ , ρ , and ω to convert each frame to head- to eye-centered coordinates. Locomotion speed ( s ) was estimated from the top-down camera feed using DeepLabCut [43]. Electrophysiology data was acquired at 30 kHz using a 11 µ m × 15 µ m multi-shank linear silicon probe (128 channels) implanted in the center of the left monocular V1, then bandpass-filtered between 0 . 01 Hz and 7 . 5 kHz , and spike-sorted with Kilosort 2.5 [44]. Single units were selected using Phy2 (see [45]) and inactive units (mean firing rate < 3 Hz ) were removed. This yielded 68, 32, and 49 active units for Mouse 1-3, respectively. To prepare the data for machine learning, all data streams were deinterlaced and resampled at 20 . 83 Hz ( 48 ms per frame; Fig. 1C). For a more detailed description of the dataset, see Appendix A and Ref. [25].

Model architecture We used a 3-layer CNN (kernel size 7, 128 × 64 × 32 channels) to encode the visual stimulus. Each convolutional layer was followed by a BatchNorm layer, a ReLU, and a Dropout layer (0.5 rate). A fully-connected layer transformed the learned visual features into a

A

D

m

64

B

ephys.

worldcam

IMU

neuron #

d

32

m

.....

eyecam

0

time (s)

9.6

visual feature vector, v (Fig. 2, top-right ). In a purely visual version of the model, v was fed into a fully-connected layer, followed by a softplus layer, to yield a neuronal response prediction.

To encode behavioral state, we constructed an input vector from different sets of behavioral variables:

S : all behavioral variables used in the Sensorium+ competition [14], consisting of running speed ( s ) , pupil size ( σ ), and its temporal derivative ( ˙ σ );

B : all behavioral variables used in [25], consisting of eye position ( θ , ϕ ), head position ( ρ , ω ), pupil size ( σ ), and running speed ( s );

D : the first-order derivatives of the variables in B , namely ˙ θ , ˙ ϕ , ˙ ω , ˙ ρ , ˙ σ , and s .

To test for interactions between behavioral variables, these sets could also include the pairwise multiplication of their elements; e.g., B × = { b i b j ∀ ( b i , b j ) ∈ B} . The input vector was then passed through a batch normalization layer and a fully connected layer (subjected to a strong L1 norm for feature selection) to produce a behavioral vector, b .

We then concatenated the vectors v , b , and their element-wise product v ⊙ b (all calculated for each individual input frame), fed them through a batch normalization layer, and input them to a 1-layer gated recurrent unit (GRU) (hidden size of 512). To incorporate temporal dynamics, we constructed different versions (GRUk) of the model that had access to k previous frames. A fully-connected layer and a softplus activation function were applied to yield the neuronal response prediction.

Training and model evaluation Since the visual input depended on the movement of the mouse and the mouse could be in very different behavioral states over the length of the recording, the data was highly inhomogeneous across time. To deal with the continuous and dynamic nature of the data, we therefore split the ∼ 1 h -long recording into 10 consecutive segments. The first 70 % of each segment were then reserved for training (including an 80-20 validation split) and the remaining 30 % for testing.

Note that it is unlikely for data to 'leak' from the train segment into the test segment. While it is possible that the mouse could have been exploring the same part of the arena at different segments of the recording, it was free to move its head, eyes, and body as it saw fit. Thus two duplicate data points could only be produced by the animal exactly duplicating the time courses of its eye, head, and body movement in the exact same location of the arena.

CNN + GRU (Vision and Behavior)

CNN (Vision only)

-n

CNN

Ius

t-1

CNN

FC+Softplus

GRU

Models were separately trained on the data from each mouse. Model parameters were optimized with Adam (batch size: 256, CNN learning rate: . 0001 , full model: . 0002 ) to minimize the Poisson loss between predicted neuronal response ( ˆ r ) and ground truth ( r ) : 1 N ∑ N i =1 (ˆ r i -r i log ˆ r i ) , where N denotes the number of recorded neurons for each mouse. We used early stopping on the validation set (patience: 5 epochs), which led all models to converge in less than 50 epochs. Due to the large number of hyper-parameters, the specific network and training settings were determined using a combination of grid search and manual exploration on a validation set (see Appendix B).

To evaluate model performance, we calculated the cross-correlation ( cc ) between a smoothed version ( 2 s boxcar filter) of the predicted and ground-truth response for each recorded neuron [25].

All models were implemented in PyTorch and trained on an NVIDIA RTX 3090 with 24GB memory. All code, data used to train the models, and weights of the trained model can be found at https: //github.com/bionicvisionlab/2023-Xu-Multimodal-Mouse-V1 .

Maximally activating stimuli We used gradient ascent [42] to discover the visual stimuli that most strongly activate a particular model neuron in our network. The visual input was initialized with noise sampled in N ( . 5 , 2) . The behavioral variables were initialized to a vector of all ones, and updated in the loop with the visual stimuli. We used the Adam optimizer to repeatedly add the gradient of the target neuron's activity with respect to its inputs. We also applied L2 regularization (weight of .02) and Laplacian regularization (weight of 0.01) [46] on the image. This procedure was repeated 6400 times. The resulting, maximally activating visual stimuli were smoothed with a Butterworth filter (low-pass, .05 cutoff frequency ratio) to reduce the impact of high-frequency noise.

Saliency map We computed a saliency map [47] of the behavioral vector for each neuron to discover which behavioral variables contributed most strongly to each model neuron's activity. We iterated through the test dataset, recorded the gradient of each behavioral input with respect to each neuron's prediction, and then averaged the gradients per neuron to obtain the saliency map.

4 Results

Mouse V1 activity is best predicted with a 3-layer CNN To determine the purely visual contribution to V1 responses, we experimented with a large number of vision architectures (see Appendix B). In the end, a vanilla 3-layer CNN (kernel size 7, 128 × 64 × 32 channels) yielded

the best cross-correlation between predicted and ground-truth responses (Table 1), outperforming the best autoencoder architecture (kernel size: 7, encoder: 64 × 128 × 256 channels, decoder: 256 × 128 × 64 channels), ResNet-18 [48] (a 20-layer CNN with the first input channel being replaced by 1), EfficientNet-B0 [49] (a 65-layer CNN with the first input channel being replaced by 1), and the Sensorium baseline [12] (a 4-layer CNN with a readout network). The greatest improvement in cross-correlation was achieved for Mouse 1.

Behavioral variables improve most neuronal predictions Once we identified the 3-layer CNN as the best visual encoder, we added the different sets of behavioral variables to the network. To allow for a fair comparison with the Sensorium+ baseline [14], we first limited ourselves to S = { σ, ˙ σ, s } , but then gradually added more behavioral variables ( B ) [25] as well the derivatives of these variables ( D ) and multiplicative pairs ( B × and {B ∪ D} × ).

The results are shown in Table 2. All models were able to outperform the Sensorium+ baseline, and the addition of behavioral variables and their interactions further improved model performance. Note that although the full model used a GRU to combine visual and behavioral features, the input sequence length was always 1 (i.e., GRU1). That being said, it is possible that the GRU learned long-term correlations that the Sensorium+ baseline model did not have access to. Nevertheless, the biggest performance improvements were gained through the addition of behavioral variables related to head and eye position (which are present in B but not in S ), their derivatives ( D ), and multiplicative interactions between these variables ( {B ∪ D} × ).

We also wondered whether the prediction of only some V1 neurons would benefit from the addition of these behavioral variables. To our surprise, the cross-correlation between predicted and ground-truth responses improved for almost all recorded V1 neurons (Fig. 3).

Table 1: Best-performing vision models, compared to the Sensorium baseline [12] (see Appendix B for more). Best-performing networks are indicated in bold. cc : cross-correlation, mean ± standard deviation across neurons ( ↑ : the higher the better), MSE: mean-squared error ( ↓ : the lower the better).

Table 2: CNN+GRU1 model, compared to the Sensorium+ baseline [12], trained on different behavioral feature sets. S = { σ, ˙ σ, s } : the set of variables used in the Sensorium+ competition [14]. B = { θ, ϕ, ω, ρ, σ, s } : the set of variables from [25]. D = { ˙ θ, ˙ ϕ, ˙ ω, ˙ ρ, ˙ σ, s } : the derivatives of B . A × = { a i a j ∀ ( a i , a j ) ∈ A} denotes the set of all multiplicative pairs. ∪ denotes the union operator. Best performing networks are indicated in bold. cc : cross-correlation, mean ± standard deviation across neurons ( ↑ : the higher the better), MSE: mean-squared error ( ↓ : the lower the better).

0.2

Mouse2

Mouse3

0.2

1.0

1.01

1.01

0.8

0.8

0.8

vis+beh

vis+beh

0.6

0.6

0.6

vis

CC

0.4

8

0.4

CC

0.4

0.2

0.2

0.2

0

0

0

0

0.2

0.4

0.6

0.8

1.0

o

0.2

0.4

0.6

0.8

1.0

0

0.2

0.4

0.6

0.8

1.0

CCvis

CCvis

CCvis

Access to longer series of data in time further improves predictive performance After we identified the full behavioral feature set ( {B ∪ D} × ) as the one yielding the best model performance, we extended the GRU's temporal dependence by allowing the input to vary from one frame ( 48 ms ) to a total of eight frames ( 384 ms ), and assessed the model's performance.

The results are shown in Table 3. The amount of temporal information needed by the model to reach peak predictive performance was similar across mice ( 288 ms , 192 ms , and 192 ms in terms of cross-correlation, 192 ms , 192 ms , and 192 ms in terms of mean-squared error, respectively). This indicates that temporal information is important for predicting dynamic neural activity. However, the dependence on temporal information has a limit, and different neurons in V1 might possess different temporal capacities.

Well-defined visual receptive fields emerge To assess whether the CNN+GRU 1 model learned meaningful visual receptive fields, we used gradient ascent (see Methods) to find the maximally activating stimulus for each neuron. Receptive fields for the 32 best-predicted neurons are shown in Fig. 4. Interestingly, most of them had well-defined excitatory and inhibitory subregions, often resembling receptive fields of orientation-selective neurons. Most excitatory and inhibitory subregions spanned approximately 30 ◦ of visual angle (the full width of the frame, 80 pixels, roughly corresponding to 120 ◦ of visual angle), which is roughly on the same order of magnitude compared to receptive field sizes typically observed in mouse V1, varying from 10 ◦ to 30 ◦ [8, 25, 50].

Receptive fields were noticeably different across mice. Whereas Mouse 1 and 3 had visual receptive fields with strongly excitatory subregions, most model neurons for Mouse 2 appeared to be inhibited

Mouse1

Mouse2

Mouse3

cC=0.875

cC=0.855

cC=0.842

cC=0.834

cC=0.784

cC=0.781

cC=0.768

ccm0.765

cC=0.827

cc=0.821

618

.771

cC=0.812

=0.806

cC=0.795

ccmO.781

CC=0.736

CC=0.700

CC=0.673

CC=0.600

cC=0.761

cC=0.759

749

0.742

CC=0.780

CC=0.776

CC=0.772

CC=0.769

cC=0.597

cC=0.591

cC=0.581

CC=0.735

CC=0.734

CC=O.733

CC=0.715

C=0.765

cC=0.758

753

CC=0.748

CC=0.567

8550=2

cC=0.519

CC=0.496

0.714

711

0.695

cc=0.691

cc=0.744

cC=O.740

cC=0.740

0.737

cC=0.486

CC=0.481

C=0.467

cC=O.437

=0.684

CC=0.660

CC=0.645

cC=0.637

cC=0.732

cC=0.715

cC=0.712

CC=0.423

cC=0.417

CC=0.406

cC=0.400

CC=0.637

627

0.626

cC=0.711

=0.701

CC=0.696

569'0=22

66E'0=23

cc=0.396

CC=O.384

CC=0.370

0.620

cC=0.616

cC=0.608

CC=0.598

6890m22

8890=23

cc=0.684

cC=0.682

cC=0.276

cC=0.255

cC=0.201

cC=0.162

597

cC=0.586

cc=0.582

cc=0.578

by visual signals (same colorbar across panels). In addition, several model neurons lacked pronounced visual receptive fields, indicating that they were more strongly driven by behavioral variables. Even though model fits were repeated with different initial values for the behavioral variables, the resulting visual receptive fields looked qualitatively the same (see Appendix C), thus demonstrating the validity of the generated receptive fields. In addition, even some of the best-predicted neurons lack a pronounced or spatially structured receptive field, implying that these neurons could be primarily driven by behavioral variables.

Analysis of behavioral saliency maps reveals different types of neurons Intrigued by the fact that some neurons lacked pronounced visual receptive fields, we aimed to analyze the influence of behavioral state on the predicted neuronal response by performing a saliency map analysis on the behavioral inputs (see Methods). Since different behavioral variables operate on different input ranges, we first standardized the saliency map activities for each behavioral variable across the model neuron population. Saliency map activities further than 1 standard deviation from the mean were then interpreted as 'driving' the neuron, allowing us to categorize each neuron as being driven by one or multiple behavioral variables (Fig. 5).

We first asked which neurons in our model were driven by which behavioral variables (Fig. 5, top ). Consistent with [25], we found a large fraction of model neurons driven by eye and head position, and smaller fractions driven by locomotion speed and pupil size. Approximately 20-30% of neurons were not driven by any of these behavioral variables, rendering their responses purely visual.

However, a particular neuron could be driven by multiple behavioral variables. Repeating the above analysis, we found that most model neurons showed mixed selectivity (i.e., responding to different categories of information, such as visual and motor signals, or stimulus and reward signals), with only a minority of cells responding exclusively to a single behavioral variable, (Fig. 5, middle ). Adding the interaction terms between behavioral variables (Fig. 5, bottom ) did not change the fact that most model V1 neurons encoded combinations of multiple behavioral variables, often relating information about the animal's eye position to head position and locomotor speed.

5 Discussion

In this paper, we propose a deep recurrent neural network that achieves state-of-the-art predictions of V1 activity in freely moving mice. We discovered that our model outperforms previous models under these more naturalistic conditions, which could be attributed to the better alignment of this data with the computations performed by the mouse visual system, based on its natural visual environment and

Mouse3

driven by

position

speed

!!dnd

visiononly

exclusively

position

drivenby

speed

-

!!dnd

multiple

.S

vision only

position

exclusively

speed

1

driven by

headxeye

headxspeed

eyexspeed

.S

eyexpupil

multiple

visiononly

0.0

0.20.40.60.8

1.00.0

0.20.4

0.6

0.8

1.0

0.0

0.20.4

0.6

0.8

1.0

fractionofneurons

fractionofneurons

fractionofneurons

behavioral characteristics. Similar to previous models, we found that a simple CNN architecture is sufficient to predict the visual response properties of cells in mouse V1.

In addition, mouse V1 is known to be strongly modulated by signals related to the movement of the animal's eyes, head, and body [21, 22, 25], which are severely restricted in head-fixed preparations. Models trained on head-fixed preparations may thus be limited in their predictive power. In contrast, our model was able to predict V1 activity on a 1-hour continuous data stream, during which the animal freely explored a real-world arena. Our analyses demonstrate the impact of the animal's behavioral state on V1 activity and reveal that most model V1 neurons exhibit mixed selectivity to multiple behavioral variables.

Accurate predictions of mouse V1 activity under natural conditions Our brains did not evolve to view stationary stimuli on a computer screen. However, most research on neural coding in vision has been conducted under head-fixed conditions, which do not mirror natural behavior and thus provide limited insight into visual processing in real-world environments. Some visual functions mediated by the ventral stream, such as identifying faces and objects, resemble this condition, but the real visual environment is constantly shifting due to self-motion, leading to dynamic activities such as navigation or object reaching, typically mediated by the dorsal stream. To truly understand visual perception in natural environments, we need to capture the computational principles when the subjects are in motion.

In this research, we take the initial steps towards this by modeling a novel data type encompassing neural activity coupled with a visual scene captured from a freely moving animal's perspective. This represents a dramatic (but, in our opinion, crucial) shift in the 'parameter space' of visual input, from static images projected on a screen to dynamic, real-world visual input.

Surprisingly, visual responses were best predicted with a standard three-layer ('vanilla') CNN (Table 1), as compared to a multitude of more sophisticated models that included autoencoders, variational autoencoders, filter bank models, and pre-trained ResNet and EfficientNet architectures. One possible explanation might be that the neurons in our dataset were selective for other behavioral inputs that we did not have access to, and that the vanilla CNN architecture imposed the fewest assumptions about how visual input contributed to the neural activity. In addition, visual receptive fields for Mouse 2 were noticeably different from the other two mice, exhibiting pronounced inhibitory subregions (Fig. 4, center ). This is consistent with the fact that the cortical probes of Mouse 2 were more superficial compared to the other two mice [25], so the recorded neurons may have both different anatomical inputs and different visual responses.

Mixed selectivity of behavioral variables Our experiments demonstrated that the models incorporating behavioral variables and their interactions performed substantially better than the models relying exclusively on visual inputs. Moreover, our saliency map analysis showed that only around 25% of model neurons could be considered purely visual, with the majority of model neurons driven by multiple behavioral variables.

This widespread mixed selectivity is consistent with previous literature suggesting that V1 neurons may be modulated by a high-dimensional latent representation of several behavioral variables related to the animal's movement, recent experiences, and behavioral goals [20]. It is also consistent with the idea of a basis function representation [51, 52], which allows a population of neurons to conjunctively represent multiple behaviorally relevant variables. Such representations are often employed by higher-order visual areas in primate cortex to implement sensorimotor transformations [17, 51, 53]. It is intriguing to find computational evidence for such a representation as early as V1 in the mouse. Future computational studies should therefore aim to study the mechanisms by which V1 neurons might construct a nonlinear combination of behavioral signals.

Limitations and future work. While our study opens a new perspective on modeling neural activity during natural conditions, there are a few limitations that need to be acknowledged. First, our data was relatively limited (around 50 neurons per animal, for 3 animals). The development of a Sensorium-style standardized dataset [14] for freely-moving mice would significantly benefit future research in this area, enabling more robust comparisons between different modeling approaches. Second, it would be beneficial to integrate other modalities that are known to be encoded in mouse V1 into the model. One such example is reward signals [54], which could provide additional information about the animal's decision-making processes and motivations during exploration.

Acknowledgments

This work was supported by the National Institute of Neurological Disorders and Stroke of the National Institutes of Health under Award Number R01-NS121919. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

References

Daniel L. K. Yamins, Ha Hong, Charles F. Cadieu, Ethan A. Solomon, Darren Seibert, and James J. DiCarlo. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences , 111(23):8619-8624, June 2014. Publisher: Proceedings of the National Academy of Sciences.

Seyed-Mahdi Khaligh-Razavi and Nikolaus Kriegeskorte. Deep Supervised, but Not Unsupervised, Models May Explain IT Cortical Representation. PLOS Computational Biology , 10(11):e1003915, November 2014. Publisher: Public Library of Science.

Umut Güçlü and Marcel A. J. van Gerven. Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream. Journal of Neuroscience , 35(27):10005-10014, July 2015. Publisher: Society for Neuroscience Section: Articles.

Santiago A. Cadena, George H. Denfield, Edgar Y. Walker, Leon A. Gatys, Andreas S. Tolias, Matthias Bethge, and Alexander S. Ecker. Deep convolutional models improve predictions of macaque V1 responses to natural images. PLOS Computational Biology , 15(4):e1006897, April 2019. Publisher: Public Library of Science.

William F. Kindel, Elijah D. Christensen, and Joel Zylberberg. Using deep learning to probe the neural code for images in primary visual cortex. Journal of Vision , 19(4):29, April 2019.

Santiago A. Cadena, Fabian H. Sinz, Taliah Muhammad, Emmanouil Froudarakis, Erick Cobos, Edgar Y. Walker, Jake Reimer, Matthias Bethge, Andreas Tolias, and Alexander S. Ecker. How well do deep neural networks trained on object recognition characterize the mouse visual system? October 2019.

Glen T Prusky, Paul W. R West, and Robert M Douglas. Behavioral assessment of visual acuity in mice and rats. Vision Research , 40(16):2201-2209, July 2000.

Cristopher M. Niell and Michael P. Stryker. Highly Selective Receptive Fields in Mouse Visual Cortex. The Journal of Neuroscience , 28(30):7520-7536, July 2008.

David Klindt, Alexander S Ecker, Thomas Euler, and Matthias Bethge. Neural system identification for large populations separating ' what' and ' where'. In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Alexander S. Ecker, Fabian H. Sinz, Emmanouil Froudarakis, Paul G. Fahey, Santiago A. Cadena, Edgar Y. Walker, Erick Cobos, Jacob Reimer, Andreas S. Tolias, and Matthias Bethge. A rotation-equivariant convolutional neural network model of primary visual cortex. December 2018.

Fabian Sinz, Alexander S Ecker, Paul Fahey, Edgar Walker, Erick Cobos, Emmanouil Froudarakis, Dimitri Yatsenko, Zachary Pitkow, Jacob Reimer, and Andreas Tolias. Stimulus domain transfer in recurrent models for large scale cortical population prediction on video. In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Konstantin-Klemens Lurz, Mohammad Bashiri, Konstantin Willeke, Akshay Jagadish, Eric Wang, Edgar Y. Walker, Santiago A. Cadena, Taliah Muhammad, Erick Cobos, Andreas S. Tolias, Alexander S. Ecker, and Fabian H. Sinz. Generalization in data-driven models of primary visual cortex. January 2021.

Shahab Bakhtiari, Patrick Mineault, Timothy Lillicrap, Christopher Pack, and Blake Richards. The functional specialization of visual cortex emerges from training parallel pathways with self-supervised predictive learning. In Advances in Neural Information Processing Systems , volume 34, pages 2516425178. Curran Associates, Inc., 2021.

Konstantin F. Willeke, Paul G. Fahey, Mohammad Bashiri, Laura Pede, Max F. Burg, Christoph Blessing, Santiago A. Cadena, Zhiwei Ding, Konstantin-Klemens Lurz, Kayla Ponder, Taliah Muhammad, Saumil S. Patel, Alexander S. Ecker, Andreas S. Tolias, and Fabian H. Sinz. The Sensorium competition on predicting large-scale mouse primary visual cortex activity, June 2022. arXiv:2206.08666 [cs, q-bio].

Andrew D. Huberman and Cristopher M. Niell. What can mice tell us about how vision works? Trends in Neurosciences , 34(9):464-473, September 2011.

James J. Gibson. The Ecological Approach to Visual Perception: Classic Edition . Psychology Press, November 2014. Google-Books-ID: 8BSLBQAAQBAJ.

M. Beyeler, N. Dutt, and J. L. Krichmar. 3D Visual Response Properties of MSTd Emerge from an Efficient, Sparse Population Code. Journal of Neuroscience , 36(32):8399-415, August 2016.

Philip R. L. Parker, Morgan A. Brown, Matthew C. Smear, and Cristopher M. Niell. Movement-Related Signals in Sensory Areas: Roles in Natural Behavior. Trends in Neurosciences , 43(8):581-595, August 2020.

Laura Busse, Jessica A. Cardin, M. Eugenia Chiappe, Michael M. Halassa, Matthew J. McGinley, Takayuki Yamashita, and Aman B. Saleem. Sensation during Active Behaviors. Journal of Neuroscience , 37(45):10826-10834, November 2017. Publisher: Society for Neuroscience Section: Symposium and Mini-Symposium.

Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Charu Bai Reddy, Matteo Carandini, and Kenneth D. Harris. Spontaneous behaviors drive multidimensional, brainwide activity. Science , 364(6437):eaav7893, April 2019. Publisher: American Association for the Advancement of Science.

Cristopher M. Niell and Michael P. Stryker. Modulation of Visual Responses by Behavioral State in Mouse Visual Cortex. Neuron , 65(4):472-479, February 2010.

Emmanouil Froudarakis, Paul G. Fahey, Jacob Reimer, Stelios M. Smirnakis, Edward J. Tehovnik, and Andreas S. Tolias. The Visual Cortex in Context. Annual Review of Vision Science , 5(1):317-339, 2019. _eprint: https://doi.org/10.1146/annurev-vision-091517-034407.

Arne F. Meyer, Jasper Poort, John O'Keefe, Maneesh Sahani, and Jennifer F. Linden. A Head-Mounted Camera System Integrates Detailed Behavioral Monitoring with Multichannel Electrophysiology in Freely Moving Mice. Neuron , 100(1):46-60.e7, October 2018.

Grigori Guitchounts, Javier Masís, Steffen B. E. Wolff, and David Cox. Encoding of 3D Head Orienting Movements in the Primary Visual Cortex. Neuron , 108(3):512-525.e4, November 2020.

Philip R. L. Parker, Elliott T. T. Abe, Emmalyn S. P. Leonard, Dylan M. Martins, and Cristopher M. Niell. Joint coding of visual input and eye/head position in V1 of freely moving mice. Neuron , September 2022.

Laura Busse, Asli Ayaz, Neel T. Dhruv, Steffen Katzner, Aman B. Saleem, Marieke L. Schölvinck, Andrew D. Zaharia, and Matteo Carandini. The Detection of Visual Contrast in the Behaving Mouse. Journal of Neuroscience , 31(31):11351-11361, August 2011. Publisher: Society for Neuroscience Section: Articles.

Sylvia Schröder, Nicholas A. Steinmetz, Michael Krumin, Marius Pachitariu, Matteo Rizzi, Leon Lagnado, Kenneth D. Harris, and Matteo Carandini. Arousal Modulates Retinal Output. Neuron , 107(3):487-495.e9, August 2020.

Cristopher M. Niell and Massimo Scanziani. How Cortical Circuits Implement Cortical Computations: Mouse Visual Cortex as a Model. Annual Review of Neuroscience , 44(1):517-546, 2021. _eprint: https://doi.org/10.1146/annurev-neuro-102320-085825.

Pieter M. Goltstein, Sandra Reinert, Tobias Bonhoeffer, and Mark Hübener. Mouse visual cortex areas represent perceptual and semantic features of learned visual categories. Nature Neuroscience , 24(10):14411451, October 2021. Number: 10 Publisher: Nature Publishing Group.

Ján Antolík, Sonja B. Hofer, James A. Bednar, and Thomas D. Mrsic-Flogel. Model Constrained by Visual Hierarchy Improves Prediction of Neural Responses to Natural Scenes. PLOS Computational Biology , 12(6):e1004927, June 2016. Publisher: Public Library of Science.

Mohammad Bashiri, Edgar Walker, Konstantin-Klemens Lurz, Akshay Jagadish, Taliah Muhammad, Zhiwei Ding, Zhuokun Ding, Andreas Tolias, and Fabian Sinz. A flow-based latent state generative model of neural population responses to natural images. In Advances in Neural Information Processing Systems , volume 34, pages 15801-15815. Curran Associates, Inc., 2021.

Aran Nayebi, Nathan C. L. Kong, Chengxu Zhuang, Justin L. Gardner, Anthony M. Norcia, and Daniel L. K. Yamins. Mouse visual cortex as a limited resource system that self-learns an ecologically-general representation. PLOS Computational Biology , 19(10):e1011506, October 2023. Publisher: Public Library of Science.

Jianghong Shi, Eric Shea-Brown, and Michael Buice. Comparison Against Task Driven Artificial Neural Networks Reveals Functional Properties in Mouse Visual Cortex. In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Jianghong Shi, Bryan Tripp, Eric Shea-Brown, Stefan Mihalas, and Michael A. Buice. MouseNet: A biologically constrained convolutional neural network model for the mouse visual cortex. PLOS Computational Biology , 18(9):e1010427, September 2022. Publisher: Public Library of Science.

Galen Pogoncheff, Jacob Granley, and Michael Beyeler. Explaining V1 Properties with a Biologically Constrained Deep Learning Architecture. In Advances in Neural Information Processing Systems . Curran Associates, Inc., November 2023.

Julie A. Harris, Stefan Mihalas, Karla E. Hirokawa, Jennifer D. Whitesell, Hannah Choi, Amy Bernard, Phillip Bohn, Shiella Caldejon, Linzy Casal, Andrew Cho, Aaron Feiner, David Feng, Nathalie Gaudreault, Charles R. Gerfen, Nile Graddis, Peter A. Groblewski, Alex M. Henry, Anh Ho, Robert Howard, Joseph E. Knox, Leonard Kuan, Xiuli Kuang, Jerome Lecoq, Phil Lesnar, Yaoyao Li, Jennifer Luviano, Stephen McConoughey, Marty T. Mortrud, Maitham Naeemi, Lydia Ng, Seung Wook Oh, Benjamin Ouellette, Elise Shen, Staci A. Sorensen, Wayne Wakeman, Quanxin Wang, Yun Wang, Ali Williford, John W. Phillips, Allan R. Jones, Christof Koch, and Hongkui Zeng. Hierarchical organization of cortical and thalamic connectivity. Nature , 575(7781):195-202, November 2019. Number: 7781 Publisher: Nature Publishing Group.

Joshua H. Siegle, Xiaoxuan Jia, Séverine Durand, Sam Gale, Corbett Bennett, Nile Graddis, Greggory Heller, Tamina K. Ramirez, Hannah Choi, Jennifer A. Luviano, Peter A. Groblewski, Ruweida Ahmed, Anton Arkhipov, Amy Bernard, Yazan N. Billeh, Dillan Brown, Michael A. Buice, Nicolas Cain, Shiella Caldejon, Linzy Casal, Andrew Cho, Maggie Chvilicek, Timothy C. Cox, Kael Dai, Daniel J. Denman, Saskia E. J. de Vries, Roald Dietzman, Luke Esposito, Colin Farrell, David Feng, John Galbraith, Marina Garrett, Emily C. Gelfand, Nicole Hancock, Julie A. Harris, Robert Howard, Brian Hu, Ross Hytnen, Ramakrishnan Iyer, Erika Jessett, Katelyn Johnson, India Kato, Justin Kiggins, Sophie Lambert, Jerome Lecoq, Peter Ledochowitsch, Jung Hoon Lee, Arielle Leon, Yang Li, Elizabeth Liang, Fuhui Long, Kyla Mace, Jose Melchior, Daniel Millman, Tyler Mollenkopf, Chelsea Nayan, Lydia Ng, Kiet Ngo, Thuyahn Nguyen, Philip R. Nicovich, Kat North, Gabriel Koch Ocker, Doug Ollerenshaw, Michael Oliver, Marius Pachitariu, Jed Perkins, Melissa Reding, David Reid, Miranda Robertson, Kara Ronellenfitch, Sam Seid, Cliff Slaughterbeck, Michelle Stoecklin, David Sullivan, Ben Sutton, Jackie Swapp, Carol Thompson, Kristen Turner, Wayne Wakeman, Jennifer D. Whitesell, Derric Williams, Ali Williford, Rob Young, Hongkui Zeng, Sarah Naylor, John W. Phillips, R. Clay Reid, Stefan Mihalas, Shawn R. Olsen, and Christof Koch. Survey of spiking in the mouse visual system reveals functional hierarchy. Nature , 592(7852):86-92, April 2021. Number: 7852 Publisher: Nature Publishing Group.

Angie M Michaiel, Elliott TT Abe, and Cristopher M Niell. Dynamics of gaze control during prey capture in freely moving mice. eLife , 9:e57458, July 2020. Publisher: eLife Sciences Publications, Ltd.

Aman B Saleem. Two stream hypothesis of visual processing for navigation in mouse. Current Opinion in Neurobiology , 64:70-78, October 2020.

Cristopher M. Niell. Cell Types, Circuits, and Receptive Fields in the Mouse Visual Cortex. Annual Review of Neuroscience , 38(1):413-431, 2015. _eprint: https://doi.org/10.1146/annurev-neuro-071714-033807.

Katrin Franke, Konstantin F. Willeke, Kayla Ponder, Mario Galdamez, Na Zhou, Taliah Muhammad, Saumil Patel, Emmanouil Froudarakis, Jacob Reimer, Fabian H. Sinz, and Andreas S. Tolias. Statedependent pupil dilation rapidly shifts visual feature selectivity. Nature , 610(7930):128-134, October 2022. Number: 7930 Publisher: Nature Publishing Group.

Edgar Y. Walker, Fabian H. Sinz, Erick Cobos, Taliah Muhammad, Emmanouil Froudarakis, Paul G. Fahey, Alexander S. Ecker, Jacob Reimer, Xaq Pitkow, and Andreas S. Tolias. Inception loops discover what excites neurons most using deep predictive models. Nature Neuroscience , 22(12):2060-2065, December 2019. Number: 12 Publisher: Nature Publishing Group.

Alexander Mathis, Pranav Mamidanna, Kevin M. Cury, Taiga Abe, Venkatesh N. Murthy, Mackenzie Weygandt Mathis, and Matthias Bethge. DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nature Neuroscience , 21(9):1281-1289, September 2018. Number: 9 Publisher: Nature Publishing Group.

Nicholas A. Steinmetz, Cagatay Aydin, Anna Lebedeva, Michael Okun, Marius Pachitariu, Marius Bauza, Maxime Beau, Jai Bhagat, Claudia Böhm, Martijn Broux, Susu Chen, Jennifer Colonell, Richard J. Gardner, Bill Karsh, Fabian Kloosterman, Dimitar Kostadinov, Carolina Mora-Lopez, John O'Callaghan, Junchol Park, Jan Putzeys, Britton Sauerbrei, Rik J. J. van Daal, Abraham Z. Vollan, Shiwei Wang, Marleen Welkenhuysen, Zhiwen Ye, Joshua T. Dudman, Barundeb Dutta, Adam W. Hantman, Kenneth D. Harris, Albert K. Lee, Edvard I. Moser, John O'Keefe, Alfonso Renart, Karel Svoboda, Michael Häusser, Sebastian Haesler, Matteo Carandini, and Timothy D. Harris. Neuropixels 2.0: A miniaturized high-density probe for stable, long-term brain recordings. Science , 372(6539):eabf4588, April 2021. Publisher: American Association for the Advancement of Science.

Philip R. L. Parker, Dylan M. Martins, Emmalyn S. P. Leonard, Nathan M. Casey, Shelby L. Sharp, Elliott T. T. Abe, Matthew C. Smear, Jacob L. Yates, Jude F. Mitchell, and Cristopher M. Niell. A dynamic sequence of visual processing initiated by gaze shifts, August 2022. Pages: 2022.08.23.504847 Section: New Results.

Pouya Bashivan, Kohitij Kar, and James J. DiCarlo. Neural population control via deep image synthesis. Science , 364(6439):eaav9436, May 2019. Publisher: American Association for the Advancement of Science.

Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps, April 2014. arXiv:1312.6034 [cs].

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition, December 2015. arXiv:1512.03385 [cs].

Mingxing Tan and Quoc V. Le. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks, September 2020. arXiv:1905.11946 [cs, stat] version: 5.

Gert Van den Bergh, Bin Zhang, Lutgarde Arckens, and Yuzo M. Chino. Receptive-field properties of V1 and V2 neurons in mice and macaque monkeys. Journal of Comparative Neurology , 518(11):2051-2070, 2010. _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/cne.22321.

Michael Beyeler, Emily L. Rounds, Kristofor D. Carlson, Nikil Dutt, and Jeffrey L. Krichmar. Neural correlates of sparse coding and dimensionality reduction. PLOS Computational Biology , 15(6):e1006908, June 2019.

S. Fusi, E. K. Miller, and M. Rigotti. Why neurons mix: high dimensionality for higher cognition. Curr Opin Neurobiol , 37:66-74, April 2016.

A. Pouget and L. H. Snyder. Computational approaches to sensorimotor transformations. Nat Neurosci , 3 Suppl:1192-8, November 2000.

Marshall G. Shuler and Mark F. Bear. Reward Timing in the Primary Visual Cortex. Science , 311(5767):1606-1609, March 2006. Publisher: American Association for the Advancement of Science.

Appendix

A Data Acquisition

The full procedure of data collection and data preprocessing is described in detail in Ref. [25], but are briefly described below for the reader's convenience.

Visual scenes from the mouse's perspective, neural activity in V1, and various behavioral variables were simultaneously recorded from three adult mice who were freely exploring an arena with a state-of-the-art head-mounted recording system [25]. The recording system consisted of high-density silicon probes, two miniature cameras and an inertial measurement unit (IMU). Electrophysiology data was acquired at 30 kHz using a 11 µ m × 15 µ m multi-shank linear silicon probe (128 channels) implanted in the center of the left monocular V1. One wide-angled camera (around 120 ◦ ) was aimed outwards to capture the visual scene available to the right eye of the mouse at 16 ms per frame ('worldcam"). A second camera was aimed at the right eye (illuminated by an infrared-LED) to record a video feed of the right eye at 30 Hz . The IMU acquired three-axis gyroscope and accelerometer data at 30 kHz . In addition, a top-down camera recorded the mouse in the arena at 60 Hz .

During experiments, mice were placed in an arena where they could move around freely for about 1 hour. The arena was approximately 48 cm long × 37 cm wide × 30 cm high. The gray floor was covered with black-and-white Legos to provide visual contrast. One wall of the arena was a monitor displaying a moving black-and-white spots stimulus, and the other three walls were covered with wallpaper with static stimuli including white noise, black-and-white high-spatial-frequency gratings, and black-and-white low-spatial-frequency gratings. In order to encourage foraging behavior during the recording sessions, small fragments of tortilla chips were sparsely distributed across the arena.

The worldcam video was downsampled to 60 × 80 pixels. DeepLabCut [43] was used to extract extract eye position ( θ , ϕ ) and pupil radius ( σ ). Pitch ( ρ ) and roll ( ω ) of the mouse's head position were extracted from the IMU. Locomotion speed ( s ) was estimated from the top-down camera feed using DeepLabCut [43]. Electrophysiology data bandpass-filtered between 0 . 01 Hz and 7 . 5 kHz , and spike-sorted with Kilosort 2.5 [44]. Single units were selected using Phy2 (see [45]) and inactive units (mean firing rate < 3 Hz ) were removed. This yielded 68, 32, and 49 active units for Mouse 1-3, respectively. To prepare the data for machine learning, all data streams were deinterlaced and resampled at 20 . 83 Hz ( 48 ms per frame; Fig. 1C).

B Vision-Only Models

B.1 Hyperparameter Tuning

We performed a grid search to find the optimal CNN kernel size (3, 5, 7, 9), number of channels (32, 64, 128, 256, 512; in various combinations), and dropout rate (0, 0.25, 0.5). While other models often rely on kernel size 3 for their CNN, we found these small kernels to lead to worse performance, perhaps due to the mouse's low-resolution vision. Kernel size 7 performed best.

We repeated the grid search for CNNs with different number of convolutional layers. The resulting 3-layer CNN with 0.5 dropout rate outperformed many differently sized networks, such as a 1-layer CNNwith 1024 channels (i.e., a shallow but wide network), a 2-layer CNN, or a 4-layer CNN. Choice of learning rates and optimizers had no notable effect on the final performance of the networks.

B.2 Autoencoder

We initially hypothesized that an autoencoder could provide regularization benefits over a 'vanilla' CNN, because the reconstruction loss might encourage the model to learn visual features that are useful for decoding. We used an encoder ϕ to map the original frame F to a vector V in the latent space, which was present at the bottleneck, while the decoder ψ then mapped the vector V from the latent space to the output.

After hyperparameter search, we settled on size 256 for the latent space vector, and the weight of the reconstruction loss relative to the Poisson loss was fixed at 0.5. Both the encoder and the decoder were 3-layer CNNs, and their numbers of channels were symmetric. However, after testing a number of autoencoders with different configurations (Table 4), we found that a simple 3-layer CNN outperformed any and all of them.

Table 4: Performance of different autoencoders. The numbers of channels in the decoder were symmetric with those of the encoder. Best performing networks are indicated in bold. cc : crosscorrelation, mean ± standard deviation across neurons ( ↑ : the higher the better), MSE: mean-squared error ( ↓ : the lower the better).

C Visual Receptive Fields

To test whether the recovered visual receptive fields are sensitive to the choice of initial values for the behavioral variables, we repeated our experiments by initializing the behavioral variables with noise sampled from the uniform distribution [ -1 , 1) . The values remained the same throughout the process of gradient ascent.

We found that different values of behavioral variables resulted in similar visual receptive fields. A representative example is shown in Fig. C.1. Although some of the absolute values changed, receptive field structure stayed qualitatively the same, with identical excitatory and inhibitory subregions to the ones reported in Fig. 4.

Mouse1

Mouse2

Mouse3

cc=0.875

cc=0.855

CC=0.842

cC=0.834

CC=0.784

CC=0.781

cC=0.768

CC=0.765

CC=0.827

0.821

.771

cc=0.812

cC=0.806

cc=0.795

cC=0.781

cC=0.736

cC=0.700

cC=0.673

cC=0.600

cC=0.761

59

42

cc=0.780

cc=0.776

cc=0.772

cC=0.769

cC=0.597

cc=0.581

cc=0.735

cC=0.734

cc=0.733

cc=0.715

C=0.765

cC=0.758

ES

cC=0.748

cC=0.567

cC=0.519

cC=0.496

.714

711

0.695

1690m23

cC=0.744

CC=0.740

CC=0.740

0.737

CC=0.486

CC=0.481

cC=0.467

cC=0.437

cC=0.684

cC=0.660

cC=0.645

cC=0.637

cc=0.732

cc=0.715

cC=0.712

cC=0.423

cc=0.417

cC=0.406

cc=O.400

CC=0.637

CC=0.630

627

=0.626

cC=0.711

cC=0.701

cC=0.696

CC=0.695

cC=0.399

cC=0.396

CC=0.384

cC=O.370

620

cc=0.616

cCm0.608

8650=23

8890=2

cC=0.684

cC=0.682

cC=0.276

cc=0.255

CC=0.201

cC=0.162

597

cC=0.586

cc=0.582

cc=0.578

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a neural network for predicting activity of V1 neurons in freely moving mice. The data consists of ~hour long electrophysiological recordings of a neuronal population while the mouse is freely exploring the space. Simultaneously with the neural activity, the experimental setup allows the authors to record the visual scenes from the mouse perspective as well as a number of behavioral variables.\n\nThe network consists of two modules: one computes visual features, the other encodes behavioral variables. The outputs of both modules are concatenated and fed into the recurrent unit (GRU) accounting for the temporal dynamics, the output of which is decoded into the predicted neural activity of each neuron in the recorded population.\n\nThe models achieve state-of-the-art performance both among the visual models and among those including behavioral variables. The authors analyze the model by computing MEIs as well as performing saliency analysis to study the effect of individual behavioral variables.\n\nSTRENGTHS:\nI think it is a very good paper.\n\n+ It is clearly written\n+ It discusses an important problem (integration of behavioral variables in predicting mouse V1 activity)\n+ The proposed model achieves a noticeable improvement over the previous state-of-the-art\n\nWEAKNESSES:\nI don't see any significant methodological weaknesses.\n\nI think the main limitation is the small size of the dataset (a few dozen neurons per mouse). For example, the diversity of the MEIs in Fig. 4 both within and across the animals is very interesting, but a fairly small dataset size doesn't allow us to draw any quantitative conclusions about these MEIs beyond visual inspection. It would be really nice to have a calcium dataset of thousands of neurons in a freely moving mouse (similar to the Sensorium one) to address this issue, but it is of course not a critique of this paper.\n\nQUESTIONS:\n- I am surprised to see that your vision-only network quite significantly outperforms the baselines despite having a standard CNN-FC architecture. Why do you think it is the case? And specifically for mouse 2, why do you think your model is able to perform so much better than the baselines?\n\n- Do you think the training data could leak into the test set in the 70-30 splits of continuous segments as you described in the paper? I guess it is not unlikely that a freely moving mouse could be exploring the same part of the arena in some of the train and test segments?\n\n- How did you set the values of the behavioral variables for the visual MEIs computation? It would be very interesting to see how the MEIs change as a function of behavioral variables as the animal explores the space around it.\n\n- Some of the previous work (e.g. [11]) explicitly assumed the existence of localized receptive fields by factorising the readout into sparse spatial masks and a feature vector. But looking at the MEIs you generated, many neurons don't have receptive fields, so such an architecture doesn't seem reasonable. Do you have any ideas why the models with an explicit receptive field assumption perform well nevertheless?\n\nLIMITATIONS:\nThe limitations are adequately addressed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors use convolutional neural networks to fit data from neuronal recordings in primary visual cortex (V1) of the mouse while the animal is freely exploring the environment. The model incorporates visual signals but also other behavioral variables (“multimodal” aspect). The proposed model provides better fits to neuronal data than other benchmark models. The neuronal and behavioral data are particularly interesting and can provide interesting insights into neuronal computations when interpreted with mechanistic models.\n\nSTRENGTHS:\nThe ability to fit neural data in a dynamic fashion is interesting and distinguishes this work from previous work in the field.\n\nNumerically, the proposed model and incorporation of behavioral variables provide better fits to the data than the alternatives tested.\nThe differences across different neurons (section starting on line 212) is quite interesting, especially if it can be connected in the future to the ongoing efforts to characterize different cell types in mice.\n\nThe neuronal and behavioral data are particularly interesting and probably the highlight of the paper. These data can provide interesting insights into neuronal computations when interpreted with mechanistic models and would be very useful for future investigators that are interested in building theories and models of brain function.\n\nWEAKNESSES:\nOne could imagine incorporating a lot of different world variables to fit neuronal responses. Here the authors choose a very reasonable set of such variables, including pupil size, head direction, moving speed. What exactly this means in terms of the function of V1 is not clear.\n\nThis kind of neural data fitting has become extremely common in the field. However, it remains unclear what kind of conclusions one can draw from this type of data fitting. \nTake the first sentence of the results. Table 1 does show that the 3-layer CNN is slightly better than other models, as the subtitle indicates. Then what? Does this mean that mouse V1 functions like a 3-layer CNN? Clearly not. There are lots of problems with 3-layer CNNs, including adversarial attacks, lack of robustness to noise, etc. These issues are not tested here. Thus, it is difficult to state what we learn about V1 function from the fact that the 3-layer CNN is slightly better than, say, ResNet-18. It is certainly better in fitting data from this experiment, what this means in the big picture of V1 computations is unclear. \n\nThe next section indicates that behavioral variables improve the data fitting, which I agree with. However, it remains very unclear that V1 neurons would have inputs that directly reflect the behavioral variables incorporated into the model. Perhaps V1 neurons could get indirect inputs that relate to pupil size, head direction, running speed, etc. From a neuroscience perspective, the critical question is to understand the anatomical inputs, the mechanisms underlying these dependences. \n\nThe notion of mixed selectivity alluded to in the discussion has been extensively used in neuroscience and remains rather useless. Neurons show “mixed selectivity” only insofar as the investigators use variables that are not directly related to the neuronal responses. Without mechanistic models, there are lots of representations that might seem as “mixed selectivity”. As a trivial example, consider a neuron that responds to 45 degrees orientation preferentially. But the investigators try to fit the responses in terms of horizontal and vertical and then claim “mixed selectivity!”. There is obviously no such thing, the neuron represents 45 degrees, period . The claims in the paper follow the same logic.\n\nQUESTIONS:\nThe differences across different neurons (section starting on line 212) is quite interesting, especially if it can be connected in the future to the ongoing efforts to characterize different cell types in mice. Expanding on this section would be useful in terms of neuroscience. What is distinct about each type of neuron, how do their properties relate to their firing rates, variability, receptive field sizes and properties, etc. This is where the paper can go from data fitting into first steps into a mechanistic understanding of V1 properties.\n\nLIMITATIONS:\nThere is no discussion about limitations, of which there are plenty.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
The authors introduce a multimodal recurrent neural network to integrates information beyond vision - behavioral and temporal dynamics processed by a separate head to explain V1 activity in mice. This model (vision + behavioral and temporal dynamics) is the state-of-the-art in prediction of V1 activity during free exploration. Further, this model is analyzed using maximal-activating stimuli and saliency maps, to obtain insights into function of mice V1 area.

STRENGTHS:
* The paper is well written and the integration of information beyond vision (or any one modality) for neural predictivity is novel, as far as I know. 

* The results are strong and convincing with proper ablation studies

WEAKNESSES:
* Vision only model seems to have fewer parameters compared to the ones with GRU on top. This makes it hard to say if the improvements come from the extra parameters (unlikely).

QUESTIONS:
* I am surprised ResNet models do not do as well the smaller CNN. The skip connections in the resnet should have resulted in the model being at least as good as the shallower model. Have you looked into why?

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors in this work propose a novel multimodal approach to design encoder models of mouse visual processing. The authors identify the limitations of unnaturalistic (head-fixed) recording, limited behavioral inputs and lack of temporal dynamics in prior recordings and models of mouse visual system and address these with new data recorded from 3 freely-moving mice that they have access to which fixes the above-mentioned issues. The multimodal architecture, a CNN-GRU encoder simultaneously encodes time-locked visual input and elaborate mouse behavioral state (running speed, eye and head position, pupil size, the first derivatives and pairwise products of these quantities). A neural response prediction readout is attached to the output of the encoder. The authors show that a 3-layer CNN is the best predictor of mouse visual responses and that behavioral input consistently outperforms a purely vision-based encoder. Further gradient ascent visualization and saliency-based analyses add interpretability to the above encoder.\n\nSTRENGTHS:\n+ The proposed work is quite original and sound in the joint-computational modeling of mouse visual and behavioral responses in freely-moving mice. The encoder architecture proposed by the authors is quite interpretable and empirically performs well to predict the neural activity data. The originality in this work in my opinion is from the modeling that the authors propose. The dataset that has been used here seems to not be original to this work but prior work referenced by the authors in the Methods section.\n+ It is (although not surprising) clear from the experimental results that joint-encoding of vision and behavior outperforms predictions from a purely vision based model. The methods and results sections have been written quite well in adequate details on preprocessing techniques, training of the encoder and metrics used to evaluate the quality of visual encoding.\n+ Additional visualization and saliency based analysis performed by the authors adds interpretability to the proposed encoder model as it shows well-defined receptive fields (mostly those model neurons with high correlation to Mouse 1 neurons though) and selectivity to various behavioral attributes.\n\nWEAKNESSES:\n- Unfortunately it seems that the dataset has been published already and is accessed by this submission; the modeling efforts built on top seem like a relatively small contribution in my opinion. Especially since the kind of architecture designed and its hyperparameters aren't justified with a systematic search. Did the authors attempt alternative architectural choices, normalization techniques other than BatchNorm, different training approaches (unsupervised / self-supervised) or other recurrent units that are not included in this submission? Please provide the required justification for the presented architectural choices.\n- Related work section seems to be quite limited and I believe it could be enhanced to further include more information about other relevant computational models of mouse visual activity.\n- Other than Figure 1 and Figure 4, the other figures seem to have poor spatial resolution and need to be improved in quality for better readability.\n- Very little information is included about the neural activity dataset, I believe it would be good to add more details about the kind of visual stimuli used and about the recording setup in the Methods section or in the supplementary information.\n- I could not find mention of whether the authors are open to sharing their models and code, this would be appreciated if the authors want to maximize the reproducibility of their work.\n\nQUESTIONS:\nPlease find my suggestions in the weaknesses section of my review.\n\nLIMITATIONS:\nLimitations have been adequately addressed in this submission.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}