Self-Attention-Based Contextual Modulation Improves Neural System Identification
Authors: Isaac Lin
Tianye Wang
Shang Gao
Tang Shiming
Tai Sing Lee
Venue: ICLR 2025 Poster (2025)
Keywords: self-attention
incremental learning
neural prediction
contextual modulation
Primary area: applications to neuroscience & cognitive science
TL;DR: Self-Attention-Based Contextual Modulation Improves Neural System Identification

Abstract: Convolutional neural networks (CNNs) have been shown to be state-of-the-art models for visual cortical neurons. Cortical neurons in the primary visual cortex are sensitive to contextual information mediated by extensive horizontal and feedback connections. Standard CNNs integrate global contextual information to model contextual modulation via two mechanisms: successive convolutions and a fully connected readout layer. In this paper, we find that self-attention (SA), an implementation of non-local network mechanisms, can improve neural response predictions over parameter-matched CNNs in two key metrics: tuning curve correlation and peak tuning. We introduce peak tuning as a metric to evaluate a model's ability to capture a neuron's top feature preference. We factorize networks to assess each context mechanism, revealing that information in the local receptive field is most important for modeling overall tuning, but surround information is critically necessary for characterizing the tuning peak. We find that self-attention can replace posterior spatial-integration convolutions when learned incrementally, and is further enhanced in the presence of a fully connected readout layer, suggesting that the two context mechanisms are complementary. Finally, we find that decomposing receptive field learning and contextual modulation learning in an incremental manner may be an effective and robust mechanism for learning surround-center interactions.

Published as a conference paper at ICLR 2025
SELF-ATTENTION--ASS CONTEXTUAL MODULA- TION IMPROVES NEURAL SYSTEM IDENTIFICATION
Isaac Lin Tianye W Shang Gaol.3, Shiming Tang', Tai Sing Lee'.
Carnegie Mellon University, ?Peking University, *Massachusetts Institute of Technology
ABSTRACT
Convolutional neural networks (CNNs) have been shown to be state-of-the-art models for visual cortical neurons. Cortical neurons in the primary visual cor- tex are sensitive to contextual information mediated by extensive horizontal and feedback connections. Standard CNNs integrate global contextual information to model contextual modulation via two mechanisms: successive convolutions and a fully connected readout layer. In this paper, we find that self-attention (SA), an implementation of non-local network mechanisms, predictions over parameter-matee matched CNNs two key metrics: tuning curve corre- lation and peak tuning. We introduce peak tuning as metric evaluate model's s ability to capture a neuron's top feature preference. We factorize networks to as- sess each context mechanism, revealing that information in the local receptive field is most important for modeling overall tuning, but surround information is critically necessary for characterizing the tuning peak. We find that self-attention can replace posterior -natial-nteggaation convolutions when learned incrementally, and is further enhanced in the presence of fully connected readout layer, sug- gesting that the two context mechanisms are complementary. Finally, we find that decomposing receptive field learning and contextual modulation learning incremental manner may be effective and robust mechanism for learning surround-ceme center interactions.
improve neural response
INTRODUCTION
Feedforward CNN models have been shown in recent years to be effective approach for modeling and predicting visual cortical neurons' responses to arbitrary natural images (Kindel et al.] 2019
Klindt al.) 2018:) Yamins & DiCarlo 2016 Zhang al. 2018 Cadena et al. 2017: Kriegeskorte
2015). Neurons in the primate visual cortex are known to have extensive horizontal and feedback
recurrent connections for mediating contextual modulation (Felleman & van Essen 1991 Markov et al. 2014). Feedforward CNNs can model the influence of contextual surround on the responses of the neurons via two mechanisms: successive convolution layers and a fully connected layer. Both can make the neural model's s responses sensitive to the global image context, outside the traditional classical receptive fields of neurons. Interestingly, feedforward CNNs trained on image classification tasks have also been shown to exhibit contextual surround modulation similar to what has been observed in ncurophysiological experiments of the visual cortex (Pan et al. 2023).
In the context of neural prediction, it is found that including the inductive bias of horizontal recurrent connections can improve the model's 's predictive capabilities (Zhang et al., 2022). On the other hand, replacing feedforward layer with recurrent layer using Markovian local kernel consistently outperforms sarameter-matched feedforward CNNs in image classification tasks (Han et al., 2018 Nayebi al. 2018 Kubilius et al. 2019 Zhang al. 2022). However, contextual modulation in the visual cortex involves both the near surround and far surround, with the far surround being mediated by top-down feedback (Angelucei & Bressloff 2006 Sasaki et al.) 2013 Shushruth et al. (2013). In addition, there is evidence that contextual mod uoddularn ulation is dynamic and highly mnage-dependent sug- gesting a flexible gating mechanism (Coen- Cagli al. 2015). Such a flexible gating mechanism can be modeled by combination of Gaussian mixture models, implemented either by image-dependee normalization (Coen-Cagli et al., 2015) or by non-local networks and the self-attention mechanism
"Corresoonding authors: Isaac Lin (isaacl 8as cmu. -du.edu), , Tai Sing Lee (tai eeeecm.. edu).

---

Published as a conference paper at ICLR 2025
2.5
8.6
(a)
20418
25246
20422
leper
Pooling (b)
80008
22469
(c)
Figure I: Macaque neuronal response dataset. (a) shows a two-photon image with cells. (b) shows feedforward CNN used model neural response. (c) shows the response of stimuli and the top 20 images that induced the strongest responses. On average, less than 0.5% of the images induce responses greater than half peak height. Each site contains around 300 neurons.
50k
in deep learning (Fei & Pitkow 2022). Self-attention- based architectures such as vision transformers
have recently been shown be effective in modeling mouse VI neurons (Li et al. (023). However, these networks are complex, with large number of layers and multiple attention heads, making it difficult isolate the key factors contributing to their superior performance.
In this paper, we demonstrate that adding a simple self-attention layer to CNN can improve neural response prediction of macaque VI neurons in two performance metries: overall tuning correlation and prediction of the tuning peaks. To understand the mechanism driving improvement, we assessed the three contextual modulation mechanisms convolutions, self-attention, and fully connected readout layer. We found that while the three context mechanisms complement one another to pro- duce the best prediction performance when used conjunction, they have specific roles. First, the fully connected layer plays critical role in peak prediction, though self-attention can further enhance it. Second, self-ateention alone can improve tuning curve correlation but is insufficient for predicting the response peak top feature preference. Third, the performance of self-attention mod- els can be greatly enhanced when the feedforward receptive fields are learned first before learning the self-attention network, rather than learning everything from scratch simultaneously. The benefits of such incremental learning dvan de Ven et al. 2022 Cotton et al. 2020) in this context suggest that decoupling the learning of feedforward receptive fields and recurrent connections may allow the system to learn a richer representation of contextual modulation. These observations provide new insights into the underlying computational mechanisms of cortical development and organization.
RELATED WORKS
Modeling neural response prediction Feedforward deep neural networks have proven effective
in modeling and predicting neural responses in early visual brain areas (Kindel et al.) 2019 Klindi
al. 2018 Yamins & DiCarlo 2016 Zhang al. 2018 Cadena et al. 2017 Kriegeskorte 2015).
However, the brain's 's visual areas contain abundant recurrent connections that are essential for gen- erating neural responses (Felleman & van Essen 1991, Markov al.) 2014 Spoerer et al.) 2020).
Incorporating biologically-Inspiree simple recurrent circuits, in the form of a Markov network, into convolutional neural networks has been shown to enhance efficiency compared to purely feedfor- ward models, achieving similar performance in image classification and neural prediction tasks Zhang et al. 2022). In the context of neural prediction, the underlying assumption that the closer model replicate the neural computation mechanisms responsible for real neuron's 's response, the more accurate the model's predictive capabilities become (Pogoncheff et al.) [023] Willeke et al.
2023 [i al. 2019)
2

---

Published as a conference paper at ICLR 2025
(8)
(b)
(c)
(d)
Input
Conv
IE-CMI
EÉ
XI-CRN
F
Pool
Input
Input
Input
Input
Conv
K Conv
V Conv
Kome 2
BN
cCPB
aCPB
aCPB
dCPB
Act
Softmax
cCPB
dCPB
@CPB
Drop
Outpul
|CPB k.3
SA
+
8CP8
6.1
SA
Sene
Conv
Drop
8CPB
Drop
8CPB
Insut
8CPB
8CPB
Drop
Drop
BN
Conv ste
FCL
8CPВ ...
8CPB
Act
Drop
Output
FCL
Output
CTL
Oulpul
Output
Output
Output
SA
$CPB vewe
FCL
CTL
Figure 2: Models explored in this study. Models are constructed from two types convolutional processing blocks (CPB): aCPB and JCPB. aCPB has fixed convolution kernel size = 5 and max pooling kernel size = 2. 3CPB takes an input convolution kernel size of k, and has no pooling layers. The two final layer readout modes are fully connected (FCL) and center hypercolumn only (CTL). Self-attention (SA) takes as input a boolean y that determines whether the value (V) vector transformed; if =Truee then V is mapped, otherwise V is equal to the input. All models with SA utilize single-headed attention. (a) shows the feedforward CNN. (b) shows the feedforward CNN augmented with self-attention. (c) shows the receptive field CNN. (d) shows the receptive field CNN augmented with self-attention.
Self-attention for global dependencies Self-attention mechanisms have recently become piv- otal component deep learning models, especially natural language processing and increasingly in computer vision tasks /Vaswani et al.) 2023 Zhao et al.) 2020 Kim et al. 2021). In computer vision, self-attention performs a weighted average operation based on the context of input features, computing attention weights dynamically through similarity function between pixel pairs (Vaswani let al. 2023 Pan et al.] 2022). This flexibility allows the attention module to adaptively focus on different regions and capture informative features (Ramachandran al. 2019). Self-attention has also been integrated with CNNs to enhance their representational power (Pan et al. 2022 Yang al. 2019 Bello et al. 2020). By enabling CNNs to consider distant spatial relationships within an image, self-attention improves the network's s ability to capture global context. This mechanism overcomes the limitations of traditional CNNs, which primarily concentrate on local features be- cause of their convolutional structure. Taking the complementary properties of convolution and self-attention, the benefits of each paradigm can be extracted by integrating the two and using self- attention to augment convolution modules (Dai et al. 2021
Yang et al.) 2019 Pan al. 2022
Cordonnier et al. 2020).
APPROACH
In this study, we developed a set of deep learning models to model V1 neural response to natural images, with the goal of evaluating the potential roles of the self-attention mechanism in neural computation within the visual cortex. We obtained dataset of neuronal responses measured us- ing two-photon imaging with GCaMP5 from two awake behaving macaque monkeys performing
3

---

Published as a conference paper at ICLR 2025
fixation task, consisting of 302 neurons from monkey 1 (MIS1) and 299 neurons from monkey 2 (M2S1), in response to 34k and 49k natural images extracted from the ImageNet dataset. The neurons were recorded across five days and tracked anatomically based on landmarks well as based on their responses to 200 fingerprint images tested every day. The images were presented in sequence with 500 ms per image preceded by 1500 second of blank screen. 'he 30k-50k images in the training set were presented once, and the 1000 images in the validation set were tested once with 10 repeats. Images were 100 100 pixels, with 30 x 30 pixels for 1 degree visual angle. The eccentricity of the recording sites were 3 degrees and 1.79 degrees, with average receptive field sizes (diameters at half-height) of about 0.75 and 0.5 degrees, respectively. We preprocessed the dataset before modeling, and notably downsampled input images to 50 x 50 pixels, yielding 15 x 15 pixels per degree visual angle. The receptive fields of neurons from each mm x mm site (the scale/size of a hypercolumn) in macaque monkeys exhibited significant overlap. These fields were mapped using oriented bars or SmoothGrad feature attribution on our deep learning model. The standard deviation of the receptive field centers, all of which are contained within the center hypercolumn of our CNN models, was less than pixel (1/15 degree visual angle). See Appendix A.1 for more details of the macaque experimental setup.
3.1 AUGMENTING FERDFORWARD CNNS WITH SELF-ATTRNTIO
First, we investigate whether incorporating self-attention into the baseline feedforward CNN model enhances neural response prediction performance. See Appendix A.3 for comparisons to other es- tablished models.
Baseline feedforward model (ff-CNN) See Figure a for the architecture. The baseline feedforward model is comprised of two a-convolutional processing blocks (uCPB) and two B- convolutional processing blocks (BCPB), followed by a fully connected readout layer (FCI.) A single Ff-CNN model is fitted to each neuron. All models described below are derived from this baseline model. Given grayscale input image with dimensions 50 x 50 pixels, the two aCPB layers with 5 x 5 kernel encode the input of size (1 50 x 50) into (c x 9) where E N is the number of channels (c E (30,32) in this study). The center hypercolumn of the post-aCPB encoding has centered effective receptive field size of 13 x 13 pixels. In other words, the center hypercolumn of the latent representation after the aCPB layers will have a 13 x 13 (or 0.8 x 0.8 degree visual angle) feedforward receptive field at the center of input 50 x 50 image. Note that the real neurons' receptive fields are contained inside the receptive field of the center hypercolumn. In the baseline model. the two aCPB layers are followed by two 9CPB layers with 3 x kernels further expand the effective receptive field of the center-hyperee hyprroouupe Finally, fE-CNN has access to the entirety of the input image in the final layer as the readout has full access to all the hypercolumns. Thus, the baseline CNN CNN has two modalities of contextual modulation convolutions and fully connected layer.
Feedforward with self-attention model (EÉ+sa-CNN) See Figure b) for the architecture. We augment ff-CNN with self-attention layer immediately after the last aCPB and before the first @CPB. This placement enables SA to act on an adequately convolved feature representation, but also be further modulated by convolutions before feeding into the final layer. We compare the per- formance of fftsa-CMN CNN against that of FE-CNN, controlling the parameter counts to be roughly equal by decreasing the number of channels, which is maintained throughout entire model, from e = 32 in the baseline CNNs to c = 30 in the self-attention models to account for the addition of the SA layer. In the context of contextual modulation, ff+ intermixes spatial interactions and inter-channel mixing across SA, the posterior 3CPBs, and the FCL.
3.2 FACTORIZING THE CONTEXTUAL MODULATION MECHANISMS
There are three mechanisms in ff ff-sa-Oe CNN mediating contextual interactions. We proceed to fac- torize ff+sa-CNN by removing the contextual modulation contributed by the 3CPBs and the FCL to assess the standalone capability of SA in incorporating surrounding context. This is accomplished by constructing a baseline receptive field model and a model where only SA is mediating horizontal connections.
4

---

Published as a conference paper at ICLR 2025
Baseline receptive field model (If-CNN) See Figure 2[c) for the architecture. We first construct the rf-CNN model, which is devoid of contextual modulation, by subtracting from ff-CNN: the kernel size in the 3CPBs are changed from 3 3 to 1 x 1 and the fully connected layer is changed to look only at the center hypercolumn (CTL.) The 1 x convolutions perform no spatial expansion before feeding into the CTL. Thus, rf-CNN is making predictions solely based the center hyper- column receptive field produced by the aCPBs, which covers the center 13 x 13 pixels of the input image.
Receptive field with self-attention model (IÉ+sa-CNN) See Figure d) for the architecture. We add self-attention to rf-CNN prior to the 3CPBs to construct rf sa CNN. Self-attention is the only mechanism for incorporating surround context in this model. The parameter counts are again controlled by reducing the number of channels from = 32 (in r Cf- to c 30 (see Appendix A. for more details). We compare the performance of the two receptive field models, alongside the feedforward models. Note that in rf+sa-CNN, y is False in the SA layer, meaning SA operates exclusively on the horizontal spatial interactions between hypercolumns without any inter-channel mixing. In contrast, ? True the SA layer of ff+sa- CNN, which allows channel mixing in SA. Channel mixing potentially provides sell-attention greater flexibility (see Appendix A.9 and A.
3.3
INCREMENTAL LEARNING: FACTORIZING THE LEARNING PROCESS
Discussed in Section comparing the four models above reveals that contextual modula- tion introduced in rf+sa-CNN via SA did not produce better performance relative to rf+CNN, despite Ff+ having clear performance improvements over ff-CNN. This is not due to the difference in channel mixing (the > parameter in SA). We hypothe- size that nottle-necking all the gradient signals solely through the center hypercolumn during backpropagation makes it difficult for the net- work to properly learn the aCPB layers and the SA layer simultaneously. Thus, we inves- tigate an incremental learning paradigm where we allow the receptive fields the oCPBs be learned before incorporating any context mech- anisms. We then incrementally add and learn self-attention layer followed by fully con- nected readout layer.
(b)
(c)
(d)
rftsa-Cor (Incr.)
= (Iner. FC2)
eÉ-CIN
[Incr. FC2)
Input
Input
Input
Input
#CPB
cCPB
cCPB
#CPB
8CPB
SA
SA
SA
Drop
8CP8
8CPB
[CPB
tlock
#CP8
Drop
Drop
CTL
8CPa
#CPB
PCPB
Output
CTL
FCL
FCL
Output
Qutput
Output
The following progression of models, rf- rfffsa-aee -a-Ns""... and ff+sa-CNN* (as shown in Figure 3) incrementally expands the capac- ity of contextual modulation, An important dis- tinction between incremental models and mod- els shown in Figure 2, marked by ", is a 1 x kernel in the 3CPB, which maintains channel mixing but removes further spatial integration through convolution. rf-CNN (shown Fig- ure aaa or Figure ((c) has information only from the center receptive field. rf+sa-CNN' (shown in Figure ((b) uses only the self- attention mechanism for contextual modula- tion. fétsa CNN* (shown in Figure c (d)) has the same surround-cente modulation rf+sa-CNN from self-attention, but al- lows spatial integration of the global context by changing the CTL to FCL the end. As hori-
Figure 3: Incremental learning models. (a) shows the baseline receptive field CNN, equivalent to Figure Zic). (b) shows (a) augmented with SA and learned incrementally; the aCPBs are taken from (a) and the remaining layers are learned. The denotes slight modification from r rf+sa CNN, Figure dd) namely y changed True. (c), (d) show the result of replacing the CTL in (b) with FCL, and learned incrementally: (c) freezes only the center hypercolumn in the FCL (FC1) whereas (d) allows the FCL. to learn freely (FCg). (c) and (d) have all other layers taken from (b). The de- notes slight modification from fftsa-CMi Fig- ure 2bb),, namely & in SCPB is changed to k 1. (simul. models are equivalent in architecture, except all blocks are learned.
5

---

Published as a conference paper at ICLR 2025
ff-CNN
ff+sa-CMM CNN
FN-CN
Real Prediction
Resporne (d6
Prediction
Prediction
Resporrse (dF/F)
2.0
2.0
2.0
CORR. 0.629
0.639
1.5
Respore
1.0
1.0
1.0
0.0
0.0
0.0
1000
1000
200
-00
1000
Image
Image
Image
Figure 4: Neuronal tuning curves ff-CNN, fftsa cnn, and rf CNN. Pearson correlation does not reflect peak tuning. Despite rf CNN having the better correlation, it is clear that DE+sa-CMi able capture the peak significantly better, at the cost of noisier overall tuning. Example shown is MISI neuron 238. See Appendix A.8 for population averages.
zontal connections in the visual cortex known to mature after the development of the receptive fields, we designed an incremental learning setup where rf-CNN first learns the receptive fields, then Of+sa-CNNY (Iner.) learns self-attention layer only after rf-CNN has already learned the aCPB receptive fields. Finally, fftsa =f+sa-CNN" (Tncr. FC1) and ff+sa-CNN" (Tncr. FC2) inherit the receptive fields and self-attention structures of (e+sa-CCNNY but differ in the change to a FCL readout. Models labelled ( (Incr.) ) are learned incrementally as such, and models labelled (S (Simul.) are traditionally trained simultaneously.
3.4 HYPERPARAMETER SELECTION AND MODEL TRAINING
Rather than splitting the evaluation set for hyperparameter selection, we partitioned our population of neurons to select hyperparameters (training and architectural). We fine-tuned, by experimenting with batch size, learning rate, epochs, number of layers, number of channels per layer, etc., model hyperparameters on a subset of 50 neurons using a relatively coarse grid search. We list key training hyperparameters here: (1) batch size 50, (2) learning rate 0.001, (3) optimizer Adam, (4) loss MSE, (5) epochs = 50. Training and computations were performed on an in-house computing cluster with GPU (NVIDIA V100 similar) nodes. Training hyperparameters were held constant across all models. Architectural hyperparameters were held constant across layers shared between models. We do not optimize hyperparameters for models other than the baseline ff-CNN.
The primary objective of this project is to demonstrate that self-attention can enhance neural re- sponse prediction relative the baseline feedforward CNN, despite hyperparameterss being opti- mized only for the baseline model. Since we show that DE+sa-CNI improves upon FE-CNN in both evaluation metrics (see Section 4.I). further hyperparameter optimization unnecessary for our objective. Instead, we are interested in understanding the reason behind this improvement. The other models tested in this study are architectural subsets of the Ef+sa-CNN, designed to dis- sect their contributions to its success. We do not anticipate any derivative models outperforming justifying holding hyperparameters constant across models for fair comparison.
3.5 MODEL EVALUATION METRICS
To quantify performance, models were evaluated two criteria, Pearson correlation and peak tuning index. Pearson correlation represents the overall tuning similarity between a model's predicted responses and the real neuron's recorded responses. The peak tuning index is used to quantify how well a model can predict and match in magnitude the strongest responses recorded by the real neuron. This lets us evaluate how well model can discriminate between, as well as model the response magnitude of, images that are strongly excitatory and images that incite a weak response.
Pearson correlation: The Pearson correlation (CORR.) is taken between neuron responses and model-predicted responses. Pearson correlation is a standard measure for evaluating neural response prediction. Other established measures, including FEV, y and CCa WWlllkee al. 2022 Cadena et al., 2017] Pospisil & Bair [002]] Zhang et al. 2022), were used as well and yielded similar results Pearson correlation (see Appendix A.3 and A.4

---

Published as a conference paper at ICLR 2025
In this paper, explored set of measures to assess the peak of neuronal tuning curve. In our macaque V1 dataset, we found that neurons exhibit sharp stimulus selectivity, consistent with findings from Tang et al. (2018a b). reinforcing the diversity and complexity of VI neurons (see Appendix A.2). We found that Pearson correlation and other standard metrics (see Appendix (A.4) successful in measuring a model's fit the overall tuning curve, but often fail represent the peak tuning preference of neurons, of which is key aspect of a neuronal function. For example, Figure demonsttaass that while CNN achieves higher correlation approximating the overall tuning curve compared to f ff+sa-CNN, the latter outperforms in modeling the neuron's 's tuning curve peak. To address this issue, we developed two new metrics better assess model's ability to capture the peak tuning of neurons.
The peak tuning index (PT) is a membership metric of the strongest predic- tions above a threshold determined by the top 1% of real responses. PT can be roughly interpreted as the percentage of the peak that model captures, under magnitude prior. The index is calculated as:
Peak tuning index:
# of top predictions 2 min (top 1 real responses) 100%
PT
of responses in the top 18
PT is divided into PT, and PTs. based on how # of top 18 predictions is defined. PT, is when predictions are jointly rank ordered with respect to the real responses. PT's is when predictions are separately rank ordered independently of the real responses. PT, is a stricter measure. Note that because we train with MSE loss, models are incentivized to minimize the absolute difference between predictions and real responses, rather than match the curvature of the tuning curve. This minimizes the risk of PT being misrepresentative due to lateral shifts in the tuning curve.
4 RESULTS
4.1 SELF-ATTENTION IMPROVES NEURAL RESPONSE PREDICTION
Table 1: Average Pearson correlation and peak tuning metrics for models trained on MISI and M2S1. Correlation SEM = 0.009 was consistent across models and monkeys. Despite =f-CNN unexpectedly outperforming r rf+sa-CNNN the difference is recovered when =f+sa-CNN is trained
incrementally (see Section 4.2
MISI
M251
Model
CORR.
#5-CNN
PTs
PTs
CORR.
FÉ-CNN
PTs
EE-CNN 0.393
0.0%
0.5
5.6 0.9
0.477
0.0%
8.6 0.9
16.2 + 1.6
#É+S2-CNN
0.418
+6.6%
5.6
10.5 1: 1.1
0.491
+3.3%
11.5 1 0.9
23.5 1: 1.8
=E-CNN 0.420
+8.6%
1.3
1.8 0.5
0.496
+4.3%
4.4
6.6 1.0
xÉ+s2-CNN
0.414
+7.2%
0.2
1.0 1: 0.3
0.486
+2.4%
0.5
5.1 1 0.8
We compared the performance of the EE+sa-CNN model to the marameter-matched baseline ff-CNN model and found that incorporating self-attention significantly improved correlation and both peak tuning metrics (see first two rows of Table D]. This indicates that self-attention enhances modeling of both the overall tuning and peak tuning aspects of the neurons, with consistent results across both monkeys.
important to note that the Ef-CNN center hypercolumn at the readout layer has a receptive field much larger than the real neuron's 's receptive field, due to successive convolutions in the aCPBs and JCPBs. Additionally, the fully connected readout layer also incorporates long-range spatial dependencies. Thus, the self-attention layer in ff+sa-CNN acts modeling horizontal connections, and provides additional performance benefits.
additional mechanism for
To better understand the role of self-attention in contextual modulation, we constructed a baseline receptive field model, ffCNNN, that is devoid of all contextual modulation mechanisms. rf-CNN's CTL readout only uses the center-hypercolume of the convolved feature space to make predictions. Note that the feedforward receptive field of the center hypercolumn after the two aCPBs is a centered 13 13 pixel portion of the input image. Moreover, between the oCPB layers and the CTL. layer,

---

Published as a conference paper at ICLR 2025
the size of the center hypercolumn receptive field does not expand due the the x 1 kernel in the @CPBs. This means that predictions are being made solely based on the center 13 x 13 pixels, which corresponds roughly to the real neuron's receptive field.
Surprisingly, we found that Cf-CNN achieved the highest correlation, indicating that models focus- ing primarily the classical receptive field offer the best fit the overall tuning curve (see Table [ However, correlation fails to reflect the model's 's shortcomings in fitting the peak of the tuning curve. Table also reveals that while CfNNNN has the highest Pearson correlation, it performs worse at capturing the tuning peak compared to ff f and ff EN-NN. Thus, we conclude that contex- tual modulation plays a crucial role in peak tuning, and that the three mechanisms in ff+sa-CNN for integrating surround information are complementary.
4.2 DISSECTING CONTEXTUAL MODULATION MECHANISMS VIA INCREMENTAL LEARNING
We explore the relative contributions of the different contextual modulation mechanisms. Specif- ically, we first ask: Is self-attention alone sufficient to model contextual modulation? To answer this question, we added SA to rfCNNN to produce rf+sa-CNN. We found that without posterior 3CPBs and FCI., the self-attention in rf+sa- is not useful, In fact, the performance is worse than =f-NNN in overall correlation and both peak tuning metrics (see Table This result is some- what unexpected, as the addition of self-attention in SS SN-CNN does improve upon ff-CNN. is possible that the spatial integration mechanisms in the 3CPBs, along with the FCL, are necessary to provide sufficient pathways for backpropagating the gradients during learning, so that both the aCPB receptive fields and self-attention kernels can be properly learned.
To test this hypothesis, we explored whether an incremental learning approach, where differ- network components are learned sequentially, could yield model that performs par with fC+ We show that learning the feedforward kernels in the oCPBs first, followed by learn- ing the self-attention layer, and finally the fully connected layer, can nearly match the performance of fftsa- CNN. This indicates that although the spatial integration by 3CPBs contributes to perfor- mance, the self-attention layer plays a more critical role in capturing horizontal interactions crucial for modeling peak tuning.
Table 2: Average Pearson correlation for models incrementally and simultaneously trained on MISI and M2S1. Correlation SEM = 0.009 was consistent across models and monkeys.
MISI
M251
Model (Tmining Method)
CORR.
A YF-ONM
CORR.
Yf-ONN
FN (Simul.) .)
0.420
0.0%
0.496
=E+ (Simul.)
0.409
-2.6%
0.480
-3.2%
=É+sa-Usa" (Iner.)
0.421
+0.6%
0.493
-0.3%
fE+ (Simul.) .)
0.416
-0.8%
U.490
-0.7%
fÉ+sa-UMm' (iner.ech
0.430
+3.0%
0.494
-0.1%
ff+5a-ONN" (Tnor.FCa)
0.414
-1.3%
0.488
-1.1%
Incremental learning offers a valuable approach for accurately assessing the potential of each spatial integration mechanism in modeling contextual modulation and evaluating their interdependence in generating an effective model. Table 2 highlights several incremental models we tested and their relative improvements. Learning the receptive field first, followed by learning self-attention in rE+SA-CNN" (Incr.) outperforms learning both simultaneously in x rf+SA-CNN* (Simul.) (see second and third rows of Table [ and middle pairs in Figure [5). This supports our hypothesis that jointly learning the cCPB receptive fields and self-attention may overwhelm the system when gradients are constrained to propagate through the center hypercolumn alone. Incremental learning can improve rf +f+sa- CNN* to match the peak prediction performance of rf- but not beyond. This suggests that, when used with a CTL readout, self-attention alone is insufficient to fully capture peak tuning.
Table 2 and Figure S demonstrate that removing the CTL restriction-i.e., allowing the read- out to access information from the hypercolumns in the final convolutional layer via FCL, as in ff+sa-CNN' -nables the network to nearly match the performance of the ff+sa-CNN. +f+sa-CNNY named such because it closely resembles ff with the only differ-
8

---

Published as a conference paper at ICLR 2025
Average PT, Index for M151
Average PTş Index for M151
Incr.FG
Simuaaneuslly Traned
Incr.FC;
Incrementily Talhed
10-
(Average)
PTs (Average)
Sinul.
Sinul.
Incr.
PCC
Sinul. Incr.
Sinul Incr.
Simul.
Simul.
rf+sa CNN* Model
Mf+sa-CNN*
CF-
If rftsa CNN
ff+sa-CNN CNN*
Model
Average PT, Index for M251
Average PTs Index for M2S1
Sialttnenslly hained
Incr. FC
25
Smmisnnessty
Incr,FC2
Indromentaly Trained
Indemmentalyy
PTy (Average)
PTs (Average)
Simal,
15
Simul.
Simul Incr.
Iner.
CC.
10
4
Simul
Simul ~ Incr.
Simul.
Cf-CNN
rf+sa- CNN
ff+sa-CHN"
rf-CNN
rf+sa- CNN*
ff+sa-CNNC
Model
Model
Figure 5: Average peak tuning indices for incrementally and simultaneously trained models. Top row: bar charts for MISI. Bot row: bar charts for M2S1. Left col: average PT, values. Right col: average PT, values. Error bars are SEM.
ence being the use a x 1 kernel instead of 3 x kernel in the 3CPBs. These findings suggest that posterior convolution is not required for spatial integration after the self-attention layer when the model is trained incrementally.
A reoccurring observation is that focusing receptive field information tends improve over- all correlation, while emphasizing contextual information enables the network to better model peak tuning. This pattern evident when comparing r FN-CNN with ff+sa+CNN in Table A similar contrast exists between fÉ+sa-CNN" [Incr.CC, and ff+sa-CNN" (Iner. FC2) In (E+sa-CNNY (Incr. FC1). ) the model inherits the center hypercolumn weights and then learns the surrounding hypercolumn contributions through the FCL. In contrast, ff+sa CNN* (Incr FC2) learns the weights of all hypercolumns in the FCL simultaneously. While the former excels in cor- relation, the latter performs better in peak tuning. This suggests that the receptive field is most important towards overall tuning, whereas surround-center interactions are key to capturing peak tuning.
The fÉ+sa-CNN" models saw an improvement in PT, and PT's over the rf+sa-CNN" models. This suggests that either the fully connected layer (FCL) plays critical role in predicting peak re- sponses, or that constraining the readout to the center hypercolumn (CTL) in rftsa CNN* limits error propagation to the self-attention block during training. As result, self-attention may be in- adequately learned in these cases, impairing the model's 's ability to effectively implement contextual modulation.
4.3 INCREMENTAL LEARNING EMPHASIZES THE CONTRIBUTION OF THE CLASSICAL RECEPTIVE FIELD
well-established neurophysiological principle is that stimuli within the classical receptive field of VI neurons are the primary driver of neural responses, while the contextual surround modulates them. We found that when the entire network trained simultaneously. as in ff+sa-CNN" (Simul.) ), performance is weaker and the network does not follow this principle. However, with incremental learning, the center hypercolumn develops into the dominant contribu- tor, as observed in 1F+sa-CNNY (Incr.FC) and ff+sa-CNN* (Iner, FC2). Figure 6 illus-

---

Published as a conference paper at ICLR 2025
FCL Decompesitin SiN
Decomposttire PCC
fi-se--5MMijerr.e
Resl
Feal
Predittion
Predition
Frecittor
Center Contibutton fresitien
130
Centor Owny bud er hed
Caner Cartrautonn freection
Suroond Cortitation
Surrannd Conr Beder to (5um)
Suround Contriuttio fredition (3am
45 15 1050
20
130
120
Figure 6: Average FCL dccomposition of Ef+sa-CNN" when trained differently. The center contri- bution (green) and the total surround contribution (pink) sum to the prediction tuning curve (orange). Plots are rank ordered with respect to predicted responses. Averages are calculated by plotting rank ordered decomposed tuning curve for each neuron, then averaging across each image. Individual contributions from hypercolumns can be found in Appendix A.5
trates the sum of connection weights from the readout, showing that in ff+sa-CNN' (Simul.) .), weights evenly distributed, whereas in the incrementally trained models, the center hypercol- umn, corresponding to the classical receptive field, is emphasized. Incremental learning fosters this center-surround division and modulation, yielding interpretable performance benefits.
DISCUSSION
CNNs are widely used and effective models for visual cortical neurons, and they inherently include two mechanisms for contextual modulation: successive convolutions and fully connected layer, which allow the global image context to be accessible to the readout. In this paper, we demon- strated that augmenting CNN models of cortical neurons with self-ateention enhances predictions of both the overall tuning curve and the tuning peak. Self-attention, resembling three-way interac- tions in probabilistic graphical models, facilitates flexible surten- interaction via contextual variables (Coen-Cagll et al. 2015 Fei & Pitkow 2022). This provides additional flexibility and complementary benefits the CNN's inherent context mechanisms. While large-scale transformer models with multiple attention heads have achieved state-of of the art performance in modeling mouse V1 neurons by capturing oangrange dependencies (Li al.) 2023), our work isolates the contribu- tions of self-attention in CNNs for modeling horizontal circuits, highlighting the dependencies and complementarity among three different mechanisms of contextual modulation.
Several key findings emerged from this work that advance our understanding of cortical computation and neural codes. First, we found that focusing on receptive field information, as r f-CNN, yields the highest Pearson correlation, alongisde other standard measures (see Appendix A.4), for overall neuronal tuning curves (see Table Tp. This suggests that the classical receptive field is the primary driver behind neuron's overall response. Our incremental learning experiments further reinforce the importance focusing first on the bottom-up information within the classical receptive field before learning the horizontal connections (see Figure 6). Second, demonstrated that contex- tual modulation is important for predicting the peak response, with self-attention playing a pivotal role. A trade-off, however, exists between the receptive field and surround modulation: RF-centric models fit overall tuning curves more accurately, while increased contextual surround modulation enhances peak tuning, though often the expense of overall tuning correlation. Incremental learn- ing, which fosters strong receptive field bias, may help even out this trade-off. This is consistent with neurophysiological evidence supporting a dominant classical receptive field and weaker sur- round modulation, with recurrent connections being fine-tuned after receptive field development.
recent CNN-based model of mouse V1 neurons revealed that the most excitable images often involve stimulus features outside the receptive fields that are consistent with the concept of pattern completion (Fu et al., 2023). Similarly, we found that models capable capturing peak tuning dis- play interpretable contextual modulation, such as association fields and pattern completion, within the self-attention module (see Appendix A.6). Additionally, incorporating a self-attention layer improved models' data efficiency (see Appendix A.7p. Further theoretical and experimental investi- gations are needed to characterize and evaluate the interactions facilitated by self-attention, so as to uncover how these mechanisms may be implemented by biological circuits.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
All procedures involving animals for generating the data of this paper were in accordance with the Guide for the Institutional Animal Care and Use Committee (IACUC) of Peking University.
ACKNOWLEDGMENTS
This work supported by NSF CISE RI 1816568 and NIH RO1 EY030226-01AI awarded to Tai Sing Lee. Isaac Lin is supported by the NSF REU supplement of RI 18 1816568. Imaging data was produced by Tianye Wang and Shiming Tang with the support of STI2030-Major Projects 20222D0204600, National Natural Science Foundation of China U1909205, and funds from the Peking-Tsinghua Center for Life Sciences ST.

---

Published as a conference paper at ICLR 2025
REFERENCES
Alessandra Angelucci and Paul C. Bressloff. Contribution of feedforward, lateral and feedback connections to the classical receptive field center and extra-classical receptive field surround of primate vI neurons. In S. Martinez-Conde, S.L. Macknik, L.M. Martinez, J.-M. Alonso, and P.U. Tse (eds.), Visual Perception, volume 154 of Progress in Brain Research, pp. 93- 120. Elsevier, 2006. doi: http://dbbao////0006600000022210 URL https: //
Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoe V. Le. Attention augmented convolutional networks, 2020.
Santiago A. Cadena, George H. Denfield, Edgar Y. Walker, Leon A. Gatys, Andreas S. Tolias, Matthias Bethge, and Alexander S. Ecker. Deep convolutional models improve predictions of macaque v1 responses to natural images. 2017. doi: 10.1101/201764.
James Cavanaugh, Wyeth Bair, and J Movshon. Nature and interaction of signals from the receptive field center and surround in macaque vI neurons. Journal of neurophysiology. 88:2530-46. 12 2002. doi: 09.1152/jn.00692.2000
Ruben Coen-Cagli, Adam Kohn, and Odelia Schwartz. Flexible gating of contextual influences in natural vision. Nature neuroscience, 18, 10 2015. doi: 10.10088mn.4128
Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self- attention and convolutional layers, 2020.
R. James Cotton, Fabian H. Sinz, and Andreas S. Tolias. Factorized neural processes for neural processes: k-shot prediction of neural responses, 2020.
Zihang Dai, Hanxiao Liu, Quoc V. Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes, 2021.
Yicheng Fei and Xaq Pitkow. Attention inference with third-order interactions. In NeurlPS '22 Workshop on All Things Attention: Bridging Different Perspectives on Attention, 2022. URL https://openrevale
net forum? pecorumidd-s200 s2VEopgE
Daniel J. Felleman and David C. van Essen. Distributed hierarchical processing in the primate cerebral cortex. Cerebral cortex, 1 1:1-47, 1991. htt. semantisccoollalaa
Jiakun Fu, Suhas Shrinivasan, Kayla Ponder, Taliah Muhammad, Zhuokun Ding, Eric Wang, Zhiwei Ding, Dat T. Tran, Paul G. Fahey, Stelios Papadopoulos, Saumil Patel, Jacob Reimer, Alexan- der S. Ecker, Xaq Pitkow, Ralf M. Haefner, Fabian H. Sinz, Katrin Franke, and Andreas S. Tolias. Pattern completion and disruption characterize contextual modulation in mouse visual cortex. bioRxiv, 2023. doi: 10. 00.110001/2222.00.11.522277 URL htl.:ttp:: htmpp://wwer //ww://wwwww///w// biorxiv.
ort////oneee///rrrr22/2////////2 53.477
Kuan Han, Haiguang Wen, Yizhen, Di Fu, Eugenio Culurciello, and Zhongming Liu. Deep predic- tive coding network with local recurrent processing for object recognition, 2018.
Kyungmin Kim, Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Zhicheng Yan, Peter Vajda, and Seon Kim. Rethinking the self-attention in vision transformers. In 2021 IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 3065-3069, 2021. doi: 10.
1109/CVPRW53099.2200 2022.00042
William F. Kindel, Elijah D. Christensen, and Joel Zylberberg. Using deep learning to probe the neural code for images in primary visual cortex. Journal of Vision, 19(4):29 29, 04 2019. ISSN 1534-7362, doi: 10. 1211166/994429 URL
David A. Klindt, Alexander Ecker, Thomas Euler, and Matthias Bethge. Neural system identifi- cation for large populations separating "what" and "where", 2018.
12

---

Published as a conference paper at ICLR 2025
Nikolaus Kriegeskorte. Deep neural networks: A new framework for modeling bio-
logical vision and brain information processing. Anmual Review of Vision Science,
(Volume 1, 2015):417-446, 2015. ISSN 2374-4650. doi: https://doi.org/0..11
annurev-vision-0R211 14-034447.
Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Ha Hong. Najib J. Majaj, Rishi Rajalingham, Elias B. Issa, Pouya Bashivan, Jonathan Prescott-Roo Kailyn Schmidt, Aran Nayebi, Daniel Bear, Daniel L. K. Yamins, and James DiCarlo. Brain-like object recognition with nigh-performing shallow recurrent anns, 2019.
Bryan M. Li, Isabel M. Cornacchia, Nathalie L. Rochefort, and Arno Onken. Vlt: large-scale mouse vI response prediction using a vision transformer, 2023. URL /ttpitttpp///arxi.orrr///// 0302203022
Ming Li, Fang Liu, Hongfei Jiang, Tai Sing Lee, and Shiming Tang. Long-term two-photon imag- ing in awake macaque monkey. Neuron, 93(5):1049-10$ 10049-0577.e,, 2017. ISSN 0896-6277 6273. doi:
https:/doi.org/100 wwwp://..emmoo.200.111111 .027. URL/tttss / /www. sci sccienediii rect com/ sci sciencc/srticceen 1/0016661371111111 730000
Zhe Li, Wieland Brendel, Edgar Y. Walker, Erick Cobos, Taliah Muhammad, Jacob Reimer, Matthias Bethge, Fabian H. Sinz, Xaq Pitkow, and Andreas S. Tolias. Learning from brains how to regularize machines, 2019.
Konstantin- Benstantin-KKemens Lurz, Mohammad Bashiri, Konstantin Willeke, Akshay Jagadish, Eric Wang, Edgar Y. Walker, Santiago Cadena, Taliah Muhammad, Erick Cobos, Andreas S. Tolias, Alexander S Ecker, and Fabian H. Sinz. Generalization in data-driven models of primary vi- sual cortex. International Conference on Learning Representations, 2021. URL https: //openzeview wwwp://ooerddddi/////.
Nikola T. Markov, Julien Vezoli, Pascal Chameau, Arnaud Falchier, René Quilodran, Cyril Huis- soud, Camille Lamy. Pierre Misery, Pascale Giroud, Shimon Ullman, Pascal Barone, Colette Dehay, Kenneth Knoblauch, and Henry Kennedy. Anatomy of hierarchy: Feedforward and feed- back pathways in macaque visual cortex. Journal of Comparative Neurology, 522(1):225-25 URL https:// wwwooniinnlbbbaan
2014. doi: htts://doi.org// 10..0////e....0//eee.eoeoo 10020000022eeeeeee 23458.
com /w// 10..//bbb/...000/ceeeee 23458
Aran Nayebi, Daniel Bear, Jonas Kubilius, Kohitij Kar, Surya Ganguli, David Sussillo, James J. DiCarlo, and Daniel L. K. Yamins. Task-driven convolutional recurrent models of the visual system, 2018.
Xu Pan, Annie DeForge, and Odelia Schwartz. Generalizing biological surround suppression based on center surround similarity via deep neural network models. bioRxiv, 2023, doi: 10. 101/
2023.03.18.533299 URL sattps://ee www. 19/2023.03.15
Xuran Pan, Chunjiang Ge, Rui Lu, Shiji Song, Guanfu Chen, Zeyi Huang, and Gao Huang. On the integration of self-attention and convolution, 2022.
Galen Pogoncheff, Jacob Granley, and Michael Beyeler. Explaining vl properties with biologically constrained deep learning architecture, 2023.
Dean A. Pospisil and Wyeth Bair. The unbiased estimation of the fraction of variance explained by model. PLOS Computational Biology, 17(8) 08 2021. doi: 10. 10.137//jurma..pch.. 1009212. URL https: www.ooo.................. /0. 1371 /aar///ouraalla pac.mnaalpppb...... 1009212
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention in vision models, 2019.
Kosei Sasaki, Elizabeth C. Cropper, Klaudiusz R. Weiss, and Jian Jing. Functional differentiation of population of electrically coupled heterogeneous elements in microcircuit. Journal of
Neuroscience, 33(1):93 105, 2013. ISSN 0270-6474. doi: 10.523//NEURSSCL.4111
URL https: www. jreuroaci wtt.rracc..rrgccootttt///////
13

---

Published as a conference paper at ICLR 2025
S. Shushruth, Lauri Nurminen, Maryam Bijanzadeh, Jennifer M. Ichida, Simo Vanni, and Alessan- dra Angelucci. Different orientation tuning of near- and sur-sund suppression in macaque primary visual cortex mirrors their tuning in human perception. The Journal of Neuroscience, 33: 106- 119, 2013. URL htw.hhttss///eet
semant cscho ar wtt.oaaaooooooooooeeeeeeoooooaaa : 13650720
Courtney J Spoerer, Tim C Kietzmann, Johannes Mehrer, Ian Charest, and Nikolaus Kriegeskorte. Recurrent neural networks can explain flexible trading of speed and accuracy in biological vi- sion. bioRxiv, 2020, doi: 10. 10116727777 URL CRLhttps:// www. ear / 677723
Shiming Tang, Yimeng Zhihao Li, Ming Li, Fang Liu, Hongfei Jiang, and Tai Sing Lee, Large- scale two-photon imaging revealed super-sparse population codes in the v1 superficial layer of awake monkeys. eLife, 7:e33700, apr 2018a. Sa. ISSN 2050-084X. doi: 10.7554/cLifc.33370 URL
https://doi.com .org//o.g/.... 10 7554 eL. fe. 33370
Shiming Tang, Tai Sing Lee, Ming Li, Yimeng, Yue Xu, Fang Liu, Benjamin Teo, and Hongfei Jiang. Complex pattern selectivity in macaque primary visual cortex revealed by large-scale two- photon imaging. Current Biology, 28(1):38-445 28..8448...... 2018b. ISSN 0960-9822. doi: https://doi. 101.0000006......0111111.. htw.icceepp1/0060092222222
URL https: / /ww.. sci encedi rect. cwm sc ence/
Gido van de Ven, Tinne Tuytelaars, and Andreas Tolias. Three types of incremental learning. Nature Machine Intelligence, 4:1-13, 12 2022. doi: 10.1038/s42256-005 01118444225600220056666 022-00568-3.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023.
Konstantin F. Willeke, Paul G. Fahey, Mohammad Bashiri, Laura Hansel, Christoph Bless- ing, Monstantin-Klemeee Lurz, Max F. Burg. Santiago Cadena, Zhiwei Ding, Kayla Pon- der, Taliah Muhammad, Saumil S. Patel, Kaiwen Deng, Yuanfang Guan, Yiqin Zhu Kaiwen Xiao, Xiao Han, Simone Azeglio, Ulisse Ferrari, Peter Neri, Olivier Marre, Adrian Hoff- mann, Kirill Fedyanin, Kirill Vishniakov, Maxim Panov, Subash Prakash, Kishan Naik, Kan- tharaju Narayanappa, Alexander Ecker, Andreas Tolias, and Fabian H. Sinz. Retrospec- tive on the sensorium 2022 competition. In Marco Ciccone, Gustavo Stolovitzky, and Jacob Albrecht (eds.), Proceedings of the NeurIPS 2022 Competitions Track, volume 220 of Pro- ceedings of Machine Learning Research, pp. 314 333. PMLR, 28 Nov-09 Dec 2022. URL https:/ http://pocceeenn/eee/ee/e www.inggssmr....mmm
Konstantin F. Willeke, Kelli Restivo, Katrin Franke, Ame F. Nix, Santiago A. Cadena, Tori Shinn, Cate Nealley, Gabrielle Rodriguez, Saumil Patel, Alexander Ecker, Fabian H. Sinz, and Andreas S. Tolias. Deep learning-driven characterization of single cell tuning in primate visual area v4 unveils topological organization. bioRxiv, 2023. doi: 10.1 10.1101/2023.0..1 12. 540591. URL htttt:://///// htt.://ww..//ooooae////a//////// 05.
Daniel Yamins and James J. DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature Neuroscience, 19:356-366 -365566666,-06 2016. URL/bttps://app http:/ppp...mmatiiiccscccccllaaa
org/CorpusID: 16970545
Baosong Yang, Longyue Wang, Derek Wong, Lidia S. Chao, and Zhaopeng Tu. Convolutional self-attention networks, 2019.
Yimeng Zhang, Tai Sing Lee, Ming Li, Fang Liu, and Shiming Tang. Convolutional neural network models of vl responses complex patterns. Journal of Computational Neuroscience, 46:33 54, 2018. /RL
Yimeng Zhang, Harold Rockwell, Sicheng Dai, Ge Huang, Stephen Tsou, Yuanyuan Wei, and Tai Sing Lee. Recurrent networks improve neural response prediction and provide insights into underlying cortical circuits, 2022. (RRLihtts:///aa// https///waxxw..ora/.aaae wttp/oagg/aaaaaaaaoea 00825
Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring self-attention for image recognition, 2020.
14

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1 ADDITIONAL DETAILS ON MACAQUE EXPERIMENTAL SETUP
We collected data using a nearly identical experimental protocol as detailed in (Tang et al. 2018a b). except that our dataset was considerably larger, including up to 34K and 49K stimulus-response pairs for the two monkeys respectively.
During each fixation task, a blank screen was presented for 1500 ms after the monkey estab- lished fixation, followed by the presentation of visual stimulus for 500 ms. 34,000 stimuli were tested in monkey and 49,000 stimuli were tested in monkey 2 over a data collection period of 5 days.Neurons were registered anatomically and also by testing& a 200 stimuli finger-print each day. To quantify the neural response, a differential image of GCamp5s calcium signals between the stim- ulus period and blank period was computed for each trial. The dF/F was then calculated based on a 200 ms to 600 ms window after stimulus onset. We note that GCamp5s is slow, but the signal has been found to be correlated with firing rate (Li et al. 2017). The training set was collected with repeat for each stimulus. The validation consists of T000 stimuli, each with 10 repeats.
A.2
VI PREFERRED FEATURES
Curves
Eyes
Textures
SineGrating
Ba
Comer
others
Figure 7: VI neuron exhibit diversity and complexity in preferred features. Individual CNNs were used model neurons' responses large of natural images. Mid row: visualizations of the optimal stimuli for fE-CNN models for 28 neurons. Neurons are clustered into 7 equally sized classes representing preferences for curves, rings (eyes), textures, grating, bars, corners, and other more complex higher order features. Top row: top 5 natural images in the validation set that elicited the largest response from real neurons in each class. Bot row: top 5 artificial stimuli that elicited the largest response from f Ef-CNN models of neurons in each class. The artificial stimuli preferences are consistent with the optimal stimuli seen in the middle row.
A driving motivation of our study was to identify the natural image features that VI neurons prefer, which include corners, curvatures, junetions, rings and other higher order features, rather than the traditional orientation and frequency tunings. While traditional artificial stimuli produce tuning curve, these stimuli rarely represent the neuron's most preferred stimulus. The studies referenced above show that the neurons' true preferences not necessarily specific orientation spatial fre- quency, and may not be well modeled by Gabor functions and traditional linear/nonlinear models. Rather, the preferred features, which encoded by the peak responses of neurons as visualized in F exhibited diversity and complexity in the tuned features. The objective of this study is not to prove or argue for the diversity and complexity of V1 neural codes, but to explore the mecha- nisms contributing to them. A key finding of this paper is that the surround contextual mechanism, implemented by a non-local network (i.e. self-attention), positively contributes the generation of the response peak, and is related to the encoding of preferred higher-order features in neurons. As such, this study aims to model the peak response of neurons and develop a novel metric evaluate the model's is accuracy in capturing the peak of each neuron's 's "natural image tuning curve."
15

---

Published as a conference paper at ICLR 2025
COMPARISONS TO OTHER ESTABLISHED MODELS A.3
M2S1 Pearson Correlation
M2SI CC
M2S1 Pearson Correlation
M2S1 CC
(C)
Modell (Vrcp
"Whee and WiNN Mede:
"What When"
Trurs stor
FC CAR
28 CMN
CN-
CNB
Hyperparameter Changes
N/A (original CC-CNN)
0,444
Model
PT.
PT,
regularization in readout layer
0.401
EE-CNN
0,159 0.023
0.290 0,041
EE CNN with:
Poisson during training
0:400
"What and where"
All dropour layers removed
0.410
Model
0.100 + 0.024
0.210 0.045
E
Figure 8: Comparing the performance of Ff-CNN. (A), (C) shows the Pearson correlation compar- isons of ff CCNNNN with the "what and where" (shared core factorized) model and the transfer learning (goal driven VGG) model, respectively. (B), (D) show CC norm squared comparisons between mod- els. (E) shows hyperparameter experiments on a subset of 50 neurons from M2S1. (F) Comparison between ff CNN and the "what and where" model on the ability to capture M2S1 tuning peaks. All results MISI are similar.
Four major classes models are found in neural response prediction literature: (1) transfer learning models, (2) single CNN feedforward models, the (3) shared core factorized ("what and where") model (Klindt et al. 2018 Lurz et al. 2021), and more recently a (4) transformer based model for mouse The first three types of models have been used in macaque V1, whereas the transformer based model has only been used in mouse V1. We experimented with the first three classes of models and found that for our dataset, the performance of baseline feedforward models (ff-CNN) comparable, and at times better, to the transfer learning model and the shared core factorized mode used the single CNN feedforward model our baseline model. We did
(see Figure (8. Thus, not compare our model with transformer based models such as ViT (L al. (223). which use deep and complex layers achieve SOTA neural response prediction. The focus of our research and our contributions are different. We demonstrate that surround contextual modulation is critical in predicting the peak responses of macaque V1 neurons response to natural images. Towards this, we used self-attention to model horizontal interactions, rather than just using an entire transformer module to achieve SOTA performance.
16

---

Published as a conference paper at ICLR 2025
A.4 COMPARISONS TO OTHER ESTABLISHED METRICS
M251, Model: TF-NNN,
M251, Model: IT+sa-CNNV,
M251, Model: r-CNN
M251. Model: rT+sa-CNN,
Pearson Correlation vs FEV
Pearson Correlation vs FEV
Pearson Correlation vs TEV
Pearson Correlation vs TEV
Comeattonn
Gorrell deom
Figure 9: Pearson correlation versus FEV scatter plot for different models. Average FEV is calcu- lated by averaging over neurons in M2S1. Note that the performance models improve relative to the ff-CNN baseline when evaluated based on FEV, following a similar trend to that of Pearson correlation.
M251, Model: ff CNN,
M251. Model: fitsa CNN, vs
M251, Model: of CNN. Correaltion C
M251, Model: rf+sa CNN, Correaltion fr
vs
P3x00n Consanoo
Fecmur
Average
Avengo
Bevringe di
Busragg 0.402
Figure 10: Pearson correlation versus r scatter plot for different models. Average r is calculated by averaging over neurons in M2S1. Note that the performance of models improve relative to the ff-CNN baseline when evaluated based 2 following similar trend to that of Pearson correla- tion.
We compute measures FEV and r for each of our models. We calculated the average FEV and r2 across all neurons in M2S1 for different models. Similar to Pearson correlation and CC norm- FEV similar across models used in our study, and comparable established
and r measures baselines (see Figure 9 and Figure 10 Despite FEV and rer taking prediction magnitudes into account, they are still unable to capture the peak tuning properties. This can be attributed to the high degree of sparsity in individual neurons' tuming curve (to natural stimuli), where only 0.4% of the stimuli above half height on average. Because standard measures of performance are heavily influenced by remaining 99% low-responding stimuli, they are not sufficient for capturing the sharp tuning curves peaks we observed (i.e. recogniing the most preferred stimuli of the neurons).
17

---

Published as a conference paper at ICLR 2025
A.5 FCL DECOMPOSITION: AVERAGE CONTRIBUTION FROM EACH HYPERCOLUMN
5000
2008
Figure 11: Average FCL. contribution from each hypercolumn for M1S1. Center hypercolumn out- lined in red. Each heatmap is independently contrast normalized. Note that the distribution in the first heatmap is evenly distributed compared to the others (see contrast scale).
T the individual contribution of the center hypercolumn with the sum of all surrounding hypercolumns. We observed an evenly distributed contribution from all hypercolumns in the si- multaneously trained model, but strong center contribution in incrementally learned models. This effect is further observed when we display the average contribution of each hypercolumn (see Figure
A.6 MODELS THAT CAN CAPTURE PRAK TUNING EXHIBIT INTERPRETABLE CONTEXTUAL. MODULATION SUCH AS ASSOCIATION FIELDS IN THE SELF-ATTENTION MODULE
"NNsa-CNN* (Simul.)
rfesa-CNN* (Incr.)
fftsa- CNN 1imul. ml
Attention highlighting for incremental learning models. Top two highest re- sponse inducing image are shown for MISI neuron 153. Note that rf+sa-CNN* (Iner), PE+sa-CNN* (Iner. FCC) ( and ff+sa-CNN* (Incr.FC, all have the same attention map due to the freezing scheme. The center hypercolumn is queried for highlighted images.
Figure 12:
Comparing the attention highlighting between self-attention models from the incremental learn- ing experiment, observe that models learned incrementally have more focused attention sus equivalent-archiiectture counterparts trained simultaneously (as shown in Figure [2]. Models with strong peak tuning, which incorporate the surround properly, displays association field ef- fects, focusing on similar patterns as present in the receptor field. Furthermore, because of the incremental freezing scheme between all incremental models, they have the same attention despite variations in the readout layer. However, incremental models display focused attention, meaning the initially trained SA representation using the CTL. (in rftsa- 1f+sa-CNN* (Incr)) allows for proper learning of attention weights. This further supports that the 30 to 1 paramter bottleneck in the CTL is not a limiting factor, and the the gain performance in the latter incremental models (in ff+sa CNN" (Incr .CC and ff+sa CNN" (Iner. FC2) are associated with the FCL.
18

---

Published as a conference paper at ICLR 2025
A.7 SELF-ATTENTION CNNS ARE DATA EFFICIENT
Table 3: Average Pearson correlation and peak tuning metrics for models trained at different data
sizes of MISI. Correlation SEM = 0.008 was consistent across models.
25% of MISI
50% of MISI
Model
CORR. of FF-CNN
PTy
PTg
ACORR, fF-CNN
PTy
PT'
FE-CNN
0.0%
0.232
1.026
0.0%
0.762
1.556
FF+Sa-CNN
+7.2%
0.927
1.656
-7.6%
1.026
2.715
rf-CNN +26.9%
0.000
0.000
+21.1%
0.000
0.000
rf+sa-CNN
+23.6%
0.000
0.000
+19.8%
0.000
0.000
We trained baseline feedforward CNN models and their counterparts with self-attention at various training dataset sizes. We conclude that percentage improvements over ff-CNN are furthered at lower data constraints (as shown in Table alluding the potential efficiency of SA in accumu- lating surround information compared to other context mechanisms. We note that r-CCNN is the most data efficient when evaluated solely Pearson correlation. However, it is important to from Table 3 that at 25% and 50%data, rf-CNN and rf+sa- Sa-s-NNNN completely fail in the peak tun- ing index, indicating that the models were entirely unable to model the response magnitude of the highest excitatory images. This lends to our claim that although rf- CNNN does well in correlation, contextual information (as is present in Ff-CNN and ff+Sa-CNN) necessary to capture peak responses.
A.8 POPULATION TUNING CURVES
#E-CNN
IÊ-CHN
#É+sa-CNN
KÉ+sa-CNN
Figure 13: Population average tuning curves for M1S1.
Differences in peak tuming can also be observed the population tuming curves (see Figure Average curves are derived by calculating rank ordered tuning curves for each neuron individually, then averaging over the image number across neurons.
A.9 V MAPPING PARAMETER Y IN SELF-ATTENTION
In the self-attention layer (as shown in Figure 2). the mapping parameter 7 allows further factor- ization of inter-channel mixing and spatial interactions. Toggling False removes the transformed value vector, and attention weights instead directly on the input representation. Note that the y pa- rameter does make difference performance wise. The only difference between rf sa CNN* and rf+sa- CNN is the presence block in the latter. rÉ+sa-CNN" has better correlation, PT,, and PT3 values, meaning allowing for the V mapping in SA allows for more flexibility, despite the lack x convolution and FCL layer in these models.
SA with 7 = True block in the former and SA with 7 = False
19

---

Published as a conference paper at ICLR 2025
A.10 CTL CHANNEL NUMBER BOTTLENECK
In models with a CTL readout, the final layer is performing a 30 to 1 or 32 to weighted sum, depending the number of channels. Thus, an issue we considered was that such a narrow final layer would inhibit proper backpropagation of error signals to upstream modules. To address the concern of 30 + mapping in the CTL layer being too tight of an initial bottleneck, we trained <f+sa-CNT with = 375 channels, so that would have a 375 + CTL mapping instead. The results were comparable the sata--NN with c 30, meaning the drop performance from rf-CNN cannot be attributed to parameter bottleneck.
A.11 IMPORTANCE OF PETT-SELF-TTTER
CHANNEL MIXING
Additionally, we tested self-attention models without post-SA convolutions (i.e. no JCPB layers) and observed sharp drops in performances compared to baseline CNNs. This suggests that inter- channel mixing is crucial in processing the output of self-attention into a interpretable representation by the readout layer. We note that transformer block in modern computer vision models employ multi-layer perceptron immediately after self attentieenon which aligns with our findings.
To compare the importance the 3 3 versus 1 1 kernel size and FCL CTL readout as a means of incorporating surround information, we compared the following models: [aCPB + aCPB 7
SA(7 True) + 3CPB(k 1)~ 3CPB(k = 1) FCL]
[aCPB - aCPB - SA(7
True) + SCPB(k = 3) BCPB(k = - CTL].
The former with 3CPB(k: 1) and FCL outperformed the latter with 3CPB(k: 3) and CTL. Thus, direct access to spatial features with a fully connected layer is stronger than convolving the surround into the center. We observe that the FCL is the strongest factor for predicting the peak responses, and is bolstered by the addition SA, as ff+sa-CNN outperforms ff-CNN in peak tuning.
A.12 RECEPTIVE FIELD CHARACTERISTICS AND POPULATION- RF DISTRIBUTION
We mapped the neural networks' receptive fields using optimally oriented short bars. The left and center panel of Figure 14 illustrates two example neurons' receptive fields, indicating that short bar outside the classical receptive fields does not elicit response greater than the baseline. The half-height maximum receptive field diameter is approximately 3 pixels (0.4 degree visual angle), corresponding to 2-STD contour diameter of about 0.8 degree visual angle, consistent with exper- imentally determined receptive field sizes. The right panel of Figure 14 displays the distribution of mapped receptive fields of neurons for one site.
distribution selected neurons
height: 0.9
0000
height:
0375
13
630
N#2799
0.50 20
623
6329
y: 23.2
x: 30.2
6005
26.
X: 27.0
-335
width 4.1
width_x:
width_y: 3.2
width y :
8025
damater degins
Figure 14: Receptive field properties of neurons and their population-level distribution. Left and Center: two sample neurons' response maps fitted with an elliptical Gaussian. Right: histogram showing the distribution of receptive field diameters (in degree visual angle) for 279 neurons.
A.13 CAPTURING MODULATORY PROPERTIES OF EXTRA-CLASSICAL RECEPTIVE FIELDS
To verify that neural networks exhibit the extra-classical receptive field contextual modulation phenomena noted by the reviewer, we replicated following experiment by (Cavanaugh et al. 2002) our digital neurons.
20

---

Published as a conference paper at ICLR 2025
The following describes the methodology of the experiment. Neurons presented with center grating stimuli optimal orientation and spatial frequency of varying diameters (center-only stim- uli), centered on their receptive fields. The diameter of the smallest center-only stimulus eliciting least 95% of the neuron's 's maximum response defines the GSF (grating summation field). Each neu- was also presented with surround grating stimuli (with gray apertures centered the receptive field) of varying sizes. The classical receptive field was estimated as the aperture diameter at which the annular stimulus response is significantly below the neuron's maximum response circular grating patch. This aperture. referred to as the AMRF, represents where surround-only stimuli do elicit significant responses above the baseline response. When the center-only stimulus sig- nificantly larger than the size of the classical receptive field, surround suppression- characteristic example extra-classical receptive field modulation observed.
shows four example neurons from Cavanaugh et al.'s study demonstrating the classical Figure 15
surround suppression phenomena. Figure 15 and TSC shows our replication of their experiment, comparing results for the baseline model (ff-CNN) and the self attention model (ff+sa-CNN). The classical receptive field size of our neurons estimated using AMRF was approximately 0.8 degree vi- sual angle. We found that when the center-only stimulus exceeded this classical receptive field size, surround suppression- hallmark of extra- extra-classical receptive field modulation was consistently observed.
Center and Surround Grating Responses of fi-CNN
Surround diameter
Suppression
(A)
Surround Only Only
80
60
0.35
40
(B)
20
AMRF
Average Neural Response
0
5
0.25
25
0.20
20
0.10
Hisperss (ps))
0.05
0.5
2.5
Diameter of Center Aperture Degree Visual Angle
100
Center and Surround Grating Responses of sa-CsN
80
Surround Only Only
0.40
0.35
(C)
Average Neural Response
0
2
8
5
0.30
0.25
50
40
30
0.15
20
10
0.05
Diameter (deg)
Diameter of Center Aparture Degree Visual Angle
Figure 15: Capturing basic modulatory nature of extra-classical receptive fields. (A) shows four example neurons from (Cavanaugh et al., 2002) study demonstrating the classical surround suppres- sion phenomena. (B), (C) shows the average population response to the center only stimuli and the surround only stimuli for two models, ff CNN and ff saNNN.
21

---

Published as a conference paper at ICLR 2025
A.14 14
PERFORMANCE COMPARISON DIFFERENT MODELS AT VARYING DATA SIZES
Table 1: M2S1 Performance Metrics when Trained with 12.5% Data
Model
CORR
PTs
PT,
ff-CNN
0.260
0.000
0,000
ff+sa-CNN
0.272
0.000
0.000
Recurrent CNN (1 layer, 7 iterations)
0.291
2.174
1.304
(A)
Table 2: M2S1 Performance Metrics when Trained with 25% Data
Model
CORR
PTs
PT,
ff-CNN
0.295
0.000
0.000
ff+sa-CNN
0.317
0.033
0.013
Recurrent CNN (1 layer, iterations)
0.298
2.174
1.304
(B)
Table 3: M2S1 Performance Metrics when Trained with 100% Data
Model
CORR
PTs
PT,
ff-CNN
0.474
17.333
8.667
ff+sa-CNN
0.498
19.667
11.000
Recurrent CNN (1 layer, 7 iterations)
0.435
5.667
3.333
(C)
Figure 16: Performance of Ef-CNN, IE+Sa-CNN, and recurrent CNN at varying data sizes. (A) shows performance metrics at 12.5% data. (B) shows performance metrics at 25% data. (C) shows performance metrics at 100% data
Our results indicate that when the full training dataset (100% data) is utilized, the feedforward CNN augmented with self-attention (ff+ SA CNN) outperforms the recurrently augmented CNN across both overall tuning metrics and peak tuning metrics. However, consistent with the findings of Zhang eet al.) (2022). the recurrent augmented CNN exhibits superior data efficiency. Specifically, when the training dataset size is reduced to 25% or less, the recurrent augmented CNN surpasses both ff C-NNN and the ff+sa-CNN performance.
A.15
CODE FOR EXPERIMENTS
The code is hosted at the github repository: htnp://giiubb.com//cccnee//a/aa
22


Reviews:
[{"review_id": "wdz4SZC96y", "reviewer": "Reviewer_Jwq3", "summary": "The manuscript asks if self-attention can improve predictions of primary visual cortex (V1) response to images. This is motivated by two facts: one is surround modulation data in V1, usually attributed to lateral and top-down feedback connections; and the other is that artificial recurrent architectures predict some aspects of V1 responses better than purely feedforward. Self-attention could augment feedforward networks with spatial context information, beyond or differently from what can be achieved with receptive field expansion due to successive layers and final fully connected layer.\n\nThe authors execute a series of well thought-out comparisons between alternative architectures and learning schemes. The main results indicate that self-attention indeed improves predictive power, particularly when models are trained incrementally (learning the receptive field first). And that this improvement is more visible in the ability to capture responses to the most driving images for the neurons, rather than the overall response correlation across all images. This is interpreted as evidence that receptive field computations are more important to capture overall tuning, whereas surround computations are modulatory.", "questions": "1) It seems important to add a performance comparison between the self-attention augmented CNNs and CNNs with recurrent layers. Or to explain why not relevant. \n\n2) Does any of the model capture the basic modulatory natural of extra-classical receptive fields, i.e. surround suppression and no response to an annular stimulus? The analysis of Fig. 6 goes in that direction but it’s not quite the same, if I interpreted it correctly.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Excellently well written and executed. The problem is situated in the relevant literature, the motivation and significance of findings are conveyed strongly, the novelty is not overstated and the design focuses sharply on the question being asked. The results are very convincing. A pleasure to read!", "weaknesses": "Minor only. \n\n1) The paper certainly adds to understanding neural system identification. But at a high level, do the result say anything about V1 contextual modulation itself? Or do the authors envision a path to that goal by fitting self-attention augmented CNNs? In any event, I don’t think this detracts from the value of the manuscript.\n\n2) Even though not systems identification, this paper https://pubmed.ncbi.nlm.nih.gov/37738258/ is very relevant and in my opinion should be discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xHih1spRJ5", "reviewer": "Reviewer_fAeC", "summary": "This paper presents a method for predicting the neural activity of neurons in the primary visual cortex of primates upon presentation of natural images. This method is based on the use of a hierarchical neural network comprising several convolutional layers and learned in a supervised manner with the aim of predicting neuronal activity. An innovation of this algorithm lies in the inclusion of a self-attention mechanism in the processing sequence. This attention mechanism could then, on the one hand, help improve prediction capabilities and, on the other, help understand the modulation mechanisms present in a biological neural network which consist in facilitating or depressing the response of neurons according to a context.", "questions": "Assuming that the limitations detailed above will be corrected, I have many questions about this work:\n\nYou use a correlation measure to compare your predictions with neurobiological observations. Can you justify this approach in relation to the observed distribution of neuronal activity? Indeed, it seems that this distribution is sparse and that another measure might be more appropriate.\n\nYou have placed the attention layer after a features layer that is supposed to represent simulated activity in a neuron in the primary visual area. As such, can you interpret the various Q, K, and V features you've inferred during learning?\n\nIt seems you're trying to predict activity on a per-frame basis, but neural activity is dynamic. Do you think you can apply this type of model, and in particular the attentional layer, to the temporal domain?\n\nFinally, you mention a homology between the neural network that learns to predict neural activity and the network of biological neurons underlying the production of this activity. This major point needs to be better introduced.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "The strength of this paper lies in the rigorous introduction of this model and the comparison of different architectures, enabling the reader to disentangle the role of different network components. This method is validated by computational simulations that quantify the correlation between observed and predicted activity. This methodology then enables predictions to be made that confirm or refute certain biological theories, such as the modulation of normal activity according to the context of neighboring neurons.", "weaknesses": "I've noted a number of weaknesses in the document that affect its comprehensibility and potential impact. Foremost, the presentation needs to be simplified. First, it is difficult to understand from the abstract that your aim is to predict neural activity. In terms of the structure of the paper, there are inconsistencies in the sequence of sections, making section 3.5 and the model evaluation seem to conclude the paper. Some parts of the methods imply knowledge of intermediate results, and a major restructuring is needed.\n\nSecondly, the model is missing some essential elements for a proper understanding. If you describe it well, the input image given to the network is not the one you want to produce. From the figure, I deduce that it's the fluorescence contrast of the image, but this needs to be written more explicitly.\n\nThirdly, the figures are not always clear: figure four, for example, neither shows nor explains what is shown here. I'm guessing that the images are sorted according to their response, and that you superimpose the prediction that is made. Figure five shows results, but with fonts too small to read. This figure can also be simplified, as many elements seem redundant. Finally, the results discussed in the discussion are interesting, as they allow biological facts to be deduced from the study of a neural network, and in particular from the inclusion of an attention layer. However, this conclusion may also be the consequence of the limitations of the various models. To be validated, the conclusion should be causally related to the mechanisms described. In particular, I'm surprised that you don't look more closely at the activity in the attention layer: what are the strength and distance of the different modulations? This would enable you to draw neurological conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "58nOSYULSO", "reviewer": "Reviewer_eVhj", "summary": "The paper demonstrates that a self-attention layer inserted in a convolutional network can improve its ability to predict neural responses to natural stimuli from macaque area V1. The neural responses were collected with two-photon imaging in two separate monkeys. Different variants of a feed-forward CNN are compared (with or without self-attention, with or without fully-connected spatial integration, with end-to-end learning or incremental pretraining and fine-tuning of specific layers). The authors conclude that self-attention can improve neural prediction accuracy, especially for explaining “peak tuning” (i.e., the top 1% of highest responses).", "questions": "* There is no clear description of the training objective for the different models compared. I assume it to be a regression of single-neuron activation from each training image, using an MSE loss, but this should be described much more explicitly.\n* The notions of “overall tuning” and “tuning peak” are used in the abstract and introduction, but these are not standard terms, and cannot be understood without some amount of explanation (e.g. “the tuning peak corresponds to the top 1% of highest responses”).\n* The Zhang (2022) reference does not include information about a journal, preprint server or conference venue.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* Fitting neural network models to brain data is an important approach to determine the biological plausibility of artificial systems, and to understand computational brain mechanisms\n* The methods are generally sound and the compared models and ablations allow us to draw conclusions about the investigated mechanisms", "weaknesses": "* The paper does not clearly state its primary objective, and the reader is left to choose between two options. The main objective could be to prove that recurrent and horizontal interactions help explain extra-classical receptive field properties in V1, with self-attention as one specific way to implement these interactions. Or, it could be to argue that self-attention (as implemented in modern neural networks with Q, K and V projections) is a good model for recurrent/horizontal interactions in area V1. I believe the former interpretation is better supported by the data, although less novel (since earlier papers have already reported that recurrent layers can improve neural predictions). However, I fear that the community would more readily jump to the latter conclusion, which is unfortunately not well supported. To make this interpretation, the self-attention models should be directly contrasted with other baselines including one (or more) recurrent layer(s) that implement horizontal interactions without self-attention. In the absence of these baselines, only the first conclusion can be supported, but this hinders the novelty of the paper’s message. In any event, the authors should clearly state which of these two goals they are targeting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TsDCGnN59A", "reviewer": "Reviewer_rqi7", "summary": "This paper explores the role of self-attention mechanisms in modeling contextual modulation in primary visual cortex (V1) neurons of macaque monkeys. Key findings and contributions include:\n\n* Adding a self-attention layer to convolutional neural networks (CNNs) improves neural response prediction of macaque V1 neurons in terms of overall tuning curve correlation and peak tuning (PT metrics). This suggests self-attention provides additional flexibility and benefits complementary to CNNs' inherent contextual modulation mechanisms (successive convolutions and fully connected layers).\n\n* The classical receptive field is the primary driver of a neuron's overall response, as models focusing on receptive field information achieve the highest correlation. However, contextual modulation, especially via self-attention, is crucial for strong and robust peak tuning.\n\n* Incremental learning, where the receptive fields are learned first followed by self-attention and fully connected layers, allows the model to properly learn the contributions of the receptive field and contextual modulation. This incremental approach leads to a receptive field-centric model more aligned with neurophysiological evidence.\n\n* Interpretable contextual modulation effects, such as association fields, emerge in the self-attention module of models that capture peak tuning well.\n\n* Self-attention augmented CNNs are more data-efficient compared to baseline CNNs.\n\nOverall, the paper provides insights into the computational role of self-attention (single head) in modeling horizontal connectivity and contextual modulation in the visual cortex, and proposes an incremental learning approach to best train networks with self-attention.", "questions": "Just a minor question: In section 3.4, I read that the loss is MSE. Many (if not most) papers in the field use a Poisson NLL loss, even with two-photon data (where, I agree that assumptions of e.g., electrophysiological recordings and Poisson counting aren't met). Could you elaborate on the use of MSE? Have you tried both and found better results with MSE?\n\nOverall, I'm happy to maintain the assigned score (I believe the paper is more of a 7 in its current state, but 7 is not among the allowed ratings) if a more extensive comparison with Li et al. (2023) is provided.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is overall well written and the results are quite convincing. Here are the points I believe are strengthening the submission:\n\n* The paper introduces a new perspective on modeling contextual modulation in V1 neurons using self-attention mechanisms, which has not been extensively explored before.\n\n* The authors systematically investigate the contributions of different contextual modulation mechanisms (convolutions, self-attention, and fully connected layers) and their interactions, providing insights into their roles in modeling neural responses.\n\n* The incremental learning approach, where receptive fields are learned before contextual modulation, is inspired by the developmental process in the visual cortex, adding biological plausibility to the model.\n\n* New evaluation metric: the peak tuning index is introduced to assess a model's ability to capture a neuron's feature preference, addressing the limitations of standard metrics like Pearson correlation.\n\n* The authors demonstrate that models with strong peak tuning display interpretable contextual modulation effects, such as association fields, in the self-attention module. I actually believe that this last point should be included in the main paper instead of the Appendix, as it brings added value by bridging previous research at the interface of neuroscience, perception, and machine learning.", "weaknesses": "**Major Weaknesses**\n\n* While the paper compares the proposed models to established CNN architectures, it does not include comparisons with more recent state-of-the-art models, such as transformer-based models, which have shown promising results in modeling mouse V1 neurons. More precisely, while Li et al. (2023) is mentioned in the introduction, no comparison with this model is provided in the paper. Although it's true that multiple attention heads may intuitively seem unnecessarily complex for the task outlined in the paper, Li et al. (2023) demonstrates very good results in predicting neural responses in mouse V1 neurons.\n\n**Minor Details**\n\n* Figure 1(b) is not readable: The axes labels, ticks, and different images are unclear. Consider resizing or reorganizing the figure differently.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)