Self-Attention-Based Contextual Modulation Improves Neural System Identification

## Abstract

Abstract Convolutional neural networks (CNNs) have been shown to be state-of-the-art models for visual cortical neurons. Cortical neurons in the primary visual cortex are sensitive to contextual information mediated by extensive horizontal and feedback connections. Standard CNNs integrate global contextual information to model contextual modulation via two mechanisms: successive convolutions and a fully connected readout layer. In this paper, we find that self-attention (SA), an implementation of non-local network mechanisms, can improve neural response predictions over parameter-matched CNNs in two key metrics: tuning curve correlation and peak tuning. We introduce peak tuning as a metric to evaluate a model’s ability to capture a neuron’s feature preference. We factorize networks to assess each context mechanism, revealing that information in the local receptive field is most important for modeling overall tuning, but surround information is critically necessary for characterizing the tuning peak. We find that self-attention can replace posterior spatial-integration convolutions when learned incrementally, and is further enhanced in the presence of a fully connected readout layer, suggesting that the two context mechanisms are complementary. Finally, we find that decomposing receptive field learning and contextual modulation learning in an incremental manner may be an effective and robust mechanism for learning surround-center interactions.

## 1 Introduction

Feedforward CNN models have been shown in recent years to be an effective approach for modeling and predicting visual cortical neurons’ responses to arbitrary natural images (Kindel et al., ; Klindt et al., ; Yamins & DiCarlo, ; Zhang et al., ; Cadena et al., ; Kriegeskorte, ). Neurons in the primate visual cortex are known to have extensive horizontal and feedback recurrent connections for mediating contextual modulation (Felleman & van Essen, ; Markov et al., ). Feedforward CNNs can model the influence of contextual surround on the responses of the neurons via two mechanisms: successive convolution layers and a fully connected layer. Both can make the neural model’s responses sensitive to the global image context, outside the traditional classical receptive fields of neurons. In the context of neural prediction, it is found that including the inductive bias of horizontal recurrent connections can improve the model’s predictive capabilities (Zhang et al., ), and that replacing a feedforward layer with a recurrent layer using a Markovian local kernel consistently outperforms parameter-matched feedforward CNNs in image classification tasks (Han et al., ; Nayebi et al., ; Kubilius et al., ; Zhang et al., ). However, contextual modulation in the visual cortex involves both the near surround and far surround, with the far surround being mediated by top-down feedback (Angelucci & Bressloff, ; Sasaki et al., ; Shushruth et al., ). In addition, there is evidence that contextual modulation is dynamic and highly image-dependent, suggesting a flexible gating mechanism (Coen-Cagli et al., ). Such a flexible gating mechanism can be modeled by a combination of Gaussian mixture models, implemented either by image-dependent normalization (Coen-Cagli et al., ) or by non-local networks and the self-attention mechanism in deep learning (Fei & Pitkow, ). Self-attention-based architectures such as vision transformers have recently been shown to be effective in modeling mouse V1 neurons (Li et al., ). However, these networks, often using a large number of layers and multiple attention heads, may be unnecessarily complex for this task.

In this paper, we demonstrate that a simple self-attention layer coupled with a CNN is sufficient in improving neural response prediction of macaque V1 neurons in two performance metrics: overall tuning correlation and prediction of the tuning peaks. To understand the mechanism driving improvement, we assessed the three contextual modulation mechanisms – convolutions, self-attention, and a fully connected readout layer. We found that while the three context mechanisms complement one another to produce the best prediction performance when used in conjunction, they have specific roles. First, the fully connected layer plays a critical role in peak prediction, though self-attention can further enhance it. Second, self-attention alone can improve tuning curve correlation but is insufficient for predicting the response peak. The performance of self-attention models can be greatly enhanced when the feedforward receptive fields are learned first before learning the self-attention network, rather than learning everything simultaneously. The benefits of such incremental learning (van de Ven et al., ; Cotton et al., ) in this context are novel, suggesting that decoupling the learning of feedforward receptive fields and recurrent connections allows the system to learn a richer representation of contextual modulation, as well as potentially providing insights towards the underlying computational organization of cortical development.

Figure: Figure 1: Macaque neuronal response dataset. (a) shows a two-photon image with cells. (b) shows the response of one neuron to $50$k stimuli and the top 20 images that induced the strongest responses. On average, less than 0.5% of the images induce responses greater than half peak height. Each site contains around 300 neurons. (c) shows a feedforward CNN used to model neural response.
Refer to caption: x1.png

## 2 Related works

#### Modeling neural response prediction

Feedforward deep neural networks have proven effective in modeling and predicting neural responses in early visual brain areas (Kindel et al., ; Klindt et al., ; Yamins & DiCarlo, ; Zhang et al., ; Cadena et al., ; Kriegeskorte, ). However, the brain’s visual areas contain abundant recurrent connections that are essential for generating neural responses (Felleman & van Essen, ; Markov et al., ; Spoerer et al., ). Incorporating biologically-inspired simple recurrent circuits, in the form of a Markov network, into convolutional neural networks has been shown to enhance efficiency compared to purely feedforward models, achieving similar performance in image classification and neural prediction tasks (Zhang et al., ). In the context of neural prediction, the underlying assumption is that the closer a model can replicate the neural computation mechanisms responsible for a real neuron’s response, the more accurate the model’s predictive capabilities become (Pogoncheff et al., ; Willeke et al., ; Li et al., ).

#### Self-attention for global dependencies

Self-attention mechanisms have recently become a pivotal component in deep learning models, especially in natural language processing and increasingly in computer vision tasks (Vaswani et al., ; Zhao et al., ; Kim et al., ). In computer vision, self-attention performs a weighted average operation based on the context of input features, computing attention weights dynamically through a similarity function between pixel pairs (Vaswani et al., ; Pan et al., ). This flexibility allows the attention module to adaptively focus on different regions and capture informative features (Ramachandran et al., ). Self-attention has also been integrated with CNNs to enhance their representational power (Pan et al., ; Yang et al., ; Bello et al., ). By enabling CNNs to consider distant spatial relationships within an image, self-attention improves the network’s ability to capture global context. This mechanism overcomes the limitations of traditional CNNs, which primarily concentrate on local features because of their convolutional structure. Taking the complementary properties of convolution and self-attention, the benefits of each paradigm can be extracted by integrating the two and using self-attention to augment convolution modules (Dai et al., ; Yang et al., ; Pan et al., ; Cordonnier et al., ).

Figure: Figure 2: Models explored in this study. Models are constructed from two types of convolutional processing blocks (CPB): $\alpha$CPB and $\beta$CPB. $\alpha$CPB has a fixed convolution kernel size $=5$ and max pooling kernel size $=2$. $\beta$CPB takes an input convolution kernel size of $k$, and has no pooling layers. The two final layer readout modes are fully connected (FCL) and center hypercolumn only (CTL). Self-attention (SA) takes as input a boolean $\gamma$ that determines whether the value (V) vector is transformed; if $\gamma=$True then V is mapped, otherwise V is equal to the input. All models with SA utilize single-headed attention. (a) shows the feedforward CNN. (b) shows the feedforward CNN augmented with self-attention. (c) shows the receptive field CNN. (d) shows the receptive field CNN augmented with self-attention.
Refer to caption: x2.png

## 3 Approach

In this study, we developed a set of deep learning models to model V1 neural response to natural images, with the goal of evaluating the potential roles of the self-attention mechanism in neural computation within the visual cortex. We obtained a dataset of neuronal responses measured using two-photon imaging with GCaMP5 from two awake behaving macaque monkeys performing a fixation task, consisting of $302$ neurons from monkey 1 (M1S1) and $299$ neurons from monkey 2 (M2S1), in response to $34$k and $49$k natural images extracted from the ImageNet dataset. The neurons were recorded across five days and tracked anatomically based on landmarks as well as based on their responses to $200$ fingerprint images tested every day. The images were presented in sequence with $500$ ms per image preceded by $1500$ second of blank screen. The $30$k-$50$k images in the training set were presented once, and the $1000$ images in the validation set were tested once with $10$ repeats. Images were $100\times 100$ pixels, with $30\times 30$ pixels for $1$ degree visual angle. The eccentricity of the recording sites were $3$ degrees and $1.79$ degrees, with average receptive field sizes (diameters at half-height) of about $0.75$ and $0.5$ degrees, respectively. We preprocessed the dataset before modeling, and notably downsampled input images to $50\times 50$ pixels, yielding $15\times 15$ pixels per degree visual angle. The receptive fields of neurons from each $1$ mm $\times$ $1$ mm site (the scale/size of a hypercolumn) in macaque monkeys exhibited significant overlap. These fields were mapped using oriented bars or SmoothGrad feature attribution on our deep learning model. The standard deviation of the receptive field centers, all of which are contained within the center hypercolumn of our CNN models, was less than 1 pixel ($1/15$ degree visual angle). See Appendix A.1 for more details of the macaque experimental setup.

### 3.1 Augmenting feedforward CNNs with self-attention

First, we investigate whether incorporating self-attention into the baseline feedforward CNN model enhances neural response prediction performance. See Appendix A.3 for comparisons to other established models.

#### Baseline feedforward model ( ff-CNN )

See Figure 2(a) for the architecture. The baseline feedforward model is comprised of two $\alpha$-convolutional processing blocks ($\alpha$CPB) and two $\beta$-convolutional processing blocks ($\beta$CPB), followed by a fully connected readout layer (FCL) . A single ff-CNN model is fitted to each neuron. All models described below are derived from this baseline model. Given a grayscale input image with dimensions $50\times 50$ pixels, the two $\alpha$CPB layers with a $5\times 5$ kernel encode the input of size $(1\times 50\times 50)$ into $(c\times 9\times 9)$ where $c\in\mathbb{N}$ is the number of channels ($c\in\{30,32\}$ in this study). The center hypercolumn of the post-$\alpha$CPB encoding has a centered effective receptive field size of $13\times 13$ pixels. In other words, the center hypercolumn of the latent representation after the $\alpha$CPB layers will have a $13\times 13$ (or $0.8\times 0.8$ degree visual angle) feedforward receptive field at the center of input $50\times 50$ image. Note that the real neurons’ receptive fields are contained inside the receptive field of the center hypercolumn. In the baseline model, the two $\alpha$CPB layers are followed by two $\beta$CPB layers with $3\times 3$ kernels to further expand the effective receptive field of the center-hypercolumn. Finally, ff-CNN has access to the entirety of the input image in the final layer as the readout has full access to all the hypercolumns. Thus, the baseline ff-CNN CNN has two modalities of contextual modulation – convolutions and a fully connected layer.

#### Feedforward with self-attention model ( ff+sa-CNN )

See Figure 2(b) for the architecture. We augment ff-CNN with a self-attention layer immediately after the last $\alpha$CPB and before the first $\beta$CPB. This placement enables SA to act on an adequately convolved feature representation, but also be further modulated by convolutions before feeding into the final layer. We compare the performance of ff+sa-CNN against that of ff-CNN, controlling the parameter counts to be roughly equal by decreasing the number of channels, which is maintained throughout entire model, from $c=32$ in the baseline CNNs to $c=30$ in the self-attention models to account for the addition of the SA layer. In the context of contextual modulation, ff+sa-CNN intermixes spatial interactions and inter-channel mixing across SA, the posterior $\beta$CPBs, and the FCL.

### 3.2 Factorizing the contextual modulation mechanisms

There are three mechanisms in ff+sa-CNN mediating contextual interactions. We proceed to factorize ff+sa-CNN by removing the contextual modulation contributed by the $\beta$CPBs and the FCL to assess the standalone capability of SA in incorporating surrounding context. This is accomplished by constructing a baseline receptive field model and a model where only SA is mediating horizontal connections.

#### Baseline receptive field model ( rf-CNN )

See Figure 2(c) for the architecture. We first construct the rf-CNN model, which is devoid of contextual modulation, by subtracting from ff-CNN: the kernel size in the $\beta$CPBs are changed from $3\times 3$ to $1\times 1$ and the fully connected layer is changed to look only at the center hypercolumn (CTL). The $1\times 1$ convolutions perform no spatial expansion before feeding into the CTL. Thus, rf-CNN is making predictions solely based on the center hypercolumn receptive field produced by the $\alpha$CPBs, which covers the center $13\times 13$ pixels of the input image.

#### Receptive field with self-attention model ( rf+sa-CNN )

See Figure 2(d) for the architecture. We add self-attention to rf-CNN prior to the $\beta$CPBs to construct rf+sa-CNN. Self-attention is the only mechanism for incorporating surround context in this model. The parameter counts are again controlled by reducing the number of channels from $c=32$ (in rf-CNN) to $c=30$ (see Appendix A.10 for more details). We compare the performance of the two receptive field models, alongside the feedforward models. Note that in rf+sa-CNN, $\gamma$ is False in the SA layer, meaning SA operates exclusively on the horizontal spatial interactions between hypercolumns without any inter-channel mixing. In contrast, $\gamma$ is True in the SA layer of ff+sa-CNN, which allows channel mixing in SA. Channel mixing potentially provides self-attention greater flexibility (see Appendix A.9 and A.11).

### 3.3 Incremental learning: factorizing the learning process

Figure: Figure 3: Incremental learning models setup. (a) shows the baseline receptive field CNN, equivalent to Figure 2(c). (b) shows (a) augmented with self-attention and learned incrementally; the $\alpha$CPBs are taken from (a) and the remaining layers are learned. The ^∗ denotes slight modification from rf+sa-CNN, Figure 2(d), namely $\gamma$ is changed to True. (c), (d) show the result of replacing the CTL in (b) with a FCL, and learned incrementally; (c) freezes only the center hypercolumn in the FCL (FC1) whereas (d) allows the FCL to learn freely (FC2). (c) and (d) have all other layers taken from (b). The ^∗ denotes slight modification from ff+sa-CNN, Figure 2(b), namely $k$ in $\beta$CPB is changed to $k=1$. (Simul.) models are equivalent in architecture, except all blocks are learned.
Refer to caption: x3.png

Discussed in Section 4.2, comparing the four models above reveals that contextual modulation introduced in rf+sa-CNN via SA did not produce better performance relative to rf+CNN, despite ff+sa-CNN having clear performance improvements over ff-CNN. This is not due to the difference in channel mixing (the $\gamma$ parameter in SA). We hypothesize that bottle-necking all the gradient signals solely through the center hypercolumn during backpropagation makes it difficult for the network to properly learn the $\alpha$CPB layers and the SA layer simultaneously. Thus, we investigate an incremental learning paradigm where we allow the receptive fields of the $\alpha$CPBs to be learned before incorporating any context mechanisms. We then incrementally add and learn a self-attention layer followed by a fully connected readout layer.

The following progression of models, rf-CNN, rf+sa-CNN^∗, and ff+sa-CNN^∗ (as shown in Figure 3), incrementally expands the capacity of contextual modulation. An important distinction between incremental models and models shown in Figure 2, marked by ^∗, is a $1\times 1$ kernel in the $\beta$CPB, which maintains channel mixing but removes further spatial integration through convolution. rf-CNN (shown in Figure 3(a) or Figure 2(c)) has information only from the center receptive field. rf+sa-CNN^∗ (shown in Figure 3(b)) uses only the self-attention mechanism for contextual modulation. ff+sa-CNN^∗ (shown in Figure 3(c)-(d)) has the same surround-center modulation as rf+sa-CNN^∗ from self-attention, but allows spatial integration of the global context by changing the CTL to FCL at the end. As horizontal connections in the visual cortex are known to mature after the development of the receptive fields, we designed an incremental learning setup where rf-CNN first learns the receptive fields, then rf+sa-CNN^∗(Incr.) learns a self-attention layer only after rf-CNN has already learned the $\alpha$CPB receptive fields. Finally, ff+sa-CNN^∗(Incr.FC1) and ff+sa-CNN^∗(Incr.FC2) inherit the receptive fields and self-attention structures of rf+sa-CNN^∗, but differ in the change to a FCL readout. Models labelled (Incr.) are learned incrementally as such, and models labelled (Simul.) are traditionally trained simultaneously.

### 3.4 Hyperparameter selection and Model training

Rather than splitting the evaluation set for hyperparameter selection, we partitioned our population of neurons to select hyperparameters (training and architectural). We fine-tuned, by experimenting with batch size, learning rate, epochs, number of layers, number of channels per layer, etc., model hyperparameters on a subset of 50 neurons using a relatively coarse grid search. We list key training hyperparameters here: (1) batch size = $50$, (2) learning rate = $0.001$, (3) optimizer = Adam, (4) loss = MSE, (5) epochs = $50$. Training and computations were performed on an in-house computing cluster with GPU (NVIDIA V100 or similar) nodes. Training hyperparameters were held constant across all models. Architectural hyperparameters were held constant across layers shared between models. We do not optimize hyperparameters for models other than the baseline ff-CNN.

The primary objective of this project is to demonstrate that self-attention can enhance neural response prediction relative to the baseline feedforward CNN, despite hyperparameters being optimized only for the baseline model. Since we show that ff+sa-CNN improves upon ff-CNN in both evaluation metrics (see Section 4.1), further hyperparameter optimization is unnecessary for our objective. Instead, we are interested in understanding the reason behind this improvement. The other models tested in this study are architectural subsets of the ff+sa-CNN, designed to dissect their contributions to its success. We do not anticipate any derivative models outperforming ff+sa-CNN, justifying holding hyperparameters constant across models for fair comparison.

### 3.5 Model evaluation

To quantify performance, models were evaluated on two criteria, Pearson correlation and peak tuning index. Pearson correlation represents the overall tuning similarity between a model’s predicted responses and the real neuron’s recorded responses. The peak tuning index is used to quantify how well a model can predict and match in magnitude the strongest responses recorded by the real neuron. This lets us evaluate how well a model can discriminate between, as well as model the response magnitude of, images that are strongly excitatory and images that incite a weak response.

Figure: Figure 4: Pearson correlation does not reflect peak tuning. Despite rf-CNN having the better correlation, it is clear that ff+sa-CNN is able to capture the peak significantly better, at the cost of a noisier overall tuning. Example shown is M1S1 neuron 238. See Appendix A.8 for population averages.
Refer to caption: x4.png

#### Pearson correlation:

The Pearson correlation (CORR.) is taken between neuron responses and model-predicted responses. Pearson correlation is a standard measure for evaluating neural response prediction. Other established measures, including FEV, $r^{2}_{er}$, and CC${}^{2}_{norm}$ (Willeke et al., ; Cadena et al., ; Pospisil & Bair, ; Zhang et al., ), were used as well and yielded similar results to Pearson correlation (see Appendix A.3 and A.4).

In this paper, we explored a set of measures to assess the peak of a neuronal tuning curve. In our macaque V1 dataset, we found that neurons exhibit sharp stimulus selectivity, consistent with findings from Tang et al. (; ), reinforcing the diversity and complexity of V1 neurons (see Appendix A.2). We found that Pearson correlation and other standard metrics (see Appendix A.4) are successful in measuring a model’s fit to the overall tuning curve, but often fail to represent the peak tuning preference of neurons, of which is a key aspect of a neuronal function. For example, Figure 4 demonstrates that while rf+CNN achieves a higher correlation in approximating the overall tuning curve compared to ff+sa-CNN, the latter outperforms in modeling the neuron’s tuning curve peak. To address this issue, we developed two new metrics to better assess a model’s ability to capture the peak tuning of neurons.

#### Peak tuning index:

The peak tuning index (PT) is a membership metric of the strongest predictions above a threshold determined by the top $1\%$ of real responses. PT can be roughly interpreted as the percentage of the peak that a model captures, under a magnitude prior. The index is calculated as:

$$ $\text{PT}=\frac{\texttt{\# of top 1\% predictions}\geq\texttt{min(top 1\% real responses)}}{\texttt{\# of responses in the top 1\%}}\times\texttt{100\%}$ $$

PT is divided into PTJ and PTS, based on how # of top 1% predictions is defined. PTJ is when predictions are jointly rank ordered with respect to the real responses. PTS is when predictions are separately rank ordered independently of the real responses. PTJ is a stricter measure. Note that because we train with MSE loss, models are incentivized to minimize the absolute difference between predictions and real responses, rather than match the curvature of the tuning curve. This minimizes the risk of PT being misrepresentative due to lateral shifts in the tuning curve.

## 4 Results

### 4.1 Self-attention improves neural response prediction

**Table 1: Average Pearson correlation and peak tuning metrics for models trained on M1S1 and M2S1. Correlation SEM $=0.009$ was consistent across models and monkeys. Despite rf-CNN unexpectedly outperforming rf+sa-CNN, the difference is recovered when rf+sa-CNN is trained incrementally (see Section 4.2).**
|  | M1S1 | M2S1 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | CORR. | $\Delta$ ff-CNN | PTJ | PTS | CORR. | $\Delta$ ff-CNN | PTJ | PTS |
| ff-CNN | $0.393$ | $0.0\%$ | $3.3\pm 0.5$ | $5.6\pm 0.9$ | $0.477$ | $0.0\%$ | $8.6\pm 0.9$ | $16.2\pm 1.6$ |
| ff+sa-CNN | $\mathbf{0.416}$ | $+6.6\%$ | $\mathbf{5.6}\pm 0.6$ | $\mathbf{10.5}\pm 1.1$ | $\mathbf{0.491}$ | $+3.3\%$ | $\mathbf{11.5}\pm 0.9$ | $\mathbf{23.5}\pm 1.8$ |
| rf-CNN | $\mathbf{0.420}$ | $+8.6\%$ | $\mathbf{1.1}\pm 0.3$ | $\mathbf{1.8}\pm 0.5$ | $\mathbf{0.496}$ | $+4.3\%$ | $\mathbf{4.4}\pm 0.6$ | $\mathbf{6.6}\pm 1.0$ |
| rf+sa-CNN | $0.414$ | $+7.2\%$ | $0.7\pm 0.2$ | $1.0\pm 0.3$ | $0.486$ | $+2.4\%$ | $3.4\pm 0.5$ | $5.1\pm 0.8$ |

We compared the performance of the ff+sa-CNN model to the parameter-matched baseline ff-CNN model and found that incorporating self-attention significantly improved correlation and both peak tuning metrics (see first two rows of Table 1). This indicates that self-attention enhances modeling of both the overall tuning and peak tuning aspects of the neurons, with consistent results across both monkeys.

It is important to note that the ff-CNN center hypercolumn at the readout layer has a receptive field much larger than the real neuron’s receptive field, due to successive convolutions in the $\alpha$CPBs and $\beta$CPBs. Additionally, the fully connected readout layer also incorporates long-range spatial dependencies. Thus, the self-attention layer in ff+sa-CNN acts as an additional mechanism for modeling horizontal connections, and provides additional performance benefits.

To better understand the role of self-attention in contextual modulation, we constructed a baseline receptive field model, rf-CNN, that is devoid of all contextual modulation mechanisms. rf-CNN’s CTL readout only uses the center-hypercolumn of the convolved feature space to make predictions. Note that the feedforward receptive field of the center hypercolumn after the two $\alpha$CPBs is a centered $13\times 13$ pixel portion of the input image. Moreover, between the $\alpha$CPB layers and the CTL layer, the size of the center hypercolumn receptive field does not expand due the the $1\times 1$ kernel in the $\beta$CPBs. This means that predictions are being made solely based on the center $13\times 13$ pixels, which corresponds roughly to the real neuron’s receptive field.

Surprisingly, we found that rf-CNN achieved the highest correlation, indicating that models focusing primarily on the classical receptive field offer the best fit to the overall tuning curve (see Table 1). However, correlation fails to reflect the model’s shortcomings in fitting the peak of the tuning curve. Table 1 also reveals that while rf-CNN has the highest Pearson correlation, it performs worse at capturing the tuning peak compared to ff+sa-CNN and ff-CNN. Thus, we conclude that contextual modulation plays a crucial role in peak tuning, and that the three mechanisms in ff+sa-CNN for integrating surround information are complementary.

### 4.2 Dissecting Contextual Modulation Mechanisms via Incremental Learning

We explore the relative contributions of the different contextual modulation mechanisms. Specifically, we first ask: Is self-attention alone sufficient to model contextual modulation? To answer this question, we added SA to rf-CNN to produce rf+sa-CNN. We found that without posterior $\beta$CPBs and a FCL, the self-attention in rf+sa-CNN is not useful. In fact, the performance is worse than rf-CNN in overall correlation and both peak tuning metrics (see Table 1). This result is somewhat unexpected, as the addition of self-attention in ff+SA-CNN does improve upon ff-CNN. It is possible that the spatial integration mechanisms in the $\beta$CPBs, along with the FCL, are necessary to provide sufficient pathways for backpropagating the gradients during learning, so that both the $\alpha$CPB receptive fields and self-attention kernels can be properly learned.

To test this hypothesis, we explored whether an incremental learning approach, where different network components are learned sequentially, could yield a model that performs on par with ff+sa-CNN. We show that learning the feedforward kernels in the $\alpha$CPBs first, followed by learning the self-attention layer, and finally the fully connected layer, can nearly match the performance of ff+sa-CNN. This indicates that although the spatial integration by $\beta$CPBs contributes to performance, the self-attention layer plays a more critical role in capturing horizontal interactions crucial for modeling peak tuning.

**Table 2: Average Pearson correlation for models incrementally and simultaneously trained on M1S1 and M2S1. Correlation SEM $=0.009$ was consistent across models and monkeys.**
|  | M1S1 | M2S1 |  |  |
| --- | --- | --- | --- | --- |
| Model (Training Method) | CORR. | $\Delta$ rf-CNN | CORR. | $\Delta$ rf-CNN |
| rf-CNN(Simul.) | $\mathbf{0.420}$ | $0.0\%$ | $\mathbf{0.496}$ | $0.0\%$ |
| rf+sa-CNN^∗(Simul.) | $0.409$ | $-2.6\%$ | $0.480$ | $-3.2\%$ |
| rf+sa-CNN^∗(Incr.) | $\mathbf{0.421}$ | $+0.6\%$ | $\mathbf{0.493}$ | $-0.3\%$ |
| ff+sa-CNN^∗(Simul.) | $0.416$ | $-0.8\%$ | $0.490$ | $-0.7\%$ |
| ff+sa-CNN^∗(Incr.FC1) | $\mathbf{0.430}$ | $+3.0\%$ | $\mathbf{0.494}$ | $-0.1\%$ |
| ff+sa-CNN^∗(Incr.FC2) | $0.414$ | $-1.3\%$ | $0.488$ | $-1.1\%$ |

Figure: Figure 5: Average peak tuning indices for incrementally and simultaneously trained models. Top row: bar charts for M1S1. Bot row: bar charts for M2S1. Left col: average PTJ values. Right col: average PTs values. Error bars are SEM.
Refer to caption: x7.png

Incremental learning offers a valuable approach for accurately assessing the potential of each spatial integration mechanism in modeling contextual modulation and evaluating their interdependence in generating an effective model. Table 2 highlights several incremental models we tested and their relative improvements. Learning the receptive field first, followed by learning self-attention in rf+SA-CNN^∗(Incr.), outperforms learning both simultaneously in rf+SA-CNN^∗(Simul.) (see second and third rows of Table 2, and middle pairs in Figure 5). This supports our hypothesis that jointly learning the $\alpha$CPB receptive fields and self-attention may overwhelm the system when gradients are constrained to propagate through the center hypercolumn alone. Incremental learning can improve rf+sa-CNN^∗ to match the peak prediction performance of rf-CNN, but not beyond. This suggests that, when used with a CTL readout, self-attention alone is insufficient to fully capture peak tuning.

Table 2 and Figure 5 demonstrate that removing the CTL restriction–i.e., allowing the readout to access information from the hypercolumns in the final convolutional layer via FCL, as in ff+sa-CNN^∗–enables the network to nearly match the performance of the ff+sa-CNN. ff+sa-CNN^∗ is named as such because it closely resembles ff+sa-CNN, with the only difference being the use of a $1\times 1$ kernel instead of a $3\times 3$ kernel in the $\beta$CPBs. These findings suggest that posterior convolution is not required for spatial integration after the self-attention layer when the model is trained incrementally.

A reoccurring observation is that focusing on receptive field information tends to improve overall correlation, while emphasizing contextual information enables the network to better model peak tuning. This pattern is evident when comparing rf-CNN with ff+sa+CNN in Table 1. A similar contrast exists between ff+sa-CNN^∗(Incr.FC1) and ff+sa-CNN^∗(Incr.FC2). In ff+sa-CNN^∗(Incr.FC1), the model inherits the center hypercolumn weights and then learns the surrounding hypercolumn contributions through the FCL. In contrast, ff+sa-CNN^∗(Incr.FC2) learns the weights of all hypercolumns in the FCL simultaneously. While the former excels in correlation, the latter performs better in peak tuning. This suggests that the receptive field is most important towards overall tuning, whereas surround-center interactions are key to capturing peak tuning.

The ff+sa-CNN^∗ models saw an improvement in PTJ and PTS over the rf+sa-CNN^∗ models. This suggests that either the fully connected layer (FCL) plays a critical role in predicting peak responses, or that constraining the readout to the center hypercolumn (CTL) in rf+sa-CNN^∗ limits error propagation to the self-attention block during training. As a result, self-attention may be inadequately learned in these cases, impairing the model’s ability to effectively implement contextual modulation.

### 4.3 Incremental learning emphasizes the contribution of the classical receptive field

Figure: Figure 6: Average FCL decomposition of ff+sa-CNN^∗ when trained differently. The center contribution (green) and the total surround contribution (pink) sum to the prediction tuning curve (orange). Plots are rank ordered with respect to predicted responses. Averages are calculated by plotting rank ordered decomposed tuning curve for each neuron, then averaging across each image. Individual contributions from hypercolumns can be found in Appendix A.5.
Refer to caption: x11.png

A well-established neurophysiological principle is that stimuli within the classical receptive field of V1 neurons are the primary driver of neural responses, while the contextual surround modulates them. We found that when the entire network is trained simultaneously, as in ff+sa-CNN^∗(Simul.), performance is weaker and the network does not follow this principle. However, with incremental learning, the center hypercolumn develops into the dominant contributor, as observed in ff+sa-CNN^∗(Incr.FC1) and ff+sa-CNN^∗(Incr.FC2). Figure 6 illustrates the sum of connection weights from the readout, showing that in ff+sa-CNN^∗(Simul.), weights are evenly distributed, whereas in the incrementally trained models, the center hypercolumn, corresponding to the classical receptive field, is emphasized. Incremental learning fosters this center-surround division and modulation, yielding interpretable performance benefits.

## 5 Discussion

CNNs are widely used and effective models for visual cortical neurons, and they inherently include two mechanisms for contextual modulation: successive convolutions and a fully connected layer, which allow the global image context to be accessible to the readout. In this paper, we demonstrated that augmenting CNN models of cortical neurons with self-attention enhances predictions of both the overall tuning curve and the tuning peak. Self-attention, resembling three-way interactions in probabilistic graphical models, facilitates flexible surround-center modulation via contextual variables (Coen-Cagli et al., ; Fei & Pitkow, ). This provides additional flexibility and complementary benefits to the CNN’s inherent context mechanisms. While large-scale transformer models with multiple attention heads have achieved state-of-the-art performance in modeling mouse V1 neurons by capturing long-range dependencies (Li et al., ), our work explicitly explores the role of self-attention in CNNs for modeling horizontal circuits, highlighting the dependencies and complementary interactions between different mechanisms of contextual modulation.

Several key findings emerged from this work that advance our understanding of cortical computation and neural codes. First, we found that focusing on receptive field information, as in rf-CNN, yields the highest Pearson correlation, alongisde other standard measures (see Appendix A.4), for overall neuronal tuning curves (see Table 1). This suggests that the classical receptive field is the primary driver behind a neuron’s overall response. Our incremental learning experiments further supports the advantage of concentrating on information within the classical receptive field in the center hypercolumn (see Figure 6). Second, we demonstrated that contextual modulation is crucial for a strong and robust peak tuning, with self-attention playing a pivotal role. A trade-off, however, exists between the receptive field and surround modulation: RF-centric models fit overall tuning curves more accurately, while increased contextual surround modulation enhances peak tuning, though often at the expense of overall tuning correlation. Incremental learning, which fosters a strong receptive field bias, may help even out this trade-off. This is consistent with neurophysiological evidence supporting a dominant classical receptive field and weaker surround modulation, with recurrent connections potentially being fine-tuned after receptive field development.

A recent CNN-based model of mouse V1 neurons revealed that the most excitable images often involve stimulus features outside the receptive fields, consistent with the concept of pattern completion (Fu et al., ). Similarly, we found that models capable of capturing peak tuning display interpretable contextual modulation, such as association fields and pattern completion, within the self-attention module (see Appendix A.6). Additionally, incorporating a self-attention layer improved models’ data efficiency (see Appendix A.7). Further theoretical and experimental investigations are needed to characterize and evaluate the interactions facilitated by self-attention, in order to uncover how these mechanisms may be implemented by biological circuits.

## Appendix A Appendix

### A.1 Additional Details on Macaque Experimental Setup

We collected data using a nearly identical experimental protocol as detailed in (Tang et al., ; ), except that our dataset was considerably larger, including up to 34K and 49K stimulus-response pairs for the two monkeys respectively.

During each fixation task, a blank screen was presented for 1500 ms after the monkey established fixation, followed by the presentation of a visual stimulus for 500 ms. 34,000 stimuli were tested in monkey 1 and 49,000 stimuli were tested in monkey 2 over a data collection period of 5 days.Neurons were registered anatomically and also by testing a 200 stimuli finger-print each day. To quantify the neural response, a differential image of GCamp5s calcium signals between the stimulus period and blank period was computed for each trial. The dF/F was then calculated based on a 200 ms to 600 ms window after stimulus onset. We note that GCamp5s is slow, but the signal has been found to be correlated with firing rate (Li et al., ). The training set was collected with 1 repeat for each stimulus. The validation set consists of 1000 stimuli, each with 10 repeats.

### A.2 V1 Preferred Features

Figure: Figure 7: V1 neuron exhibit diversity and complexity in preferred features. Individual CNNs were used to model neurons’ responses to a large set of natural images. Mid row: visualizations of the optimal stimuli for ff-CNN models for 28 neurons. Neurons are clustered into 7 equally sized classes representing preferences for curves, rings (eyes), textures, grating, bars, corners, and other more complex higher order features. Top row: top 5 natural images in the validation set that elicited the largest response from real neurons in each class. Bot row: top 5 artificial stimuli that elicited the largest response from ff-CNN models of neurons in each class. The artificial stimuli preferences are consistent with the optimal stimuli seen in the middle row.
Refer to caption: x14.png

A driving motivation of our study was to identify the natural image features that V1 neurons prefer, which include corners, curvatures, junctions, rings and other higher order features, rather than the traditional orientation and frequency tunings. While traditional artificial stimuli produce a tuning curve, these stimuli rarely represent the neuron’s most preferred stimulus. The studies referenced above show that the neurons’ true preferences are not necessarily a specific orientation or spatial frequency, and may not be well modeled by Gabor functions and traditional linear/nonlinear models. Rather, the preferred features, which are encoded by the peak responses of neurons as visualized in Figure 7, exhibited diversity and complexity in the tuned features. The objective of this study is not to prove or argue for the diversity and complexity of V1 neural codes, but to explore the mechanisms contributing to them. A key finding of this paper is that the surround contextual mechanism, implemented by a non-local network (i.e. self-attention), positively contributes to the generation of the response peak, and is related to the encoding of preferred higher-order features in neurons. As such,this study aims to model the peak response of neurons and develop a novel metric to evaluate the model’s accuracy in capturing the peak of each neuron’s ”natural image tuning curve.”

### A.3 Comparisons to Other Established Models

Figure: Figure 8: Comparing the performance of ff-CNN. (A), (C) shows the Pearson correlation comparisons of ff-CNN with the ”what and where” (shared core factorized) model and the transfer learning (goal driven VGG) model, respectively. (B), (D) show CC norm squared comparisons between models. (E) shows hyperparameter experiments on a subset of 50 neurons from M2S1. (F) Comparison between ff-CNN and the ”what and where” model on the ability to capture M2S1 tuning peaks. All results on M1S1 are similar.
Refer to caption: extracted/5956319/big3.png

Four major classes of models are found in neural response prediction literature: (1) transfer learning models, (2) single CNN feedforward models, the (3) shared core factorized (“what and where”) model (Klindt et al., ; Lurz et al., ), and more recently a (4) transformer based model for mouse V1. The first three types of models have been used in macaque V1, whereas the transformer based model has only been used in mouse V1. We experimented with the first three classes of models and found that for our dataset, the performance of our baseline feedforward models (ff-CNN) are comparable, and at times better, to the transfer learning model and the shared core factorized mode (see Figure 8). Thus, we used the single CNN feedforward model as our baseline model. We did not compare our model with transformer based models such as ViT (Li et al., ), which use deep and complex layers to achieve SOTA neural response prediction. The focus of our research and our contributions are different. We demonstrate that surround contextual modulation is critical in predicting the peak responses of macaque V1 neurons in response to natural images. Towards this, we used self-attention to model horizontal interactions, rather than just using an entire transformer module to achieve SOTA performance.

### A.4 Comparisons to Other Established Metrics

Figure: Figure 9: Pearson correlation versus FEV scatter plot for different models. Average FEV is calculated by averaging over neurons in M2S1. Note that the performance of models improve relative to the ff-CNN baseline when evaluated based on FEV, following a similar trend to that of Pearson correlation.
Refer to caption: extracted/5956319/fevmean.png

Figure: Figure 10: Pearson correlation versus $r^{2}_{er}$ scatter plot for different models. Average $r^{2}_{er}$ is calculated by averaging over neurons in M2S1. Note that the performance of models improve relative to the ff-CNN baseline when evaluated based on $r^{2}_{er}$, following a similar trend to that of Pearson correlation.
Refer to caption: extracted/5956319/r2er.png

We compute measures FEV and $r^{2}_{er}$ for each of our models. We calculated the average FEV and $r^{2}_{er}$ across all neurons in M2S1 for different models. Similar to Pearson correlation and $CC^{2}_{norm}$, FEV and $r^{2}_{er}$ measures are similar across models used in our study, and are comparable to established baselines (see Figure 9 and Figure 10). Despite FEV and $r^{2}_{er}$ taking prediction magnitudes into account, they are still unable to capture the peak tuning properties. This can be attributed to the high degree of sparsity in individual neurons’ tuning curve (to natural stimuli), where only 0.4% of the stimuli above half height on average. Because standard measures of performance are heavily influenced by remaining 99% low-responding stimuli, they are not sufficient for capturing the sharp tuning curves peaks we observed (i.e.recognizing the most preferred stimuli of the neurons).

### A.5 FCL decomposition: average contribution from each hypercolumn

Figure: Figure 11: Average FCL contribution from each hypercolumn for M1S1. Center hypercolumn outlined in red. Each heatmap is independently contrast normalized. Note that the distribution in the first heatmap is evenly distributed compared to the others (see contrast scale).
Refer to caption: x15.png

Figure 6 plots the individual contribution of the center hypercolumn with the sum of all surrounding hypercolumns. We observed an evenly distributed contribution from all hypercolumns in the simultaneously trained model, but a strong center contribution in incrementally learned models. This effect is further observed when we display the average contribution of each hypercolumn (see Fig 11).

### A.6 Models that can capture peak tuning exhibit interpretable contextual modulation such as association fields in the self-attention module

Figure: Figure 12: Attention highlighting for incremental learning models. Top two highest response inducing image are shown for M1S1 neuron 153. Note that rf+sa-CNN^∗(Incr), ff+sa-CNN^∗(Incr.FC1), and ff+sa-CNN^∗(Incr.FC2) all have the same attention map due to the freezing scheme. The center hypercolumn is queried for highlighted images.
Refer to caption: x18.png

Comparing the attention highlighting between self-attention models from the incremental learning experiment, we observe that models learned incrementally have a more focused attention versus equivalent-architecture counterparts trained simultaneously (as shown in Figure 12). Models with strong peak tuning, which incorporate the surround properly, displays association field effects, focusing on similar patterns as present in the receptor field. Furthermore, because of the incremental freezing scheme between all incremental models, they have the same attention despite variations in the readout layer. However, incremental models display a focused attention, meaning the initially trained SA representation using the CTL (in rf+sa-CNN^∗(Incr)) allows for proper learning of attention weights. This further supports that the $30$ to $1$ paramter bottleneck in the CTL is not a limiting factor, and the the gain in performance in the latter incremental models (in ff+sa-CNN^∗(Incr.FC1) and ff+sa-CNN^∗(Incr.FC2)) are associated with the FCL.

### A.7 Self-attention CNNs are data efficient

**Table 3: Average Pearson correlation and peak tuning metrics for models trained at different data sizes of M1S1. Correlation SEM $=0.008$ was consistent across models.**
|  | 25% of M1S1 | 50% of M1S1 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model | $\Delta$ CORR. of ff-CNN | PTJ | PTS | $\Delta$CORR. of ff-CNN | PTJ | PTS |
| ff-CNN | $0.0\%$ | $0.232$ | $1.026$ | $0.0\%$ | $0.762$ | $1.556$ |
| ff+sa-CNN | $+7.2\%$ | $0.927$ | $1.656$ | $+7.6\%$ | $1.026$ | $2.715$ |
| rf-CNN | $+26.9\%$ | $0.000$ | $0.000$ | $+21.1\%$ | $0.000$ | $0.000$ |
| rf+sa-CNN | $+23.6\%$ | $0.000$ | $0.000$ | $+19.8\%$ | $0.000$ | $0.000$ |

We trained baseline feedforward CNN models and their counterparts with self-attention at various training dataset sizes. We conclude that percentage improvements over ff-CNN are furthered at lower data constraints (as shown in Table 3), alluding to the potential efficiency of SA in accumulating surround information compared to other context mechanisms. We note that rf-CNN is the most data efficient when evaluated solely on Pearson correlation. However, it is important to see from Table 3 that at $25\%$ and $50\%$data, rf-CNN and rf+sa-CNN completely fail in the peak tuning index, indicating that the models were entirely unable to model the response magnitude of the highest excitatory images. This lends to our claim that although rf-CNN does well in correlation, contextual information (as is present in ff-CNN and ff+sa-CNN) is necessary to capture peak responses.

### A.8 Population tuning curves

Figure: Figure 13: Population average tuning curves for M1S1.
Refer to caption: x21.png

Differences in peak tuning can also be observed in the population tuning curves (see Figure 13). Average curves are derived by calculating rank ordered tuning curves for each neuron individually, then averaging over the image number across neurons.

### A.9 V mapping parameter γ 𝛾 \gamma italic_γ in self-attention

In the self-attention layer (as shown in Figure 2), the V mapping parameter $\gamma$ allows further factorization of inter-channel mixing and spatial interactions. Toggling $\gamma=$ False removes the transformed value vector, and attention weights instead directly on the input representation. Note that the $\gamma$ parameter does make a difference performance wise. The only difference between rf+sa-CNN^∗ and rf+sa-CNN is the presence of a SA with $\gamma$ = True block in the former and a SA with $\gamma$ = False block in the latter. rf+sa-CNN^∗ has better correlation, PTJ, and PTS values, meaning allowing for the V mapping in SA allows for more flexibility, despite the lack of a $3\times 3$ convolution and FCL layer in these models.

### A.10 CTL channel number bottleneck

In models with a CTL readout, the final layer is performing a $30$ to $1$ or $32$ to $1$ weighted sum, depending on the number of channels. Thus, an issue we considered was that such a narrow final layer would inhibit proper backpropagation of error signals to upstream modules. To address the concern of a $30\rightarrow 1$ mapping in the CTL layer being too tight of a an initial bottleneck, we trained rf+sa-CNN with $c=375$ channels, so that it would have a $375\rightarrow 1$ CTL mapping instead. The results were comparable to the rf+sa-CNN with $c=30$, meaning the drop in performance from rf-CNN cannot be attributed to a parameter bottleneck.

### A.11 Importance of post-self-attention channel mixing

Additionally, we tested self-attention models without post-SA convolutions (i.e. no $\beta$CPB layers) and observed sharp drops in performances compared to baseline CNNs. This suggests that inter-channel mixing is crucial in processing the output of self-attention into a interpretable representation by the readout layer. We note that transformer block in modern computer vision models employ a multi-layer perceptron immediately after self-attention, which aligns with our findings.

To compare the importance of the $3\times 3$ versus $1\times 1$ kernel size and FCL vs CTL readout as a means of incorporating surround information, we compared the following models:
[$\alpha$CPB $\rightarrow$ $\alpha$CPB $\rightarrow$ SA($\gamma=$ True) $\rightarrow$ $\beta$CPB($k=1$)$\rightarrow$ $\beta$CPB($k=1$) $\rightarrow$ FCL] $\overset{\text{vs}}{\iff}$ [$\alpha$CPB $\rightarrow$ $\alpha$CPB $\rightarrow$ SA($\gamma=$ True) $\rightarrow$ $\beta$CPB($k=3$)$\rightarrow$ $\beta$CPB($k=3$) $\rightarrow$ CTL].

The former with $\beta$CPB($k=1$) and FCL outperformed the latter with $\beta$CPB($k=3$) and CTL. Thus, direct access to all spatial features with a fully connected layer is stronger than convolving the surround into the center. We observe that the FCL is the strongest factor for predicting the peak responses, and is bolstered by the addition SA, as ff+sa-CNN outperforms ff-CNN in peak tuning.

### A.12 Code for experiments

The code is hosted at the github repository: https://anonymous.4open.science/r/sacnn/

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 8
presentation: 4
questions: 1) It seems important to add a performance comparison between the self-attention augmented CNNs and CNNs with recurrent layers. Or to explain why not relevant. 

2) Does any of the model capture the basic modulatory natural of extra-classical receptive fields, i.e. surround suppression and no response to an annular stimulus? The analysis of Fig. 6 goes in that direction but it’s not quite the same, if I interpreted it correctly.
soundness: 4
strengths: Excellently well written and executed. The problem is situated in the relevant literature, the motivation and significance of findings are conveyed strongly, the novelty is not overstated and the design focuses sharply on the question being asked. The results are very convincing. A pleasure to read!
summary: The manuscript asks if self-attention can improve predictions of primary visual cortex (V1) response to images. This is motivated by two facts: one is surround modulation data in V1, usually attributed to lateral and top-down feedback connections; and the other is that artificial recurrent architectures predict some aspects of V1 responses better than purely feedforward. Self-attention could augment feedforward networks with spatial context information, beyond or differently from what can be achieved with receptive field expansion due to successive layers and final fully connected layer.

The authors execute a series of well thought-out comparisons between alternative architectures and learning schemes. The main results indicate that self-attention indeed improves predictive power, particularly when models are trained incrementally (learning the receptive field first). And that this improvement is more visible in the ability to capture responses to the most driving images for the neurons, rather than the overall response correlation across all images. This is interpreted as evidence that receptive field computations are more important to capture overall tuning, whereas surround computations are modulatory.
weaknesses: Minor only. 

1) The paper certainly adds to understanding neural system identification. But at a high level, do the result say anything about V1 contextual modulation itself? Or do the authors envision a path to that goal by fitting self-attention augmented CNNs? In any event, I don’t think this detracts from the value of the manuscript.

2) Even though not systems identification, this paper https://pubmed.ncbi.nlm.nih.gov/37738258/ is very relevant and in my opinion should be discussed.

---

confidence: 5
contribution: 4
initial_rating: 6
presentation: 2
questions: Assuming that the limitations detailed above will be corrected, I have many questions about this work:

You use a correlation measure to compare your predictions with neurobiological observations. Can you justify this approach in relation to the observed distribution of neuronal activity? Indeed, it seems that this distribution is sparse and that another measure might be more appropriate.

You have placed the attention layer after a features layer that is supposed to represent simulated activity in a neuron in the primary visual area. As such, can you interpret the various Q, K, and V features you've inferred during learning?

It seems you're trying to predict activity on a per-frame basis, but neural activity is dynamic. Do you think you can apply this type of model, and in particular the attentional layer, to the temporal domain?

Finally, you mention a homology between the neural network that learns to predict neural activity and the network of biological neurons underlying the production of this activity. This major point needs to be better introduced.
soundness: 3
strengths: The strength of this paper lies in the rigorous introduction of this model and the comparison of different architectures, enabling the reader to disentangle the role of different network components. This method is validated by computational simulations that quantify the correlation between observed and predicted activity. This methodology then enables predictions to be made that confirm or refute certain biological theories, such as the modulation of normal activity according to the context of neighboring neurons.
summary: This paper presents a method for predicting the neural activity of neurons in the primary visual cortex of primates upon presentation of natural images. This method is based on the use of a hierarchical neural network comprising several convolutional layers and learned in a supervised manner with the aim of predicting neuronal activity. An innovation of this algorithm lies in the inclusion of a self-attention mechanism in the processing sequence. This attention mechanism could then, on the one hand, help improve prediction capabilities and, on the other, help understand the modulation mechanisms present in a biological neural network which consist in facilitating or depressing the response of neurons according to a context.
weaknesses: I've noted a number of weaknesses in the document that affect its comprehensibility and potential impact. Foremost, the presentation needs to be simplified. First, it is difficult to understand from the abstract that your aim is to predict neural activity. In terms of the structure of the paper, there are inconsistencies in the sequence of sections, making section 3.5 and the model evaluation seem to conclude the paper. Some parts of the methods imply knowledge of intermediate results, and a major restructuring is needed.

Secondly, the model is missing some essential elements for a proper understanding. If you describe it well, the input image given to the network is not the one you want to produce. From the figure, I deduce that it's the fluorescence contrast of the image, but this needs to be written more explicitly.

Thirdly, the figures are not always clear: figure four, for example, neither shows nor explains what is shown here. I'm guessing that the images are sorted according to their response, and that you superimpose the prediction that is made. Figure five shows results, but with fonts too small to read. This figure can also be simplified, as many elements seem redundant. Finally, the results discussed in the discussion are interesting, as they allow biological facts to be deduced from the study of a neural network, and in particular from the inclusion of an attention layer. However, this conclusion may also be the consequence of the limitations of the various models. To be validated, the conclusion should be causally related to the mechanisms described. In particular, I'm surprised that you don't look more closely at the activity in the attention layer: what are the strength and distance of the different modulations? This would enable you to draw neurological conclusions.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: * There is no clear description of the training objective for the different models compared. I assume it to be a regression of single-neuron activation from each training image, using an MSE loss, but this should be described much more explicitly.
* The notions of “overall tuning” and “tuning peak” are used in the abstract and introduction, but these are not standard terms, and cannot be understood without some amount of explanation (e.g. “the tuning peak corresponds to the top 1% of highest responses”).
* The Zhang (2022) reference does not include information about a journal, preprint server or conference venue.
soundness: 2
strengths: * Fitting neural network models to brain data is an important approach to determine the biological plausibility of artificial systems, and to understand computational brain mechanisms
* The methods are generally sound and the compared models and ablations allow us to draw conclusions about the investigated mechanisms
summary: The paper demonstrates that a self-attention layer inserted in a convolutional network can improve its ability to predict neural responses to natural stimuli from macaque area V1. The neural responses were collected with two-photon imaging in two separate monkeys. Different variants of a feed-forward CNN are compared (with or without self-attention, with or without fully-connected spatial integration, with end-to-end learning or incremental pretraining and fine-tuning of specific layers). The authors conclude that self-attention can improve neural prediction accuracy, especially for explaining “peak tuning” (i.e., the top 1% of highest responses).
weaknesses: * The paper does not clearly state its primary objective, and the reader is left to choose between two options. The main objective could be to prove that recurrent and horizontal interactions help explain extra-classical receptive field properties in V1, with self-attention as one specific way to implement these interactions. Or, it could be to argue that self-attention (as implemented in modern neural networks with Q, K and V projections) is a good model for recurrent/horizontal interactions in area V1. I believe the former interpretation is better supported by the data, although less novel (since earlier papers have already reported that recurrent layers can improve neural predictions). However, I fear that the community would more readily jump to the latter conclusion, which is unfortunately not well supported. To make this interpretation, the self-attention models should be directly contrasted with other baselines including one (or more) recurrent layer(s) that implement horizontal interactions without self-attention. In the absence of these baselines, only the first conclusion can be supported, but this hinders the novelty of the paper’s message. In any event, the authors should clearly state which of these two goals they are targeting.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Just a minor question: In section 3.4, I read that the loss is MSE. Many (if not most) papers in the field use a Poisson NLL loss, even with two-photon data (where, I agree that assumptions of e.g., electrophysiological recordings and Poisson counting aren't met). Could you elaborate on the use of MSE? Have you tried both and found better results with MSE?

Overall, I'm happy to maintain the assigned score (I believe the paper is more of a 7 in its current state, but 7 is not among the allowed ratings) if a more extensive comparison with Li et al. (2023) is provided.
soundness: 3
strengths: The paper is overall well written and the results are quite convincing. Here are the points I believe are strengthening the submission:

* The paper introduces a new perspective on modeling contextual modulation in V1 neurons using self-attention mechanisms, which has not been extensively explored before.

* The authors systematically investigate the contributions of different contextual modulation mechanisms (convolutions, self-attention, and fully connected layers) and their interactions, providing insights into their roles in modeling neural responses.

* The incremental learning approach, where receptive fields are learned before contextual modulation, is inspired by the developmental process in the visual cortex, adding biological plausibility to the model.

* New evaluation metric: the peak tuning index is introduced to assess a model's ability to capture a neuron's feature preference, addressing the limitations of standard metrics like Pearson correlation.

* The authors demonstrate that models with strong peak tuning display interpretable contextual modulation effects, such as association fields, in the self-attention module. I actually believe that this last point should be included in the main paper instead of the Appendix, as it brings added value by bridging previous research at the interface of neuroscience, perception, and machine learning.
summary: This paper explores the role of self-attention mechanisms in modeling contextual modulation in primary visual cortex (V1) neurons of macaque monkeys. Key findings and contributions include:

* Adding a self-attention layer to convolutional neural networks (CNNs) improves neural response prediction of macaque V1 neurons in terms of overall tuning curve correlation and peak tuning (PT metrics). This suggests self-attention provides additional flexibility and benefits complementary to CNNs' inherent contextual modulation mechanisms (successive convolutions and fully connected layers).

* The classical receptive field is the primary driver of a neuron's overall response, as models focusing on receptive field information achieve the highest correlation. However, contextual modulation, especially via self-attention, is crucial for strong and robust peak tuning.

* Incremental learning, where the receptive fields are learned first followed by self-attention and fully connected layers, allows the model to properly learn the contributions of the receptive field and contextual modulation. This incremental approach leads to a receptive field-centric model more aligned with neurophysiological evidence.

* Interpretable contextual modulation effects, such as association fields, emerge in the self-attention module of models that capture peak tuning well.

* Self-attention augmented CNNs are more data-efficient compared to baseline CNNs.

Overall, the paper provides insights into the computational role of self-attention (single head) in modeling horizontal connectivity and contextual modulation in the visual cortex, and proposes an incremental learning approach to best train networks with self-attention.
weaknesses: **Major Weaknesses**

* While the paper compares the proposed models to established CNN architectures, it does not include comparisons with more recent state-of-the-art models, such as transformer-based models, which have shown promising results in modeling mouse V1 neurons. More precisely, while Li et al. (2023) is mentioned in the introduction, no comparison with this model is provided in the paper. Although it's true that multiple attention heads may intuitively seem unnecessarily complex for the task outlined in the paper, Li et al. (2023) demonstrates very good results in predicting neural responses in mouse V1 neurons.

**Minor Details**

* Figure 1(b) is not readable: The axes labels, ticks, and different images are unclear. Consider resizing or reorganizing the figure differently.

Meta-review: This paper describes deep learning model for predicting neural activity in primary visual cortex that incorporates convolutional layers as well as self-attention. It shows that self-attention can indeed improve performance relative to feed-forward neural networks, particularly when models are trained incrementally. This represents is a novel approach to modeling V1 data, and offers the potential to help understand nonlinear information processing in real neural circuits. The reviewers praised the paper for its novelty, clarity, motivation, and significance, and I'm pleased to report that it has been accepted to ICLR.  Congratulations! Please revise the manuscript to address all reviewer comments and suggestions, especially the points about figure clarity and legibility.  (I also agree with the reviewers that it would be nice to see a comparison to the Li et al. (2023), if possible).

Decision: Accept (Poster)