Flexible Context-Driven Sensory Processing in Dynamical Vision Models
Authors: Lakshmi Narasimhan Govindarajan
Abhiram Iyer
Valmiki Kothare
Ila R Fiete
Venue: NeurIPS 2024 poster (2024)
Keywords: convolutional RNNs
feedback
top-down modulation
neural dynamics
psychophysics
Primary area: neuroscience_and_cognitive_science

Abstract: Visual representations become progressively more abstract along the cortical hierarchy. These abstract representations define notions like objects and shapes, but at the cost of spatial specificity. By contrast, low-level regions represent spatially local but simple input features. How do spatially non-specific representations of abstract concepts in high-level areas flexibly modulate the low-level sensory representations in appropriate ways to guide context-driven and goal-directed behaviors across a range of tasks? We build a biologically motivated and trainable neural network model of dynamics in the visual pathway, incorporating local, lateral, and feedforward synaptic connections, excitatory and inhibitory neurons, and long-range top-down inputs conceptualized as low-rank modulations of the input-driven sensory responses by high-level areas. We study this ${\bf D}$ynamical ${\bf C}$ortical ${\bf net}$work ($DCnet$) in a visual cue-delay-search task and show that the model uses its own cue representations to adaptively modulate its perceptual responses to solve the task, outperforming state-of-the-art DNN vision and LLM models. The model's population states over time shed light on the nature of contextual modulatory dynamics, generating predictions for experiments. We fine-tune the same model on classic psychophysics attention tasks, and find that the model closely replicates known reaction time results. This work represents a promising new foundation for understanding and making predictions about perturbations to visual processing in the brain.

Flexible Context-Driven Sensory Processing in Dynamical Vision Models
Lakshmi Narasimhan Govindarajan 1.344 Abhiram Tyer 2, 3,4 Valmiki Kothare', and Ila 3.4
'Department of Brain and Cognitive Sciences, MIT, Cambridge, MA *Department of Electrical Engineering and Computer Science, MIT, Cambridge, MA "McGovem Institute for Brain Research, MIT, Cambridge, MA K. Lisa Yang Integrative Computational Neuroscience (ICoN), MIT, Cambridge, MA, (lakshmin, abiyer, valmiki, Falmiki,fietti fieteelmit..eee
Abstract
Visual representations become progressively more abstract along the cortical hier- archy. These abstract representations define notions like objects and shapes, but at the cost of spatial specificity. By contrast, low-level regions represent spatially local but simple input features. How do spatially non-specific representations abstract concepts in high-level areas flexibly modulate the low-level sensory repre- sentations in appropriate ways to guide context-driven and goal-directed behaviors across range of tasks? We build biologically motivated and trainable neural network model of dynamics in the visual pathway, incorporating local, lateral, and feedforward synaptic connections, excitatory and inhibitory neurons, and long- range top-down inputs conceptualized as low-rank modulations of the input-driven sensory responses by high-level areas. We study this Dynamical Cortical network (DCner) in a visual cue-delay-searde task and show that the model uses its own cue representations
adaptively modulate its perceptual responses to solve the task, outperforming state-of tifte-of-th-att the-art art DNN vision and LLM models. The model's 's population states over time shed light on the nature of contextual modulatory dy- namics, generating predictions for experiments. We fine-tune the same model on classic psychophysics attention tasks, and find that the model closely replicates known reaction time results. This work represents promising new foundation for understanding and making predictions about perturbations to visual processing the brain.
Introduction
We readily use abstract cues to modulate our sensory perception. These include cues to attend to abstract high-level features (find Waldo; count the number of hoop shots, etc.) or low-level features (find the red items, vertically oriented bars, etc.). Such cue-based modulations of the visual pathway allow us locate items of interest more rapidly and accurately, and to perform goal-directed computations.
Understanding how top-down modulatory cues are represented and then interact with bottom-up sensory-drive neural responses has been longstanding goal in computational cognitive neuro- science 114. Extensive psychophysical experiments [5] and studies showing that individual neurons whose receptive fields are aligned with the attentional cue exhibit gain modulation 6 shed light
"Denotes equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

on the phenomenon. However, fundamental circuit-level and conceptual problem remains open: what is the nature of the "modulatory homunculus" that knows, given various goals and context cues, which low-level representations to modulate, in which combination and which topographic part of the representational space in different processing layers? We lack a cohesive computational framework to link these two levels of representation. Simultaneously, we lack models of sensory processing that fully take into account the recurrent and temporally unfolding nature of computation in the brain; thus, they fall short of explaining phenomena like reaction time variations with task difficulty and the sharpening of perception as information is integrated within a trial.
We combine known architectures of visual cortex with advances in machine learning to introduce a biophysically-inas -nnspire model, the Dynamical Cortical network (DCner). to solve cue-delay-visur search tasks (Figure. D. The model is endowed with several relevant properties of biological circuits, including separate (tuned) excitatory and (weakly-tuned) inhibitory populations, lateral inhibition (intra-area recurrence), and neuron types with distinct learnable time constants. We operationalize the "modulatory homunculus" as multiplicative low-rank perturbations from higher-order cortical and thalamic areas. Specifically, these low-rank perturbations arise from the model's own sensory responses from earlier times within the trial.
Contributions. In this work, we focus on analyzing and interpreting the internal dynamics and behavioral modes of our biophysically inspired model on suite of visual cue-mediated tasks.
We introduce vis-count, novel, challenging, and parametrically generated cue-delay-visute visuull search task. A visual cue (consisting of a color, a shape, or a color-shape conjunction) specifies which objects in a subsequently presented scene of geometric objects to count.
We present a biologically realistic model of the brain's visual system, the Dynamical Cortical network (DCner), that is relatively shallow with local and top-down feedback and separate E/ neurons, which capable of top-down attentional modulation based on previously presented cues and processes information over time. Our framework is among the first to link levels of analyses from physiology to behavior via computations and is a necessary first step toward hypothesis generation for neuroscience.
Our model outperforms state-of-the- -ft-----ttt-ttttt- standard DNNs and LLMs on the task, while being inter- pretable and having orders of magnitude fewer parameters.
We perform in silico electrophysiology of the circuit's ' population dynamics to show that cue-based modulation drives a divergence of the bottom-up responses from the uncued case.
We can fine-tune the same model new stimulus sets corresponding to classic human psy- chophysics attention tasks. Reaction time analogues in our model closely replicate experimental observations.
In sum, these contributions suggest that our approach is promising framework for modeling the brain's visual processing dynamics, one that replicates many key attentional phenomena and that can generate testable hypotheses and predictions about circuit mechanisms.
2 Related Work
Stimulus computable models of visual cognition. Modeling top-down contextual effects on bottom-up sensory processing during visual search is of longstanding interest in the vision sciences community. There are large number of phenomenological models of visual search and associated reaction time findings 111 but only recently have models been able to operate directly on high dimensional sensory inputs [3) The most promising approach involves augmenting pre-attentive ventral stream models with controllers either via attention maps (13] or multiplicative modulation factors on network activities [7]. However, these models cannot be used to faithfully study sensory processing dynamics because they are purely feedforward. Recurrent models of early sensory processing with lateral feedback and distinct excitatory and inhibitory populations [8] 20),, and top-down feedback (2I) have shown promise in accounting for contextual visual computations and human reaction times on visual cognitive tasks. However, to our knowledge, stimulus-computable recurrent vision models compatible with cue-delay-targei paradigms do not exist. More generally, models have tended to either be stimulus-com cimpuus-computable or grounded biological realism, but usually not both.

---

Model readout
Low-rank modulation
Weakly Tuned Inhibition
Tuned excitation
Higher-order
layer
V2
Bottom-up inputs
Intra-area feedback
Modulatory inputs
Figure 1: Low-rank modulations to drive context-aware processing, We present a biologically motivated, end-to-end trainable network model of dynamics in the visual pathway. Layers in the model are parameterized by recurrent (tuned) Excitatory (E) and (weakly tuned) Inhibitory (I) neural populations that interact bidirectionally with a higher-order layer in a low-rank manner. The low-rank modulatory factors serve to extract abstract context cues from sensory responses for subsequently driving neural dynamics into context-appropriate dynamical regimes.
Low-rank interactions. Neuronal networks in the brain can be involved disparate computations simuitaneously [22]. Computational neuroscience research has focused understanding how such neural networks can represent task-specific information and, relatedly, its consequence on the system's overall behavior. Theoretical results suggest that the coexistence of structured low-rank connectivity and random connectivity in networks can enable multi-task computations and expand the dynamic range of the network's 's functional capability (7] An alternate but non-orthogonal viewpoint is that low-rank control inputs from an external system can switch a network's 's input-output mappings doneext-dependently [28 30). Most of the computational work in this regard is, however, rooted in the motor domain, with Schmid and Neumann being a recent exception.
Neurophysiology of attentional control. Biological attention upmodulates task-relevant information by filtering sensory responses using "templates". Empirical work shows that attentional templates are primarily represented in higher order cortical areas 3 and that such templates can be learned rapidly on per-task basis [35]. Morcover, the nature of such attentional filtering is known to be cell-type and layer-specific [36], with theories emphasizing the role of top-down mediation of specific GABAergic interneurons B77. In addition to cortical feedback inputs, higher-ordse thalamic nuclei
also known convey contextual inputs to sensory regions either via direct projections [38 33 indirectly through the frontal cortices [40 41]. Building biological sensory processing models that account for all these disparate findings is of primary importance the Neuroscience community.
3 General methods
3.1 DCnet Model and training details
DCnet comprises two core components: biologically motivated sensory perception stream and higher-ordes area that interacts bidirectionally with it (Figure. [ Each of the four sensory areas our model are organized retinotopically as hypercolumns consisting of distinct excitatory and inhibitory neural subpopulations that obey Dale's law [42]. Excitatory pyramidal neurons receive bottom-up and recurrent lateral excitatory inputs as well as short-range lateral inhibitory inputs from interneurons in the same area. Interneurons receive bottom-up and lateral excitatory inputs. Finally, pyramidal neurons project in a feedforward manner to their downstream area. The ratio between excitatory and inhibitory neurons is 4 as observed in cortical areas 43

---

b
Ours
Convolutional tesplitti Cimpltett Resffen- uimption) VIT 015 (implcct)
How many objects in the presented scene obey the visual property depicted in the cue?
1.0
0.99
0.95
1.0
cotor
Answer
tnai
Test Accuracy
0.23
0.75
0.5
tial
0.25
Answar
traal
0.0
color trials
shape
conjunction trials
novel conjunction cues
Cue phase
Scene phase
Decision
trials
time
Figure 2: Explicit context-guided modulation of sensory dynamics is necessary for learning generalizable solutions. a. We introduce vis-count, a parametric, visually-cued, delayed search task. On each trial, models are cued with a visual attribute (either a color, shape, conjunction of the two), and after a delay, a scene is presented. The task is to count and report the number of cue-consistent objects the scene. b. On each of the three types of trails, consistently outperforms state-of-te saate-of-the-an art standard DNNs (Section. 3.2 Baselines) on novel held-out scenes. Implicit models refer to the condition where cues and scenes are presented simultaneously. The red dashed line denotes chance performance. c. A harder test of generalization novel scenes and cues reveal that our model is robust to such variations, unlike performant implicit models.
find that our model
The higher-order layer receives pyramidal input from each area and, in turn, modulates inter-area projections in low-rank manner operationalized as follows.
r1: r2 = <(h) WT + br: + b2
# Compute the linear projections
m ={(h:) [r & r2]
# Compute the modulating factors
1/+ = h m
Execute the modulation
Here, hy E RCXHxW is the excitatory population readout of area E [1..4] at time I; E(-) the spatial average pooling operator; @ denotes outer product and @ denotes pointwise scaling.
Neurons have learnable cell-type specific time constants. We dispense of traditional ML operations such as BatchNorm or LayerNorm, designed to impart stability during training. Our model has 1.8M parameters that we learn via gradient descent on task-performance objective. full mathematical specification of the model is provided in Appendix. A.1 A.1
3.2 Baselines
We instantiate baseline models of varieties. First, consider "Convolutional RNN" model (~ 8.8M params) that uses traditional 6-Layer convolutional backbone feeding into a gated recurrent unit (GRU) 42 with N 2048 neurons (Appendix, BB. We construct this baseline evaluate the benefits of an explicit modulatory mechanism such as the higher-order layer in our model. Second, we consider four standard deep feedforward neural network architectures. As these models do not operate on spatiotemporal inputs, we condense trials to single time point by stacking cues and scenes together. Given the lack of explicit cue followed by scene phase for these models, term them implicit. Cues and scenes are presented at the same time. These baselines provide an upper bound on the expected performance of DCnet and verify that our chosen task
con-tiiiial computational challenge. In our experiments, we consider the following implicit models: a 6- 6-Layer CNN (~ 2.8M params); ResNet- (45) 11.5M params); VIT-B/16 [46] (~ 86M params), vision transformer with a patch size of 16px; Swin-T 47 28M params), hierarchical vision
transformer. Furthermore, we also ran experiments on the zero-shot generalization abilities of an LLM (Appendix.) C

---

a
b
Target
Target
0 2 4
01234 4
t-0
Cued
Uncued
Time
Time
Cue
Cue
-mm Ja
Target
Target
01 2 3 3
Cued
Time
Time
Uncued
Cue
Cue
-2-- K0.0 50
Figure 3: Neural network dynamics reveal context-dependert behavior. We perform dimensional- ity reduction on late layer model activities and visualize neural trajectories (Gray for individual trials; Dark-colored for trial averages) for different experimental manipulations. The solid dot indicates the trial start. a. For a fixed cue (color trials visualized here), network dynamics reflect the extraction and preservation of task-relevant information (object counts) while being invariant to task-irrelevant bottom-up responses from the different scenes. b. Matched cued vs. uncued trials for the same scene (inset) reveal divergence of the bottom-up responses driven by the low-rank modulations.
4 vis-count: A parametric cued visual search task
4.1 Task and stimuli
We draw inspiration from Clevr 48
synthetic dataset for language-mediate visual reasoning and construct vis-count, a parametric visually-cued delayed search task. On each trial, we challenge models count and report the number of geometric objects in a visual scene with features consistent with cue provided earlier (Figure. 2 Cues can be simple colors, geometric shapes, a conjunction of the two. Scenes comprised 10 geometric 3D shapes of varying sizes, colors, and material properties. By construction, there can be 0-5 cue-cuusee tee-coniittent objects in the scene. We counterbalanced dataset so that distribution target counts was uniform across trial types. Cues and scenes were rendered at 320 x 240px and resized to 128 128px for model training and evaluation. In total, our training (validation) dataset comprised of 384K (38K) trials. As with Clevr, we detect and discard scenes with fully occluded objects.
For DCnet and the Convolutional RNN bascline, cues and scenes are visual inputs provided to Layer 1 of these models. Cues are persistently provided to the network for the first T discrete time steps followed by scenes for the next T time steps. The output activities the last layer the last time step (t 27)) are transformed into logits, and a supervision signal is provided via a cross-entis entsrpy loss, with ground truth labels counted from 0 to For the "implicit" baseline models, cues and scenes are slucked together and presented simultaneously.
4.2 Results
We report the results of our model evaluations and comparisons to baselines in Figure. 2p-c. Our model consistently and significantly outperforms state-of-the-art DNN vision models when evaluated on trials with novel held-out scenes, achieving 99%, 95%, 73% accuracy the color, shape, and conjunction trials, respectively. Additionally, we perform harder generalization test by synthesizing trials with novel cues and novel scenes (c.g., test model on blue and green colors, cube
5

---

A
B
C
D
E
Figure 4: A selectivity gradient emerges across the cortical layers for the different visual cues. We sequentially lesion the modulatory synapses in trained DCnet from the higher-order layer to each sensory area, starting from the top-most (D) to the bottom-most (A) sensory area, while documenting task performance on each trial type. (E) indicates the intact DCner. While late lesions have minimal effect on performance in color trials, the opposite is true for shape trials. Early areas in the DCnet exhibit color selectivity, while later areas exhibit shape selectivity.
1.0
Model Accuracy
Color
0.75
(Early selectivity)
0.5
Conjunctions
Shape
(Late selectivity)
0.25
Random chance
0.0
A
B
C
D
E
shapes, and conjunctions thereof, all unseen in the training set). Our model demonstrated the best generalization performance (55%) while the performant baseline models (with orders of magnitude more parameters) dropped to chance (16.67%). Sample model results are visualized in Appendix. A.S
Model errors, too, can be particularly enlightening. Consider the last example in Figure. A.5 When cued find "blue cylinders", the model mistakenly appears include either the cyan cylinder the occluded blue cube its count, either of which would reflect an illusory conjunction [49]. Illusory conjunctions arise due to failures of crature-based attention and have been extensively studied in the human cognition literature. The presence of such pathologies in our model exposes an opportunity to gain potential insights into the neural implementations of feature-based attention.
In the following sections, we report analyses of our model's 's learned dynamics and internal represen- tations that support contextual guidance.
What modulate? Low-rank structures support optimal cue representations. daily life, "cues" are not specified to as abstract rules but rather as high-dimensional sensory inputs. How do we construct a representational space where abstract rules (derived from sensory responses) and the sensory responses themselves are distinct yet coexist? Following insights from electrophysiology [29] and prior theoretical work (3 28 we hypothesized that learning to inject derived cue information back into sensory representations as low-rank perturbations will aid in optimally partitioning the sensory representational space.
DCner's task performance indicated that it had indeed learned task task-optimal representations. Here, we perform dimensionality reduction the activities of pyramidal neurons in the last layer of DCner to probe how this optimality emerges.
First, observe that cued uncued dynamics individual trials become progressively divergent and nearly orthogonal with time (Figure. B3). This indicates that the bidirectional interactions likely promote the formation of low-dimensional activity subspaces embedded within the higher- dimensional ambient activity space. Second, we see that the bottom-up responses to the same set of scenes are differentially modulated appropriate target subspaces based the cue (Figure. (pp) We believe that this happens through the inactivation of context-irrelevan subspaces, resulting in invariance current task-ine dask-irrelevant features. As additional consideration, note that training DCnet without this top-down feedback mechanism brings performance down to 38% highlighting the importance of this interaction in our framework.
Where to modulate? cortical gradient for feature selectivity. After extracting abstract context rules from sensory responses, question remains: At what level of the representational hierarchy must the perturbation be applied? We perform lesion analysis on the trained DCnet to probe this question (Figure. 4..
We sequentially "turn off" modulations, area by area, and observe their detrimental effects on overall function as determined by task performance. We implement this by setting the modulating factors to

---

a
DCnet
DCnet (trineed without inhibition)
b
Early Layer
Number of Neurons Number of Noushs
Number Nevorr
DCver Orained wi innitoonn
DCnet
Number of Nouroos
Late Layer
com
higher
higherr
exoressivity
Time 125
Ausccomeiatinn
Dynamic Range
Figure 5: In silico electrophysiology sheds light on tuning properties and excitation-nnhibition dynamics. We drive network activity in DCnef with uncorrelated, time-varying Gaussian noise inputs. (a) We depict randomly chosen neuron in early and late sensory areas of DCnet. Each E/I neuron pair shown here has matched receptive fields. Learned time constants reflect fast/slow integration early/late areas, revealing a macroscopic gradient. By contrasting DCner trained with and without inhibition, we find that inhibition plays a crucial role in (b) imparting stability and (c) expanding the dynamic range of computations in the network. More quantitative details are presented in Appendix. A.4
1. We hypothesized that if the modulation of pyramidal neuron activity in a given area is essential for the system's function, then lesioning this perturbation will result a maximal drop in performance. Interestingly, this analysis yielded differential results per trial type (Figure. 4 For the color trials, we find that lesioning modulations in the early areas had the largest impact on performance, while for the shape trials, it was the late modulations that proved critical. In contrast, modulation strength seemed relatively uniform across the sensory areas.
Taken together, these results offer two insights. First, opposing cortical gradients for color and shape selectivity emerge and coexist in the sensory areas despite the model only being supervised "count". Second, leveraging this selectivity gradient, the higher-order area learns apply appropriate low-rank modulations. We believe that the low-rank nature the context- based modulation, not only within but also across areas, is fundamental to generalization.
Exeitation-Inhibttio Dynamics. The amplification of context-relevant sensory representations must be balanced to support stable dynamics [50]. The neural underpinnings of this balance and its compu- tational role in feature gain modulation have previously only been studied phenomenologically Here, we leverage DCnet to investigate how the circuit-level properties discovered through optimization can support overall network function.
We probe the cell-type specific neural dynamics in DCnel by driving network activity with uncorre- lated, time-varying Gaussian noise inputs. First, despite fewer interneurons the model, inhibitory interactions play crucial role in imparting stability (Figure. (5)) and expanding the dynamic range of computations expressed by the network when compared inhibition (Figure. (5)). Second, we detect the presence of co-tuned excitation and inhibition (Ap- pendix. A.44. Empirically, co-tuning is known be a common organizational principle across the sensory areas. These findings imply the critical role of interneurons in cortical amplification dynamies underlying context-des deepndeet computations. Finally, we observe that neurons learn to integrate information faster in early vs. late areas, revealing the emergence of a macroscopic gradient neural timescales (Figure. 5p.
version of DCner trained without
Model psychophysics on parametric cued feature searches
feature search a variant the general visual search problem in which "target" is defined by a single discriminative feature (5). Parametric variations along (or orthogonal to) this discriminative feature axis help shed light on the mechanisms of top-down contextual guidance and its impact on behavior. Here, we consider two feature search tasks from the human psychophysics literature.
7

---

Task
b
specified target present the scene (YES/NO)
Low ditractor heterogeneity High heterogeneity
0.15
Target
Mean Model Entropy
Scene
77
7
74
17
4
74
Target present trials
Target absent trials
Heterogeneity distractors in task-irrelevant dimension
Figure 6: Slower reaction times/larger output entropy with more heterogeneous distractors. a. Task description. b. We parametrically vary the heterogeneity in distractor orientation (similar to (5)). In both target-present and target-absent trials, we find that the DCner's s output entropy is significantly different between low distractor heterogeneity vs. high distractor heterogeneity trials. Error bars represent the standard error of the mean.
General methods. To perform model psychophysics, we start from the DCnet model trained on vis-count. We replace its readout do binary classification (target present/absent) and fine-tune the model on each task below. Furthermore, to derive a measure of model "reaction time," we compute the entropy of DCnet's decision outputs after evolving dynamics for the duration of each trial.
5.1 The role distractor heterogeneity
Task and stimuli. A target feature's bottom-up salience (pop-out) does not survive variations in irrelevant distractor features. (Figure. (p) [5]. To test this, construct cued-feature search trails where models search for an L/T (target) in grid with fixed number of Ts/Ls respectively (scene). Targets and scenes rendered and oriented uniformly at random between [0, 2T).. The distractor heterogeneity level (w) for every trial was uniform random between 00, 1]. Consequently, distractor orientations in the scene were uniform random between [ - w + w 2 with 0 [0, 2x)) being the mean distractor orientation. The target present in 50% of the trails. Our training (test) dataset comprised of 32K (8K) trials.
128 128px canvas. Targets (L/T) presented at the center
Results. When fine tuned on this task, DCnet achieved an overall accuracy of 97% on the test trials. Moreover, faithful to the human psychophysics results, find that the entropy of DCnet outputs significantly different for low (w <= 0.5) vs. high (w > 0.5) distractor heterogeneity for both target-present (Mann-Whine 301376., p < .001). While target-absent trials yield higher reaction times in humans when compared to surget-present trials, don't see this in model's output entropy. This possibly reflects an imperfect choice for model reaction time measure, an aspect of our work that we look to extend upon in future work.
U = 302947., p .001) and target-absent trials Mannn-Whtnee
5.2
The role target-distractor feature differences in the presence of distractors
Task and stimuli. In a classic critical color difference task, human participants were required to detect target that diffeered from distracting stimuli only in color [51]. Search times were measured for varying color differences as a function of display density (the number of distractors). We parametrically synthesize stimuli to recreate this task (Figure. bb).).
Target and distractor stimuli were chosen be circular discs of radii 10px. Cues were rendered at the center of a 128 x 128px canvas at a target color chosen at random from among four perceptually uniform color spaces. Search scenes consisted of 1 distractor stimuli whose color differed from the target color at one of 10 preset target-distractor difference levels chosen uniformly at random. The target was present in 50% of the trails. In total, our training (test) dataset comprised of 32K (8K) trials.

---

the specified target present in the scene (YES/NO)
torges High targes-listractrr feature difference
Target
Scene
distractooo teature difference
Mean Medel Entropy
re0.35
Moan Model Entropy
YES
0..0.1
Target-distractor feature difference
Number of distractors
Number of distractors
Figure 7: Differential reaction times/output entropy predictions for varying levels of target- distractor differences. a. Task description. b. We parametrically vary both the number of distracting stimuli in the scene and the target-distractor color difference [5]). c. DCner's output entropy is invariant (linear) to the number of distracting stimuli when the feature difference is high (low), capturing the psychophysical phenomenon of "popout." d. Model entropy as function of target- distractor feature difference when marginalized over the number of distracting stimuli. Error bars represent the standard error of the mean.
Results. When fine tuned on this task, DCnet achieved overall accuracy of 95% the test trials. Moreover, DCnet recapitulates two key findings from the psychophysics literature.
We probe DCner's output entropy (a proxy for model RT) function of the number of distractors in the scene. We find that for trials in which the target-distractor color difference was high, model entropy was invariant the number of distracting stimuli (Pearson's the target-distractor color difference was low, model entropy increased linearly with the number of distracting stimuli in the scene (Pearson's DCner's 's output entropy as function of the target-distractor feature difference. These results faithfully
-0.11). On trials where
0.55). We find similar result when we probe
replicate behavioral effects reported in prior literature (5] (1 52
As we alluded to in Section 5.T] our reaction time (RT) metric is one among many possible choices. To take step further, implement and test another RT metric inspired by evidential learning (EDL) theory as proposed in [20). We verify that our primary conclusions from this experiment hold even in the context of this new metric. Finetuned on this task. Model RTs, computed as a time-averaged uncertainty measure, increased linearly with the number of distractors in the low target-distractor color difference trials (Pearson's 's = 0.92) and was invariant to the number of distractors on high target-distractor color difference trials (Pearson's 0.2).
the EDL objective. DCnet achieves 86% accuracy
Discussion
The advent highly-performant ventral stream models visual perception is rapidly revolutionizing visual neuroscience research. Stimulus computable models operating directly high dimensional sensory inputs yield benefits as hypothesis generators for scientific discovery. (53 However, there exist two fundamental axes of dissonance, which are potential rate limitors. First, the emphasis on building "bigger and better" models promotes divergence from biological realism [56]. Second, it is evident that the extent of biological visual capabilities spans space bigger than that entails only feedforward perceptual modules 5 58). calls for accounting for the cooperative dynamics between perceptual and cognitive processes [59]
In this work, we aim to bridge this gap by introducing a computational framework that emphasizes biological realism and conceptualizes interactions between perceptual dynamics and abstract cognitive demands while being scalable and compuptlle We present the Dynamical Cortical network (DCnet): trainable neural network model of visual dynamics incorporating local, lateral, and feedforward synaptic connections, excitatory and inhibitory neurons, and long-range top-down inputs conceptualized as low-rank modulations of the input-driven sensory responses by high-level areas.

---

We start by studying the ubility and behavior of DCner to operate in a visually-cued search paradigm. DCnet not only outperforms fate-of-the-aa DNN models, but its population states over time reveal the computational structure and neural underpinnings of contextual modulatory dynamics, generating predictions for experiments. Furthermore, we fine-tune the same model to perform two classic 2AFC attention psychophysics tasks. Reaction time analogs from DCnet strikingly recapitulate core tenets of feature-based attentional modulation. We find that DCnet responses are sensitive to target-distracto feature differences, heterogeneity of irrelevant distractor features, and display density,
Overall, these contributions suggest that our approach is a promising framework for modeling the brain's 's visual cortical dynamics, one that replicates key neural and behavioral signatures of contextual attentional modulation.
Limitations and future directions. In this work, we take the first step towards building an overcom- plete model of cortical circuitry. Palpable omissions from our current framework include long-range inter-area feedback and compartmental separation of feedforward and feedhack inputs. We hope to continue building on this framework in the future to include these components that will open up novel avenues for computational neuroscience. Additionally, extracting reaction time measures from large-scale recurrent models is discipline of its own [20] We adopt simple reaction time metric here that suffices for our current purpose, but we plan to include other sophisticated comparisons
in future work. Lastly, we have restricted our purview to visual sensory processing. The concept of contextual modulation, however, is pervasive across sensory modalities. We are excited about extending our ideas to other sensory domains.
Broader impact. Artificially intelligent models with enhanced visual capabilities are now pervasive in our daily lives. The not-so-hidden cost we pay enjoy these models' benefits is their carbon footprint on our environment. Building energy- parameter-, and sample-efficient models that are also performant is non-negotiable going forward. Understanding context-aware behavior of utmost importance for neuroscience research as its failure modes are associated with several psychiatric and neuropathologies. We do not anticipate any negative impact that our work would create.
Acknowledgments and Disclosure of Funding
We thank the McGovern Institute and the K. Lisa Yang Integrative Computational Neuroscience (ICoN) Center for supporting and funding this research. We are grateful to Mark Harnett, Jim DiCarlo, and Bob Desimone for enlightening conversations.
References
[1] Victor AF Lamme and Pieter R Roelfsema. The distinct modes of vision offered by feedforward and recurrent processing. Trends in neurosciences, 23(11):571-579, 2000.
[2] Karl Friston. A theory of cortical responses. Philosophical transactions of the Royal Society B: Biological sciences, 360(1456):11-883) 2005.
[3] Christopher Summerfield and Floris De Lange. Expectation in perceptual decision making: neural and computational mechanisms. Nature Reviews Neuroxcience, 15(11):745-756 2014.
[4] Moshe Bar, Karim S Kassam, Avniel Singh Ghuman, Jasmine Boshyan, Annette M Schmid, Anders M Dale, Matti S Hämäläinen, Ksenija Marinkovic, Daniel I. Schacter, Bruce R Rosen, et al. Top-down facilitation of visual recognition. Proceedings of the national academy of sciences, 103(2):449-454, 2006.
[5] Jeremy M Wolfe and Todd S Horowitz. What attributes guide the deployment of visual attention and how do they do it? Nature reviews neuroscience, 495--09, 2004.
[6] Charles D Gilbert and Wu Li. Top-down influences on visual processing. Nature Reviews Neuroscience, (K5):330-363, 2013.
[7] Charles D Gilbert and Torsten N Wiesel. Receptive field dynamics adult primary visual cortex. Nature, 356(6365):150-1155 1992.
[8] Jeremy M Wolfe. Guided search 6.0: An updated model of visual search. Psychonomic Bulletin & Review, 1864):1000-1099 2021.
10

---

[9] Jeremy M Wolfe, Kyle R Cave, and Susan L Franzel. Guided search: an alternative to the feature integration model for visual search. Journal of Experimental Psychology: Human perception and performance, 15(3): 419, 1989.
[10] Seth Herd and Randall C C'RReilly. Serial visual search from a parallel model. Vision Research, 45(24): 2987-2992, 2005.
[11] Sharat Chikkerur, Thomas Serre, Cheston Tan, and Tomaso Poggio. What and where: A bayesian inference theory of attention, Vision research, 50(22):2233-2244 2010.
[12] Kyle R Cave. The featuregate model of visual selection. Psychological research, 62(2):182 194, 1999.
[13] Mengmi Zhang. Jiashi Feng, Keng Teck Ma, Joo Hwee Lim, Qi Zhao, and Gabriel Kreiman. Finding any waldo with zero-shot invariant and efficient visual search. Nature communications, 91):3773, 2018.
[14] Shashi Kant Gupta, Mengmi Zhang, Chia-Chien Wu, Jeremy Wolfe, and Gabriel Kreiman. Visual search asymmetry: Deep nets and humans share similar inherent biases, Advances in neural information processing systems, 34:6946-6959, 2021.
[15] Yupei Chen, Zhibo Yang. Seoyoung Ahn, Dimitris Samaras, Minh Hoai, and Gregory Zelinsky. Coco- search18 fixation dataset for predicting goal-directed attention control. Scientific reports, 11(1):8776. 2021.
[16] Zenglin Shi, Ying Sun, and Mengmi Zhang. Training-free object counting with prompts. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 323-331, 2024.
[17] Grace W Lindsay and Kenneth D Miller. How biological attention mechanisms improve task performance in large-scale visual system model. Eliff,, 7:e38105, 2018
[18] Drew Linsley, Alckh Karkada Ashok, Lakshmi Narasimhan Govindarajan, Rex Liu, and Thomas Serre. Stable and expressive recurrent vision models. Advances in Neural Information Processing Systems, 33: 10456-10467, 2020.
[19] Vijay Veerabadran, Srinivas Ravishankar, Yuan Tang, Ritik Raina, and Virginia de Sa. Adaptive recurrent vision performs shot computation scaling unseen difficulty levels. Advances in Neural Information Processing Systems, 36, 2024.
[20] Lore Goetschalckx, Lakshmi Narasimhan Govindarajan, Alekh Karkada Ashok, Aarit Ahuja, David Sheinberg, and Thomas Serre. Computing a human-like reaction time metric from stable recurrent vision models. Advances in Neural Information Processing Systems, 36, 2024.
[21] Junkyung Kim, Drew Linsley, Kalpit Thakkar, and Thomas Serre. Disentangling neural mechanisms for perceptual grouping. arXiv preprint 1rXiv:1906.01556 2019.
[22] Wacl Asaad, Gregor Rainer, and Earl Miller. Neural activity the primate prefrontal cortex during associative learning. Neuron. 21(6):1399-1407 1998.
[23] Francesca Mastrogiuseppe and Srdjan Ostojic. Linking connectivity, dynamics, and computations in low-rank recurrent neural networks. Neuron, 99(3):609-66)) 2018
[24] Friedrich Schuessler, Alexis Dubreuil, Francesca Mastrogiuseppe. Srdjan Ostojic, and Omri Barak. Dy- namics of random recurrent networks with correlated low-rank structure. Physical Review Research, 2(1): 013111, 2020.
[25] Stefano Recanatesi, Ulises Pereira-Obilln Obilinovic, Masayoshi Murakami, Zachary Mainen, and Luca Mazzu- cato. Metastable attractors explain the variable timing of stable behavioral action sequences. Neuron, 110 (1):139-153) 2022.
[26] Alfonso Renart, Rubén Moreno-Bote, Xiao-Jing Wang, and Néstor Parga. Mcan-driven and fluctuation- driven persistent activity in recurrent networks. Neural computation, 19(1):1-46, 2007.
[27] Lea Duncker, Laura Driscoll, Krishna V Shenoy, Mancesh Sahani, and David Sussillo. Organizing recurrent network dynamics by cosk-compttation to enable continual learning. Advances in neural information processing systems, 33:14387-14397, 2020.
[28] Laureline Logiaco, LF Abbott, and Sean Escola. Thalamic control of cortical dynamics in model of flexible motor sequencing. Cell reports, 35(9), 2021.

---

[29] Kaushik J Lakshminarasinhan. Marjorie Xie, Jeremy D Cohen, Britton A Sauerbrei, Adam W Hantman. Ashok Litwin- Kumar, and Sean Escola. Specific connectivity optimizes learning thalamocortical loops. Cell Reports, 43(4), 2024.
[30] Daniel Schmid and Heiko Neumann. Thalamo-cortical interaction for incremental binding in mental contour-tracing. bioRxiv, pages 2023-12, 2023.
[31] Robert Desimone and John Duncan, Neural mechanisms of selective visual attention, Annual review of neuroscience, 18(1):193-222, 1995.
[32] Narcisse Bichot, Matthew Heard, Ellen M DeGennaro, and Robert Desimone. A source for feature- based attention in the prefrontal cortex. Neuron, 88(4):832-84) 844,, 2015.
[33] Andrew F Rossi, Narcisse P Bichot, Robert Desimone, and Leslic G Ungerleider. Top-down attentional deficits in macaques with lesions of lateral prefrontal cortex. Journal of Neuroscience, 27(42): 1314, 2007.
[34] Yang Zhou and David J Freedman. Posterior parietal cortex plays a causal role in perceptual and categorical decisions. Science, (65(6444))180-115) 2019.
[35] Caroline Jahn, Nikola T Markov, Britney Morea, Nathaniel D Daw, R Beeket Ebitz, and Timothy J Buschman. Learning attentional templates for value-based decision-making. Cell, 2024.
[36] Georgios Spyropoulos, Marius Schneider, Jochem van Kempen, Mare Alwin Gieselmann, Alexander Thiele, and Martin Vinck. Distinct feedforward and feedback pathways for cell-type specific attention effects in macaque v4. bioRxiv, pages 2022-11, 2022.
[37] Siyu Zhang, Min Xu, Tsukasa Kamigaki, Johnny Phong Hoang Do, Wei-Cheng Chang, Sean Jenvay, Kazunari Miyamichi, Liqun Luo, and Yang Dan. Long-range and local eircuits for top-down modulation of visual cortex processing. science, 345(6197)) (566617))660-6665 2014.
[38] Wenzhi Sun, Zhongchao Tan, Brett D Mensh, and Na Ji. Thalamus provides layer 4 of primary visual cortex with orientation-and direction-tuned inputs. Nature neuroxcience, 19(2):008-315, 2016.
[39] Morgane M Roth, Johannes Dahmen, Dylan Muir, Fabia Imhof, Francisco Martini, and Sonja B Hofer. Thalamic nuclei convey diverse contextual information to layer 1 of visual cortex. Nature neuroscience, 19 (2) 2277 2016.
[40] Domenico G Guarino, Andrew Davison, Yves Frégnac, and Ján Antolik. The cortico-thalamic loop attunes competitive lateral interactions across retinotopic and orientation preference maps. BioRxiv, pages 2022-12, 2022.
[41] Anthony D Lien and Massimo Scanziani. Tuned thalamic excitation is amplified by visual cortical circuits. Nature neuroscience, 16(9):1315-132,, 2013.
[42] John C Eccles, Fatt, and K Koketsu. Cholinergic and inhibitory synapses a pathway from motor-axon collaterals to motoneurones. The Journal of physiology. 126(3):524, 1954.
[43] D Fitzpatrick, JS Lund, DE Schmechel, and AC Towles. Distribution of gabaergic neurons and axon terminals in the macaque striate cortex. Journal of Comparative Neurology, 264(1):73-91, 1987.
[44] Junyoung Chung, Caglar Gulcchre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 1412.355,, 2014.
[45] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and patlern recognition, pages 770-778, 2016.
[46] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. arXiv preprint arXiv:2010. 11999, 2020.
[47] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[48] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2901-2910 2017.
12

---

[49] Anne Treisman and Hilary Schmidt. Illusory conjunctions in the perception of objects. Cognitive psychology, 14(1):107-141, 1982.
[50] Jagruti J Pattadkal, Boris V Zemelman, Ila Fiete, and Nicholas J Priehe. Primate nencortex performs balanced sensory amplification. Newron, 112(4):661-675, 2024.
[51]
Allen L Nagy and Robert R Sanchez. Critical color differences determined with a visual search task. JOSA A, 1(7):120--2217, 1990.
[52] Allen I. Nagy and Rohert R Sanchez. Chromaticity and luminance as coding dimensions in visual search. Human Factors, 346):601-611, 1992.
[53] Pouya Bashivan, Kohitij Kar, and James J DiCarlo. Neural population control via deep image synthesis. Science, xeu(6449 2019.
[54] Hidenori Tanaka, Aran Nayebi, Niru Maheswaranathan, Lane Mcintosh, Stephen Baccus, and Surya Ganguli. From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction. Advances neural information processing systems, 32, 2019.
[55] Kohitij Kar, Jonas Kubilius, Kailyn Schmidt, Elias Issa, and James J DiCarlo. Evidence that recurrent cir- cuits are critical to ventral stream's execution core object recognition behavior. Nature neuroscience. (2(6))774-983, 2019.
[56] Drew Linsley, Ivan F Rodriguez Rodriguez, Thomas Fel, Michael Arcaro, Saloni Sharma, Margaret Livingstone, and Thomas Serre. Performance-optimizzd deep neural networks are evolving into worse models of inferotemporal visual cortex. Advances in Neural Information Processing Systems, 36, 2024.
[57] Gabriel Kreiman and Thomas Serre. Beyond the feedforward sweep: feedback computations in the visual cortex. Annals of the New York Academy of Sciences, (464(1):222-2444 2020.
[58] Caspar M Schwiedrzik and Winrich A Freiwald. High-level prediction signals in a low-level area of the macaque face- pre-processing hierarchy. Newron, 96(1):89 97,9999777 2017.
[59] Petra Vetter and Albert Newen. Varieties of cognitive penetration in visual perception. Consciousness and cognition, 27:62-75, 2014.
13

---

A Dynamical Cortical network (DCnet)
A.1 Architectural specification
We instantiate DCnet recurrent convolutional neural network model following the primate visual cortex's anatomical and neurophysiologica properties. DCnel implements local, lateral, and feedforward synaptic connections, excitatory and inhibitory neurons, and long-range top-down inputs conceptualized as low-rank modulations of the input-driven sensory responses by high-level areas.
Specified below are the governing dynamics for the neurons in our model. Neurons in our model are either excitatory (E) or inhibitory (D) denotes the network layer (brain area). a' and 3 are indices used to describe excitatory and inhibitory cell types, respectively, in layer I. (I,, v) describes specific spatial location. 20) is the feedforward input to layer l. h refers to neuron's instantaneous state.
d
- 11 (1...)) + ge (0 (()) (1) a,,
TB
dt
(=(()) ME .19 xx,) S(W
input -pput-p=? =p ( + [ ME + [W In" a, + bF.))
oi
111 ( gz((2() mig 11 x.,
Zg
gr(20) (1 ,t, s(w!) imput-112 [ W NT ( Is ,,.y + br
TE and The are cell-type specific learnable neural time constants. Synaptic connections W's are sparse matrices on which we impose translational invariance (details below). These are, practice, realized convolutions. be_, and big. excitatory and inhibitory cell-type specific learnable thresholds. f(.) is non-linear activation function. We use the hyperbolic tangent as our activation function f. An average pooling operation Pool is applied to layer pyramidal outputs (hg) to increase the receptive field size by a factor of two.
(0 (hg [t]) O r'((h -T)) t T << T
2(+ 11
( (0) (t))
We make discrete time approximations to train model. The cue is presented first to the network (z())))) and the dynamics are unrolled for T steps followed by scene presentation (z)) for another T steps. While the scene is presented, inputs to each layer are modulated as follows, where <(.) is pooling operator that computes the average activity per cell type across all (z, y). r'(e is the low-rank modulation function defined as follows:
r(e) e Oa ee + bı] @ [Wi2e"' + b2))
Here, Wi,1, W7.2 learnable linear projections and b b2 learnable biases. denotes outer product and @ denotes pointwise scaling. By construction, the output of [Wr,ee" +b, WW +ba]
a low-rank matrix.
A.2 Implementational details
This section provides specific details on model parameters detailed in Section A.
All layers have the following convolutional kernels (Ws specified in the governing equations): Input to excitation (W (E)) excitation to excitation (WR--E), excitation to inhibition (WE--1). inhibition to excitation (Wi+E), and input to inhibition (W (Wingur-1). The dimensionality of these convolutions is listed layer-wise below, along with the kernel and padding shapes for all convolutions in that layer.
Input size
# exc. cell types (Nz)
inh. cell types (N+)
Kernel size, Padding
Layer
3 x 128 128
16
4
(5, 5), (2, 2)
Layer
64 64
(5, (2, 2)
Layer
33 x 32 x 32
64
16
(5, (2,
Layer
128 x 16 x 16
128
(3,3), (1,
14

---

char Pnserbtttoonoo of
Mart Hrandey Eraniin
distibutioon of
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer
175
152
133
Tox (mS)
Tyolms)
10.0
0.3
5.0
2.
2.5
10 call typo
322
type
Figure 8: An emergent macroscopic gradient in neuron time constants. Cell-type specific integration time constants in DCnet are learnable parameters. While initialize the r's uniformly (blue) pre-training, we observe layer-dependence post-training for both the (a) excitatory, and (b) inhibitory cell types.
The outputs from the last layer are transformed via a linear readout parameterized as fully connected layer (1024 x 6) into logits, which we supervise with CrossEntropy loss. The model does not include standard embellishments like explicit normalization layers, model ensembling, test-time augmentations, etc.
A.3 Training details
All models were trained on A100 GPUs for 100 epochs each. An experimental run took anywhere from 4-8 hours to complete. We used AdamW optimizer (momentum=009,, 31 0.9, 33 0.999), a one-cycle learning rate scheduler with a warm-up period of 30 epochs and maximum learning rate of 4e 4. DCnet was 4 layers deep (~~ 1.8M learnable parameters) and was trained with batches of 256 samples. Code and datasets can be found here: Project repository.
A.4 Mechanistic interpretability analyses
In the interest of mechanistic interpretability, we perform series of experiments on DCnet. We drive activity by uncorrelated, time-varying Gaussian noise inputs. We then compute:
1. The Lag-1 autocorrelation measure of stability of the excitatory neurons (Fig. 5p). We find that DCnet excitatory neurons are significantly more stable than excitatory neurons in the DCnel (trained w/o inhibition) model. Kolmogorov-Sminn Smirroo test confirmed the significant difference (statistic=1.0,) p .001).
2. The Dynamic Range (DR) of excitatory cells in each layer of DCnel and compare that to the DR of corresponding excitatory cells from DCnet (trained w/o inhibition) (Fig. (Fig. DR is computed as the Interquartile range (a measure of statistical dispersion) of neuron's s activity over time period of 128ms when driven by noise. We take the mean over 64 trials for each neuron and plot this distribution per layer in Fig. 5t. We find that excitatory neurons in the DCnet model have significantly higher DR across layers compared to DCner (trained w/o inhibition), suggesting the role of inhibitory interactions in expanding the range of computations carried out by each neuron. Kolmogorov-Smming these significant differences (Layer I (statistic=0.667, < .001); Layer (statisti==0.99, p < .001); Layer (statistic=1.f .0, p < .001); Layer 4 (statistic=0.97, p .001)).
test confirmed
3. The E-I correlation coefficient as a measure of co-tuning in DCnet. We find that the average (across neurons) E-I correlation is as follows: -0.076 (Layer 1), 0.766 (Layer 2), 0.699 (Layer 3). 0.535 (Layer 4).
Additionally, we also highlight macroscopic gradient in learned time constants across the layers of DCnet (Fig. 8).
15

---

A.5 Sample model outputs
Cues
Scenes
Answer
Model
4
4
Color Trials
3
3
4
5
5
Shape Trials
4
4
2
1
Conjunction Trials
2
Figure Model outputs. Visualizing example DCner predictions on out-of-distribution (held-out) scenes across trial types.
16

---

A.6 DCnet on visual object recognition
In follow-up experiment, we train the sensory backbone of DCner on visual object recognition task. On CIFAR-10, we report a test accuracy of 84.79% We highlight that our model does not include standard embellishments used in machine learning, including explicit normalization layers, model ensembling, test-time augmentations, etc. This is first step that strongly demonstrates the potential of our framework to scale up.
654
car
10
921
2
bird
31
813
32
26
11
cet
000
deer
9
43
42
830
18
23
22
dog
755
frog
7
38
34
10
14
884
3
17
24
27
24
872
200
ship
truck
20
49
5
10
1
13
092
plane
harbe
ship
truck
Predicted label
Figure 10: DCnet performance on visual object recognition. Confusion matrix of DCner outputs on the CIFAR-10 test set. Overall model performance 84.79%.
B Convolutional RNN Baseline
Layer
Input Size
Operations
Layer
3 x 128 x 128
Conv2d (8 x x 5 x 5), ReLU, AvgPoo12d 5, stride 2)
Layer?
8 x x 64
Conv2d (16 x 8 x 5), ReLU, AvgPoo12d x 5, stride
Layer
16 x 32 x 32
Conv2d (32 x 16
5), ReLU, AvgPoo12d (3 3, stride 2)
Layer
32 x 16 x 16
Conv2d (64 x x x 3), ReLU, AvgPool2d (3 x 3, stride 2)
Layer
8 x
Conv2d (128 x 64 x x 3), ReLU, AvgPoo1266 (2
stride 2)
Layer
128 x 4
Conv2d (128 128 x 3), ReLU, AvgPoo12d (1 x 1, stride
Projection
Fully connected (2048 inputs. 2048 outputs), ReLU, GRU (2048 inputs, 2048 outputs)
Readout
Fully connected (2048 inputs, 6 outputs)
Table Bascline model details
Convolution filters are mentioned as: (output channels x input channels x kernel height x kernel width). Pooling layers are mentioned as: (kernel height x kernel width, stride). Inputs to every layer feature extractor for the
were normalized with LayerNorm. The convolutional backbone serves GRU network. We train this baseline model by passing in the cue for the first T time steps followed by the scene for the next T timesteps. For our experiments, we set T = 3. A cross-entropy loss is used for the readout activities at the final time step.
17

---

C
Zero-shot testing of LLM
Prompt Bere a task that takes two images, cuc scenc. Here are the instructions: is color, return count of how many objects the scene have that
1. If the color. 2. IE the cue is orange colored shape, return count of all objects in the scene that have that shape. 3. If the ave is a non-orange colored shape, return count You will only need follow of the instructions above. Don't write code. Taere is no need to explain the logic of your counting. Simply provide the final answer.
objects
shape color.
GPT4 Sure, can help with that. Please upload the two images have: the oue image and the scene Image, and 1'll proceed according to your instructions.
ChaNGPT
ChaRSPT
The supaplooste Acceromplt instuct Ploceet scens have
intructioon, will oount objects
Fomes suung calondlope Auoand og yoyorr iotracinnn Roourial injoda stee that have thosame shepe tecces
pul aik Aoord mylo par nutracie Aicco somee how oote
Diylindrr the soane image.
2 rijects pollos tnw amaacces
ChanGPT
cas pupppp voloed Accerinng par indrucaion, alsdeectratt somes that how bote thesame sapp odea harcue.
shhorrr of
There in tpaple cabe sanna iage
The orange coloned Acco toygtoyyu inttuction stotccca soore that ths ahaps of cac.
stceis
Figure 11: ChatGPT4 experiments We prompted GPT4 to solve solve vis-count. Given a color, shape, conjunction cue, asked the LLM identify how many objects in the provided scene match the cue's properties. Out of 30 trials, GPT4 achieved success rate 37%.
18

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes. all claims made in the abstract and introduction reflect the paper's
contributions. We make clear the strengths of our work and also the limitations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, the limitations of our work is discussed in the Discussion section of paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
19

---

The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex: system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper does not contain any purely mathematical or theoretical results. There are no mathematical proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, the details about our experimental setup are provided in the Supplementary Section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
20

---

If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes, we provide link to our code that anonymized. We ensure that the code can be used to reproduce our main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
pwt.aoooommisiiooooocoocc for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
21

---

Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Hyperparameters are provided in our Supplementary Section. Further details regarding our experiments can also be found in our code, which we fully disclose in our Supplementary Section as well.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report all statistically significant measures for our experiments, These can be seen in our Results section.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Discussion about the resources used to train our model are discussed in the Supplementary Section.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
22

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttp:://neurii::
Answer:
[Yes]
Justification:
We have verified that work follows the NeurlPS Code of Ethics fully and
completely.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work does not have a clear societal impact and thus we do not discuss this in our paper. This work is with regards building biologically plausible vision models that can serve as hypothesis generators for real world experiments. These models will not be used by the average everyday user or affect everyday society in any regard.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
23

---

Our model poses no such risks. We do not train large models that have collected user data, nor do we present such models in our work.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Baselines implemented by other research papers have been fully credited. All code is properly documented and cited, as well.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, papprswi wwwode.tthcc....m com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide the code for exactly reproducing the datasets that we generate. This reproduction process is well documented and shared within the code that we share.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing research or collecting humans in any way.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This research does not involve crowdsourcing or conducting experi- ments/research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "sG1o5Chgkr", "reviewer": "Reviewer_BT2x", "summary": "The paper presents a model for how high levels of (presumably cortical) representation modulate lower levels.  A multilayer neural network with recurrent connections both within each layer and between layers is trained on a visual cue-delay-search task.  The phenomenology of the model is then studied and characterized, showing how low level representations are modulated by task demands, which makes predictions for experiments.", "questions": "see above", "limitations": "see above", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Mechanisms motivated by behavioral and physiological results\n- Evaluations based on psychophysical tasks\n- Relatively small compared to other deep learning models\n- Attempts to understand recurrent interactions in deep networks, currently missing from much of the literature.", "weaknesses": "- The framing of the central question of the paper in terms of a “modulatory homunculus” that must decide what features to attend to in lower-levels seems unnatural and odd.  It seems exclusively oriented toward modeling a specific laboratory task.  But what about natural vision?   It would seem these recurrent connections must always be in play for myriad tasks in daily life.  It would have been more compelling to see the model trained on a broader range of tasks - e.g., visual scene analysis - rather than this specific laboratory task.\n- There are many recurrent interactions in the model, but the results focus more on characterizing the phenomenology of the model as opposed to what is learned by the recurrent weight matrices, which from a mechanistic point of view would seem most interesting.  It would also be worthwhile to start with a minimal model that demonstrates the principles of what is being learned, as opposed to throwing it all in the kitchen sink of the model shown in Figure 1.\n- Because the model has so many components and it is not well-explained in the main body (including not defining all the variables in the equation - see below), by the time you get to results there is still no intuition for how the model is cued and performs the task, or how it learns.\n- Unclear if the comparison to models without time dynamics and explicit cuing is informative.\n\nOther comments:\n- homunculus is misspelled both times\n- 65: I don’t think there are explicit testable predictions/potential experiments that come from the model anywhere in the paper\n- 110: Even if there is more model specification in appendix, one should at least define all the variables and give basic intuition for the equations. What is r_1, r_2 ?  W? Where are the time constants? Not clear how the variables correspond to interneurons, lateral connections, etc. What is the role of the spatial pooling operator (not explained even in the appendix)?\n- Fig 2b: does not seem to actually outperform ResNet, even though it seems like it might have an advantage over ResNet because of cuing (though this is unclear because they don't explain cuing)\n- 120: Besides the Conv RNN model (maybe?), are these comparable baselines if they have no temporal dynamics?\n- 144: Don't seem to describe the novel cues and scenes anywhere. How different are they from the training data? Is a 55% accuracy impressive?\n- 148: How do they know it included the cyan cylinder and not another object (and therefore conclude it's an illusory conjunction)? Could it also be confused by the occlusion of the blue cube?\n- 169: \"We believe\"... this whole paragraph seems like speculation based on looking at Fig. 2, but no actual proof/argument?\n- 184: Unclear what \"cortical gradients\" means, I'm assuming trajectory in PCA space? Is it that interesting/surprising that trajectories are different for color and shape given the model presumably would have to disentangle in order to get high accuracy?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and insightful comments. We provide a point-by-point response below. Your comments have made the manuscript stronger. We hope that we have sufficiently addressed your concerns enough to revise your score.\n\n>**Model mechanisms are motivated by physiology/behavior and an attempt is made to understand recurrent interactions in deep neural networks which is missing from much of the literature.**\n\nWe are particularly happy that the reviewer pointed this out. One of our motivations in pursuing this line of work is to tackle the increasing divergence between successful deep learning approaches that model visual computations and biological visual processing. We will make sure to highlight this in our discussion.\n\n> **What about natural vision? Why focus on a “laboratory task”?**\n\nThanks for raising this point. It is true that feedback processes in the visual system are critical for a variety of functions. Having said that, we wish to highlight that our approach, to the best of our knowledge, is among the first to tackle contextual cueing paradigms in a realistic manner. \n\nOur focus was on the problem of how models can learn sensory representations of “cue”-ed attributes to look for in a “scene”. Please note that these attributes may not directly be present in the visual scene. For example, when cued for a specific color, that color may (or may not) exist in the scene under various lighting conditions, shadows, occlusions, object textures, material properties, and at different scales. To do this, one needs to be able to generate systematic variations on a theme. Although vis-count may not be “naturalistic” in a traditional sense, it contains non-trivial variations that are important to study. We highlight that there is a lack of similar datasets for naturalistic visual search at scale.\n\nWe do hope to build towards naturalistic scale, and as a first step, we now try training our sensory backbone on naturalistic object recognition tasks (without standard ML bells and whistles such as model ensembling, test time augmentations, etc.). We report an accuracy of 84.79% on CIFAR-10 (Fig. R4). This is only a preliminary result, but ones which strongly demonstrate the potential of our framework to scale up.\n\n> **The focus is on phenomenology as opposed to gaining mechanistic insights.**\n\nPoint well taken. We believe these are non-orthogonal interesting things to study. We have now performed a host of quantitative analyses to gain insights into the role of inhibitory influence in determining the dynamic range and stability of the excitatory neurons and co-tuning (please see our response to **qySN**). Additionally, we also highlight a macroscopic gradient in learned time constants in Fig. R1.\n> **``Kitchen sink” approach to Figure 1.**\n\nWe wish to point out that the anatomical constraints that we include are simplified forms of well-studied elements in biophysics. Our ablation studies make it clear that the design constraints here (separate excitatory and inhibitory populations, top-down feedback, learnable dynamics, etc.) are both necessary for model performance and facilitate comparisons to relevant neuroscience literature.\n\n> **Intuition for how the model is cued and performs the task, or how it learns**\n\nWe apologize for the lack of clarity in this aspect. With our revised model description (provided in the general response), we have tried to make this point clearer.\n\nBoth cues and scenes are visual inputs ($\\in \\mathbb{R}^{128 \\times 128 \\times 3}$) provided to Layer 1 of our model. Cues are persistently provided to the network for the first $T$ discrete time steps followed by scenes for the next $T$ time steps. The output activities of the last layer at the last time step ($t = 2T$) is transformed into logits and a supervision signal is provided via a cross entropy loss (ground truth labels are counts from 0 to 5). \n\nIntuitively, our model learns to “up-modulate” features in the scene that resemble features of the cue, and ultimately use these up-modulated features to inform its final output. To do so, it needs to learn a disentangled feature basis (shared between cues and scenes as they are processed by the same backbone) because aspects of the same scene can be up- or down- modulated differentially based on the cue. This is what we try to expose in Figure 3.\n\n> **Unclear if the comparison to models without time dynamics and explicit cuing is informative**\n\nWe believe that these comparisons are important particularly because it helps understand the upper bounds of expected performance. Our “implicit” cueing condition is one where the cue and scene are provided to the model **at that same time**. In practice, for a model without time dynamics, such as the ResNet and the Transformers, this is realized by stacking these two inputs before passing them into the model. This allows for the model to perform direct comparisons between the features of these two inputs, as opposed to time-delayed comparisons that happen in the explicit-cuing condition. Hence, this provides a potential upper-bound on training performance. Interestingly, while some of these implicit models can learn the task well (such as the ResNet in Fig. 2b), they fail to generalize (Fig. 2c).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w1JgKx12Tt", "reviewer": "Reviewer_7BUE", "summary": "The authors introduce DCNet, a hierarchical recurrent vision model that draws inspiration from the structure and function of biological visual circuits. This novel architecture consists of excitatory and inhibitory bottom-up neurons modulated by a (low-rank) representation of a higher area, analogous to the higher cortical and thalamic regions of the brain. Through their proposed model dynamics, the authors demonstrate emergent task-driven contextualization of bottom-up responses. Experimental results show that DCNet successfully replicates human behavioral findings on various classic psychophysics-inspired visual cue-delay search tasks, highlighting the model's potential in understanding and mimicking human visual perception.", "questions": "Please refer to my review above.", "limitations": "The authors have addressed limitations adequately.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The paper presents a noteworthy contribution with the proposed DCNet architecture, which creatively integrates biological anatomical insights with deep neural networks. The authors provide a clear and compelling motivation for their design, effectively addressing the need for contextual modulation of visual responses.\n+ The experimental results demonstrate the promise of DCNet as a model of human visual search behavior. Notably, the findings in Figure 2 (vis-count) and Figure 6 (visual search with distractors) showcase DCNet's ability to learn generalizable contextual visual search solutions, outperforming baseline models and aligning with human behavioral patterns.\n+ A particularly intriguing aspect of the paper is the emergence of repeated trajectories and attractor states for cued stimuli, as seen in Figure 3. This phenomenon offers valuable insights and warrants further exploration.\n+ Figure 4 is a promising visualization validating that task-relevant information is stored in the low-rank higher area modulation. It is interesting to see how the early layer lesions are impacting shape selectivity, which one expects to emerge in higher layers of the perception stream.", "weaknesses": "- While the proposed DCNet model is intriguing, it would be beneficial to more explicitly highlight its anatomical constraints in relation to prior work. Conversely, the authors could further emphasize the novelty and significance of the low-rank modulation aspect, which appears to be a unique contribution. Additional signposting would help to clarify its impact on the observed results.\n- To strengthen the paper, the authors may consider including ablation studies to dissect the contributions of individual components within the DCNet architecture. This would provide valuable insights into which elements are crucial for the model's high performance and alignment with human behavior.\n- The authors cite relevant models in references [18-21], but it is unclear how these models perform on the vis-count task. A more comprehensive comparison would be helpful, including a detailed discussion of the differences between these models and DCNet. This would enable a more nuanced understanding of the proposed model's advantages and limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their majorly positive comments. We believe that the suggestions raised by the reviewer will increase the quality of the manuscript and we now provide additional analyses and responses to these.\n\n> **Explicitly highlight DCNet’s anatomical constraints relative to prior work**\n\nThanks for this comment. Also, in line with comments from the other reviewers, we now provide a full model specification in the general response. We will make sure to highlight and signpost anatomical facets of our model that are unique including: distinct E and I subpopulations, lateral recurrent feedback, learnable neuron time constants per cell-type, and long-range top-down feedback conceptualized as multiplicative low-rank perturbations.\n\n> **Ablation studies to dissect individual components of DCNet architecture**\n\nWe agree with the reviewer. In this regard, we now perform two ablation studies. \n\nFirst, we train a version of the model where we lesion the Inhibitory cell populations in every layer. We analyze the internal input-driven dynamics of this model variant in Fig. R2. We train DCNet (Lesioned Inhibition) on vis-count (color condition; for the purposes of time). While DCNet (Lesioned Inhibition) learns the task ($95$% accurate), we find that neurons have a significantly smaller dynamic range and are comparatively less stable (details of these analyses are given in our response to **qySN**) when driven for longer time periods. \n\nSecond, we train a version of the model with lesioned top-down feedback. DCNet (Lesioned Top-down) drops significantly in performance to $38$%, further highlighting the importance of top-down feedback.\n\n> **Unclear how relevant models [18 - 21] perform on vis-count task and differences between those models and DCNet**\n\nThanks for raising this point. We did not evaluate these models on vis-count as none of these models work within a cueing-paradigm. We will, however, make sure to highlight aspects of these models that are similar to our work. [18, 19, 20] are single-layer convolutional recurrent neural network models with lateral feedback and distinct E/I populations. [21] is an extension of [20] to include a feature processing hierarchy. All these models included ML-esque normalization operations (such as BatchNorm or LayerNorm) to impart stability during training and none of these models included cell-type specific learnable integration constants as well as time-delayed top-down modulation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kN69JCoa2K", "reviewer": "Reviewer_qySN", "summary": "The authors present a convolutional dynamical systems model if visual processing with separate excitatory and inhibitory populations in each layer. They add a low rank modulation of each layer by a factor that is computed as a linear map of the layer activities. This is meant to represent top down modulations by some higher level area. The authors train their network on visual search tasks, including a new one which requires participants to count the number of objects in a scene that match a target in color, shape, or both. The context modulation is required for good performance of the model and the model reproduces two effects from the visual search literature.", "questions": "Did the authors explore variants of the model where the modulation depends on all layers, has a memory or similar ideas? For a higher order representation, the current implementation seems extremely reduced.\n\nDid the authors run any quantitative comparisons between their model and any neural or psychophysical measurements.", "limitations": "I agree with the authors that their work is unlikely to have negative societal impact. Limitations of their work could have been discussed more carefully though.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The presented model has a relatively high level of biological detail by including a split in excitatory and inhibitory pools, temporal dynamics and a low-rank top down mechanism.\nAlso the network is image computable, can be trained for reasonably complex tasks and the authors compare to some reaction time data.", "weaknesses": "All results to evaluate the network are qualitative, in contrast to claims of the authors that their network closely replicates known reaction time results. \nWhile higher distractor heterogeneity, which makes humans slower, makes the network more uncertain (6c), target absent trials usually yield longer action times while the entropy of the network is lower here. So even the qualitative result is not as in humans.\nThe positive slope in 7c seems to rely heavily on the data point at 1 distractor, does this actually keep growing?\nFor 7d I would love some comparison to human data as the feature difference does not tell me much such that I cannot judge whether this actually matches human behaviour.\nAlso, It is customary to look at the accuracy of responses as well somewhere.\n\nWhile the proposed network does perform better than standard DNNs the difference is small, especially to ResNet-18. Also the generalization gap is quite substantial for the proposed network as well. So overall the performance improvement is marginal.\n\nWhile the authors comment on the dynamics observed in their model in Figure 5 and the accompanying text, I am not really following this part. Proper analyses of the dynamics should contain some quantitative measures of the dynamics and comparisons of these to neural data. Just looking at a set of randomly chosen activity traces is insufficient I think.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive comments. We believe that some of the suggestions raised by the reviewer will increase the quality of the manuscript and we now provide additional analyses and responses to these.\n\n> **Relatively high level of biological detail. The network is image computable and can be trained for reasonably complex tasks**\n\nWe thank the reviewer for this note. As we noted in our response to *BT2x*, one of our goals was to tackle the increasing divergence between successful deep learning approaches that model visual computations and biological visual processing.\n\n> **Direct reaction time matches to human data**\n\nWe agree with the reviewer that target-absent trials yield higher reaction times in humans while we don’t see this in our model output entropy. We wish to highlight that model entropy is far from the perfect RT metric that one can extract from neural dynamics. In fact, the subject of computing model RTs is an active research area (Spoerer et al. (2020); Goetschalckx et al. (2024); Subramanian et al. (2022); Graves (2016) to name a few), and exploration of different metrics is beyond the scope of this work. Our goal here was to expose the potential for studying model RTs in the context of cued-paradigms (through DCNet). Cued-contextual paradigms constitute a wealth of data in human psychophysical research but to our knowledge, there is no current modeling approach to study this in a naturalistic manner.\n\nAs for Fig. 7c: We limit the number of distractors to 6 to account for the canvas size and placement of the objects within it. The reviewer’s point is well taken. Our response is the same as above – entropy isn’t the perfect metric. We genuinely hope to extend on this in future work.\n\nHuman data comparison to 7d: While generating our stimuli, we used perceptually uniform color spaces to stay close to the experimental paradigm discussed in Wolfe and Horowitz (2004) Fig. 5a-e. We agree that this isn’t a direct comparison to human RTs and emphasize that we wished to only show human-like trends (which is strongly present in our analysis). Exact comparisons to human RTs is beyond the scope of this work, but it is something we are really interested in exploring in future work.\n\nWe will add the limitations of our chosen RT metric as part of the limitations and outlook for the future in our manuscript. \n\n> **Customary to look at the accuracy of responses as well somewhere**\n\nThanks for the comment. We do refer to the accuracy of model responses in the manuscript.\n\nL222 “DCnet achieved an overall accuracy of 97% on the test trials”.\n\nL238 “When fine tuned on this task, DCnet achieved an overall accuracy of 95% on the test trials”.\n\n> **Overall the performance improvement is marginal**\n\nAs the reviewer points out, the proposed network performs marginally better than ResNet-18 in-distribution (Fig. 2b), the difference is much greater when tested out-of-distribution (Fig. 2c) – our model drops by $\\sim 0.2\\%$ while ResNet-18 drops by $\\sim 0.6\\%$. We wish to highlight that performance was not our only criteria of desire. Rather, we wanted to build a competitive and biologically-faithful framework that will allow us to study the dynamics of contextual-cueing phenomenon.\n\n> **Proper analyses of the dynamics...activity traces is insufficient.**\n\nWe fully agree with the reviewer and thank them for raising this point. While comparisons to neural data is beyond the scope of this rebuttal timeline, we perform several analyses to better understand and quantify the internal dynamics of our model. We missed the point with Fig. 5, but will now update it to include the analyses we detail below (also in Fig. R2).\n\nAs originally presented in Fig.5, we drive DCNet activity by uncorrelated, time-varying Gaussian noise inputs. We then compute:\n\n1. The *Dynamic Range* (DR) of excitatory cells in each layer of DCNet and compare that to the DR of corresponding excitatory cells from DCNet (Lesioned Inhibition) (Fig. R2c). DR is computed as the Interquartile range (a measure of statistical dispersion) of a neuron’s activity over a time period of $128 ms$ when driven by noise. We take the mean over 64 trials for each neuron and plot this distribution per-layer in Fig. R2c. We find that excitatory neurons in the DCNet model have a significantly higher DR across layers compared to DCNet (Lesioned Inhibition) suggesting the role of inhibitory interactions in expanding the range of computations carried out by each neuron. A Kolmogorov–Smirnov test confirmed these significant differences (Layer 1 (statistic=0.667, p < .001); Layer 2(statistic=0.99, p < .001); Layer 3(statistic=1.0, p < .001); Layer 4(statistic=0.97, p < .001)).\n\n1. The lag 1 autocorrelation as a measure of *stability* of the excitatory neurons (Fig. R2b). We find that DCNet excitatory neurons are significantly more stable than excitatory neurons in the DCNet (Lesioned Inhibition) model. A Kolmogorov–Smirnov test confirmed the significant difference (statistic=1.0, p < .001).\n\n1. The E-I correlation coefficient as a measure of co-tuning in DCNet. We find that the average (across neurons) E-I correlation is as follows: -0.076 (Layer 1), 0.766 (Layer 2), 0.699 (Layer 3), 0.535 (Layer 4). This confirms the visual intuition provided in Fig. R2a that E-I co-tuning is weaker in early compared to late layers.\n\n> **Did the authors explore variants of the model...extremely reduced.**\n\nThe reviewer makes an excellent suggestion. Inspired by this, we implement and test a new form of top-down modulation that pools information from all cortical layers before computing time-delayed top-down modulating factors as a non-linear transformation (one per layer) of this pooled input. This model achieved an overall accuracy of $92.35\\%$ on vis-count, which is better than the metrics we report in the manuscript. However, we note that more work needs to be done to understand this better as the biological analogues of this process is not apparent.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lxFltI3mrS", "reviewer": "Reviewer_BdqG", "summary": "Artificial neural networks are loosely based on biological neural networks. In this study, the authors construct neural networks whose structures are based on general principles of visual signal pathway such as retinotopy. They used this model to ask if the context-dependent feedback mimicking top-down signals in the brain could help artificial neural networks perform cue-delay tasks more efficiently.", "questions": "Please see the weakness section.", "limitations": "The authors provided the limitations of their study.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The authors constructed a simple but effective model (DCNet) that can capture the gist of interplay between low-order sensory areas and high-level cognitive areas. They showed that their model can outperform conventional deep neural networks and large language models, although DCNet has a small number of learnable parameters. Also, they performed simulated legion experiments and psychophysics experiments, providing intriguing similarities between biological visual systems and DCNet.\n\nThe strong influence of top-down (feedback in their study) on sensory signal processing is already well known, but this study provides some evidence that such top-down modulation can be used to build a new type of deep neural networks. The authors tested DCNet on a single task only, but this study may still contribute to advancing bio-inspired computing.", "weaknesses": "The paper is well written and easy to follow, but this reader finds it a little strange that the authors did not specify basic information on their model or baseline models. First, they provided mathematical descriptions of the recurrent “neuron” in the model, but it remains unclear how they implement these neurons. Based on the given source codes and the appendix, this reader thinks that individual neurons are 2D convolutional filters and that the intermediate states are used to realize recurrent behaviors. Additionally, 2D convolutional filters have multiple parameters and are not listed in the manuscript. I did find some parameters in the source codes, but they are not good enough. Second, the authors should provide more details on the baseline models. They mentioned “Traditional 6-Layer convolutional backbone feeding into a gated recurrent unit (GRU) [44] with N = 2048 neurons”, but the meaning of traditional 6-layer convolutional backbone is unclear, and they did not explain how 2048 GRU neurons provide the final answers. Once again, the authors need to provide pertinent details of each component. Third, the modulation signals are computed with two linear projects r1 and r2. It looks like they were estimated from all 4 layers, but it is unclear how they are concatenated to create r1 and r2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their overall positive comments. We have addressed your comments below including full model and baseline specifications. We hope that we have sufficiently addressed your concerns enough to revise your score.\n\n> **Capture the gist of interplay between low-order sensory areas and high-level cognitive areas. Top down modulation can be used to build a new class of DNNs**\n\nWe sincerely thank the reviewer for this positive comment. This was indeed one of our primary motivations to pursue this line of work.\n\n> **Did not specify basic information on their model**\n\nFirst, we would like to thank the reviewer for checking our source code. We also acknowledge that we could have done a better job of explaining our model components. We provide a full model description in the general response (and will include this information as part of our manuscript). Here, we provide more specific details on exact model parameters.\n\nLayer 1 (Input size 3 x 128 x 128): \nExcitatory cell types (16), Inhibitory cell types (4). \nKernel (5, 5), Padding (2, 2)\n\nLayer 2 (Input size 16 x 64 x 64): \nExcitatory cell types (32), Inhibitory cell types (8)\nKernel (5, 5), Padding (2, 2)\n\nLayer 3 (Input size 32 x 32 x 32): \nExcitatory cell types (64), Inhibitory cell types (16)\nKernel (5, 5), Padding (2, 2)\n\nLayer 4 (Input size 128 x 16 x 16): \nExcitatory cell types (128), Inhibitory cell types (32)\nKernel sizes (3, 3), Padding (1, 1)\n\nFinal Readout: Fully connected (1024 inputs, 6 outputs)\n\nAll layers have the following convolutional kernels ($\\textbf{W}$s specified in the governing equations): Input to excitation, excitation to excitation, excitation to inhibition, inhibition to excitation, and input to inhibition. The input and output dimensionalities of these convolutions are listed layer-wise above, along with the kernel and padding shapes for all convolutions in that layer.\n\n> **More details on the baseline models**\n\nConvolution filters are mentioned as: output channels x input channels x spatial size (height, width). Pooling layers are mentioned as: kernel size (height x width), stride. Inputs to every layer were normalized with LayerNorm.\n\nLayer 1 (Input size 3 x 128 x 128): Conv2d (8 x 3 x 5 x 5), ReLU, AvgPool2d (5, 5, stride 2).\n\nLayer 2 (Input size 8 x 64 x 64): Conv2d (16 x 8 x 5 x 5), ReLU, AvgPool2d (5, 5, stride 2). \n\nLayer 3 (Input size 16 x 32 x 32): Conv2d (32 x 16 x 5 x 5), ReLU, AvgPool2d (3, 3, stride 2). \n\nLayer 4 (Input size 32 x 16 x 16): Conv2d (64 x 32 x 3 x 3), ReLU, AvgPool2d (3, 3, stride 2). \n\nLayer 5 (Input size 64 x 8 x 8): Conv2d (128 x 64 x 3 x 3), ReLU, AvgPool2d (2, 2, stride 2). \n\nLayer 6 (Input size 128 x 4 x 4): Conv2d (128 x 128 x 3 x 3), ReLU, AvgPool2d (1, 1, stride 1).\n\nProject convolutional outputs to GRU: Fully connected (2048 inputs, 2048 outputs), ReLU\n\nRecurrence: GRU (2048 inputs, 2048 outputs)\n\nFinal Readout: Fully connected (2048 inputs, 6 outputs)\n\nThe convolutional backbone serves as a feature extractor for the GRU network. We train this baseline model by passing in the cue for the first $T$ time steps followed by the scene for the next $T$ timesteps. For our experiments we set $T=3$. A CrossEntropy Loss is used on the readout activities at the final timestep. \n\nWe will include these details in the Appendix.\n\n> **Unclear modulation signal is formed from concatenating r1 and r2s**\n\nWe apologize for this confusion. The modulatory signals are computed in a layer-matched manner and hence there is no issue of concatenation. \n\nHowever, we include in this rebuttal experimental results (in response to **qySN**) for a variant of the modulation signal that pools from all layers before computing the modulation factors. For this implementation, we linearly projected excitatory neuron activities from every layer into the dimension of interest (based on which layer is being modulated), followed by a sigmoided linear-nonlinear transformation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their time in reading our manuscript and for their extensive feedback. In this general response, we address some common themes across the reviews. We provide detailed answers to specific reviewers' comments in subsequent responses. To go with this rebuttal, we also provide a PDF with additional figures labeled **Fig. R1-R4**.

First, we sincerely thank the reviewers for their generally positive feedback. All reviewers noted the importance and novelty of our model construction, as well as the utility in understanding recurrent dynamics in deep networks and the promise of building a new class of biologically motivated models with top-down modulation.

One common critique across reviews pointed to the lack of details with respect to the model specification as well as quantitative analyses.  

Towards addressing questions raised by the reviewers, we perform and include several new analyses in this rebuttal. Specifically, 

1. We perform two kinds of ablation studies on DCNet. We train one version of DCNet with its inhibitory populations lesioned and one version of DCNet with top-down feedback lesioned. (**7BUE**)
1. We perform a series of mechanistic interpretability analyses to better understand the learned cell type specific time constants in our model, quantify the dynamic range and stability of excitatory neurons (in the presence and absence of inhibitory interactions), and quantify the degree of co-tuning between the excitatory and inhibitory populations in the model. (**BT2x**, **qySN**)
1. We train the DCNet backbone model on CIFAR10, an object recognition task, and report a competitive performance of $84.79$%. (**BT2x**)
1. We implement an alternative form of top-down modulation that factors in information from all layers. (**qySN**)
1. We include a detailed model specification (below) as requested by multiple reviewers (**BT2x**, **BdqG**).

We hope you agree that our manuscript has improved through your feedback and that our findings will have a significant impact on computational cognitive neuroscience.



### Model specification

Neurons in our model are either excitatory (exc) or inhibitory (inh). $i$ and $j$ are indices used to describe excitatory and inhibitory cell types respectively. $(x, y)$ describes a specific spatial location. $l$ denotes the network layer. $z^{(l)}$ is the feedforward input to layer $l$. $h$ refers to a neuron's instantaneous state. Specified below are the governing dynamics for the neurons in our model.

$$
\tau\_{\text{exc}\_i}^{(l)} \frac{d h\_{\text{exc}\_i}^{(l, x, y)}}{dt} = -h\_{\text{exc}\_i}^{(l, x, y)} + g\_{\text{exc}} ( z^{(l)}, h\_{\text{exc}}^{(l)}, h\_{\text{inh}}^{(l)}, i, x, y )
$$

$$
g\_{\text{exc}} ( z^{(l)}, h\_{\text{exc}}^{(l)}, h\_{\text{inh}}^{(l)}, i, x, y ) = f( [ W\_{\text{input} \to \text{exc}}^{(l)} z^{(l)} + \lfloor W\_{\text{exc} \to \text{exc}}^{(l)} \rfloor\_{{}\_{{}\_+}} h\_{\text{exc}}^{(l)} +
\lfloor W\_{\text{inh} \to \text{exc}}^{(l)} \rfloor\_{{}\_{{}\_-}} h\_{\text{inh}}^{(l)} ]\_{i, x, y} + b\_{\text{exc}\_i})
$$

$$
\tau\_{\text{inh}\_j}^{(l)} \frac{d h\_{\text{inh}\_j}^{(l, x, y)}}{dt} = -h\_{\text{inh}\_j}^{(l, x, y)} + g\_{\text{inh}} ( z^{(l)}, h\_{\text{exc}}^{(l)}, h\_{\text{inh}}^{(l)}, j, x, y )
$$

$$
g\_{\text{inh}} ( z^{(l)}, h\_{\text{exc}}^{(l)}, h\_{\text{inh}}^{(l)}, j, x, y ) = f( [ W\_{\text{input} \to \text{inh}}^{(l)} z^{(l)} + \lfloor W\_{\text{exc} \to \text{inh}}^{(l)} \rfloor\_{{}\_{{}\_+}} h\_{\text{exc}}^{(l)}
 ]\_{j, x, y} + b\_{\text{inh}\_j})
$$

$\tau\_{\text{exc}\_i}^{(l)}$ and $\tau\_{\text{inh}\_j}^{(l)}$ are cell-type specific learnable neural time constants. Synaptic connections $\boldsymbol{W}$s are sparse matrices on which we impose translational invariance. These are, in practice, realized as convolutions. $b\_{\text{exc}\_i}$ and $b\_{\text{inh}\_j}$ are excitatory and inhibitory cell-type specific thresholds. $f(.)$ is a non-linear activation function. We use the hyperbolic tangent as our activation function $f$. An average pooling operation $\texttt{Pool}$ is applied to layer pyramidal outputs ($h\_{\text{exc}}$) to increase the receptive field size by a factor of two.

$$
z^{(l + 1)}[t] = \texttt{Pool} ( h\_{\text{exc}}^{(l)} [t]) \odot \Gamma(\xi(h\_{\text{exc}}^{(l)}[t - T])) \\ \texttt{if}\\ t\\  \geq T \\ \texttt{else}\\  \texttt{Pool} ( h\_{\text{exc}}^{(l)} [t]) 
$$

We make discrete time approximations to train our model. The cue is presented first to the network ($z^{(1)}[0])$) and the dynamics are unrolled for $T$ steps followed by scene presentation ($z^{(1)}[T])$) for another $T$ steps. While the scene is presented, inputs to each layer are modulated as follows, where  $\xi(.)$ is a pooling operator that computes the average activity per cell type across all $(x,y)$. $\Gamma(\textbf{e})$ is the low-rank modulation function defined as follows:

$$
\Gamma(\textbf{e}) = \textbf{e} \odot \sigma \left( [\textbf{W}\_{l,1} \textbf{e}^{T}  + \textbf{b}\_1] \otimes [\textbf{W}\_{l,2} \textbf{e}^{T} + \textbf{b}\_2 ] \right)
$$

Here, $\textbf{W}\_{l,1}, \textbf{W}\_{l,2}$ are learnable linear projections and $\textbf{b}\_1, \textbf{b}\_2$ are learnable biases. $\otimes$ denotes outer product and $\odot$ denotes pointwise scaling. By construction, the output of $[\textbf{W}\_{l,1} \textbf{e}^{T}  + \textbf{b}\_1] \otimes [\textbf{W}\_{l,2} \textbf{e}^{T} + \textbf{b}\_2 ]$ is a low-rank matrix.

Meta-review: This paper introduces a neural network framework designed to understand and implement the role of contextual modulation in visual processing. Given the extensive presence of feedback and lateral connections in the visual system and the critical role of contextual processing in both natural and artificial neural networks, I believe this work marks a significant advancement. It provides a framework for future studies exploring contextual processing through modulatory connections in both neuroscience and deep learning.

The paper received largely positive reviews. The reviewers highlighted the novel perspective and framework introduced by the authors, including the model's success in qualitatively replicating human search behavior, learning contextual search tasks, and outperforming other baseline models. However, the reviewers also raised several significant concerns regarding the study's scope and the need for additional analyses. Specifically, they pointed out issues such as the relevance of the work in naturalistic vision settings, the absence of mechanistic explanations for the model's behavior, and the lack of comparisons with similar models. From my reading of the discussions, these concerns do not appear to have been fully addressed by the authors.

While these comments are important and could certainly improve the paper, I believe that, considering the generally positive feedback from the reviewers, the paper merits acceptance as a poster presentation.
Decision: Accept (poster)