Flexible Context-Driven Sensory Processing in Dynamical Vision Models

Authors: narasimhan  lakshmi
  govindarajan
abhiram  iyer
valmiki  kothare
ila  fiete
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:59:21

Abstract: Visual representations become progressively more abstract along the cortical hierarchy. These abstract representations define notions like objects and shapes, but at the cost of spatial specificity. By contrast, low-level regions represent spatially local but simple input features. How do spatially non-specific representations of abstract concepts in high-level areas flexibly modulate the low-level sensory representations in appropriate ways to guide context-driven and goal-directed behaviors across a range of tasks? We build a biologically motivated and trainable neural network model of dynamics in the visual pathway, incorporating local, lateral, and feedforward synaptic connections, excitatory and inhibitory neurons, and long-range top-down inputs conceptualized as low-rank modulations of the input-driven sensory responses by high-level areas. We study this ${\bf D}$ynamical ${\bf C}$ortical ${\bf net}$work ($DCnet$) in a visual cue-delay-search task and show that the model uses its own cue representations to adaptively modulate its perceptual responses to solve the task, outperforming state-of-the-art DNN vision and LLM models. The model's population states over time shed light on the nature of contextual modulatory dynamics, generating predictions for experiments. We fine-tune the same model on classic psychophysics attention tasks, and find that the model closely replicates known reaction time results. This work represents a promising new foundation for understanding and making predictions about perturbations to visual processing in the brain.

Main paper: 
 introduction: 
 We readily use abstract cues to modulate our sensory perception. These include cues to attend to abstract high-level features (find Waldo; count the number of hoop shots, etc.) or low-level features (find the red items, vertically oriented bars, etc.). Such cue-based modulations of the visual pathway allow us to locate items of interest more rapidly and accurately, and to perform goal-directed computations.Understanding how top-down modulatory cues are represented and then interact with bottom-up sensory-driven neural responses has been a longstanding goal in computational cognitive neuroscience [1] [2] [3] [4] . Extensive psychophysical experiments [5] and studies showing that individual neurons whose receptive fields are aligned with the attentional cue exhibit a gain modulation [6, 7] shed light on the phenomenon. However, a fundamental circuit-level and conceptual problem remains open: what is the nature of the "modulatory homunculus" that knows, given various goals and context cues, which low-level representations to modulate, in which combination and which topographic part of the representational space in different processing layers? We lack a cohesive computational framework to link these two levels of representation. Simultaneously, we lack models of sensory processing that fully take into account the recurrent and temporally unfolding nature of computation in the brain; thus, they fall short of explaining phenomena like reaction time variations with task difficulty and the sharpening of perception as information is integrated within a trial.We combine known architectures of visual cortex with advances in machine learning to introduce a biophysically-inspired model, the Dynamical Cortical network (DCnet), to solve cue-delay-visual search tasks (Figure . 1). The model is endowed with several relevant properties of biological circuits, including separate (tuned) excitatory and (weakly-tuned) inhibitory populations, lateral inhibition (intra-area recurrence), and neuron types with distinct learnable time constants. We operationalize the "modulatory homunculus" as multiplicative low-rank perturbations from higher-order cortical and thalamic areas. Specifically, these low-rank perturbations arise from the model's own sensory responses from earlier times within the trial.Contributions. In this work, we focus on analyzing and interpreting the internal dynamics and behavioral modes of our biophysically inspired model on a suite of visual cue-mediated tasks.• We introduce vis-count, a novel, challenging, and parametrically generated cue-delay-visual search task. A visual cue (consisting of a color, a shape, or a color-shape conjunction) specifies which objects in a subsequently presented scene of geometric objects to count.• We present a biologically realistic model of the brain's visual system, the Dynamical Cortical network (DCnet), that is relatively shallow with local and top-down feedback and separate E/I neurons, which is capable of top-down attentional modulation based on previously presented cues and processes information over time. Our framework is among the first to link levels of analyses from physiology to behavior via computations and is a necessary first step toward hypothesis generation for neuroscience.• Our model outperforms state-of-the-art standard DNNs and LLMs on the task, while being interpretable and having orders of magnitude fewer parameters.• We perform in silico electrophysiology of the circuit's population dynamics to show that cue-based modulation drives a divergence of the bottom-up responses from the uncued case.• We can fine-tune the same model on new stimulus sets corresponding to classic human psychophysics attention tasks. Reaction time analogues in our model closely replicate experimental observations.In sum, these contributions suggest that our approach is a promising framework for modeling the brain's visual processing dynamics, one that replicates many key attentional phenomena and that can generate testable hypotheses and predictions about circuit mechanisms. 
related work: 
 Stimulus computable models of visual cognition. Modeling top-down contextual effects on bottom-up sensory processing during visual search is of longstanding interest in the vision sciences community. There are a large number of phenomenological models of visual search and associated reaction time findings [8] [9] [10] [11] [12] , but only recently have models been able to operate directly on high dimensional sensory inputs [13] [14] [15] [16] . The most promising approach involves augmenting pre-attentive ventral stream models with controllers either via attention maps [13] or multiplicative modulation factors on network activities [17] . However, these models cannot be used to faithfully study sensory processing dynamics because they are purely feedforward. Recurrent models of early sensory processing with lateral feedback and distinct excitatory and inhibitory populations [18] [19] [20] , and top-down feedback [21] have shown promise in accounting for contextual visual computations and human reaction times on visual cognitive tasks. However, to our knowledge, stimulus-computable recurrent vision models compatible with cue-delay-target paradigms do not exist. More generally, models have tended to either be stimulus-computable or grounded in biological realism, but usually not both. Low-rank interactions. Neuronal networks in the brain can be involved in disparate computations simultaneously [22] . Computational neuroscience research has focused on understanding how such neural networks can represent task-specific information and, relatedly, its consequence on the system's overall behavior. Theoretical results suggest that the coexistence of structured low-rank connectivity and random connectivity in networks can enable multi-task computations and expand the dynamic range of the network's functional capability [23] [24] [25] [26] [27] . An alternate but non-orthogonal viewpoint is that low-rank control inputs from an external system can switch a network's input-output mappings context-dependently [28] [29] [30] . Most of the computational work in this regard is, however, rooted in the motor domain, with Schmid and Neumann being a recent exception.Neurophysiology of attentional control. Biological attention upmodulates task-relevant information by filtering sensory responses using "templates". Empirical work shows that attentional templates are primarily represented in higher order cortical areas [31] [32] [33] [34] and that such templates can be learned rapidly on a per-task basis [35] . Moreover, the nature of such attentional filtering is known to be cell-type and layer-specific [36] , with theories emphasizing the role of top-down mediation of specific GABAergic interneurons [37] . In addition to cortical feedback inputs, higher-order thalamic nuclei are also known to convey contextual inputs to sensory regions either via direct projections [38, 39] or indirectly through the frontal cortices [40, 41] . Building biological sensory processing models that account for all these disparate findings is of primary importance to the Neuroscience community.3 General methods 
dcnet model and training details: 
 DCnet comprises two core components: a biologically motivated sensory perception stream and a higher-order area that interacts bidirectionally with it (Figure . 1). Each of the four sensory areas in our model are organized retinotopically as hypercolumns consisting of distinct excitatory and inhibitory neural subpopulations that obey Dale's law [42] . Excitatory pyramidal neurons receive bottom-up and recurrent lateral excitatory inputs as well as short-range lateral inhibitory inputs from interneurons in the same area. Interneurons receive bottom-up and lateral excitatory inputs. Finally, pyramidal neurons project in a feedforward manner to their downstream area. The ratio between excitatory and inhibitory neurons is 4 : 1, as observed in cortical areas [43] .Figure 2 : Explicit context-guided modulation of sensory dynamics is necessary for learning generalizable solutions. a. We introduce vis-count, a parametric, visually-cued, delayed search task. On each trial, models are cued with a visual attribute (either a color, shape, or a conjunction of the two), and after a delay, a scene is presented. The task is to count and report the number of cue-consistent objects in the scene. b. On each of the three types of trails, we find that our model consistently outperforms state-of-the-art standard DNNs (Section. 3.2; Baselines) on novel held-out scenes. Implicit models refer to the condition where cues and scenes are presented simultaneously. The red dashed line denotes chance performance. c. A harder test of generalization on novel scenes and cues reveal that our model is robust to such variations, unlike performant implicit models.The higher-order layer receives pyramidal input from each area and, in turn, modulates inter-area projections in a low-rank manner operationalized as follows.r 1 ; r 2 = ξ(h l t )W T l,1 + b 1 ; ξ(h l t )W T l,2 + b 2 # Compute the linear projections m l t = ξ(h l t ) [r 1 ⊗ r 2 ]# Compute the modulating factorsh l t+∆ = h l t+∆-ϵ ⊙ m l tHere, h l t ∈ R C×H×W is the excitatory population readout of area l ∈ [1..4] at time t; ξ(.) is the spatial average pooling operator; ⊗ denotes outer product and ⊙ denotes pointwise scaling.Neurons have learnable cell-type specific time constants. We dispense of traditional ML operations such as BatchNorm or LayerNorm, designed to impart stability during training. Our model has ∼ 1.8M parameters that we learn via gradient descent on a task-performance objective. A full mathematical specification of the model is provided in Appendix. A.1.We instantiate baseline models of two varieties. First, we consider a "Convolutional RNN" model (∼ 8.8M params) that uses a traditional 6-Layer convolutional backbone feeding into a gated recurrent unit (GRU) [44] with N = 2048 neurons (Appendix. B). We construct this baseline to evaluate the benefits of an explicit modulatory mechanism such as the higher-order layer in our model. Second, we consider four standard deep feedforward neural network architectures. As these models do not operate on spatiotemporal inputs, we condense trials to a single time point by stacking cues and scenes together. Given the lack of an explicit cue followed by scene phase for these models, we term them implicit. Cues and scenes are presented at the same time. These baselines provide an upper bound on the expected performance of DCnet and verify that our chosen task is a non-trivial computational challenge. In our experiments, we consider the following implicit models: a 6-Layer CNN (∼ 2.8M params); ResNet-18 [45] (∼ 11.5M params); ViT-B/16 [46] (∼ 86M params), a vision transformer with a patch size of 16px; Swin-T [47] (∼ 28M params), a hierarchical vision transformer. Furthermore, we also ran experiments on the zero-shot generalization abilities of an LLM (Appendix. C).  
vis-count: a parametric cued visual search task 4.1 task and stimuli: 
 We draw inspiration from Clevr [48] , a synthetic dataset for language-mediated visual reasoning and construct vis-count, a parametric visually-cued, delayed search task. On each trial, we challenge models to count and report the number of geometric objects in a visual scene with features consistent with a cue provided earlier (Figure . 2a ). Cues can be simple colors, geometric shapes, or a conjunction of the two. Scenes comprised 3 -10 geometric 3D shapes of varying sizes, colors, and material properties. By construction, there can be 0-5 cue-consistent objects in the scene. We counterbalanced the dataset so that the distribution of target counts was uniform across trial types. Cues and scenes were rendered at 320 × 240px and resized to 128 × 128px for model training and evaluation. In total, our training (validation) dataset comprised of ∼ 384K (38K) trials. As with Clevr, we detect and discard scenes with fully occluded objects.For DCnet and the Convolutional RNN baseline, cues and scenes are visual inputs provided to Layer 1 of these models. Cues are persistently provided to the network for the first T discrete time steps followed by scenes for the next T time steps. The output activities of the last layer at the last time step (t = 2T ) are transformed into logits, and a supervision signal is provided via a cross-entropy loss, with ground truth labels counted from 0 to 5. For the "implicit" baseline models, cues and scenes are stacked together and presented simultaneously.We report the results of our model evaluations and comparisons to baselines in Figure . 2b-c . Our model consistently and significantly outperforms state-of-the-art DNN vision models when evaluated on trials with novel held-out scenes, achieving 99%, 95%, 73% accuracy on the color, shape, and conjunction trials, respectively. Additionally, we perform a harder generalization test by synthesizing trials with novel cues and novel scenes (e.g., we test our model on blue and green colors, cube Figure 4 : A selectivity gradient emerges across the cortical layers for the different visual cues. We sequentially lesion the modulatory synapses in a trained DCnet from the higher-order layer to each sensory area, starting from the top-most (D) to the bottom-most (A) sensory area, while documenting task performance on each trial type. (E) indicates the intact DCnet. While late lesions have a minimal effect on performance in color trials, the opposite is true for shape trials. Early areas in the DCnet exhibit color selectivity, while later areas exhibit shape selectivity. shapes, and conjunctions thereof, all unseen in the training set). Our model demonstrated the best generalization performance (55%) while even the most performant baseline models (with orders of magnitude more parameters) dropped to chance (16.67%). Sample model results are visualized in Appendix. A.5.Model errors, too, can be particularly enlightening. Consider the last example in Figure . A.5. When cued to find "blue cylinders", the model mistakenly appears to include either the cyan cylinder or the occluded blue cube in its count, either of which would reflect an illusory conjunction [49] . Illusory conjunctions arise due to failures of feature-based attention and have been extensively studied in the human cognition literature. The presence of such pathologies in our model exposes an opportunity to gain potential insights into the neural implementations of feature-based attention.In the following sections, we report analyses of our model's learned dynamics and internal representations that support contextual guidance.What to modulate? Low-rank structures support optimal cue representations. In daily life, "cues" are not specified to us as abstract rules but rather as high-dimensional sensory inputs. How do we construct a representational space where abstract rules (derived from sensory responses) and the sensory responses themselves are distinct yet coexist? Following insights from electrophysiology [29] and prior theoretical work [23, 24, 28] , we hypothesized that learning to inject derived cue information back into sensory representations as low-rank perturbations will aid in optimally partitioning the sensory representational space.DCnet's task performance indicated that it had indeed learned task-optimal representations. Here, we perform dimensionality reduction on the activities of pyramidal neurons in the last layer of DCnet to probe how this optimality emerges.First, we observe that cued vs. uncued dynamics on individual trials become progressively divergent and nearly orthogonal with time (Figure . 3b ). This indicates that the bidirectional interactions likely promote the formation of low-dimensional activity subspaces embedded within the higherdimensional ambient activity space. Second, we see that the bottom-up responses to the same set of scenes are differentially modulated to appropriate target subspaces based on the cue (Figure . 3a ). We believe that this happens through the inactivation of context-irrelevant subspaces, resulting in invariance to current task-irrelevant features. As additional consideration, we note that training DCnet without this top-down feedback mechanism brings performance down to 38%, highlighting the importance of this interaction in our framework.Where to modulate? A cortical gradient for feature selectivity. After extracting abstract context rules from sensory responses, a question remains: At what level of the representational hierarchy must the perturbation be applied? We perform lesion analysis on the trained DCnet to probe this question (Figure . 4 ).We sequentially "turn off" modulations, area by area, and observe their detrimental effects on overall function as determined by task performance. We implement this by setting the modulating factors to 1. We hypothesized that if the modulation of pyramidal neuron activity in a given area is essential for the system's function, then lesioning this perturbation will result in a maximal drop in performance. Interestingly, this analysis yielded differential results per trial type (Figure . 4 ). For the color trials, we find that lesioning modulations in the early areas had the largest impact on performance, while for the shape trials, it was the late modulations that proved critical. In contrast, modulation strength seemed relatively uniform across the sensory areas.Taken together, these results offer two insights. First, opposing cortical gradients for color and shape selectivity emerge and coexist in the sensory areas despite the model only being supervised to "count". Second, leveraging this selectivity gradient, the higher-order area learns to apply appropriate low-rank modulations. We believe that the low-rank nature of the context-based modulation, not only within but also across areas, is fundamental to generalization.Excitation-Inhibition Dynamics. The amplification of context-relevant sensory representations must be balanced to support stable dynamics [50] . The neural underpinnings of this balance and its computational role in feature gain modulation have previously only been studied phenomenologically. Here, we leverage DCnet to investigate how the circuit-level properties discovered through optimization can support overall network function.We probe the cell-type specific neural dynamics in DCnet by driving network activity with uncorrelated, time-varying Gaussian noise inputs. First, despite fewer interneurons in the model, inhibitory interactions play a crucial role in imparting stability (Figure . 5b ) and expanding the dynamic range of computations expressed by the network when compared to a version of DCnet trained without inhibition (Figure . 5c ). Second, we detect the presence of co-tuned excitation and inhibition (Appendix. A.4). Empirically, co-tuning is known to be a common organizational principle across the sensory areas. These findings imply the critical role of interneurons in cortical amplification dynamics underlying context-dependent computations. Finally, we observe that neurons learn to integrate information faster in early vs. late areas, revealing the emergence of a macroscopic gradient in neural timescales (Figure . 5a, 8 ). 
model psychophysics on parametric cued feature searches: 
 A feature search is a variant of the general visual search problem in which a "target" is defined by a single discriminative feature [5] . Parametric variations along (or orthogonal to) this discriminative feature axis help shed light on the mechanisms of top-down contextual guidance and its impact on behavior. Here, we consider two feature search tasks from the human psychophysics literature. General methods. To perform model psychophysics, we start from the DCnet model trained on vis-count. We replace its readout to do binary classification (target present/absent) and fine-tune the model on each task below. Furthermore, to derive a measure of model "reaction time," we compute the entropy of DCnet's decision outputs after evolving dynamics for the duration of each trial.Task and stimuli. A target feature's bottom-up salience (pop-out) does not survive variations in irrelevant distractor features. (Figure . 6b) [5] . To test this, we construct cued-feature search trails where models search for an L/T (target) in a grid with a fixed number of Ts/Ls respectively (scene). Targets and scenes were rendered on a 128 × 128px canvas. Targets (L/T) were presented at the center and oriented uniformly at random between [0, 2π). The distractor heterogeneity level (ω) for every trial was uniform random between [0, 1]. Consequently, distractor orientations in the scene were uniform random between [θ -ω π 2 , θ + ω π 2 ] with θ ∈ [0, 2π) being the mean distractor orientation. The target was present in 50% of the trails. Our training (test) dataset comprised of 32K (8K) trials.Results. When fine tuned on this task, DCnet achieved an overall accuracy of 97% on the test trials. Moreover, faithful to the human psychophysics results, we find that the entropy of DCnet outputs is significantly different for low (ω <= 0.5) vs. high (ω > 0.5) distractor heterogeneity for both target-present (Mann-Whitney U = 302947., p < .001) and target-absent trials (Mann-Whitney U = 301376., p < .001). While target-absent trials yield higher reaction times in humans when compared to target-present trials, we don't see this in our model's output entropy. This possibly reflects an imperfect choice for a model reaction time measure, an aspect of our work that we look to extend upon in future work.Task and stimuli. In a classic critical color difference task, human participants were required to detect a target that differed from distracting stimuli only in color [51] . Search times were measured for varying color differences as a function of display density (the number of distractors). We parametrically synthesize stimuli to recreate this task (Figure . 7a-b ).Target and distractor stimuli were chosen to be circular discs of radii 10px. Cues were rendered at the center of a 128 × 128px canvas at a target color chosen at random from among four perceptually uniform color spaces. Search scenes consisted of 1 -7 distractor stimuli whose color differed from the target color at one of 10 preset target-distractor difference levels chosen uniformly at random. The target was present in 50% of the trails. In total, our training (test) dataset comprised of 32K (8K) trials. Results. When fine tuned on this task, DCnet achieved an overall accuracy of 95% on the test trials. Moreover, DCnet recapitulates two key findings from the psychophysics literature.We probe DCnet's output entropy (a proxy for model RT) as a function of the number of distractors in the scene. We find that for trials in which the target-distractor color difference was high, model entropy was invariant to the number of distracting stimuli (Pearson's r = -0.11). On trials where the target-distractor color difference was low, model entropy increased linearly with the number of distracting stimuli in the scene (Pearson's r = 0.55). We find a similar result when we probe DCnet's output entropy as a function of the target-distractor feature difference. These results faithfully replicate behavioral effects reported in prior literature [5, 51, 52] .As we alluded to in Section 5.1, our reaction time (RT) metric is one among many possible choices. To take a step further, we implement and test another RT metric inspired by evidential learning (EDL) theory as proposed in [20] . We verify that our primary conclusions from this experiment hold even in the context of this new metric. Finetuned on the EDL objective, DCnet achieves 86% accuracy on this task. Model RTs, computed as a time-averaged uncertainty measure, increased linearly with the number of distractors in the low target-distractor color difference trials (Pearson's r = 0.92) and was invariant to the number of distractors on high target-distractor color difference trials (Pearson's r = 0.2).Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: Yes, we provide a link to our code that is anonymized. We ensure that the code can be used to reproduce our main experimental results. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: Our work does not have a clear societal impact and thus we do not discuss this in our paper. This work is with regards to building biologically plausible vision models that can serve as hypothesis generators for real world experiments. These models will not be used by the average everyday user or affect everyday society in any regard. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
discussion: 
 The advent of highly-performant ventral stream models of visual perception is rapidly revolutionizing visual neuroscience research. Stimulus computable models operating directly on high dimensional sensory inputs yield benefits as hypothesis generators for scientific discovery. [53] [54] [55] . However, there exist two fundamental axes of dissonance, which are potential rate limitors. First, the emphasis on building "bigger and better" models promotes a divergence from biological realism [56] . Second, it is evident that the extent of biological visual capabilities spans a space bigger than one that entails only feedforward perceptual modules [1, 57, 58] . It calls for accounting for the cooperative dynamics between perceptual and cognitive processes [59] .In this work, we aim to bridge this gap by introducing a computational framework that emphasizes biological realism and conceptualizes interactions between perceptual dynamics and abstract cognitive demands while being scalable and stimulus-computable. We present the Dynamical Cortical network (DCnet): a trainable neural network model of visual dynamics incorporating local, lateral, and feedforward synaptic connections, excitatory and inhibitory neurons, and long-range top-down inputs conceptualized as low-rank modulations of the input-driven sensory responses by high-level areas.We start by studying the ability and behavior of DCnet to operate in a visually-cued search paradigm. DCnet not only outperforms state-of-the-art DNN models, but its population states over time reveal the computational structure and neural underpinnings of contextual modulatory dynamics, generating predictions for experiments. Furthermore, we fine-tune the same model to perform two classic 2AFC attention psychophysics tasks. Reaction time analogs from DCnet strikingly recapitulate core tenets of feature-based attentional modulation. We find that DCnet responses are sensitive to target-distractor feature differences, heterogeneity of irrelevant distractor features, and display density.Overall, these contributions suggest that our approach is a promising framework for modeling the brain's visual cortical dynamics, one that replicates key neural and behavioral signatures of contextual attentional modulation.Limitations and future directions. In this work, we take the first step towards building an overcomplete model of cortical circuitry. Palpable omissions from our current framework include long-range inter-area feedback and compartmental separation of feedforward and feedback inputs. We hope to continue building on this framework in the future to include these components that will open up novel avenues for computational neuroscience. Additionally, extracting reaction time measures from large-scale recurrent models is a discipline of its own [20] . We adopt a simple reaction time metric here that suffices for our current purpose, but we plan to include other sophisticated comparisons in future work. Lastly, we have restricted our purview to visual sensory processing. The concept of contextual modulation, however, is pervasive across sensory modalities. We are excited about extending our ideas to other sensory domains.Broader impact. Artificially intelligent models with enhanced visual capabilities are now pervasive in our daily lives. The not-so-hidden cost we pay to enjoy these models' benefits is their carbon footprint on our environment. Building energy-, parameter-, and sample-efficient models that are also performant is non-negotiable going forward. Understanding context-aware behavior is of utmost importance for neuroscience research as its failure modes are associated with several psychiatric and neuropathologies. We do not anticipate any negative impact that our work would create.We instantiate DCnet as a recurrent convolutional neural network model following the primate visual cortex's anatomical and neurophysiological properties. DCnet implements local, lateral, and feedforward synaptic connections, excitatory and inhibitory neurons, and long-range top-down inputs conceptualized as low-rank modulations of the input-driven sensory responses by high-level areas.Specified below are the governing dynamics for the neurons in our model. Neurons in our model are either excitatory (E) or inhibitory (I). l denotes the network layer (brain area). α l and β l are indices used to describe excitatory and inhibitory cell types, respectively, in layer l. (x, y) describes a specific spatial location. z (l) is the feedforward input to layer l. h refers to a neuron's instantaneous state.EQUATIONEQUATIONEQUATIONEQUATIONEQUATIONI , β l , x, y) = f ([W (l) input→I z (l) + ⌊W (l) E→I ⌋ + h (l) E ] β l ,x,y + b I β l ) τ E α land τ I β l are cell-type specific learnable neural time constants. Synaptic connections W 's are sparse matrices on which we impose translational invariance (details below). These are, in practice, realized as convolutions. b E α l and b I β l are excitatory and inhibitory cell-type specific learnable thresholds. f (.) is a non-linear activation function. We use the hyperbolic tangent as our activation function f . An average pooling operation Pool is applied to layer pyramidal outputs (h E ) to increase the receptive field size by a factor of two.z (l+1) [t] = Pool(h (l) E [t]) ⊙ Γ(ξ(h (l) E [t -T ])) t ≥ T Pool(h (l) E [t]) t < TWe make discrete time approximations to train our model. The cue is presented first to the network (z (1) [0])) and the dynamics are unrolled for T steps followed by scene presentation (z (1) [T ])) for another T steps. While the scene is presented, inputs to each layer are modulated as follows, where ξ(.) is a pooling operator that computes the average activity per cell type across all (x, y). Γ(e) is the low-rank modulation function defined as follows:Γ(e) = e ⊙ σ [W l,1 e T + b 1 ] ⊗ [W l,2 e T + b 2 ]Here, W l,1 , W l,2 are learnable linear projections and b 1 , b 2 are learnable biases. ⊗ denotes outer product and ⊙ denotes pointwise scaling. By construction, the output of[W l,1 e T +b 1 ]⊗[W l,2 e T +b 2 ]is a low-rank matrix.This section provides specific details on model parameters detailed in Section A.1. All layers have the following convolutional kernels (Ws specified in the governing equations): Input to excitation (W input→E ), excitation to excitation (W E→E ), excitation to inhibition (W E→I ), inhibition to excitation (W I→E ), and input to inhibition (W input→I ). The dimensionality of these convolutions is listed layer-wise below, along with the kernel and padding shapes for all convolutions in that layer. The outputs from the last layer are transformed via a linear readout parameterized as a fully connected layer (1024 × 6) into logits, which we supervise with a CrossEntropy loss. The model does not include standard embellishments like explicit normalization layers, model ensembling, test-time augmentations, etc.All models were trained on A100 GPUs for 100 epochs each. An experimental run took anywhere from 4-8 hours to complete. We used an AdamW optimizer (momentum=0.9, β 1 = 0.9, β 2 = 0.999), a one-cycle learning rate scheduler with a warm-up period of 30 epochs and a maximum learning rate of 4e -4. DCnet was 4 layers deep (∼ 1.8M learnable parameters) and was trained with batches of 256 samples. Code and datasets can be found here: Project repository.In the interest of mechanistic interpretability, we perform a series of experiments on DCnet. We drive activity by uncorrelated, time-varying Gaussian noise inputs. We then compute:1. The Lag-1 autocorrelation as a measure of stability of the excitatory neurons (Fig. 5b ). We find that DCnet excitatory neurons are significantly more stable than excitatory neurons in the DCnet (trained w/o inhibition) model. A Kolmogorov-Smirnov test confirmed the significant difference (statistic=1.0, p < .001). 2. The Dynamic Range (DR) of excitatory cells in each layer of DCnet and compare that to the DR of corresponding excitatory cells from DCnet (trained w/o inhibition) (Fig. 5c ). DR is computed as the Interquartile range (a measure of statistical dispersion) of a neuron's activity over a time period of 128ms when driven by noise. We take the mean over 64 trials for each neuron and plot this distribution per layer in Fig. 5c . We find that excitatory neurons in the DCnet model have a significantly higher DR across layers compared to DCnet (trained w/o inhibition), suggesting the role of inhibitory interactions in expanding the range of computations carried out by each neuron. A Kolmogorov-Smirnov test confirmed these significant differences (Layer 1 (statistic=0.667, p < .001); Layer 2 (statistic=0.99, p < .001); Layer 3 (statistic=1.0, p < .001); Layer 4 (statistic=0.97, p < .001)).3. The E-I correlation coefficient as a measure of co-tuning in DCnet. We find that the average (across neurons) E-I correlation is as follows: -0.076 (Layer 1), 0.766 (Layer 2), 0.699 (Layer 3), 0.535 (Layer 4).Additionally, we also highlight a macroscopic gradient in learned time constants across the layers of DCnet (Fig. 8 ).Prompt Here is a task that takes two images, a cue and a scene. Here are the instructions:1. If the cue is a color, return a count of how many objects in the scene have that color. 2. If the cue is an orange colored shape, return a count of all objects in the scene that have that shape. 3. If the cue is a non-orange colored shape, return a count of all objects in the scene that have that shape and color. You will only need to follow 1 of the 3 instructions above. Don't write code.There is no need to explain the logic of your counting. Simply provide the final answer.GPT4 Sure, I can help with that. Please upload the two images you have: the cue image and the scene image, and I'll proceed according to your instructions.ChatGPT4 experiments We prompted GPT4 to solve solve vis-count. Given a color, shape, or conjunction cue, we asked the LLM to identify how many objects in the provided scene match the cue's properties. Out of 30 trials, GPT4 achieved a success rate of 37%.• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [NA] Justification: Our paper does not involve crowdsourcing research or collecting humans in any way. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: This paper introduces a neural network framework designed to understand and implement the role of contextual modulation in visual processing. Given the extensive presence of feedback and lateral connections in the visual system and the critical role of contextual processing in both natural and artificial neural networks, I believe this work marks a significant advancement. It provides a framework for future studies exploring contextual processing through modulatory connections in both neuroscience and deep learning.

The paper received largely positive reviews. The reviewers highlighted the novel perspective and framework introduced by the authors, including the model's success in qualitatively replicating human search behavior, learning contextual search tasks, and outperforming other baseline models. However, the reviewers also raised several significant concerns regarding the study's scope and the need for additional analyses. Specifically, they pointed out issues such as the relevance of the work in naturalistic vision settings, the absence of mechanistic explanations for the model's behavior, and the lack of comparisons with similar models. From my reading of the discussions, these concerns do not appear to have been fully addressed by the authors.

While these comments are important and could certainly improve the paper, I believe that, considering the generally positive feedback from the reviewers, the paper merits acceptance as a poster presentation.

Reviews: review: paper_summary: paper_summary: The paper presents a model for how high levels of (presumably cortical) representation modulate lower levels.  A multilayer neural network with recurrent connections both within each layer and between layers is trained on a visual cue-delay-search task.  The phenomenology of the model is then studied and characterized, showing how low level representations are modulated by task demands, which makes predictions for experiments.
strength_weakness: strength_weakness: - The framing of the central question of the paper in terms of a “modulatory homunculus” that must decide what features to attend to in lower-levels seems unnatural and odd.  It seems exclusively oriented toward modeling a specific laboratory task.  But what about natural vision?   It would seem these recurrent connections must always be in play for myriad tasks in daily life.  It would have been more compelling to see the model trained on a broader range of tasks - e.g., visual scene analysis - rather than this specific laboratory task.
- There are many recurrent interactions in the model, but the results focus more on characterizing the phenomenology of the model as opposed to what is learned by the recurrent weight matrices, which from a mechanistic point of view would seem most interesting.  It would also be worthwhile to start with a minimal model that demonstrates the principles of what is being learned, as opposed to throwing it all in the kitchen sink of the model shown in Figure 1.
- Because the model has so many components and it is not well-explained in the main body (including not defining all the variables in the equation - see below), by the time you get to results there is still no intuition for how the model is cued and performs the task, or how it learns.
- Unclear if the comparison to models without time dynamics and explicit cuing is informative.

Other comments:
- homunculus is misspelled both times
- 65: I don’t think there are explicit testable predictions/potential experiments that come from the model anywhere in the paper
- 110: Even if there is more model specification in appendix, one should at least define all the variables and give basic intuition for the equations. What is r_1, r_2 ?  W? Where are the time constants? Not clear how the variables correspond to interneurons, lateral connections, etc. What is the role of the spatial pooling operator (not explained even in the appendix)?
- Fig 2b: does not seem to actually outperform ResNet, even though it seems like it might have an advantage over ResNet because of cuing (though this is unclear because they don't explain cuing)
- 120: Besides the Conv RNN model (maybe?), are these comparable baselines if they have no temporal dynamics?
- 144: Don't seem to describe the novel cues and scenes anywhere. How different are they from the training data? Is a 55% accuracy impressive?
- 148: How do they know it included the cyan cylinder and not another object (and therefore conclude it's an illusory conjunction)? Could it also be confused by the occlusion of the blue cube?
- 169: "We believe"... this whole paragraph seems like speculation based on looking at Fig. 2, but no actual proof/argument?
- 184: Unclear what "cortical gradients" means, I'm assuming trajectory in PCA space? Is it that interesting/surprising that trajectories are different for color and shape given the model presumably would have to disentangle in order to get high accuracy?
questions: questions: see above
limitations: limitations: see above
paper_summary: paper_summary: The authors introduce DCNet, a hierarchical recurrent vision model that draws inspiration from the structure and function of biological visual circuits. This novel architecture consists of excitatory and inhibitory bottom-up neurons modulated by a (low-rank) representation of a higher area, analogous to the higher cortical and thalamic regions of the brain. Through their proposed model dynamics, the authors demonstrate emergent task-driven contextualization of bottom-up responses. Experimental results show that DCNet successfully replicates human behavioral findings on various classic psychophysics-inspired visual cue-delay search tasks, highlighting the model's potential in understanding and mimicking human visual perception.
strength_weakness: strength_weakness: - While the proposed DCNet model is intriguing, it would be beneficial to more explicitly highlight its anatomical constraints in relation to prior work. Conversely, the authors could further emphasize the novelty and significance of the low-rank modulation aspect, which appears to be a unique contribution. Additional signposting would help to clarify its impact on the observed results.
- To strengthen the paper, the authors may consider including ablation studies to dissect the contributions of individual components within the DCNet architecture. This would provide valuable insights into which elements are crucial for the model's high performance and alignment with human behavior.
- The authors cite relevant models in references [18-21], but it is unclear how these models perform on the vis-count task. A more comprehensive comparison would be helpful, including a detailed discussion of the differences between these models and DCNet. This would enable a more nuanced understanding of the proposed model's advantages and limitations.
questions: questions: Please refer to my review above.
limitations: limitations: The authors have addressed limitations adequately.
paper_summary: paper_summary: The authors present a convolutional dynamical systems model if visual processing with separate excitatory and inhibitory populations in each layer. They add a low rank modulation of each layer by a factor that is computed as a linear map of the layer activities. This is meant to represent top down modulations by some higher level area. The authors train their network on visual search tasks, including a new one which requires participants to count the number of objects in a scene that match a target in color, shape, or both. The context modulation is required for good performance of the model and the model reproduces two effects from the visual search literature.
strength_weakness: strength_weakness: All results to evaluate the network are qualitative, in contrast to claims of the authors that their network closely replicates known reaction time results. 
While higher distractor heterogeneity, which makes humans slower, makes the network more uncertain (6c), target absent trials usually yield longer action times while the entropy of the network is lower here. So even the qualitative result is not as in humans.
The positive slope in 7c seems to rely heavily on the data point at 1 distractor, does this actually keep growing?
For 7d I would love some comparison to human data as the feature difference does not tell me much such that I cannot judge whether this actually matches human behaviour.
Also, It is customary to look at the accuracy of responses as well somewhere.

While the proposed network does perform better than standard DNNs the difference is small, especially to ResNet-18. Also the generalization gap is quite substantial for the proposed network as well. So overall the performance improvement is marginal.

While the authors comment on the dynamics observed in their model in Figure 5 and the accompanying text, I am not really following this part. Proper analyses of the dynamics should contain some quantitative measures of the dynamics and comparisons of these to neural data. Just looking at a set of randomly chosen activity traces is insufficient I think.
questions: questions: Did the authors explore variants of the model where the modulation depends on all layers, has a memory or similar ideas? For a higher order representation, the current implementation seems extremely reduced.

Did the authors run any quantitative comparisons between their model and any neural or psychophysical measurements.
limitations: limitations: I agree with the authors that their work is unlikely to have negative societal impact. Limitations of their work could have been discussed more carefully though.
paper_summary: paper_summary: Artificial neural networks are loosely based on biological neural networks. In this study, the authors construct neural networks whose structures are based on general principles of visual signal pathway such as retinotopy. They used this model to ask if the context-dependent feedback mimicking top-down signals in the brain could help artificial neural networks perform cue-delay tasks more efficiently.
strength_weakness: strength_weakness: The paper is well written and easy to follow, but this reader finds it a little strange that the authors did not specify basic information on their model or baseline models. First, they provided mathematical descriptions of the recurrent “neuron” in the model, but it remains unclear how they implement these neurons. Based on the given source codes and the appendix, this reader thinks that individual neurons are 2D convolutional filters and that the intermediate states are used to realize recurrent behaviors. Additionally, 2D convolutional filters have multiple parameters and are not listed in the manuscript. I did find some parameters in the source codes, but they are not good enough. Second, the authors should provide more details on the baseline models. They mentioned “Traditional 6-Layer convolutional backbone feeding into a gated recurrent unit (GRU) [44] with N = 2048 neurons”, but the meaning of traditional 6-layer convolutional backbone is unclear, and they did not explain how 2048 GRU neurons provide the final answers. Once again, the authors need to provide pertinent details of each component. Third, the modulation signals are computed with two linear projects r1 and r2. It looks like they were estimated from all 4 layers, but it is unclear how they are concatenated to create r1 and r2.
questions: questions: Please see the weakness section.
limitations: limitations: The authors provided the limitations of their study.

Comments: title: further experiments with a better motivated reaction time metric
feedback on our rebuttal
Thank you!
Any response to our rebuttal?
Read Rebuttal and keep rating
Rebuttal, Part 3
Rebuttal, Part 2
comment: Per your comment and the current timeframe of the discussion period, we have implemented a reaction time metric inspired by evidential learning (EDL) theory, as proposed in Goetschalckx et al. (2024). Furthermore, we render a version of the task presented in Fig. 7 with up to 15 distractors (we reduce the radii of the discs to achieve this while maintaining the overall canvas size). 

We test our model (that we trained with EDL on the original dataset) **zero-shot** on the new stimuli to study the effect of increasing the number of distractors on the model reaction time. Our model achieves an impressive $\sim 70$% OOD generalization accuracy. We present results from "correct" trials below. We find that the linear RT trend for the low target-distractor difference trials firmly holds as we increase the number of distractors. The slope for low target-distractor trials is positive and significantly higher than that obtained for the high target-distractor trials. 

| # distractors     | $\xi_{cRNN}$ (Low T-D difference) | $\xi_{cRNN}$ (High T-D difference) |
| ----------- | ----------- | ----------- |
| 1 | 0.11 | 0.15 |
| 3 | 0.21 | 0.16 |
| 5 | 0.37 | 0.20 |
| 7 | 0.51 | 0.21 |
| 9 | 0.58 | 0.24 |
| 11 | 0.69 | 0.24 |
| 13 | 0.76 | 0.28 |
| 15 | 0.77 | 0.31 | 

We hope we’ve convinced the reviewer of our framework's usefulness and general flexibility/applicability, regardless of the particular reaction time metric we choose to incorporate (a vast and ongoing area of research). 

We would appreciate if the reviewer could update their score if this response eases their concerns.
Thanks again for your positive evaluation of our manuscript. We hope you had a chance to review our rebuttal and the analyses presented there in. If you believe we had addressed your concerns, we would appreciate it if you can increase your score. Many thanks!
Thank you very much for your feedback and your revised evaluation recommending acceptance. In particular, thank you for highlighting the future promise of such work to serve as a bridge between neuroscientists and AI researchers.
We hope you had a chance to review our rebuttal. Given your detailed feedback and suggestions, we believe our paper has improved. Please let us know if there is anything else that we can clarify. We kindly ask you to consider revising your score if you agree that the primary concerns raised in the review were addressed.
Thank you for the clarification. This model stands between biologically realistic models, which focus on simulating realistic brain activity, and functional models, which are engineered to perform specific tasks. This type of model is rare and should be encouraged to foster meaningful interactions between neuroscience and AI research communities. As the author provided more clarification, I think it could be possible for this model to serve as a reference for those who are interested in building such models. Thus, I would like to increase my score to 6.
I just read through the authors responses and am going to keep the rating I gave initially, I thank the reviewers for their additional details.

In particular like to see that the model with attention driven by all layers works better, as I think that fits our knowledge about biology better than feedback computed separately per layer.

However, I still think proper quantitative comparisons are necessary to show that this model does indeed behave like a biological brain. If the authors believe that entropy is a bad predictor for RT, they should compare it something else like a confidence score and extract a sensible measure to predict RTs. They cite papers on how to do this. And for the dynamics I would really like to see some direct comparisons showing in what respects the dynamics are similar or different to some concrete actual measurements of brain data.

The limited evaluation in the rating text matches my impression quite well.
> **169: "We believe"... this whole paragraph seems like speculation based on looking at Fig. 2, but no actual proof/argument?**

We believe that the reviewer is referring to Fig. 3 here, and not Fig. 2. In Fig. 3a, we consider model trajectories of the *same* several hundred scenes (individual trajectories shown in gray) when modulated by four different cues (subpanels). Our point is better illustrated by Fig. R3. When a given scene is cued with different colors, dynamics are driven to context-relevant states though the bottom-up responses from the scene are exactly the same. This is possible only by the inactivation of context-irrelevant subspaces.
> **homunculus is misspelled both times**

Thank you for pointing this out. We have made the correction.

> **No explicit testable predictions/potential experiments that come from the model anywhere in the paper (L65)**

We believe our framework is among the first to link physiology to behavior (through computations) and is a necessary first step towards hypothesis generation. We will clarify this further in the manuscript.

> **Lack of clarity in model specification**

We apologize for this confusion. We take this comment seriously and we now spell out the entirety of our model formulation in the general response. We are happy to answer any further questions the reviewer might have.

> **Fig 2b: does not seem to actually outperform ResNet, even though it seems like it might have an advantage over ResNet because of cuing (though this is unclear because they don't explain cuing)**

We would like to clarify this point. In Fig. 2b what is shown is the performance of the models when the cues are in-distribution. As we point out in our answer above, the implicitly-cued models are supposed to serve as a potential performance **upper-bound**. The implicit-cue condition is computationally easier compared to the explicit-cue condition. We point the reviewer to Fig. 2c which shows that on a generalization condition, when the cues are out-of-distribution the same ResNet model falls to chance. 

> **Besides the Conv RNN model (maybe?), are these comparable baselines if they have no temporal dynamics?**

Thanks for raising this point. As we discussed above, the baselines were primarily constructed for two reasons. First, we wanted to understand performance bounds and verify that vis-count is a non-trivial computational challenge. Second, we wanted to study the computational implications (and benefits) of two components of our framework: temporal dynamics, and top-down feedback. The Conv RNN model has temporal dynamics, but no top-down modulation. The feedforward baselines have neither.

> **144: Don't seem to describe the novel cues and scenes anywhere. How different are they from the training data? Is a 55% accuracy impressive?**

Thanks for raising this point. The scenes are split into a training and test set. They are i.i.d. samples from our data generation process. None of the scenes used for evaluation were seen by the model during training. We perform two types of tests with the cues. The weak generalization experiment contained cues (colors, shapes, conjunctions) that the model had seen during training (but for different scenes). The stronger generalization experiment used cues never encountered in training. Specifically, for our stronger generalization experiment, we keep the color cues green and blue; the shape cue cube; and conjunctions of these out of the training set.

Chance performance for all of these tasks are $16.67$%. An accuracy of $55$% is well above chance and significantly higher than the performance of models with orders of magnitude more parameters (Fig. 2c; hatched data).

We will add these clarifications to the manuscript.

> **148: How do they know it included the cyan cylinder and not another object (and therefore conclude it's an illusory conjunction)? Could it also be confused by the occlusion of the blue cube?**

The reviewer is correct in pointing this out. Identifying either the blue-cube or the cyan-cylinder would constitute a “binding error”. We highlighted this example since the feature dissimilarity between the target object and distractors (apart from the cyan-cylinder and blue-cube) are very high and is unlikely to be the cause of the error. However, the reviewer’s general point is well taken and we will perform an in-depth model explainability analysis. In the meantime, we will reword this sentence to reflect that this is a possibility and not a certainty.

“When cued to find “blue cylinders", the model mistakenly appears to include either the cyan cylinder or the occluded blue cube in its count; which would count as an illusory conjunction”.


> **184: Unclear what "cortical gradients" means, I'm assuming trajectory in PCA space? Is it that interesting/surprising that trajectories are different for color and shape given the model presumably would have to disentangle in order to get high accuracy?**

We believe there is a misunderstanding here. In Fig. 4 we show the results of a lesion experiment we performed, where we systematically cut-out modulations from each layer of the model. Data presented in Fig. 4 are not trajectories in the PCA space. We find that lesioning later layer modulations in the model significantly impacted accuracy on the “shape” trials but not the “color” trials. Lesioning early layer modulation significantly impacted accuracy on “color” trials. From these we conclude that color selectivity emerges early while shape selectivity emerges late. The selectivity profile across layers of the model is what we refer to as a “cortical gradient”.