PenguinMan/ARXIV huggingface dataset:
How pupil responses track value-based decision-making during and after reinforcement learning
Authors: Joanne C Van Slooten, Sara Jahfari, Tomas Knapen, Jan Theeuwes
Journal: PLOS Computational Biology
Accepted: 2018-11-08T00:00:00Z


Cognition can reveal itself in the pupil, as latent cognitive processes map onto specific pupil responses. For instance, the pupil dilates when we make decisions and these pupil size fluctuations reflect decision-making computations during and after a choice. Surprisingly little is known, however, about how pupil responses relate to decisions driven by the learned value of stimuli. This understanding is important, as most real-life decisions are guided by the outcomes of earlier choices. The goal of this study was to investigate which cognitive processes the pupil reflects during value-based decision-making. We used a reinforcement learning task to study pupil responses during value-based decisions and subsequent decision evaluations, employing computational modeling to quantitatively describe the underlying cognitive processes. We found that the pupil closely tracks reinforcement learning processes independently across participants and across trials. Prior to choice, the pupil dilated as a function of trial-by-trial fluctuations in value beliefs about the to-be chosen option and predicted an individual’s tendency to exploit high value options. After feedback a biphasic pupil response was observed, the amplitude of which correlated with participants’ learning rates. Furthermore, across trials, early feedback-related dilation scaled with value uncertainty, whereas later constriction scaled with signed reward prediction errors. These findings show that pupil size fluctuations can provide detailed information about the computations underlying value-based decisions and the subsequent updating of value beliefs. As these processes are affected in a host of psychiatric disorders, our results indicate that pupillometry can be used as an accessible tool to non-invasively study the processes underlying ongoing reinforcement learning in the clinic.
Author summary: It has long been known that the pupil dilates when we decide. These pupil dilations have predominantly been linked to arousal. However, reward-related processes may trigger pupil dilations as well, as dilations have been linked to activity in the dopaminergic midbrain, a region important for reward processing and reinforcement learning. Using a learning task and a computational model to quantitatively describe the cognitive processes that drive reinforcement learning behavior, we show that the pupil closely tracks different aspects of the reinforcement learning process. Prior to making a value-based choice, pupil dilation reflected the value of the soon-to-be-chosen option. After receiving choice feedback, early dilation reflected uncertainty about the value of recent choice options, while late constriction reflected how strongly an outcome violated current value beliefs. These findings provide the novel insight that the pupil can be used to track value-based decision-making, opening up a new method for online tracking of reinforcement learning processes.
Visual deprivation independent shift of ocular dominance induced by cross-modal plasticity
Authors: Manuel Teichert, Marcel Isstas, Lutz Liebmann, Christian A Hübner, Franziska Wieske, Christine Winter, Konrad Lehmann, Jürgen Bolz
Journal: PLOS ONE
Accepted: 2019-02-25T00:00:00Z


There is convincing evidence that the deprivation of one sense can lead to adaptive neuronal changes in spared primary sensory cortices. However, the repercussions of late-onset sensory deprivations on functionality of the remaining sensory cortices are poorly understood. Using repeated intrinsic signal imaging we investigated the effects of whisker or auditory deprivation (WD or AD, respectively) on responsiveness of the binocular primary visual cortex (V1) in fully adult mice. The binocular zone of mice is innervated by both eyes, with the contralateral eye always dominating V1 input over ipsilateral eye input, the normal ocular dominance (OD) ratio. Strikingly, we found that 3 days of WD or AD induced a transient shift of OD, which was mediated by a potentiation of V1 input through the ipsilateral eye. This cross-modal effect was accompanied by strengthening of layer 4 synapses in V1, required visual experience through the ipsilateral eye and was mediated by an increase of the excitation/inhibition ratio in V1. Finally, we demonstrate that both WD and AD induced a long-lasting improvement of visual performance. Our data provide evidence that the deprivation of a non-visual sensory modality cross-modally induces experience dependent V1 plasticity and improves visual behavior, even in adult mice.
Sensory coding and contrast invariance emerge from the control of plastic inhibition over emergent selectivity
Authors: René Larisch, Lorenz Gönner, Michael Teichmann, Fred H Hamker
Journal: PLOS Computational Biology
Accepted: 2021-10-15T00:00:00Z


Visual stimuli are represented by a highly efficient code in the primary visual cortex, but the development of this code is still unclear. Two distinct factors control coding efficiency: Representational efficiency, which is determined by neuronal tuning diversity, and metabolic efficiency, which is influenced by neuronal gain. How these determinants of coding efficiency are shaped during development, supported by excitatory and inhibitory plasticity, is only partially understood. We investigate a fully plastic spiking network of the primary visual cortex, building on phenomenological plasticity rules. Our results suggest that inhibitory plasticity is key to the emergence of tuning diversity and accurate input encoding. We show that inhibitory feedback (random and specific) increases the metabolic efficiency by implementing a gain control mechanism. Interestingly, this led to the spontaneous emergence of contrast-invariant tuning curves. Our findings highlight that (1) interneuron plasticity is key to the development of tuning diversity and (2) that efficient sensory representations are an emergent property of the resulting network.
Author summary: Synaptic plasticity is crucial for the development of efficient input representation in the different sensory cortices, such as the primary visual cortex. Efficient visual representation is determined by two factors: representational efficiency, i.e. how many different input features can be represented, and metabolic efficiency, i.e. how many spikes are required to represent a specific feature. Previous research has pointed out the importance of plasticity at excitatory synapses to achieve high representational efficiency and feedback inhibition as a gain control mechanism for controlling metabolic efficiency. However, it is only partially understood how the influence of inhibitory plasticity on excitatory plasticity can lead to an efficient representation. Using a spiking neural network, we show that plasticity at feed-forward and feedback inhibitory synapses is necessary for the emergence of well-distributed neuronal selectivity to improve representational efficiency. Further, the emergent balance between excitatory and inhibitory currents improves the metabolic efficiency, and leads to contrast-invariant tuning as an inherent network property. Extending previous work, our simulation results highlight the importance of plasticity at inhibitory synapses.
Thunderstruck: The ACDC model of flexible sequences and rhythms in recurrent neural circuits
Authors: Cristian Buc Calderon, Tom Verguts, Michael J Frank
Journal: PLOS Computational Biology
Accepted: 2022-01-21T00:00:00Z


Adaptive sequential behavior is a hallmark of human cognition. In particular, humans can learn to produce precise spatiotemporal sequences given a certain context. For instance, musicians can not only reproduce learned action sequences in a context-dependent manner, they can also quickly and flexibly reapply them in any desired tempo or rhythm without overwriting previous learning. Existing neural network models fail to account for these properties. We argue that this limitation emerges from the fact that sequence information (i.e., the position of the action) and timing (i.e., the moment of response execution) are typically stored in the same neural network weights. Here, we augment a biologically plausible recurrent neural network of cortical dynamics to include a basal ganglia-thalamic module which uses reinforcement learning to dynamically modulate action. This “associative cluster-dependent chain” (ACDC) model modularly stores sequence and timing information in distinct loci of the network. This feature increases computational power and allows ACDC to display a wide range of temporal properties (e.g., multiple sequences, temporal shifting, rescaling, and compositionality), while still accounting for several behavioral and neurophysiological empirical observations. Finally, we apply this ACDC network to show how it can learn the famous “Thunderstruck” song intro and then flexibly play it in a “bossa nova” rhythm without further training.
Author summary: How do humans flexibly adapt action sequences? For instance, musicians can learn a song and quickly speed up or slow down the tempo, or even play the song following a completely different rhythm (e.g., a rock song using a bossa nova rhythm). In this work, we build a biologically plausible network of cortico-basal ganglia interactions that explains how this temporal flexibility may emerge in the brain. Crucially, our model factorizes sequence order and action timing, respectively represented in cortical and basal ganglia dynamics. This factorization allows full temporal flexibility, i.e. the timing of a learned action sequence can be recomposed without interfering with the order of the sequence. As such, our model is capable of learning asynchronous action sequences, and flexibly shift, rescale, and recompose them, while accounting for biological data.
Spike burst-pause dynamics of Purkinje cells regulate sensorimotor adaptation
Authors: Niceto R Luque, Francisco Naveros, Richard R Carrillo, Eduardo Ros, Angelo Arleo
Journal: PLOS Computational Biology
Accepted: 2019-01-08T00:00:00Z


Cerebellar Purkinje cells mediate accurate eye movement coordination. However, it remains unclear how oculomotor adaptation depends on the interplay between the characteristic Purkinje cell response patterns, namely tonic, bursting, and spike pauses. Here, a spiking cerebellar model assesses the role of Purkinje cell firing patterns in vestibular ocular reflex (VOR) adaptation. The model captures the cerebellar microcircuit properties and it incorporates spike-based synaptic plasticity at multiple cerebellar sites. A detailed Purkinje cell model reproduces the three spike-firing patterns that are shown to regulate the cerebellar output. Our results suggest that pauses following Purkinje complex spikes (bursts) encode transient disinhibition of target medial vestibular nuclei, critically gating the vestibular signals conveyed by mossy fibres. This gating mechanism accounts for early and coarse VOR acquisition, prior to the late reflex consolidation. In addition, properly timed and sized Purkinje cell bursts allow the ratio between long-term depression and potentiation (LTD/LTP) to be finely shaped at mossy fibre-medial vestibular nuclei synapses, which optimises VOR consolidation. Tonic Purkinje cell firing maintains the consolidated VOR through time. Importantly, pauses are crucial to facilitate VOR phase-reversal learning, by reshaping previously learnt synaptic weight distributions. Altogether, these results predict that Purkinje spike burst-pause dynamics are instrumental to VOR learning and reversal adaptation.
Author summary: Cerebellar Purkinje cells regulate accurate eye movement coordination. However, it remains unclear how cerebellar-dependent oculomotor adaptation depends on the interplay between Purkinje cell characteristic response patterns: tonic, high frequency bursting, and post-complex spike pauses. We explore the role of Purkinje spike burst-pause dynamics in VOR adaptation. A biophysical model of Purkinje cell is at the core of a spiking network model, which captures the cerebellar microcircuit properties and incorporates spike-based synaptic plasticity mechanisms at different cerebellar sites. We show that Purkinje spike burst-pause dynamics are critical for (1) gating the vestibular-motor response association during VOR acquisition; (2) mediating the LTD/LTP balance for VOR consolidation; (3) reshaping synaptic efficacy distributions for VOR phase-reversal adaptation; (4) explaining the reversal VOR gain discontinuities during sleeping.
Integrated neural dynamics of sensorimotor decisions and actions
Authors: David Thura, Jean-François Cabana, Albert Feghaly, Paul Cisek
Journal: PLOS Biology
Accepted: 2022-09-29T00:00:00Z


Recent theoretical models suggest that deciding about actions and executing them are not implemented by completely distinct neural mechanisms but are instead two modes of an integrated dynamical system. Here, we investigate this proposal by examining how neural activity unfolds during a dynamic decision-making task within the high-dimensional space defined by the activity of cells in monkey dorsal premotor (PMd), primary motor (M1), and dorsolateral prefrontal cortex (dlPFC) as well as the external and internal segments of the globus pallidus (GPe, GPi). Dimensionality reduction shows that the four strongest components of neural activity are functionally interpretable, reflecting a state transition between deliberation and commitment, the transformation of sensory evidence into a choice, and the baseline and slope of the rising urgency to decide. Analysis of the contribution of each population to these components shows meaningful differences between regions but no distinct clusters within each region, consistent with an integrated dynamical system. During deliberation, cortical activity unfolds on a two-dimensional “decision manifold” defined by sensory evidence and urgency and falls off this manifold at the moment of commitment into a choice-dependent trajectory leading to movement initiation. The structure of the manifold varies between regions: In PMd, it is curved; in M1, it is nearly perfectly flat; and in dlPFC, it is almost entirely confined to the sensory evidence dimension. In contrast, pallidal activity during deliberation is primarily defined by urgency. We suggest that these findings reveal the distinct functional contributions of different brain regions to an integrated dynamical system governing action selection and execution.

Recent theoretical models suggest that deciding about actions and executing them are not mediated by completely distinct neural mechanisms but are instead two modes of a single system. Analysis of neural activity in several cortical and subcortical regions reveals how they implement an integrated dynamical system that governs deliberation, selection, and execution of actions.
Left parietal tACS at alpha frequency induces a shift of visuospatial attention
Authors: Teresa Schuhmann, Selma K Kemmerer, Felix Duecker, Tom A de Graaf, Sanne ten Oever, Peter De Weerd, Alexander T Sack
Journal: PLOS ONE
Accepted: 2019-11-12T00:00:00Z

Background: Voluntary shifts of visuospatial attention are associated with a lateralization of parieto-occipital alpha power (7-13Hz), i.e. higher power in the hemisphere ipsilateral and lower power contralateral to the locus of attention. Recent noninvasive neuromodulation studies demonstrated that alpha power can be experimentally increased using transcranial alternating current stimulation (tACS). Objective/Hypothesis: We hypothesized that tACS at alpha frequency over the left parietal cortex induces shifts of attention to the left hemifield. However, spatial attention shifts not only occur voluntarily (endogenous/ top-down), but also stimulus-driven (exogenous/ bottom-up). To study the task-specificity of the potential effects of tACS on attentional processes, we administered three conceptually different spatial attention tasks. Methods: 36 healthy volunteers were recruited from an academic environment. In two separate sessions, we applied either high-density tACS at 10Hz, or sham tACS, for 35–40 minutes to their left parietal cortex. We systematically compared performance on endogenous attention, exogenous attention, and stimulus detection tasks. Results: In the endogenous attention task, a greater leftward bias in reaction times was induced during left parietal 10Hz tACS as compared to sham. There were no stimulation effects in either the exogenous attention or the stimulus detection task. Conclusion: The study demonstrates that high-density tACS at 10Hz can be used to modulate visuospatial attention performance. The tACS effect is task-specific, indicating that not all forms of attention are equally susceptible to the stimulation.
Analogous computations in working memory input, output and motor gating: Electrophysiological and computational modeling evidence
Authors: Rachel Rac-Lubashevsky, Michael J Frank
Journal: PLOS Computational Biology
Accepted: 2021-04-17T00:00:00Z


Adaptive cognitive-control involves a hierarchical cortico-striatal gating system that supports selective updating, maintenance, and retrieval of useful cognitive and motor information. Here, we developed a task that independently manipulates selective gating operations into working-memory (input gating), from working-memory (output gating), and of responses (motor gating) and tested the neural dynamics and computational principles that support them. Increases in gating demands, captured by gate switches, were expressed by distinct EEG correlates at each gating level that evolved dynamically in partially overlapping time windows. Further, categorical representations of specific maintained items and of motor responses could be decoded from EEG when the corresponding gate was switching, thereby linking gating operations to prioritization. Finally, gate switching at all levels was related to increases in the motor decision threshold as quantified by the drift diffusion model. Together these results support the notion that cognitive gating operations scaffold on top of mechanisms involved in motor gating.
Author summary: How do humans decide which information is relevant to attend to in memory, which cognitive operation to take, and when? Flexibly updating, maintenance and retrieval of relevant information from working memory (WM) are thought to be managed by gating computations in the frontostriatal network, supporting higher order learning and cognitive flexibility. Using the reference-back-2 task, we tested the key properties of gating. Namely that they are selective (“content-addressable”) and that principles of cognitive “actions” (including input gating of WM, output gating from WM) are scaffold on top of the motor gating operations. Using trial-by-trial EEG indexing and quantitative computational modeling (the hierarchical drift-diffusion model) we showed that action selection at all three levels of gating have separable neural signatures but they operate partly in parallel, such that decisions about a response are processed to some degree even while the identity of the cognitive rule were uncertain. Furthermore, we showed analogous computations across levels of gating as selection of WM representation and of motor action lead to increase in the estimated decision threshold and to enhanced neural coding of the selected information thereby providing a novel link between WM gating and WM prioritization.
Correcting the hebbian mistake: Toward a fully error-driven hippocampus
Authors: Yicong Zheng, Xiaonan L Liu, Satoru Nishiyama, Charan Ranganath, Randall C O’Reilly
Journal: PLOS Computational Biology
Accepted: 2022-09-19T00:00:00Z


The hippocampus plays a critical role in the rapid learning of new episodic memories. Many computational models propose that the hippocampus is an autoassociator that relies on Hebbian learning (i.e., “cells that fire together, wire together”). However, Hebbian learning is computationally suboptimal as it does not learn in a way that is driven toward, and limited by, the objective of achieving effective retrieval. Thus, Hebbian learning results in more interference and a lower overall capacity. Our previous computational models have utilized a powerful, biologically plausible form of error-driven learning in hippocampal CA1 and entorhinal cortex (EC) (functioning as a sparse autoencoder) by contrasting local activity states at different phases in the theta cycle. Based on specific neural data and a recent abstract computational model, we propose a new model called Theremin (Total Hippocampal ERror MINimization) that extends error-driven learning to area CA3—the mnemonic heart of the hippocampal system. In the model, CA3 responds to the EC monosynaptic input prior to the EC disynaptic input through dentate gyrus (DG), giving rise to a temporal difference between these two activation states, which drives error-driven learning in the EC→CA3 and CA3↔CA3 projections. In effect, DG serves as a teacher to CA3, correcting its patterns into more pattern-separated ones, thereby reducing interference. Results showed that Theremin, compared with our original Hebbian-based model, has significantly increased capacity and learning speed. The model makes several novel predictions that can be tested in future studies.
Author summary: Exemplified by the famous case of patient H.M. (Henry Molaison) whose hippocampus was surgically removed, the hippocampus is critical for learning and remembering everyday events—what is typically called “episodic memory.” The dominant theory for how it learns is based on the intuitive principle stated by Donald Hebb in 1949, that neurons that “fire together, wire together”—when two neurons are active at the same time, the strength of their connection increases. We show in this paper that using a different form of learning based on correcting errors (error-driven learning) results in significantly improved episodic memory function in a biologically-based computational model of the hippocampus. This model also provides a significantly better account of behavioral data on the testing effect, where learning by testing with partial cues is better than learning with the complete set of information.
The contribution of the basal ganglia and cerebellum to motor learning: A neuro-computational approach
Authors: Javier Baladron, Julien Vitay, Torsten Fietzek, Fred H Hamker
Journal: PLOS Computational Biology
Accepted: 2023-03-13T00:00:00Z


Motor learning involves a widespread brain network including the basal ganglia, cerebellum, motor cortex, and brainstem. Despite its importance, little is known about how this network learns motor tasks and which role different parts of this network take. We designed a systems-level computational model of motor learning, including a cortex-basal ganglia motor loop and the cerebellum that both determine the response of central pattern generators in the brainstem. First, we demonstrate its ability to learn arm movements toward different motor goals. Second, we test the model in a motor adaptation task with cognitive control, where the model replicates human data. We conclude that the cortex-basal ganglia loop learns via a novelty-based motor prediction error to determine concrete actions given a desired outcome, and that the cerebellum minimizes the remaining aiming error.
Author summary: In this study, we aimed to better understand motor learning using a neuro-computational approach. While previous work emphasized different learning regimes of the various components, the main novelty of our study is the interplay of its components. Notably, we show that the model accounts well for motor adaptation data of experiments that involve a cognitive strategy.