Striatal Prediction Error Modulates Cortical Coupling

Both perceptual inference and motor responses are shaped by learned probabilities. For example, stimulus-induced responses in sensory cortices and preparatory activity in premotor cortex reflect how (un)expected a stimulus is. This is in accordance with predictive coding accounts of brain function, which posit a fundamental role of prediction errors for learning and adaptive behavior. We used functional magnetic resonance imaging and recent advances in computational modeling to investigate how (failures of) learned predictions about visual stimuli influence subsequent motor responses. Healthy volunteers discriminated visual stimuli that were differentially predicted by auditory cues. Critically, the predictive strengths of cues varied over time, requiring subjects to continuously update estimates of stimulus probabilities. This online inference, modeled using a hierarchical Bayesian learner, was reflected behaviorally: speed and accuracy of motor responses increased significantly with predictability of the stimuli. We used nonlinear dynamic causal modeling to demonstrate that striatal prediction errors are used to tune functional coupling in cortical networks during learning. Specifically, the degree of striatal trial-by-trial prediction error activity controls the efficacy of visuomotor connections and thus the influence of surprising stimuli on premotor activity. This finding substantially advances our understanding of striatal function and provides direct empirical evidence for formal learning theories that posit a central role for prediction error-dependent plasticity.

Related papers:

title: Learning the value of information in an uncertain world
abstract: Our decisions are guided by outcomes that are associated with decisions made in the past. However, the amount of influence each past outcome has on our next decision remains unclear. To ensure optimal decision-making, the weight given to decision outcomes should reflect their salience in predicting future outcomes, and this salience should be modulated by the volatility of the reward environment. We show that human subjects assess volatility in an optimal manner and adjust decision-making accordingly. This optimal estimate of volatility is reflected in the fMRI signal in the anterior cingulate cortex (ACC) when each trial outcome is observed. When a new piece of information is witnessed, activity levels reflect its salience for predicting future outcomes. Furthermore, variations in this ACC signal across the population predict variations in subject learning rates. Our results provide a formal account of how we weigh our different experiences in guiding our future actions.

title: Integrated Bayesian models of learning and decision making for saccadic eye movements☆
abstract: The neurophysiology of eye movements has been studied extensively, and several computational models have been proposed for decision-making processes that underlie the generation of eye movements towards a visual stimulus in a situation of uncertainty. One class of models, known as linear rise-to-threshold models, provides an economical, yet broadly applicable, explanation for the observed variability in the latency between the onset of a peripheral visual target and the saccade towards it. So far, however, these models do not account for the dynamics of learning across a sequence of stimuli, and they do not apply to situations in which subjects are exposed to events with conditional probabilities. In this methodological paper, we extend the class of linear rise-to-threshold models to address these limitations. Specifically, we reformulate previous models in terms of a generative, hierarchical model, by combining two separate sub-models that account for the interplay between learning of target locations across trials and the decision-making process within trials. We derive a maximum-likelihood scheme for parameter estimation as well as model comparison on the basis of log likelihood ratios. The utility of the integrated model is demonstrated by applying it to empirical saccade data acquired from three healthy subjects. Model comparison is used (i) to show that eye movements do not only reflect marginal but also conditional probabilities of target locations, and (ii) to reveal subject-specific learning profiles over trials. These individual learning profiles are sufficiently distinct that test samples can be successfully mapped onto the correct subject by a naïve Bayes classifier. Altogether, our approach extends the class of linear rise-to-threshold models of saccadic decision making, overcomes some of their previous limitations, and enables statistical inference both about learning of target locations across trials and the decision-making process within trials.

title: A Dual Role for Prediction Error in Associative Learning
abstract: Confronted with a rich sensory environment, the brain must learn statistical regularities across sensory domains to construct causal models of the world. Here, we used functional magnetic resonance imaging and dynamic causal modeling (DCM) to furnish neurophysiological evidence that statistical associations are learnt, even when task-irrelevant. Subjects performed an audio-visual target-detection task while being exposed to distractor stimuli. Unknown to them, auditory distractors predicted the presence or absence of subsequent visual distractors. We modeled incidental learning of these associations using a Rescorla–Wagner (RW) model. Activity in primary visual cortex and putamen reflected learning-dependent surprise: these areas responded progressively more to unpredicted, and progressively less to predicted visual stimuli. Critically, this prediction-error response was observed even when the absence of a visual stimulus was surprising. We investigated the underlying mechanism by embedding the RW model into a DCM to show that auditory to visual connectivity changed significantly over time as a function of prediction error. Thus, consistent with predictive coding models of perception, associative learning is mediated by prediction-error dependent changes in connectivity. These results posit a dual role for prediction-error in encoding surprise and driving associative plasticity.

title: A Neural Representation of Prior Information during Perceptual Inference
abstract: Perceptual inference is biased by foreknowledge about what is probable or possible. How prior expectations are neurally represented during visual perception, however, remains unknown. We used functional magnetic resonance imaging to measure brain activity in humans judging simple visual stimuli. Perceptual decisions were either biased in favor of a single alternative (A/ approximately A decisions) or taken without bias toward either choice (A/B decisions). Extrastriate and anterior temporal lobe regions were more active during A/ approximately A than A/B decisions, suggesting multiple representations of prior expectations within the visual hierarchy. Forward connectivity was increased when expected and observed perception diverged ("prediction error" signals), whereas prior expectations fed backward from higher to lower regions. Finally, the coincidence between expected and observed perception activated orbital prefrontal regions, perhaps reflecting the reinforcement of prior expectations. These data support computational and quantitative models proposing that a visual percept emerges from converging bottom-up and top-down signals.

title: Effect of deviant probability and interstimulus/interdeviant interval on the auditory N1 and mismatch negativity in the cat auditory cortex.
abstract: In passive oddball paradigm the effects of changes in interstimulus/interdeviant interval (ISI; IDI) and deviant probability were investigated on mismatch negativity (MMN), auditory N1 wave and the exogenous P1 component of the auditory event-related potential in the cat. An epidural electrode matrix was chronically implanted over the auditory fields of the neocortex, and the amplitudes of the aforementioned components were measured in the location of their amplitude maxima. Dependence of the MMN both on the ISI and IDI as well as deviant probability was revealed, while the amplitude of the P1 and N1 showed dependence merely on the ISI. This method can be used for separation of the two negative, often overlapping components in the cat.

title: The effect of stimulus probability on the somatosensory mismatch field
abstract: We investigated the effect of deviant stimulus probability on the somatosensory magnetic mismatch negativity (MMNm) using an electrical two-point stimulation. First, we determined the discrimination threshold (DT) of the two-point distance. We applied standard stimuli at a distance that subjects felt as one point and deviant stimuli at a distance that subjects definitely felt as two points. We used three deviant stimulus probabilities, 10, 30, and 50%. The components peaking around 30–70 ms (first component) and 150–250 ms (fourth component) following deviant stimuli were significantly larger than those following standard stimuli in 10% condition, but not in 30 or 50% condition. The equivalent current dipole (ECD) was located in the contralateral primary somatosensory cortex (cSI) for the first component, and in the cSI and in the contralateral secondary somatosensory cortex (cSII) for the fourth component. The peak amplitude of the MMNm decreased as the probability of the deviant stimulus increased. The Somatosensory MMNm was affected by deviant stimulus probability similar to an auditory mismatch negativity (MMN).

title: Determinants of laser-evoked EEG responses: pain perception or stimulus saliency?
abstract: Although laser-evoked electroencephalographic (EEG) responses are increasingly used to investigate nociceptive pathways, their functional significance remains unclear. The reproducible observation of a robust correlation between the intensity of pain perception and the magnitude of the laser-evoked N1, N2, and P2 responses has led some investigators to consider these responses a direct correlate of the neural activity responsible for pain intensity coding in the human cortex. Here, we provide compelling evidence to the contrary. By delivering trains of three identical laser pulses at four different energies, we explored the modulation exerted by the temporal expectancy of the stimulus on the relationship between intensity of pain perception and magnitude of the following laser-evoked brain responses: the phase-locked N1, N2, and P2 waves, and the non-phase-locked laser-induced synchronization (ERS) and desynchronization (ERD). We showed that increasing the temporal expectancy of the stimulus through stimulus repetition at a constant interstimulus interval 1) significantly reduces the magnitudes of the laser-evoked N1, N2, P2, and ERS; and 2) disrupts the relationship between the intensity of pain perception and the magnitude of these responses. Taken together, our results indicate that laser-evoked EEG responses are not determined by the perception of pain per se, but are mainly determined by the saliency of the eliciting nociceptive stimulus (i.e., its ability to capture attention). Therefore laser-evoked EEG responses represent an indirect readout of the function of the nociceptive system.

title: Seeing smells: imaging olfactory learning in bees
abstract: Using calcium imaging as bees learn to associate odors with a reward, Faber and colleagues show that learning changes activity patterns at the earliest olfactory site in the brain.

title: A theory of cortical responses
abstract: This article concerns the nature of evoked brain responses and the principles underlying their generation. We start with the premise that the sensory brain has evolved to represent or infer the causes of changes in its sensory inputs. The problem of inference is well formulated in statistical terms. The statistical fundaments of inference may therefore afford important constraints on neuronal implementation. By formulating the original ideas of Helmholtz on perception, in terms of modern-day statistical theories, one arrives at a model of perceptual inference and learning that can explain a remarkable range of neurobiological facts. It turns out that the problems of inferring the causes of sensory input (perceptual inference) and learning the relationship between input and cause (perceptual learning) can be resolved using exactly the same principle. Specifically, both inference and learning rest on minimizing the brain's free energy, as defined in statistical physics. Furthermore, inference and learning can proceed in a biologically plausible fashion. Cortical responses can be seen as the brain’s attempt to minimize the free energy induced by a stimulus and thereby encode the most likely cause of that stimulus. Similarly, learning emerges from changes in synaptic efficacy that minimize the free energy, averaged over all stimuli encountered. The underlying scheme rests on empirical Bayes and hierarchical models of how sensory input is caused. The use of hierarchical models enables the brain to construct prior expectations in a dynamic and context-sensitive fashion. This scheme provides a principled way to understand many aspects of cortical organization and responses. The aim of this article is to encompass many apparently unrelated anatomical, physiological and psychophysical attributes of the brain within a single theoretical perspective. In terms of cortical architectures, the theoretical treatment predicts that sensory cortex should be arranged hierarchically, that connections should be reciprocal and that forward and backward connections should show a functional asymmetry (forward connections are driving, whereas backward connections are both driving and modulatory). In terms of synaptic physiology, it predicts associative plasticity and, for dynamic models, spike-timing-dependent plasticity. In terms of electrophysiology, it accounts for classical and extra classical receptive field effects and long-latency or endogenous components of evoked cortical responses. It predicts the attenuation of responses encoding prediction error with perceptual learning and explains many phenomena such as repetition suppression, mismatch negativity (MMN) and the P300 in electroencephalography. In psychophysical terms, it accounts for the behavioural correlates of these physiological phenomena, for example, priming and global precedence. The final focus of this article is on perceptual learning as measured with the MMN and the implications for empirical studies of coupling among cortical areas using evoked sensory responses.

title: Free-energy and the brain
abstract: If one formulates Helmholtz’s ideas about perception in terms of modern-day theories one arrives at a model of perceptual inference and learning that can explain a remarkable range of neurobiological facts. Using constructs from statistical physics it can be shown that the problems of inferring what cause our sensory inputs and learning causal regularities in the sensorium can be resolved using exactly the same principles. Furthermore, inference and learning can proceed in a biologically plausible fashion. The ensuing scheme rests on Empirical Bayes and hierarchical models of how sensory information is generated. The use of hierarchical models enables the brain to construct prior expectations in a dynamic and context-sensitive fashion. This scheme provides a principled way to understand many aspects of the brain’s organisation and responses. In this paper, we suggest that these perceptual processes are just one emergent property of systems that conform to a free-energy principle. The free-energy considered here represents a bound on the surprise inherent in any exchange with the environment, under expectations encoded by its state or configuration. A system can minimise free-energy by changing its configuration to change the way it samples the environment, or to change its expectations. These changes correspond to action and perception, respectively, and lead to an adaptive exchange with the environment that is characteristic of biological systems. This treatment implies that the system’s state and structure encode an implicit and probabilistic model of the environment. We will look at models entailed by the brain and how minimisation of free-energy can explain its dynamics and structure.

title: Predictive coding under the free-energy principle
abstract: This paper considers prediction and perceptual categorization as an inference problem that is solved by the brain. We assume that the brain models the world as a hierarchy or cascade of dynamical systems that encode causal structure in the sensorium. Perception is equated with the optimization or inversion of these internal models, to explain sensory data. Given a model of how sensory data are generated, we can invoke a generic approach to model inversion, based on a free energy bound on the model's evidence. The ensuing free-energy formulation furnishes equations that prescribe the process of recognition, i.e. the dynamics of neuronal activity that represent the causes of sensory input. Here, we focus on a very general model, whose hierarchical and dynamical structure enables simulated brains to recognize and predict trajectories or sequences of sensory states. We first review hierarchical dynamical models and their inversion. We then show that the brain has the necessary infrastructure to implement this inversion and illustrate this point using synthetic birds that can recognize and categorize birdsongs.

title: Anticipatory activity of motor cortex neurons in relation to direction of an intended movement.
abstract: 1. Monkeys were trained to 1) hold a handle in a central zone midway between "push" and "pull" while awaiting 2) an instruction telling them how to respond to a subsequent 3) perturbation, which triggered the instructed movement and was followed by 4) a reward if the movement was correct. 2. There were two sorts of instructions: push and pull. When the pull instruction had preceded the perturbation, the monkey responded to the perturbation by pulling, whereas after a push instruction, the monkey responded to the perturbation by pushing. 3. Recordings in pre- and postcentral sensorimotor cortex revealed instruction-induced changes of neuronal activity during the period intervening between the instruction and the perturbation-triggered movement. Effects of the instruction were differential depending on which of the two instructions was given, such differential responses to the instruction being detected in 61% of precentral pyramidal tract neurons (PTNs), 44% of precentral non-PTNs, and 11% of postcentral neurons. 4. Since motor cortex PTN axons end on alpha and gamma motoneurons and on interneurons of the spinal cord, changes of PTN activity with "intention" or "motor set" provide a mechanism for suprasegmental control and presetting of spinal cord reflex excitability specific to the nature of an impending movement.

title: Set-related neuronal activity in the premotor cortex of rhesus monkeys: effects of changes in motor set
abstract: Previous studies have suggested that the premotor cortex plays a role in motor preparation. We have tested this hypothesis in macaque monkeys by examining neuronal activity during an enforced, 1.5—3.0 s delay period between the presentation of an instruction for movement and the onset of that movement. Two targets for movement were available to the monkey, one on the left and one on the right. Illumination of one of the targets served as the instruction for a forelimb movement. It is known that there are cells in the premotor cortex that have directionally specific, sustained activity increases or decreases following such instructions. If the premotor cortex is involved in the preparation for movement in a particular direction, then changing the target from one to the opposite side during the delay period should lead to a pronounced change in sustained neuronal activity. Further, removing the instruction, while still requiring movement to the target, should have little or no sustained effect. Seventy cells showed the predicted activity patterns, thus supporting the view that the premotor cortex plays a role in motor preparation.

title: Prior information in motor and premotor cortex: activity during the delay period and effect on pre-movement activity.
abstract: In instructed-delay (ID) tasks, instructional cues provide prior information about the nature of a movement to execute after a delay. Neuronal responses in dorsal premotor cortex (PMd) during the instructed-delay period (IDP) between the CUE and subsequent GO signals are presumed to reflect early planning stages initiated by the prior information. In contrast, in multiple-choice reaction-time (RT) tasks, all motor planning and execution processes must occur after the GO signal. These assumptions predict that neuronal planning correlates recorded during the IDP of ID trials should share common features with early post-GO activity in RT trials, and that those response components need not be recapitulated after the GO signal of ID trials. These two predictions were tested by comparing activity recorded in RT and ID tasks from 503 neurons in PMd and caudal (MIc) and rostral (MIr) primary motor cortex. The incidence and strength of directionally tuned IDP activity declined progressively from PMd to MIc. The directional tuning of activity during the IDP of ID trials was more similar to that in the reaction-time epoch (RTE) of RT trials than after movement onset, especially in PMd. A modulation of post-GO activity was often observed between RT and ID trials and was confined mainly to the RTE. This effect was also most prominent in PMd. The most common change was a reduction in intensity of short-latency phasic responses to the GO signal between RT and ID trials, especially in PMd cells with a short-latency phasic response to CUE signals. However, the largest group of cells in each area showed no large change in peak RTE activity between RT and ID trials, whether they were active in the IDP or not. Since early phasic CUE-related responses are least likely to be recapitulated after the GO signal in ID trials, they may be a neuronal correlate of an early planning stage such as response selection. Tonic IDP responses, which are not as strongly associated with a post-GO reduction in activity, may be related to other aspects of motor planning and preparation. Finally, a major component of the movement-related activity in both MI and PMd is not susceptible to modification by prior information and is indivisibly coupled temporally to movement execution.

title: The effect of conditional probability of the response signal on the simple reaction time.
abstract: With simple reaction time schedules using irregular series of interstimulus intervals (or preparatory periods), the relationship between the response latency and the duration of interstimulus intervals depends on two factors: (1) Time uncertainty increasing when time estimation is made more difficult by lengthening the interval duration. (2) Probability of signal occurrence, increasing with time (conditional probability). In order to investigate the role of the latter factor time uncertainty has been made equal for all time intervals by introducing a second signal with a periodicity equal to a sub-multiple of the duration of the different intervals. It is shown that the responses latency is inversely related to the duration of the interstimulus intervals; this relationship is absent in a control experiment where the second signal is not given. The worsening of the average performance is explained in terms of probability, the role of a differential learning in the two situations is discussed and the exact shape of the curves is analyzed.

title: Information theory, novelty and hippocampal responses: unpredicted or unpredictable?
abstract: Shannon's information theory provides a principled framework for the quantitative analysis of brain responses during the encoding and representation of event streams. In particular, entropy measures the expected uncertainty of events in a given context. This contextual uncertainty or unpredictability may, itself, be important for balancing [bottom-up] sensory information and [top-down] prior expectations during perceptual synthesis. Using event-related functional magnetic resonance imaging (fMRI), we found that the anterior hippocampus is sensitive to the entropy of a visual stimulus stream. In contrast, activity in an extensive bilateral cortico-thalamic network was dictated by the surprise or information associated with each particular stimulus. In short, we show that the probabilistic structure or context in which events occur is an important predictor of hippocampal activity.

title: Influence of Uncertainty and Surprise on Human Corticospinal Excitability during Preparation for Action
abstract: Actions are guided by prior sensory information [1–10], which is inherently uncertain. However, how the motor system is sculpted by trial-by-trial content of current sensory information remains largely unexplored. Previous work suggests that conditional probabilities, learned under a particular context, can be used preemptively to influence the output of the motor system [11–14]. To test this we used transcranial magnetic stimulation (TMS) to read out corticospinal excitability (CSE) during preparation for action in an instructed delay task [15, 16]. We systematically varied the uncertainty about an impending action by changing the validity of the instructive visual cue. We used two information-theoretic quantities to predict changes in CSE, prior to action, on a trial-by-trial basis: entropy (average uncertainty) and surprise (the stimulus-bound information conveyed by a visual cue) [17–19]. Our data show that during preparation for action, human CSE varies according to the entropy and surprise conveyed by visual events guiding action. CSE increases on trials with low entropy about the impending action and low surprise conveyed by an event. Commensurate effects were observed in reaction times. We suggest that motor output is biased according to contextual probabilities that are represented dynamically in the brain.

title: Temporal Prediction Errors in a Passive Learning Task Activate Human Striatum
abstract: Functional MRI experiments in human subjects strongly suggest that the striatum participates in processing information about the predictability of rewarding stimuli. However, stimuli can be unpredictable in character (what stimulus arrives next), unpredictable in time (when the stimulus arrives), and unpredictable in amount (how much arrives). These variables have not been dissociated in previous imaging work in humans, thus conflating possible interpretations of the kinds of expectation errors driving the measured brain responses. Using a passive conditioning task and fMRI in human subjects, we show that positive and negative prediction errors in reward delivery time correlate with BOLD changes in human striatum, with the strongest activation lateralized to the left putamen. For the negative prediction error, the brain response was elicited by expectations only and not by stimuli presented directly; that is, we measured the brain response to nothing delivered (juice expected but not delivered) contrasted with nothing delivered (nothing expected).

title: Temporal Difference Models and Reward-Related Learning in the Human Brain
abstract: Temporal difference learning has been proposed as a model for Pavlovian conditioning, in which an animal learns to predict delivery of reward following presentation of a conditioned stimulus (CS). A key component of this model is a prediction error signal, which, before learning, responds at the time of presentation of reward but, after learning, shifts its response to the time of onset of the CS. In order to test for regions manifesting this signal profile, subjects were scanned using event-related fMRI while undergoing appetitive conditioning with a pleasant taste reward. Regression analyses revealed that responses in ventral striatum and orbitofrontal cortex were significantly correlated with this error signal, suggesting that, during appetitive conditioning, computations described by temporal difference learning are expressed in the human brain.

title: Prediction Error during Retrospective Revaluation of Causal Associations in Humans fMRI Evidence in Favor of an Associative Model of Learning
abstract: Associative learning theory assumes that prediction error is a driving force in learning. A competing view, probabilistic contrast (PC) theory, is that learning and prediction error are unrelated. We tested a learning phenomenon that has proved troublesome for associative theory--retrospective revaluation--to evaluate these two models. We previously showed that activation in right lateral prefrontal cortex (PFC) provides a reliable signature for the presence of prediction error. Thus, if the associative view is correct, retrospective revaluation should be accompanied by right lateral PFC activation. PC theory would be supported by the absence of this activation. Right PFC and ventral striatal activation occurred during retrospective revaluation, supporting the associative account. Activations appeared to reflect the degree of revaluation, predicting later brain responses to revalued cues. Our results support a modified associative account of retrospective revaluation and demonstrate the potential of functional neuroimaging as a tool for evaluating competing learning models.

title: Temporal difference models describe higher-order learning in humans
abstract: The ability to use environmental stimuli to predict impending harm is critical for survival. Such predictions should be available as early as they are reliable. In pavlovian conditioning, chains of successively earlier predictors are studied in terms of higher-order relationships, and have inspired computational theories such as temporal difference learning. However, there is at present no adequate neurobiological account of how this learning occurs. Here, in a functional magnetic resonance imaging (fMRI) study of higher-order aversive conditioning, we describe a key computational strategy that humans use to learn predictions about pain. We show that neural activity in the ventral striatum and the anterior insula displays a marked correspondence to the signals for sequential learning predicted by temporal difference models. This result reveals a flexible aversive learning process ideally suited to the changing and uncertain nature of real-world environments. Taken with existing data on reward learning, our results suggest a critical role for the ventral striatum in integrating complex appetitive and aversive predictions to coordinate behaviour.

title: Temporal Difference Modeling of the Blood-Oxygen Level Dependent Response During Aversive Conditioning in Humans: Effects of Dopaminergic Modulation
abstract: The prediction error (PE) hypothesized by the temporal difference model has been shown to correlate with the phasic activity of dopamine neurons during reward learning and the blood-oxygen level dependent (BOLD) response during reward and aversive conditioning tasks. We hypothesized that dopamine would modulate the PE related signal in aversive conditioning and that haloperidol would reduce PE related activity, while an acute dose of amphetamine would increase PE related activity in the ventral striatum.



Healthy participants took an acute dose of amphetamine, haloperidol, or placebo. We used functional magnetic resonance imaging (fMRI) to measure the BOLD signal while they carried out an aversive conditioning task, using cutaneous electrical stimulation as the unconditioned stimulus (US) and yellow and blue circles as conditioned stimulus (CS+ and CS-, respectively).



Prediction error related BOLD activity was seen only in the ventral striatum in the placebo subjects. The subjects given amphetamine showed a wider network of PE related BOLD activity, including the ventral striatum, globus pallidus, putamen, insula, anterior cingulate, and substantia nigra/ventral tegmental area. Haloperidol subjects did not show PE related activity in any of these regions.



Our results provide the first demonstration that the modulation of dopamine transmission affects both the physiological correlates and PE related BOLD activity during aversive learning.

title: Decision theory, reinforcement learning, and the brain
abstract: Decision making is a core competence for animals and humans acting and surviving in environments they only partially comprehend, gaining rewards and punishments for their troubles. Decision-theoretic concepts permeate experiments and computational models in ethology, psychology, and neuroscience. Here, we review a well-known, coherent Bayesian approach to decision making, showing how it unifies issues in Markovian decision problems, signal detection psychophysics, sequential sampling, and optimal exploration and discuss paradigmatic psychological and neural examples of each problem. We discuss computational issues concerning what subjects know about their task and how ambitious they are in seeking optimal solutions; we address algorithmic topics concerning model-based and model-free methods for making choices; and we highlight key aspects of the neural implementation of decision making.

title: Reinforcement learning: The Good, The Bad and The Ugly
abstract: Reinforcement learning provides both qualitative and quantitative frameworks for understanding and modeling adaptive decision-making in the face of rewards and punishments. Here we review the latest dispatches from the forefront of this field, and map out some of the territories where lie monsters.

title: Object perception as Bayesian inference.
abstract: We perceive the shapes and material properties of objects quickly and reliably despite the complexity and objective ambiguities of natural images. Typical images are highly complex because they consist of many objects embedded in background clutter. Moreover, the image features of an object are extremely variable and ambiguous owing to the effects of projection, occlusion, background clutter, and illumination. The very success of everyday vision implies neural mechanisms, yet to be understood, that discount irrelevant information and organize ambiguous or noisy local image features into objects and surfaces. Recent work in Bayesian theories of visual perception has shown how complexity may be managed and ambiguity resolved through the task-dependent, probabilistic integration of prior object knowledge with image features.

title: When a good fit can be bad
abstract: How should we select among computational models of cognition? Although it is commonplace to measure how well each model fits the data, this is insufficient. Good fits can be misleading because they can result from properties of the model that have nothing to do with it being a close approximation to the cognitive process of interest (e.g. overfitting). Selection methods are introduced that factor in these properties when measuring fit. Their success in outperforming standard goodness-of-fit measures stems from a focus on measuring the generalizability of a model's data-fitting abilities, which should be the goal of model selection.

title: Comparing dynamic causal models
abstract: This article describes the use of Bayes factors for comparing dynamic causal models (DCMs). DCMs are used to make inferences about effective connectivity from functional magnetic resonance imaging (fMRI) data. These inferences, however, are contingent upon assumptions about model structure, that is, the connectivity pattern between the regions included in the model. Given the current lack of detailed knowledge on anatomical connectivity in the human brain, there are often considerable degrees of freedom when defining the connectional structure of DCMs. In addition, many plausible scientific hypotheses may exist about which connections are changed by experimental manipulation, and a formal procedure for directly comparing these competing hypotheses is highly desirable. In this article, we show how Bayes factors can be used to guide choices about model structure, both concerning the intrinsic connectivity pattern and the contextual modulation of individual connections. The combined use of Bayes factors and DCM thus allows one to evaluate competing scientific theories about the architecture of large-scale neural networks and the neuronal interactions that mediate perception and cognition.

title: Comparing hemodynamic models with DCM
abstract: The classical model of blood oxygen level-dependent (BOLD) responses by Buxton et al. [Buxton, R.B., Wong, E.C., Frank, L.R., 1998. Dynamics of blood flow and oxygenation changes during brain activation: the Balloon model. Magn. ] has been very important in providing a biophysically plausible framework for explaining different aspects of hemodynamic responses. It also plays an important role in the hemodynamic forward model for dynamic causal modeling (DCM) of fMRI data. A recent study by Obata et al. [Obata, T., Liu, T.T., Miller, K.L., Luh, W.M., Wong, E.C., Frank, L.R., Buxton, R.B., 2004. Discrepancies between BOLD and flow dynamics in primary and supplementary motor areas: application of the Balloon model to the interpretation of BOLD transients. NeuroImage 21, 144–153] linearized the BOLD signal equation and suggested a revised form for the model coefficients. In this paper, we show that the classical and revised models are special cases of a generalized model. The BOLD signal equation of this generalized model can be reduced to that of the classical Buxton model by simplifying the coefficients or can be linearized to give the Obata model. Given the importance of hemodynamic models for investigating BOLD responses and analyses of effective connectivity with DCM, the question arises which formulation is the best model for empirically measured BOLD responses. In this article, we address this question by embedding different variants of the BOLD signal equation in a well-established DCM of functional interactions among visual areas. This allows us to compare the ensuing models using Bayesian model selection. Our model comparison approach had a factorial structure, comparing eight different hemodynamic models based on (i) classical vs. revised forms for the coefficients, (ii) linear vs. non-linear output equations, and (iii) fixed vs. free parameters, ε, for region-specific ratios of intra- and extravascular signals. Using fMRI data from a group of twelve subjects, we demonstrate that the best model is a non-linear model with a revised form for the coefficients, in which ε is treated as a free parameter.

title: Modeling Geometric Deformations in EPI Time Series
abstract: Even after realignment there is residual movement-related variance present in fMRI time-series, causing loss of sensitivity and, potentially, also specificity. One cause is the differential deformation of the sampling matrix, by field inhomogeneities, at different object positions, i.e., a movement-by-inhomogeneity interaction. This has been addressed previously by using empirical field measurements. In the present paper we suggest a forward model of how data is affected by an inhomogeneous field at different object positions. From this model we derive a method to solve the inverse problem of estimating the field inhomogeneities and their derivatives with respect to object position, directly from the EPI data and estimated realignment parameters. The field is modeled as a linear combination of cosine basis fields, which facilitates a fast way of implementing the necessary matrix operations. Simulations suggest that the solution is tractable and that the fields are estimable given the deformed images and knowledge of the relative positions at which they have been acquired. An experiment on a subject performing voluntary movements in the scanner yielded plausible estimates of the deformation fields and their application to "unwarp" the time series significantly reduced movement-related variance.

title: Image Distortion Correction in fMRI: A Quantitative Evaluation
abstract: A well-recognized problem with the echo-planar imaging (EPI) technique most commonly used for functional magnetic resonance imaging (fMRI) studies is geometric distortion caused by magnetic field inhomogeneity. This makes it difficult to achieve an accurate registration between a functional activation map calculated from an EPI time series and an undistorted, high resolution anatomical image. A correction method based on mapping the spatial distribution of field inhomogeneities can be used to reduce these distortions. This approach is attractive in its simplicity but requires postprocessing to improve the robustness of the acquired field map and reduce any secondary artifacts. Furthermore, the distribution of the internal magnetic field throughout the head is position dependent resulting in an interaction between distortion and head motion. Therefore, a single field map may not be sufficient to correct for the distortions throughout a whole fMRI time series. In this paper we present a quantitative evaluation of image distortion correction for fMRI at 2T. We assess (i) methods for the acquisition and calculation of field maps, (ii) the effect of image distortion correction on the coregistration between anatomical and functional images, and (iii) the interaction between distortion and head motion, assessing the feasibility of using field maps to reduce this effect. We propose that field maps with acceptable noise levels can be generated easily using a dual echo-time EPI sequence and demonstrate the importance of distortion correction for anatomical coregistration, even for small distortions. Using a dual echo-time series to generate a unique field map at each time point, we characterize the interaction between head motion and geometric distortion. However, we suggest that the variance between successively measured field maps introduces additional unwanted variance in the voxel time-series and is therefore not adequate to correct for time-varying distortions.

title: An automated method for neuroanatomic and cytoarchitectonic atlas-based interrogation of fMRI data sets
abstract: Analysis and interpretation of functional MRI (fMRI) data have traditionally been based on identifying areas of significance on a thresholded statistical map of the entire imaged brain volume. This form of analysis can be likened to a "fishing expedition." As we become more knowledgeable about the structure-function relationships of different brain regions, tools for a priori hypothesis testing are needed. These tools must be able to generate region of interest masks for a priori hypothesis testing consistently and with minimal effort. Current tools that generate region of interest masks required for a priori hypothesis testing can be time-consuming and are often laboratory specific. In this paper we demonstrate a method of hypothesis-driven data analysis using an automated atlas-based masking technique. We provide a powerful method of probing fMRI data using automatically generated masks based on lobar anatomy, cortical and subcortical anatomy, and Brodmann areas. Hemisphere, lobar, anatomic label, tissue type, and Brodmann area atlases were generated in MNI space based on the Talairach Daemon. Additionally, we interfaced these multivolume atlases to a widely used fMRI software package, SPM99, and demonstrate the use of the atlas tool with representative fMRI data. This tool represents a necessary evolution in fMRI data analysis for testing of more spatially complex hypotheses.

title: Circular analysis in systems neuroscience: the dangers of double dipping
abstract: A neuroscientific experiment typically generates a large amount of data, of which only a small fraction is analyzed in detail and presented in a publication. However, selection among noisy measurements can render circular an otherwise appropriate analysis and invalidate results. Here we argue that systems neuroscience needs to adjust some widespread practices to avoid the circularity that can arise from selection. In particular, 'double dipping', the use of the same dataset for selection and selective analysis, will give distorted descriptive statistics and invalid statistical inference whenever the results statistics are not inherently independent of the selection criteria under the null hypothesis. To demonstrate the problem, we apply widely used analyses to noise data known to not contain the experimental effects in question. Spurious effects can appear in the context of both univariate activation analysis and multivariate pattern-information analysis. We suggest a policy for avoiding circularity.

title: Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans
abstract: Theories of instrumental learning are centred on understanding how success and failure are used to improve future decisions. These theories highlight a central role for reward prediction errors in updating the values associated with available actions. In animals, substantial evidence indicates that the neurotransmitter dopamine might have a key function in this type of learning, through its ability to modulate cortico-striatal synaptic efficacy. However, no direct evidence links dopamine, striatal activity and behavioural choice in humans. Here we show that, during instrumental learning, the magnitude of reward prediction error expressed in the striatum is modulated by the administration of drugs enhancing (3,4-dihydroxy-l-phenylalanine; l-DOPA) or reducing (haloperidol) dopaminergic function. Accordingly, subjects treated with l-DOPA have a greater propensity to choose the most rewarding action relative to subjects treated with haloperidol. Furthermore, incorporating the magnitude of the prediction errors into a standard action-value learning algorithm accurately reproduced subjects' behavioural choices under the different drug conditions. We conclude that dopamine-dependent modulation of striatal activity can account for how the human brain uses reward prediction errors to improve future decisions.

title: Neuronal coding of prediction errors.
abstract: Associative learning enables animals to anticipate the occurrence of important outcomes. Learning occurs when the actual outcome differs from the predicted outcome, resulting in a prediction error. Neurons in several brain structures appear to code prediction errors in relation to rewards, punishments, external stimuli, and behavioral reactions. In one form, dopamine neurons, norepinephrine neurons, and nucleus basalis neurons broadcast prediction errors as global reinforcement or teaching signals to large postsynaptic structures. In other cases, error signals are coded by selected neurons in the cerebellum, superior colliculus, frontal eye fields, parietal cortex, striatum, and visual system, where they influence specific subgroups of neurons. Prediction errors can be used in postsynaptic structures for the immediate selection of behavior or for synaptic changes underlying behavioral learning. The coding of prediction errors may represent a basic mode of brain function that may also contribute to the processing of sensory information and the short-term control of behavior.

title: Nonlinear dynamic causal models for fMRI
abstract: Models of effective connectivity characterize the influence that neuronal populations exert over each other. Additionally, some approaches, for example Dynamic Causal Modelling (DCM) and variants of Structural Equation Modelling, describe how effective connectivity is modulated by experimental manipulations. Mathematically, both are based on bilinear equations, where the bilinear term models the effect of experimental manipulations on neuronal interactions. The bilinear framework, however, precludes an important aspect of neuronal interactions that has been established with invasive electrophysiological recording studies; i.e., how the connection between two neuronal units is enabled or gated by activity in other units. These gating processes are critical for controlling the gain of neuronal populations and are mediated through interactions between synaptic inputs (e.g. by means of voltage-sensitive ion channels). They represent a key mechanism for various neurobiological processes, including top-down (e.g. attentional) modulation, learning and neuromodulation. This paper presents a nonlinear extension of DCM that models such processes (to second order) at the neuronal population level. In this way, the modulation of network interactions can be assigned to an explicit neuronal population. We present simulations and empirical results that demonstrate the validity and usefulness of this model. Analyses of synthetic data showed that nonlinear and bilinear mechanisms can be distinguished by our extended DCM. When applying the model to empirical fMRI data from a blocked attention to motion paradigm, we found that attention-induced increases in V5 responses could be best explained as a gating of the V1-->V5 connection by activity in posterior parietal cortex. Furthermore, we analysed fMRI data from an event-related binocular rivalry paradigm and found that interactions amongst percept-selective visual areas were modulated by activity in the middle frontal gyrus. In both practical examples, Bayesian model selection favoured the nonlinear models over corresponding bilinear ones.

title: Visual responses in the dorsal premotor area F2 of the macaque monkey
abstract: This study aimed to determine the presence of neurons responding to visual stimuli in area F2 of the dorsal premotor cortex of the macaque monkey. In order to delimit the sector in which visually responsive neurons are located, the somatotopic organization of area F2 was studied with intracortical microstimulation and single neuron recording. The results showed that: (1) in area F2 there is a significant percentage of visually responsive neurons (15.9% of all recorded neurons); (2) area F2 is excitable with a low-threshold current (average 28.1 µA) and has a somatotopic representation of the whole body, except the face; and (3) most visually driven neurons (n=130 out of 169) are concentrated within the rostrolateral sector of the forelimb representation of area F2, thus providing for the first time functional support for the neuroanatomical evidence that the visual input to area F2 is mostly restricted to this sector.

title: Connections of inferior temporal areas TEO and TE with parietal and frontal cortex in macaque monkeys.
abstract: Inferior temporal cortex is perhaps the highest visual processing area and much anatomical work has focused on its connections with other visual areas in temporal and occipital cortex. Here we report connections of inferior temporal cortex with regions in the frontal and parietal lobes. Inferior temporal areas TEO and TE were injected with WGA-HRP and 3H-AA, respectively, or vice versa, in 1-week-old infant and 3-4-year-old adult monkeys (Macaca mulatta). The results indicated that whereas TEO has more extensive connections with parietal areas, TE has more extensive connections with prefrontal areas. Thus, in the intraparietal sulcus, area TEO is connected with areas LIPd, LIPv, and V3A, and with the as yet undefined region between LIPv and V3A, whereas the connections of TE are predominantly with LIPd, and to a lesser extent with LIPv. In the prefrontal cortex, area TE is connected with areas 8 and 45 in the inferior limb of the anterior bank of the arcuate sulcus, with area 12 on the inferior prefrontal convexity, and with areas 11 and 13 on the orbital surface. By contrast, the connections of area TEO are limited to areas 8, 45, and 12. Furthermore, within prefrontal cortex, the projections from areas TEO and TE terminate in different layers in areas 8 and 45, such that those from TEO terminate in all layers, whereas those from TEO terminate in layers I and V/VI only. In contrast to the connections of areas TEO and TE with various medial temporal-lobe and subcortical structures, which are immature in infant monkeys (Webster et al., 1991, 1993b), the connections with parietal and prefrontal areas appear adult-like as early as 1 week of age.

title: Cortical connections of the macaque caudal ventrolateral prefrontal areas 45A and 45B.
abstract: We have found that the 2 architectonic subdivisions of the prefrontal area 45, 45A and 45B, display connectivity patterns that clearly distinguish them from one another and from their neighboring architectonic areas. Area 45A is primarily connected to the frontal areas 45B, 12l, caudal 12r, 12o, 10, rostrodorsal 46, 9/8B, 44, 8/FEF (frontal eye field), and the SEF (supplementary eye field), temporal area IPa, and unique among all the studied areas, to the superior temporal polysensory (STP) area and auditory parabelt areas. Area 45B displayed much stronger frontal connections with the oculomotor areas 8/FEF, 8r, and the SEF than those of area 45A, primary connections with areas 12l, caudal 12r, 12o, and 8B, and unlike area 45A, with areas ventrorostral 46, rostral 12r, 12m, and 13m. Temporal connections were all virtually confined to areas IPa, intermediate TEa/m, and TE. Additional labeling was found in lateral intraparietal area. Our data suggest that 45A and 45B are 2 distinct areas, possibly playing a differential role in nonspatial information processing: area 45A corresponds to the prefrontal sector for which a role in communication behavior and homology with the human area 45 was proposed, whereas area 45B is a distinct prearcuate area, possibly affiliated to the oculomotor frontal system.

title: Superior area 6 afferents from the superior parietal lobule in the macaque monkey
abstract: Superior area 6 of the macaque monkey frontal cortex is formed by two cytoarchitectonic areas: F2 and F7. In the present experiment, we studied the input from the superior parietal lobule (SPL) to these areas by injecting retrograde neural tracers into restricted parts of F2 and F7. Additional injections of retrograde tracers were made into the spinal cord to define the origin of corticospinal projections from the SPL. The results are as follows: 1) The part of F2 located around the superior precentral dimple (F2 dimple region) receives its main input from areas PEc and PEip (PE intraparietal, the rostral part of area PEa of Pandya and Seltzer, [1982] . Area PEip was defined as that part of area PEa that is the source of corticospinal projections. 2) The ventrorostral part of F2 is the target of strong projections from the medial intraparietal area (area MIP) and from the dorsal part of the anterior wall of the parietooccipital sulcus (area V6A). 3) The ventral and caudal parts of F7 receive their main parietal input from the cytoarchitectonic area PGm of the SPL and from the posterior cingulate cortex. 4) The dorsorostral part of F7, which is also known as the supplementary eye field, is not a target of the SPL, but it receives mostly afferents from the inferior parietal lobule and from the temporal cortex. It is concluded that at least three separate parietofrontal circuits link the superior parietal lobule with the superior area 6. Considering the functional properties of the areas that form these circuits, it is proposed that the PEc/PEip‐F2 dimple region circuit is involved in controlling movements on the basis of somatosensory information, which is the traditional role proposed for the whole dorsal premotor cortex. The two remaining circuits appear to be involved in different aspects of visuomotor transformations.  1998.

title: Projections from the superior temporal sulcus to the agranular frontal cortex in the macaque
abstract: The aim of this study was to investigate the organization of the projections from the superior temporal sulcus (STS) to the various areas forming the agranular frontal cortex. Injections of retrograde neuronal tracers were made in the various agranular areas, in nine macaque monkeys. The results showed that two rostral premotor areas, F6 (pre‐SMA) and F7, and the ventrorostral part of area F2 (F2vr) are targets of projections from the upper bank of the STS (uSTS). F6 and the dorsorostral part of F7 (supplementary eye field, SEF) are targets of projections from the rostral part of the uSTS, corresponding to the so‐called ‘superior temporal polysensory area’ (STP). In contrast, the ventral part of area F7 (not including the SEF) and F2vr are targets of afferents from the caudal part of the uSTS. Ventral F7 is the target of weak afferents from the caudalmost and dorsalmost part of the uSTS (area 7a), whilst F2vr is the target of projections from a relatively more rostral and ventral sector of the uSTS, close to the fundus of the sulcus. This sector should correspond to area MST. In conclusion, F6 and SEF receive high order information from STP, whereas ventral F7 and F2vr receive information from areas of the dorsal visual stream.

title: Parietal inputs to dorsal versus ventral premotor areas in the macaque monkey: evidence for largely segregated visuomotor pathways
abstract: The lateral premotor cortex plays a crucial role in visually guided limb movements. Visual information may reach this cortical region from the parietal cortex, the highest stage in the dorsal visual stream. Anatomical studies indicate that the parietal projections to the dorsal (PMd) and ventral (PMv) premotor areas arise from separate parietal regions, supporting the notion of parallel visuomotor pathways. We tested the degree of segregation of these pathways by injecting retrograde tracers into PMd and PMv in the same monkeys, under physiological control. Eleven injections were made in four animals, and the analysis of retrograde labelling revealed that parietal cells projecting to PMd and those projecting to PMv are largely segregated. The strongest projections to PMd arise from the superior parietal lobule, including the medial intraparietal area (MIP), PEc and PGm, and the parieto-occipital area. These areas were devoid of labelling following injections into PMv, which receives its major projections from the anterior intraparietal area (AIP), area PEip, the anterior portion of the inferior parietal gyrus (area 7b), and the somatosensory areas. In addition to their strong projections to PMv, areas 7b and PEip send minor projections to PMd as well. Additional projections to PMd arise from the ventral intraparietal area and the inferior parietal lobule. The present findings are direct anatomical evidence for largely segregated visuomotor pathways linking parietal cortex with PMd and PMv.

title: Cortical connections of the visuomotor parietooccipital area V6Ad of the macaque monkey
abstract: Area V6A, a functionally defined region in the anterior bank of the parietooccipital sulcus, has been subdivided into dorsal and ventral cytoarchitectonic fields (V6Ad and V6Av). The aim of this study was to define the cortical connections of the cytoarchitectonic field V6Ad. Retrograde and bidirectional neuronal tracers were injected into the dorsal part of the anterior bank of parietooccipital sulcus of seven macaque monkeys (Macaca fascicularis). The limits of injection sites were compared with those of cytoarchitectonic fields. The major connections of V6Ad were with areas of the superior parietal lobule. The densest labeling was observed in the medial intraparietal area (MIP). Areas PEc, PGm, and V6Av were also strongly connected. Labeled cells were found in medial parietal area 31; in cingulate area 23; in the anterior (AIP), ventral (VIP), and lateral (LIP) intraparietal areas; in the inferior parietal lobule (fields Opt and PG); and in the medial superior temporal area (MST). In the frontal lobe, the main projection originated from F2, although labeled cells were also found in F7 and area 46. Preliminary results obtained from injections in nearby areas PEc and V6Av revealed connections different from those of V6Ad. In agreement with functional data, the strong connections with areas where arm‐reaching activity is represented suggest that V6Ad is part of a parietofrontal circuit involved in the control of prehension, and connections with AIP specifically support an involvement in the control of grasping. Connections with areas LIP and Opt are likely related to the oculomotor activities observed in V6Ad.  2009.

title: Neural computation of log likelihood in control of saccadic eye movements
abstract: THE latency between the appearance of a visual target and the start of the saccadic eye movement made to look at it varies from trial to trial to an extent that is inexplicable in terms of ordinary 'physiological' processes such as synaptic delays and conduction velocities. An alternative interpretation is that it represents the time needed to decide whether a target is in fact present: decision processes are necessarily stochastic, because they depend on extracting information from noisy sensory signals1. In one such model2, the presence of a target causes a signal in a decision unit to rise linearly at a rate r from its initial value s 0 until it reaches a fixed threshold 0, when a saccade is initiated. One can regard this decision signal as a neural estimate of the log likelihood of the hypothesis that the target is present, the threshold being the significance criterion or likelihood level at which the target is presumed to be present. Experiments manipulating the prior probability of the target's appearing confirm this notion: the latency distribution then changes in the way expected if s 0 simply reflects the prior log likelihood of the stimulus.

title: A framework for mesencephalic dopamine systems based on predictive Hebbian learning
abstract: We develop a theoretical framework that shows how mesencephalic dopamine systems could distribute to their targets a signal that represents information about future expectations. In particular, we show how activity in the cerebral cortex can make predictions about future receipt of reward and how fluctuations in the activity levels of neurons in diffuse dopamine systems above and below baseline levels would represent errors in these predictions that are delivered to cortical and subcortical targets. We present a model for how such errors could be constructed in a real brain that is consistent with physiological results for a subset of dopaminergic neurons located in the ventral tegmental area and surrounding dopaminergic neurons. The theory also makes testable predictions about human choice behavior on a simple decision-making task. Furthermore, we show that, through a simple influence on synaptic plasticity, fluctuations in dopamine release can act to change the predictions in an appropriate manner.

title: Computational roles for dopamine in behavioural control
abstract: Neuromodulators such as dopamine have a central role in cognitive disorders. In the past decade, biological findings on dopamine function have been infused with concepts taken from computational theories of reinforcement learning. These more abstract approaches have now been applied to describe the biological algorithms at play in our brains when we form value judgements and make choices. The application of such quantitative models has opened up new fields, ripe for attack by young synthesizers and theoreticians.

title: On the programming and reprogramming of actions.
abstract: Actions are often selected in the context of ongoing movement plans. Most studies of action selection have overlooked this fact, implicitly assuming that the motor system is passive prior to presentation of instructions triggering movement selection. Other studies addressed action planning in the context of an already present motor plan, but focused mostly on inhibition of a prepotent response under fierce time pressure. Under these circumstances, inhibition of previous motor plans and selection of a new response become temporally intermingled. Here, we explore how the presence of earlier motor plans influences cerebral effects associated with action selection, separating in time movement programming, reprogramming, and execution. We show that portions of parietofrontal circuits, including intraparietal sulcus and left dorsal premotor cortex, are systematically involved in programming motor responses, their activity being indifferent to the presence of earlier motor plans. We identify additional regions recruited when a motor response is programmed in the context of an existing motor program. We found that several right-hemisphere regions, previously associated with response inhibition, might be better characterized as involved in response selection. Finally, we detail the specific role of a right precentral region in movement reprogramming that is involved in inhibiting not only actual responses but also motor representations.

title: Transformation of a Virtual Action Plan into a Motor Plan in the Premotor Cortex
abstract: Before preparing to initiate a forthcoming motion, we often acquire information about the future action without specifying actual motor parameters. The information for planning an action at this conceptual level can be provided with verbal commands or nonverbal signals even before the associated motor targets are visible. Under these conditions, the information signifying a virtual action plan must be transformed to information that can be used for constructing a motor plan to initiate specific movements. To determine whether the premotor cortex is involved in this process, we examined neuronal activity in the dorsal premotor cortex (PMd) of monkeys performing a behavioral task designed to isolate the behavioral stages of the acquisition of information for a future action and the construction of a motor plan. We trained the animals to receive a symbolic instruction (color and shape of an instruction cue) to determine whether to select the right or left of targets to reach, despite the physical absence of targets. Subsequently, two targets appeared on a screen at different locations. The animals then determined the correct target (left or right) based on the previous instruction and prepared to initiate a reaching movement to an actual target. The experimental design dissociated the selection of the right/left at an abstract level (action plan) from the physical motor plan. Here, we show that activity of individual PMd neurons initially reflects a virtual action plan transcending motor specifics, before these neurons contribute to a transformation process that leads to activity encoding a motor plan.

title: Orienting of Attention*
abstract: Bartlett viewed thinking as a high level skill exhibiting ballistic properties that he called its “point of no return”. This paper explores one aspect of cognition through the use of a simple model task in which human subjects are asked to commit attention to a position in visual space other than fixation. This instruction is executed by orienting a covert (attentional) mechanism that seems sufficiently time locked to external events that its trajectory can be traced across the visual field in terms of momentary changes in the efficiency of detecting stimuli. A comparison of results obtained with alert monkeys, brain injured and normal human subjects shows the relationship of this covert system to saccadic eye movements and to various brain systems controlling perception and motion. In accordance with Bartlett's insight, the possibility is explored that similar principles apply to orienting of attention toward sensory input and orienting to the semantic structures used in thinking.

title: Cerebral correlates of alerting, orienting and reorienting of visuospatial attention: an event-related fMRI study
abstract: The identification of brain systems contributing to different aspects of visuospatial attention is of both clinical and theoretical interest. Cued target detection tasks provide a simple means to dissociate attentional subcomponents, such as alerting, orienting or reorienting of attention. Event-related functional magnetic resonance imaging (fMRI) was used to study neural correlates of these distinct attentional processes. Volunteers were scanned while performing a centrally cued target detection task. Four different types of trials (no cue, neutral cue, valid cue and invalid cue trials) with targets appearing either in the right or left hemifield were randomly intermixed. Behaviourally, the data provided evidence for alerting, spatial orienting and reorienting of attention. Neurally, the alerting effect was seen in bilaterally increased extrastriatal blood oxygenation level-dependent (BOLD) activity in neutral as compared to no cue trials. Neural correlates of spatial orienting were seen in anterior cingulate cortex, which was more active during valid as compared to neutral cue trials. Neural correlates of reorienting of attention, that is, higher BOLD activity to invalid as compared to validly cued trials were evident in several brain regions including left and right intraparietal sulcus, right temporo-parietal junction and middle frontal gyrus bilaterally. The data suggest that frontal and parietal regions are specifically involved in reorienting rather than orienting attention to a spatial position. Alerting effects were seen in extrastriate regions which suggest that increased phasic alertness results in a top-down modulation of neural activity in visual processing areas.

title: Dopamine neurons report an error in the temporal prediction of reward during learning
abstract: Many behaviors are affected by rewards, undergoing long-term changes when rewards are different than predicted but remaining unchanged when rewards occur exactly as predicted. The discrepancy between reward occurrence and reward prediction is termed an 'error in reward prediction'. Dopamine neurons in the substantia nigra and the ventral tegmental area are believed to be involved in reward-dependent behaviors. Consistent with this role, they are activated by rewards, and because they are activated more strongly by unpredicted than by predicted rewards they may play a role in learning. The present study investigated whether monkey dopamine neurons code an error in reward prediction during the course of learning. Dopamine neuron responses reflected the changes in reward prediction during individual learning episodes; dopamine neurons were activated by rewards during early trials, when errors were frequent and rewards unpredictable, but activation was progressively reduced as performance was consolidated and rewards became more predictable. These neurons were also activated when rewards occurred at unpredicted times and were depressed when rewards were omitted at the predicted times. Thus, dopamine neurons code errors in the prediction of both the occurrence and the time of rewards. In this respect, their responses resemble the teaching signals that have been employed in particularly efficient computational learning models.

title: Dissociable Systems for Gain- and Loss-Related Value Predictions and Errors of Prediction in the Human Brain
abstract: Midbrain dopaminergic neurons projecting to the ventral striatum code for reward magnitude and probability during reward anticipation and then indicate the difference between actual and predicted outcome. It has been questioned whether such a common system for the prediction and evaluation of reward exists in humans. Using functional magnetic resonance imaging and a guessing task in two large cohorts, we are able to confirm ventral striatal responses coding both reward probability and magnitude during anticipation, permitting the local computation of expected value (EV). However, the ventral striatum only represented the gain-related part of EV (EV+). At reward delivery, the same area shows a reward probability and magnitude-dependent prediction error signal, best modeled as the difference between actual outcome and EV+. In contrast, loss-related expected value (EV−) and the associated prediction error was represented in the amygdala. Thus, the ventral striatum and the amygdala distinctively process the value of a prediction and subsequently compute a prediction error for gains and losses, respectively. Therefore, a homeostatic balance of both systems might be important for generating adequate expectations under uncertainty. Prevalence of either part might render expectations more positive or negative, which could contribute to the pathophysiology of mood disorders like major depression.

title: Human striatal activation reflects degree of stimulus saliency
abstract: Salient stimuli are characterized by their capability to perturb and seize available cognitive resources. Although the striatum and its dopaminergic inputs respond to a variety of stimuli categorically defined as salient, including rewards, the relationship between striatal activity and saliency is not well understood. Specifically, it is unclear if the striatum responds in an all-or-none fashion to salient events or instead responds in a graded fashion to the degree of saliency associated with an event. Using functional magnetic resonance imaging, we measured activity in the brains of 20 participants performing a visual classification task in which they identified single digits as odd or even numbers. An auditory tone preceded each number, which was occasionally, and unexpectedly, substituted by a novel sound. The novel sounds varied in their ability to interrupt and reallocate cognitive resources (i.e., their saliency) as measured by a delay in reaction time to immediately subsequent numerical task-stimuli. The present findings demonstrate that striatal activity increases proportionally to the degree to which an unexpected novel sound interferes with the current cognitive focus, even in the absence of reward. These results suggest that activity in the human striatum reflects the level of saliency associated with a stimulus, perhaps providing a signal to reallocate limited resources to important events.

title: Anticipation of novelty recruits reward system and hippocampus while promoting recollection
abstract: The dopaminergic midbrain, which comprises the substantia nigra and ventral tegmental area (SN/VTA), plays a central role in reward processing. This region is also activated by novel stimuli, raising the possibility that novelty and reward have shared functional properties. It is currently unclear whether functional aspects of reward processing in the SN/VTA, namely, activation by unexpected rewards and cues that predict reward, also characterize novelty processing. To address this question, we conducted an fMRI experiment during which subjects viewed symbolic cues that predicted either novel or familiar images of scenes with 75% validity. We show that SN/VTA was activated by cues predicting novel images as well as by unexpected novel images that followed familiarity-predictive cues, an ‘unexpected novelty’ response. The hippocampus, a region implicated in detecting and encoding novel stimuli, showed an anticipatory novelty response but differed from the response profile of SN/VTA in responding at outcome to expected and ‘unexpected’ novelty. In a behavioral extension of the experiment, recollection increased relative to familiarity when comparing delayed recognition memory for anticipated novel stimuli with unexpected novel stimuli. These data reveal commonalities in SN/VTA responses to anticipating reward and anticipating novel stimuli. We suggest that this anticipatory response codes a motivational exploratory novelty signal that, together with anticipatory activation of the hippocampus, leads to enhanced encoding of novel events. In more general terms, the data suggest that dopaminergic processing of novelty might be important in driving exploration of new environments.

title: Striatal Activity Underlies Novelty-Based Choice in Humans
abstract: The desire to seek new and unfamiliar experiences is a fundamental behavioral tendency in humans and other species. In economic decision making, novelty seeking is often rational, insofar as uncertain options may prove valuable and advantageous in the long run. Here, we show that, even when the degree of perceptual familiarity of an option is unrelated to choice outcome, novelty nevertheless drives choice behavior. Using functional magnetic resonance imaging (fMRI), we show that this behavior is specifically associated with striatal activity, in a manner consistent with computational accounts of decision making under uncertainty. Furthermore, this activity predicts interindividual differences in susceptibility to novelty. These data indicate that the brain uses perceptual novelty to approximate choice uncertainty in decision making, which in certain contexts gives rise to a newly identified and quantifiable source of human irrationality.

title: Functional architecture of basal ganglia circuits: neural substrates of parallel processing
abstract: Concepts of basal ganglia organization have changed markedly over the past decade, due to significant advances in our understanding of the anatomy, physiology and pharmacology of these structures. Independent evidence from each of these fields has reinforced a growing perception that the functional architecture of the basal ganglia is essentially parallel in nature, regardless of the perspective from which these structures are viewed. This represents a significant departure from earlier concepts of basal ganglia organization, which generally emphasized the serial aspects of their connectivity. Current evidence suggests that the basal ganglia are organized into several structurally and functionally distinct 'circuits' that link cortex, basal ganglia and thalamus, with each circuit focused on a different portion of the frontal lobe. In this review, Garrett Alexander and Michael Crutcher, using the basal ganglia 'motor' circuit as the principal example, discuss recent evidence indicating that a parallel functional architecture may also be characteristic of the organization within each individual circuit.

title: Multiple reward signals in the brain
abstract: The fundamental biological importance of rewards has created an increasing interest in the neuronal processing of reward information. The suggestion that the mechanisms underlying drug addiction might involve natural reward systems has also stimulated interest. This article focuses on recent neurophysiological studies in primates that have revealed that neurons in a limited number of brain structures carry specific signals about past and future rewards. This research provides the first step towards an understanding of how rewards influence behaviour before they are received and how the brain might use reward information to control learning and goal-directed behaviour.

title: Effects of motor preparation and spatial attention on corticospinal excitability in a delayed-response paradigm
abstract: The preparation of motor responses during the delay period of an instructed delay task is associated with sustained neural firing in the primate premotor cortex. It remains unclear how and when such preparation-related premotor activity influences the motor output system. In this study, we tested modulation of corticospinal excitability using single-pulse transcranial magnetic stimulation (TMS) during a delayed-response task. At the beginning of the delay interval participants were either provided with no information, spatial attentional information concerning location but not identity of an upcoming imperative stimulus, or information regarding the upcoming response. Behavioral data indicate that participants used all information available to them. Only when information concerning the upcoming response was provided did corticospinal excitability show differential modulation for the effector muscle compared to other task-unrelated muscles. We conclude that modulation of corticospinal excitability reflects specific response preparation, rather than non-specific event preparation.

title: Expectancy Induces Dynamic Modulation of Corticospinal Excitability
abstract: Behavioral studies using motor preparation paradigms have revealed that increased expectancy of a response signal shortens reaction times (RTs). Neurophysiological data suggest that in such paradigms, not only RT but also neuronal activity in the motor structures involved is modulated by expectancy of behaviorally relevant events. Here, we directly tested whether expectancy of a response signal modulates excitability of the corticospinal system used in the subsequent movement. We combined single- and paired-pulse transcranial magnetic stimulation (TMS) over the primary motor cortex with a simple RT task with variable preparatory delays. We found that, in line with typical behavioral observations, the subjects' RTs decreased with increasing response signal expectancy. TMS results revealed a modulation of corticospinal excitability in correspondence with response signal expectancy. Besides an increased excitability over the time-course of the preparatory delay, corticospinal excitability transiently increased whenever a response signal was expected. Paired-pulse TMS showed that this modulation is unlikely to be mediated by excitability changes in interneuronal inhibitory or facilitatory networks in the primary motor cortex. Changes in corticospinal synchronization or other mechanisms involving spinal circuits are candidates mediating the modulation of corticospinal excitability by expectancy.

title: THE BASAL GANGLIA: FOCUSED SELECTION AND INHIBITION OF COMPETING MOTOR PROGRAMS
abstract: The basal ganglia comprise several nuclei in the forebrain, diencephalon, and midbrain thought to play a significant role in the control of posture and movement. It is well recognized that people with degenerative diseases of the basal ganglia suffer from rigidly held abnormal body postures, slowing of movement, involuntary movements, or a combination of these a abnormalities. However, it has not been agreed just what the basal ganglia contribute to normal movement. Recent advances in knowledge of the basal ganglia circuitry, activity of basal ganglia neurons during movement, and the effect of basal ganglia lesions have led to a new hypothesis of basal ganglia function. The hypothesis states that the basal ganglia do not generate movements. Instead, when voluntary movement is generated by cerebral cortical and cerebellar mechanisms, the basal ganglia act broadly to inhibit competing motor mechanisms that would otherwise interfere with the desired movement. Simultaneously, inhibition is removed focally from the desired motor mechanisms to allow that movement to proceed. Inability to inhibit competing motor programs results in slow movements, abnormal postures and involuntary muscle activity.

title: Differential Responses in Human Striatum and Prefrontal Cortex to Changes in Object and Rule Relevance
abstract: Event-related functional magnetic resonance imaging was used to measure blood oxygenation level-dependent responses in 16 young healthy human volunteers during performance of an attentional switching task. The task allowed the separate investigation of lower-order switching between concrete objects and higher-order switching between abstract task rules. Significant signal change in the ventral striatum was demonstrated on trials when subjects switched between objects but not when subjects switched between abstract task rules. In contrast, signal change in the lateral prefrontal cortex (PFC) was observed during all switch trials. The switch-related responses were not contaminated by task difficulty, because the greatest signal change was observed during the relatively easy switch trials, which required both lower-order and higher-order switching at the same time. The present data suggest that mechanisms of inhibitory response control in frontostriatal systems are organized according to distinct levels of abstraction. Specifically, the response selection computation carried by the ventral striatum, which projects to the orbitofrontal cortex and the medial PFC, is restricted to the transformation of concrete stimulus exemplar information into motor responses, whereas the adaptive function of the lateral PFC extends to the transformation of abstract task-rule representations into action.

title: The Human Striatum is Necessary for Responding to Changes in Stimulus Relevance
abstract: Various lines of evidence suggest that the striatum is implicated in cognitive flexibility. The neuropsychological evidence has, for the most part, been based on research with patients with Parkinson's disease, which is accompanied by chemical disruption of both the striatum and the prefrontal cortex. The present study examined this issue by testing patients with focal lesions of the striatum on a task measuring two forms of cognitive switching. Patients with striatal, but not frontal lobe lesions, were impaired in switching between concrete sensory stimuli. By contrast, both patient groups were unimpaired when switching between abstract task rules relative to baseline nonswitch trials. These results reveal a dissociation between two distinct forms of cognitive f lexibility, providing converging evidence for a role of the striatum in f lexible control functions associated with the selection of behaviorally relevant stimuli.

title: Making Working Memory Work: A Computational Model of Learning in the Prefrontal Cortex and Basal Ganglia
abstract: The prefrontal cortex has long been thought to subserve both working memory (the holding of information online for processing) and executive functions (deciding how to manipulate working memory and perform processing). Although many computational models of working memory have been developed, the mechanistic basis of executive function remains elusive, often amounting to a homunculus. This article presents an attempt to deconstruct this homunculus through powerful learning mechanisms that allow a computational model of the prefrontal cortex to control both itself and other brain areas in a strategic, task-appropriate manner. These learning mechanisms are based on subcortical structures in the midbrain, basal ganglia, and amygdala, which together form an actor-critic architecture. The critic system learns which prefrontal representations are task relevant and trains the actor, which in turn provides a dynamic gating mechanism for controlling working memory updating. Computationally, the learning mechanism is designed to simultaneously solve the temporal and structural credit assignment problems. The model's performance compares favorably with standard backpropagation-based temporal learning mechanisms on the challenging 1-2-AX working memory task and other benchmark working memory tasks.

title: Synaptic Plasticity and Dysconnection in Schizophrenia
abstract: Current pathophysiological theories of schizophrenia highlight the role of altered brain connectivity. This dysconnectivity could manifest 1) anatomically, through structural changes of association fibers at the cellular level, and/or 2) functionally, through aberrant control of synaptic plasticity at the synaptic level. In this article, we review the evidence for these theories, focusing on the modulation of synaptic plasticity. In particular, we discuss how dysconnectivity, observed between brain regions in schizophrenic patients, could result from abnormal modulation of N-methyl-D-aspartate (NMDA)-dependent plasticity by other neurotransmitter systems. We focus on the implication of the dysconnection hypothesis for functional imaging at the systems level. In particular, we review recent advances in measuring plasticity in the human brain using functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) that can be used to address dysconnectivity in schizophrenia. Promising experimental paradigms include perceptual and reinforcement learning. We describe how theoretical and causal models of brain responses might contribute to a mechanistic understanding of synaptic plasticity in schizophrenia.