Behavioral and Brain Functions

Substantial evidence suggests that the phasic activities of dopaminergic neurons in the primate midbrain represent a temporal difference (TD) error in predictions of future reward, with increases above and decreases below baseline consequent on positive and negative prediction errors, respectively. However, dopamine cells have very low baseline activity, which implies that the representation of these two sorts of error is asymmetric. We explore the implications of this seemingly innocuous asymmetry for the interpretation of dopaminergic firing patterns in experiments with probabilistic rewards which bring about persistent prediction errors. In particular, we show that when averaging the non-stationary prediction errors across trials, a ramping in the activity of the dopamine neurons should be apparent, whose magnitude is dependent on the learning rate. This exact phenomenon was observed in a recent experiment, though being interpreted there in antipodal terms as a within-trial encoding of uncertainty. Introduction There is an impressively large body of physiological, imaging, and psychopharmacological data regarding the phasic activity of dopaminergic (DA) cells in the midbrains of monkeys, rats and humans in classical and instrumental conditioning tasks involving predictions of future rewards [1-5]. These data have been taken to suggest [6,7] that the activity of DA neurons represents temporal difference (TD) errors in the predictions of future reward [8,9]. This TD theory of dopamine provides a precise computational foundation for understanding a host of behavioural and neural data. Furthermore, it suggests that DA provides a signal that is theoretically appropriate for controlling learning of both predictions and reward-optimising actions. Some of the most compelling evidence in favour of the TD theory comes from studies investigating the phasic activation of dopamine cells in response to arbitrary stimuli (such as fractal patterns on a monitor) that predict the proximate availability of rewards (such as drops of juice). In many variants, these have shown that with training, phasic DA signals transfer from the time of the initially unpredictable reward, to the time of the earliest cue predicting a reward. This is exactly the expected outcome for a temporal-difference based prediction error (eg. [1,2,1013]). The basic finding [7] is that when a reward is unexpected (which is inevitable in early trials), dopamine cells respond strongly to it. When a reward is predicted, however, the cells respond to the predictor, and not to the now-expected reward. If a predicted reward is unexpectedly omitted, then the cells are phasically inhibited at the normal time of the reward, an inhibition which reveals the precise timing of the reward prediction [10], and whose temporal metrics Published: 04 May 2005 Behavioral and Brain Functions 2005, 1:6 doi:10.1186/1744-9081-1-6 Received: 12 February 2005 Accepted: 04 May 2005 This article is available from: http://www.behavioralandbrainfunctions. com/content/1/1/6 © 2005 Niv et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.