ReviewReward , Motivation , and Reinforcement Learning brain

brain report errors in the prediction of reward has been a powerful (though not undisputed) organizing force for a wealth of experimental data (see Schultz et al., 1997; Peter Dayan1,3 and Bernard W. Balleine2,3 Gatsby Computational Neuroscience Unit University College London Schultz, 2002 [this issue of Neuron]; Montague and 17 Queen Square Berns, 2002 [this issue of Neuron]). This theory derives London WC1N 3AR from reinforcement learning (Sutton and Barto, 1998), United Kingdom which shows how a particular form of the error (called 2 Department of Psychology and the temporal difference error) can be used to learn prethe Brain Research Institute dictions of reward delivery and also how the predictions University of California, Los Angeles can be used to learn to choose an adaptive course of Box 951563 action in terms of maximizing reward and minimizing Los Angeles, California 90095 punishment (Houk et al., 1995; Suri and Schultz, 1998, 1999). A popular reinforcement learning model called the actor-critic has been used to offer an account of the There is substantial evidence that dopamine is inmain subdivisions of the midbrain dopamine system. In volved in reward learning and appetitive conditioning. one version of this, the dopamine cells in the ventral However, the major reinforcement learning-based thetegmental area (VTA) and the substantia nigra pars comoretical models of classical conditioning (crudely, prepacta (SNc) report the same prediction error, but turn diction learning) are actually based on rules designed it to two different uses. VTA dopamine cells are associto explain instrumental conditioning (action learning). ated with the critic, controlling the learning of values Extensive anatomical, pharmacological, and psychoheld in the basolateral nucleus of the amygdala and the logical data, particularly concerning the impact of moorbitofrontal cortex. SNc dopamine cells are associated tivational manipulations, show that these models are with the actor, controlling the learning of actions in comunreasonable. We review the data and consider the petitive cortico-striato-thalamo-cortical loops (Alexaninvolvement of a rich collection of different neural sysder and Crutcher, 1990). tems in various aspects of these forms of conditioning. Here, we evaluate the actor-critic model of the dopaDopamine plays a pivotal, but complicated, role. mine system from the perspective of the substantial psychological and neurobiological data on motivation, Behavioral psychologists have long made a set of diswhich is key to the modern view of reward learning. The tinctions between classical/Pavlovian conditioning and actor-critic model can be seen as a particular form of instrumental/operant conditioning. Both forms of condione of the better accepted motivational theories, namely tioning concern the ways that animals learn to predict incentive learning theory (Bindra, 1974, 1978; Bolles, and respond to important events in their environments, 1975; Toates 1986, 1994), in which stimuli associated such as the delivery of appetitive and aversive stimuli with positive reinforcers such as food or water (which (food/water and mild electric shocks, respectively). In are known as Pavlovian excitors) are thought to act classical conditioning, the outcomes are provided whatas conditioned incentives for instrumental conditioning. ever the animals do, and so any changes in behavior This is the conditioned reinforcement effect, that if a rat presumably reflect innately specified reactions to preobserves that a particular light always comes on just dictions of the outcomes. Conversely, in instrumental before it receives some food, it will then learn to press conditioning, whether or not an animal gets a reward or a lever in order to get that light, even if in those learning punishment depends on the actions it performs. Instrutrials no food is provided. Despite this solid foundation, mental conditioning is thus closely related to the engithe actor-critic model fails to take into account large neering theory of optimal control and the computer scibodies of work (recently reviewed from a theoretical ence theory of reinforcement learning, which both study perspective in Berridge, 2001; Dickinson and Balleine, how systems of any sort can choose their actions to 2002) on the many psychological differences between maximize rewards or minimize punishments. The operaPavlovian and instrumental conditioning in terms of motional distinction between classical and instrumental tivational processes and action determination and also conditioning may seem like the sort of thing about which on the neural underpinnings of these differences. only dyed-in-the-wool behaviorists could care. HowThese data, together with those arguing against a ever, critical issues turn on it, such as the organization simple conflation of Pavlovian conditioned responses, of goal-directed behavior and motivational control, e.g., goal-directed instrumental actions, and habitual instruthe way that animals head for water if thirsty but food mental actions, force substantial changes to both asif hungry. In this review, we argue that powerful and pects of the actor-critic model. Most critically, the founpredictive theories of the neural basis of appetitive condations of the relationship between Pavlovian and ditioning have failed to respect important aspects of the instrumental conditioning in the existing dopaminergic distinction and point the way toward their improvement. model are decimated by recent data suggesting that The idea that dopamine cells in the vertebrate midthere are at least two independent, largely anatomically distinct, reward processes, only one of which appears to be sensitive to dopaminergic manipulations. Modifi3 Correspondence: dayan@gatsby.ucl. ac.uk (P.D.), balleine@