Action-modulated midbrain dopamine activity arises from distributed control policies
Authors: Jack Lindsey
Ashok Litwin-Kumar
Venue: NeurIPS 2022 Accept (2022)
Keywords: neuroscience
reinforcement learning
dopamine
basal ganglia
off-policy learning

Abstract: Animal behavior is driven by multiple brain regions working in parallel with distinct control policies. We present a biologically plausible model of off-policy reinforcement learning in the basal ganglia, which enables learning in such an architecture. The model accounts for action-related modulation of dopamine activity that is not captured by previous models that implement on-policy algorithms.  In particular, the model predicts that dopamine activity signals a combination of reward prediction error (as in classic models) and "action surprise," a measure of how unexpected an action is relative to the basal ganglia's current policy. In the presence of the action surprise term, the model implements an approximate form of $Q$-learning.  On benchmark navigation and reaching tasks, we show empirically that this model is capable of learning from data driven completely or in part by other policies (e.g. from other brain regions).  By contrast, models without the action surprise term suffer in the presence of additional policies, and are incapable of learning at all from behavior that is completely externally driven.  The model provides a computational account for numerous experimental findings about dopamine activity that cannot be explained by classic models of reinforcement learning in the basal ganglia.  These include differing levels of action surprise signals in dorsal and ventral striatum, decreasing amounts movement-modulated dopamine activity with practice, and representations of action initiation and kinematics in dopamine activity. It also provides further predictions that can be tested with recordings of striatal dopamine activity.

Action-modulated midbrain dopamine activity arises from distributed control policies
Jack Lindsey Department of Neuroscience Columbia University
Ashok Litwin-Kumar Department of Neuroscience Columbia University
New York, NY jackullindsey@@mail.co com
New York, NY
1 l.iiiinn-kumrr@lllluubi.e
Abstract
Animal behavior is driven by multiple brain regions working in parallel with dis- tinet control policies. We present a biologically plausible model of off- off-pooicy reinforcement learning the basal ganglia, which enables learning in such an architecture. The model accounts for action-related modulation of dopamine activ- ity that is not captured by previous models that implement on-policy algorithms. In particular, the model predicts that dopamine activity signals combination of reward prediction error (as in classic models) and "action surprise," a measure how unexpected action is relative to the basal ganglia' current policy. In the presence the action surprise term, the model implements an approximate form of Q-learning. On benchmark navigation and reaching tasks, show empirically that this model is capable of learning from data driven completely or part by other policies (e.g. from other brain regions). By contrast, models without the action surprise term suffer in the presence of additional policies, and are incapable of learning all from behavior that is completely externally driven. The model pro- vides a computational account for numerous experimental findings about dopamine activity that cannot be explained by classic models of reinforcement learning in the basal ganglia. These include differing levels of action surprise signals in dorsal and ventral striatum, decreasing amounts of ceemmnttmmodllteed dopamine activity with practice, and representations of action initiation and kinematics in dopamine activity. also provides further predictions that can be tested with recordings of striatal dopamine activity.
Introduction
An extensive body of work has sought to account for the function of the basal ganglia using the computational framework of reinforcement learning (RL), in particular RL algorithms for action selection and value learning 30 47! 29 58). Striatal neurons within the basal ganglia integrate diverse inputs, including projections from across the cerebral cortex. The activity of these neurons, in particular neurons in the dorsal striatum, plays key role in action selection 44]. On the other hand, neurons in the ventral striatum have been shown to encode the learned value of stimuli 10 43 45
The phasic activity of midbrain dopamine neurons projecting to the striatum gates plasticity at cortico- striatal synapses (63 g M and may therefore modulate the learning of actions and values. Indeed, many RL algorithms, the learning of state-to-action mappings ("policies") and value estimates modulated by scalar factor known as the "advantage" or "reward prediction error" (RPE), which measures deviations in attained reward from expectations based on learned value estimates. Numerous experiments have shown that striatal dopamine activity encodes an RPE-like signal [30 43 26
"Code for experiments is provided https///hhhaaooo/////ccss//oooon
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Collectively, these findings suggest model in which the basal ganglia implements an online actor- critic RL algorithm. where ventral and dorsal striatal subregions play the roles of critic and actor,
respectively, and dopaminergic activity encodes the advantage (RPE) signal 30 This model has
been extended variety of ways to incorporate more biological detail [56 57
55
ideas
from model-based RI. [5]. distributional RL [K]] and ml. [61
Despite these promising links, there remain challenges the view of basal ganglia as implementing an actor-critic RL algorithm. In this work, we address two such challenges. First, dopamine activity in the basal ganglia observed to encode other information beside RPEs [g particularly signals relating to movement initiation and vigor [12 67]. Second, the classic actor-critic model of the basal ganglia is an on-policy RL algorithm ganglia's 's policy. However, motor control in the brain is distributed, with multiple brain regions including the motor cortex and cerehellum exerting influence on behavior [9 64 52 G1. We show that classic actor-critic models of the basal ganglia fail in this scenario and argue that the ability to learn from off- policy experience is essential to models of basal ganglia learning.
designed to learn from experiences driven by the basal
We present a biologically plausible model of off--policy RL in continuous action spaces, Our model differs from the classic actor-critic model of the basal ganglia by adding an additional action-related term to dopamine activity. We call this term "action surprise," as it measures deviation of the action an agent takes in particular state from the typical action that the output of the basal ganglia would select in that state. We show mathematically that, with this addition, the algorithm implements an approximate form of Q-learning in continuous action spaces proposed by [26]. Using simulations, we show that action surprise is essential to effective learning when controllers other than the basal ganglia also contribute to behavior. Thus, aation-related activity need not be understood as an independent function of midbrain dopamine neurons separate from their role in learning, but rather as necessary component of an algorithm that enables off-policy RL. This action surprise model accounts for several experimental findings about movement-related dopamine activity in the basal ganglia. Finally, we describe predictions our model regarding dopamine activity.
2 Background: Connection between policy gradient algorithms and cortico-stritall plasticity
Here we briefly introduce the connection between on-policy policy gradient algorithms and synaptic plasticity rules in the striatum. In what follows, we assume familiarity with standard notation and concepts in RL; for brief review, see Appendix A.I Policy gradient algorithms update the parameters 0p of the policy according to
A x Ve, log (og)(ar)s:)d,
(I)
where 6, an estimate of the advantage function A(s, at) Q(s,, ar) V(s:).
Throughout this work we assume that policies (als)) are parameterized by Gaussian distributions a, (s) ).). For simplicity, will assume o; for all i; however, our results are easily extended to the case of heterogeneous o, Letting ,₂ be the parameters of (s)s,, we have
~0 x 7² 8c (ar - ((:))) ((s)).)).
(2)
Supposing our policy is parameterized by linear map from feature representation of st, (u(s)) W Wa(((s) this becomes:
AW _ (a. uls:)) (T..)
This is a "three-factor" learning rule 200 for synapse Waj obtained by multiplying presynaptic activity (St).. a postsynaptic term (az); - (st), measuring the deviation of the sampled action from the typical action in this state, and a third factor o,. RL models of the basal ganglia assume that (at): - (s,)s) is available to the postsynaptic neuron and that de is signaled by dopamine release in the striatum [58] EE! Experiments have indeed shown that a coincidence of dopamine release and pre and post-synaptic neural activity enables plasticity cortico-striatal synapses 63 Biological implementations of this learning rule are discussed further in Appendix A.2
In actor-critic models the & factor is
8, = T(+1 + + (s:))
(4)
2

---

often referred to as "reward prediction error" (RPE). Here, V is an estimate of the value function output by a "critic" network, separate from the policy network, which learns its parameters 0y using temporal difference (TD) learning:
A0v o de Vo, (s.).
(5)
In models of the basal ganglia, the ventral striatum is often assigned the role of the critic, as it is
Importantly, TD learning uses the same quantity 6G for learning the value function as for policy learning. Hence, scalar 6, signal measuring RPE and broadcast across the striatum supports learning for both the actor and critic. RPE captures key experimental features of striatal dopamine reesvity-rssoonees to unexpected reward or reward- preaddctreeetvveeeeee cues, no significant response to expected reward, and suppression response to unexpected lack of reward [50 43 26
implicated in value-learning but less in controlling actions 4 14
3 Off-policy RL through action surprise signals in dopamine activity
The actor-critic algorithm described above is an on-policy algorithm. Hence, the algorithm may perform suboptimally, or fail altogether, if the actions used for learning are not (always) sampled from the learned policy. Off-policy algorithms are often used to enable learning from replay buffer of past experiences, expert demonstrations, and/or a separate exploration policy [2]. In a biological
context, we argue that the fact that other brain regions can exert control over behavior independently of the basal ganglia further motivates off-policy RL. Indeed, we will show empirically that standard on-policy algorithms suffer when other controllers exert (partial) control over an agent's 's behavior.
3.1 Action-sensitive dopamine activity arises from parameterized Q-learning
While there are a variety of approaches to off-policy RL, many require learning an estimate of the Q-function ((s, a) rather than (s).. Q-learning iteratively minimizes the following loss function:
M++1 Q(Ss, ar)
Me+1 T++1 + max Q(5+11
Computing the quantity maxa Q(s, a) directly is intractable in high-dimensional, continuous action spaces. A variety of approaches to this problem have been proposed. Here focus approach adopted by [23), which involves restricting the form of the Q-function estimate to a family of functions whose maximum is easy compute (we discuss other techniques for continuous Q-learning and off-policy RI., which we argue are less biologically realistic, in Appendix A.3
We parameterize the Q-function as follows:
1 ( a) = V(s) - ||a (())
For now we treat the the scaling factor a as a fixed hyperparameter; however, it can also be learned online (see Appendix A.4b. This parameterization is a special case of the one proposed in [X]
primary insight of our work is the observation that, under the parameterization of Eq. gradient updates of the loss function of Eq. yield biologically plausible actor-critic algorithm with action- sensitive dopamine activity. In particular, taking the gradient of the Q-learning loss function with respect the parameters By and 0₂ of V and μr, respectively, yields the following learning updates:
Adv x ot Vav (st).
(9)
1
A0 μ ot (a vo,s)))Vo.
(10)
where
1 ((s))= 2z ||ar u(s)))?
8t T++l + +
(11)
= 6, + 0²
la, ?1
(12)
3

---

These update equations are the same as those of the on-policy advantage actor-critic algorithm, but with one additional term added the dopaminergic signal. Now this signal 8 represents the sum of classic RPE and ||a u())ll which measures the deviation of the sampled action from the action most likely to be chosen by the actor. We call this term "action surprise." It can be interpreted as a reduction in the predicted value ((: as) actions that deviate from the basal ganglia's 's policy.
We note that our model is agnostic to whether action surprise is encoded by the same neurons as RPE. It may be encoded by separate neurons as long as they release dopamine in the same areas as RPE-signaling dopamine. We also note that the update equation (Eq. for the actor uses the action a, taken by the agent. Biologically, this requires an efferent copy of the agent's action (taking into account the influence of other controllers) be sent to the striatal projection neurons representing action in the basal ganglia. This architecture is consistent with the presence of pathways from motor cortex, thalamus, and cerebellum to striatal projection neurons connections involved in our model is depicted Fig.
37 25
36
A schematic of the
It is easy to misinterpret the effect of the dopaminergic action surprise as encouraging the basal ganglia's 's policy to imitate those other controllers. This is in fact the case. Later, we show empir- ically that this model robustly enables off-policy learning even in the presence of poorly performing external controllers. Below, however, we show how imitation learning can be implemented.
State features (s,)
Reward
Basal ganglia actor-critic system
Dopamine 6,'
External controller
Actor (dorsal striatum) Crilic (ventral strialum)
Value (Re)
Efference
copy
Action surprise
|la p(ss)]l
Action
Figure 1: Schematic of model architecture. Actions are driven in parallel by an actor-critic architecture in the basal ganglia (gray region) and an external controller, representing other brain regions (orange). Both systems receive state information via a feature representation ((s{) (e.g. cortico-striatal projection neuron activity). The actor-critic module performs RL with weight updates modulated by 6,,, which a combination of both RPE and action surprise. Dashed lines indicate architectural features of the action surprise model that are not present in the classic actor-critic model.
3.2 Differential action surprise signals in dorsal and ventral striatum
The formulation above predicts that the same dopamine signal 8+ is broadcast to both the actor and critic. However, the ratio movement-related to reward-related modulation of dopamine activity appears vary across the striatum, with more weveemettrrlatee activity in dorsal regions [27] In our modeling framework, this corresponds to the action surprise term being weighted more strongly in the actor (dorsal striatum) than the critic (ventral striatum). Such asymmetry is produced by adding an additional term to the weight updates of the actor, proportional to
a,
(s:)) (a, Vo,
(13)
This update is aligned with the gradient with respect to 0₂ of the following loss function:
Coup E ||a, (=
(14)
with the learning rate itself governed by the magnitude of action surprise. Hence, these updates encourage the basal ganglia policy to imitate the agent's 's behavioral policy steps with large action surprise. Heuristically, steps with large action surprise are more likely have been driven by an external controller. An additional action surprise contribution to actor-modulating dopamine activity

---

can therefore be interpreted as an approximate supervised learning signal encouraging imitation of the external controller's 's policy. We show in simulations that this effect helps the basal ganglia more rapidly consolidate expert policies of other controllers.
4 Experimental Setup
Tasks. We simulated two simple continuous control tasks to demonstrate the role of the action surprise term in off--ooliy learning. See Fig. A for illustrations.
Open-field navigation: An agent is positioned in two-dimensional square environment with con- tinuous coordinates. On each trial, the initial position of the agent and a goal location are randomly sampled. The agent controls its and y acceleration.
Two-joint: An agent controls a two-joint arm with two equal-length segments. The arm position and a target location are randomly sampled at the beginning of each trial. The agent outputs torques at each the joints order to move the most peripheral point of the arm toward the target.
In both tasks the cost incurred by the agent at each time step is proportional to the inverse of its squared distance from the goal location g at each time step, plus penalties for its squared velocity and acceleration (summed across joints in the two-joint arm case)
dx(t) dt
d x'x(t) dt
Tt= [|x(() gll a1
a2
(15)
We also experimented with a version of the loss function in which rewards for reaching the goal location are sparse and binary, rather than continuous, which gave similar results (see Appendix [A.7].
Model architecture. For both tasks we used a neural network architecture with a single hidden layer. The state inputs the network consisted of the agent's (angular) position and (angular) velocity, as well as the position of the target location, a total of six scalar variables. Each variable was represented
one-hot vector by discretizing its domain into 10 equally spaced bins. These vectors were concatenated and used as the network input. The hidden layer had size 256 and used ReLU nonlinearities. Input weights to the hidden layer were fixed random (Kaiming uniform) initializations. We fixed these weights to avoid the complexity of biological implementations of backpropagation, since this is not the focus of our work (however. we note that the action surprise model can be applied to deep networks by backpropagating gradients through additional layers). Thus, the hidden layer activations served as a fixed feature representation Φbs) of the environment state, and all learning occurs in output weights W,, and Wv which output actions μ = W 4 (s) and value estimates V V= wy- Φ0s)), respectively. We refer to this network as the basal ganglia network.
External controllers and training protocol. To model the influence of other brain regions on behavior, we introduced an additional neural network that also exerted control over the agent's 's actions (Fig. [ orange). To vary the performance of this controller's 's policy, we trained the controller on the task with backpropagation for different numbers of steps: 0 (random policy). 2,000 (intermediate policy), or 100,000 (expert policy). We also varied the degree to which the external controller influences behavior. In the fully on-policy case, the external controller is not used. In the fully off-policy case, the external controller entirely drives the behavior. In the case of partial control, each action is sampled from the basal ganglia's 's network or the external controller with probability 0.5 each. Additionally, we implemented an alternative partial control mechanism, in which the average of the basal ganglia network output and the external controller output is used.
Throughout training, Gaussian exploration noise is added to the output of both networks at each step. In all cases, for the action surprise model, the action a, used for the actor update in Eq. is the action taken by the agent, taking into account the contributions of both controllers and the exploration noise.
On-policy baseline models. We compared the action surprise model with two baselines in which dopamine neurons signal pure RPE. The first described by Eqs. and where the action used the actor update is simply the action a, taken by the agent, as in the action surprise model. As discussed above, this approach is potentially unstable in the presence of other controllers due to the mismatch between the behavioral policy from which actions are sampled and the basal ganglia network's 's policy, As second, potentially more competitive baseline, we instead used the output of the basal ganglia network alone, including exploration noise, in place of a, in Eq. We refer to this as the "no efferent
5

---

copy" model since the basal ganglia is blind to the action ultimately taken by the agent. In this version, the influence of the external controller effectively treated as part of the environment. We note that this approach has no ability even in principle to learn from fully off-policy data, but can learn when the basal ganglia network exerts partial control over actions.
Hyperparameters. For all models we optimized the learning rate and magnitude of exploration noise as typerparameters. For the action surprise model we optimized the coefficient & of the action surprise term as a hyperparameter. To ensure strong baselines we allowed the actor and critic learning rates be optimized separately for the RPE-only models. See Appendix A.5 for details.
Experimental Results
A
B
C
Open-field navigation
RPE (w/ efferent)
RPE (wio efferent)
RPE actinn surprise
Goal
Cost
10k
Two-joint arm
30
Goal
Cost
10k
Episodes
Figure 2: A: Depictions of simulated tasks. B: Example trials in which the action surprise model successfully reaches the target but the RPE-only models fail to. C: Performance of the three models when the basal ganglia shares control of behavior with an external controller. Dark lines indicate mean performance over three runs, and faint individual lines (in this case difficult to see due tight overlap) indicate individual runs. Performance shown reflects policy of the basal ganglia network and external controller. Dashed line indicates the performance of the external controller alone.
We first analyzed performance for the case of actions partially driven by the basal ganglia network and partially driven by the external controller, intended as a representative model of behavioral control distributed across brain regions. Fig, 2 shows that the basal ganglia policy attained superior learning speed and performance than RPE-only models (shown here is a representative case using the intermediate-leved external controller). Fig. 2 shows example trials in which this performance advantage manifests very clearly, with the action surprise model successfully reaching the target while the other models fail completely. The action surprise model remained advantageous when we varied the expertise of the external controller (Fig. (A). The same was true when we assessed combined performance of the basal ganglia network and the external controller (Fig. 3), rather than performance of the basal ganglia network by itself. Notably, the RPE-only model with efferent copy failed to learn in some cases. We found similar results when the contributions of the basal ganglia network and external controller were averaged rather than combined by sampling (Appendix A.6)
We next examined the fully off-policy case. The RPE-only model with efferent copy failed to learn from off- ool-col data in all cases, as expected, as the action term used in the update for this model is uncorrelated with the actions taken by the external controller. Interestingly, the RPE-only model with efferent copy also failed (catastrophically) to learn in all cases. The action surprise model, by contrast, learned both tasks regardless of the quality of the external controller's 's policy (Fig BC).
potential concern is that the action surprise model's improvement for off- off-policy learning comes at the expense of on-policy learning performance. However, we found that even in the case of fully

---

A
B
D
RFE effereni RFE (wel offenent)
Partial ganglia control (basal ganglia pollcy)
Partial basal ganglia control (combined pollcy)
RFE + action surphse
Fully off-policy
On-policy
Open-field
Two-joint
Open-field
Two-joint
Open-field
Two-joint
Open-field
Two-joint
30
30
30
30
30
30
30
30
Expert
Cost
10k
10k
10k
10k
10k
10k
10k
10k
Episodes
Intermediate
30
30
30
30
features d(s.)
Cost
(mc)
10k
10k
10k
Artion surpron | - M(5=3])*
Reward predizton
Open-field
Two-joint
Random
30
30
30
30
30
Stonger action 30
Cost
Control
supprion to artor
10k
Epieodes
Eplaodes
Episodes
Episodes
Figure 3: A: Performance of the three models, in Fig. _C, shown for external controllers with varying levels of expertise (random, intermediate, expert; trained via backpropagation on the task for different numbers of steps). Middle row corresponds to Fig. 2 B: Same as A, but showing the performance of the combined policy of the basal ganglia network and external controller. C: Same as A, but in the case where all behavior is driven by the external controller during learning. D: Per- formance of the RPE and RPE+action surprise models for fully on-policy learning on the two tasks. E: Top: Schematic of model in which the contribution of the action surprise term to dopamine activity is weighted more strongly in the actor than in the critic (green arrow). Bottom: comparison of this model to the original action surprise model in the case of partial basal ganglia control and fully off- off-policy learning with the expert controller (black traces are the same as in the top row of panel E). In this example the action surprise term is weighted times as strongly in the actor than the critic.
on-policy learning, with no external controller, the action surprise model matched and indeed slightly outperformed the RPE-only model (Fig. BP).
We also tested the variant of the action surprise model in which an additional update (Eq. 13) is applied to the actor, corresponding to a higher action surprise coefficient in the actor (dorsal striatum) than in the critic (ventral striatum). We found that this addition improves learning in the presence of expert external controllers (Fig. DE), consistent with accelerated consolidation of the expert policy.
Biological implications of the action surprise model
The action surprise model explains several features of midbrain dopamine activity and also makes several testable predictions, which we outline below.
Nonspecific encoding of action. A central property of the model is that the action surprise term ||ay - -s(ss))' not action-specific-ii reflects only scalar information about the agent's action, even in high-dimensional action spaces, and does not distinguish between two equally surprising actions. This contrasts with the representation in striatal projection neurons forming the output of the actor network, which specify movement commands. Indeed, experimental recordings have found detailed encoding of kinematics in striatal projection neurons [6]] but only coarse movement-relate signals in dopamine activity that do not reliably distinguish between movement types 12 3S This distinction is consistent with our model and inconsistent with models that explain dopamine activity in terms of specific motor commands.
modulated
Decrease metion-modulated dopamine activity with learning. The action surprise model predicts that movement-related dopamine activity is lower when the basal ganglia's policy more closely matches the agent's 's actions. Before the basal ganglia has learned an effective policy, actions driven by an expert external controller will typically provoke large action surprise. Once the basal ganglia
7

---

A
B
C
Open-field navigation
0.8
2.5
12
0.7
0.0
action surprise
6
Aciion surprise
Acion surprise
Action surprise
Magnituda
0.5
Acceleration
2.5
5
2-
0.4
6-
0.0
0.3
2.5
4
0.2
0.0
0.1
5
2-
0.0
0.0
0-
10k
10
10
0.0
2.5
5.0
10.0
Episodes
Time
Time
Acceleration
Two-joint arm
0.8
2.5
12
0.7
10
Magnitude action surprise
0.6
Action surprise
Action surprise
Action surprise
0.0
Acceleration
0.5
2.5
5
0.4
0.0
0.3
2.5
4
0.2
0.0
2-
2
0.1
2.5
0.0
0.0
0-
0.
10k
10
0.0
2.5
5.0
7.5
10.0
Episodes
Time
Time
Acceleration
Figure 4: A: Mean and standard deviation of action surprise term in the dopamine response over the course of learning, in the case where the basal ganglia shares partial control with an expert controller. In all these examples = was set to 0.5 and and the exploration noise to 1.0. B: Traces of the action surprise signal (black) and the the agent's acceleration (red) or rotational acceleration averaged across joints in the two-joint arm case for five randomly sampled example trials. C: Mean and 95% confidence interval (across trials) of action surprise function of the time within a trial. D: Scatter plots (showing individual steps, across many episodes) of the action surprise term in the model dopamine response vs. magnitude of agent acceleration.
network has learned an expert policy itself, action surprise will typically be lower. Thus, we predict reduction in the magnitude of action surprise signals over the course of learning. We observed this phenomenon in both simulated tasks (Fig. EA. This prediction is supported by findings that dopaminee activity coinciding with lever-press movements decreases with repeated task practice 2
Context-dependent correlation between dopamine activity and acceleration. Several studies have shown that striatal dopamine activity correlates with movement speed and acceleration 60
40 4
have shown that activity in moovement-responsi midbrain dopamine neurons is modulated primarily movement at initiation (and in some cases at movement offsett but less so during ongoing movements. Our model gives rise to a complex relationship between movement and action- driven dopamine signals, with the two appearing correlated on some trials but not others (Fig. 4B) On average, action-driven dopamine activity is highest at trial onset (Fig. BC), and it is positively correlated with the agent's 's acceleration (Fig. FD). Our results suggest that the representations of movement initiation and kinematic information attributed to dopamine activity in prior work may actually emerge as byproducts of an action surprise computation.
Others 27 BS
Neural circuits underlying the computation of action surprise. The action surprise model requires information about the agent's action be available to midbrain dopamine neurons for computation of the action surprise term. This is consistent with the presence of pathways from motor cortex and cerebellum midbrain dopamine neurons 12 39
It also consistent with perturbation experiments
finding a causal effect of cortical perturbations and lesions on striatal dopamine activity
B3 54
We note that the specific form of the action surprise term in our model, ||a {=(s) is somewhat arbitrary, and other measures of distance between and ps() may equally suitable. We also note that our model is agnostic as to whether the action surprise and RPE components of dopamine activity
8

---

are represented in the same individual neurons. Empirically, RPE-signalling and movement signaalig dopamine neurons appear to comprise distinct but partially overlapping populations in the same striatal subregions [[2], though data on this question is limited.
Considerations for future experimental work. We intend this work to motivate experimental work that can support, challenge, or refine our model. Doing so requires careful experimental design. First, the effects of action-related dopamine activity on learning are easily overlooked in the context of simple tasks in which animals learn to associate cues or actions with immediate reward. In such tasks, standard RPE-based models may be adequate to achieve good performance. Second, disentangling "action surprise" as we define it from other related quantities (imitiation, acceleration, motivation, etc.) requires a rich action space, while many animal RL studies involve a small, discrete set of actions (e.g. binary choice). Hence, motor tasks that require sequences of movements involving continuous control
are most appropriate for our purposes. Testing predictions of the model also involves disentangling the behavioral contributions of the hasal ganglia and other brain regions and analyzing these in concert with simultaneously recorded dopamine activity and behavior. While the technology exists to conduct such simultaneous recordings, the model would also benefit from the ability to causally modulate the surprisingness of actions, for instance using catch trials with different target movements.
such experiment has been performed. Tests of
7 Discussion
Our work demonstrates a new link between action-related midbrain dopamine activity and reinforce- ment learning in the basal ganglia. While each of these topics has received extensive treatment in the neuroscience literature, they have typically been studied separately. Midbrain dopamine activity is known to causally affect movement initiation and invigoration in real time 12
47
Consequently,
the function of such activity is often regarded as a motivational signal 655 or as a contributor to action selection 117. separate from dopamine's ' role in reinforcement learning. However, action and RPE-related dopamine release coincide in the same striatal subregions, often even in the same Thus, it is likely that both action-related and RPE-related dopamine activity impact cortico-striatal synaptic plasticity and learning. Excitingly, concurrent experimental work provides direct evidence for an action surprise-like signal in dopamine neurons projecting to the tail of the striatum, and for a causal influence of these neurons on task learning [22] Our results show that such an influence is a necessary component of architecture capable of off-policy learning.
2
neurons, and with comparable magnitude (60 49 12 38
I8
We note that our proposal does not preclude alternative roles for dopamine activity. Indeed, action- related dopamine signals are observed to both precede and follow movement [12]. Early responses may participate in motivation and action selection while lingering responses responses (extending into the critical plasticity window for cortico-striatal synapses 666 may aid in learning. Whether similar signals can achieve both functions is important direction for future modeling work. Prior work [17 has suggested that dopaminergic action selection signals may correspond to imagined RPEs of simulated action outcomes according to a learned model of the environment, which may correlate strongly with action surprise if actions are partially controlled by a model-based system.
Our model suggests that distributed control of behavior by many regions plays an important role in learning in the basal ganglia. number of experiments have observed differential contributions from multiple brain regions during learning. For instance, some motor skills are observed to recruit the motor cortex early in learning before being consolidated into the basal ganglia B1 28 These results suggest that regions other than the basal ganglia may be more adept at flexibly adapting to novel tasks, while the basal ganglia specializes in consolidating well-practiced skills. Other studies have observed parallel contributions of model-based and model-free reinforcement learning strategies in the same task [53 II revealing arbitration mechanisms leverage the ad advantages each B4 [2). The flexibility afforded by off-policy RL algorithms enables the basal ganglia to benefit from complementary learning and control strategies adopted by other neural circuits. Exploring how off off-policy RL algorithms can best leverage these diverse sources of expertise is fruitful avenue for extensions to our model. For instance, action surprise signals may be modulated by the confidence of external controllers in order more efficiently learn from expert behavior.
Our work also has implications for reinforcement learning- The introduction of a biologically plausible off-policy reinforcement learning algorithm, involving local learning rules and a single
9

---

modulatory factor, enables the deployment of off oolicc RL on plasticity- enabled neuromorphic hardware 14. Moreover, we anticipate that insights into how neural circuits learn from off- policy behavior governed by distributed and diverse controllers will provide inspiration for RI. algorithms.
Acknowledgments and Disclosure of Funding
This work supported by NSF NeuroNex Award DBI-1707398 and The Gatsby Foundation (GAT3708). ALK was also supported by the McKnight, Burroughs-Wellcome Wellcome, and Mathers Founda- tions. JL was also supported by the DOE CSGF (DE DCOSC002)3477. The authors declare no competing interests.
References
[1] Martín F Adrover, Jung Hoon Shin, Cesar Quiroz, Sergi Ferré, Julia Lemos, and Veronica Alvarez. Prefrontal cortex-driven dopamine signals in the striatum show unique spatial and pharmacological properties. Journal of Neuroscience, 40(39):510--552, 2020.
[2] Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6):26-38, 2017.
[3] Gregory Ashby, Benjamin Turner, and Jon Horvitz. Cortical and basal ganglia con- tributions to habit learning and automaticity. Trends in cognitive sciences, 14(5):208-215, 2010.
[4] Joseph W Barter, Suellen Li, Dongye Lu, Ryan A Bartholomew, Mark A Rossi, Charles Shoemaker, Daniel Salas-Meza, Erin Gaidis, and Henry H Yin. Beyond reward prediction errors: the role of dopamine movement kinematics. Frontiers in integrative neuroscience, 9:39, 2015.
[5] Andreea C Bostan, Richard P Dum, and Peter L Strick. The basal ganglia communicate with the cerebellum. Proceedings of the national academy of sciences, 107((8):8452 8456, 2010.
[6] Andreea C Bostan and Peter L Strick. The basal ganglia and the cerebellum: nodes in integrated network. Nature Reviews Neuroscience, 19(6):338-333 -300,, 2018.
[7] Michael Boyeson and Dennis M Feeney. Striatal dopamine after cortical injury. Experimental neurology, 89(2):479-483, 1985.
[8] Joshua Brown, Daniel Bullock, and Stephen Grossberg. How the basal ganglia use parallel excitatory and inhibitory learning pathways selectively respond to unexpected rewarding cues. Journal of Neuroscience, 19(23);:0002-005 1999.
[9] Paolo Calabresi, Paolo Gubellini, Diego Centonze, Barbara Picconi, Giorgio Bernardi, Karima Chergui, Per Svenningsson, Allen A Fienberg, and Paul Greengard. Dopamine and camp- regulated phosphoprotein 32 kda controls both striatal long-term depression and long-term potentiation, opposing forms of synaptic plasticity. Journal of Neuroscience, 20(22):4433 8453-4411 2000.
[10] Rudolf N Cardinal, John A Parkinson, Jeremy Hall, and Barry Everitt. Emotion and motivation: the role of the amygdala, ventral striatum, and prefrontal cortex. Neuroscience & Biobehavioral Reviews, 26(3):321-355 2002.
[11] José L Contreras- Vidal and Wolfram Schultz. A predictive reinforcement model of dopamine neurons for learning approach behavior. Journal of computational neuroscience, 6(3):191-214, 1999.
[12] Joaquim Alves Da Silva, Fatuel Tecuapetla, Vitor Paixão, and Rui M Costa. Dopamine neuron activity before action initiation gates and invigorates future movements. Nature, 554((691)):44 248, 2018.
[13] Will Dabney, Zeb Kurth-Nelson, Naoshige Uchida, Clara Kwon Starkweather, Demis Hassabis, Rémi Munos, and Matthew Botvinick. A distributional code for value dopamine-based reinforcement learning. Nature, 577 2020.
10

---

[14] Mike Davies, Narayan Srinivasa, Tsung-Han Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam. Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. leee Micro, 38(1))82-99, 2018.
[15] Nathaniel D Daw, Samuel J Gershman, Ben Seymour, Peter Dayan, and Raymond Dolan. Model-based influences on humans' choices and striatal prediction errors. Neuron, (9(6):1204 1215, 2011.
[16] Ashesh K Dhawale, Steffen BE Wolff, Raymond Ko, and Bence P Ölveczky. The basal ganglia control the detailed kinematics of learned motor skills. Nature neuroscience, 2021.
[17] Kenji Doya. What are the computations of the cerebellum, the basal ganglia and the cerebral cortex? Neural networks, 12(7-8):911977, 1999.
[18] Ben Engelhard, Joel Finkelstein, Julia Cox, Weston Fleming, Hee Jae Jang, Sharon Oreelas, Sue Ann Koay, Stephan Y Thiberge, Nathaniel D Daw, David W Tank, et al. Specialized coding of sensory, motor and cognitive variables in vta dopamine neurons. Nature, 570(7762):509-551, 2019.
[19] Cornelia Exner, Janka Koschack, and Eva Irle. The differential role of premotor frontal cortex and basal ganglia in motor sequence learning: evidence from focal basal ganglia lesions. Learning & Memory, 96):376-386, 2002.
[20] Nicolas Frémaux and Wulfram Gerstner. Neuromodulated spike-timing-dependeee plasticity, and theory of three-factor learning rules. Frontiers in neural circuits, 9:85, 2016.
[21] Jan Gläscher, Nathaniel Daw, Peter Dayan, and John P O' O'Doherty. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron, 66(4):585-595, 2010.
[22] Francesca Greenstreet, Hernando Martinez Vergara, Sthitapranjya Pati, Laura Schwarz, Mathew Wisdom, Fred Marbach, Yvonne Johansson, Lars Rollik, Theodore Moskovitz, Claudia M Clopath, et al. Action prediction error: value-free dopaminergic teaching signal that drives stable learning. bioRxiv, 2022.
[23] Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. In International conference on machine learning, pages 2829- 2838. PMLR, 2016.
[24] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018
[25] Suzanne N Haber. Corticostriatal circuitry. Dialogues in clinical neuroscience, 2022.
[26] James Houk and James Adams. 13 a model of how the basal ganglia generate and use neural signals that. Models of information processing in the basal ganglia, page 249, 1995.
[27] Mark W Howe and Daniel A Dombeck. Rapid signalling in distinct dopaminergic axons during locomotion and reward. Nature, 535(7613):505-510, 2016
[28] Eun Jung Hwang, Jeffrey E Dahlen, Yvonne Yuling Hu, Karina Aguilar, Bin Yu, Madan Mukundan, Akinori Mitani, and Takaki Komiyama. Disengagement motor cortex from movement control during long-term learning. Science advances, 5(10):eaa00001, 2019.
[29] Makoto Ito and Kenji Doya. Multiple representations and algorithms for reinforcement learning in the cortico-base ganglia circuit. Current opinion in neurobiology, 21(3):368-33 3703,33333733 201
[30] Daphna Joel, Yacl Niv, and Eytan Ruppin. Actor-critic models of the basal ganglia: New anatomical and computational perspectives. Neural networks, 15(4-6):535-54) 2002.
[31] Risa Kawai, Timothy Markman, Rajesh Poddar, Raymond Ko, Antoniu I. Fantana, Ashesh K Dhawale, Adam Kampff, and Bence Ölveezky. Motor cortex required for learning but not for executing motor skill. Neuron, 86(3):800-812, 2015.

---

[32] Dongjae Kim, Geon Yeong Park, Sang Wan Lee, et al. Task complexity interacts with state- space uncertainty in the arbitration between model-based and model-free learning. Nature communications, 10(1):1-14, 2019.
[33] Polina Kosillo, Yan-Feng Zhang, Sarah Threlfell, and Stephanie J Cragg. Cortical control of stri- atal dopamine transmission via striatal cholinergic interneurons. Cerebral cortex, 26(1 1):4160- 4169, 2016.
[34] Sang Wan Lee, Shinsuke Shimojo, and John P P'DDoherty. Neural computations underlying arbitration between model-based and model-free learning. Neuron, 81(3):687-699 2014.
[35] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: Ieci::150000 (15.509999771 2015.
[36] Nikolaus R McFarland and Suzanne N Haber. Convergent inputs from thalamic motor nuclei and frontal cortical areas to the dorsal striatum in the primate. Journal of Neuroxcience, 20(10):378833813 2000.
[37] Sarah Melzer, Mariana Gil, David E Koser, Magdalena Michael, Kee Wui Huang, and Hannah Monyer. Distinct corticostriatal gabaergic neurons modulate striatal output neurons and motor activity. Cell reports, 19(5):1045-1055 2017.
[38] Marcelo D Mendonça, Joaquim Alves da Silva, Ledia F Hernandez, Ivan Castela, José Obeso, and Rui M Costa. Transient dopamine neuron activity precedes and encodes the vigor of contralateral movements. bioRxiv, 2021.
[39] William Menegas, Joseph F Bergan, Sachie K Ogawa, Yoh Isogai, Kannan Umadevi Venkataraju, Pavel Osten, Naoshige Uchida, and Mitsuko Watabe-Uchida. Dopamine neurons projecting to the posterior striatum form anatomically distinct subclass. elife, 4:e10032, 2015.
[40] Nicolas Meuleau, Leonid Peshkin, and Kee-Eung Kim. Exploration in gradient-based reinforce- ment learning. 2001.
[41] Kevin GC Mizes, Jack Lindsey, Sean Escola, and Bence P Ölveczky. Similar striatal activity exerts different control over automatic and flexible motor sequences. bioRxiv, 2022.
[42] Ali Mohebi, Jeffrey R Pettibone, Arif A Hamid, Jenny-Marie T Wong. Leah T Vinson, Tommaso Patriarchi, Lin Tian, Robert Kennedy, and Joshua D Berke. Dissociable dopamine dynamics for learning and motivation. Nature, 570(7759):65-70, 2019.
[43] P Read Montague, Peter Dayan, and Terrence J Sejnowski. A framework for mesencephalic dopamine systems based on predictive hebbian learning. Journal of neuroscience, 16(5):1936- 1947, 1996.
[44] Yael Niv. Reinforcement learning in the brain. Journal of Mathematical Psychology, 53(3):139- 154, 2009.
[45] John O'Doherty, Peter Dayan, Johannes Schultz, Ralf Deichmann, Karl Friston, and Raymond J Dolan. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. science, 45445669)4452-444, 2004.
[46] Mark G Packard and Barbara J Knowlton. Learning and memory functions of the basal ganglia. Annual review of neuroscience, (5(11:5633550 2002.
[47] Babita Panigrahi, Kathleen Martin, Yi Li, Austin R Graves, Alison Vollmer, Lars Olson, Brett D Mensh, Alla Y Karpova, and Joshua T Dudman. Dopamine is required for the neural representation and control of movement vigor. Cell, 162(6):1418-14)) 1430,, 2015.
[48] Doina Precup. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series, page 80, 2000.
[49] Corey B Puryear, Min Jung Kim, and Sheri JY Mizumori. Conjunctive encoding of movement and reward by ventral tegmental area neurons in the freely navigating rodent. Behavioral neuroscience, 124(2):234, 2010.
12

---

[50] Wolfram Schultz, Peter Dayan, and P Read Montague, A neural substrate of prediction and reward. Science, 275(5306):1593-1599 1997.
[51] Weixing Shen, Mare Flajolet, Paul Greengard, and D James Surmeier. Dichotomous dopaminer- gic control of striatal synaptic plasticity. Science, 321(589)):84 001-851, 2008.
[52] Maria Caterina Silveri. Contribution of the cerebellum and the basal ganglia to language production: Speech, word fluency, and sentence innstruction-evidenee evidence from pathology. The Cerehellum, 20(2):282-22 294, 2021.
[53] Peter Smittenaar, Thomas HB FitzGerald, Vincenzo Romei, Nicholas Wright, and Raymond Dolan. Disruption of dorsolateral prefrontal cortex decreases model- in favor of model-free control in humans. Neuron, (14):994-919, 2013.
[54] Antonio P Strafella, Tomás Paus, Maria Fraraccio, and Alain Dagher. Striatal dopamine release induced by repetitive transcranial magnetic stimulation of the human motor cortex. Brain, (16(12):209-2613 2003.
[55] Roland E Suri, J Bargas, and MA Arbib. Modeling functions of striatal dopamine modulation in learning and planning. Neuroscience, 103(1):65-85, 2001.
(56) Roland E Suri and Wolfram Schultz. Learning of sequential movements by neural network model with dopamine-like reinforcement signal. Experimental brain research, 121(3):50--354, 1998.
[57] Roland E Suri and Wolfram Schultz. A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task. Neuroscience, 91(3):871-890, 1999.
[58] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[59] Eiji Uchibe. Cooperative and competitive reinforcement and imitation learning for mixture of heterogeneous learning modules. Frontiers in neurorobotics, 12:61, 2018.
[60] Dong V Wang and Joe Z Tsien. Conjunctive processing of locomotor signals by the ventral tegmental area neuronal population. PloS one, 6(1):e16528, 2011.
[61] Jane X Wang, Zeb Kurth-Nelson, Dharshan Kumaran, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Demis Hassabis, and Matthew Botvinick. Prefrontal cortex as a meta-reinforcement learning system. Nature nearoscience, 2166))860-868, 2018.
[62] Mitsuko Watabe-Uchida. Lisa Zhu, Sachie K Ogawa, Archana Vamanrao, and Naoshige Uchida. Whole-brain mapping of direct inputs to midbrain dopamine neurons. Neuron, 74(5):858-873, 2012.
[63] JR Wickens, AJ Begg, and GW Arbuthnott. Dopamine reverses the depression of rat cor- ticostriatal synapses which normally follows high-firquency stimulation of cortex in vitro. Neuroscience, 70(1):1-5, 1996.
[64] Dirk Wildgruber, Hermann Ackermann, and Wolfgang Grodd. Differential contributions of motor cortex, basal ganglia, and cerebellum to speech motor control: effects of syllable repetition rate evaluated by fmri. Neuroimage, 13(1):101-109, 2001.
[65] Roy A Wise. Dopamine, learning and motivation. Nature reviews neuroscience. 5(6):483-494 2004.
[66] Sho Yagishita, Akiko Hayashi- -Tkkai, Graham CR Ellis- Hidetoshi Urakubo, Shin Ishii, and Haruo Kasai. critical time window for dopamine actions on the structural plasticity of dendritic spines. Science, 1666660)):11661666 2014.
[67] Alexandre Zénon, Sophie Devesse, and Etienne Olivier. Dopamine manipulation affects response vigor independently of opportunity cost. Journal of Neuroscience, 36(37):9516-955 2016.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No] Our work is not primarily intended advance the state of the art RL any particular engineering application, but rather to provide basic scientific insights about computation in the brain. As such, we regard the potential for negative societal impacts of this work to be limited.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Section 3.1, in particular Equations (6) and (8)
(b) Did you include complete proofs of all theoretical results? [Yes] Section 3.1
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Appendix A.5.
(c) Did report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix A.S.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material URL? [Yes] Code will be provided as a URL upon publication.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're
using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "XpTNOtH1x9E", "reviewer": "Reviewer_YRYS", "summary": "The paper proposes a new learning rule based on action surprise, which is meant to facilitate off policy reinforcement learning and explain a variety of biological results from DA neuron and basal ganglia recordings, some of which (such as DA coding of novelty and other action related codes) were not thought to be directly linked to reinforcement learning. The authors provide some empirical results comparing the proposed model to other alternatives and show relevant correlates to some biological results, but not direct comparison to behaviour.", "questions": "To improve the paper, the authors could do the following:\n- provide more examples about the representation of mu(s_t) and how could it be investigated in neural recordings for example (i.e. is there any specific rather than vague/indirect evidence for such representation)\n- address the issue of biological/behavioural relevance of tasks and particularly training methods, as they seem more common in machine learning RL, not behavioural RL (although open field tasks are definitely used widely, but not sure if in exactly the same way), paying particularly care for biological realism of timelines, training protocols and learning rules (e.g. is backprop biologically realistic?..)\n- perform direct comparison with behavioural learning data or at the very least discuss how exactly it could be done, or if not, why not.\n- perform more meaningful comparisons with other models, if authors feel that such comparisons are needed for the paper (it seems in many cases those other models simply don't work, with only general theoretical explanation provided why not).\nAs a general comment, I'm quite skeptical of the widespread practice of comparing different models in modelling papers, especially when it's used not as a means to select between several alternatives equally considered by authors, but showcase of the proposed model's superior performance to other existing models, as implementation details of such alternative models may not be chosen with the best intention to make them perform as well as possible, and various creative choices (such as best formalisation of states, actions, rewards) and/or fine tuning may be needed for that purpose, so I think absolute assessment and direct comparison with behavioural or physiological data as opposed to other models are much more useful.\n\nMinor issue: why is there a green line in fig. 3E bottom?", "limitations": "There was some discussion of limitations, but not really the main limitations/weaknesses of work as I see them (detailed above).", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think the proposed idea is indeed very interesting and has great potential to link a variety of seemingly disparate biological results regarding DA and basal ganglia physiology. I also think the authors perform a great literature review and brilliant integration with relevant findings in literature. The clarity of the proposed model is sufficient, although it's a bit abstract at times, e.g. it would be great to see how the mu(s_t) and a_t - mu(s_t) terms would look in practice, as that's the key element of the model and perhaps not sufficiently explained with examples.\n\nMy main problem with the paper is that although it seems to clearly target computational cognitive neuroscience (as opposed to machine learning) audience, the type of tasks, training protocols and measurements seem to correspond a lot more to \"machine learning\" RL, as opposed to what is or could be done in behavioural experiments, whether animal or human. The type of (distance/error dependent) cost function that is presumably used has little in common with that is used in reinforcement learning, where rewards are often scarce and don't provide constant feedback. Training protocols requiring 100k episodes (or anything of that order) are also totally unrealistic in biology, at least if you interpret this as behavioural trials done in learning (and even if you try to relate it to replay of sequences and similar data, 100k is still quite a stretch). There is no behavioural data provided for these 2 tasks that could suggest based on what timeline or mechanisms animals actually learn in these cases, and comparisons with other models (particularly examples, like in Fig.2b given without much explanation of why this happens) are not very informative. While I don't doubt that authors' model can perform well as asserted and their idea (if not necessarily the implementation) may be a great way to reconcile some disparately looking biological data, I think the effort to link model performance to actual experiments is insufficient. It's true that there is not a great deal of data of direct relevance to off-policy RL in behavioural neuroscience, but I doubt that nothing could be found which could be compared more directly.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EIv9DIXG8d2", "reviewer": "Reviewer_QgmR", "summary": "The authors explains the role of dopamine response at action initiation based on an off-policy actor critic architecture.\n\n\n", "questions": "Dopamine neuron's cue response is quite fast, often preceding action execution. The present framework assumes the use of efference copy of finally selected action somewhere in the brain. Does the timing really match up?\n\nHere is a conceptual question. Action value equation (8) assumes any action that deviates from mu(s) are worse. On the other hand, the advantage equation (12) favors action that deviates from mu(s). Where does this apparent inconsistency come from? Maybe it is a matter of whether you apply (8) to V(s(t)) or V(s(t+1)).\n\nUchibe proposed an off-policy RL scheme with mixture of controllers, as in the present setting, but uses important sampling, which gives more weights on actions similar to one's own; apparently opposite to what is done by action surprise. Could you comment on the relationship or advantages/weakness?\nUchibe E (2018). Cooperative and Competitive Reinforcement and Imitation Learning for a Mixture of Heterogeneous Learning Modules. Front Neurorobot, 12, 61. https://doi.org/10.3389/fnbot.2018.00061\n\n", "limitations": "The approximation of action value by (8) with fixed sigma would limit the model's applicability.\n\nI updated my score after reading the paper [22].\n\n", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper makes an important point that the brain's action selection is distributed, so that learning should be off-policy.\n\nThe dopamine response at action initiation promotes imitation learning of external controller, which works nice if the external controller is doing a fair job, but can screw up the basal ganglia learning if others are doing badly.\n\nDopamine response at action initiation had been predicted upon finding an high advantage action in a model-based framework (Doya 1999, Figure 7). This should also be discussed.\nDoya K (1999). What are the computations of the cerebellum, the basal ganglia, and the cerebral cortex. Neural Networks, 12, 961-974. https://doi.org/10.1016/S0893-6080(99)00046-5\n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5HaVTkl_9Ij", "reviewer": "Reviewer_15y6", "summary": "the currently dominant view of RL in the basal ganglia. The authors then show how Q-learning with a particular state-action value function parametrization yields an RL algorithm that is very close to the actor-critic model while being able to learn off-policy. Importantly, this new model has an \"action-surprise\" term that also allows the model to explain the existence of movement-related modulation in the striatum. The authors show that the model can produce performance advantages. Finally, they discuss how the model might be implemented biologically.", "questions": "See comments above.", "limitations": "The authors don't really address limitations of their approach. I don't see any issues with negative societal impact.", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I really liked this paper. It presented a novel theoretical idea that jointly solves both empirical and computational problems. It was clearly written and compellingly argued.\n\nMy only major comment is that I wish the authors had simulated actual neuroscience experiments rather than (or in addition to) their toy setups (the two-joint arm and open field). This feels like a missed opportunity to show how the model can directly explain the relevant empirical data.\n\nRelated to the last point, there are experiments that measure dopamine when other action-related areas are manipulated (e.g., inactivated, stimulated, lesioned). It seems like this model should make predictions about these experiments. Similarly, if an animal's actions are influenced in unexpected ways, then the prediction is that this should register in the dopamine signal (though one would have to be careful to control for sensory effects); I'm not aware of studies that look specifically at this question, but it seems like a clear prediction.\n\nMinor comments:\n\n- p. 1: \"decreasing amounts movement-modulated dopamine activity\" -> \"decreasing amounts of movement-modulated dopamine activity\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept