Neural networks with motivation
Authors: Sergey A. Shuvaev
Ngoc B. Tran
Marcus Stephenson-Jones
Bo Li
Alexei A. Koulakov
Venue:  (2020)
Keywords: neuroscience
brain
motivation
learning
reinforcement learning
recurrent neural network
deep learning

Abstract: How can animals behave effectively in conditions involving different motivational contexts? Here, we propose how reinforcement learning neural networks can learn optimal behavior for dynamically changing motivational salience vectors. First, we show that Q-learning neural networks with motivation can navigate in environment with dynamic rewards. Second, we show that such networks can learn complex behaviors simultaneously directed towards several goals distributed in an environment. Finally, we show that in Pavlovian conditioning task, the responses of the neurons in our model resemble the firing patterns of neurons in the ventral pallidum (VP), a basal ganglia structure involved in motivated behaviors. We show that, similarly to real neurons, recurrent networks with motivation are composed of two oppositely-tuned classes of neurons, responding to positive and negative rewards. Our model generates predictions for the VP connectivity. We conclude that networks with motivation can rapidly adapt their behavior to varying conditions without changes in synaptic strength when expected reward is modulated by motivation. Such networks may also provide a mechanism for how hierarchical reinforcement learning is implemented in the brain.

Under review as a conference paper at ICLR 2020
NEURAL NETWORKS WITH MOTIVATION
Anonymous authors
Paper under double-blind review
ABSTRACT
How can animals behave effectively in conditions involving different motivational contexts? Here, we propose how reinforcement learning neural networks can learn optimal behavior for dynamically changing motivational salience vectors. First, we show that Q-learning neural networks with motivation can navigate in envi- ronment with dynamic rewards. Second, we show that such networks can learn complex behaviors simultaneously directed towards several goals distributed in environment. Finally, we show that in Pavlovian conditioning task, the responses of the neurons in our model resemble the firing patterns of neurons in the ventral pallidum (VP), basal ganglia structure involved in motivated behaviors. We show that, similarly real neurons, recurrent networks with motívation are composed of two oppositely-tunee classes of neurons, responding to positive and negative rewards. Our model generates predictions for the VP connectivity. We conclude that networks with motivation can rapidly adapt their behavior to varying condi- tions without changes in synaptic strength when expected reward modulated by motivation. Such networks may also provide a mechanism for how hierarchical reinforcement learning is implemented in the brain.
INTRODUCTION
Motivation is a cognitive process that propels individual's behavior towards away from particular object. perceived event, or outcome (Zhang et al., 2009). Mathematically, motivation can be viewed as subjective modulation of the perceived reward value before the reward received. Therefore, it reflects an organism's 's wanting of the reward before the outcome is actually achieved.
Computational models for motivated behavior, which are best represented by reinforcement learning (RL) models, are mostly concerned with the learning aspect of behavior. However, fluctuations in physiological states, such as confidence and motivation, can also profoundly affect behavior (Zhang et al. 2009). Modeling such factors is thus an important goal in computational neuroscience and is in the early stages of mathematical description (Berridge 2012).
Here build neural network theory for motivational modulation of behavior based on Q-learning and apply this theory to mice performing Pavlovian conditioning task in which experimental obser- vations of neural responses obtained in the ventral pallidum (VP) available. We show that motivated RL model both learns to correctly predict dnnivatioo-dependenn rewards in the Pavlovian conditioning task and is consistent with responses of neurons in the VP. In particular, show that, similarly to the VP neurons, Q-learning neural networks contain two oppositely- of neurons responsive to positive and negative rewards. In the model, these two populations form a push-pull network that helps maintain motivation motivation-dependenn variables when inputs are missing. Our RL-based model both consistent with experimental data and predicts the structure of the VP networks. We thus argue that motivation leads to complex behaviors which may add an extra level of complexity machine learning approaches and consistent with biological data.
populations
RESULTS
Motivation is defined mathematically as a need- deeeddeendddnt modulation of the perceived reward value depending animal's extrinsic intrinsic conditions (Zhang et al.) 2009). Thus, rats, which normally repelled by high levels of salt in their food, may become attracted to a salt-containing solution following salt-free diet (Berridge 2012). To model this observation, |Brridge & Schulkin

---

Under review as a conference paper at ICLR 2020
1989) have proposed that the perceived reward r; received at time t is not absolute, but modulated by an internal variable reflecting the level of motivation, which we will call here μ.. The perceived level of the reward Ft as function of motivation μ can be expressed by the following equation:
F. = =((r,, (4)
(1)
In the simplest example, the reward, associated with salt is given by n = pr.- Baseline motiva-
tion towards salt can be defined by μ 1, leading to the perceived reward of ig = -rt < 0. Thus, normally the presence of salt in the diet is undesired. In the salt-free condition, the motivation changes to f +1, leading to the subjective reward of F; +rx 2 O. Thus salt-containing diet becomes attractive. In reality, the function F(...) defining the impact of motivation on a perceived reward is complex (Zhang al. 2009). including the dependence on multiple factors described by a motivation vector μ.. Individual components of this vector describe various needs experienced by the organism, such thirst (e.g- 11), appetite (122), etc. In this study, we explore the computa- tional impact of motivation vector in the context of RL and investigate the brain circuits that might implement these computations.
Our approach to motivation is based on Q-learning (Watkins & Dayan 1992). which relies on an agent estimating Q-function, defined as the sum of future rewards given an action a, chosen in state 3; at time point t: ((3 a1) = Er= r(5 (g) ) (here and below, we omit averaging for simplicity). Here < y S
the discounting factor that keeps the sum from diverging, and balances preference of short- versus long-term rewards. If a correct Q-function is known, a rational agent picks an action that maximizes future rewards: Ot t- argmara (55 a). In case of motivation as reward values are affected by the motivation vector Jī, for the Q-function, we obtain:
in equation [
( at 11) = Σ F( (at)""
Here F(B Me4rr/ (as) is the motivation j7-dependent perceived reward obtained in a state Feur reached at time L + T given action O chosen at time L.
The state of the agent 5, and its motivation jī distinct. The motivation is slowly changing variable, that on average is not affected substantially by single action. For example, the animal's s appetite does not change substantially during a single trial. At the same time, the actions selected by the animal lead to immediate changes of the animal's 's state &t. Recent research in neuroscience suggests that motivation and state may be represented and computed separately in the mammalian brain. Whereas motivation is usually attributed to the regions of the reward system, such as the VP (Berridge& & Schulkin 1989; Berridge 2012). the state is likely to be computed elsewhere, e.g. in the hippocampus (Eichenbaum et al. 1999), or cortex. In RL, an agent's state and motivation may have different mathematical representations. In the examples below, the state variable is given by a one-hot vector, while motivation is represented by full vector. Two arguments of the Q-function, 5, and J7, are therefore distinet. Finally, in hierarchical RI. implementation, motivation is provided by higher level network, while information about the state is generated externally.
Although the Q-function with motivation (equation 2 is similar to the Q-function in goal- conditioned RL (Schaul al. 2015; Andrychowizz et 2017), the underlying learning dynamics is different. Motivated behavior pursues multiple distributed sources of dynamic rewards. The Q- function therefore accounts for the future motivation dynamics. This way, an agent with motivation chooses what reward to pursue- - making it also different from RL with subgoals (Sutton et al., 1999). Behavior with motivation therefore involves minimum to no handcrafted features, which suggests that motivation could provide a step towards general methods that leverage computation a goal identified by Richard [Sutton| (019).
As in the case of standard Q-leariing the action chosen by a rational agent maximizes the sum of the expected future perceived rewards, i.e. dt + Q a, mT) To learn a correct Q-function, one can use the Temporal Difference (TD) method (Sutton & Barto (1998), ). If the Q-function is learned perfectly, it satisfies the recursive relationship ( a, μt) = f( 17)) + y max ((5++1 0e+1. 1)) For incompletely learned notivation-dependeer Q-function, the TD error 8 is non-zero:
8 7 F(5,, (1)) + 7 max Q( 01,1, ( Q(5e, Ge; pe)
act1
TD error can be used to update motivation-deepnderr functioo directly or to train neural networks to optimize their policy. Q-function depends on the new set of variables j7 that evolve following
2

---

Under review as a conference paper at ICLR 2020
their rules. These variables reflect fluctuations in physiological psychological states that substantially change the reward function and, therefore, can generate flexible behaviors dependent on animals' ongoing needs. We trained neural networks via backpropagation of the TD error (equa- tion B). an approach employed in deep Q-learning (Mnih et al. 2015). Below we present several examples in which neural networks could be trained solve motivation-dependent tasks.
2.1 THE FOUR DEMANDS TASK
Consider the example in Figure I An agent navigates in 6x6 square gridworld separated into four 3x3 subdivisions (rooms) (Figure [T]). The environment was inspired by the work of [Sutton et al. (1999): however, the task is different, as described below. In each room, the agent recerves one and only one type of reward rn (3t, ye). where n 1...4 (Figure 3). These rewards can be viewed as four different resources, such water, food, sleep, and work. Motivation is described in this system by a 4D vector zī defining affinity of the agent for each of these resources. When the agent enters room number n, the corresponding resource in the room is consumed, the agent receives rewards defined by Ft = Pa. and the corresponding component of the motivation vector μ4, is reset to (Figure IP. On the next time step, motivations in all four rooms are increased by one, i.c. Jhn + An 1, which reflects additional "wanting" of the resource induced by the "growing appetite". After a prolonged period of building up appetite, the motivation towards resource saturates fixed maximum value of 0, which becomes parameter of this model, determining the behavior.
C
motivation
environment
reward
A
14 water
M4 food
(x,
xxx,y)
A
/4
A
one-room binge
E two-room binge
migration
sleep /4
work H4
9
7(x,y)
4(x,))
(,6,,)
= 0,1,O,
1=1,1,
>= 9,0,1,9,0...
F=0.5
=110/333.33.
Figure 1: The Four Demands example. (A) An agent populates a environment separated into four rooms. Each room is associated with its own reward and motivation (water, food, sleep, work). (B) The perceived reward is scalar product between the motivation vector and the reward vector as illustrated. (C) Components of the 4D motivation vector functions of time. When agent When the agent not in room, motivation increases by at each time step until saturation (D-F) Potential strategies in our model: (D) one-room binge, (E) two-room binge, and (F) migration.
room, motivation is reset zero.
What are the potential behaviors of the agent? Assume, that the maximum allowed motivation is large, and does not influence results. If the agent always stays in the same room (one-room binge strategy, Figure DD), the rewards received by the agent consist of a sequence of zeros and ones, i.e. 0, 1, 0, 1, ... (in model, after the motivation is set to zero, it is increased by time step). The average reward corresponding to this strategy is therefore Fone-room binge 1/2. The average reward two-room binge strategy, Figure [). In this case, the sequence of rewards received by the agent described by the sequence of ones and the average reward Ftwo-room bénge 1. Two-room binging therefore outperforms the one-room binge strategy. Finally, the agent can migrate by moving cycle through all four rooms (Figure [ In this case, the agent spends three steps in each room and the overall period of migration is 12 steps. During these three steps, the agent receives the 10/3). Thus, migration strategy is more beneficial for the agent than both of the binging strategies. Migration, however, is affected by the maximum allowed motivation value 6. When < 9, the benefits of migration strategy
the next
be increased, if the agent jumps from room room each time step (a
rewards of 9 (the agent left this room nine steps ago), then O, and (Fmigration
reduced. For A 1, for example, migration yields the reward rate of just = 2/3, which is below the return of the two-room binging. Thus, our model should
Emigrationn/ p= |0=1 display various behaviors depending on 0.
3

---

Under review as a conference paper at ICLR 2020
A
40 inputs
100
100
100
outputs
bingg
puemau вбеeлле
Balinn Woo0-oo
A deeaa www.oooo
motivated
((stay)
non-motivated random walk
((up)
~V
0
(2down)
sfwater)
Qtleft)
uilood) plskoop)
(righh)
p(work)
10
100
maximum allowed motivation value,
Figure 2: (A) The architecture of the 3-layer fully connected network computing the Q-function Q 8. μi) (B) The average reward per time step received by the network trained with maximum allowed motivation value 0 (blue circles motivation is provided walk). For small 0s, the motivated network displays two-room binge behavior, while for larger 0s, migration dominates. Under the same conditions, the non-motivated network mostly displays two-room binge bchavior.
input to the network: yellow motivation: orange random
We trained a simple feedforward neural network (Figure AA) to generate behaviors using the state vector and the 4D vector of motivations as inputs. The network computed Q-values for five possible actions (up, down, left, right, stay), using TD method and backpropagating the d signal. The binary 36D (6x6) one-hot state vector represented the the agent's position. The network was trained times for different values of the maximum allowed motivation value A. As expected, the behavior displayed by the network depended on this parameter. The phase diagram of the agent's behaviors (Figure 2B, blue circles) shows that the agent successfully discovered the migration strategy and two-room binge strategies for high and low values correspondingly. For intermediate values of 0 (1.7 0 3), the network discovered delayed two-room binging strategy, in which it spent extra step in one of the room. The networks with motivation can also display a variety of complex behaviors for different motivation dynamics, such binging, addiction, withdrawal, etc. In one example, by increasing the maximum motivatiuon value for one of the demands ("smoking"), we trained networks to display "smoking addiction" (Figure A.B))
addiction model
8
=1
102r
02-10
time
smoking
Figure 3: Addiction model: (A) addiction motivation schedule. In the first three rooms, motivations are allowed to grow to the value of 1. In the fourth ("smoking" room), the motivation may grow to the value of 10. (B) Strategy learned by the network: spend 5 steps in the room #2, then visit the room #4 ("smoking")
Does motivation contribute to learning optimal strategies? To address this question, we performed similar set of simulations, except the motivation input to the network was suppressed (μ 0)- Although the input to such "non-motivated" networks was sufficient recover the optimal strategies, in most of the simulations the agents exercised two-room binging (Figure 2 yellow circles). The migration strategy, despite being optimal in 3/4 of the simulations, was successfully learned only by single agent out of 41. Moreover, the performance of the non-motivated networks often yielded that of the random walk (Figure orange circles). We conclude that motivation may facilitate learning by providing additional cues for temporal credit assignment in the rewards. Overall, suggest that motivation is helpful in generating complex ongoing behaviors based on simple conditions.
2.2 THE TRANSPORT NETWORK TASK
In the next example, the agent navigaties in a system of roads connecting cities (Figure 4A). The goal of the agent is visit certain subset of the target cities. The visiting order is not important, but the agent is supposed to use the route of minimal length. This problem is similar to the vehicle routing problem (Dantzig & Ramser 1959) (we do not require agents return to the city of origin).
4

---

Under review as a conference paper at ICLR 2020
A
B
0
5
agent
targets
Figure 4: The transport network example. An agent (black dot) navigates in network of roads connecting the cities cach associated with its own binary motivation. The perceived reward is equal to the value of the motivation vector f the position the agent, less the distance traveled. When the agent visits city with motivation (red circle), the motivation toward this city is reset (A-D) The steps of the agent through the network (black arrows) and the corresponding motivation vectors.
The task continues until Ó.
We trained neural network that receives the agent's state (position) and the motivation vector as inputs, then computes the Q-values for all available actions (connected cities) for the given position (Figure 55).. In every city, the agent receives a reward equal to the value of the motivation vector the position of the agent. The network is also negatively rewarded at every link between cities in proportion to the length of this link. We trained the network using TD method by backpropagating the TD error. Trained neural networks produced behaviors that closely match the shortest path solution (Figure (5.. In 82% of the test examples, the agent traveled the shortest path. In the remaining cases, the paths chosen by agents are close the shortest path solution. Overall, suggest that networks with motivation can solve complex transport problems. In doing so, the agent instructed to perform any particular goal, but instead learns set next target autonomously.
inputs
200
outputs
10'
path length
shortest path
Qicity I)
Q(city 2)
@=
10"
M(cityy 1) p(city 2)
Qicity
p(city N)
10"
10'
10
10
10"
10
iteration
Figure 5: Training a neural networks to find the shortest route to visit a subset of target cities using the motiva- tion framework. (A) One-layer neural network computing the Q-values. (B) Test performance on 50 sets of 3 random targets (blue). The precomputed shortest path solutions (red). The policy converges iteration 3 10
2.3 RESPONSES OF THE VP NEURONS PAVLOVIAN CONDITIONING TASK
To explore how motivation may be implemented in the brain, we trained 3 mice associate the specific cues (sound tones) with the different rewards (Fi In the experiment, the animals received one of five possible rewards: large or small positive reward (a drop of water); large or small negative reward (an air puff); or zero reward nothing at all. Trials containing positive or negative rewards combined with zero reward trials were separated into different blocks. During these blocks of trials, the animal was expected to be motivated and demotivated respectively. In course of the training, the animals learned anticipate both positive and negative rewards.
To relate behavior to the underlying neuronal circuits, we recorded the activity of the neurons in the VP brain area implicated in computing motivation (Berridge Schulkin 1989). The recordings were made while the mice were performing this task (Figure Overall, we obtained 149 well- isolated single neurons that showed task-related responses (Figure(6C) Our data suggests that the VP contains 2 large populations of oppositely-tune -tuned neurons, activated by positive and negative (Figure (D,E) rewards. To gain insight into a potential explanation for this phenomenon, we investigated artificial neural networks with motivation that were subjected similar conditions mice.
5

---

Under review as a conference paper at ICLR 2020
A
negative motivation block of trials
sample negative neuron in the brain
time
cue 1
cue
cue
newwad
50
M
x
firing rate (Hz)
B
positive motivation block of trials
1sec
40
0+
30
cue 4
20
N++
0
10
neuronal firing rates brain (VP)
sample positive neuron in the brain
mixed positive negalive
F
cue
reward
50
firing rate (H2)
neurons
40
30
20
10
0-
0+
F+
r++
time
time (s)
2
3
-2
time (s)
Figure 6: (A, B) A behavioral task for the recording. Trials were separated into the blocks during which only negative rewards (air puffs) only positive rewards (water drops) were delivered. (C) Responses of neurons recorded in 3 mice clustered into the negative motivation, positive motivation, and neurons of mixed sensitivity. (D-G) Activity sample neurons responding to negative (D, E) or positive (F, G) rewards.
As the Pavlovian conditioning task includes time variable (Figure 6 we chose to use recurrent neural network (RNN) as a basis our model, as suggested by Sutton & Barto (987). 7) The RNN received 2 inputs. One input described the cue function of time within trial (Figure 77. representing the state of the animal. Another input described motivation (constant within the entire trial) to indicate whether agent is in positive (μx +1 negative (μ² 1) block of trials.
The network has learned to accurately predict the trial outcome based on the cue (Figure B). For example, in the negative block of trials (μ+ 1)), before cue is presented value of future reward (ur, 8t) starts from low negative value, in an expectation of future negative reward. As the cue arrives, the expected value of future reward V/ represents the expected outcome. For example, in the trials with large negative reward (the leftmost column in Figure the network adjusts its expectation lower value after the cue arrives (s -08).. For trials with small negative reward (second column), no adjustment is necessary, and, therefore, reward expectation V/2 remains unaffected by the cue. V; decreases slightly after the cue arrives due the temporal discount 7 0.9. For negative-rewan trials ( column 3), in the negative block of trials, the expected reward increases after the cue arrives, due to the optimistic prediction. In positive block of trials (μ = +1, Figure BB columns 4-6), the behavior of the network is the same, except for the sign. Overall, our model yields reward expectations V that accurately reflect motivation and future rewards.
0), the expected
We then examined the responses of neurons in the model. We clustered the responses using unsu- pervised clustering algorithm (Sinakevitch et al. 2018). The neural population contained two large groups of oppositely tuned cells (Figure 7C), elevating their activity in positive and negative reward trials respectively. agreement with the experimental observations in the brain (Figure(6C)) Overall, we find close correspondence between activity of neurons in the artificial and biological networks.
What might be the functional significance of the two oppositely tuned neural populations? We found that the negative reward neurons (Figure 7D, blue cluster) tend form excitatory connections with each other, and so do the positive reward neurons (red cluster). Oppositely tuned cell, on the other hand, tend to inhibit each other (Figure FE.F). Thus, RNN our model yields a prediction for the structure of connectivity in the VP in the brain. Such connectivity helps maintaining the information about reward expectation within the trial. Indeed, in the Pavlovian conditioning task, cue and reward are separated by a temporal delay. During the delay, the network is supposed to maintain the infor- mation about upcoming reward, and, thus, acts as a working memory network (Her et al. 2016), which keeps reward expectation in its persistent activity. This persistent activity can be seen both the responses of individual neurons in the VP in the brain (Figure (C-E) and the RNN neurons in the model (Figure 7C). Previous studies in working memory and decision decision-making tasks (Machens jet al., 20055] Wong et al.] 2207): Her et al.) 2016) suggest that such parametric persistent activity can be maintained by two groups of oppositely tuned neurons, in the network architecture called

---

Under review as a conference paper at ICLR 2020
the "push-pull" circuit. This is exactly what find in RNN (Figure 7 Memory is main- tained in push-pull circuits via positive feedback. The positive feedback is produced by two forms of connectivity. First, similarly tuned neurons excite each other, as in Figure 7p. Second, oppositely tuned neurons inhibit each other, which introduces effective self-excitation via disinhibition. Over- all, we show that, similarly to real neurons, recurrent networks with motivation composed of two oppositely-tume classes of neurons, responding to positive and negative rewards. Our model also generates predictions for the structure of the VP connectivity.
2 inputs
40
output
B
inputs and outputs of the model
A
8(/) cwe(/)
8
+,0400
resuren
V
C
neuronal activations the model
mixed positive negalive
11+11
neurons
recurrent weigths
mixed posilive negative
postsynaple neuron
10
time
10
E
Wi: t-SNE
M:: scheme
positive
negative
-10
-102
presynaptic neuron
approach
avoid
Figure 7: (A) The architecture of the RNN computing the Q-function in the Pavlovian conditioning task. Inputs and outputs of the RNN for each type. Inputs: motivation, cue, reward. Outputs: sample V- function. Bottom row: TD error. Trial types (left to right): large negative reward, small negative reward, no negative reward, no positive reward, small positive reward, large positive reward. (C) Sample responses of neurons in the RNN clustered into the negative motivation (blue), positive motivation (red) and neurons of mixed sensitivity (other). Recurrent connectivity matrix. (E) t-SN representation. (F) Push-pull eircuit.
DISCUSSION
Motivation has been defined previously as the need-based modulation of reward magnitude. Here propose an RL approach to the neural networks that can be trained to include motivation into the calculation of action. We consider a diverse set of example networks that can solve different problems following similar pattern. We train such networks using TD rule via conventional back- propagation. We find that the networks can learn optimal behaviors, including behaviors that reflect complex scenarios of future motivation changes. When compared to the responses of neurons in the mouse brain, our neural network model can accurately predict behavioral outcomes, demonstrates similar patterns of neuronal responses, and generates predictions for network connectivity.
We trained our networks to compute future motivation-deeeendent reward in the Pavlovian condi- tioning task. Connecting RL and, in particular, TD methods was matter of the extensive research, reviewed by Suttonn & Barto (1987). We found that the neurons in the RNNs trained recogmize motivation can be clustered into oppositely tuned pop- ulations: neurons increasingly active in positive and negative reward trials. In agreement with this finding, we found similar two groups of neurons in motivation-depeeddntt estimates of reward (Richard et al.) 2016). Thus, neural networks with motivation, trained to perform in realistic tasks, develop responses similar to those the brain.
Pavlovian conditioning tasks
mouse VP: basal ganglia region implicated
The recurrent network structure in this Pavlovian conditioning case is compatible with the conven- tional models working memory. The information about upcoming reward once supplied by a is maintained in the network due the positive recurrent feedback. This feedback produced by inhibition between two oppositely tuned populations neurons, i.c. positive and negative reward sensitive cells. Thus, the experimentally observed presence of particular neural populations may be

---

Under review as a conference paper at ICLR 2020
consequence of the functional requirements on the network to maintain persistent variables within a trial. This function is reflected in both neural responses and architecture. Our findings present a generative hypothesis for how information about trial outcome is maintaned in the brain networks.
In recent work, Keramati & Gutkin (2014) show that homeostatic RL explains prominent motivation- related behavioral phenomena including anticipatory responding (Mansfiel & Cunningham 1980), dose-dependent reinforcement and potentiating effect of deprivation (Hodos) 1961). inhibitory effect of irrelevant drives (Dickinson & Balleine| 22022) etc. Although homeostatic RL defines the rewards as the gradients of the cost function with a fixed point, the theoretical predictions generalize to the models with linear, or approximately linear, multiplicative motivation. We therefore expect the behaviors of our models to be consistent with the large body of experimental data mentioned above.
Motivation offers framework compatible with other methods in machine learning, such as R- learning, goal-conditioned RL, and hierarchical RL (HRL). In R-learning. (Sinakevitch et al. 2018 Schwartz 1993), the cumulative sum of future rewards is computed with respect to the average level. The average reward level ís a slowly changing variable computed across several trials, which makes similar to motivation. In goal-conditioned RL- - the closest counterpart to RL with motivation the Q-function depends on three parameters: (5 (4 g). where y is the current static goal. In the motivation framework, multiple dynamic goals are present at the same time, and it is up to an agent to decide which one to pursue. HRL methods include the options framework (Sutton & Barto 1998 Sutton et al., 1999,, RL with subgoals (Sutton et al., 1999), feudal RI. (Dayan & Hinton 2000 Bacon & Precup 2018), and others. In HRI., complex tasks are solved by breaking them into
smaller, more manageable pieces. HRL approaches have several advantages compared to traditional RL, such as transfer of knowledge from already learned tasks and the ability to faster learn solutions to complex tasks. Although HRL methods are computationally efficient and generate behaviors separated into multiple levels of organization which resemble animals' behavior a mapping of HRL methods to brain networks is missing. Here, we suggest that motivation offers way for HRL algorithms to be implemented in the brain. case of motivation, both manager and lower-level actor nerworks receive the same reward, which makes motivated networks different from e.g. their feudal counterparts (Dayan & Hinton 2000 Bacon& & Precup, 2018).
As described above, actions in the motivation-based RL are selected on the basis of Q-function Q(ee, a, (u). An action a, selected at certain time maximizes the Q-function, representing the total expected future reward, and leads to the transition of the agent to the new state: 8 - at - $1+1- Because of the dependence of the Q-function on motivation, the action choice depends on the variable J representing motivation in our framework. We argued above that motivation allows RL to have the flexibility of a rapid change in behavioral policy when the need of an animal fluctuates. The same mechanism can be used to implement HRI., if motivation ju is supplied by another, higher-level "manager" network with its own Q-function, Q (Ie a()), p ).. When the higher-level network picks an action a;" it leads to a change in the motivational state for the lower-level network: μ - (1) + Mt+1 thus rapidly changing the behavior of the latter. The "manager" network could on its own be controlled by higher-level manager via its own motivation (*) Such decision hierarchy may include several management levels, with the dynamics of motivation on level / determined
via Q-function computed on level l 4 1: Q(++1) (1) (0--)),,(001) and n4!") + a, (I+1) + 1441-
Although HRI. outside the scope of this project, we suggest that motivation-basee RI studied here may link the neurobiology of adaptive behaviors to developments in machine learning.
Overall, we suggest that motivation-based networks may generate complex ongoing behaviors that can adapt to dynamic changes in an organism's 's demands. Thus, neural networks with motivation can both encompass more complex behaviors than networks with a fixed reward function and be mapped onto animals's circuits that control rewarded behaviors. Since animal performance in real- istic conditions depends on the states of satiety, wakefulness, etc., our approach should help build more realistic computational models that include these variables. Importantly, when we compared the responses of neurons in the mouse brain to our model, our neural network model can accurately predict behavioral outcomes, demonstrates similar patterns of neuronal responses, and generates predictions for network connectivity. In particular, our model explains why basal ganglia neurons form two classes: tuned to positive and negative rewards. In our model, these classes emerge from the need to maintain the information about future beward within the trial using positive recurrent feedback. Thus, networks with motivation considered here give imporant insights into the mecha- nisms of signal processing in brain reward circuits.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Picter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, pp. 5048-5058 2017.
P.L. Bacon and D. Precup. Constructing temporal abstractions autonomously in reinforcement learn- ing. Al Magazine, 39:39-50, 2018.
K.C. Berridge. From prediction error to incentive salience: mesolimbic computation of reward motivation. Eur J Neurosci, 35:1124-1143, 2012.
K.C. Berridge and J. Schulkin. Palatability shift a salt-associated incentive during sodium deple- tion. Q Exp Psychol B, 41:121-138, 1989.
G.B. Dantzig and J.H. Ramser. The truck dispatching problem. Management science, 6:80-91, 1959.
P. Dayan and G Hinton. Feudal reinforcement learning. 2000.
Anthony Dickinson and Bernard Balleine. The role of learning in the operation of motivational systems. Stevens' handbook of experimental psychology, 2002.
2002.
Howard Eichenbaum, Paul Dudchenko, Emma Wood, Matthew Shapiro, and Heikki Tanila. The hippocampus, memory, and place cells: spatial memory or a memory space? Neuron, 23(2): 209-226, 1999.
E.S. Her, N. Huh. Kim, and W. Jung. Neuronal activity in dorsomedial and dorsolateral striatum under the requirement for temporal credit assignment. Sci Rep, 6:27056, 2016.
William Hodos. Progressive ratio measure of reward strength. Science, 134(3483):943-90 944, 1961.
Mehdi Keramati and Boris Gutkin. Homeostatic reinforcement learning for integrating reward col- lection and physiological stability. Elife, 3:e0481 1, 2014.
C.K. Machens, R. Romo, and C.D. Brody. Flexible control of mutual inhibition: a neural model of two-interval discrimination. Science,
2005.
James G Mansfield and Christopher I. Cunningham. Conditioning and extinction of tolerance the hypothermic effect of ethanol in rats. Journal of Comparative and Physiological Psychology, 94 (5):962 1980.
V. Mnih, K. Kavukcuoglu, D. Silver, A.A. Rusu, J. Veness, M.G. Bellemare, A. Graves, M. Ried- miller, A.K. Fidjeland, G. Ostrovski, and S. Petersen. Human-level control through deep rein- forcement learning. Nature, 518:529-533, 2015.
J.M. Richard, F. Ambroggi. P.H. Janak, and H.L. Fields. Ventral pallidum neurons encode incentive value and promote cuc-clicited instrumental actions. Neuron, 90:1 106-11331,,,, 2016.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International conference on machine learning, pp. 1312-1320, 2015.
Schwartz. A reinforcement learning method for maximizing undiscounted rewards. Proceed- ings of the Tenth International Conference Machine Learning (ICML '93), pp. 298-305. IEEE Press, 1993.
I. Sinakevitch, G.R. Bjorklund. J.M. Newbern, R.C. Gerkin, and B.H. Smith. Comparative study of chemical neuroanatomy of the olfactory neuropil in mouse, honey bee, and human. Biol Cybern, 11::127-140, 2018.
Richard Sutton and Andrew G Barto. A comporal-differeme model of classical conditioning. In Proceedings of the ninth annual conference of the cognitive science society, pp. 255-378. Seattle, WA, 1987.

---

Under review as a conference paper at ICLR 2020
R.S. Sutton. The bitter lesson. 2019.
R.S. Sutton and A.G. Barto. Reinforcement learning: an introduction. MIT Press, Cambridge,
Mass., 1998.
R.S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112:181-211, 1999.
C.J. Watkins and P. Dayan. Q-learning. Machine learning, 8(3-4):27--292, 1992.
K.F. Wong, A.C. Huk, M.N. Shadlen, and X.J. Wang. Neural circuit dynamics underlying accumu- lation of time-narying evidence during perceptual decision making. Front Comput Neurosci, 1:6, 2007.
J. Zhang, K.C. Berridge, A.J. Tindell, K.S. Smith, and J.W. Aldridge. A neural computational model of incentive salience. PLoS Comput Biol, 5:e1000437, 2009.
A APPENDIX - METHODS
A.1 THE FOUR DEMANDS TASK
To optimize the behaviors the Four Demands task, we implemented feedforward neural network as described below. On the input, the network received an agent's state and motivation. The state variable contained an agent's position, which was represented by 36-dimensional one-hot vector. The motivation represented by 4-dimensional integer vector. From both state and motivation variables, we subtracted the mean values. To balance the contributions of state and motivation to the network, normalized their variances 1 and 9 respectively, since the ratio of the number of these variables is 4/36 (in case of non-motivated agents, we set the motivation variable to zero). The inputs of the network were propagated through three hidden layers (100 sigmoid units each), and an output layer (5 linear units). We trained the network to compute the Q-values of the possible actions: move left, right, up, down, or stay.
On every iteration, we picked an action, corresponding to the largest network output (Q-value). With probability s, we replaced the selected action with random action (s-greedy policy; E decreased exponentially from 0.5 to 0,05 throughout simulation; in case of random walk agents, we set s = 1). the selected action resulted step through "wall", the position remained unchanged; otherwise we updated the agent's position. For the agent's 's new position, we computed the perceived reward (F" fT"),, and used Bellman equation (7 0.9) to compute TD error. We then backpropagated the TD error through the network to update its weights (initialized using Xavier rule). We performed 10a training itcrations with the learning rate decreasing exponentially from 10 to 3 10 10-3.
We trained the network using various motivation schedules as follows. Each component of the motivation was increased by one on every iteration. a component of motivation μn, reached the threshold 8,, we stopped increasing this component any further. If the reward of type was consumed on current iteration, we dropped the corresponding component of motivation fn to zero. For motivated, non not-oviiva,ed, and random walk agents, we trained 41 model each (123 models total) with motivation thresholds 0, = 02 03 = 04 ranging from 1 to 100, spaced exponentially, one training run per unique 0 value. To mimic addiction, we also trained model with 0 0₂ 0g 1, and 04 = 10. For each run, we displayed sequences of agent's 's locations to establish correspondence between policies and average reward rates.
A.2 THE TRANSPORT NETWORK TASK
To build an environment for the transport network task, we defined the locations for 10 "cities" by sampling and y coordinates from the standard normal distribution (0, 1). For these locations, we computed Delaunay triangulation to define a network of the roads between the cities. For each road (Delaunay graph edge), computed its length the Euclidean distance between two cities it connects. We then selected multiple random subsets of 3 cities to be visited by an agent: the training set (10' target subsets), and the testing set (50 different target subsets).
10

---

Under review as a conference paper at ICLR 2020
To navigate the transport network, we implemented a feedforward neural network as described be- low. On the imput, the network received an agent's 's state and motivation. The state variable contained an agent's position, which was represented by 10-dimensional one-hot vector. The motivation was represented by a 10-dimensional binary vector. To specify the agent's targets, we initialized the motivation vector with 3 non-zero components Mii---Piis. corresponding the target cities 41.--33- The inputs of the network were propagated through hidden layer (200 Leaky ReLU units; leak 0.01), and an output layer (10 linear units). We trained the network to compute the Q-values of the potential actions (visiting each of the cities).
On every iteration within a task episode, we picked an action go from the current city to one of the immediately connected cities, then we updated the current position. To choose the action, we used the softmax policy (33 = 0.5) over the Q-values of the available moves. When the motivation Mi towards the new position j was non-zero, we yielded the reward of 5, and dropped the motivation H4 to On every iteration, we reduced the reward by the distance travelled within this iteration. The task episode terminated when all the components of motivation were equal zero. On every iteration, we used Bellman equation (7 0.9) to compute the TD error. We backpropagated the TD error through the network to update its weights (initialized using Xavier rule). Overall, performed training on 10* task episodes with the learning rate 10 10-?, 2 To assess the model performance, we evaluated the model on the testing set and compared the resulting path lengths the precomputed shortest path solutions.
A.3 PAVLOVIAN CONDITIONING TASK
To build a circuit model of motivation in Pavlovian conditioning task, we implemented a recurrent neural network. We trained the network on terminating sequences of 20 iterations, representing time within individual trials. On the input, the network received an agent's state and motivation. The state variable contained cue (conditioned stimulus; CS), which (#0.0, +0.4, =..3.. Depending on iteration, the state variable was equal either to the CS (iterations 6-9 out of 20), to zero (elsewhere). The motivation variable μ #1 was equal to the sign of the CS; it was constant throughout the entire sequence of 20 iterations. The inputs of the network were propagated through a recurrent layer (40 sigmoid units), and output layer (1 linear unit). We trained the network to compute the V-values for each iteration within the sequence.
chose randomly from
On every iteration, computed reward, reflecting the reward (unconditioned stimulus; US). De- pending on iteration, the reward was equal either to the CS (iterations 15-16), or to zero (elsewhere). We used the rewards in Bellman equation 7 0.9 to compute a TD error for every iteration. We then backpropagated the TD errors through time to update the network's 's weights (initially drawn from the uniform distribution U0-10-5, 10-9)). We performed training on 10 minibatches of 20 sequences each, with the learning rate of 10-1
We then clustered the recurrent neurons after training follows. First, for every neuron we com- puted 6 average activations, corresponding to the unique types of trials (positive/negative motivation with zero/small//arge reward). Then, we used the average activations to compute a correlation matrix for the neurons. Finally, we processed the correlation matrix with the watershed algorithm (marker- based; = 0.04), hence clustered the recurrent neurons. To examine the connectivity between the clusters, used the weights of the recurrent neurons to compute applied t-SNE in dimensions (p = =30), and color-coded the neurons with respect to the clusters.
correlation matrix. We then


Reviews:
[{"review_id": "ryed1wPvKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper builds a model of motivation-dependent learning.  A motivation channel is provided as an additional input to and RL-based learning system (essentially concatenated to state information), similar to goal-conditioned approaches (as the authors mention).  The motivational variables evolve according to their own rules, and are designed/interpreted as biological motivations such as water, food, sleep and work.  While the narrative is interesting, I lean towards reject as I believe it failed to deliver on what it promised.\n\nIn the first experiment, the satisfaction of these motivations are mapped onto a 4-room setting, where being in each room satisfies a motivation.  The choice to map the four rooms to biological drives is cute, but possibly confusing/misleading since this navigation problem really has nothing to do with these biological drives. A claim is that by providing the motivation as input to the policy, it is more robustly (across seeds) able to learn the \"migration\" (i.e. cycling) behavior among the rooms.  In a second example, a similar problem is solved involving navigation on a graph.\n\nThe final, most substantial example, is a policy trained to solve a simple, abstract version of a behavioral task. In this setting, a motivation channel was again used.  However, the motivation channel value is now fixed to one of two discrete values, essentially meaning it is simply a task-label variable, a paradigm that has already been applied in the context of simple models of neuroscience tasks, e.g. see Song et al. 2017 \"Reward-based training of recurrent neural networks for cognitive and value-based tasks\".  \n\nThere is a bit of a mixed framing overall as to whether it is being claimed that the \"motivation\" being passed as an input is a fundamental contribution to AI/RL (I think it is not), versus the computational modeling of biological motivation.  I think the people qualified to judge whether the computational model is a worthwhile model of motivation specifically are probably a narrower set of computational neuroscientists.  I do think there is value in the kind of computational modeling performed, involving establishing a relationship between training a neural network to solve a behavioral task and comparing this with real neural data.  This paradigm already becoming increasingly popular within computational neuroscience.  However, while I find the results slightly interesting, but not very significant, as someone interested in the biology of motivation, I question whether the nature of these contributions would be of broad interest at this venue.  \n\nMore fundamentally, I don't believe there is a meaningful ML/AI/RL contribution, and I have some issues with the presentation of the first two examples.  While I do like the narrative inspiring these problems, I find the implementations of the problems too simplified to really be meaningfully related to their inspiration (in terms of motivated behaviors).  Rather than really model motivation as part of the policy architecture, the authors have proposed a solution to modeling motivation that makes motivation a feature of the environment.  Essentially, the reward provided by the environment depends on an extra latent variable and by hiding this (in the cases where the policy does not see motivation inputs), it is quite likely that it becomes too difficult for the value function to predict what is happening (the environment has become partially observed).  This seems less a setting where motivation channels solve a problem, and more just an example of an environment that has more complex rules for generating rewards being more challenging to learn about, especially if latent variables are not available to the value function.  Critically, it has not been shown that motivational systems are useful for artificial agents, rather the tasks themselves have been designed to attempt to be models of biological motivation.  \n\nPersonally, I am interested in motivated behaviors and think that future AI developments should take note of this field, but again, the present work does not provide actionable insights into implementing an artificial motivation system.  At the same time, this work does not provide interesting enough neurobiological results for those to stand on their own either.\n\nMinor clarification:\n\n\"trained to perform in realistic tasks\" -- the task is very simple.  I would consider this a fairly abstract model of the task.  \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "rJgOwFa1oB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a computational model of motivation for Q learning and relates it to biological models of motivation. Motivation is presented to the agent as a component of its inputs, and is encoded in a vectorised reward function where each component of the reward is weighted. This approach is explored in three domains: a modified four-room domain where each room represents a different reward in the reward vector, a route planning problem, and a pavlovian conditioning example where neuronal activations are compared to mice undergoing a similar conditioning.\n\nReview Summary:\nI am uncertain of the neuroscientific contributions of this paper. From a machine learning perspective, this paper has insufficient details to assess both the experimental contributions and proposed formulation of motivation. It is unclear from the discussion of biological forms of motivation, and from the experimental elaboration of these ideas, that the proposed model of motivation is a novel contribution. For these reasons, I suggest a reject.\n\nThe Four Rooms Experiment:\n\nIn the four-rooms problem, the agent is provided with a one-hot encoding representing which cell it the agent is located in within the grid-world. The reward given to the agent is a combination of the reward signal from the environment (a one-hot vector where the activation is dependent on the room occupied by the agent) and the motivation vector, which is a weighting of the rooms. One agent is given access to the weighting vector mu in its state vector: the motivation is concatenated to the position, encoding the weighting of the rooms at any given time-step. The non-motivated agent does not have access to mu in its state, although its reward is weighted as the motivated agent’s is. The issue with this example is that the non-motivated agent does not have access to the information required to learn a value-function suitable to solve this problem. By not giving the motivation vector to non-motivated agent, the problem has become a partially observable problem, and the comparison is now between a partially observable and fully observable setting, rather than a commentary on the difference between learning with and without motivation.\n\nIn places, the claims made go beyond the results presented. How do we know that the non-motivated network is engaging in a \"non-motivated delay binge\"? We certainly can see that the agent acquires an average reward of 1, but it is not evident from this detail alone that the agent is engaging in the behaviour that the paper claims. \n\nMoreover, the network was trained 41 times for different values of the motivation parameter theta. Counting out the points in figure 2, it would suggest that the sweep was over 41 values of theta, which leaves me wondering if the results represent a single independent trial, or whether the results are averaged over multiple trials. Looking at the top-right hand corner I see a single yellow dot (non-motivated agent) presented in line with blue (motivated agent) suggesting that the point is possibly an outlier. Given this outlier, I’m led believe that the graph represents a single independent trial. A single trial is insufficient to draw conclusions about the behaviour of an agent. \n\nThe Path Routing Experiment:\n\nIn the second experiment, where a population of agents is presented in fig 5, it is claimed that on 82% of the trials, the agent was able to find the shortest path. Looking at the figure itself, at the final depicted iteration, all of the points are presented in a different colour and labelled “shortest path”. The graph suggests that 100% of the agents found the shortest path. The claim is made that for the remaining 18% of the agents, the agents found close to the shortest path—a point not evident in the figures presented.\n\n\nPavlovian Conditioning Experiment:\n\nIn the third experiment, shouldn’t Q(s) be V(s)? In this setting, the agent is not learning the value of a state action pair, but rather the value of a state. Moreover, the value is described as Q(t), where t is the time-step in the trial; however, elsewhere in the text it is mentioned that the state is not simply t, but contains also the motivation value mu.  \n\nThe third experiment does not have enough detail to interpret the results. It is unclear how many trials there were for both of the prediction settings. It is unclear whether the problem described is a continuing problem or a terminating prediction problem—i.e., whether after the conditioned stimulus and unconditioned stimulus are presented to the agent, does the time-step (and thus the state) reset to 0, or does time continue incrementing? If it is a terminating prediction problem, it is unclear whether the conditioned stimulus and unconditioned stimulus were delivered on the same time-steps for each independent trial. If I am interpreting the state-construction correctly, the state is incrementing by one on each time-step; this problem is effectively a Markov Reward Process where the agent transitions from one state to the next until time stops with no ability to transition to previous states.\n\nIn both the terminating and continuing cases, the choice of inputs is unusual. What was the motivation for using the time-step as part of the state construction?\n\nHow is the conditioned stimulus formulated in this setting? It is mentioned that it is a function of time, but there are no additional details.\n\nFrom reading the text, it is unclear whether fig 7b/c presents activations over multiple independent trials or a single trial.\n\nGeneral Thoughts on Framing:\n\nThis paper introduces non-standard terms without defining them first. For example, TD error is introduced as Reward Prediction Error, or RPE: a term that is not typically used in the Reinforcement Learning literature. To my understanding, there is a hypothesis about RPE in the brain in the cognitive science community; however, the connection between this idea in the cognitive science literature and its relation to RL methods is not immediately clear.\n\nTemporal Difference learning is incorrectly referred to as \"Time Difference\" learning (pg 2). \n\nNotes on technical details:\n\n- The discounting function gamma should be 0<= gamma <=1, rather than just <=1.\n\n- discounting not only prevents the sum of future rewards from diverging, but also plays an important role in determining the behaviour of an agent---i.e., the preference for short-term versus long-term rewards.\n\n- pg 2 \"the motivation is a slowly changing variable, that is not affected substantially by an average action\" -- it is not clear from the context what an average action is. \n\n- Why is the reward r(s|a), as opposed to r(s,a)?\n\nNotes on paper structure:\n\n- There are some odd choices in the structure of this paper. For instance, the second section---before the mathematical framing of the paper has been presented---is the results section. \n\n- In some sentences, citations are added where no claim is being made; it is not clear what the relevance of the citation is, or what the citation is supporting. E.g., “We chose to use a recurrent neural network (RNN) as a basis for our model” following with a citation for Sutton & Barto, 1987.\n\n- In some sentences, citations are not added where substantial claims are being made. E.g, “The recurrent network structure in this Pavlovian conditioning is compatible with the conventional models of working memory”. This claim is made, but it is never made clear what the conventional computational models of working memory are, or how they fit into the computational approaches proposed.\n\n- Unfortunately, a number of readers in the machine learning community might be unfamiliar with pavlovian conditioning and classical conditioning. Taking the time to unpack these ideas and contextualise them for the audience might help readers understand the paper and its relevance.\n\n- Figure 7B may benefit from displaying not just the predicted values V(s), but a plot of the prediction over time in comparison to the true expected return.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "B1gtEtq2YH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors investigate mechanisms underlying action selection in artificial agents and mice. To achieve this goal, they use RL to train neural networks to choose actions that maximize their temporally discounted sum of future rewards. Importantly, these rewards depend on a motivation factor that is itself a function of time and action; this motivation factor is the key difference between the authors' approach and \"vanilla\" RL. In simple tasks, the RL agent learns effective strategies (i.e., migrating between rooms in Fig. 1, and minimizing path lengths for the vehicle routing problem in Fig. 5). \n\nThe authors then apply their model to a task in which the agent is presented with sound cues. Depending on the trial block, the reward for the given cue is either zero, positive, or negative; the authors suggest that these varying reward values correspond to varying motivational states. In this setting, the model learns to have two populations of units; each selective to either positive or negative rewards. Recurrent excitation within populations and mutual inhibition between populations define the learned dynamics.\n\nFinally, the authors train mice on this same task, and record from neurons in area VP. Those neurons show a similar structure to the RNN: subpopulations of neurons respond to either positive or negative rewards.\n\nFirst, I'd like to thank the authors for the excellent clarity of this paper. It was very clear, and interesting to read.  \n\nI have some suggestions for how to deepen the connection between the model and the experiment, and some concerns about the necessity of the motivation framework to the Pavlovian task:\n\n1) The authors make the prediction that neurons in VP should show (functional) connectivity matching that learned by their model. This could be tested in their data. If that prediction is true, then one should see positive noise correlations for neuron pairs of the same preference (i.e., within the same pool, defined by spiking more for positive, or for negative rewards), and negative noise correlations for pairs of neurons with different preferences (i.e., one neuron in each pool).\n\n2) A recent preprint by Sederberg and Nemenman (doi: https://doi.org/10.1101/779223) argued against over-interpreting the stimulus selectivity of neurons in recurrent circuits. They showed that, even in randomly connected (untrained) networks: a) neurons show either positive or negative selectivity; and b) neuron pairs with selectivity for the same stimulus (or task) feature tend to excite each other, and neuron pairs with opposite selectivity tend to inhibit each other.  Given that finding, I wonder how compelling is the match between the mouse data and the RL agent (Figs. 6 and 7): could randomly-connected untrained networks show similar phenomena as in the mouse (Fig. 6)?\n\nI'm not asking if the untrained network can duplicate all the details of the trained one in Fig. 7. Just whether the mouse data could be recapitulated by a simpler (no training) model.\n\n3) For the Pavlovian conditioning in the RL agent, I'm not sure I'd describe this as changing motivation. It seems instead that the (external) reward contingency really changes between states. So the fact that the same network can make predictions in both cases seems more like metalearning than motivation-based action selection. For this reason, it's hard for me to connect the two halves of the paper: the first half has nice ideas on motivation-based action selection, while the second one has no apparent action selection, and hence no mechanism for the agent's motivation to matter.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a deep RL framework that incorporates motivation as input features, and is tested on 3 simplified domains, including one which is presented to rodents. 

While R2 found the paper well-written and interesting to read, a common theme among reviewer comments is that it’s not clear what the main contribution is, as it seems to simultaneously be claiming a ML contribution (motivation as a feature input helps with certain tasks) as well as a neuroscientific contribution (their agent exhibited representations that clustered similarly to those in animals). In trying to do both, it’s perhaps doing both a disservice. 

I think it’s commendable to try to bridge the fields of deep RL and neuroscience, and this is indeed an intriguing paper. However any such paper still needs to have a clear contribution. It seems that the ML contributions are too slight to be of general practical use, while the neuroscientific contributions are muddled somewhat. The authors several times mentioned the space constraints limiting their explanations. Perhaps this is an indication that they are trying to cover too much within one paper. I urge the authors to consider splitting it up into two separate works in order to give both the needed focus. 

I also have some concerns about the results themselves. R1 and R3 both mentioned that the comparison between the non-motivated agent and the motivated agent wasn’t quite fair, since one is essentially only given partial information. It’s therefore not clear how we should be interpreting the performance difference. Second, why was the non-motivated agent not analyzed in the same way as the motivated agent for the Pavlovian task? Isn’t this a crucial comparison to make, if one wanted to argue that the motivational salience is key to reproducing the representational similarities of the animals?  (The new experiment with the random fixed weights is interesting, I would have liked to see those results.) For these reasons and the ones laid out in the extensive comments of the reviewers, I’m afraid I have to recommend reject.

Decision: Reject