Neural networks with motivation

1 INTRODUCTION . Motivation is a cognitive process that propels an individual ’ s behavior towards or away from a particular object , perceived event , or outcome ( Zhang et al. , 2009 ) . Mathematically , motivation can be viewed as subjective modulation of the perceived reward value before the reward is received . Therefore , it reflects an organism ’ s wanting of the reward before the outcome is actually achieved . Computational models for motivated behavior , which are best represented by reinforcement learning ( RL ) models , are mostly concerned with the learning aspect of behavior . However , fluctuations in physiological states , such as confidence and motivation , can also profoundly affect behavior ( Zhang et al. , 2009 ) . Modeling such factors is thus an important goal in computational neuroscience and is in the early stages of mathematical description ( Berridge , 2012 ) . Here we build a neural network theory for motivational modulation of behavior based on Q-learning and apply this theory to mice performing Pavlovian conditioning task in which experimental observations of neural responses obtained in the ventral pallidum ( VP ) are available . We show that our motivated RL model both learns to correctly predict motivation-dependent rewards in the Pavlovian conditioning task and is consistent with responses of neurons in the VP . In particular , we show that , similarly to the VP neurons , Q-learning neural networks contain two oppositely-tuned populations of neurons responsive to positive and negative rewards . In the model , these two populations form a push-pull network that helps maintain motivation-dependent variables when inputs are missing . Our RL-based model is both consistent with experimental data and predicts the structure of the VP networks . We thus argue that motivation leads to complex behaviors which may add an extra level of complexity to machine learning approaches and is consistent with biological data . 2 RESULTS . Motivation is defined mathematically as a need-dependent modulation of the perceived reward value depending on animal ’ s extrinsic or intrinsic conditions ( Zhang et al. , 2009 ) . Thus , rats , which are normally repelled by high levels of salt in their food , may become attracted to a salt-containing solution following salt-free diet ( Berridge , 2012 ) . To model this observation , Berridge & Schulkin ( 1989 ) have proposed that the perceived reward rt received at time t is not absolute , but is modulated by an internal variable reflecting the level of motivation , which we will call here µ . The perceived level of the reward r̃t as a function of motivation µ can be expressed by the following equation : r̃t = r̃ ( rt , µ ) ( 1 ) In the simplest example , the reward , associated with salt is given by r̃t = µrt . Baseline motivation towards salt can be defined by µ = −1 , leading to the perceived reward of r̃t = −rt < 0 . Thus , normally the presence of salt in the diet is undesired . In the salt-free condition , the motivation changes to µ = +1 , leading to the subjective reward of r̃t = +rt ≥ 0 . Thus salt-containing diet becomes attractive . In reality , the function r̃ ( ... ) defining the impact of motivation on a perceived reward is complex ( Zhang et al. , 2009 ) , including the dependence on multiple factors described by a motivation vector ~µ . Individual components of this vector describe various needs experienced by the organism , such as thirst ( e.g . µ1 ) , appetite ( µ2 ) , etc . In this study , we explore the computational impact of motivation vector in the context of RL and investigate the brain circuits that might implement these computations . Our approach to motivation is based on Q-learning ( Watkins & Dayan , 1992 ) , which relies on an agent estimating Q-function , defined as the sum of future rewards given an action at chosen in a state ~st at time point t : Q ( ~st , at ) = ∑∞ τ=0 r ( ~st+τ |at ) γτ ( here and below , we omit averaging for simplicity ) . Here 0 < γ ≤ 1 is the discounting factor that keeps the sum from diverging , and balances preference of short- versus long-term rewards . If a correct Q-function is known , a rational agent picks an action that maximizes future rewards : at ← argmaxaQ ( ~st , a ) . In case of motivation in equation 1 , as reward values are affected by the motivation vector ~µ , for the Q-function , we obtain : Q ( ~st , at , ~µ ) = ∞∑ τ=0 r̃ ( ~st+τ , ~µt+τ |at ) γτ ( 2 ) Here r̃ ( ~st+τ , ~µt+τ |at ) is the motivation ~µ-dependent perceived reward obtained in a state ~st+τ reached at time t+ τ given action at chosen at time t. The state of the agent ~st and its motivation ~µ are distinct . The motivation is a slowly changing variable , that on average is not affected substantially by a single action . For example , the animal ’ s appetite does not change substantially during a single trial . At the same time , the actions selected by the animal lead to immediate changes of the animal ’ s state ~st . Recent research in neuroscience suggests that motivation and state may be represented and computed separately in the mammalian brain . Whereas motivation is usually attributed to the regions of the reward system , such as the VP ( Berridge & Schulkin , 1989 ; Berridge , 2012 ) , the state is likely to be computed elsewhere , e.g . in the hippocampus ( Eichenbaum et al. , 1999 ) , or cortex . In RL , an agent ’ s state and motivation may have different mathematical representations . In the examples below , the state variable is given by a one-hot vector , while motivation is represented by a full vector . Two arguments of the Q-function , ~st and ~µ , are therefore distinct . Finally , in hierarchical RL implementation , motivation is provided by a higher level network , while information about the state is generated externally . Although the Q-function with motivation ( equation 2 ) is similar to the Q-function in goalconditioned RL ( Schaul et al. , 2015 ; Andrychowicz et al. , 2017 ) , the underlying learning dynamics is different . Motivated behavior pursues multiple distributed sources of dynamic rewards . The Qfunction therefore accounts for the future motivation dynamics . This way , an agent with motivation chooses what reward to pursue – making it also different from RL with subgoals ( Sutton et al. , 1999 ) . Behavior with motivation therefore involves minimum to no handcrafted features , which suggests that motivation could provide a step towards general methods that leverage computation – a goal identified by Richard Sutton ( 2019 ) . As in the case of standard Q-learning , the action chosen by a rational agent maximizes the sum of the expected future perceived rewards , i.e . at ← argmaxaQ ( ~st , a , ~µ ) . To learn a correct Q-function , one can use the Temporal Difference ( TD ) method ( Sutton & Barto ( 1998 ) ) . If the Q-function is learned perfectly , it satisfies the recursive relationship Q ( ~st , at , ~µ ) = r̃ ( ~st , ~µt ) + γmaxat+1 Q ( ~st+1 , at+1 , ~µt+1 ) . For an incompletely learned motivation-dependent Q-function , the TD error δ is non-zero : δ = r̃ ( ~st , ~µt ) + γmax at+1 Q ( ~st+1 , at+1 , ~µt+1 ) −Q ( ~st , at , ~µt ) ( 3 ) TD error can be used to update motivation-dependent Q-function directly or to train neural networks to optimize their policy . Q-function depends on the new set of variables ~µ that evolve following their own rules . These variables reflect fluctuations in physiological or psychological states that substantially change the reward function and , therefore , can generate flexible behaviors dependent on animals ’ ongoing needs . We trained neural networks via backpropagation of the TD error ( equation 3 ) , an approach employed in deep Q-learning ( Mnih et al. , 2015 ) . Below we present several examples in which neural networks could be trained to solve motivation-dependent tasks . 2.1 THE FOUR DEMANDS TASK . Consider the example in Figure 1 . An agent navigates in a 6x6 square gridworld separated into four 3x3 subdivisions ( rooms ) ( Figure 1A ) . The environment was inspired by the work of Sutton et al . ( 1999 ) ; however , the task is different , as described below . In each room , the agent receives one and only one type of reward rn ( xt , yt ) , where n = 1 ... 4 ( Figure 1B ) . These rewards can be viewed as four different resources , such as water , food , sleep , and work . Motivation is described in this system by a 4D vector ~µ defining affinity of the agent for each of these resources . When the agent enters a room number n , the corresponding resource in the room is consumed , the agent receives rewards defined by r̃t = µn , and the corresponding component of the motivation vector µn is reset to zero ( Figure 1C ) . On the next time step , motivations in all four rooms are increased by one , i.e . µn ← µn+1 , which reflects additional “ wanting ” of the resource induced by the “ growing appetite ” . After a prolonged period of building up appetite , the motivation towards a resource saturates at a fixed maximum value of θ , which becomes a parameter of this model , determining the behavior . What are the potential behaviors of the agent ? Assume , that the maximum allowed motivation θ is large , and does not influence our results . If the agent always stays in the same room ( one-room binge strategy , Figure 1D ) , the rewards received by the agent consist of a sequence of zeros and ones , i.e . 0 , 1 , 0 , 1 , . . . ( in our model , after the motivation is set to zero , it is increased by one on the next time step ) . The average reward corresponding to this strategy is therefore r̄one−room binge = 1/2 . The average reward can be increased , if the agent jumps from room to room on each time step ( a two-room binge strategy , Figure 1E ) . In this case , the sequence of rewards received by the agent is described by the sequence of ones and the average reward is r̄two−room binge = 1 . Two-room binging therefore outperforms the one-room binge strategy . Finally , the agent can migrate by moving in a cycle through all four rooms ( Figure 1F ) . In this case , the agent spends three steps in each room and the overall period of migration is 12 steps . During these three steps , the agent receives the rewards of 9 ( the agent left this room nine steps ago ) , then 0 , and 1 ( r̄migration = 10/3 ) . Thus , migration strategy is more beneficial for the agent than both of the binging strategies . Migration , however , is affected by the maximum allowed motivation value θ . When θ < 9 , the benefits of migration strategy are reduced . For θ = 1 , for example , migration yields the reward rate of just r̄migration|θ=1 = 2/3 , which is below the return of the two-room binging . Thus , our model should display various behaviors depending on θ . We trained a simple feedforward neural network ( Figure 2A ) to generate behaviors using the state vector and the 4D vector of motivations as inputs . The network computed Q-values for five possible actions ( up , down , left , right , stay ) , using TD method and backpropagating the δ signal . The binary 36D ( 6x6 ) one-hot state vector represented the the agent ’ s position . The network was trained 41 times for different values of the maximum allowed motivation value θ . As expected , the behavior displayed by the network depended on this parameter . The phase diagram of the agent ’ s behaviors ( Figure 2B , blue circles ) shows that the agent successfully discovered the migration strategy and two-room binge strategies for high and low values of θ correspondingly . For intermediate values of θ ( 1.7 < θ < 3 ) , the network discovered a delayed two-room binging strategy , in which it spent an extra step in one of the room . The networks with motivation can also display a variety of complex behaviors for different motivation dynamics , such as binging , addiction , withdrawal , etc . In one example , by increasing the maximum motivatiuon value for one of the demands ( ” smoking ” ) , we trained networks to display ” smoking addiction ” ( Figure 3A , B ) . Does motivation contribute to learning optimal strategies ? To address this question , we performed a similar set of simulations , except the motivation input to the network was suppressed ( µ = 0 ) . Although the input to such “ non-motivated ” networks was sufficient to recover the optimal strategies , in most of the simulations the agents exercised two-room binging ( Figure 2B , yellow circles ) . The migration strategy , despite being optimal in 3/4 of the simulations , was successfully learned only by a single agent out of 41 . Moreover , the performance of the non-motivated networks often yielded that of the random walk ( Figure 2B , orange circles ) . We conclude that motivation may facilitate learning by providing additional cues for temporal credit assignment in the rewards . Overall , we suggest that motivation is helpful in generating complex ongoing behaviors based on simple conditions .

Summary: The authors investigate mechanisms underlying action selection in artificial agents and mice. To achieve this goal, they use RL to train neural networks to choose actions that maximize their temporally discounted sum of future rewards. Importantly, these rewards depend on a motivation factor that is itself a function of time and action; this motivation factor is the key difference between the authors' approach and "vanilla" RL. In simple tasks, the RL agent learns effective strategies (i.e., migrating between rooms in Fig. 1, and minimizing path lengths for the vehicle routing problem in Fig. 5).