Neural networks with motivation

1 INTRODUCTION . Motivation is a cognitive process that propels an individual ’ s behavior towards or away from a particular object , perceived event , or outcome ( Zhang et al. , 2009 ) . Mathematically , motivation can be viewed as subjective modulation of the perceived reward value before the reward is received . Therefore , it reflects an organism ’ s wanting of the reward before the outcome is actually achieved . Computational models for motivated behavior , which are best represented by reinforcement learning ( RL ) models , are mostly concerned with the learning aspect of behavior . However , fluctuations in physiological states , such as confidence and motivation , can also profoundly affect behavior ( Zhang et al. , 2009 ) . Modeling such factors is thus an important goal in computational neuroscience and is in the early stages of mathematical description ( Berridge , 2012 ) . Here we build a neural network theory for motivational modulation of behavior based on Q-learning and apply this theory to mice performing Pavlovian conditioning task in which experimental observations of neural responses obtained in the ventral pallidum ( VP ) are available . We show that our motivated RL model both learns to correctly predict motivation-dependent rewards in the Pavlovian conditioning task and is consistent with responses of neurons in the VP . In particular , we show that , similarly to the VP neurons , Q-learning neural networks contain two oppositely-tuned populations of neurons responsive to positive and negative rewards . In the model , these two populations form a push-pull network that helps maintain motivation-dependent variables when inputs are missing . Our RL-based model is both consistent with experimental data and predicts the structure of the VP networks . We thus argue that motivation leads to complex behaviors which may add an extra level of complexity to machine learning approaches and is consistent with biological data . 2 RESULTS . Motivation is defined mathematically as a need-dependent modulation of the perceived reward value depending on animal ’ s extrinsic or intrinsic conditions ( Zhang et al. , 2009 ) . Thus , rats , which are normally repelled by high levels of salt in their food , may become attracted to a salt-containing solution following salt-free diet ( Berridge , 2012 ) . To model this observation , Berridge & Schulkin ( 1989 ) have proposed that the perceived reward rt received at time t is not absolute , but is modulated by an internal variable reflecting the level of motivation , which we will call here µ . The perceived level of the reward r̃t as a function of motivation µ can be expressed by the following equation : r̃t = r̃ ( rt , µ ) ( 1 ) In the simplest example , the reward , associated with salt is given by r̃t = µrt . Baseline motivation towards salt can be defined by µ = −1 , leading to the perceived reward of r̃t = −rt < 0 . Thus , normally the presence of salt in the diet is undesired . In the salt-free condition , the motivation changes to µ = +1 , leading to the subjective reward of r̃t = +rt ≥ 0 . Thus salt-containing diet becomes attractive . In reality , the function r̃ ( ... ) defining the impact of motivation on a perceived reward is complex ( Zhang et al. , 2009 ) , including the dependence on multiple factors described by a motivation vector ~µ . Individual components of this vector describe various needs experienced by the organism , such as thirst ( e.g . µ1 ) , appetite ( µ2 ) , etc . In this study , we explore the computational impact of motivation vector in the context of RL and investigate the brain circuits that might implement these computations . Our approach to motivation is based on Q-learning ( Watkins & Dayan , 1992 ) , which relies on an agent estimating Q-function , defined as the sum of future rewards given an action at chosen in a state ~st at time point t : Q ( ~st , at ) = ∑∞ τ=0 r ( ~st+τ |at ) γτ ( here and below , we omit averaging for simplicity ) . Here 0 < γ ≤ 1 is the discounting factor that keeps the sum from diverging , and balances preference of short- versus long-term rewards . If a correct Q-function is known , a rational agent picks an action that maximizes future rewards : at ← argmaxaQ ( ~st , a ) . In case of motivation in equation 1 , as reward values are affected by the motivation vector ~µ , for the Q-function , we obtain : Q ( ~st , at , ~µ ) = ∞∑ τ=0 r̃ ( ~st+τ , ~µt+τ |at ) γτ ( 2 ) Here r̃ ( ~st+τ , ~µt+τ |at ) is the motivation ~µ-dependent perceived reward obtained in a state ~st+τ reached at time t+ τ given action at chosen at time t. The state of the agent ~st and its motivation ~µ are distinct . The motivation is a slowly changing variable , that on average is not affected substantially by a single action . For example , the animal ’ s appetite does not change substantially during a single trial . At the same time , the actions selected by the animal lead to immediate changes of the animal ’ s state ~st . Recent research in neuroscience suggests that motivation and state may be represented and computed separately in the mammalian brain . Whereas motivation is usually attributed to the regions of the reward system , such as the VP ( Berridge & Schulkin , 1989 ; Berridge , 2012 ) , the state is likely to be computed elsewhere , e.g . in the hippocampus ( Eichenbaum et al. , 1999 ) , or cortex . In RL , an agent ’ s state and motivation may have different mathematical representations . In the examples below , the state variable is given by a one-hot vector , while motivation is represented by a full vector . Two arguments of the Q-function , ~st and ~µ , are therefore distinct . Finally , in hierarchical RL implementation , motivation is provided by a higher level network , while information about the state is generated externally . Although the Q-function with motivation ( equation 2 ) is similar to the Q-function in goalconditioned RL ( Schaul et al. , 2015 ; Andrychowicz et al. , 2017 ) , the underlying learning dynamics is different . Motivated behavior pursues multiple distributed sources of dynamic rewards . The Qfunction therefore accounts for the future motivation dynamics . This way , an agent with motivation chooses what reward to pursue – making it also different from RL with subgoals ( Sutton et al. , 1999 ) . Behavior with motivation therefore involves minimum to no handcrafted features , which suggests that motivation could provide a step towards general methods that leverage computation – a goal identified by Richard Sutton ( 2019 ) . As in the case of standard Q-learning , the action chosen by a rational agent maximizes the sum of the expected future perceived rewards , i.e . at ← argmaxaQ ( ~st , a , ~µ ) . To learn a correct Q-function , one can use the Temporal Difference ( TD ) method ( Sutton & Barto ( 1998 ) ) . If the Q-function is learned perfectly , it satisfies the recursive relationship Q ( ~st , at , ~µ ) = r̃ ( ~st , ~µt ) + γmaxat+1 Q ( ~st+1 , at+1 , ~µt+1 ) . For an incompletely learned motivation-dependent Q-function , the TD error δ is non-zero : δ = r̃ ( ~st , ~µt ) + γmax at+1 Q ( ~st+1 , at+1 , ~µt+1 ) −Q ( ~st , at , ~µt ) ( 3 ) TD error can be used to update motivation-dependent Q-function directly or to train neural networks to optimize their policy . Q-function depends on the new set of variables ~µ that evolve following their own rules . These variables reflect fluctuations in physiological or psychological states that substantially change the reward function and , therefore , can generate flexible behaviors dependent on animals ’ ongoing needs . We trained neural networks via backpropagation of the TD error ( equation 3 ) , an approach employed in deep Q-learning ( Mnih et al. , 2015 ) . Below we present several examples in which neural networks could be trained to solve motivation-dependent tasks . 2.1 THE FOUR DEMANDS TASK . Consider the example in Figure 1 . An agent navigates in a 6x6 square gridworld separated into four 3x3 subdivisions ( rooms ) ( Figure 1A ) . The environment was inspired by the work of Sutton et al . ( 1999 ) ; however , the task is different , as described below . In each room , the agent receives one and only one type of reward rn ( xt , yt ) , where n = 1 ... 4 ( Figure 1B ) . These rewards can be viewed as four different resources , such as water , food , sleep , and work . Motivation is described in this system by a 4D vector ~µ defining affinity of the agent for each of these resources . When the agent enters a room number n , the corresponding resource in the room is consumed , the agent receives rewards defined by r̃t = µn , and the corresponding component of the motivation vector µn is reset to zero ( Figure 1C ) . On the next time step , motivations in all four rooms are increased by one , i.e . µn ← µn+1 , which reflects additional “ wanting ” of the resource induced by the “ growing appetite ” . After a prolonged period of building up appetite , the motivation towards a resource saturates at a fixed maximum value of θ , which becomes a parameter of this model , determining the behavior . What are the potential behaviors of the agent ? Assume , that the maximum allowed motivation θ is large , and does not influence our results . If the agent always stays in the same room ( one-room binge strategy , Figure 1D ) , the rewards received by the agent consist of a sequence of zeros and ones , i.e . 0 , 1 , 0 , 1 , . . . ( in our model , after the motivation is set to zero , it is increased by one on the next time step ) . The average reward corresponding to this strategy is therefore r̄one−room binge = 1/2 . The average reward can be increased , if the agent jumps from room to room on each time step ( a two-room binge strategy , Figure 1E ) . In this case , the sequence of rewards received by the agent is described by the sequence of ones and the average reward is r̄two−room binge = 1 . Two-room binging therefore outperforms the one-room binge strategy . Finally , the agent can migrate by moving in a cycle through all four rooms ( Figure 1F ) . In this case , the agent spends three steps in each room and the overall period of migration is 12 steps . During these three steps , the agent receives the rewards of 9 ( the agent left this room nine steps ago ) , then 0 , and 1 ( r̄migration = 10/3 ) . Thus , migration strategy is more beneficial for the agent than both of the binging strategies . Migration , however , is affected by the maximum allowed motivation value θ . When θ < 9 , the benefits of migration strategy are reduced . For θ = 1 , for example , migration yields the reward rate of just r̄migration|θ=1 = 2/3 , which is below the return of the two-room binging . Thus , our model should display various behaviors depending on θ . We trained a simple feedforward neural network ( Figure 2A ) to generate behaviors using the state vector and the 4D vector of motivations as inputs . The network computed Q-values for five possible actions ( up , down , left , right , stay ) , using TD method and backpropagating the δ signal . The binary 36D ( 6x6 ) one-hot state vector represented the the agent ’ s position . The network was trained 41 times for different values of the maximum allowed motivation value θ . As expected , the behavior displayed by the network depended on this parameter . The phase diagram of the agent ’ s behaviors ( Figure 2B , blue circles ) shows that the agent successfully discovered the migration strategy and two-room binge strategies for high and low values of θ correspondingly . For intermediate values of θ ( 1.7 < θ < 3 ) , the network discovered a delayed two-room binging strategy , in which it spent an extra step in one of the room . The networks with motivation can also display a variety of complex behaviors for different motivation dynamics , such as binging , addiction , withdrawal , etc . In one example , by increasing the maximum motivatiuon value for one of the demands ( ” smoking ” ) , we trained networks to display ” smoking addiction ” ( Figure 3A , B ) . Does motivation contribute to learning optimal strategies ? To address this question , we performed a similar set of simulations , except the motivation input to the network was suppressed ( µ = 0 ) . Although the input to such “ non-motivated ” networks was sufficient to recover the optimal strategies , in most of the simulations the agents exercised two-room binging ( Figure 2B , yellow circles ) . The migration strategy , despite being optimal in 3/4 of the simulations , was successfully learned only by a single agent out of 41 . Moreover , the performance of the non-motivated networks often yielded that of the random walk ( Figure 2B , orange circles ) . We conclude that motivation may facilitate learning by providing additional cues for temporal credit assignment in the rewards . Overall , we suggest that motivation is helpful in generating complex ongoing behaviors based on simple conditions .

Summary: This paper presents a computational model of motivation for Q learning and relates it to biological models of motivation. Motivation is presented to the agent as a component of its inputs, and is encoded in a vectorised reward function where each component of the reward is weighted. This approach is explored in three domains: a modified four-room domain where each room represents a different reward in the reward vector, a route planning problem, and a pavlovian conditioning example where neuronal activations are compared to mice undergoing a similar conditioning.