Visceral Machines: Risk-Aversion in  Reinforcement Learning with Intrinsic Physiological Rewards

1 INTRODUCTION . The human autonomic nervous system ( ANS ) is composed of two branches . One of these , the sympathetic nervous system ( SNS ) , is “ hard-wired ” to respond to potentially dangerous situations often reducing , or by-passing , the need for conscious processing . The ability to make rapid decisions and respond to immediate threats is one way of protecting oneself from danger . Whether one is in the African savanna or driving in Boston traffic . The SNS regulates a range of visceral functions from the cardiovascular system to the adrenal system ( Jansen et al. , 1995 ) . The anticipatory response in humans to a threatening situation is for the heart rate to increase , heart rate variability to decrease , blood to be diverted from the extremities and the sweat glands to dilate . This is the body ’ s “ fight or flight ” response . While the primary role of these anticipatory responses is to help one prepare for action , they also play a part in our appraisal of a situation . The combination of sensory inputs , physiological responses and cognitive evaluation form emotions that influence how humans learn , plan and make decisions ( Loewenstein & Lerner , 2003 ) . Intrinsic motivation refers to being moved to act based on the way it makes one feel . For example , it is generally undesirable to be in a situation that causes fear and thus we might choose to take actions that help avoid these types of contexts in future . This is contrasted with extrinsic motivation that involves explicit goals ( Chentanez et al. , 2005 ) . Driving is an everyday example of a task in which we commonly rely on both intrinsic and extrinsic motivations and experience significant physiological changes . When traveling in a car at highspeed one may experience a heightened state of arousal . This automatic response is correlated with the body ’ s reaction to the greater threats posed by the situation ( e.g. , the need to adjust steering more rapidly to avoid a pedestrian that might step into the road ) . Visceral responses are likely to preempt accidents or other events ( e.g. , a person will become nervous before losing control and hitting someone ) . Therefore , these signals potentially offer an advantage as a reward mechanism compared to extrinsic rewards based on events that occur in the environment , such as a collision . This paper provides a reinforcement learning ( RL ) framework that incorporates reward functions for achieving task-specific goals and also minimizes a cost trained on physiological responses to the environment that are correlated with stress . We ask if such a reward function with extrinsic and intrinsic components is useful in a reinforcement learning setting . We test our approach by training a model on real visceral human responses in a driving task . The key challenges of applying RL in the real-world include the amount of training data required and the high-cost associated with failure cases . For example , when using RL in autonomous driving , rewards are often sparse and skewed . Furthermore , bad actions can lead to states that are both catastrophic and expensive to recover from . While much of the work in RL focuses on mechanisms that are task or goal dependent , it is clear that humans also consider the feedback from the body ’ s nervous system for action selection . For example , increased arousal can help signal imminent danger or failure to achieve a goal . Such mechanisms in an RL agent could help reduce the sample complexity as the rewards are continually available and could signal success or failure before the end of the episode . Furthermore , these visceral signals provide a warning mechanism that in turn could lead to safer explorations . Our work is most closely related to that in intrinsically motivated learning ( Chentanez et al. , 2005 ; Zheng et al. , 2018 ; Haber et al. , 2018 ; Pathak et al. , 2017 ) that uses a combination of intrinsic and extrinsic rewards and shows benefits compared to using extrinsic rewards alone . The key distinction in our work is that we specifically aim to build intrinsic reward mechanisms that are visceral and trained on signals correlated with human affective responses . Our approach could also be considered a form of imitation learning ( Ross et al. , 2011 ; Ross & Bagnell , 2014 ; Ho & Ermon , 2016 ; Chang et al. , 2015 ) as we use the signal from a human expert for training . However , a difference is that our signal is an implicit response from the driver versus an explicit instruction or action which might commonly be the case in imitation learning . The structural credit assignment problem , or generalization problem , aims to address the challenge posed by large parameter spaces in RL and the need to give the agent the ability to guess , or have some intuition about new situations based on experience ( Lin , 1992 ) . A significant advantage of our proposed method is the reduced sparsity of the reward signal . This makes learning more practical in a large parameter space . We conduct experiments to provide empirical evidence that this can help reduce the number of epochs required in learning . In a sense , the physiological response could be considered as an informed guess about new scenarios before the explicit outcome is known . The challenge with traditional search-based structured prediction is the assumptions that must be made in the search algorithms that are required ( Daumé et al. , 2009 ) . By training a classifier using a loss based on the human physiological response this problem can potentially be simplified . The core contributions of this paper are to : ( 1 ) present a novel approach to learning in which the reward function is augmented with a model learned directly from human nervous system responses , ( 2 ) show how this model can be incorporated into a reinforcement learning paradigm and ( 3 ) report the results of experiments that show the model can improve both safety ( reducing the number of mistakes ) and efficiency ( reducing the sample complexity ) of learning . In summary , we argue that a function trained on physiological responses could be used as an intrinsic reward or value function for artificially intelligent systems , or perhaps more aptly artificially emotionally intelligent systems . We hypothesize that incorporating intrinsic rewards with extrinsic rewards in an RL framework ( as shown in Fig 1 ) will both improve learning efficiency as well as reduce catastrophic failure cases that occur during the training . 2 BACKGROUND . 2.1 SYMPATHETIC NERVOUS SYSTEM . The SNS is activated globally in response to fear and threats . Typically , when threats in an environment are associated with a “ fight of flight '' response the result is an increase in heart rate and perspiration and release of adrenaline and cortisol into the circulatory system . These physiological changes act to help us physically avoid danger but also play a role in our appraisal of emotions and ultimately our decision-making . A large volume of research has found that purely rational decision-making is sub-optimal ( Lerner et al. , 2015 ) . This research could be interpreted as indicating that intrinsic rewards ( e.g. , physiological responses and the appraisal of an emotion ) serve a valuable purpose in decision-making . Thus , automatic responses both help people act quickly and in some cases help them make better decisions . While these automatic responses can be prone to mistakes , they are vital for keeping us safe . Logical evaluation of a situation and the threat it presents is also important . Ultimately , a combination of intrinsic emotional rewards and extrinsic rational rewards , based on the goals one has , is likely to lead to optimal results . 2.2 REINFORCEMENT LEARNING . We consider the standard RL framework , where an agent interacts with the environment ( described by a set of states S ) , through a set of actions ( A ) . An action at at a time-step t leads to a distribution over the possible future state p ( st+1|st , at ) , and a reward r : S × A → R. In addition , we start with a distribution of initial states p ( s0 ) and the goal of the agent is to maximize the discounted sum of future rewards : Rt = ∑∞ i=t γ i−tri , where γ is the discount factor . Algorithms such as Deep Q-Networks ( DQN ) learn a Neural-Network representation of a deterministic policy π : S → A that approximates an optimal Q-function : Q∗ ( s , a ) = Es′∼p ( ·|s , a ) [ r ( s , a ) + γmaxa′∈AQ∗ ( s′ , a′ ) ] . The application of RL techniques to real-world scenarios , such as autonomous driving , is challenging due to the high sample complexity of the methods . High-sample complexity arises due to the creditassignment problem : it is difficult to identify which specific action from a sequence was responsible for a success or failure . This issue is further exacerbated in scenarios where the rewards are sparse . Reward shaping ( Ng et al. , 1999 ; Russell , 1998 ) is one way to deal with the sample complexity problem , in which heuristics are used to boost the likelihood of determining the responsible action . We contrast sparse episodic reward signals in RL agents with physiological responses in humans . We conject that the sympathetic nervous system ( SNS ) responses for a driver are as informative and useful , and provide a more continuous form of feedback . An example of one such SNS response is the volumetric change in blood in the periphery of skin , controlled in part through vasomodulation . We propose to use a reward signal that is trained on a physiological signal that captures sympathetic nervous system activity . The key insight being that physiological responses in humans indicate adverse and risky situations much before the actual end-of-episode event ( e.g . an accident ) and even if the event never occurs . By utilizing such a reward function , not only is the system able to get a more continuous and dense reward but it also allows us to reason about the credit assignment problem . This is due to the fact that an SNS response is often tied causally to the set of actions responsible for the eventual success or failure of the episode . Our work is related , in spirit , to a recent study that used facial expressions as implicit feedback to help train a machine learning systems for image generation ( Jaques et al. , 2018 ) . The model produced sketches that led to significantly more positive facial expressions when trained with input of smile responses from an independent group . However , this work was based on the idea of Social Learning Theory ( Bandura & Walters , 1977 ) and that humans learn from observing the behaviors of others , rather than using their own nervous system response as a reward function . 3 THE PROPOSED FRAMEWORK . Our proposal is to consider a reward function that has both an extrinsic component r and an intrinsic component r̃ . The extrinsic component rewards behaviors that are task specific , whereas the intrinsic component specifically aims to predict a human physiological response to SNS activity and reward actions that lead to states that reduce stress and anxiety . The final reward r̂ then is a function that considers both the extrinsic as well as intrinsic components r̂ = f ( r , r̃ ) . Theoretically , the function f ( · , · ) can be fairly complex and one possibility would be to parameterize it as a neural network . For simplicity , we consider linear combinations of the extrinsic and intrinsic rewards in this paper . Formally , lets consider an RL framework based on a DQN with reward r. We propose to use a modified reward r̂ that is a convex combination of the original reward r and a component that mirrors human physiological responses r̃ : r̂ = λr + ( 1− λ ) r̃ ( 1 ) Here λ is a weighting parameter that provides a trade-off between the desire for task completion ( extrinsic motivation ) and physiological response ( intrinsic motivation ) . For example , in an autonomous driving scenario the task dependent reward r can be the velocity , while r̃ can correspond to physiological responses associated with safety . The goal of the system then is to complete the task while minimizing the physiological arousal response . The key challenge now is to build a computational model of the intrinsic reward r̃ given the state of the agent . In the rest of the paper we focus on the autonomous driving scenario as a canonical example and discuss how we can model the appropriate physiological responses and utilize them effectively in this framework . One of the greatest challenges in building a predictive model of SNS responses is the collection of realistic ground truth data . In this work , we use high-fidelity simulations ( Shah et al. , 2018 ) to collect physiological responses of humans and then train a deep neural network to predict SNS responses that will ultimately be used during the reinforcement learning process . In particular , we rely on the photoplethysmographic ( PPG ) signal to capture the volumetric change in blood in the periphery of the skin ( Allen , 2007 ) . The blood volume pulse waveform envelope pinches when a person is startled , fearful or anxious , which is the result of the body diverting blood from the extremities to the vital organs and working muscles to prepare them for action , the “ fight or flight ” response . Use of this phenomenon in affective computing applications is well established and has been leveraged to capture emotional responses in marketing/media testing ( Wilson & Sasse , 2000 ) , computer tasks ( Scheirer et al. , 2002 ) and many other psychological studies ( L. Fredrickson & Levenson , 1998 ; Gross , 2002 ) . The peripheral pulse can be measured unobtrusively and even without contact ( Poh et al. , 2010 ; Chen & McDuff , 2018 ) , making it a good candidate signal for scalable measurement . We leverage the pulse signal to capture aspects of the nervous system response and our core idea is to train an artificial network to mimic the pulse amplitude variations based on the visual input from the perspective of the driver . To design a reward function based on the nervous system response of the driver in the simulated environment we collected a data set of physiological recordings and synchronized first person video frames from the car . Using this data we trained a convolutional neural network ( CNN ) to mimic the physiological response based on the input images . Fig 2 shows a section of the recorded blood volume pulse signal with pulse peaks highlighted , notice how the waveform envelope changes . Reinforcement Learning Environments : We performed our experiments in AirSim ( Shah et al. , 2018 ) where we instantiated an autonomous car in a maze . The car was equipped with an RGB camera and the goal for the agent was to learn a policy that maps the camera input to a set of controls ( discrete set of steering angles ) . The agent ’ s extrinsic reward can be based on various driving related tasks , such as keeping up the velocity , making progress towards a goal , traveling large distances , and can be penalized heavily for collisions . Fig 2 shows example frames captured from the environment . The maze consisted of walls and ramps and was designed to be non-trivial to navigate for the driver . Intrinsic Reward Architecture : We used a CNN to predict the normalized pulse amplitude derived from the physiological response of the driver . The image frames from the camera sensor in the environment served as an input to the network . The input frames were downsampled to 84 × 84 pixels and converted to grayscale format . They were normalized by subtracting the mean pixel value ( calculated on the training set ) . The network architecture is illustrated in Fig 3 . A dense layer of 128 hidden units preceded the final layer that had linear activation units and a mean square error ( MSE ) loss , so the output formed a continuous signal from 0 to 1 . Training the Reward Network : We recruited four participants ( 2 male , 2 female ) to drive a vehicle around the maze and to find the exit point . All participants were licensed drivers and had at least seven years driving experience . For each participant we collected approximately 20 minutes ( ∼24,000 frames at a resolution of 256 × 144 pixels and frame rate of 20 frames-per-second ) of continuous driving in the virtual environment ( for a summary of the data see Table 1 ) . In addition , the PPG signal was recorded from the index finger of the non-dominant hand using a Shimmer31 GSR+ with an optical pulse sensor . The signal was recorded at 51.6Hz . The physiological signals were synchronized with the frames from the virtual environment using the same computer clock . A standard custom peak detection algorithm ( McDuff et al. , 2014 ) was used to recover the systolic peaks from the pulse waveform . The amplitudes of the peaks were normalized , to a range 0 to 1 , across the entire recording . Following the experiment the participants reported how stressful they found the task ( Not at all , A little , A moderate amount , A lot , A great deal ) . The participants all reported experiencing some stress during the driving task . The frames and pulse amplitude measures were then used to train the CNN ( details in the next section ) . The output of the resulting trained CNN ( the visceral machine ) was used as the reward ( r̃ = CNN Output ) in the proposed framework .

Summary: Starting from the hypothesis that humans have evolved basic autonomic visceral responses that influence decision making in a meaningful way and that these are at work in driving a car, the authors propose to use such signals within the RL framework. This is accomplished by augmenting the RL reward function with a model learned directly from human nervous system responses. This leads to a