Soft Actor-Critic with Inhibitory Networks for Faster Retraining

1 INTRODUCTION . In reinforcement learning ( RL ) , for all but the simplest tasks , agents behavior must be optimized with respect to multiple goals and constraints ( Sutton & Barto , 2018 ) . It is common in practice for new objectives and constraints to arise after already having trained an agent on existing ones . In order for the behavior of the agent to account for these additional constraints , retraining is required . Within the context of deep RL , retraining an agent to new constraints gives problems in balancing exploitation of previously learned skills with learning new skills . In this paper , we present a method for efficiently retraining an agent to acquire new skills in a similar environment . Existing approaches to solving the problem of learning new skills include the use of hierarchical RL structures ( Dayan & Hinton , 1993 ; Barto & Mahadevan , 2003 ) , such as the options framework ( Sutton et al. , 1999 ; Comanici & Precup , 2010 ) , universal value functions ( Schaul et al. , 2015 ) , option-critic ( Bacon et al. , 2017 ) , FeUdal networks ( Vezhnevets et al. , 2017 ) , and data-efficient hierarchical RL ( Nachum et al. , 2018 ) . Other methods use multiple policies and value functions each of which are optimized for simple objectives that can then be composed to achieve complex objectives ( Van Seijen et al. , 2017 ; Sahni et al. , 2017 ; Haarnoja et al. , 2017 ; Hansen et al. , 2020 ; Barreto et al. , 2020 ) . In this paper we propose to address the problem through the use of multiple value functions to provide a complex evaluative input to a single policy network . By applying different value functions in a state dependent fashion , the reward provided to the policy network during training can remain the same as in prior training when appropriate , and can switch to a different reward when the situation indicates new constraints or goals . The mechanism we propose to govern the output of the composite value function is based upon neuroscientific research on inhibitory control ( Diamond , 2013 ) . The brain uses inhibition to interrupt ongoing goal-directed action when unexpected events or conflicts arise . The horse-race model of Logan et al . ( 2015 ) , which describes the behavioral response to a goal and its inhibition in terms of dual processes , is supported by many behavioral and neurobiological studies ( Verbruggen & Logan , 2009 ; Shenoy et al. , 2011 ; Ide et al. , 2013 ; Schall et al. , 2017 ) . We implement this inhibitory concept in soft actor-critic ( SAC ) algorithm ( Haarnoja et al. , 2018a ) by using an additional value network ( inhibitory ) , as opposed to retraining the previously learned value network ( ongoing ) , to learn the new skill evaluation . Additionally , we propose the use of an inhibitory policy network to control inhibition . We call this SAC with inhibitory networks approach SAC-I . SAC-I is distinct from previous value composition work ( Haarnoja et al. , 2017 ; Van Niekerk et al. , 2019 ) , since it proposes a specific mechanism to train and compose value networks and generate a single policy network focused on fast and improved retraining . We hypothesize that retraining of RL agents can be accelerated by creating independent and mutually inhibitory evaluative processes that will change the reward function used during learning in a state dependent manner . SAC provides two important features for the SAC-I : a replay buffer that can be partitioned into episodic memories related to each evaluative-learning process ( Botvinick et al. , 2019 ) and an automated entropy estimation ( Haarnoja et al. , 2018b ) , which allows computing two separate temperature parameters and exploring actions differently . There are two main contributions in this work . First , we develop the SAC-I architecture for accelerated retraining , that encompasses the use of inhibitory networks for the control of multiple evaluative networks . This approach modifies SAC methods by separating the learning process , which includes training multiple value functions , storing episodic replay buffers , estimating distinct temperature parameters , and learning an inhibition policy when necessary ( Section 3 ) . Second , we provide a detailed validation showing the different components of SAC-I and its improvements over SAC in two modified environments from OpenAI Gym ( Section 4 ) . The LunarLanderContinuous-v2 with a bomb appearing randomly resembles the classic stop-signal paradigm ( Logan et al. , 2015 ; Verbruggen & Logan , 2009 ) in inhibitory control studies . A mixed version of the BipedalWalkerHardcore-v3 , highlights the out-performance of SAC-I over SAC , which is not able to successfully solve the task . 2 BACKGROUND AND RELATED WORK . 2.1 MAXIMUM ENTROPY REINFORCEMENT LEARNING . In RL , knowing the best way to explore while exploiting is non-trivial , environment-dependent , and still an active area of research ( Hong et al. , 2018 ) . Maximum entropy RL theory provides a principled way to address this particular challenge , and has been a key element in many of the recent RL advancements , providing improved exploration and faster learning ( Thomas , 2014 ; Schulman et al. , 2017a ; Haarnoja et al. , 2017 ; 2018a ; b ; Ziebart , 2010 ) . Given a Markov decision process ( MDP ) with a set of states S , a set of actions A , a transition function T and a reward function R , forming a tuple < S , A , T , R > ( Puterman , 1994 ) , a stochastic policy π : S → A is a mapping from states to probabilities of selecting each possible action , where π ( a|s ) represents the probability of choosing action a given state s. In maximum entropy RL , as in an MDP , the goal is to find the optimal policy π∗ that provides the highest expected sum of rewards , while additionally maximizing the entropy of each visited state , leading to the expression ( Ziebart , 2010 ) : π∗ = arg max π ∑ t E ( st , at ) ∼ρπ [ rt + αH ( π ( · |st ) ) ] , ( 1 ) where α is the temperature parameter that controls the stochasticity of the optimal policy , ρπ is the state-action marginal of the trajectory distribution induced by the policy , rt is a shorthand for the environmental reward r ( st , at ) at time t , and H ( π ) represents the entropy of the policy , E [ −log ( π ( a|s ) ) ] . This approach allows a state-wise balance between exploitation and exploration . For states with high reward , a low entropy policy is permitted while , for states with low reward , high entropy policies are preferred , leading to greater exploration . The discount factor γ is omitted in the equation for simplicity since it leads to a more complex expression for the maximum entropy case ( Thomas , 2014 ) . But it is required for the convergence of infinite-horizon problems , and it is included in our final algorithm . 2.2 SOFT ACTOR-CRITIC . Soft actor-critic ( SAC ) ( Haarnoja et al. , 2018a ) is one of the most successful maximum entropy RL methods and has become a common baseline algorithm in most of the RL libraries , outperforming state-of-the-art methods Haarnoja et al . ( 2018a ; b ) . Like the deep deterministic policy gradient ( DDPG ) approach ( Lillicrap et al. , 2016 ) , SAC is a model-free and off-policy method , using a replay buffer , where the policy and value functions are approximated using neural networks . In addition , it incorporates a policy entropy term into the objective function facilitating exploration , similar to soft Q-learning ( Haarnoja et al. , 2017 ) . Similar to trust region policy optimization ( TRPO ) ( Schulman et al. , 2015 ) and proximal policy optimization ( PPO ) Schulman et al . ( 2017b ) , SAC uses a stochastic policy and is known to be more stable than DDPG . In short , SAC combines the best of DDPG ( sample efficiency ) and TRPO/PPO ( stability through stochastic policies ) . As expressed in Equation 1 , the SAC policy/actor is trained with the objective of maximizing the expected cumulative reward and the action entropy at a particular state . The critic is the soft Q-function and , following the Bellman equation , is expressed by : Q ( st , at ) = rt + γ Est+1∼p [ V ( st+1 ) ] , where p represents the state transition probability , and the soft value function is parameterized by the Qfunction : V ( st+1 ) = Eat∼π [ Q ( st+1 , at+1 ) −α log π ( at+1|st+1 ) ] . The soft Q-function is trained to minimize the following objective function given by the mean squared error between predicted and observed state-action values : JQ = E ( st , at ) ∼D [ 1 2 ( Q ( st , at ) − ( rt + γ Est+1∼p [ V̄ ( st+1 ) ] ) ) 2 ] , ( 2 ) whereD denotes the replay buffer , and V̄ is the target value function ( Mnih et al. , 2015 ) . Finally , the policy is updated to minimize the KL-divergence between the policy and the exponentiated stateaction value function ( Haarnoja et al. , 2018b ) , and can be expressed by : Jπ = Est∼D [ Eat∼π [ α log π ( at|st ) −Q ( st , at ) ] ] . ( 3 ) 2.3 INHIBITORY CONTROL Inhibitory control , also known as response inhibition , is a critical component of the executive functions and refers to the ability to modify ongoing actions in response to unexpected and dynamically changing task demands ( Aron , 2007 ; Diamond , 2013 ) . In Shenoy et al . ( 2011 ) , inhibitory control is formalized as a rational decision-making problem , and a computational model using Bayesian inference and stochastic control tools is proposed and validated by behavioral data from humans and animals . Using a widely adopted paradigm known as the stop-signal task ( Logan et al. , 2015 ) , authors show that the optimal policy , whether to go or stop , systematically depends on accumulating sensory evidence , which supports the hypothesis that the brain is implementing statistically optimal decision-making ( Shenoy et al. , 2011 ) . Figure 1 depicts the two pro- cesses involved in the stop-signal task . The Go process starts with a go signal followed by a stop signal which triggers the stop process . The stop process will dominate when its activation is larger than the Go process activation . The key assumption is that both processes are stochastically independent , as supported by behavioral studies ( Verbruggen & Logan , 2009 ) . Stop-signal reaction time ( SSRT ) is defined as the time necessary to respond to the stop stimulus . In further work using functional MRI , the anterior cingulate cortex , a region in the brain implicated in a variety of cognitive control functions , is shown to activate proportionally to a Bayesian prediction error between predicted and observed events resembling the temporal-difference methods in reinforcement learning ( Ide et al. , 2013 ) . These previous computational models indicate that when dealing with unexpected events such as the stop signal the brain implements a dual-process model driven by prediction error and responds to the conflict in an optimal way .

Summary: The authors present SAC-I, which is a modified version of SAC designed for retraining of some existing agent on an updated environment or task. Inspired by the concept of inhibitory control from neuroscience, they learn a separate Q functions and optionally a policy for inhibition. They define some inhibitory reward $r_I$ for each environment to enable it, and make comparison with SAC on modified versions of LunarLanderContinuous and BipedalWalkerHardcore-v3.