Soft Actor-Critic with Inhibitory Networks for Faster Retraining

1 INTRODUCTION . In reinforcement learning ( RL ) , for all but the simplest tasks , agents behavior must be optimized with respect to multiple goals and constraints ( Sutton & Barto , 2018 ) . It is common in practice for new objectives and constraints to arise after already having trained an agent on existing ones . In order for the behavior of the agent to account for these additional constraints , retraining is required . Within the context of deep RL , retraining an agent to new constraints gives problems in balancing exploitation of previously learned skills with learning new skills . In this paper , we present a method for efficiently retraining an agent to acquire new skills in a similar environment . Existing approaches to solving the problem of learning new skills include the use of hierarchical RL structures ( Dayan & Hinton , 1993 ; Barto & Mahadevan , 2003 ) , such as the options framework ( Sutton et al. , 1999 ; Comanici & Precup , 2010 ) , universal value functions ( Schaul et al. , 2015 ) , option-critic ( Bacon et al. , 2017 ) , FeUdal networks ( Vezhnevets et al. , 2017 ) , and data-efficient hierarchical RL ( Nachum et al. , 2018 ) . Other methods use multiple policies and value functions each of which are optimized for simple objectives that can then be composed to achieve complex objectives ( Van Seijen et al. , 2017 ; Sahni et al. , 2017 ; Haarnoja et al. , 2017 ; Hansen et al. , 2020 ; Barreto et al. , 2020 ) . In this paper we propose to address the problem through the use of multiple value functions to provide a complex evaluative input to a single policy network . By applying different value functions in a state dependent fashion , the reward provided to the policy network during training can remain the same as in prior training when appropriate , and can switch to a different reward when the situation indicates new constraints or goals . The mechanism we propose to govern the output of the composite value function is based upon neuroscientific research on inhibitory control ( Diamond , 2013 ) . The brain uses inhibition to interrupt ongoing goal-directed action when unexpected events or conflicts arise . The horse-race model of Logan et al . ( 2015 ) , which describes the behavioral response to a goal and its inhibition in terms of dual processes , is supported by many behavioral and neurobiological studies ( Verbruggen & Logan , 2009 ; Shenoy et al. , 2011 ; Ide et al. , 2013 ; Schall et al. , 2017 ) . We implement this inhibitory concept in soft actor-critic ( SAC ) algorithm ( Haarnoja et al. , 2018a ) by using an additional value network ( inhibitory ) , as opposed to retraining the previously learned value network ( ongoing ) , to learn the new skill evaluation . Additionally , we propose the use of an inhibitory policy network to control inhibition . We call this SAC with inhibitory networks approach SAC-I . SAC-I is distinct from previous value composition work ( Haarnoja et al. , 2017 ; Van Niekerk et al. , 2019 ) , since it proposes a specific mechanism to train and compose value networks and generate a single policy network focused on fast and improved retraining . We hypothesize that retraining of RL agents can be accelerated by creating independent and mutually inhibitory evaluative processes that will change the reward function used during learning in a state dependent manner . SAC provides two important features for the SAC-I : a replay buffer that can be partitioned into episodic memories related to each evaluative-learning process ( Botvinick et al. , 2019 ) and an automated entropy estimation ( Haarnoja et al. , 2018b ) , which allows computing two separate temperature parameters and exploring actions differently . There are two main contributions in this work . First , we develop the SAC-I architecture for accelerated retraining , that encompasses the use of inhibitory networks for the control of multiple evaluative networks . This approach modifies SAC methods by separating the learning process , which includes training multiple value functions , storing episodic replay buffers , estimating distinct temperature parameters , and learning an inhibition policy when necessary ( Section 3 ) . Second , we provide a detailed validation showing the different components of SAC-I and its improvements over SAC in two modified environments from OpenAI Gym ( Section 4 ) . The LunarLanderContinuous-v2 with a bomb appearing randomly resembles the classic stop-signal paradigm ( Logan et al. , 2015 ; Verbruggen & Logan , 2009 ) in inhibitory control studies . A mixed version of the BipedalWalkerHardcore-v3 , highlights the out-performance of SAC-I over SAC , which is not able to successfully solve the task . 2 BACKGROUND AND RELATED WORK . 2.1 MAXIMUM ENTROPY REINFORCEMENT LEARNING . In RL , knowing the best way to explore while exploiting is non-trivial , environment-dependent , and still an active area of research ( Hong et al. , 2018 ) . Maximum entropy RL theory provides a principled way to address this particular challenge , and has been a key element in many of the recent RL advancements , providing improved exploration and faster learning ( Thomas , 2014 ; Schulman et al. , 2017a ; Haarnoja et al. , 2017 ; 2018a ; b ; Ziebart , 2010 ) . Given a Markov decision process ( MDP ) with a set of states S , a set of actions A , a transition function T and a reward function R , forming a tuple < S , A , T , R > ( Puterman , 1994 ) , a stochastic policy π : S → A is a mapping from states to probabilities of selecting each possible action , where π ( a|s ) represents the probability of choosing action a given state s. In maximum entropy RL , as in an MDP , the goal is to find the optimal policy π∗ that provides the highest expected sum of rewards , while additionally maximizing the entropy of each visited state , leading to the expression ( Ziebart , 2010 ) : π∗ = arg max π ∑ t E ( st , at ) ∼ρπ [ rt + αH ( π ( · |st ) ) ] , ( 1 ) where α is the temperature parameter that controls the stochasticity of the optimal policy , ρπ is the state-action marginal of the trajectory distribution induced by the policy , rt is a shorthand for the environmental reward r ( st , at ) at time t , and H ( π ) represents the entropy of the policy , E [ −log ( π ( a|s ) ) ] . This approach allows a state-wise balance between exploitation and exploration . For states with high reward , a low entropy policy is permitted while , for states with low reward , high entropy policies are preferred , leading to greater exploration . The discount factor γ is omitted in the equation for simplicity since it leads to a more complex expression for the maximum entropy case ( Thomas , 2014 ) . But it is required for the convergence of infinite-horizon problems , and it is included in our final algorithm . 2.2 SOFT ACTOR-CRITIC . Soft actor-critic ( SAC ) ( Haarnoja et al. , 2018a ) is one of the most successful maximum entropy RL methods and has become a common baseline algorithm in most of the RL libraries , outperforming state-of-the-art methods Haarnoja et al . ( 2018a ; b ) . Like the deep deterministic policy gradient ( DDPG ) approach ( Lillicrap et al. , 2016 ) , SAC is a model-free and off-policy method , using a replay buffer , where the policy and value functions are approximated using neural networks . In addition , it incorporates a policy entropy term into the objective function facilitating exploration , similar to soft Q-learning ( Haarnoja et al. , 2017 ) . Similar to trust region policy optimization ( TRPO ) ( Schulman et al. , 2015 ) and proximal policy optimization ( PPO ) Schulman et al . ( 2017b ) , SAC uses a stochastic policy and is known to be more stable than DDPG . In short , SAC combines the best of DDPG ( sample efficiency ) and TRPO/PPO ( stability through stochastic policies ) . As expressed in Equation 1 , the SAC policy/actor is trained with the objective of maximizing the expected cumulative reward and the action entropy at a particular state . The critic is the soft Q-function and , following the Bellman equation , is expressed by : Q ( st , at ) = rt + γ Est+1∼p [ V ( st+1 ) ] , where p represents the state transition probability , and the soft value function is parameterized by the Qfunction : V ( st+1 ) = Eat∼π [ Q ( st+1 , at+1 ) −α log π ( at+1|st+1 ) ] . The soft Q-function is trained to minimize the following objective function given by the mean squared error between predicted and observed state-action values : JQ = E ( st , at ) ∼D [ 1 2 ( Q ( st , at ) − ( rt + γ Est+1∼p [ V̄ ( st+1 ) ] ) ) 2 ] , ( 2 ) whereD denotes the replay buffer , and V̄ is the target value function ( Mnih et al. , 2015 ) . Finally , the policy is updated to minimize the KL-divergence between the policy and the exponentiated stateaction value function ( Haarnoja et al. , 2018b ) , and can be expressed by : Jπ = Est∼D [ Eat∼π [ α log π ( at|st ) −Q ( st , at ) ] ] . ( 3 ) 2.3 INHIBITORY CONTROL Inhibitory control , also known as response inhibition , is a critical component of the executive functions and refers to the ability to modify ongoing actions in response to unexpected and dynamically changing task demands ( Aron , 2007 ; Diamond , 2013 ) . In Shenoy et al . ( 2011 ) , inhibitory control is formalized as a rational decision-making problem , and a computational model using Bayesian inference and stochastic control tools is proposed and validated by behavioral data from humans and animals . Using a widely adopted paradigm known as the stop-signal task ( Logan et al. , 2015 ) , authors show that the optimal policy , whether to go or stop , systematically depends on accumulating sensory evidence , which supports the hypothesis that the brain is implementing statistically optimal decision-making ( Shenoy et al. , 2011 ) . Figure 1 depicts the two pro- cesses involved in the stop-signal task . The Go process starts with a go signal followed by a stop signal which triggers the stop process . The stop process will dominate when its activation is larger than the Go process activation . The key assumption is that both processes are stochastically independent , as supported by behavioral studies ( Verbruggen & Logan , 2009 ) . Stop-signal reaction time ( SSRT ) is defined as the time necessary to respond to the stop stimulus . In further work using functional MRI , the anterior cingulate cortex , a region in the brain implicated in a variety of cognitive control functions , is shown to activate proportionally to a Bayesian prediction error between predicted and observed events resembling the temporal-difference methods in reinforcement learning ( Ide et al. , 2013 ) . These previous computational models indicate that when dealing with unexpected events such as the stop signal the brain implements a dual-process model driven by prediction error and responds to the conflict in an optimal way .

Summary: This paper introduces an agent architecture with two independent policy networks (called ‘go’ and ‘stop’) and an assignment mechanism consisting of a network or a rule that activates one of the two every step. The two policy networks are in principle both soft actor-critic (SAC) agents, although the authors explore several variations on the degree of independence between them, with e.g. hyperparameters being tied in some cases. Each policy network is trained on replayed data from the subset of states that has been assigned to it by the assignment mechanism.