Soft Actor-Critic with Inhibitory Networks for Faster Retraining
Authors: Jaime S. Ide
Daria Micovic
Adrian P Pope
Michael John Guarino
Kevin Alcedo
David Rosenbluth
Venue: ICLR 2022 Submitted (2022)
Keywords: soft actor-critic
SAC
maximum entropy RL
inhibitory response
cognitive control

Abstract: Reusing previously trained models is critical in deep reinforcement learning to speed up training of new agents. However, it is unclear how to acquire new skills when objectives and constraints are in conflict with previously learned skills. Moreover, when retraining, there is an intrinsic conflict between exploiting what has already been learned and exploring new skills. In soft actor-critic (SAC) methods, a temperature parameter can be dynamically adjusted to weight the action entropy and balance the explore $\times$ exploit trade-off. However, controlling a single coefficient can be challenging within the context of retraining, even more so when goals are contradictory. In this work, inspired by neuroscience research, we propose a novel approach using inhibitory networks to allow separate and adaptive state value evaluations, as well as distinct automatic entropy tuning. Ultimately, our approach allows for controlling inhibition to handle conflict between exploiting less risky, acquired behaviors and exploring novel ones to overcome more challenging tasks. We validate our method through experiments in OpenAI Gym environments. 

Under review as a conference paper at ICLR 2022
SOFT ACTOR-CRITIC WITH INHIBITORY NETWORKS FOR FASTER RETRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Reusing previously trained models is critical in deep reinforcement learning to speed up training of new agents. However, is unclear how to acquire new skills when objectives and constraints are in conflict with previously learned skills. Moreover, when retraining, there is an intrinsic conflict between exploiting what has already been learned and exploring new skills. In soft actor-critic (SAC) meth- ods, a temperature parameter can be dynamically adjusted to weight the action en- tropy and balance the explore X exploit trade-off. However, controlling single coefficient can be challenging within the context of retraining, even more so when goals are contradictory. In this work, inspired by neuroscience research, pro- pose a novel approach using inhibitory networks to allow separate and adaptive state value evaluations, well distinct automatic entropy tuning. Ultimately, our approach allows for controlling inhibition to handle conflict between exploit- ing less risky, acquired behaviors and exploring novel ones to overcome more challenging tasks. We validate our method through experiments in OpenAl Gym environments.
INTRODUCTION
In reinforcement learning (RL), for all but the simplest tasks, agents behavior must be optimized with respect to multiple goals and constraints (Sutton & Barto 2018). It is common in practice for new objectives and constraints to arise after already having trained an agent on existing ones. In order for the behavior of the agent to account for these additional constraints, retraining is required. Within the context of deep RI., retraining an agent new constraints gives problems balaneing exploitation of previously learned skills with learning new skills. In this paper, we present method for efficiently retraining an agent to acquire new skills in a similar environment.
Existing approaches to solving the problem of learning new skills include the use of hierarchical RL structures (Dayan Hinton 1993 Barto & Mahadevan 2003), such as the options framework (Sutton et al. 1999| Comanici & Precup 2010). universal value functions (Schaul al., 2015),
option-critic (Bacon et al.] (2017). FeUdal networks (Vezhnevets et al., 2017). and data-efficient hierarchical RI. (Nachum al. 2018). Other methods multiple policies and value functions each of which are optimized for simple objectives that can then be composed to achieve complex objectives (Van Seijen et al., 2017 Sahni et al. 2017 Haarnoja al.) [017]] Hansen et al. 2020)] Barreto et al. 2020). In this paper we propose to address the problem through the use of multiple value functions to provide complex evaluative input to single policy network. By applying different value functions in a state dependent fashion, the reward provided to the policy network during training can remain the same as prior training when appropriate, and can switch different reward when the situation indicates new constraints or goals.
The mechanism we propose to govern the output of the composite value function is based upon neuroscientific research on inhibitory control (Diamond 20133. The brain uses inhibition to interrupt ongoing goal-directed action when unexpected events or conflicts arise. The horse-race model of Logan et al. (2015), which describes the behavioral response to a goal and its inhibition in terms of dual processes, is supported by many behavioral and neurobiological studies /Verbruggen & Logan 2009: Shenoy et al. 2011 TolT:lak Ide et al. 2013 Schall et al. (017)) We implement this inhibitory concept in soft actor-critic (SAC) algorithm (Haarnoja et al. 2018a) by using an additional value network (inhibitory), as opposed retraining the previously learned value network (ongoing). to

---

Under review as a conference paper at ICLR 2022
learn the new skill evaluation. Additionally, propose the use of inhibitory policy network to control inhibition. We call this SAC with inhibitory networks approach SAC-I. SAC-I is distinct from previous value composition work (Haamoja et al. 2017: Van Niekerk et al.) 2019p, since it proposes a specific mechanism to train and compose value networks and generate a single policy network focused fast and improved retraining. We hypothesize that retraining of RL agents can be accelerated by creating independent and mutually inhibitory evaluative processes that will change the reward function used during learning in a state dependent manner. SAC provides two important features for the SAC-I: a replay buffer that can be partitioned into episodic memories related to each celluative-eeamingg process (Botvinick et al. 2019) and an automated entropy estimation (Haarnojal er al. 2018b), which allows computing two separate temperature parameters and exploring actions differently.
There are two main contributions in this work. First, we develop the SAC-I architecture for accelerated retraining, that encompasses the use of inhibitory networks for the control of multi- ple evaluative networks. This approach modifies SAC methods by separating the learning pro- cess, which includes training multiple value functions, storing episodic replay buffers, estimat- ing distinct temperature parameters, and learning inhibition policy when necessary (Section B). Second, we provide a detailed validation showing the different components of SAC-I and its improvements over SAC in modified environments from OpenAl Gym (Section 4. The Lunarl.anderCom aanderContinouusss
with a bomb appearing randomly resembles the classic stop-signal paradigm (Logan et al., 2015; Verbruggen & Logan 2009) in inhibitory control studies. A mixed version of the Bipedal SeopedalWalkeer WalkerHardeore. which is not able successfully solve the task.
highlights the out-performance of SAC-I over SAC,
BACKGROUND AND RELATED WORK
2.1 MAXIMUM ENTROPY REINFORCEMENT LEARNING
In RL. knowing the best way to explore while exploiting is non-trivial, environment-dependdent and still an active area of research (Hong al., 2018). Maximum entropy RL theory provides principled way to address this particular challenge, and has been a key element in many of the recent RL advancements, providing improved exploration and faster learning (Thomas 2014 Schulman iet al. 2017a; Haarnoja et al. 2017: 2018a|b) Ziebart 2010). Given Markov decision process (MDP) with a set of states S, set actions A. transition function T and a reward function R, forming tuple < S, A. R > (Puterman 1994), stochastic policy π S - A is mapping from states to probabilities of selecting each possible action, where π (a]ss) represents the probability of choosing action a given state In maximum entropy RL, as in MDP, the goal is to find the optimal policy n² that provides the highest expected sum of rewards, while additionally maximizing the entropy of each visited state, leading the expression (Ziebart 2010):
arg max Σ E (ss [ra + (H((T(( (s:)))
where is the temperature parameter that controls the stochasticity of the optimal policy, Prr is the state-action marginal of the trajectory distribution induced by the policy, rt is a shorthand for the environmental reward r(se, 0t) time f, and H(T)) represents the entropy of the policy, [l- This approach allows state-wise balance between exploitation and exploration. For states with high reward, low entropy policy permitted while, for states with low reward, high entropy policies are preferred, leading to greater exploration. The discount factor 7 is omitted in the equation for simplicity since it leads more complex expression for the maximum entropy case (Thomas) 2014). But it is required for the convergence of infinite-horizon problems, and it is included our final algorithm.
2.2 SOFT ACTOR- -CIITI
Soft actor-critic (SAC) (Haarnoja al.) 2018a) is one of the most successful maximum entropy RL methods and has become a common baseline algorithm in most of the RL libraries, outperform- ing otate-of-the-art methods Haarnoja al. (2018 Sa b). Like the deep deterministic policy gradient (DDPG) approach (Lillicrap et al. SAC is model-free free and off off-po method, using a replay buffer, where the policy and value functions are approximated using neural networks. In addition,
2

---

Under review as a conference paper at ICLR 2022
it incorporates a policy entropy term into the objective function facilitating exploration, similar to soft Q-learning (Haarnoja et al., 2017). Similar to trust region policy optimization (TRPO) (Schul- man et al. 2015) and proximal policy optimization (PPO) [Schulman et al. (017b). SAC uses stochastic policy and is known to be more stable than DDPG. In short, SAC combines the best of DDPG (sample efficiency) and TRPO/PPO (stability through stochastic policies). As expressed in Equation D the SAC policy/actor is trained with the objective of maximizing the expected cu- mulative reward and the action entropy at a particular state. The critic is the soft Q-function and, following the Bellman equation, is expressed by: Q(sr, a.) = + y Be.ts-pllV (st+1)), where represents the state transition probability, and the soft value function is parameterized by the Q- function: (s+++)) En.~- (Q( (8+11) a log π (ax+1 | The soft Q-function is trained to minimize the following objective function given by the mean squared error between predicted and observed state-action values:
Jq B (Q((), ai) - (re + Be V ((L+)))))
(2)
where DD denotes the replay buffer, and V is the target value function (Mnih al. 2015). Finally, the policy is updated to minimize the KI. Kl.-divergence between the policy and the exponentiated state- action value function (Haarnoja et al. 2018b). and can be expressed by: Bare aa log T (a))ss)) ((ss.:a))]
Jx = E D
(3)
2.3 INIIBITORY CONTROL
Inhibitory control, also known as response inhibition, is critical compo- nent of the executive functions and refers to the ability to modify ongo- ing actions in response to unexpected and dynamically changing task de- mands (Aron] 2007 Diamond 2013). In [Shenoy et al. (201T). inhibitory control is formalized as a rational periiioonmmainn problem, and a com- putational model using Bayesian inference and stochastic control tools is proposed and validated by behavioral data from humans and animals. Using widely adopted paradigm known the stop-signal task (Logan iet al. 2215),, authors show that the optimal policy, whether to go or stop, systematically depends on accumulating sensory evidence, which sup- ports the hypothesis that the brain is implementing statistically optimal depicts the two pro- cesses involved in the stop-signal task. The Go process starts with go signal followed by stop signal which triggers the stop process. The stop process will dominate when its activation is larger than the Go process activation. The key assumption is that both processes are stochastically inde- pendent, as supported by behavioral studies (Verbruggen & Logan 2009). Stop-signal reaction time (SSRT) is defined the time necessary to respond to the stop stimulus. In further work using func- tional MRI, the anterior cingulate cortex, a region in the brain implicated in a variety of cognitive control functions, is shown to activate proportionally to Bayesian prediction error between pre- dicted and observed events resembling the temporal-difference methods in reinforcement learning (Ide al., 2013). These previous computational models indicate that when dealing with unexpected events such as the stop signal the brain implements a dual-process model driven by prediction error and responds the conflict in an optimal way.
activation
Stop
SSRT
Go
Stop Signal
Figure 1:
Dual process
model in the stop-signal task (SST).
recision-making (Shenoy et al., 2011). Figure [
2.4 RELATED WORK
Value function composition. In value function composition (Haarnoja et al. 2017 Van Niekerk al. 2019). the goal is to model new task by composing value functions previously trained on sub-tasks. In Todorov (2009), they show that for linearly solvable MDPs (Todorov) 2007), pre- trained value function estimators can be optimally composed and solved. The composition is union of tasks, an "OR" composition, and is defined by taking the softmax over the component reward signals. Van Niekerk et al. (2019) extends this result to the standard and entropy regularized RL settings. Haarnoja et al. (017) defines composition rule that approximately solves the intersection of tasks in the entropy regularized setting, an "AND" composition, with the composed signal as an average over the constituent rewards. Despite the similarities, different since composes a previously learned value function with one which is newly trained, and moreover does not involve combining their value estimates.
approach is fundamentally
3

---

Under review as a conference paper at ICLR 2022
Hybrid reward architecture. In Hybrid Reward Architecture (HRA) (Van Seijen al. 2017), the goal is to learn a complex task by decomposing its reward, and training separate value functions for each component. The total reward is replaced by an equivalent representation the sum, "AND" composition, of decomposed constituett rewards. They show that HRA learns more efficiently than the deep Q-network algorithm (DQN) (Mnih al.] 2015) when both algorithms have otherwise identical network architectures. SAC-I is similar to HRA in the training of multiple Q networks using different rewards, however essentially different since does not aim to combine the reward values. In our approach, Q networks are trained independently and used to provide specialized action values depending on the state, which is defined by the inhibition rule.
Transfer learning in RL. Broadly speaking, transfer learning in RL consists of transferring the knowledge gained in one task to improve the learning performance in a related, but different task Taylor& & Stone, 2009; Lazaric 2012). This knowledge can be some type of learned representation (Rusu et al.J 2016b), reward shaping (Brys et al.] (013), demonstration (Schaal 1996). model dy- namics (Ammar et al. 2012), policy (Rusu al. 2016a). Our work is within the large field of transfer learning, however we are primarily focused on transferring learned value and policy func- tions among identical aspects of task, while learning new skills (value functions) and retraining the previously learned policy within the similar environment.
3 METHODS
3.1 SAC-I: SAC WITH INHIBITORY NETWORKS
Inhibitory control is traditionally defined as the ability to stop ongoing or planned cognitive or motor processes, overriding impulsive ha- bitual responses (Aron) 2007). One possible RL implementation is at the executive level by switching between two policies using a hierar- chical architecture. However, this approach re- quires having to train multiple policies, and more complex hierarchical model. We propose a computationally less expensive approach and
Episodic Memory
Evaluative System
Go networks
Stop networks
Replay Butter
Rapley Buffer- -R
(,(Aa)
Q,(a.)
(_((A.)
min(q, 9.)
minie., 9..)
Policyts)
Inhibitory
Policy
Executive System
update
Figure 2: SAC-I policy update.
do the switching at the Q network level, since will ultimately drive the change of the habitual re- sponse action. We implement inhibitory control in this broader sense at a higher cognitive level, where the action execution (policy) is updated by the evaluative processes that precede execution.
We apply inhibitory control to the SAC algorithm by having multiple, competing value functions (critics) that take turns depending on the current task demand. Rather than having a Q network that learns the Go task and later the Stop task, we keep the previous Q network that knows to evaluate the Go task and train a new Q-I network that learns the Stop task. Thus, both Q networks are trained independently. Here, the term "inhibitory" refers to new evaluative process replacing ongoing one. In Figure we depict the SAC policy update with inhibitory networks. Go networks estimate ongoing evaluation (which led to pre-trained skills), while the Stop networks estimate the value of the new event, leading the policy to learn new skills. Additionally, we introduce the use of an in- hibitory policy network to learn the inhibition policy responsible for deciding how and when to use the outcomes from the multiple Q networks. Alternatively, to avoid having to train an additional policy, a parametric stste- inhibition rule can be implemented using knowledge about the en- vironment or updated constraints. We use Q networks to estimate all the value functions, following the implementation in Haarnoja et al. (2018b), and the clipped double-Q learning trick, introduced in twin delayed DDPG (TD3) (Fujimoto et al. 2018) to avoid their overestimation.
3.2 EPISODIC MEMORY THROUGH SELECTIVE REPLAY BUFFER
In neuroscience research, episodic memory refers to the brain's 's ability to recollect past experiences, and is an important component of learning (Tulving) 2002). In recent years, it has been applied to DRL as a normaaraet framework to retrieve past successful experiences to improve sample efficiency (Blundell et al. 2016 Lin et al.) 2018 Botvimick al. 2019) to avoid catastrophic forgetting (sell & Cosgun 2018). In this work, we implement episodic memory in its simplest form; partition of the state space according to inhibition rule or policy that yields partition
4

---

Under review as a conference paper at ICLR 2022
of the replay buffer. Let S = (Sp.S) be partition of the state space. The corresponding replay buffer partition is D = {DR, } where set membership of a tuple (#k, a, rk,, s') is parameterized by 8g for k E {R, [] where sn E Sn and S, Sy. For case of notation, s' = St+1+ and the next state can belong to any of the partitions, s' E S. Note that inhibitory states have an associated inhibitory reward r'y and regular states have the associated reward rn- These corresponding rewards are stored in the replay buffers but the replay buffers are not parameterized by them. The loss function of each Q network is computed using memories from the corresponding replay buffer as expressed by:
Jq. ((,a))>D 2
(Qu(s.a) (rk + 7 B. Exrp (V
for k IR
(4)
single replay buffer that contains both the regular and the inhibitory wards (s, a, Tn. TI, F1, s'), and sample a tuple containing the regular (s, o, Tn, O, s') or the inhibitory (s, a, 0, rr, s') rewards depending on the updated Q network, we would have sparser rewards partie- ularly for the Q-I network. Therefore, having separate memories promotes faster learning (as shown in Figure 4 and, importantly, also allows the estimation of separate entropy parameters.
Notice that if
3.3 AUTOMATED DUAL ENTROPY ESTIMATION
The SAC algorithm is sensitive to the a temperature changes depending on the environment, reward scale and training stage, as shown in the initial paper (Haarnoja et al.) 2018a). To address this issue, the same authors propose to automatically adjust the temperature parameter by formulating the problem with a dual objective; maximize entropy while satisfying a minimum entropy constraint (Haarnoja al. 2018b). The goal is still maximizing the cumulative expected reward (Equation[] [ but the average entropy of the policy is now constrained by a minimum value. The full derivation of the dual optimization problem is given in Haarnoja et al. (2018b). In practice, the optimization is performed recursively as follows: at time f, given the current estimate ag, the optimal policy क estimated described in Equation I Subsequently, given Ty, the a is approximated using neural network. While in the standard SAC, there is single a parameter, in SAC-I, we propose estimating two separate a temperature parameters, a and a,, allow distinct action entropy for the previously learned and the new skills, respectively. This is implemented by training two separate a networks, with regular states 8R SR distinguished from inhibitory states 87 E S₁, with losses given by:
Jas = Eawal I- log (aa sx) - az Hol, for l E {R. where Ho is the minimum expected entropy. The policy loss function is composed of two terms:
(5)
J== Σ IE(R.I)
Ee Sn [Ea fax log (a)ss) Q& (sk.))))]
3.4 INHIBITORY POLICY NETWORK
In our proposed SAC-I algorithm (Figure 2 for the cases in which a state-dependent inhibition rule is not defined, we propose training inhibitory policy network my. This network can be trained as an automated hard switch or a soft modulator between the regular and inhibitory Q networks. The inhibitory policy network is a stand-alone agent with its own Q networks (not shown in Figure 2). however it shares the same replay buffers. The loss functions are the standard ones as defined in Equations 2 and 3 Ultimately, the goal of the inhibitory policy network to maximize the environment' s reward by learning to choose between the Go and Stop networks and/or by modulating the Stop network. We show the implementation of both cases next Section 4
SAC-I is summarized in Algorithm 1.
4 EXPERIMENTS AND RESULTS
In order to show two different use-cases of SAC-I algorithm, as well to evaluate it as way to speed up training during transfer learning, we use continuous tasks from the Box2D simulator, OpenAl Gym (Brockman et al. 20166). LumarLanderrontiiuuus-- v2 and Bipedal HipedalWWlkerHHrddoreeww. Importantly, we include custom modifications to them to emulate the scenarios in which retraining is necessary. By adding random bomb in the LunarL.ander task, we show the advantages of using
5

---

Under review as a conference paper at ICLR 2022
Algorithm 1: Soft Actor Critic with Inhibitory Networks (SAC-I)
Initialize Qny Ra Qi Q2 policy n. policy T,, on and as networks parameters; Initialize the target RR, RRa Q1, and Q1, networks weights; Initialize the replay bufters DR and Dy; for each episode do
for each environment step do
Given 8t, sample oe from (8)) and (s1+1, vi) from the environment; Use an inhibition rule or inhibitory policy my classify st: if st E SR. push (st, or, RR St+1) to DR;
else if 8, E S₁ push (St, a, [1,,8141 to Df; end
for each gradient step do Sample batch of memories from from (Dn. D1}; for k E {R. I do
Update Qs, and Qka (Equation 4h. ag (Equation 5). and Qs, and Qa (soft-update); end
Update the policy network a (Equation 6
If an inhibitory policy network is used, update ny and the associated Q networks;
end end
SAC-I, compared to standard SAC, when retraining with conflicting goals, similar to what happens in a stop-signal task (Logan et al., 2015). i.e. stopping an ongoing action (to land) whenever an unexpected event occurs (to avoid bomb). In the experiments with SipedalWWlkerHarddoree train agents in simpler version of the task (BipedalWaller-v3), and retrain them in the more complex task. We show how SAC-I can help transfer learning and adjust inhibitory control. In all the experiments provided in this section, used five random seeds to account for the variability during training and compare agents trained across a fixed number of steps. All the hyperparameters are available Table (Appendix).
4.1 LUVARLANDER WITH BOMB
Environment. The original version of Lunarl LanderContinuous--2 is modified order to include a bomb that appears randomly within a region above the landing pad (Figure Like the original version, includes the environment reward (moving from the top of the frame to the landing pad: 100-140 points, each leg contact: + +10, crashing: -100, successful landing: +100, firing engine: -0.3 per frame), but additionally it includes a penalty for hitting the bomb (-150) and a time penalty (-0.1 per frame) to motivate landing quickly possible (like in SST). Importantly, the bomb coordinates are included in the observation state only after the bomb appears so the agent does not know about its existence beforehand. Further details can be found in Appendix.
(a) Episode start
(b) Go trial: landing
(c) Stop trial: bomb appears
(d) Stop trial: landing
Figure 3: AaniaaaaaarConiuuuuus/// with Bomb. (a) Episode starts with the lander placed ran-
domly on top of the frame around the center. (b) Go trial: refers to an episode without bomb, and is exactly as the original environment. (c) Stop trial: refers to episodes in which bomb randomly appears close to the landing pad (between the two flags). (d) It shows successful landing during stop trial after avoiding hitting the bomb. By default, the bomb appears in 50% of the episodes.
Experimental design. Initially, standard SAC agent is trained in the original environment aannLaannneCommmuuuus22222 This agent, which we will call "baseline agent", takes about 250K steps to reach average cumulative reward of 200 and is able to successfully land in most of the episodes. All the network weights are transferred to the retrained agent. The Q-I network is trained from scratch. In this experiment, the inhibition rule/policy works switch between the

---

Under review as a conference paper at ICLR 2022
regular and inhibitory Q networks. We show both cases, user-defined inhibition rule and learned inhibitory policy network. Experiments are performed in a single machine with Intel Core 17-9850H CPU@2.60Hz x 12, Quadro RTX3000, RAM 16GB. The average rewards reported in the figures represent the average of the episode reward, including the bomb penalty, over the last 100 episodes. Agents are trained for fixed number of 2K episodes, approximately 500K steps.
Advantage of retraining. In this experiment, we show how the performance of standard SAC agent is boosted by retraining it using weights from the baseline agent, which already knows how to safely land. In order to learn bomb avoidance skills, we shaped the original reward with an additional penalty given by the expression rbomb.prozy
-1e4 x (d, -0.3), where de is defined as the agent's s distance to the center of the bomb (r). yb). The idea is to have a field-type avoidance mechanism and give a penalty proportional bomb proximity. Average reward results are depicted in Figure orange and blue lines. For retraining, all the network weights are loaded from the baseline agent and updated in the new task with bomb. The retrained agent (blue line) clearly learns to complete the task (land and avoid the bomb) faster than the agent trained from scratch (orange line). In about 300K steps, starts to avoid the bomb and at 500K steps is mostly able to complete the task successfully. The agent from scratch takes 1.25M steps to learn the task.
Effect of using episodic memory and dual alpha in SAC-I. We parse out the distinct contributions of different components of SAC-I by training agents with and without episodic mem- ory and the dual temperature parameters. In these versions, we use a user defined inhibition rule given by y > 1/b and do < 0.3. If these conditions are met, the inhibitory Q-I net-
200
Avergee Reward
SAC traind from stretth
work is used with the same shaping used for the SAC agent, ri = Nbomb prozy. Results are shown in Figure 4 The SAC- I vanilla agent (green line) is trained without episodic mem- ory and with a single a parameter. The SAC-I with episodic memory (purple line) is trained with separate experience re- play buffer for the inhibitory states, but uses only a single en- tropy parameter a. The SAC-I agent with episodic memory and dual a (red line) has faster learning, reaching an average reward of 200 after 300K steps, and highlighting the importance of these com- ponents
trannd from busdine
-200
S4C-1 opirad: menary
epasd: namarry dal
Steps
1a5
Figure 4: Comparison of different SAC and SAC-I agents.
SAC-I performance for different conflict levels. The brain's inhibitory control is known to be modulated by levels of conflict between ongoing processes (Braver al., 2001). For instance in the SST, changing the frequency of stop signals, which is associated with expectation, alters the response time to go signals. In a similar way, we investigate whether and how the frequency of stop trials (episodes with bomb) impacts the agent's learning, other than the overall performance which expected to decrease for increased occurrence of bombs. We hypothesize that the bomb frequency will impact SAC-I less than SAC because it trains separate Q networks for different skills involved in the task (i.e. landing and bomb avoidance). Both SAC and SAC-I agents are trained with varying bomb frequencies, and results are shown in Figure
5
As expected, the overall performance
300
Bomb 25%
300
Bomb 50%
Bomb 75%
Average Neward
200
Average Reweee
200
Average Reward
200
100
100
180
-100
1000
SAC SAC-I
-300
-300
0
Steps
1n5
Stros
Steps
Figure 5: Perforrance of SAC-I and SAC agents for different bomb frequencies.
decreases for higher frequency. SAC-I agent reaches an average reward of 200 in about 200K, 300K and 600K for frequencies 25%, 50% and 75%, respectively. For the SAC agent, it is clear that bomb frequency not only affects performance but also the training progression. For example, significant drop in performance is observed around step 50K for Bomb 75% as well as for 50% although less. This likely happens because learning to avoid the bomb interferes with its initial ability to land.
Further SAC-I agents are all trained with episodic memory and dual a parameters estimation.
7

---

Under review as a conference paper at ICLR 2022
Interestingly, although both agent's performances converge asymptotically, the difference is larger for Bomb 50% case, in which the uncertainty is the highest. This is likely because SAC and SAC-I agents learn different strategies to avoid the bomb. While the SAC agent adopts an optimal global strategy accordingly to different bomb frequencies (i.e, if bomb is frequent, slow down when task starts), the SAC-I agent keeps the same landing strategy independent of bomb frequency and learns to avoid the bomb when it appears.
Training without reward shaping. Further, we examined the training of SAC and SAC-I agents without using any reward
200
250
shaping for the bomb avoidance. Both agents use the same reward r
Average Rovare
200
+ Tbomb. where ro is the original reward and Thomb is a sparse penalty for hitting the bomb. Result are shown in Figure(6] Surprisingly, the SAC* agent without shap- ing (yellow line) performed better than the one with, mean- ing that shaping is negatively impacting its training. In con- trast, the SAC-I* agent without shaping (gray line) success- fully learns the task with similar speed to the one with shaping (red line). likely because it keeps separate critic networks to evaluate landing and bomb avoidance. In further experiments,
150
100
50
SAC
shaptng)
-30
SAC
estarottes imsoiton
Figure SAC and SAC-I agents with and without shaping. we observed that different reward shaping can boost SAC-I's performance even more, but not SAC (see results with conservative shaping in Appendix, Figure I0P.
Learning when to inhibit. In this experiment, we show an agent trained with an inhibitory policy network SAC-I## and compare it to an agent trained with an inhibition rule SAC-I* The SAC-I* agent (gray line) uses a simple inhibition rule ("whenever bomb appears") to switch between the regular and inhibitory Q networks (Figure 2] While, the SAC-I** agent (brown line) learns an inhibition policy, i.e. the best timing to inhibit. This SAC-I** agent (brown line) performs as good as the SAC-I# and outperforms the SAC agents (Figure
4.2 BIPEDAL
In this evaluation, show different use of inhibitory networks, control the magnitude of inhi- bition. To create the retraining scenario, first, we train a standard SAC agent in an easier environ- ment BipedalWalker-v3 (Figure 77)) in which the goal is to walk through plain terrain, and use its weights to retrain SAC and SAC-I agents in Bipedal Wanped/WWakerraardooee--- (Figure 7D-d). In the Bipedal task, agents naturally get negative "inhibitory" reward whenever they are stuck. We use SAC-I to learn as well as to weight that negative reward.
(a) Plain terrain
(b) Stuck position
(c) Fall
(d) Overcoming obstacles
BipedalWalker environment. (a) Plain terrain: agent has to learn to walk forward; (b) Stuck position: agent starts to receive negative reward since spending energy without moving forward; (c) Fall: agent can fall because it loses balance, stumbles itself, or fall into a hole; (d)
Figure 7:
Overcoming obstacles: the agent learns avoiding the holes or going over the blocks.
The Bipedal AapedalWllkrHHardooeeey is challenging task, known to be unsolvable for many of the sim- pler non-recurrent DRL architectures or model-free RL methods (Wei Ying. 2021). We solve the task using a standard two-layer dense-network architecture, and adopt two strategies: removing the fell-penalty and creating a cumulative version of the task reward. Otherwise, the task is unsolvable with SAC algorithm (see Figure II in Appendix).
Learning how to inhibit. In Figure g we show the training performance of SAC and two versions of SAC-I agents learning SipedalWalkerHardooee.33 baseline. The SAC agent reward consists of the raw environment reward тo, but with a cumulative reward and without the fall penalty]] expressed by r To Tstuck rfoll-
All agents retrained from
"SAC and SAC-I agents from Figure are also shown to facilitate comparison, using the same color coding. "Mimicking life, harsh fall penalty seems block learning. See Figure 11 in Appendix.
8

---

Under review as a conference paper at ICLR 2022
The cumulative reward is defined by Tstuck In==o re-is ac- counted if it is negative, i.c. the agent is at "stuck" state. The cumulative reward provides a less noisy feedback. The pri- mary goal of SAC-I (standard) is to separately train Q-I critic network specialized
250
200
150
modeling the new reward structure, i.e. rI To + Tstuck ffall- The SAC-I* agent (adaptive) learns an inhibitory policy network estimate the weight so that ry* To + w x Tssuck Tfall- This SAC-I* agent with adap- tive inhibition (green line) outperforms the other agents, and the difference is highlighted in the mixed version of the task (see further Figure 1.
Stape
Figure 8: Performance of SAC and SAC-I agents.
Mixed version of AmpedalWalkrrHHardooee-z. To replicate the scenario in which there are go and stop episodes, we retrain the baseline agent on modified task mixing both the BipedalWalker- v3 (Go trial) and Bipedal Walkerrrreeerrrrreeeee (Stop trial). For each episode, we randomly choose in which environment the agent should perform. Interestingly, this makes the learning even more challenging, because the obstacles are sparser in time and the agent has to explore and exploit while learning the easy and hard versions of the task simultaneously. In Figure | show that using inhibitory networks is critical to successfully learn the mixed task. Both agents are retrained using the same reward structure as presented in the previous section. Although the task gets easier as the percent of go trials is increased, from 10% to 30%, we observe that it is more difficult to learn the mixed version of the task since there are less stop trials to learn from (for instance, compare the two plots in the middle column). Also, we observe some training instability for 70% stop trial, likely because the interference between learning the Stop and Go trials (3rd column). Interestingly, we observe that the SAC-I* agent with adaptive inhibition has more stability across the stop training sessions. For Stop 90%, the averages of the reward standard deviation are 47.5 and 27.5 for SAC-I and SAC-I* agents, respectively. For Stop 70%, the averages are 56.3 and 31.7, respectively. Finally, we observe that, unlike the SAC agent, only the SAC-I agents are able to successfully learn the stop trials (2nd column). Note, they are trained using the same reward structure (previous section).
Total
Stop trials
Go trials
300
Stop trial 90%
Axeraee Mexeee
Prenge Revsard
200
200
100
-3000
-100.0
-108
30000
Stags
103
Steps
306
300
Stop bieel TONe
Average Honard
.00
100
SAC SAC-I (stancird)) SAC-1" (adaptive)
0.5
3.0
-10g
0.5
2.0
3.5 2.0 255
3.0
3.5
-10g 0
0.5
1.0
1.5
2.0
25
30
35
Figure 9: SAC and SAC-I agents performance in the mixed version of the WalpedalWakerrHrrddooe. v3. First row shows results for the task with 90% Stop trials (hardcore), while the second, with 70%. From the left to the right, rewards are averaged across all, only Stop and only Go trials.
CONCLUSIONS
In this work, we draw parallel between neuroscience research in inhibitory control and reinforce- ment learning of competing value functions. We propose an SAC algorithm using inhibitory net- works with a particular focus on retraining the agent to acquire a new skill, while still exploiting previously learned abilities. Through our experiments, we show that SAC-I agents are able to main- tain higher rewards from the beginning of retraining since they keep a previous ability compared to retrained SAC agents (Figure (). With SAC-I, we advance the use of SAC methods by introduc- ing the use of multiple value networks with respective episodic replay buffers, as well as distinct entropy parameter estimation for skills at different training stages. In Figure 9 show SAC-I is able to successfully learn the mixed version of the Bipedal WalkerHardcore-- -33, otherwise unsolvable with the standard SAC. We believe that the experiments and results presented in this paper offer proof of concept of the advantages using SAC methods with inhibitory networks (SAC-D) for faster retraining.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Haitham B. Ammar, Karl Tuyls, Matthew E. Taylor, Kurt Driessens, and Gerhard Weiss. Reinforce- ment learning transfer via sparse coding. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems Volume 1, AAMAS '12. pp. 383390, Rich- land, SC, 2012. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 0981738117.
Adam R. Aron. The neural basis of inhibition in cognitive control. The Neuroscientist, 13(3):
128, 2007. doi: 10.1177/107355407292222 PMID: 17519365.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the Thirty-Firy AAAI Conference on Artificial Intelligence, AAAI117, pp. 17261734. AAAI Press, 2017.
André Barreto, Shaobo Hou, Diana Borsa, David Silver, and Doina Precup. Fast reinforcement learning with generalized policy updates. Proceedings of the National Acudemy of Sciences, 117
(48):30079-30087, 2020. ISSN 0027-8424. doi: 10. 00.1077/pnas.1907771177 1907370117. URL https:
7/ //www. /ww..pnas. org/ wwntenttii7/ /30079
Andrew G.
Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learn- ing. Discrete Event Dynamic Systems, 13(12):4177, 2003. ISSN 0924-6703. 1022140919877.
doi: 10. 10.10A//A
Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z Leibo, Jack Rae, Daan Wierstra, and Demis Hassabis. Model-free episodic control, 2016.
Matthew Botvinick, Sam Ritter, Jane X Wang, Zeb Kurth-Nelson, Charles Blundell, and Demis Hassabis. Reinforcement learning, fast and slow. Trends in cognitive sciences, 2385):00842, May 2019. ISSN 1364-6613. doi: 10.1116).iiss.201..22.. URL httthttps/:/ddo org/10. 2016/j.tice 201 99. 02. 006
Todd S. Braver, Deanna M. Barch, Jeremy R. Gray, David L. Molfese, and Avraham Snyder. Ante- rior Cingulate Cortex and Response Conflict: Effects of Frequency, Inhibition and Errors. Cere- bral Cortex, 1199)825-836, 09 2001. ISSN 1047-3211. doi: (0.1093//erroo//11...22 URL
ht https: doi http:://doo.orggg///p 1ttp:///..109//crrro/111111 9.825 cercor/1c..11
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540. 2016.
Tim Brys, Anna Harutyunyan, Matthew Taylor, and Ann Nowé. Policy transfer using reward shaping. In Proceedings of the 2015 International Conference on Autonomous Agents and Multi- agent Systems, AAMAS '15 pp. 181188. International Foundation for Autonomous Agents and Multiagent Systems, 2015. ISBN 978 7781450334136.
G. Comanici and D Precup. Optimal policy switching algorithms for reinforcement learning. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2010), pp. 709-714, 714, 2010.
Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. In S. Hanson, J. Cowan, and C. Giles (eds.), Advances in Neural Information Processing Systems, volume Morgan- Kaufmann, 1993.
Adele Diamond. Executive functions. Annual Review of Psychology, 64:135-68 2013.
S. Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In Pruceedings of the 35th International Conference on Machine Learning, volume 80, pp. 1591. PMLR, 2018
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep easrd policies. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1352-1361, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR.
10

---

Under review as a conference paper at ICLR 2022
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and An- dreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870, Stockholmsmssan, Stockholm Sweden, 10-15 Jul 2018a. PMLR. Jttp::tt::/pproeeeee 8b. html
press/
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications. CoRR, abs/1812.05505, 2018b. URL http: wrg. org/abs/ /
1812. 05905
Steven Hansen, Will Dabney, Andre Barreto, David Warde-Farley, Tom Van de Wiele, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. In Interna- fional Conference Learning Representations, 2020. URL
net/
Borumii
Zhang- Wei Hong. Tzu- Yun Shann, Shih- Yang Su, Yi-Hsiang Chang, Tsu-Jui Fu, and Chun- Yi Lee. Diversity-driven exploration strategy for deep reinforcement learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Biani -iannhi, and R. Garnett (eds.), Advances in Neural Infor- mation Processing Systems, volume 31. Curran Associates, Inc., 2018.
Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: new approach of reward shaping. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 15931-15941, Curran As- sociates, Inc., 2020. URL /ttp/ttts///////ceeaissssss neurips cc wttpppppp/p////E1LLLLL pdf
J.S. Ide, P. Shenoy, A.J. Yu, and C.S. Li. Bayesian prediction and evaluation in the anterior cingulate cortex. Neurosci, 3335):2039447,22
David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In Sheila A.
Mcllraith and Kilian Q. Weinberger (eds.), Proceedings of the Thirty-Second AAAI Conference Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI- 18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. AwA.AAAAAAAAAAAAAAAAAAAAAAATAAAT
AAAI Press, 2018. URL
https: / www www..aaai..oommmmmmmmm....... org/ ccte wwwps/iinxxx... http:///aAAA/lAAAaaaaaaaaa/a////
Alessandro Lazaric. Transfer in Reinforcement Leariigg:: A Framework and a Survey, pp. 143-173. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. ISBN 978-3-641--2745--3 doi: 10. 1007/
978-3-642-27645-333 URL htttss://dd/t http://ooooooo////////// 1007
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Represenaations, (ICLR 2016), 2016. URL htap://///xvvv.. wwe.xivvvrrr/.. org/ www.bb//13. 02971
Zichuan Lin, Tianqi Zhao, Guangwen Yang, and Lintao Zhang. Episodic memory deep g-networks. In Proceedings of the 27th International Joint Conference Artificial Intelligence, IJCAI'18, pp- 24332439. AAAI Press, 2018. ISBN 9780999241 127.
G. D. Logan, M. Yamaguchi, J. D. Schall, and T. J. Palmeri. Inhibitory control in mind and brain 2.0: Blocked-input models of saccadic countermanding. Psychological Review, 155(2):115-147, 2015. doi: https://doi.or// 103.00000/00082233333333
Andrew Martin. Fear of Failure in Learning, pp. 1276-1278. Springer US, Boston, MA, 2012. ISBN 778-1-4419-1428-- doi: 10.
266. URL httpttttts/oooo org/
T0.10077978--144419-444---
Volodymyr Mnih, Koray Kavukcuoglu. David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen,
11

---

Under review as a conference paper at ICLR 2022
Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learn- ing. Nature, 118:529-5333 2015. ISSN 00280836. URL http://tt..o.........oo..1 naturel 42.66
O. Nachum, S. Gu, H. Lee, and S. Levine. Data-efficient hierarchical reinforcement learning. Ad- vances in Neural Information Processing Systems, pp. 3303-3313, 2018.
Martin L. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 1994.
Andrei A. Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distil- lation, 201
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. CoRR, (bs//600.04661,, 2016b.
abs /00.6666..0 04671
Himanshu Sahni, Saurabh Kumar, Farhan Tejani, and Charles Isbell Jr. Learning compose abs/1711 abs/1711.11289 11289
skills. CoRR, abs/1711.11289) 2017. URL http::t/axxi...oo/// http:://aaaai..ooooo//ss///1111.
Stefan Schaal. Learning from demonstration. Proceedings the 9th International Conference on Neural Information Processing Systems, NIPS pp. 10401046, Cambridge, MA, USA, 1996.
MIT Press.
J.D. Schall, T.J. Palmeri, and G.D. Logan. Models of inhibitory control. Philos Trans R Soc Lond B Biol Sci., 372(1718), 04 2017. doi: 1t1.//////t.2216..11111.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1312-132,, Lille, France, 07-09 Jul 2015. PMLR.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. Francis Bach and David Blei (eds.), Proceedings the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1889-1897, Lille, France, 07-09 2015. PMLR.
John Schulman, Pieter Abbeel, and Xi Chen. Equivalence between policy gradients and soft q- learning. CoRR, abs/1704.06440. 2017a.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017b. URL http:://rxiv
1707.06347
B. Seymour, T. Singer, and R Dolan. The neurobiology of punishment. Nat Rev Neurosci, 8:300- 311. 2007. URL :ttttps//dot: wtt.:/i.oorr111.111113..1
Pradeep Shenoy, RP Rao, and AJ Yu. A rational decision making framework for inhibitory control. In Advances in Neural Information Processing Systems (NIPS), volume 23, pp. 2146-2154, MIT Press, 2011.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction reinforcement learning. Artif. Intell., 112(12):181211, 1999. ISSN 0004-3702. doi:
Matthew E. Taylor and Peter Stone. Transfer learning for reinforcement learning domains: survey. Journal of Machine Learning Research, 10()):6633-163 2009.
12

---

Under review as a conference paper at ICLR 2022
Philip Thomas. Bias in natural actor-critic algorithms. In Eric P. Xing and Tony Jebara (eds.), Pro- ceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 441-448, Bejing, China, 22-24 Jun 2014. PMLR.
Emanuel Todorov. Linearly-solvable markov decision problems. In B. Schölkopf, J. Platt, and T. Hoffman (eds.), Advances in Neural Information Processing Systems, volume 19. MIT Press, 2007. URL p neurips ottp://eeeraaaaeeereooo.ooeerrra cc/ps///////e/////////eee//
pdf
Emanuel Todorov. Compositionality of optimal control laws. In Y. Bengio, D. Schuurmans, J. Laf- ferty, C. Williams, and A. Culotta (eds.), Advances Neural Information Processing Systems,
volume 22. Curran Associates, Inc., 2009. URL pttttpes/pproceeinggs www.ipsurp.... ec/ pdÉ.
Endel Tulving. Episodic memory: from mind to brain. Annual Review of Psychology, 53:1-25, 2002.
Benjamin Van Niekerk, Steven James, Adam Earle, and Benjamin Rosman. Composing value functions in reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 Proceed- ings of Machine Learning Research, pp. 6401-6409. PMLR, 09-15 Jun 2019. URL http:
//proceedings , mlr. wwwppppwssss.sseeepp niekerk19a.....
o html
Harm Van Seijen, Mehdi Fatemi, Joshua Romoff, Romain Laroche, Tavian Barnes, and Jeffrey Tsang. Hybrid reward architecture for reinforcement learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso-
ciates, Inc., 2017. URL httpsss/pproceeiinggss... neerips.. wtc.issscc/eee
pdf
Juan Vargas, Lazar Andjelic, and Amir Barati Farimani. Effects of sparse rewards of different magnitudes in the speed of learning of model-based actor critic methods, 2020.
F. Verbruggen and G.D. Logan. Models of response inhibition in the stop-signal and stop-change paradigms. Neurosci Biobehav Rev., 33:647-666 61, 2009. doi: n0.1016f..euuborre.00000
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, pp. 35403549. JMLR.org. 2017.
Honghao Wei and Lei Ying. Fork: A forward-looking actor for model-free reinforcement learning, 2021.
B. D. Ziebart. Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy. PhD thesis, Machine Learning Dpt., Carnegie Mellon University, Pittsburgh, PA, 2010.
13

---

Under review as a conference paper at ICLR 2022
A
APPENDIX
BROADER IMPACTS
Efficiency in retraining is useful in situations where training is computationally expensive can cause damage and wear to instruments like robotic arms. Our work proposes a method of efficient retraining of agents by reusing trained policies and value function estimators. Ultimately, we con- tribute to the body of work that prioritizes creative reuse of models.
Table List of common apper-parameters used in the SAC training.
Parameter
Value
Optimizer
Adam
Replay buffer size
1.0e6
Number of hidden layers
1 (LunnrLaander), 2 (Bipedal Walker)
Number of hidden neurons
256
Batch size Learning rate
64 and 128
5,0e-4
Discount factor y
0.99
Soft T
1.0c-3
Entropy target T
-3
Activation function
ReLU
Target update interval
Number of episodes before training πт
0 (LunarLander)). 100 (BipedalWalker)
SUPPLEMENTARY EXPERIMENTS AND RESULTS
AunarlLaneerCotiinuuu--22 with bomb. The observation space consists of the original states (related to position, angle, velocities and leg contact) plus 4 values indicating the upper left and bottom right coordinates of the bomb zone (a box with a width of 10% of the screen size). The agent assumes horizontal coordinates E [-1, 1] and vertical coordinates y E [0,1.4, where y = 0 corresponds to the ground. The bomb is randomly placed with center coordinates Ie E [-0.2, 0.2] and ye 10 0.5), and appears with default frequency of 50% (stop trials), which creates an environment with the highest uncertainty. In the stop trials, the bomb appears randomly after the agent achieves an altitude between the range [0.9, 1.1]. Importantly, the bomb coordinates are included in the observation state only after the bomb appears so that the agent does not know about its existence beforehand. If the bomb is absent, four dummy negative values are provided in the state. The episode ends if the agent hits the bomb (i.c. agent inside the bomb zone) or the ground, successfully lands or reaches the maximum length of 1000 steps.
Conservative reward shaping. In LunarLanderComtinuonr v2 with bomb, it is expected that how the inhibitory reward ri is shaped, as described in the main text, influences the agents" training. Indeed, we tested a few shaping variants and observed differences. However, results comparing SAC- I vs. SAC were qualitatively similar to the ones presented so far. One potential issue with the current reward shaping to avoid the bomb is that it does not consider any of the aspects involved in landing. Therefore, we present another variant of the rı which includes some of the aspects of the raw re- ward such rewarding getting close to the landing pad, slow- ing down, and penalizing "aggressive" maneuvers (large ve- locities and vertical angles). The more conservative reward shaping is given by the sum of the following components
300
200
Avarage Rewward
150
50
54C
joonannatie shasing)
-100
szaps
Figure 10: Comparison of SAC and SAC-I agents with conserva- tive shaping.
ry = r2 + ry + Tangle + ruel, where rn 1/ de + 0.1) + 0.77, = -33 (d, - 0.05)2, Tangle = -angle?, Fvel -2 x (v + uy + o2 vangte), and (dz, dly) are the horizontal and vertical distances to the bomb, respectively. In short, the inhibitory reward has maximum value 0 and com- ponents with quadratic forms. Coefficients are adjusted to have r, < and a balance between the different rewards. If the inhibition rule given by conditions d₂ 0.2 and y Yb is met, the Q-I
14

---

Under review as a conference paper at ICLR 2022
network is used along with the inhibitory reward rI. In this version, the bomb avoidance is driven by the horizontal distance dz. independent of dy, making the agent more conservative. Results are shown in Figure TO. Unlike SAC, the SAC-I is benefited by the shaping and it offers a mechanism to include additional shaping without interfering previously learned Q networks.
BipedalWalker environments. In both Bipeda/Walker-v3 and Bipedal HipedalWalkerHHaaddooeeeyy tasks, the primary goal is to reach the end of the track using the least amount of energy as possible. The agent gets dense reward for moving forward (total 300+ points up to the end) and -100 for falling. There is a small cost for applying motor torque and moving the joints, so agents with optimal movements score more. The observation state has a total of 24 variables split between 14 related to the agent's body (including the angles and velocities of hull, hip joints, knee joints, vertical and horizontal speeds) and 10 LIDAR sensor readings. Everything the agent knows about the environment is through the sensors, so it does not know about obstacles or holes until they are scanned by the sensor, neither about its position in the environment since no coordinates are given. Note that sensor readings are relative to the agent's position and angle. These characteristics make the hardcore version significantly more challenging than the plain terrain version. The agent has legs, each leg has 2 joints, and therefore the action space has size 4 with continuous values within the range of [-., Each episode ends whenever the agent reaches the end of the track, falls reaches the maximum of 2000 steps. Some informal discussion regarding the difficulty in solving the task can be found in https://aistaccexchangg wttp:annnnoooo/uettoooos///s//// (iaeeettion//348//aaass-s- 1 -o--o----ov---onnii------ hww--oppnn----door--eeeeee cictoorr-bipdal-wwaaeee wwwh-lerrwiit---ehe-----lee thir---tierr--mmmmm.m
BipedalWalke experimental design. The standard SAC agent is trained in the BipedalWalker-Vy be used as a baseline agent in further training in the hardcore version. Agents are trained with three different seeds for 3K episodes, and the one with the best average reward is selected as the baseline. This agent reaches an average over the last 100 episodes above 300+ in about 0.5M steps similar to the results reported in Wei & Ying (20TT) All the network weights are transferred to the retrained agent. The Q-I network is also retrained from previous weights. For the BipedalWalker Hardorra,,,,,, we train agents with five seeds for 5K episodes, and compute the average reward across runs.
Effect of removing the fall penalty. In neuroscience, punishments (negative rewards) are known to produce aversive conditioning prediction of aversive events that leads to defensive ag- gressive Pavlonian actions (Seymour et al. 2007), what in turn can cause fear of failure and procrastination in learning (Martin) 2012). Negative rewards are often successfully used in RL. However, there are studies showing that intermixing positive and sparse negative rewards might have adverse effects in learning, reported in recent work using DDPG in robotic environ- ment Vargas et al. (2020). Moreover, optimally shaping reward is still an open area of re- search (Hu al., 2020), and it is likely depend the particular DRL method being used. Interestingly, in our experiments in HipedalWalkerHardeon v3, we observe that the SAC agents are greatly benefited from removing the fall penalty from the raw environment reward. We observe that, with the fall penalty, often the agent learns to simply stop in front of the block by fear of falling (which likely a local minimum), never exploring enough to over- come the large obstacles. In Figure [T] we illustrate the ef- fects of removing the fall penalty and including the stuck penalty. Removing the fall penalty (-100) included in origi- nal reward/penalty provided by the environment allowed the agent to explore riskier actions to overcome obstacles. The SAC agent without stuck penalty is also unable to learn the task successfully (purple line). It should be noted that retrain- ing from baseline agent is also critical to speed-up training and make possible solving the task. Average reward represents the average over the last 100 episodes of the raw environmental reward.
300
250
200
Avarage Roward
150
100
SAC+ tala -derc
-100
1.0
Steps
360
Figure 11: Agent performance in SipedallWaleerHarddoeee-22 re- trained using SAC algorithm with (brown) and without (blue line) the fall penalty.
Training from scratch in the mixed version of the BipedalWalkerHarddcee.aa show the average reward during training for the SAC and SAC-I agents, in the environment with 90% stop trials. Training the SAC-I agent from scratch is slower than retraining from the baseline agent (Figure 10, Stop trial 90%). However, the SAC-I agent still able to solve the hardcore task (Stop trials). Mcanwhile, the SAC agent is only able to solve the simple task (Go trials). The advantage
In Figure 12 we
15

---

Under review as a conference paper at ICLR 2022
of retraining is evident during the initial training steps. The retrained SAC-I agent completes the simple task from the beginning of the training (compare with Figure 10, Stop trial 90%). The SAC-I agent trained from scratch learns to complete the Go trials only after 1M steps.
Total
Stop trials (90%)
Go trials (10%)
300
Eustage Ransed
200
200
Zawrngg Rawseed
200
150
150
100
SAC SAC-I
-30
-50
100
-100
Steps
169
0.5
1.0
1.5
Steps
2.0
2.5
3.0
Figure 12: SAC and SAC-I agents performance in the mixed version of BipedalWallerffrrdorrvv3.3 Agents are trained from scratch. Stop trials are 90% the total number of episodes.
16


Reviews:
[{"review_id": "lY0Rekb_ndR", "reviewer": "Reviewer_y7rr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the proposed method is sound and the paper includes experiments that well illustrate the problem setting. However, I have several concerns regarding clarity of presentation, novelty of the contribution, and the experiments.\n\nRegarding clarity, If I understand the paper correctly, the proposed method is essentially a hierarchical policy with discrete primitives that get activated one at a turn by a high level policy, called inhibitory network. As presented, the framework is limited to two primitives, called “Go” and “Stop” , though an extension to any number of primitives follows trivially. Many of the details still remain unclear to me (as I will explain below), and it is very possible that I have missed important ones.\n\nIn particular, I would suggest including a concrete example of the realization of the method, with the connection to hierarchical policies right in the introduction; and improving the Methods section by including more details and being more careful with the notation and claims. Currently there are many aspects that are not accurate or not addressed at all. Here are some examples:\n* What is the exact role of the inhibitory network (e.g., what is $q$ in Figure 2)? \n* Is the network learned, and if it is, how do you learn it and what is the objective for learning it? \n* When the inhibitory network is learned, the split of the data between the two replay pools will be affected, or does it?\n* In the first paragraph of page 9, it is explained that SAC-I* learns the inhibition rule by adjusting a reward weight parameter $w$. How does this relate to what is presented in Figure 2? \n* What if $s$ and $s’$ belong to different partitions, in which replay buffer the corresponding transition goes?\n* In Equation 1, conditional expectation depends on the variable $t$ that is visible only inside the expectation so it cannot be a conditioner.\n* In the first paragraph of page 3, $\\rho_pi$ is defined as the state marginal, though it should represent the dynamics.\n* Are there separate policies for the Go and the Stop behaviors, or just a single one as in Figure 2? If there is only a single policy, then what is the purpose of having two temperatures $\\alpha_R$ and $\\alpha_I$ (Section 3.3)?\n\nRegarding contributions, it is not clear how the approach is different from hierarchical RL with discrete low level behaviors. There is indeed a difference in how the data is split in two replay pools, each used to train a separate Q function, but a better justification is needed to explain why this difference is an important one. I think the work would substantially benefit from reformulating the approach as an HRL method, carefully pinpointing the improvements and contributions over existing HRL literature, and evaluating their relative importance. The connection to inhibition in the brain is interesting, but as presented at the moment, hides the connection to HRL.\n\nRegarding the experiments, there is some mismatch between the figures and the text. For example, Figure 5 shows that training without the inhibitory network is more efficient. However, the text suggests the opposite. Similarly, Figure 9 suggests that pure SAC performs comparably to SAC-I* (adaptive), whereas SAC_I (standard) fails, but the text claims that “only the SAC-I agents are able to successfully learn the stop trials.”  Also comparing SAC to SAC-I is not completely fair as SAC-I makes use of domain knowledge in terms of handcrafted inhibition rule. Comparing to SAC-I* that learns the inhibition rule (if I got this right) is well justified, but no details are given regarding how it is learned.\n\nMinor comments:\n* The sentence: “ For states with high reward, a low entropy policy is permitted while, for states with low reward, high entropy policies are preferred, leading to greater exploration” is inaccurate. The magnitude of the reward and the entropy of a state are not related. Instead, the entropy of a state is determined by the relative magnitude of the expected future returns for different actions.\n* Vague sentence, can you please reword: “Stop networks estimate the value of the unexpected event, leading the policy to learn new skills.”\n* The experiment section mentions that “... the bomb coordinates are included in the observation state only after the bomb appears…”. How is this implemented in practice? Does the network allow a varying number of observations?\n", "summary_of_the_paper": "The paper proposes inhibitory networks for reinforcement learning. Inhibitory networks choose a behavior among a set of low-level behaviors to be executed at the current state. The inhibitory network can take the form of a handcrafted rule (i.e., execute policy X if some state feature is active, else execute policy Y), or can be learned similarly to hierarchical policies. The method is implemented on top of soft actor-critic and evaluated in two OpenAI Gym environments, LunarLander and BipedalWalkerHardcore with some non-default modifications, and compared to a baseline standard SAC implementation.", "summary_of_the_review": "The clarity of the paper is not sufficient for assessing its contributions and how it positions itself with respect to the prior literature. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "klnmpQknoFK", "reviewer": "Reviewer_dDqD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strong Points:**\n\nThis paper investigates a set of environments and tasks for multi-task / transfer learning. While these environments are fairly standard (from OpenAI gym), the tasks themselves, from what I can tell, are not widely used. I see these transfer tasks as a useful contribution of the paper, which may become adopted in future papers.\n\nThe paper appears to be technically correct, employing a modified version of SAC to investigate transfer learning tasks. The authors examine ablations of their proposed method to isolate the contributions from each aspect. The authors also vary aspects of the environment to investigate how this affects transfer performance.\n\nThe authors attempt to bring in motivation from inhibitory control in neuroscience. While I feel that this aspect needs improvement (see below), I appreciate the effort to port these concepts into deep RL and machine learning.\n\n\n**Weak Points:**\n\nThe empirical evaluation is weak and could be improved in a number of ways. For instance, the main baseline used in the paper is pre-trained SAC. A more appropriate baseline would be to compare with alternative methods for multi-task or continual learning. Despite citing several lines of work in the introduction, e.g., various forms of hierarchical RL and compositional value functions, these other baseline are absent from the evaluation. Another weakness of the empirical evaluation is that SAC-I is evaluated on only two environments from similar domains. While I agree that these are interesting tasks, it may be helpful to also evaluate in settings used by previous works in continual RL, enabling a proper comparison. Finally, the evaluation entirely focuses on task performance. A more complete empirical evaluation would include further analysis on the mechanics underlying the inhibitory network and how it affects the resulting policy.\n\nThe method itself is difficult to follow, due the number of variations presented. Overall, the presentation of the method comes across as somewhat ad hoc. For instance, in some cases, an inhibition rule is defined, whereas in other cases, this is learned by a separate inhibition policy. In both environments, the authors use various redefinitions of the reward functions for shaping and learning the task. Separate replay buffers are used for the different tasks, along with separate temperature factors. The Q-value is fed into the policy network, which is an odd design choice that seems poorly motivated. In total, it’s difficult for the reader to assess where this method fits into the broader landscape of previous works and whether the proposed design choices are novel and/or reasonable. Fixing these issues would involve more of a first-principles approach to describing the issues with previous works in hierarchical / multi-task / continual RL and proposing and analyzing well-founded methods for overcoming these challenges.\n\nThere is a considerable amount of work in related areas of multi-task RL, non-stationary environments, online/continual learning, etc. that is largely overlooked. These are vast, interconnected areas, so I would not expect an extensive literature review. However, I would expect a more in-depth discussion beyond what is currently present in Section 2.4. This discussion would help to situate the proposed method.\n\nThere appear to be some inconsistencies in the empirical presentation. SAC outperforms SAC-I in Figure 5, although the accompanying text states otherwise. Similarly, SAC performs nearly as well as SAC-I* (adaptive) in Figure 9, again, in disagreement with the text. Note: the plotting colors in Figure 9 are also inconsistent with Figure 8. My guess is that these are both errors. These errors (and perhaps others) need to be fixed.\n\nThe authors motivate the proposed method by describing connections to inhibitory control, from neuroscience. While I’m generally supportive of neuroscience inspiration in machine learning, it’s not clear what these sections/paragraphs provide for the paper. That is, the details of SAC-I are so disconnected from the specifics of the neuroscience research that this motivation effectively amounts to ‘animals can switch between multiple tasks.’ This doesn’t strike me as any more helpful than the motivation from existing works in hierarchical RL, e.g., options. By framing this work in terms of neuroscience, it may needlessly complicate the presentation, raising a host of questions, such as whether separate replay buffers and the ability to accurately classify tasks are biologically valid. I would suggest that the authors either a) largely remove these sections or b) show more specifically how the inhibitory control literature can inform RL beyond what currently exists.\n\n**Additional feedback:**\n\nSection 2.1: This notation is imprecise / sloppy. For instance, in Eq. 1, the environment dynamics do no appear, and $r$ is not defined.\n\nSection 2.2: Missing citation for PPO. It would be worth unpacking the last sentence (regarding KL minimization), or at least pointing to a source that describes this, such as Levine’s 2018 tutorial.\n\nSection 2.3: It’s not clear that anterior cingulate cortex resembles TD learning. To my knowledge, this is more in-line with model-based estimation, not model-free.\n\nSection 3.1: By conditioning the policy on $Q$, this effectively becomes a mixture of policies, which is equivalent to a hierarchical policy. Ideally, connections and differences from previous works in hierarchical RL should be discussed. It’s not clear that this diagram really matches Algorithm 1.\n\nSection 4.1: It’s not clear what the reward shaping analysis really shows. It seems like the point is that SAC-I is less negatively affected by the shaping reward.", "summary_of_the_paper": "This paper presents a method for transferring a pre-trained agent to a new task. The authors refer to this method as SAC-I, as they are combining ‘inhibition’ with soft actor-critic (SAC). The basic idea is to train a separate set of Q-networks and potentially an inhibitory policy network. These are then used with the existing policy. Various agents are presented on two Box2D environments from OpenAI gym. SAC-I generally improves over SAC.", "summary_of_the_review": "Currently, the paper is lacking an adequate discussion of related works, as well as appropriate baselines beyond SAC. As the authors are proposing a new method for transfer learning, a comparison with existing works is necessary for proper empirical evaluation. Given that this method shares some similarities with hierarchical RL, these may be useful baselines. Looking to these baselines could also provide additional tasks and environments explored in previous works. Without these improvements, the proposed method is difficult to evaluate, as the larger context is missing. For these reasons, I cannot recommend acceptance at this time.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fMWTjHi4U2s", "reviewer": "Reviewer_6ibM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of two independent policies is interesting, but the use of the words ‘inhibitory’ and ‘stop-policy’ is a bit confusing. I’m not sufficiently familiar with the neuroscience literature to judge this well, but my understanding was that inhibition is about the suppression of reflexive actions, rather than about what to do instead. The agent presented here just has to choose between two policies.\n\nOne question that the two-policy situation raises is why to stop at two. Depending on the task, it might be that a different number is optimal. Another question relates to the effectiveness of the extra learning capacity (weights & updates): what if that capacity of the decider network and the second policy was allocated to a single policy instead? I could imagine that the factorization employed here can be more effective, although that probably depends on the environment, but it would be good to know instead of imagining. The experiments are done in environments that effectively seem to have two tasks, which might be why having two policies works well in practice. But what if the environment mixes three tasks? Or 20?\n\nThe method presented here is strongly reminiscent of the options framework - which is being cited - with a different approach to choosing start and stop states. It would help place the paper more clearly in the overall RL literature if this similarity and difference was discussed more explicitly.\n\nIt is not entirely clear to me what is happening in the case called adaptive SAC-I; is there an extra network being trained to estimate the weight $w$? And if so, what is the loss for this network?\n\nCould the authors comment on the use of SAC as a starting point / base training method? I understand that it is a very effective maximum entropy method, but it is not necessarily the state of the art in RL tasks in general. Both the motivation from inhibition in the neuroscience literature, and the effective implementation of a two-policy agent are fairly applicable to many reinforcement learning techniques, not just SAC.\n\nOther questions and remarks:\n- Figure 4: how are the agents trained if they don’t have a replay buffer (episodic memory)? Algorithm 1 seems to indicate that no training will happen without.\n- Apart from ‘inhibition’, the term ‘episodic memory’ is also a bit of a misnomer when seen against the use of that term in the literature: here it is just used as a replay buffer, if I understand the paper correctly. There is no notion of retrieving information from the memory within a given step here, which is commonly the case when the term ‘episodic memory’ is used.\n\nOn the whole, the paper does not entirely convince me that the proposed architecture will generalize well. Including experiments on a wider range of environments would address that concern. Including other baselines than SAC would also help to clarify the value of the contribution.\n", "summary_of_the_paper": "This paper introduces an agent architecture with two independent policy networks (called ‘go’ and ‘stop’) and an assignment mechanism consisting of a network or a rule that activates one of the two every step. The two policy networks are in principle both soft actor-critic (SAC) agents, although the authors explore several variations on the degree of independence between them, with e.g. hyperparameters being tied in some cases. Each policy network is trained on replayed data from the subset of states that has been assigned to it by the assignment mechanism.\n", "summary_of_the_review": "The proposed architecture is potentially interesting, even if it seems to go against Sutton’s “Bitter Lesson” by specializing an architecture for a specific case, rather than relying on learning. It does look brittle due to the hardwired nature of the choices that are being made; in particular the number of policy modules in the agent and the division of states across the policies. I am open to being convinced these aspects don’t make the agent brittle, and would raise my score in that case, but that would require more extensive experimentation and baselines.\n\nThere are some misnomers, or uncommon uses of terms: mainly ‘inhibition’, but also ‘episodic memory’. Otherwise the paper is well-presented; changing the terminology to be more in line with the literature would make it mostly easy to read.\n\nMy general recommendation at this point is to review the agent architecture and see if this is a reasonable point to present results. It might be interesting to see what this kind of structure does to agent performance on a wider array of tasks, and how those tasks are solved. If it does well generally, this would be a strong paper, but in its current form it looks a bit unfinished.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_VcDsF5H-aA", "reviewer": "Reviewer_tm8g", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n* Considering that RL is about sequential decision making and its importance in developing human-like agents, approaching the problem with a neuroscientific view might be a helpful and meaningful direction. I believe it could be a more considerable submission if the need for human knowledge is minimized and its empirical performance is improved (see my comments on the weaknesses below, please).\n\n**Weaknesses**\n\n* The concept of retraining should be more formal or specific.\n\n  In Sec.2.4, the authors state that \"we are primarily focused on transferring learned value and policy functions among identical aspects of a task, while learning new skills (value functions) and retraining the previous learned policy within the similar environment\", but its difference from transfer learning in RL is still not clear, as usual transfer learning in RL also involves transferring learned components. I think \"identical aspects of a task\" doesn't clarify the dissimilarity.\n\n  I suggest the authors to discuss which components of the original MDP can change and why and how retraining is different from common practices of transfer learning in RL.\n\n* My biggest concern is that this work seems to require or rely on human engineering of reward functions too much (regardless of learning an inhibitory policy or not).\n\n  Especially, it looks that regular rewards $r_R$ are designed to contain mixed, conflicting signals or no signals about dangers (e.g., bombs), while inhibitory rewards $r_I$ are configured to provide meaningful signals about the dangers. For instance, in the paragraph \"Advantage of retraining.\" of Sec.4.1 for the LunarLanderContinuous experiments, the authors state that the penalty, $r_{bomb.proxy}$, which is defined using the agent's distance to the bomb, is combined with the original reward and given to the agent for the retraining. Given that the bombs appear close to the landing pad (Fig.3), the regular rewards $r_R$ would be a mixture of two competing signals. On the other hand, the inhibitory rewards $r_I$ defined in the paragraph \"Effect of using episodic memory and dual alpha in SAC-I.\" are just $r_{bomb.proxy}$, which means the SAC-I agents receive the clean, decoupled signals about the bombs.\n\n  I'm aware that the authors suggest it as part of the problem setup, but assuming the specific problem, I think the contribution of this work is not very significant. Compared to the normal case with a single reward function, this problem setup requires defining and adjusting two reward functions properly, which is not free (i.e., humans' job) and can be more complex.\n\n  I believe that removing or minimizing the requirement of manual reward function engineering could be a good way to improve this work.\n\n* Little more details about the inhibitory policy would be nice. I would like to see clarification especially on the following points: Does it output binary actions to choose between regular and inhibitory modules? What exactly does \"soft modulator\" from Sec.3.4 mean?\n\n* The performance improvement over SAC seems to be marginal in many cases, considering the shaded area of the plots. Increasing the number of runs might help make the results more clear, but I also suggest finding ways to improve its performance.\n\n**Minor writing suggestions**\n\n* Other methods use multiple \"policy\" -> \"policies\" (Sec.1)\n* First, we develop the SAC-I architecture for accelerated retraining, \"that\" encompasses -> \"which\" (Sec.1)\n* which is not able to \"successfully solve the task\" -> \"solve the task successfully\" (Sec.1 and similar occurrences)\n* \"previous\" learned -> \"previously\" (Sec.1, Sec.2.4)\n* Broadly speaking, transfer learning in RL consists of \"transfering\" the knowledge -> \"transferring\" (Sec.2.4)\n* to improve the learning performance in a related, but \"different, task\" -> \"different task\" (Sec.2.4)\n* \"over estimation\" -> \"overestimation\" (Sec.3.1)\n* The inhibitory policy network is a \"stand alone\" agent -> \"stand-alone\" (Sec.3.4)\n* with \"particular focus\" -> \"a particular focus\" (Sec.5)", "summary_of_the_paper": "The authors present SAC-I, which is a modified version of SAC designed for retraining of some existing agent on an updated environment or task. Inspired by the concept of inhibitory control from neuroscience, they learn a separate Q functions and optionally a policy for inhibition. They define some inhibitory reward $r_I$ for each environment to enable it, and make comparison with SAC on modified versions of LunarLanderContinuous and BipedalWalkerHardcore-v3.", "summary_of_the_review": "Although I appreciate the attempt to bring another neuroscientific concept into RL, I'm mainly worried about the significance of the reward function engineering in the presented results and thus a smaller importance of the proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a hierarchical policy architecture with two substituent policies, "go" and "stop", and a controller mechanism for switching between them on every step (either a rule or a learned network), taking inspiration from neuroscience concepts of inhibition. Both are trained via Soft Actor Critic on the subset of states assigned to them and comparisons are made against a baseline. The use case targeted is the repurposing of pre-trained agents to new or updated environments.

Reviewers regarded the method as sound, technically correct, and involving illustrative experiments (although perhaps picking problems too carefully adapted to the solution being presented), and were positive on the general direction of taking inspiration from neuroscience. Reviewer y7rr found the details unclear, recommended more focus on a concrete realization of the general method, and questioned the differences with more traditional hierarchical RL; while many specific inquiries were addressed the reviewer's broad concerns about contextualization remained. Reviewer dDqD had similar concerns around confusing presentation and positioning within the broader literature on "multi-task RL, non-stationary environments, online/continual learning, etc.", and the discussion unfolded similarly -- many specific concerns addressed but fundamental issues remaining. 6ibM, like y7rr, raised the question of why one should stop at 2 policies rather than N policies, noted the under-discussed relationship to options, and questioned the starting point of SAC, and while this was clarified to be about value functions rather than policies, the reviewer still thought this was an ill-justified choice that rendered the system "brittle", and remained unhappy with baseline choices not extending beyond SAC-based agents. tm8g had similar concerns about clarity and in particular that reward engineering seemed central; the authors clarified that this was not the case.

There is wide consensus among qualified reviewers that the presentation (and in particular situating the method with respect to prior work) is inadequate for publication, and I am inclined to agree. As y7rr put it, "evaluating its importance and correctness is hard" without adequate context on the relationships to in particular existing work on hierarchical, multi-task and continual learning. While the direction appears interesting, unfortunately the hard work of contextualizing one's contribution is an utterly essential part of the scientific enterprise; without it we risk retreading well-explored terrain while merely wearing slightly different boots. I encourage the authors to further clarify their presentation incorporating the valuable feedback from the reviewers on this aspect.
Decision: Reject