CAT-SAC: Soft Actor-Critic with Curiosity-Aware Entropy Temperature
Authors: Junfan Lin
Changxin Huang
Xiaodan Liang
Liang Lin
Venue:  (2021)

Abstract: The trade-off between exploration and exploitation has long been a crucial issue in reinforcement learning~(RL). Most of the existing RL methods handle this problem by adding action noise to the policies, such as the Soft Actor-Critic (SAC) that introduces an entropy temperature for maximizing both the external value and the entropy of the policy. However, this temperature is applied indiscriminately to all different environment states, undermining the potential of exploration. In this paper, we argue that the agent should explore more in an unfamiliar state, while less in a familiar state, so as to understand the environment more efficiently. To this purpose, we propose \textbf{C}uriosity-\textbf{A}ware entropy \textbf{T}emperature for SAC (CAT-SAC), which utilizes the curiosity mechanism in developing an instance-level entropy temperature. CAT-SAC uses the state prediction error to model curiosity because an unfamiliar state generally has a large prediction error. The curiosity is added to the target entropy to increase the entropy temperature for unfamiliar states and decrease the target entropy for familiar states. By tuning the entropy specifically and adaptively, CAT-SAC is encouraged to explore when its curiosity is large, otherwise, it is encouraged to exploit. Experimental results on the difficult MuJoCo benchmark testify that the proposed CAT-SAC significantly improves the sample efficiency, outperforming the advanced model-based / model-free RL baselines.

Under review as a conference paper at ICLR 2021
CAT-SAC: SOFT ACTOR-CRITIC WITH CURIOSITY- AWARE ENTROPY TEMPERATURE
Anonymous authors Paper under double-blind review
ABSTRACT
The rrfde-off between exploration and exploitation has long been a crucial is- sue in reinforcement learning (RL). Most of the existing RL methods handle this problem by adding action noise to the policies, such as the Soft Actor-Critic (SAC) (Haarnoja et al. 2018a by that introduces an entropy temperature for maxi- mizing both the external value and the entropy of the policy. However, this temper- ature applied indiscriminately to all different environment states, undermining the potential of exploration. In this paper, we argue that the agent should explore more in an unfamiliar state, while less in familiar state, so as to understand the environment more efficiently. To this purpose, we propose Curiosity-Awant entropy Temperature for SAC (CAT-SAC), which utilizes the curiosity mecha- nism in developing an instance-level entropy temperature. CAT-SAC uses the state prediction error to model curiosity because an unfamiliar state generally has large prediction error. The curiosity is added to the target entropy to increase the entropy temperature for unfamiliar states and decrease the target entropy for familiar states. By tuning the entropy specifically and adaptively, CAT-SAC is encouraged to explore when its curiosity is large, otherwise, it is encouraged to exploit. Experimental results on the difficult MuJoCo benchmark testify that the proposed CAT-SAC significantly improves the sample efficiency, outperforming the advanced model-based model-free free RL baselines.
INTRODUCTION
Deep reinforcement learning (RL), with high-capacity deep neural networks (DNNs), has been ap- plied to solve various complex decision-making problems, including video games (Mnih al. 2015
Vinyals et al. 2019), chess games (Silver et al. 2017 2018) as well as robotic manipulation ( (Kalash-
nikov et al. 2018 However, it is notorious for its high sample inefficiency (Kaiser et al. 2019). Even when solving a simple task, RL needs substantial interaction data to improve the policy. One of the major obstacles for achieving sample-efficienee is the difficulty of balancing exploration and exploitation. The RL. agent needs to explore the environment to collect useful information as well as exploit the acquired knowledge to improve its policy. Most of the existing works either use intrinsic rewards (e.g.. bount-based bonuses (Bellemare al.] (016)) and state prediction error (Pathak et al.) 2017 Burda et al. 2018b) to strengthen exploration, or augment the action value with the entropy to control the proportion of exploration dynamically (Haarnoja al. 2017 Ziebart et al.) 2008). Among these approaches, Soft Actor Critic (SAC) (Haarnoja et al. 201,2018, b) achieves the superior performance on MuJoCo (Todorov al. 2012), OpenAl gym (Brockman al. 2016) benchmark with complex continuous control tasks.
Specifically, SAC aims to maximize expected value while also maximizing the entropy:
n argmnax Σ Eor (=(((-))))),,
(I)
where r, is the environment reward at timestep i, is the state, or is the action, Par is the distribution of trajectory w.r.t policy म, 7L is the entropy of the policy and a is the entropy temperature that weights the importance of the entropy term versus the external reward. The entropy temperature critical SAC since different values determine diverse patterns of agent behavior. Small a may lead the agent to over-optimize the state-action value and develop greedy policy. Due to the lack of

---

Under review as a conference paper at ICLR 2021
exploration, the agent with a small a is likely to get stuck in local optimal. On the other hand, over- optimize the entropy with large a causes the agent to act nearly uniformly and therefore hampers the ability to exploit the environment. To choose reasonable temperature automatically, Haarnoja et al.] (2018b) introduce a target entropy term and extends SAC to a constrained optimization problem, which performs empirically well on MuloCo tasks (Brockman et al., 2016).
However, the target entropy is applied to all transitions equally during training, which neglects the particularity of different states - some states require less exploration while others need more (Tokic) 2010). The particularity of different states for exploration has been discovered for a long time. One of the important particularity the internal curiosity w.r.t. the environment states. Intuitively, when playing a new game, humans make many different attempts at the beginning, and after they under- stand the basic logic of the game, they will not spend extra time trying the things they already know. This implies that better exploration strategy is to explore when the agent is in unfamiliar states and exploit when it in the familiar states. Therefore, optimizing the policy with global target entropy without considering the particularity of different states may instead hamper the exploration of the agent. Researchers use internal curiosity as an auxiliary reward to encourage the exploration of the agent (Bellemare et al. 2016; Tang et al. 2017; Ostrovski et al., (017). Among these methods, Burda et al. (201 8b) propose Random Network Distillation (RND) (Burda et al. (2018)) to remedy the "noisy TV' problem (Pathak et al. 2017 which predicts the output of fixed random network and demonstrates the excellent exploration ability on games with image input. Although |Haarooj ct al. (2018b) have also considered the association between the curiosity and the entropy, they do not apply any relevant guidance to their proposed strategy of automatically tuning entropy temperature.
In order dynamically adjust the exploration-exploiittion strategy regarding the curiosity about states, we make the first attempt to introduce the curiosity to the target entropy term of SAC so that SAC can actively enlarge its entropy at unfamiliar states while reducing its entropy at familiar states. We named our method as CAT-SAC - SAC with Curiosity-A Aware entropy Temperature, by seamlessly introducing the curiosity to augment the target entropy term. Specifically, CAT-SAC first augments the target entropy term by adding the zero mean curiosity. By this means, the agent is encouraged to explore more in an unfamiliar state as its target entropy is large (with positive curiosity) and to explore less at familiar state as its target entropy is small (with negative curiosity). Then, CAT-SAC adopts an instance-level entropy temperature (state)) to replace the original global entropy temperature so to learn different entropy temperatures for different states. The instance- level entropy temperature is curiosity awaar aware as it is supervised by the curiosity augmented target entropy. As for the curiosity model, current advanced methods e.g. RND (Burda et al.) 2018b) may fail to model the curiosity for feature input (e.g., position, velocity). One of the major reasons is that different states with dramatic visual difference may have little difference in feature so that RND has difficulty in distinguishing different state with feature inputs and fails to model the curiosity. To cope with this problem, we propose a novel curiosity model X-RND, which first synthesizes 'unvisited' feature states from the collected data by blending two visited states with random weight entry by entry. With the visited and the 'unvisited' states, X-RND learns to separate the 'unvisited' from the visited by contrastive self-supervised learning (He et al. 2020). By doing so, X-RND successfully prevents the curiosity about the 'unvisited' states from the influence of the visited states.
In summary, the main contributions of this paper are as follows: We propose novel CAT-SAC model, which enables the agent to better trade-off the exploration versus exploitation according to different curiosity about states; To model curiosity for feature inputs. we propose new curiosity model, X-RND, optimized by contrastive self-supervised learning. Experimental results testify that the proposed method significantly improves the sample-efficiency on complex and difficult con- tinuous control tasks of the MuJoCo benchmark against the state-of-art model-based / model-free methods.
2 RELATED WORK
Exploration an essential issue for effective reinforcement learning (Kakade & Langford 2002). and the problem of exploration has been widely studied (Wunder et al. 2010 Haarnoja et al.
20]8a Pathak et al. 2017 Lee et al. 2020: Zhang et al., 2020 Sekar et al., 2020 Stadie et al,
When predicting the dynamics, the curiosity model is prone to produce a large error w.r. noisy observation, resulting in a large curiosity.
2

---

Under review as a conference paper at ICLR 2021
2018
Nachum et al.
2019). The main researches exploration include noise-based exploration
Wunddrr et al. 2010 Tokic 2010, Mnih et al., 2015: Lillicrap et al. 2013), maximum entropy of policy exploration (Harmojj al.) 2017 201 Sa) and intrinsic bonus exploration (Pathak et al. 2017 Burda et al., 2018a| Achiaam & Sastry 2017, Burda et al. 2018b). Noise based exploration aims balance the ratio of exploration/expootation by introducing random policy noise. The most relevant to CAT-SAC is the maximum entropy exploration and the intrinsic bonus.
Maximum Entropy RL. Maximum entropy RL optimizes policies by maximizing the expected return and the expected entropy of the policy simultaneously (Haarnoja et al., 2017) (Haarnoja al. 2018a). This framework has been used in many fields. such as reverse reinforcement learning Ziebart et al. (2008) and optimal control (Roy et al. 2013).. Recently, the maximum entropy frame- work has been successfully incorporated into Q-learning and policy gradient methods improve sample efficiency, such as soft Q-learning algorithm (Haarnoja et al., (017) and soft actor-critic (SAC) algorithm (Haarnoja et al., 2018a 2019). Soft Q-learning learns stochastic energy-base -based policies with approximate inference via stein variational gradient descent (SVGD) (Liu & Wang 2016). However, approximate inference procedures become complex in continuous action spaces. Haarnoja al. (2018)) present convergence proof for policy iteration in the maximum entropy framework, known as soft actor-critic (SAC), which avoids the complexity and potential instability associated with the approximate inference of soft Q-learning. [Hаarooja et al. (2019) further pro- pose an extension to the soft actor-critic algorithm that the temperature parameter can be optimized adaptively.
Intrinsic Bonus. Intrinsic bonus exploration methods encourage the agent to explore by introducing an intrinsic bonus an auxiliary reward. One of the common intrinsic bonuses is defined based on visit-count that indicates how many times the agent performed action a at state s (Bellemare et al.) 2016
Tang et al., 2017: Ostrovski et al., 2017; Fox et al., 2018). Another class of intrinsic bonus calculated according to errors of predicting dynamics (aathak et al.] 2017] [Burda et al., 2018a] Achiam & Sastry, 2017, Houthooft et al.] 2016). They first learn forward dynamics to predict the next observation given the current observation and agent's 's action. The prediction error represents the agent's ' familiarity about the state. These approaches focus on maximizing the prediction error as intrinsic reinforcement learning objective. However, maximizing such prediction errors of for- warding dynamics will tend to cause the agent to be attracted to stochastic transitions (Burda et al.) 2018b), e.g., the "noisy TV". |Burda et al. 2018b) propose random network distillation (RND) bonus to handle this undesirable stochasticity. RND predicts the output of a fixed randomly initialized neu- ral network based on the current observation. Therefore, the predicted output deterministic.
Even though intrinsic bonus methods have obtained a signifi- cant improvement on (semi-)sparse tasks, they also introduce exploration biases the policy, which might cause the policy to improve in the opposite direction against the original reward function. Such a "misdirected" problem becomes even severe on dense-reward tasks since the original policy could have performed much better. Therefore, there are also plenty undirected exploration methods, such as e-greedy (Sutton & Barto 2018) and posterior sampling (Osband al., 2013 Osband & Van Roy] 2015 Kurutach et al., 2018 Wang & Bal 2019e.These methods focus on adjusting the variance of policy to explore better. As studied (Tokic. 2010), increasing policy variance at unfamiliar states cause the policy to explore better. Similar ideas are also be observed in bootstrapping methods (Os- band et al. 22016; 2018) and upper-bound-comee cooudd-cofiieecc exploration methods (Ciosek et al., [2019] Lee al. (020). which tend to have larger action variance at uncertainty states. In this paper, propose to introduce the unfamiliarity of states to automatically tune the entropy temperature to improve the exploration of SAC.
Undirected Exploration Strategy.
3 PRELIMINARIES
In this section, we introduce the background and annotations of reinforcement learning and the two key ingredients of our method, i.e., SAC (Haarnoja et al. 2018a b) and RND (Burda et al., 2018b).
3.1 REINFORCEMENT LEARNING
We consider a standard RL framework where an agent interacts with an environment according to the observation in discrete time. Formally, the agent observes state 8t (e.g., position, velocity)
3

---

Under review as a conference paper at ICLR 2021
from the environment each time step t and then chooses action de according its policy л. The environment returns a reward 7t and the next state $141 ~ Pe Qe). where Pe is the environment transition mapping. The return of trajectory from timestep is 7r -k= 7 where y E [0. 1) is the discount factor. RI aims to optimize the policy to maximize the expected return from each state 8t.
3.2 SOFT ACTOR-CRITED
SAC (Haarnoja et al. 2018a) is off-policy actor-critic method based on the maximum entropy RL. framework (Ziebart et al. 2008), which aims to maximize both the return and the policy entropy, as described in Equ. (T). uses a neural network with parameter for Q-value Qu. and uses anther neural network parameterized by & for the policy Td. To update these parameters, SAC alternates between a soft policy evaluation and a soft policy improvement. At the soft policy evaluation step, 8 is updated by minimizing the following cost function:
Leritie (0) Es, anD [11Q ac) - Qtarget (st, ()))
where Q Qtarget (st, as) = re +
Pet ((L+11 - a log T (a
(3)
where an experience replay buffer, a is the entropy temperature, and Q is the target Q function whose parameters are periodically updated from the learned Qo. At the soft policy improvement step, the policy with its parameters ø is updated by minimizing the following objective:
Cacror (0) Be Ja log (a - Qo(st. at)].
(4)
Here, the policy models Gaussian with mean and variance to handle continuous control problems. In order to automatically tune Haarnoja et al. (2018b) introduces a constraint target entropy H, so that a is learned to meet the constraints during the whole training process:
((a) = Bay. log (arlse) + R2)]-
(5)
3.3 RANDOM DISTILLATION NETWORK
Random Network Distillation (RND) Burda al. 2018b), prediction-basee method for encour- aging exploration through curiosity, which manages to exceed average human performance on Mon- tezuma's 's Revenge (Badia et al., 20200) The RND exploration bonus is defined as the state prediction error of a neural (st) predicting features of the state given by fixed randomly initialized neural network f(xx). The exploration bonus and the loss function are:
cw (8t) = (x)) - [())]1 and L(w) = Ee,,-(((((()
(6)
In unfamiliar states, it's hard for fw, to predict the output of f. and hence the prediction error c(st) is high. By measuring how hard it is to predict the output of f on these states, RND incentivizes visiting unfamiliar states.
CAT-SAC
Curiosity-Awar entropy Temperature for Soft Actor-Critic. In the design of CAT-SAC, we integrate three key ingredients, i.c.. the curiosity augmented target entropy, the instance-level entropy temperature, and the curiosity model X-RND for feature input, into a single framework.
In this section, we present CAT-SAC:
4.1 CURIOSITY AUGMENTED TARGET ENTROPY
Haarnoja et al. 2018b) 8b) reformulates SAC from the maximum entropy RL framework to a constraint satisfaction problem by introducing the target entropy to tune the entropy temperature automatically. However, applying the target entropy to all transitions neglects the particularity of different states, ie.. unfamiliar state often requires more exploration. To enable the agent increase the entropy
4

---

Under review as a conference paper at ICLR 2021
when it's in unfamiliar state, we inject the curiosity to the target entropy. Specifically, first normalize the curiosity c(s) w.r.t state s, and then add it to the original target entropy H:
h(s) H. c(s) P
where j and are the running mean and running standard deviation of c(s) respectively. From the Eq,.(7), the expectation of the proposed curiosity augmented target entropy E.[h(s)] = H. therefore the new target entropy is consistent with the original one in expectation. In this manner, the target entropy in an unfamiliar state is pushed above the expectation and thus the agent is encouraged to explore more in that state; the contrary, the target entropy in the familiar state is pressed down to regulate the behavior of the agent to exploit efficiently.
4.2 INSTANCE-LEVED ENTROPY TEMPERATURE
The entropy temperature is the most crucial component in either SAC or CAT-SAC, as it is the bridge connecting the target entropy and the policy. Therefore, to really cast different target entropy to the policy at different states, we need to turn the original global entropy temperature a into an instance-level entropy temperature (s))), whose is:get
argmin E. 6
log (a(a))))) h(s))))
(8)
we can observe that if is state-independent, then the expectation term in Equ.(8) will be equivalent to the original of Equ.(5). In this case, the effect of the curiosity augmented target entropy will not be reflected in the policy as its expectation equal to the original target entropy.
From Equ.
However, assigning different entropy temperature for each individual state independently makes the optimization of Equ. trivial: the entropy temperature (sg(s;) become either extremely high or extremely low according to the sign of log T (a ( +s,,). It further over-weights log T (a.(ss)) of Equ. which influences back the optimization
Equ. The consequence of these optimization processes is that - log T ( 8x) is optimized to approach h((se) rapidly. As mentioned by Haarnoja ct al.] (2018b). forcing the entropy to be a target entropy will hamper the flexibility of the policy. To prevent the (st) from doing so, instead of st,
the curiosity c(s:) the context of the entropy temperature, which projects different states to compact values so as to cluster those states with the same level of unfamiliarity share the same entropy temperature:
as(si) g5(c(st)),
where Is linear layer with parameter 5. More details about the Is is presented in Appendix. To further alleviate the problem mentioned above, we project the zero-mean c(st) into its closest integer to reduce the space of the entropy temperature. One can treat our method as a stratified SAC regarding different discrete curiosity levels.
4.3 X-RND BASED ON SELF-SUPERVISED CONTRASTIVE LEARNING
In our framework, the curiosity values of familiar states should relatively smaller than those of unfamiliar states. BBurda et al. (2018b) have demonstrated that RND works well in modeling curios- ity for hard exploration tasks with image input. However, we found that RND works poorly with feature input. Especially for the features of unvisited states that are close to the visited states', RND is likely to produce low prediction errors. Therefore, the curiosity model based on state prediction error like RND, besides minimizing the prediction error Equ.(6) of the visited states, may need extra supervision to learn how to maintain the prediction error of the unvisited states to a certain extent. To this end, we develop X-RND, which includes an additional contrastive loss (He al., 2020)] Schmidhuber 2015) to push the curiosity values of potential unseen states s' above a margin value. The loss function of X-RND is formulated as:
((w) E Eg, (s)(s) 3maxxmm (s))
(10)
where is the curiosity margin for unvisited states and 3 is the hyperparameter that weighs the importance of the contrastive loss term. order to obtain the unvisited states s', X-RND constructs the potential unvisited states from the visited states. Specifically, X-RND randomly samples set of
5

---

Under review as a conference paper at ICLR 2021
CARANC
0900
(a) Ant
(b) Cheetah
(C) Walker
(d) Hopper
Figure 1: Evaluation curves of SAC HHarnojj et al., 2018 and CAT-SAC (a) Ant, (b) Cheetah, (c) Walker, and (d) Hopper environments. The solid line and shaded regions represent the mcan and the standard deviation across six runs, respectively.
states from the replay buffer, and then randomly fuses any two different states with different weights.
Formally, 8' = t (82-81). where 81, 82 ~ D, 81 * 82 and ~ [0 1.5] [s| is the dimension of state feature and (1(0, 1.5]]=| is the |s] |s]-dimensional uniform distribution with range [0. 1.5] By doing so, the synthesized 'unvisited" states still obey the similar statistical characteristics as the visited states 3] By pushing the prediction errors of these self-constructed states above the curiosity margin, X-RND significantly improves the performance of the curiosity model with feature inputs, as demonstrated in the next section.
5 EXPERIMENTS
We begin by demonstrating the effectiveness of the curiosity-aware exploration strategy on difficult continuous control OpenAl Gym environments (Brockman et al., 2016) Sec. [.. against current advanced model-based / model-free baselines. To better understand the benefit of the proposed curiosity model for feature inputs, we design a minimal grid-world environment to provide a con- cise and straightforward comparison between X-RND and RND in Sec. 5.2 After that, we ablate different choices of the important components and hyperparameters of CAT-SAC in Sec. 5.3
5.1 MuJoCo RESULTS
CAT-SAC is evaluated on four continuous control tasks from OpenAI Gym (Brockman al. 2016).
ie.. Ant, Walker, Cheetah as well as Hopper. Following the same experiment settings of SUN- RISE (Lee et al. 2020). we compare CAT-SAC with METRPO (Kurutach et 2018), combina- tion of TRPO (Schulman et al.) 2015) and ensembles of dynamics models; PETS (Chua et al.) 2018),
an advanced masel- (MB) RL method based on ensembles of dynamics models: POPLIN- (Wang & Bal (019), a of-e-oof-the-at MBRI. method; POPLIN-A (Wang & Ba 2019). a variant of POPLIN-P by injecting noise to the action space. We also compare CAT-SAC with three state- of-the-art model-free RL methods, TD3 (Fujimoto et al.) 2018), SAC (Haarnoja et al., (2018) and SUNRISE (Lee et al., 2020). For our method, we focus on the comparison between SAC and CAT-SAC. The whole algorithm for CAT-SAC is described in Alg. [ in the Appendix. Following the experiment setups in SUNRISE, the curves and results in our experiments are calculated after 200K timesteps. Each experiment is conducted four times with the mean and the standard deviation reported. More experimental details are presented in the Appendix.
As shown in Tab. T CAT-SAC outperforms all of the baselines all tasks. As for the performance on Cheetah and Walker, CAT-SAC makes a breakthrough that exceeding 6000 and 2000 on aver- age, respectively, within 200K environment frames. On tasks Ant and Hopper, we observe that the performance of SAC and CAT-SAC almost coincides at the early stage, however, CAT-SAC gets
With upper bound larger than 1, the value of the synthesized state be beyond the range of the visited states.
AAthoogh a few synthesized states possibly exist in the visited states, it would not have great impact since the visited states are much more likely to be sampled during training that their prediction error would be pulled back in expectation.

---

Under review as a conference paper at ICLR 2021
Table 1: Performance on OpenAl Gym 200K timesteps. The results show the mean and standard deviation across six runs Henderson et al.] (2017). For baselines, we report the best number in SUNRISE (Lee al.
2020). The best results are highlighted in bold.
Cheetah
Walker
Hopper
Ant
METRPO (2018)
2283.7 + 900.4
-1609.3 + 657.5
1272.5 500.9
282.2 18
PETS 2018
2288.4 + 1019.0
282.5 501.6
114.9 + 621.0
621.0
1165.5 + 226.9
POPLIN-A 2019)
1562.8 1136.7
-105.0 249.8
2
1148.4 438.3
POPLIN-P 2019)
4235.0 = 1133.0
=
2330.1 + 320.9
TD3 (2018)
3015.7 + 969.8
-516.4 812.2
1816.6 994.8
870.1 283.8
SUNRISH (2020)
5370.6 + 483.1
1926.2 + 694.8
2601.9 306.5
1627.0 + 292.7
SAC (2018b)
5470.9 600.8
1419.3 = 364.8
1341.2 = 694.9
1055.3 = 292.5
CAT-SAC
6159.7 865.6
2666.2 + 633.0
3011.1 + 274.1
2444.8 + 856.3
92
0.
0.16
0.16
011
0.14
0.1
001
0.04
19 20
30
10
30
#Visited
Avisited stile
(a) Swiss Roll Maze
(b) RND
(c) X-RND
Figure 2: Demonstration on (a) the Swiss Roll maze environment, (b) prediction prediction error of X-RND. The agent starts from the blue block and moves to the green block. (b)(c) shows the state prediction errors of all movable blocks (y-axis) given the number of visited blocks (x-axis) to the agent.
of RND (c)
away from the flat slope quickly and improves faster, which implies CAT-SAC is more capable of exploiting acquired knowledge and execute efficient exploration. In comparison with all base- lines, CAT-SAC is more conpute-efficiet since it does not require any ensemble of dynamics and planning during the evaluation phase.
5.2 X-RND VERSUS RND WITH FEATURE INPUTS
We present simple example to highlight the effectiveness and robustness of the proposed curiosity model X-RND for feature input. introduce a grid-world environment, Swiss Roll maze, as de- picted in Fig. 2a The agent is moving from the blue block to the green block. Every time the agent arrives at a new block, both RND and X-RND train on features (i.c., position) all visited blocks. Since aim to analyze the problem of RND with feature inputs and the advantage of X-RND, simply adopt the agent with the optimal policy for moving. We record the curiosity values of all movable blocks of both methods w.r.t. the amount of unique visited states and plots them in Fig. 2b and Fig. 2c respectively. More experimental details are presented in the Appendix.
In Fig. 2bland and Fig. 2c values of each present the curiosity values of all visited unvisited. The curiosity values are clipped within the curiosity margin (m = 0.2 for this toy experiment). Ideally, values in the upper triangle of the figure should be closed 0, while values in the lower triangle should be closed to the curiosity margin. From Fig. 2b)and and Fig. 2c we can observe that, when the agent visits 10 unique states (index at x-axis), RND has faultily regarded the distant and unvisited states (index from 30 to the last at y-axis) as visited states and produces low prediction errors for them; while X-RND successfully sustains the curiosity for these states. more appealing property of X-RND is that X-RND is more like to form sharp contrast between the explored and unexplored areas (with a clear diagonal). It is more important to have accurate curiosity prediction for the near unvisited states than keeping the curiosity about the very distant state high, since those states are more likely to be visited. This phenomenon implies the synthesized states are likely to fall the border between the explored and unexplored.

---

Under review as a conference paper at ICLR 2021
3500
0.2
3500
-15
3500
0.2
3500
15
3000
3000
5000
Average return
0.0
Avernge retumn
Average relurn
0.00
2500
2500
2500
2000
2000
Avernge
1500
1500
1500
1500
1000
1000
1000
500
500
500
0
0
50
100
150
50
100
150
50
100
150
50
100
1:0
Enviromment steps (K)
Environment steps (K)
Environment steps (K)
Environment steps (K)
(a) Ant (3)
(b) Ant (172)
(c) Hopper (3)
(d) Hopper (m)
Figure 3: Evaluation curves of CAT-SAC with different values of hyperparameters and m. The
solid line and shaded regions represent the mean and the standard deviation across four runs.
CARSAC 304
CAREAN RCM
35D
CATENE
2403
80
(a) Ant (Comp.)
(b) Hopper (Comp.)
(c) Ant (Bonus)
(d) Hopper (Bonus)
Figure 4: Evaluation curves for CAT-SAC (a) Ant and (b) Hopper tasks. The solid line and
shaded regions represent the mean and standard deviation, respectively, across four runs.
5.3 ABLATION STUDY
In this section, ablate the effects of different values of the hyperparameters as well as different components of CAT-SAC on Ant and Hopper, which should be representative of all tasks.
Effects of different hyperparameters. We analyze the effects of different values of the hyperpa- rameters CAT-SAC, i.e., the curiosity margin m and the importance weight / in Equ. Notice that when 3 equals O, the X-RND in SAC is the same as RND. As depicted in Fig. Ba and Fig. Bc CAT-SAC with positive 3 achieves better performance on both tasks, which means that better cu- riosity model X-RND indeed help improve the performance of CAT-SAC. However, we also observe that the larger 3 is not necessary to bring more improvement. We suppose the reason is that large importance weight on the contrastive loss may cause the RND loss for familiar states difficult to decrease. As for the choice of the curiosity margin m, we found that CAT-SAC is more sensitive to different values of m on task Ant (Fig- 3b). The reason might also is that m determines the range of the curiosity and therefore influence the distribution of c(s). which is the critical component in CAT-SAC. But overall, with different values of 3 and m, CAT-SAC still outperforms SAC, which implies that our method consistently gains improvement.
Individual effect of each component. In our full CAT-SAC, curiosity from X-RND is discretized into integer and normalized to have zero mean, and then added to the original target entropy. To evaluate the effect of each component, we conduct experiment on i) real, which does not discretize the curiosity 4.2 ii) RND/ICM, which use RND/ICM to predict curiosity; (iii) rev, which subtracts the curiosity from the target effect. Among these settings, only rev is contrary to our motivation, that's. the agent should explore more in unfamiliar states. Fig. 4a and Fig. F show the performance of CAT-SAC under different settings. From the results, we observe that turning off any component would decrease the performance of the full version. And except for rev, CAT-SAC with different component turning off still outperforms the baselines with a sharp margin. It implies that our moti- vation actually helpful in developing a better all the proposed techniques, we obtain the best performance in all environments. From this compar- ison, we see that all proposed components have a positive effect on improving the cample-fffciencyy of CAT-SAC. Besides, we also conducted experiments that only augments SAC with intrinsic bonus, ie., RND and ICM, to understand the performance of these methods. From the results shown in Fig. 4c and Fig. 4d found that the performance of either RND ICM is diverse on different
trade-off strategy. By utilizing
8

---

Under review as a conference paper at ICLR 2021
CAUSAC
750
3km DATSNG $4C
140
(a) Bipeda/Walker
Bestrrmmnne (10X)
maps
(b) 5-step interval (mild)
(c) 10-step interval (difficult)
Figure 5: Evaluation curves for different approaches on (a) 5-step interval (mild) and (b) 10-step in- terval (difficult) tasks. The solid line and shaded regions represent the mean and standard deviation, respectively, across six runs.
tasks. Especially task Ant, these methods seem to lead the policy to improve in the opposite direction, hampering the sample efficiency.
5.4 THE PERFORMANCE UNDER SPARSE-REWARE SETTINGS
On sparse-reward tasks, since the agent could not receive immediate feedback from the environ- ment, they usually stuck in a local optimal. Intrinsic bonuses (c.g. RND (Burda et al. (2018b) and ICM (Pathak et al. 2017 adjust the target of the agent by augmenting the environment reward with intrinsic prediction errors to encourage to form a meaningful and directional exploration behavior. Different from these methods, our methods apply curiosity in adjusting the variance of the policy, which still obeys the original target of SAC. Thus, similar to SAC, our method is not expected to solve searse-reward tasks. Nevertheless, it is still interesting to see whether tuning the action entropy at different states w.r.t. curiosity can still gain improvement on sparse-reward tasks.
To this end, we adopt classical Box2d (Catto 2011 task, BipedalWalker (Fig- Saj. which is quired to learn an agent to move ahead as far as it can. In our experiments, we only reward the agent if the agent reaches the anchor locations. The interval distance between the two closest anchor locations determines the difficulty of the task. In the revised BipedalWalkerr the agent is required learn to move without any feedback on its posture, and only when reaching the anchor locations will the agent be rewarded. For more environment and training details, please refer to Appendix.
In our experiments, we design two tasks with different intervals. They are 1) 5-step interval: mild exploration task with 5-step-length interval; 2) 10-step interval: difficult exploration task with 10-step-length interval. From Fig. [5b] we observe that on the mild task, both SAC and CAT- SAC can obtain the optimal effortlessly and CAT-SAC improves faster. However, we increase the interval to steps, both undirected exploration methods SAC and CAT-SAC perform poorly. However, CAT-SAC can sometimes get rid of the local optimal and achieves better performance in expectation. On this difficult task, we also conduct experiments with directed exploration methods, i.e. RND and ICM. As expected, these methods achieve significantly better performances shown in Fig. 5c| While, as discussed in the last section, introducing exploration biases like RND and ICM not necessary to improve the sample efficiency, especially on dense-reward tasks.
CONCLUSION
In this paper, we present CAT-SAC, a simple framework unified curiosity and entropy maximization RL. In particular, CAT-SAC encourages a large entropy in unfamiliar states and small entropy in familiar states for better trade-off between exploration and exploitation. To model curiosity for feature input, also propose
prediction-based model, X-RND. Our experiments show that the full CAT-SAC consistently improves the performances of SAC and outperforms state-of-the. RI. algorithms in complex control tasks.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Joshua Achiam and Shankar Sastry. Surprise-based intrinsic motivation for deep reinforcement learning, arXiv preprint
2017.
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. arXiv preprint ar wrXv:2003. 2020.
Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in neural information processing systems, pp. 1471-1479, 2016.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint orXiv:1606.01540. 2016.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros Large-scale study of curiosity-driven learning. In International Conference Learning Repre- sentations, 8a.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. International Conference on Learning Representations, 2018b.
Erin Catto. Box2d: 2d physics engine for games. URL: http://www. box2d. org (18.11. 2019), 2011.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement leamn- ing in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pp. 4754 4765, 2018.
Kamil Ciosek, Quan Vuong. Robert Loftin, and Katja Hofmann. Better exploration with optimistic actor critic. In Advances in Neural Information Processing Systems, pp. 1787-1798, 2019.
Lior Fox, Leshem Choshen, and Yonatan Loewenstein. Dora the explorer: Directed outreaching reinforcement action-selection. In International Conference on Learning Representations, 2018.
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in
actor-critic methods. arXiv preprint arXiv: 10Xiv:180.00947, 2018.
Tuomas Haamoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. arXiv preprint arXiv: (002.08770.8166 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-eritic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning, pp. 1861-1870, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv:1812.0500 2018b.
Tuomas Haarnoja, Sehoon Ha, Aurick Zhou, Jie Tan, George Tucker, and Sergey Levine. Learning to walk via deep reinforcement learning. In Proceedings of Roboties: Science and Systems, FreiburgimBreisgauu Germany, June 2019. doi: 10.15607/RSS.22199XXMM
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. arXiv preprint arXiv:/709 779700906660 2017.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck. and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pp. 1109-1111 1177, 2016.
10

---

Under review as a conference paper at ICLR 2021
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. arXiv preprint
00374, 2019.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In ICML, volume 2. pp. 267-274, 2002.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint urXiv:1806.10292. 2018.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint arXiv: 10592, 2018
Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: simple unified frame- work for ensemble learning in deep reinforcement learning. arXiv preprint arXiv:2007.04938 2020.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint
2015.
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances in neural information processing systems, pp. 2378-2388 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 5 518(7540::5295533 2015.
Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine Why does hierarchy (sometimes) work so well in reinforcement learning? arXiv preprint arXiv:1909. 10618, 2019.
Ian Osband and Benjamin Van Roy. Bootstrapped thompson sampling and deep exploration. arXiv preprint S.Kiv:5577000000 2015.
Ian Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. In Advances in Neural Information Processing Systems, pp. 3003-3011, 2013.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29:4026-4033 2016.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. 2018
Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International Conference Machine Learning, pp. 2721-2730, 2017.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16-17, 2017.
Nicholas Roy, Paul Newman, and Siddhartha Srinivasa. On stochastic optimal control and reinforce- ment learning by approximate inference. 2013.
Jürgen Schmidhuber. Deep learning neural networks: An overview. Neural networks, 61:85-117, 2015.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
Ramanan Sekar. Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. arXiv preprint arXiv:2005.05961 2020.
11

---

Under review as a conference paper at ICLR 2021
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-333 359-359, 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self self-play. Science, 362(6419):1140 1144, 2018.
Bradly C Stadie, Ge Yang, Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever. Some considerations on learning to explore via meta-reinforremmnt learning. arXiv preprint arXiv:1803.01111 118, 2018.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schul- man, Filip DeTurek, and Pieter Abbeel. exploration: A study of count-based exploration for deep reinforcement learning. In Advances in neural information processing systems, pp. 2753- 2762, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Michel Tokic. Adaptive e-greedy exploration in reinforcement learning based value differences. In Annual Conference on Artificial Intelligence, pp. 203-210, Springer, 2010.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czameeki, Michaël Mathicu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-334 2019.
Tingwu Wang and Jimmy Ba. Exploring model-based planning with policy networks. arXiv preprint arXiv:1906.08664 2019.
Michael Wunder, Michael L Littman, and Monica Babes. Classes of multiagent q-learning dynamics with epsilon-greedy exploration. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 1167-1174. Citeseer, 2010.
Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value dis- agreement. arXiv preprint arXiv:2006.09641 2020.
Brian Zicbart, Andrew Maas, Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1433 Chicago, II, USA, 2008.
12

---

Under review as a conference paper at ICLR 2021
A CAT-SAC: EXPERIMENTAL SETUPS
Environments. We evaluate the performance of CAT-SAC on four complex environments from OpenAl Gym (Brockman et al. 2016), Ant, Cheetah, Walker, and Hopper. Observation in these tasks is in the form of features.
Training details. We consider combination of SAC and CAT-SAC using the publicly caaptuuub.ooom/uumm/iaiaattt
released implementation repository tions hyperparameters and architectures in order match the implementation of RLKit http://ittuub.oom//icchh//e/eeee
with slight modifica-
The complete algorithm is depicted in Alg. [ The detailed hy- perparameters used in CAT-SAC is presented in Tab.
Algorithm Soft Actor-Critic with Curiosity-Aware Entropy Temperature
1: Input: initial policy parameters Φ, Q-function parameters 01, 02 empty replay buffer 'D
2: Set target parameters equal to main parameters 0 t- 01, Ouag. 2 + 02
3: repeat
4: Observe state and select action ~ TT(-1s))
5: Execute a, in the environment
6: Observe next state 8t+1, reward and done signal de to indicate whether 8 is terminal
7: Store (8t, de; Tt. 8541, de) in replay buffer
8: if de is true then
9:
Reset environment state.
10:
Sample set of states {s} from D and sythesize set of states {s'}
11:
Update fu by optimizing Equ.
12:
Calculate the mean f and standard deviation of {c(s))
13:
end if
if it's time update then
15:
Randomly sample a batch of transitions, B = {st, rz, St+1, d)} from D Calculate c(st) and (s(ss) for the rest optimizations
17:
Update Q-functions by one step of gradient descent using Vo, To (0;) for i=1,2
18:
Update policy by one step of gradient ascent using V [ Σ.EB Lactor
19:
Update target networks with Orarg, + porang + (1 p)0, for i=1,2
20:
Construct curiosity augmented target entropy h(sr)) by Equ.
21:
Update as by one step of gradient descent using V. L())
end if
23:
until convergence
Details of the instance-level entropy temperature. order automatically tune the entropy temperature in a similar way as (Haarnoja et al.) (0018)). we do not adopt deep neural networks for the instance-level entropy temperature. Instead, we initialize list of scalar variables. Each variable is assigned to integer index, ranging from - Iarrget entropy| to |target entropy]. Given the discrete curiosity input, we match it with the variable with the closest index, and then the same optimization process in (Haarnoja et al. 2018b) is conducted to optimize these variables. As for the input under the real setting in Sec. 5.3 we match with the two variables with the closest indexes and calculate the weights of these variables
that the sum of the weighted indexes assembles the curiosity value. Then the sum of the weighted variables is treated as the entropy temperature. The implementations of different approaches depicted in Fig.
13

---

Under review as a conference paper at ICLR 2021
S2
St
C2
Key:
-1/42
1122
N-1
4
+++1
-1:42
7:-2
R-1
Parameter:
loga
logar
loga
Vanilla
Real value
Discrete value
Figure 6: The sketches of log a. The left one is the implementation of vanilla SAC (Haarnoja al, 2018b). And the others are proposed in our paper to model instance-level entropy temperature. The middle one explain how we handle real-value curiosity. And the right one first rounds the curiosity into the closest integer and then choosing the matched entropy temperature.
Table Values of hypeparaantters used in this paper.
Hyperparameter
Value
Hyperparameter
Value
#training frames
2e7
#frames between evaluations
le3
Backbone
MLP
Non-lincarity
ReLU
Backbone (g)
Vector
(X-) X-)RND feature size
128
Hidden layers
2
Optimizer
Adam
Hidden layers ((X))RND) f)
4
Critic target update freq
Hidden units
256
Learning rate
3e-4
Batch size
256
Discount y
0.99
Initial temperature
Curioisty margin m
10
#batch for update (X-)RND
100
Importance weight 3
0.1
Polyak factor 7 Replay buffer size
0.005
Target entropy
-| action space
le6
Evaluation episodes
10
Hidden layer (S.R.)
1
Hidden units (S.R.)
16
Importance weight (S.R.)
0.5
(X- RRD feature size (S.R.)
16
Curiosity margin (S.R.)
0.2
Batch size (S.R.)
Hidden layer (Bipedal.)
2
Hidden units (Bipedal.)
16
Importance (Bieedl)))
(X-)RND feature size (Bipedal.)
Curiosity margin (Bipedal.) m
0.3
Batch size (Bipedal.)
128
B SWISS ROLL MAZE: EXPERIMENTAL SETUPS
Environment. As depicted Sec. 5.2 In this environment, only position features are provided.
Training details. Most of the hyperparameters take the same values as CAT-SAC, presented in Tab. For some hyperparameters with different values, we tag them with '(SRR)) in the table. The visualization results of the prediction error of RND and X-RND presented in Fig, and Fig, g respectively. From these figures, we can observe the X-RN R-ND can maintain the curiosity about distant states with feature input.
14

---

Under review as a conference paper at ICLR 2021
Figure 7:
The prediction errors of all blocks of RND. These figures are obtained by using RND to predict the curiosity about all states after training with different numbers of visited states. From left to right and from top to bottom, the number of visited states is increasing.
Figure 8: The prediction enrors of all blocks of X-RND. These figures are obtained by using X-RND to predict the curiosity about all states after training with different numbers visited states. From left to right and from top bottom, the number of visited states is increasing.
C BIPEDALWALKKRR: EXPERIMENTAL. SETUPS
Environment. We adopted the open-released BipedalWalker and revised it to the sparse-reward task by removing its original reward function. The revised version rewards the agent the agent reach predefined anchor locations. These locations placed along the track evenly with fixed interval. Since there are no posture reward, an agent required to explore how to walk to these anchor locations. We also augmented the original state feature with the hull's location (x, y) of the agent.
Training details. Most of the hyperparameters take the same values as CAT-SAC, presented in Tab. For some hyperparameters with different values, we tag them with 'Biipdall.' in the table.
15


Reviews:
[{"review_id": "r4wNW3w6VD7", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces an algorithm that augments SAC with Curiosity-Aware Temperature, to enable more efficient exploration. Previous versions of SAC had a fixed entropy temperature which had to be tuned or an automatic tuning mechanism that was not state-specific. The paper proposes that exploration can be improved if temperature is state-dependent, based on curiosity, or unfamiliarity of the state.\n\nAuthors introduce curiosity to the target temperature such that the entropy is large in unfamiliar states, promoting exploration, and small in familiar states, encouraging more exploitation. To enable this, the authors introduce three components: 1) target entropy that is augmented with curiosity, 2) curiosity and hence state based entropy, 3) X-RND that adds contrastive loss to ensure more robust computation of curiosity.\n\nCuriosity is based on prediction error of states, using the idea from Random Distillation Network (RND) (Burda et al. 2018b). This curiosity term is normalized such that in expectation it corresponds to the original target entropy. The instance-level temperature also uses this curiosity to map states with similar level of unfamiliarity to similar temperature value. X-RND is a technique that the authors develop in order to overcome previous difficulties that RND had on feature inputs. \n\nIn their benchmark experiments they show that their method CAT-SAC shows superiority compared to SAC as well as other baseline methods. They also show results on a toy domain how X-RND can estimate curiosity more robustly than RND.\n\nThe authors have done a fair job introducing curiosity to enable better exploration by varying the target entropy at state-level. Disregarding minor grammar errors I think it is structured nicely. The idea is interesting but I would recommend reject as the experiments need to be conducted more rigorously. \n\nBenchmark experiments are conducted only with 4 runs on Mujoco environments that are known to have high variance based on random seeds. Henderson et al. 2018 (https://arxiv.org/pdf/1709.06560.pdf) show that on HalfCheetah, the same algorithm can have significantly different performance, between two groups of 5 random seeded runs. More runs need to be conducted in order to show credible performance improvements.\nFurthermore, currently CAT-SAC surpasses SAC in all four Mujoco domains at 200k steps of training, but SAC results in the original paper (Haarnoja et al. 2018b) show SAC reaching avg return of 2000 in Hopper and 3200 in Walker. This also shows that current results where SAC performs worse than CAT-SAC could have been due to variance in random seeds.\n\nComparison of CAT-SAC to other baseline method performance seems less appropriate too. Results are based on SUNRISE paper (Lee et al. 2020) which has not been peer-reviewed and which have also done only 4 runs each. \n\nLastly, demonstration of X-RND seems to show that it is able to remember states it has visited and keep curiosity for remaining unvisited states. It looks like after the state is visited once the curiosity drops from 0.2 to 0.02 immediately. I’m not sure if it would be desirable to have curiosity drop suddenly after visiting it only once. To me X-RND seems like a mechanism with a replay buffer that limits neural networks from generalizing and making it tabular-like to output low curiosity only for states it has visited. X-RND certainly performs better than RND in this toy domain but I’m not sure if it adds that much value to the overall paper, where it is about having instance-level entropy that encourages exploration.\n\nFor other minor details, I think the plots and labels in Figure 2(b) (c)  are confusing.I think the x-axis should be index of each state, and prediction error of each state the label for the colormap. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "khvXYitj567", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n\nThis work proposes to incorporate curiosity into the entropy temperature of Soft Actor-Critic, and applies a modified version of Random Network Distillation as their curiosity model. Its key insight is that the entropy temperature in Soft Actor-Critic should encourage the agent to explore unfamiliar states more and familiar states less, rather than globally encourage some expected target entropy. To this end, they make the target entropy and temperature both state-dependent, using a curiosity model based on Random Network Distillation. \n\n\n\nStrengths: \n\n- The insight is described well and motivates the need for a curiosity-aware entropy temperature for further sample-efficiency.\n\n- The paper demonstrates good experimental results across four OpenAI Gym tasks evaluated after 200K environment steps. There is also a thorough ablation study to illustrate the importance of each component of the approach.\n\n\n\nWeaknesses:\n\n- The justification of the instance-level entropy temperature in Section 4.2 could be clearer. As is, it’s unclear why the instance-level entropy is not redundant after the curiosity augmented target entropy, and vice versa. It’s also not well justified why the curiosity value is rounded and then re-scaled to form the entropy temperature. \n\n- While the method achieves good results on the standard tasks, I am still curious about its performance on pure exploration tasks, and hard exploration tasks such as sparse-reward settings. In the former, it would be great to quantitatively measure the unique states visited and to provide a comparison to RND.\n\n- While the modification to RND is intended to correct for curiosity at unseen states, the selection of unseen states is based on a heuristic: unseen states are created as combinations of pairs of seen states.\n\n\n\nRecommendation:\n\nI am recommending to reject this paper. I think parts of the method can be more clearly explained and justified. There is also potential to better highlight the strengths of the proposed method in the experimental results, i.e., by including pure/hard exploration tasks.\n\n\n\nQuestions:\n\n- The learning curves in Figure 1 evaluate SAC and CAT-SAC for 200K environment steps. I’m curious about the behavior after more steps & at convergence, and how many steps are necessary for convergence.\n\n- Are beta and m tuned for each task?\n\n- What does “the instance-level entropy temperature may weaken the connection across states” in Section 4.2 mean?\n\n\n\nUpdates after Reading Authors' Response:\n\nThank you for the detailed clarification and the new results. From the response, it seems that the proposed modifications to SAC do not and are not meant to solve the exploration problem in sparse-reward tasks. Instead, its aim is to improve sample efficiency on standard dense-reward tasks. The impact of the work then feels quite limited: the proposed modifications are specific to SAC and do not meaningfully improve performance on the sparse-reward task. For these reasons, I will keep my original evaluation.", "extra_scores": {}, "extra_text": {}}, {"review_id": "OfFjowdEWnx", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper tries to improve exploration in continuous control tasks by augmenting Soft Actor-Critic (SAC) with a curiosity module that increases the target entropy for unfamiliar states and decreases the target entropy for familiar states. The curiosity module is implemented using RND combined with a contrastive loss function (X-RND). To incorporate curiosity into SAC, the entropy coefficient is learned and dependent on the prediction error (coming from X-RND) of the state. The overall method (CAT-SAC) is tested on mujoco and swiss roll maze. Additionally, the paper includes ablation studies showing the benefit of X-RND and that of discretization of curiosity.\n\nNovelty: The main novel component is incorporating curiosity into entropy to increase the target entropy in unfamiliar states and decrease it in familiar states. \n\nPros: CAT-SAC does improve over SAC on mujoco domains and it helps to have X-RND component instead of vanilla RND.\n\nCons: The ablation studies and the evaluation feels incomplete. \n(1) Rather than adding curiosity to the entropy, what if we added it to the reward function? How well that perform? That should be a baseline. \n(2) RND doesn’t work that well with feature inputs. What about other notions of curiosity? How does dynamics model prediction error work in input space? There should be 2 baselines: (i) how well dynamics model prediction error work when added to reward function? (ii) how well it works when incorporated into entropy?\n(3) The method should be evaluated on harder exploration tasks (eg: ant gather and swimmer gather from Variational Information Maximizing Exploration (VIME; Houthooft et. al.) paper) and compared with VIME (or other exploration algorithms eg: why does hierarchy work so well? (nachum et. al.))\n\nReasons for score: Weighing the above pros and cons, I vote for rejecting. \n\nQuestions: Given c(s_t)is a scalar (as it’s prediction error), is g_{\\delta}(c(s_t)) just m(c(s_t)) + b?\n\nI am happy to reconsider my score if the above concerns are addressed. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "-VtAhx5F6vk", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Contribution**: For better exploration, the authors propose to use curiosity to set state dependent target entropies with SAC, with the goal of inducing more diverse behavior at unfamiliar states. They use RND to provide a curiosity score, which after normalizing, is used to adjust the state dependent target entropy. Due to RND performing poorly as a curiosity measure with state-based representations instead of image based, introduce a variant X-RND which additionally uses a contrastive loss to improve the curiosity mechanism. They demonstrate benefits over regular SAC on standard Mujoco Gym benchmarks.\n\n**Prior work in exploration**: While increasing policy entropy in regions of low  confidence makes intuitive sense, it is not actually clear simply being noisier in the face of uncertainty actually leads to good exploration (for example https://arxiv.org/abs/1306.0940 argues that these \"dithering\" style exploration is inefficient). It would be good to add additional comparisons to other methods for augmenting SAC to perform better exploration. For example, OAC https://arxiv.org/abs/1910.12807 learns upper confidence bounds on Q-values and uses the optimistic Q values for an exploration policy. It could be interesting to also explore how well CAT-SAC compares to just using the (X-)RND curiosity score directly as a bonus for exploration. In general, the related work is lacking in discussion of classical RL exploration methods like UCB-style bonuses and posterior sampling.\n\n**Conclusion**: Overall, I like the work. It appears technically sound, and presents a fairly simple and intuitive way to adjust exploration with SAC to better handle unfamiliar states (as far as any undirected exploration methods do at least). I would like to see a few more comparisons against other exploration techniques in deep RL, and perhaps some experiments ont asks outside of the standard gym benchmarks that focus more on the exploration problem itself rather than control (for example some sparse reward tasks). \n\nFairly minor points: \nLaTeX error: There are several instances of log in math mode that should use \\log instead.\nI would also like to see extended learning curves of CAT-SAC vs SAC, particularly until we see the performance of each method saturate. It would be interesting to see if the better exploration from CAT-SAC allows it to converge to higher performing policies as well as learning faster.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The concept of increasing entropy in novel states to promote exploration appears to be quite interesting. I do appreciate this idea, and I would encourage the authors to study it further. I think the reviewers also agree with this. Unfortunately, the paper as written has a number of issues: (1) the theoretical motivation for this is quite weak -- it intuitively makes sense, but for a published paper, we need more than intuition; (2) the empirical results are not especially compelling, it seems like the authors have to kind of thread the needle in arguing that they are concerned specifically with dense reward exploration -- a less chartable view is that the method just doesn't work all that well compared to other exploration settings in problems that present a major exploration challenge. The reviewers generally found the evidence in favor of the method to be a bit questionable. Therefore, in the balance, while the paper presents what I think is a really nice idea, it still needs work to justify both theoretically and empirically. I would encourage the authors to flesh out this work more -- I think with a bit more work, it could be a really nice paper, but for now it's probably not quite ready.

A few more comments (which did not influence the decision, but I recommend addressing in the future):

- Presenting exploration and efficiency results in a table, like Table 1, is not good. It hides actual patterns in performance, especially if you are talking about exploration. To the authors' credit, it appears that this trend was started by prior work (e.g., Kimin Lee et al.), but it was bad scholarship then, and it's bad scholarship now -- it's quite easy to pick a checkpoint where a given method looks better than all the other methods (which might be why some prior work opted for this format), but it's misleading to the reader and should not be done.

- I don't agree with Reviewer 2's comments about random seeds. Certainly it's better to have more random seeds than less, but this doesn't appear out of line with the standard in the field. That said, the results in Figure 4 do look quite close, so trying an actual statistical significance test might be a good idea (again, this didn't influence my decision -- the standard in RL holds that 4-6 seeds is plenty, and we have to review work by the current standard in the field).
Decision: Reject