Stochastic Policy Optimization with Heuristic Information for Robot Learning
Authors: SEONGHYUN KIM
Ingook Jang
Samyeul Noh
Hyunseok Kim
Venue: CoRL2021 Poster (2021)
Keywords: Robot manipulation
Reinforcement learning
Stochastic policy

Abstract: Stochastic policy-based deep reinforcement learning (RL) approaches have remarkably succeeded to deal with continuous control tasks. However, applying these methods to manipulation tasks remains a challenge since actuators of a robot manipulator require high dimensional continuous action spaces. In this paper, we propose exploration-bounded exploration actor-critic (EBE-AC), a novel deep RL approach to combine stochastic policy optimization with interpretable human knowledge. The human knowledge is defined as heuristic information based on both physical relationships between a robot and objects and binary signals of whether the robot has achieved certain states. The proposed approach, EBE-AC, combines an off-policy actor-critic algorithm with an entropy maximization based on the heuristic information. On a robotic manipulation task, we demonstrate that EBE-AC outperforms prior state-of-the-art off-policy actor-critic deep RL algorithms in terms of sample efficiency. In addition, we found that EBE-AC can be easily combined with latent information, where EBE-AC with latent information further improved sample efficiency and robustness.

Stochastic Policy Optimization with Heuristic Information for Robot Learning
Seonghyun Kim, Ingook Jang, Samyeul Noh, Hyunseok Kim"
Electronics and Telecommunications Research Institute
218 Gajeong-ro, Daejeon, Korea {kim-sh, ingook, samuel, hertzkim hertzkiimSetril Setri. re.
Stochastic policy-based deep reinforcement learning (RL) approaches have remarkably succeeded to deal with continuous control tasks. However, ap- plying these methods to manipulation tasks remains challenge since actuators of a robot manipulator require high dimensional continuous action spaces. In this paper, we propose exploration-bounder exploration actor-critic (EBE-AC), novel deep RL approach to combine stochastic policy optimization with inter- pretable human knowledge. The human knowledge defined as heuristic in- formation based both physical relationships between robot and objects and binary signals of whether the robot has achieved certain states. The proposed ap- proach, EBE-AC, combines an off-policy actor-critic algorithm with entropy maximization based on the heuristic information. On a robotic manipulation task, demonstrate that EBE-AC outperforms prior state-of-the-an off off-policy actor- critic deep RL algorithms in terms of sample efficiency. In addition, we found that EBE-AC be easily combined with latent information, where EBE-AC with la- tent information further improved sample efficiency and robustness.
Abstract:
Keywords: Robot manipulation, Reinforcement learning, Stochastic policy
Introduction
A challenge to human-level AI [1] is still ongoing research area and recently deep reinforcement learning (RL) approaches have remarkably shown outstanding results beyond human in the fields of board and video games [2, 3]. As greatly unexplored area which adapts deep RL, robot manipula- tion is one of the challenging applications due to complex nonlincar dynamics and huge continuous action spaces [4]-[7]. Various and stable robot manipulation skills in real- wal-world environments [8), where the laws of physics are strictly applied, are required to solve complex tasks, such as moving, grasping, pushing, and placing. It is, therefore, still challenge to solve complex tasks through deep RL approaches.
In front of the advanced dexterity capabilities of human hands, learning the dexterity with func- tionality to autonomously pick and place objects has always been a long-standing challenge for robotics [9, 10]. To solve complex manipulation tasks, various approaches have been presented to utilize human knowledge, which includes learning from human demonstrations and optimizing hyper-parameteed empirically [11]-[18]. The first improves sample efficiency of learning by provid- ing good expert demonstration data to solve a given task [13] [13]-[16]. In general, the human demon- strations be used not only loss function of inverse RL but also reusable experience data of ooolpy RL. However, it is expensive to obtain real-world demonstrations for every single task.
As the second approach utilizing human knowledge, empirical hyper-parameter tuning [19], has helped optimize learning progress since hyper-parameters are highly related to not only neural net- work architectures but also reward designs in the deep RL. On the whole, hyper-parameter tuning widely used to ensure that learning algorithms gain good performance, however, it is vulnerable to initial hyper-pance parameter settings for each task [18]. This results in the inefficiency of the learning process because it is necessary to perform hyper-paramedt tuning every task.
ccrrresponding author
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

A heuristic (informed) strategy using problem-specific knowledge has been generally known as a more efficient solution than an uninformed strategy using only the problem itself [20]. Since manipulation skills are under the laws of physics, informed human knowledge, such as moving in the direction of narrowing the distance, being in alignment with the target object, and grasping the center of gravity of the object, can be helpful to find solutions efficiently in robot manipulation.
In this paper, we propose novel deep RL approach combine stochastic policy optimization with human knowledge, which is exploration-bounded exploration actor-critic (EBE-AC). EBE-AC ap- plies interpretable and reusable human knowledge, heuristic information based on both physical re- lationships between robot and objects and binary signals of whether the robot has achieved certain states, an off- policy actor-critic algorithm. The main contributions of this paper are summarized as follows:
We design a novel information theoretic metric in order for EBE-AC to have dependencies between actions and heuristic information.
We propose e-exploration method using two types of stochastic policies, exploration and bounded exploration, in order to store diverse experiences a shared replay buffer.
We propose a temperature optimization for given entropies corresponding to the two poli- cies, in order to balance between the two.
To verify the performance of EBE-AC, we conducted experiments on a robotic manipulation task, Pick-and-Lift. The experimental results show that EBE-AC outperforms prior state-of-the-art off- policy actor-critic deep RL algorithms in terms of sample efficiency. In addition, we found that EBE-AC can easily combined with latent information as DIAYN [21] does, where EBE-AC with latent information further improved sample efficiency and robustness.
2 Related Work
Due to the difficulty of obtaining stable and steady improvement of deterministic policy in applying complex and high dimensional tasks, stochastic policy optimization based on information theory has been examined in problems with continuous state-action spaces [22]-[24]. Many works show that applying a stochastic policy's entropy to an objective function results in good performances. In particular, regularizing both policy and value functions with entropy shows the convergence and stability improvements during training processes.
Through maximum entropy RL [23], a policy is encouraged to perform diverse actions to widely explore state space while maximizing expected returns. Owing to the diverse actions being able to maximize expected returns, the learned policy can have multiple optimal solutions and obtain stable performance regardless of random initial states. In addition, the maximum entropy approaches are also robust environmental disturbances such as the presence of adversaries and dynamics change over time because these works focus on the entropy of the action distribution for a given state.
In the unsupervised emergence of diverse skills dictating the states [21], the mutual information between states and latent variables is leveraged to autonomously discover diverse skills. To ensure the different skills being distinguishable, the mutual information between actions and skills should be minimized for a given state. By utilizing the mutual information as metric of the objective function, a stochastic policy can be also stably and steadily improved under the consideration of additional latent variables.
These prior works utilize rich exploration to obtain various experiences at the beginning of learn- ing, and tend reduce the exploration as the learning progresses in order to acquire good expected returns stably. However, when a policy is stuck in inevitable local optimums which must be passed to complete a given complex task, the policy needs utilize rich exploration in order to escape the local optimums even in the middle of learning. In other words, it is necessary that policy simul- taneously pursues rich exploration and stable expected returns throughout the learning period. This paper proposes a novel way to combine human knowledge with a stochastic policy by leveraging the information theoretic metrics. Unlike reducing the mutual information between actions and la- tent variables in [21], the proposed method increases the mutual information between actions and heuristic information based on human knowledge. As a result, the dependency between actions and heuristic information will be guaranteed and experiences obtained by actions are related to heuristic

---

information. Based on the metric, e-exploration method using two types of stochastic policies is pro- posed in order maintain rich exploration throughout the learning period. Based on the proposed method, the sample efficiency is also improved and the stochastic policy achieves the robustness to complex tasks.
3 Preliminaries
3.1 Notation
Markov decision process is defined by the tuple (S,A,,,r), where 8 is state space, A is continuous action space, p S x A x S - R+ is transition probability distribution, and r S A - is a reward function. In episodic tasks, a discounted return at time is given as G. = Σx " > ax), where T' is an episode horizon, y is discount factor, and r(*k, ax) a reward for the state-action pair (8k, as). A value function for policy ([ae|sr) is represented as Ux (8x) = 7 (Gr].
3.2
Objective function
We consider combining RL with the information theoretic metrics, which encourage policy optimization in both ways to diverse actions and specific actions in given state. Therefore, we propose an objective function learn policy conditioned on heuristic information according to human knowledge, where the policy can select actions considering not only the states but also the heuristic information.
First, we define an information theoretic reward as
T(st, az) = r(st, az) + w. I ze|st) + wz w.hp, (celst: r(s,, 0.) w. (h (ailst) hps (ar|st. 2e)) + wyhpp (1,)));2)..
(I)
(2)
where is heuristic information, I [a,; (st) is mutual information between a, and 2, for the given St. h-(-) is an entropy for the distribution p. W, and wa are positive temperatures, and ps and Ps are distributions of action a conditioned on 8, and 21), respectively. The distributions Ps and are induced by policy и. The heuristic information 2, defined by external function = f(xx) based 8t. where f(-) can be physical information between robot and objects in addition to binary signals representing specific states.?
Maximizing the reward F(sr, a;) in Equation (1) means that policy has to find an optimal solution to maximize the reward from the environment while guaranteeing high relation between a, and and diverse actions for given information and zt. In episodic tasks, discounted return for T(st, ar) is derived as
T
G.
Σ
Ge (a.ss))))s) (w, w. W. (a))s),2),
(3)
k=t+1
where H,(-) Ekme+1 7* k-t-1 (-) is discounted sum of entropy.
By using the discounted return in Equation (3), the objective function is defined as follows:
(st) = [G,] (s.) + w.H, (a.ls:) (w- w. w, (a.1s,,)),
(4)
where H.,-) = E, (H,()))
To maximize the objective function in Equation (4) with a simple and fast way to implement, this paper designs e-exploration based policy using two types of stochastic policies as follows:3
(ar(s))), for Pm (m( T=(ag(st,2)) for Pm((2 =
(5)
Detailed examples of heuristic information are described in Appendix D.3.
"In comparison with an é-greedy method which uses policy- based actions and random actions with e prob- ability for exploration effects, the e-exploration method uses two types of policies having different exploration characteristies measured in entropies where one learns to maximize return with rich explorations and the other learns to maximize return with bounded explorations.

---

Obervation Reward
Palicy Selection
Policy Models
Update Tomperatures
(4,4119)
Pm(())
Update
[a, s,
Soft Q-Functions
Environment
f(s,)
(5,
ction
4r,
Replay Buffer
Update Soft Q-functions
(fte No On Puuts (10 92
T emperratures
W2
W,
Figure 1:
EBE-AC Algorithm: EBE-AC algorithm consists of two types of policies. Each policy shares the replay buffer and updates its own soft Q-function in an off-policy soft actor-critic(SACC manner. After updating the soft Q-function and policies, the temperatures are updated.
where (x) P(m xz is a probability mass function for selecting policy Tz, and e is positive constant. By Equation (5). the objective function in Equation (4) reformulated as follows:
Jam (st) (s. (s. un (5c) + H x
(1 {) (8:) + w. H. (atlst)}
+ c(Ur, (8t) + (w, - H.)H. (as 3e, 2r)3 = J (
(6)
where
H N. (ar (s)) for (w- - w/ H n, (ael st, zt), for =
Thus, the maximization problem for Equation (6) is represented
max TT
JT,,,, (8t); Vs, E.
Exploration-Buunded Exploration Actor-Critic
4.1 Problem decomposition
The maximization problem in Equation (8) is decomposed into two sub-problems: a maximization problem with exploration entropy and maximization problem with bounded exploration entropy for the policies n, and _,, respectively.
4.1.1 Learning with exploration entropy
For the policy n., the problem in Equation (8) is reduced as
max (st,; Vsr E
Rx
where Jr, (5t) = Uz, (st) + w. H. (a.)st). By representing Jr, (st) Tr. (st), Equation (9) can be derived as follows:
Jz, (st) Gn (8t) E,, [T,, (8r, u+) + wahn, (arls:)]
(10)
where Z.. (s,, ax) r(s, az) + y TT W.. (st=1)) is the soft Q-function for n. [23].* Because the entropy term w,,r, (s) incentivizes the policy to choose actions with a high variance, the policy learns to be exploratory for given states.
'Detailed derivations of Equation (10) described in Appendix A.

---

max 4.1.2 Learning with bounded exploration entropy
For the policy T., the problem in Equation (8) is reduced as
max (sc); 4ss S,
(11)
where Jn. (8t) = Ux, (8e) + (wy - w x; (ag zt). Without loss of generality, ex (sp) can be rewritten as Ux2 (st) = Ux= (st, 24) because 24 f((1) is a deterministic function of 83. Through the same procedure in Section 4.1.1 with a replacement s/ = (st: =t) Jn. (st) is derived as Jx. (st) = Ün (s() = Ex, M7n (5), ax) + (w, - w. Mr,, (ar s/]],
(12)
where u,, (8() and Zn az) are the soft value function and soft Q-function for T. under the given s = (st, 2t), respectively.
Unlike the entropy term waht, (ar(se) in Equation (10) which incentivizes the policy to have rich exploration, the entropy term (W: ), (a policy has rich exploration. The variance of the policy's 's actions depends on whether (w; w.) is positive or not. For example, when (w-- wx,) is negative, the policy gets a penalty for actions with a high variance. Thus, the policy learned from Equation (12) has bounded exploration. The bounded exploration reduces an action variance such that the policy focuses on exploring around given state and does not explore unnecessarily widely. In perspective of experience data for learning, detailed experiences from T. can supplement widely obtained experiences from g..
(s) in Equation (12) does not guarantee that the
4.2 Automating entropy adjustment
Because an entropy of policy is affected by not only its own distribution but also the type of task and the magnitude of rewards, the entropy of the policy can vary during training. For this reason, the temperatures W, and w; need to be set according to these considerations. As shown in [25], a temperature can be adjusted by optimizing dual problem for constrained optimization problem.
First, the objective function in Equation (4) can be represented by soft Q-function as follows:
(s:) = E [0= (st, az) + wphp. (asls)) + (w= w., h. (ael [&t, 21)
(13)
where T ax) r(st, ar) + 1 (8 and Ür (8t) = E, [Gr] are the soft Q-function and soft value function for π, respectively.
For the objective function in Equation (13), a constrained optimization problem can be derived as
Ex, 1@x (c): s.t. E. [/p, (axl0e) - hp, (arlsr, (00()),,,)) =x)] 2 0,
(14)
"
(15)
Ear lhp. (ar]st,2()) 2 0.
(16)
For the constrained optimization problem, each entropy can be the bound condition to each other, i.e., hpp (a.|s;) is the upper bound from the perspective of hps (arlst, 2x) and hp, (a, (8c, =)) is the lower bound from the perspective of hpp (aels:). Because hp, (ar(s;) does not have an upper bound, the policy is able to pursue diverse actions only using the state. Using the state and heuristic in- formation, however, the policy is limited to get diverse actions due to the upper bound hp, (a, st). Thus, for the case of the given state and heuristic information, the policy has to find balanced actions between the upper bound hp, (t) and the lower bound 0.
A dual problem for the constrained optimization problem is defined as min ( a.) w,h,, (a. (s:) (w,, w. w.)hp, (ael (dc]s1, 21),
(17)
where wr, and w. are the Lagrange multipliers. The primal problem for and the dual problem for (wa, wz) can be solved by using gradient descent method recursively. In practice, the soft Q-function is modeled by function approximation based on deep RL. Under the consideration of a parameterized soft Q-function for a fixed policy, Tr (st, at) does not affect to the Lagrange multipliers. Then, the optimal Lagrange multipliers for the given policy can be obtained by
w. =
arg min Jw: 22,100
(18)
where = [We,, W=] and
(19) After obtaining the optimal Lagrange multipliers, the optimization for the policy is performed for the given Lagrange multipliers. Therefore, π and are updated by such a recursive optimization.
Jw Ex, hp, (ae(st) + (w, - w. hh) (ag (8t, 2e)].
5

---

Algorithm EBE-AC
Initialize parameters Om and om for E {8, 2).
2: Initialize temperatures w.
3: Set: On On and D 0.
4: for each iteration do
5: for each environment step do
6: Select behavior policy index: my ~ Pem (2)
7: Sample action: ae Tm, (ae(8t)
8: Step environment: ($1+1, p(s 7e
9: Store transition:
D (m², St, ag Pe, 8441)
10: end for
11: for each gradient step do
12:
Update soft Q-functions:
13:
8m t 8m AVe.. Jo.. for m E {s, using Equation (20)
14:
Update policies:
15:
Om f 6m AV Jon for m {s,2] 2) using Equations (10) and (12)
16:
Update temperatures:
17:
W s- w - -VVwJw using Equation (19)
18:
Update target soft Q-functions:
19: Om < 70m + (1 - 7m for m E {s,2]
20: end for
21:
end for
4.3 Exploration-bounded exploration actor-critic algorithm
To solve the optimization problems in Section 4.1, actor-critic method with function approximation adopted. For function approximations, parameterization method applied soft Q-function To (st, at) and policy The for m {s, 2} in Equations (10) and (12). As shown in [25], the parameterized soft Q-function qo... (st. ae) is trained to minimize the soft Bellman residual
_ Ju = E.) [ (To... (st, at) (r(se, az) + 7B,,, 111 -p (o
(s++1)))
(20)
where D is replay buffer, To... (st, at) is qo, (st, ai) for and To, (5% at) for m
To, (8141, ((41) + wohre, (ai1s141), for m = To_ ($/41, a(+1) (w, wa)hn (arl)(+1), for m m=
Ja (5t+1) =
and 6m is the parameters of a target soft Q-function obtained as an exponentially moving average of the soft Q-function parameters. After obtaining the parameterized soft Q-functions, the parameter- ized policies and temperatures are updated sequentially.
These updates summarized in Algorithm The soft Q-functions and policies updated in the same manner with SAC [25]. As SAC uses two soft Q-functions to mitigate positive bias in a policy improvement, we also use soft Q-functions for each policy. For the simplicity Algorithm we omit detailed description of the two soft Q-functions for each policy.
5 Experiments
In order to validate our method, we implement robotic manipulation task, Pick-and-Lift. modified from that of RLBench [26], where the task environment consists of a single UR3 arm with 6 DoF including the two-finger gripper, cube object reward Pick Lick-and-Lift as the sum of dense and sparse rewards, in detail the rewards for reaching and moving sub-tasks are given by dense terms of distances among the robot's gripper, the cube and the target position, and the rewards for grasping is given by sparse terms of the binary signal according to whether not succeed in grasping."
table, and target position. We design the
DDetails of Pick-and- aad-Liji Lift described in Appendix D.

---

2,00
EBE-ACI0
200
1.75
EBE-ACIO DIAIN
1.50
DIAYN
1.25
1,00
Episode
0-
EBE-AC09.1)
EBE-AC(O.2) w/ DIYN
0.25
-20
DIANN
0.00
2000
4000
6000
8000 10000 12000 Number of episedes
14000 16000
2000
4000
6000
8000 10000 12000 14000 16000 Number of episodes
(a) Episode rewards
(b) Success rates
Figure 2: Training curves for Pick-and-Liff according to the number of episodes. In our experiments, the performance measured by learning from scratch without any pre-trained models.
To demonstrate our method, we evaluate EBE-AC compared to SAC [2C and DIAYN [21], state-of- the-art methods in the field of stochastic policy based deep RL. In addition, we evaluate EBE-AC with DIAYN to show effects of our method combining with latent information used in DIAYN.
5.1 Policy evaluation
We evaluate EBE-AC(e) and EBE-AC(e) combined with DIAYN, where is the probability defined in Equation (5). Figure represents training curves for episode rewards and success rates. In Figure 2a, range of the episode reward in y-axis is roughly divided into three cases as follows:
Range 1. (episode reward < 0): The robot tries reach the cube.
Range 2. (0 < episode reward < 100): The robot tries grasp the cube.
Range 3. (100 < episode reward): The robot tries to move the cube to the target position.
Compared with SAC and DIAYN, EBE-AC methods reach Range 2 faster than the other methods. SAC can reach Range 3 and means that SAC are stuck in local optimums Range 2 during training. EBE-AC(O.) and EBE-AC(O.2) with DIAYN methods reach Range 3 faster than DIAYN. Especially, EBE-AC(0.2) with DIAYN is noticeably faster than DIAYN.
In Figure 2b which represents the success rate, it is also observed that only the methods reaching Range 3 have meaningful success rates. EBE-AC(0.2) with DIAYN outperforms the other methods and EBE-AC(O.1) 1) has better success rates than DIAYN. Since EBE-AC is composed of the policy with exploration entropy and the policy with bounded exploration entropy, it can be combination of SAC's policy and the proposed policy with bounded exploration entropy, where the policies are linked by the temperature optimization. Through the composition characteristies of EBE-AC and experimental results for the fast convergences, therefore, it is shown that our method able to help the prior works in terms of the sample efficiency.
the
5.2 Temperature Comparison
Figure represents an analysis for temperatures according the number of episodes, where the temperatures of SAC and DIAYN are optimized in the same manner with SAC [25].
As shown in Figure temperatures of all the methods decrease rapidly the beginning of learning. The degradation of temperatures affects to reducing the proportion of entropy in a soft value function
Empiricalll, we could gain the best performance for EBE-AC methods with the values e = 0.1 and 0.2 for EBE-AC and EBE-AC with DIAYN, respectively. Results for various are described in Appendix D.5.
Detaill of episode rewards are described Appendix D.6. For readability of Figure 3, we omit (EBE-AC(0.2) with DIAYN because EBE-AC(O.)) is enough to explain
the temperatures.

---

0.8
W, of EBE-ACIO. 1)
0.7
Wy of EBE AC|O.1)
0.6
(W, - W.) EBE-AC(O.1)
W, SAC
Temperature
Wo of DIAYN
0.4
0.3
0.1
0.0
-0.1
6000
8000
10000
Number of episodes
Figure Compared to SAC and DIAYN, EBE-AC(0.1) utilizes the temperatures more dynamic than SAC and DIAYN.
which is a sum of soft Q-function and entropy. means that the learning takes into account expected returns approximated by the soft Q-function more importantly.
The value w is always higher than that of w, and leads to the value (w-- w,) be positive. Thus, the entropy term in Equation (1 1) affects Tæ to maintain exploratory. In addition, it is observed that w, in Equation (10) is higher than (w;- - w.). ). means that n. is affected from the entropy and pursues more exploration than π.. The temperatures of EBE-AC(O.)) increase rapidly at about 3000 and 8000 x-axis, which is consistent with the increase of episode reward Figure 2a. This feature is also observed for DIAYN about 5000 and 10000 in x-axis. Therefore, EBE-AC(0.1) considers weighted entropies more than SAC and DIAYN, and more exploratory than SAC and DIAYN. For this reason, EBE-AC(O.1) has better sample efficiency than SAC and DIAYN.
6 Conclusion
In this paper, we proposed exploration-boundee exploration actor-critic (EBE-AC), a novel deep RL. algorithm that can optimize stochastic policy through the use of heuristic information. The heuris- tic information includes interpretable human knowledge related to physical relationships between robot and objects, which can be reused various learning tasks. In robotic manipulation task, demonstrated that the proposed algorithm outperformed state-of- state-of-the-ar off-policy actor-critic deep RL algorithms, such as SAC and DIAYN, in terms of sample efficiency. addition, we extended the algorithm by means of latent information as DIAYN such that the extended algorithm can further improve sample efficiency and robustness.

---

Acknowledgments
This work was supported by Electronics and Telecommunications Research Institute (ETRI) grant funded by the Korean government. [21ZR] 100, Study of Hyper- Thinking Internet Technology by autonomous connecting, controlling, and evolving ways]
References
[1] J. McCarthy. From here to human-level ai. Artificial Intelligence, 171(18):1174-1182 2007.
[2] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathicu, A. Dudzik, Chung, D. H. Choi, R. Powell, T.. Ewalds, P. Georgiev, al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 375(7782):350-354 2019.
[3] C. Berner, G. Brockman, B. Chan, V. Cheung, P. Debiak, C. Dennison, D. Farhi, Q. Fis- cher, S. Hashme, C. Hesse, R. Józefowicz, S. Gray, C. Olsson, J. Pachocki, M. Petrov, H. P. de Oliveira Pinto, J. Raiman, T. Salimans, J. Schlatter, J. Schneider, S. Sidor, I. Sutskever, J. Tangg F. Wolski, and S. Zhang. Dota 2 with large scale deep reinforcement learning. CoRR, abs/1912.06680, 2019.
[4] Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. Mastering the game of go with deep neural networks and tree search. Nature, 529:484 489, Jan. 2016.
[5] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1-44 4-, 2016.
[6] S. Gu, E. Holly, T. Lillicrap, and S. Levine. Deep reinforcement learning for robotic manipu- lation with asynchronous off off-policy updates. 2017 IEEE International Conference on Robotics and Automation, pages 3389-3398 Aug. 2017.
[7] L. Fan, Y. Zhu, J. Zhu, Z. Liu, O. Zeng, A. Gupta, Creus-Costa S. Savarese, and L. Fei-Fei. Surreal: Open- Open-source source reinforcement learning framework and robot manipulation benchmark. Proceedings of The 2nd Conference on Robot Learning, 87:767-782, Oct. 2018
[8] K. Pertsch, Lee, and J. Lim. Accelerating reinforcement learning with learned skill priors. Conference Robot Learning, 2020.
[9] A. Billard and D. Kragic. Trends and challenges in robot manipulation. Science, 364(6446), 2019. doi: 10. 126//ccience..at84444
[10] R. Jeong, T. Springenberg. J. Kay, D. Zheng, Y. Zhou, A. Galashov, N. Heess, and Nori. Learning dexterous manipulation from suboptimal experts. Conference on Robot Learning, 2020.
[11] T. Welschehold, C. Dornhege, and W. Burgard. Learning manipulation actions from human demonstrations. 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 3772-3777, 2016.
[12] C. Finn, Levine, and P. Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. Proceedings of The 33rd International Conference Machine Learning, 48:49-58, 20-22 Jun 2016.
[13] C. Florensa, D. Held, M. Wulfmeier, M. Zhang, and P. Abbeel. Reverse curriculum generation for reinforcement learning. Proceedings of the 1st Annual Conference on Robot Learning, 78: 482-495, Nov. 2017.
[14] M. Vecerík, T. Hester, J. Scholz, Wang, O. Pietquin, B. Piot, N. Heess, T. Rothörl, Lampe, and M. A. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics 2017.
problems with sparse rewards. CORR, abbs/1707.083
[15] Hester, M. Vecerik, O. Pietquin, Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, I. Osband, G. Dulac-Arnodd, J. Agapiou, J. Leibo, and A. Gruslys. Deep 4- learning from demonstrations. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018.

---

[16] A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel. Overcoming explo- ration in reinforcement learning with demonstrations. 2018 IEEE International Conference on Robotics and Automation, pages 2292262999 2018.
[17] T. Lillicrap, J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Con- tinuous control with deep reinforcement learning. arXiv preprint arXiv:1509.0299 2015.
[18] Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. Deep reinforcement learning that matters. Proceedings of the AAAI Conference Artificial Intelligence, 32(1): 1861-1870, Apr. 2018.
[19] M. Kim. Supervised learning-basse ddos attacks detection: Tuning hyperparameters. ETRI Journal, 4165):560-573, 2019. do..htpss//doo.orr//044111 4000:
[20] J. Russell and Norvig. Artificial Intelligence: modern approach. Pearson, edition,
2009.
[21] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. International Conference on Learning Representations, 2019.
[22] T. Haarnoja, H. Tang, P. Abbeel, and S. Levine. Reinforcement learning with deep energy- based policies. Proceedings of the 34th International Conference Machine Learning. 70: 1352-1361, Aug. 2017.
[23] T. Haarnjja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. Proceedings of the 35th International Conference on Machine Learning, 80:1861-1870, Jul. 2018.
[24] B. O'Donoghue, R. Munos, K. Kavukcuoglu, and V. Mnih. Combining policy gradient and q-learning. arXiv preprint arXiv.1611.01626633 2017.
[25] Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta,
P. Abbeel, and S. Levine. Soft actor-critic algorithms and applications. CoRR, abs/ htts//812.055000 2018.
[26] S. James, Z. Ma, D. R. Arrojo, and A. J. Davison. Rlbench: The robot learning benchmark learning environment. IEEE Roboties and Automation Letters, 5(2):3019-33) 2020. doi: 10.109//RA.2200.2977777...
10


Reviews:
[{"review_id": "vtMCp7TK9cw", "reviewer": "Reviewer_p2oA", "summary": "The authors propose to augment the observation with heuristic information in deep reinforcement learning.\nBecause the heuristic information might not always help the agent, they are learning two policies at the same time.\nOne is using the heuristic information while the other is not.\nThey performed experiments with ablation studies showing that their approach was learning faster.\n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThe paper is easy to understand and well written.\n\nWeaknesses:\nThe approach is a bit ad hoc. There is no real justification for picking one or the other policy.\n\nIn Fig. 5 DIAYN w/ z and EBE-AC(0.2) w/ DIAYN are actually quite close.\nI suspect that the small gain of EBE-AC(0.2) w/ DIAYN is actually due to the fact that some information in z impedes the learning.\nFor instance, if I have not grasped the object yet f_distance(object, target position) and f_fingers will not help to make a decision.\nAccordingly, once the object is already grasped, f_distance(gripper, object) is not helping.\n\nTwo missing baselines would be to learn two policies with SAC both without z and both with z. To see if the gain is not coming from a higher expressive power (2 policies).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am not convinced by the method because it somehow tries to tackle an ill-defined problem.\nUsing this method, it is assumed that the information in z is not always useful. In this case, it would be easier to detect which information of z is useful at which times and condition the policy on that. Rather than learning two policies at the same time, it seems easier to learn a succession of policies. For example, one to catch the object with a well-defined z, then another to move it to the target with another z definition.\n", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "Some effort for the readability of the curves should be made.\n\nTypos: \n-line 105 are induced \n-line 151 need \n-line 240 maintain \n-line 252 improve", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {}}, {"review_id": "mWoO7rrDobN", "reviewer": "Reviewer_48sx", "summary": "The paper presents an information theoretic objective function to achieve both maximum entropy policy optimization and the maximum expected mutual information between the policy conditioned on heuristic information and unconditioned policy. The objective is then reformulated to two soft Q functions and all parameters are optimized in an actor critic way including two temperature parameters which control the degree of the maximum entropy and mutual information. In experimental results, the proposed method outperforms the state of the art soft actor critic algorithms like SAC and DIAYN in a pick-and-lift robot simulator.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n    1. The information-theoretic objective takes human heuristic information into account which can guide exploration into unknown or a more reasonable region.\n    2. There are two hyperparameters called temperature in the objective function. One encourages the actions to have rich exploration, the other one can encourage or bound exploration automatically. A dual problem is formulated to optimize the two temperature parameters.\n    3. The experimental result is based on a pick-and-lift robot simulator. The performance of the proposed method outperforms existing soft actor critic method in both performance and speed of convergence.\nCons:\n    1. The most interesting part of this paper is the temperature parameter which is optimized automatically. It would be great if the author can show the evolution of the temperature parameters during the training phase and explain why such results happen.\n    2. Because this approach used step-based exploration rather than trajectory exploration. So it’s questionable whether there is temporally consistency in the robot motions, which is a big issue to apply this method on real robot tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper presents an information theoretic objective function to achieve both maximum entropy policy optimization and the maximum expected mutual information between the policy conditioned on heuristic information and unconditioned policy. And two temperature parameters as well as the policy parameters are optimized automatically to encourage or bound the agent’s exploration automatically. Overall, this paper is marginally above the acceptance threshold.\n\n\n### Post Rebuttal Comment: \nI thank the reviewers for their comments. I keep my evaluation as is because the paper is missing real world experiments.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "1. The experiments are not based on real robot systems, it might be better to transfer the existing result to a real robot system.\n2. Figure 3-4 needs some improvement. It’s hard to tell the settings of each curve. And the legend should be consistent with all plots. "}}, {"review_id": "b44jYg6hDkK", "reviewer": "Reviewer_UVE3", "summary": "This paper presents a method for guiding exploration during reinforcement learning using binary and continuous heuristics.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n-- Incorporating human knowledge into learning is an important area of research\n-- The authors have included significant formal detail about their approach\n\nWeaknesses\n\n-- The related work section is highly inadequate.  \n-- The paper was very difficult to read and needs significant proofreading\n-- The evaluation fails to provide strong justification for the proposed method\n\nSuggestions for Improvement\n\n-- Include other methods that incorporate human knowledge into learning as baselines\n-- Validate the method across more than task (ideally including real-world robot experiments as well as simulation)\n-- Provide significantly more context to relate the contribution to prior work and the state-of-the-art\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The general idea of using heuristics to speed learning (especially in robotics) is good, but the paper needs significant revisions to make it easier to follow, better situate the work relative to prior work, and more fully demonstrate that the approach improves over prior work.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Poor: Difficult to follow and understand.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "-- Please discuss the proposed work in more detail in relation to the works cited. What are the contributions of the proposed work? How does it advance the state-of-the-art?\n-- The notation in the algorithmic sections needs to be revised; the p(m=[s|z]) notation in equations 5 and 7 is particularly strange\n-- From equation 5, it seems that this is using an epsilon-greedy approach to exploration (guided by the heuristic) versus exploitation (guided by the current best policy).  If this is true, please just say that.\n-- Consider moving some of the formalization from section 4 to the appendix (especially background information not critical to your contribution) and providing more detail on the heuristics used in the experiments \n-- The charts in Figure 3 are very difficult to read and interpret\n-- It would be helpful to give an intuition for what you're doing in section 4, especially where and how the heuristics come in; right now it's very difficult to follow\n-- The entire first paragraph can be removed"}}, {"review_id": "4dxQ7zAVilS", "reviewer": "Reviewer_oVi6", "summary": "In this paper, the authors propose the \"exploration-bounded exploration actor-critic\" (EBE-AC) algorithm, a deep RL algorithm that can optimize a stochastic policy through the use of heuristic information. Such heuristic information includes prior human knowledge about physical relationships (e.g., between a robot and objects). The authors implement and evaluate their approach on a robotic manipulation task, Pick-and-Lift.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "PROS:\n* The paper is well written and clearly structured\n* The math is easy to follow\n* The idea is interesting\n* The experimental evaluation is sound and it includes the relevant baselines\n* All the hyper-parameters and details are provided in the supplementary material for reproducing the experiments\n\nCONS:\n* This is not really a negative point, but it would be interesting to see another application of the proposed algorithm, eventually in a different domain.\n* Plots could be improved for readability.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is well written and well structured. The idea is clearly described by means of mathematical formulations, and the experimental evaluation is convincing. All the details for reproducing the experiments are provided in the supplementary material.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "* Can you point out some different domain/application examples?\n* Improved plots would be good", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new approach for policy optimization. The main contribution lying at the information theoretic objective function that targets to achieve both maximum entropy policy optimization and the maximum expected mutual information between the policy conditioned on heuristic information and unconditioned policy. It is suggested that there should be more details on the choice of  the heuristics used in the experiments. And it would be great if there are additional validation results  across more tasks or discussions on the use of the proposed formulation for different domain/application examples.

After the rebuttal: The authors have made great effort in providing additional results and clarifying concerns from the reviewers. There is one critic that the proposed method has addressed an ill-defined problem, which I think can be an interesting direction for the authors to look at for future researches. On the other side, the proposed method still makes sense and is accompanied with a good set of experimental results that demonstrate some of the claims. 
Decision: Accept (Poster)