Hindsight Trust Region Policy Optimization
Authors: Hanbo Zhang
Site Bai
Xuguang Lan
Nanning Zheng
Venue:  (2020)
Keywords: Hindsight
Sparse Reward
Reinforcement Learning
Policy Gradients

Abstract: As reinforcement learning continues to drive machine intelligence beyond its conventional boundary, unsubstantial practices in sparse reward environment severely limit further applications in a broader range of advanced fields. Motivated by the demand for an effective deep reinforcement learning algorithm that accommodates sparse reward environment, this paper presents Hindsight Trust Region Policy Optimization (HTRPO), a method that efficiently utilizes interactions in sparse reward conditions to optimize policies within trust region and, in the meantime, maintains learning stability. Firstly, we theoretically adapt the TRPO objective function, in the form of the expected return of the policy, to the distribution of hindsight data generated from the alternative goals. Then, we apply Monte Carlo with importance sampling to estimate KL-divergence between two policies, taking the hindsight data as input. Under the condition that the distributions are sufficiently close, the KL-divergence is approximated by another f-divergence. Such approximation results in the decrease of variance and alleviates the instability during policy update.  Experimental results on both discrete and continuous benchmark tasks demonstrate that HTRPO converges significantly faster than previous policy gradient methods. It achieves effective performances and high data-efficiency for training policies in sparse reward environments.

Under review as a conference paper at ICLR 2020
HINDSIGHT TRUST REGION POLICY OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
As reinforcement learning continues to drive machine intelligence beyond its con- ventional boundary, unsubstantial practices in sparse reward environment severely limit further applications in a broader range of advanced fields. Motivated by the demand for an effective deep reinforcement learning algorithm that accommo- dates sparse reward environment, this paper presents Hindsight Trust Region Pol- icy Optimization (HTRPO), a method that efficiently utilizes interactions in sparse reward conditions to optimize policies within trust region and, in the meantime, maintains learning stability. Firstly, we theoretically adapt the TRPO objective function, in the form of the expected return of the policy, to the distribution of hindsight data generated from the alternative goals. Then, we apply Monte Carlo with importance sampling to estimate KL-divergence between two policies, tak- ing the hindsight data as input. Under the condition that the distributions are sufficiently close, the KL-divergence is approximated by another f f-divergence. Such approximation results in the decrease of variance and alleviates the insta- bility during policy update. Experimental results both discrete and continu- ous benchmark tasks demonstrate that HTRPO converges significantly faster than previous policy gradient methods. It achieves effective performances and high data-efficiency for training policies in sparse reward environments.
INTRODUCTION
Reinforcement Learning has been heuristic approach confronting a great many real-world prob-
lems from playing complex strategic games (Mnih et al., 2015 Silver et al. 2016 Justesen et al.,
2019 to the precise control of robots(Levine et al. 2016: Mahler & Goldberg [017]] Quillen al,
2018 in which policy gradient methods play very important roltonS et al. 2000 Deisenroth
ct al. 2013). Among them, the ones based on trust region including Trust Region Policy Opti- mization (Schulman al. 2015a) and Proximal Policy Optimization (Schulman et al. 2017) have achieved stable and effective performances on several benchmark tasks. Later on, they have been verified in variety of applications including skill Caariin(Naagaaang
al.) 2018), multi-agent (Gutra et al.] (201)). imitation learning(Ho et al.) 20166). and have been investigated further to be combined with more advanced techniques CachhniqesSNNaaaae Nachum al. 2017 Houthooft et al. 2016 Heess ct al. 2017).
One unresolved core issue in reinforcement learning is efficiently training the agent in sparse reward environments, in which the agent is given a distinctively high feedback only upon reaching the desired final goal state. On one hand, generalizing reinforcement learning methods to sparse reward scenarios obviates designing delicate reward mechanism, which is known as reward shaping (et al. 1999); on the other hand, receiving rewards only when precisely reaching the final goal states also guarantees that the agent can focus on the intended task itself without any deviation.
Despite the extensive use of policy gradient methods, they tend to be vulnerable when dealing with sparse reward scenarios. Admittedly, policy gradient may work in simple and sufficiently rewarding environments through massive random exploration. However, since it relies heavily on the expected return, the chances in complex and sparsely rewarding scenarios become rather slim, which often makes unfeasible to converge to a policy by exploring randomly.
Recently, several works have been devoted to solving the problem of sparse reward. mainly applying either hierarchical reinforcement learning (Kulkarni et al., 20166) Vezhnevets et al., 2017 Le et al. 2018 Marino et al. 2019) a hindsight methodology, including Hindsight Experience Replay

---

Under review as a conference paper at ICLR 2020
Andrychowicz al.) 2017). Hindsight Policy Gradient (Rauber et al. 2019) and their extensions Fang et a 2019 Levy et al. 2019). The idea of Hindsight Experience Replay(HER) is to regard the ending states obtaind through the interaction under current policy as alternative goals, and therefore generate more effective training data comparing to that with only real goals. Such augmentation overcomes the defects of random exploration and allows the agent to progressively move towards intended goals. It is proven to be promising when dealing with sparse reward reinforcement learning problems.
For Hindsight Policy Gradient(HPG), it introduces hindsight to policy gradient approach and im- proves sample efficiency in sparse reward environments. Yet, its learning curve for policy update still oscillates considerably. Because it inherits the intrinsic high variance of policy gradient meth- ods which has been widely studied in |Schulman et al.] (2015b), Gu et al. (2016) and Wu et al. (2018). Furthermore, introducing hindsight to policy gradient methods would lead to greater vari- ance (Rauber et al., 2019). Consequently, such exacerbation would cause obstructive instability during the optimization process.
To design an advanced and efficient on-policy reinforcement learning algorithm with hindsight expe- rience, the main problem is the contradiction between on-policy data needed by the training process and the severely oof-po hindsight experience we can get. Moreover, for TRPO, one of the most significant property is the approximated monotonic converging process. Therefore, how these ad- vantages can be preserved when the agent is trained with hindsight data also remains unsolved.
In this paper, propose methodology called Hindsight Trust Region Policy Optimization (HTRPO). Starting from TRPO, a hindsight form of policy optimization problem within trust region theoretically derived, which can be approximately solved with the Monte Carlo estimator using severely off-policy hindsight experience data. HTRPO extends the effective and monotonically iter- ative policy optimization procedure within trust region accommodate sparse reward environments. In HTRPO, both the objective function and the expectation of KL divergence between policies are estimated using generated hindsight data instead of on-policy data. To overcome the high variance and instability in KI. divergence estimation, another f-divergence is applied to approximate KI. divergence, and both theoretically and practically, it is proved to be more efficient and stable
We demonstrate that on several benchmark tasks, HTRPO can significantly improve the performance and sample efficiency in sparse reward scenarios while maintains the learning stability. From the experiments, we illustrate that HTRPO can be neatly applied to not only simple discrete tasks but continuous environments as well. Besides, it is verified that HTRPO can be generalized to different hyperparameter settings with little impact performance level.
2 PRELIMINARIES
Reinforcement Learning Formulation and Notation. Consider the standard infinite-horizon inforcement learning formulation which can be defined by tuple AS,AA n,, Po, r, 7). S represents the set of states and A denotes the set of actions. : + P(A) a policy that represents an agent's behavior by mapping states to a probability distribution over actions. Po denotes the distribution of the initial state Sy. Reward function r : S + defines the reward obtained from the environment and 7 (0, 1) is a discount factor. In this paper, the policy is differentiable function regarding parameter We follow the standard formalism of state-action value function Q(s, a), state value
function (s) and advantage function A(s, a) in Sutton & Barto (2018). We also adopt the defini- tion of 7-discounted state visitation distribution as (s = (1 7) Ei P = s) (Ho et al, 2016), in which the coefficient 1 y is added to keep the integration of (s(() as 1. Correspondingly, 7-discounted sate-actionn visitation distribution (Ho et al. 2016), also known as occupancy measure (Ho& & Ermon| 2016), is defined po(s, a) (s( x (a(ss)) in which (als)) stands for the policy under parameter& t.
Trust Region Policy Optimization
Schulman et al. (2015a) proposes iterative trust region method that effectively optimizes policy by maximizing the per-iteration policy improvement. The optimization problem proposed in TRPO can be formalized as follows:
max (0)
(1)
2

---

Under review as a conference paper at ICLR 2020
s.t. E D ( S empplo)
in which Pa(s) = 2 s). 0 denotes the parameter of the new policy while @ is that of the old one. Trajectory represented by $1, 11, $2, 02,.... The objective function LTRPO(0) can be given out in the form of expeted return:
To(als)
( s.ampgle.a)
E
(T(aas) Ag(s.a)
(3)
Hindsight Policy Gradient(HPG). After generalizing the concept of hindsight, Rauber et al. (2019)
combines the idea with policy gradient methods. Though goal-conditioned reinforcement learning has been explored for a long time and actively investigated in recent works(Peters & Schaal 2008
Schaul al., 2015 Andrychowicz al., 20177] Nachum et al. 2018 Held al. (0.18 Nair al. 2018 Veeriah et al. 2018). HPG firstly extends the idea of hindsight to goal-conditioned policy gradient and shows that the policy gradient can be computed in expectation over all goals. The goal-conditioned policy gradient is derived as follows:
T-1
V (en = (r)P(Tla)
E
Σ Ve log ma(a st, Ag ar at.9)
(4)
1=1
Then, by applying hindsight formulation, rewrites goal-conditioned policy gradient with trajecto- ries conditioned on some other goal g' using importance sampling (Bishop. 2016) to improve sample efficiency in sparse-reward scenarios.
In this paper, we propose an approach that introduces the idea of hindsight to TRPO, called Hind- sight Trust Region Policy Optimization(HTRPOO aiming to further improve policy performance and sample efficiency for reinforcement learning with sparse rewards. In Section 3 and Section 4 we demonstrate how to redesign the objective function and the constraints starting from TRPO respectively.
3 EXPECTED RETURN AND POLICY GRADIENTS OF HTRPO
In order to apply hindsight methodology, this section presents the main steps for the derivation of HTRPO objective function. Starting from the original optimization problem in TRPO, the objective function can be written in the following variant form:
xe (T)(()))))
(g
E
Ag(se; oe)
t== TT(ae)ss)
r>pa(r)
The derivation process this variant form is shown explicitly Appendix A.I and in Schulman et al. (2015a).
Given the expression above, we consider the goal-conditioned objective function of TRPO as a premise for hindsight formulation. Similar to equation 4 Lg can be correspondingly given out in the following form:
To Ta((alss (ag TT (a(aa(st.))
La E
Ag(st, at g)
(6)
(T19)
For the record, though it seems that equation 6 makes it possible for off-policy learning, it can be used the objective only when policy To is close the old policy Tg. i.e. within the trust region. Using severely off off-policy data like hindsight experience will make the learning process diverge. Therefore, importance sampling need be integrated to correct the difference of the trajectory distribution caused by changing the goal. Based on the coll-conditioned form of the objective function, the following theorem gives out the hindsight objective function conditioned on some goal g' with the distribution correction derived from importance sampling.
Theorem 3.1 (HTRPO Objective Function). For the original goal g and an alternative goal g', the object function of HTRPO (g( is given by:
T Ta(ak (k. g') na Ta(esse,) g') Ag ( an.s
(g)
E g'
E (epp(Tls)
ΣI
(7)
Te (aka 8k.
1=
3

---

Under review as a conference paper at ICLR 2020
in which, 7 $1, $2, 0g, ..., St, dg-
Appendix A.2 presents an explicit proof on how the hindsight-forne objective function derives from equation 6 It will be solved under KL divergence expectation constraint, which will be discussed in detail in Section 4 Intuitively, equation / provides a way to compute the expected return in terms of the advantage with -ww-goal-conditooned hindsight experiences which are generated from interactions directed by old goals.
Naturally, Theorem 3.2 gives out the gradient of HTRPO objective function that will be applied to solve the optimization problem. Detailed steps of computing the gradient is presented in Appendix A.3
Theorem 3.2 (Gradient of HTRPO Objective Function). For the original goal and an alternative goal g'. the gradient (0) of HTRPO object function with respect to 0 given by the following expression:
20 EI
Ta(ak ((kk|skig') g') Vonooaaaaaa Ta(ax Sk. 9) Tz S1, g²)
VoTo(ailse.9)) Ag(ss,,nng
VoLg(0)=
(8)
o' (-pp(T|o)
in which 7 81, 01, $2; 02 .... Sp, Gg.
4 EXPECTATION OF KL DIVERGENCE ESTIMATION
This section firstly demonstrates some techniques, with striet proof, that can be used to estimate the expectation of KI KI-divergence and further reduce the variance, and then presents how hindsight is applied to the constraint function of TRPO.
In TRPO, the KL divergence expectation under pa(s) estimated by averaging all the values of KL divergence. When they are respectively conditioned on all states collected using the old policy, this kind of estimation is exactly Monte Carlo estimation which is unbiased. However, when we only have access to hindsight experience data, the state distribution may inevitably change and the previous method for estimating the expectation of KL divergence is no longer valid. To solve this problem, we firstly transform the KL divergence to an expectation under occupancy measure P3(8,a) = Pg(s) x (a)s). It can be estimated using collected state-action pair (s, a), whose changed distribution be corrected by importance sampling. Then, by making use of another f-divergence, the variance of estimation theoretically proved to be reduced so as to facilitating a more stable training.
The constraint function KL-divergence can be naturally converted to logarithmic form. Ap- pendix B. [.I provides more explicit version of this conversion.
Theorem 4.1 (Logarithmic Form of Constraint Function). Given two policies (alss) and (a(a)))
the expectation of their KL-divergence over states Pg(s) is written as:
E [KK[(Tg(ab>]]]<(( (m)
E [lo (ogTa(aas) - log To (als)]
However, simply expanding the KL-ddvergence into logarithmic form still leaves several problems unhandled. Firstly, the variance remains excessively high, which would cause considerable insta- bility during the learning process. Secondly, current estimation of KL-divergence is of possible negativity. If encountering negative expectation of KL-divergence, the learning process would result in fatal instability.
The following Theorem 4.2 describes a technique to reduce the variance and Theorem 4.3 gives out the strict proof for the decrease of variance.
Theorem 4.2 (Approximation of Constraint Function). For policy TT(aas) and (a(a))) and for Tz(a(s).
17 (
[log log [a(ss)]]
E s s. sp
(1og
I (log (T(ass) - log =o({as)))
8,arpp(s.a)
+
R
[o(n))]]
(10)
8,0 s,0-p)(s,a)
4

---

Under review as a conference paper at ICLR 2020
Theorem 4.2 demonstrates that when 0 and @ is of limited difference, the expectation of log (T(a)s) log Ta(aas can be sufficiently estimated by the expectation of its square. The proof provided in Appendix B.2 fact, (s.arp((s.a) [ (g((ss) log {o{a]s))²] the expectation of an f- divergence, where f(x) = alog x)?. Noticeably, f(r) is a strictly convex function when x E 00), and f(1) 0.
Moreover, is noteworthy that there are two corresponding major improvements through this kind of estimation. Firstly, it is guaranteed to reduce the variance which leads more stable performance. This merit will be explained in detail Theorem 4.3. Another significant improvement manifested in the elimination of negative KL-divergence, since the estimation presents itself in the form of square which is always non-negative.
Theorem 4.3 (Variance of Constraint Function). For policy TT(als) and ITe (als), let Var denotes the variance of a variable. For any action a E A and any state & E S, when log TT(alss) - log To(als) E [-0.5, 0.3], then
(log TT(aaa))) - log (a(a))))
Var (s0m)
<
Var e,0mpy(3.0)
[log (a(s)) - log )
(1I)
2
Theorem 4.3 illustrates that there is decrease from the variance log (a(()s)) log (als)) to variance of its square, and furthermore indicates that the variance is effectively reduced. The proof is given in detail in Appendix B.3 In fact, the closer it is between @ and 0, the more the variance decreases.
Based on Theorem 4.1 to Theorem 4.3, in this paper, adopt the following form of constraint condition:
(log Ta(als) log =o(cls))) cc
(12)
(E
In Theorem 4.4, demonstrate that hindsight can also be introduced to the constraint function. The proof follows the methodology similar to that in Section and is deducted explicitly Appendix
B.4
Theorem 4.4 (HTRPO Constraint Function). For the original goal g and an alternative goal g', the constraint between policy Ta s) and policy (a(ss) is given by:
Te (anlskkg') (log (a(ar)s),))) - log logro(ads) se' (13)
E
E (>pa(T
2
Te To((x(ok.9)
t=
k=1
which d'
1-
Theorem 4.4 implies the practicality of using hindsight data under condition g' to estimate the ex- pectation. From all illustration above, give out the final form of the optimization problem for HTRPO:
max E 8
EI t=
T (an sk: 9)
To s1,g") To(a, gt.
Ag(oe,ar,gg)
(14)
o
(rll))
T (a
Ta Ta(ox|sk.g' (sk, 9')
s.t.
E
E (rlp)
(log T (g(arss g') log To < (15) (a sk.g)
1=0
The solving process for HTRPO optimization problem is explicitly demonstrated in Appendix Cadd the complete algorithm procedure is included in Appendix
5 EXPERIMENTS
This section demonstrates the validation of HTRPO on several sparse reward benchmark tasks] design of our experiments aims to conduct an in-depth investigation in the following aspects:
The
The code and video be found httpt:t//////. //gi thub. wwt.hh........eeeeeeeeee ht//://////////////////0
5

---

Under review as a conference paper at ICLR 2020
Initial Aray
Target Aray
(a) Flipping
(b) Empty Maze
(c) Four Rooms
(d) Fetch
Figure Demonstration of experimental environments
How is the effectiveness of HTRPO?
How does each component of HTRPO contribute to its effectiveness?
How is the performance of policy gradient methods trained with hindsight data in continu- ous environments?
How sensitive is HTRPO network architecture and some key parameters?
5.1 EXPERIMENTAL SETTINGS
We implement HTRPO on a variety of reinforcement learning environments, including Bit Flipping, Grid World and Fetch. Among them, Bit Flipping, Grid World, Fetch Reach and Fetch Push implemented as descrete-action environments while we also conduct continuous version of experi- ments in Fetch Reach, Fetch Push and Fetch Slide. A glimpse of these environments demonstrated in Figure [ while the detailed introductions are included in Appendix F.I] The reward mechanisms are intentionally modified sparse reward regulations. Besides, for continuous version of Fetch experiments, we apply an additional policy entropy bonus to encourage more exploration.
For each trail of interaction, reward for the agent is set as the remaining number of time steps plus one, and all goals during exploration are chosen uniformly at random for both training and evalua- tion. During the training process, we terminate one episode either when the maximum number of time steps has elapsed or when the goal state is reached. We evaluate agents' performance by docu- menting 10 learning trails the form of average return and their corresponding standard deviation.
In Bit Flipping and Grid World environments, the network architecture is of two hidden layers, each with hyperbolic tangent units; Fetch environment, for both discrete and continuous implemen- tations, the network contains two 256-unit hidden layers. For all environments mentioned above, we compare HTRPO with HPG (Rauber et al., 2019) and TRPO (Schulman et al.] (2015a), which chosen the baseline algorithms. Since HPG is never applied to continuous environments in Rauber et al. (2019), we implement HPG to be adapted to continuous environments. Note that the way scale the time axis is significantly different from that in RRuber et al. (2019). Instead of regarding a certain number of training batches as interval between evaluation steps, we directly uses the accumulated time steps the agent takes while interacting with the environments throughout episodes and batches.
Besides comparing with baselines, we also ablate each component HTRPO to investigate how significant it is for the final performance. To be specific, we adopt the "vanilla" estimation of KL- divergence which we call "HTRPO with KI.1" instead of the proposed one Section 4 we also observe the performance of our algorithm without weighted importance sampling, which is denoted as "HTRPO without WIS" in this paper.
5.2 COMPARATIVE ANALYSIS
In discrete environments, test both the official version of HPG released in Rauber et al. (2019) and our HPG implementation while for continuous environments of Fetch, we only test our HPG due the lack of surpport for continuous tasks in Rauber al. (2019). We apply input normalization in

---

Under review as a conference paper at ICLR 2020
HTRPO
our HPG
official HPG
HTRPO with KL1
HTRPO without WIS
TRPO
10
Average Raluun
3
15
10
Time (1e4)
Time (1e4)
Time steps (1e4)
(a) 8-Bit Flipping
(b) 16-Bit Flipping
(c) Empty Maze
18
Average Return
15
18
2
0
8
18
20
Time steps (1e4)
Time steps (1e5)
Time steps (1e5)
(d) Four Rooms
(c) Discrete Fetch Reach
(f) Discrete Fetch Push
Figure 2: Evaluation curves for discrete environments. The full lines represent the average evalua- tion over 10 trails and the shaded regions represent the corresponding standard deviation.
HTRPO
our HPG
HTRPO with KL
HTRPO without WIS
TRPO
50
Average Return
30
25
25
20
30
20
15
20
10
4
5
12
20
16
20
Time steps (1e5)
Time steps (1e5)
Time steps (1e5)
(a) Countinuous Fetch Reach
(b) Countinuous Fetch Push
(c) Countinuous Fetch Slide
Figure 3: Evaluation curves for continuous environments. The full lines represent the average eval-
uation over 10 trails and the shaded regions represent the corresponding standard deviation.
the continuous Fetch environments for better performance. However, for fair comparison with the official HPG, we do not employ this trick in any of the discrete environments.
The evaluation curves for the trained policy are demonstrated in Figure 2 and and the training
curves and success rate of these experiments are supplemented in Appendix hyperparameters for all experiments are listed Appendix E. From results demonstrated in Rauber et al. (2019), the officially released version of HPG eventually converges to similar performances with that of HTRPO discrete environments, but sometimes, unlike our HPG, still far from converging under this time-step evaluation setting. This kind of distinction in converging speed between our HPG and the official HPG may be caused by the reduction of noises, since we use TD- error to update policies instead of the return corrected by importance sampling, which is adopted in HPG. Thus, for the fairness of comparison, in the following analysis, we mainly compare the properties between HTRPO and our HPG.
Detailed settings of
How is the effectiveness of HTRPO?
From the results forms HPG significantly. Aside from assuring a good converging property, the sample efficiency of HTRPO also exceed that of HPG, for it reaches higher average return within less time in most
that in both discrete and continuous environments, HTRPO outper-

---

Under review as a conference paper at ICLR 2020
environments. As for TRPO, though it can converge in several simple tasks like Bit Flipping, Grid World and continuous Fetch Reach, it remains incompetent in dealing with complex control tasks including Fetch Push and Fetch Slide, in all of which HTRPO can learn good policy. The reason is that for TRPO, it is basically impossible to acquire a positive reward at the beginning of the training in such environments, which makes the policy updates meaningless.
How does each component of HTRPO contribute to its effectiveness?
In both Figure 2 and Figure "HTRPO with KL1" and "HTRPO without WIS" performs much worse than the complete version of HTRPO. When we estimate the KL-divergence using the "vanilla" KI--divergence defined equation 9 it causes severe instability, which means that the estimated KL-divergence can be negative with an unacceptable probability. Considering the prac- ticality of the experiment, the corresponding iteration will be skipped without any updates of the policy in this senario. Given the phenomenon stated above, the final performance of "HTRPO with KL1" is much worse and more unstable all environments. As for the study of Weighted Impor- tance Sampling, it is widely known for significantly reducing the variance (Bishop) (016), which once again proved by the results of "HTRPO without WIS". Admittedly, we can see that the performance of "HTRPO without WIS" matches the full version of HTRPO in several simple envi- ronments in Figure 2 Push and Fetch Slide, the variance is detrimentally larger than that in simple environments, In short, the performance of "HTRPO without WIS" has a severe degradation comparing to the full version of HTRPO.
(a)-(d) and Figure (a). However, for more complex environments like Fetch
How is the performance of policy gradient methods trained with hindsight data in continuous environments?
As mentioned in Plappert et al. 2018)) it still remains unexplored that to what extent the policy gradient methods trained with hindsight data can solve continuous control tasks. In this section, will provide the answer. We implement HTRPO in continuous control tasks including Fetch Reach, Fetch Push and Fetch Slide. HPG tested as well for comparison. From the results, we can see that with the help of input normalization, HPG can learn a valid policy in continuous control tasks. Still, HTRPO performs much better than HPG in all three environments, benefiting from faster and more stable convergence. As illustrated in Figure 3 HTRPO eventually achieves an average success rate 92% for Fetch Push and 82.5% for Fetch Slide.
How sensitive is HTRPO to network architecture and some key parameters?
To study the sensitivity of HTRPO to different network architectures, we observe the performance of HTRPO with different network settings. From the results demonstrated in Appendix F221 HTRPO achieves commendable performances with all three different network architectures while HPG only converges under certain settings. As for the sensitivity of HTRPO to key parameters, we mainly ob- serve the impact of different number of alternative goals. Based on the learning curves in Appendix we can see that Hindishgt TRPO with more alternative goals achieves better converging speed.
F2.2
6 CONCLUSION
We have extended the monotonically converging on-policy algorithm TRPO accommodate sparse reward environments by adopting the hindsight methodology. The optimization problem in TRPO scrupulously derived into hindsight formulation and, when the KL-divergence in the constraint function is small enough, it can be tactfully approximated by another f-divergence in order to re- duce estimation variance and improve learning stability. Experimental results on a variety of en- vironments demonstrate effective performances of HTRPO, and validate its sample efficiency and stable policy update quality in both discrete and continuous scenarios. Therefore, this work reveals HTRPO's vast potential in solving sparse reward reinforcement learning problems.
ACKNOWLEDGMENTS
We greatly acknowledge all the fundings in support of this work.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, pp. 5048-50585 2017.
C.M. Bishop. Pattern Recognition and Machine Learning. Information Science and Statistics. Springer New York, 2016. ISBN 9781 493938438
S. Boyd, S.P. Boyd, L. Vandenberghe, and Cambridge University Press. Convex Optimization. Berichte über verteilte messysteme. Cambridge University Press, 2004. ISBN 9780521833783.
Mare Peter Deisenroth, Gerhard Neumann, Jan Peters, et al. A survey policy search for robotics. Foundations and Trends in Robotics, 2(1-2):1-142 2013.
Meng Fang. Cheng Zhou, Bei Shi, Boqing Gong, Weitao Xi, Tianzhou Wang, Jia Xu, and Tong Zhang. DHER: Hindsight experience replay for dynamic goals. In International Conference on Learning Representations, 2019.
Shixiang Gu, Timothy Lillicrap, Zoubin Ghahramani, Richard Turner, and Sergey Levine. Q-prop: 2016.
Sample-efficient policy gradient with off-policy critic. arXiv preprint
Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems, pp. 66-83. Springer, 2017.
Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang. SM Eslami, Martin Riedmiller, et Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv: aeli:02070000200 2017.
David Held, Xinyang Geng, Carlos Florensa, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in neural information processing systems, pp. 4565-4573, 2016.
Jonathan Ho, Jayesh Gupta, and Stefano Ermon. Model-free imitation learning with policy opti- mization. In International Conference on Machine Learning, pp. 2760-2769, 2016.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pp. 1109-1117, 2016.
Niels Justesen, Philip Bontrager, Julian Togelius, and Sebastian Risi. Deep learning for video game playing. IEEE Transactions on Games, 2019.
Tejas Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in neural information processing systems, pp. 3675-3683, 2016.
Hoang Le, Nan Jiang, Alekh Agarwal, Miroslav Dudik, Yisong Yue, and Hal Daumé, III. Hierar- chical imitation and reinforcement learning. Proceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pp. 2917-2926, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. The Journal of Machine Learning Research, 17(1):133--373, 2016.
Andrew Levy, Robert Platt, and Kate Saenko. Hierarchical reinforcement learning with hindsight. In International Conference on Learning Representations, 2019.
Jeffrey Mahler and Ken Goldberg. Learning deep policies for robot bin picking by simulating robust grasping sequences. In Conference robot learning, pp. 515-524, 2017.

---

Under review as a conference paper at ICLR 2020
Kenneth Marino, Abhinav Gupta, Rob Fergus, and Arthur Szlam. Hierarchical RL using ensem- ble of proprioceptive periodie policies. In International Conference on Learning Representations, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Trust-pel: An off-policy trust region method for continuous control. arXiv preprint 177.:01707,1191,, 2017.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, pp. 3303-3313, 2018.
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pp. 7559-7566. IEEE, 2018.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Vi- sual reinforcement learning with imagined goals. In Advances in Neural Information Processing Systems, pp. 9191-9200, 2018
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, volume 99, pp. 278-287, 1999.
Jan Peters and Stefan Schaal. Reinforcement learning of motor skills with policy gradients. Neural networks, 21(4):682-697, 2008.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Pow- ell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, al. Multi-goal reinforce- ment learning: Challenging robotics environments and request for research. arXiv preprint arXiv: 09444, 2018
Michael JD Powell and J Swann. Weighted uniform samplinga monte carlo technique for reducing variance. IMA Journal of Applied Mathematics, 2(3):228-236, 1966.
Deirdre Quillen, Eric Jang, Ofir Nachum, Chelsea Finn, Julian Ibarz, and Sergey Levine. Deep reinforcement learning for vision- robotic grasping: A simulated comparative evaluation of off- methods. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 6284-6291. IEEE, 2018
Paulo Rauber, Avinash Ummadisingu, Filipe Mutz, and Jürgen Schmidhuber. Hindsight policy gradients. In International Conference on Learning Representations, 2019.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International Conference on Machine Learning, pp. 1312-1320, 2015.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889-1897, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint 10X0::1506602438,, 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 107.007777. 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):484, 2016.
Ricahrd S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. Adaptive Computation and Machine Learning series. MIT Press, 2018. ISBN 9780262352703.
10

---

Under review as a conference paper at ICLR 2020
Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1):181 211, 1999. ISSN 0004-3702.
Richard Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural informa- tion processing systems. pp. 1057-1063, 2000.
Vivek Veeriah, Junhyuk Oh, and Satinder Singh. Many- goals reinforcement learning. arXiv preprint arXiv:1806.0960,, 2018.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg. David Silver, and Koray Kavukcuoglu. FeUdal networks for hierarchical reinforcement learning. In Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceed- ings of Machine Learning Research, pp. 3540-3549, International Convention Centre, Sydney, Australia, 06-11 Aug 2017.
Cathy Wu, Aravind Rajeswaran, Yan Duan, Vikash Kumar, Alexandre M Bayen, Sham Kakade, Igor Mordatch, and Pieter Abbeel. Variance reduction for policy gradient with action-dependent factorized baselines. arXiv preprint 1803v 07246, 2018

---

Under review as a conference paper at ICLR 2020
A PROOF FOR OBJECTIVE FUNCTION
A.1 DERIVIATION FROM EQUATION 3 TO EQUATION
With no influence the optimal solution, multiply equation 3 by constant 1-7-
To (a|s)
Lg(0)
Ap(s, a)
1 Pg (p
(T((ass)
E
7 P(s =
To (a)ss)
Ap(s,a)
ne (T((as) (
$ES
1-7
Ta(a st)) TT( (s))
E
Ap(st, as)
t=0
Ag(8e,0t) at)
(16)
(T) 1=0
A.2 THEOREM 3.1
Theorem 3.1 (HTRPO Objective Function). For the original goal g and an alternative goal g', the object function of HTRPO (0) is given by:
EI
Ta (a skk skk
ne (a(alser
Aglde, an.s))
Lg( = E
E
(a 8k, g) T6 (a(al10t,g")
(T)p(T(s))
in which, 7 81, a1 82, 022 ... St, Cle
Proof. Starting from equation 6 for every time step the expectation, denote
no (a |8,, 9) ?
Lg(0,1) = E
E (Tpp((rl))
vg( Ag(se, ar
(17)
so that
Lx(0) Σ (.0 t).
(18)
1=
Split every trajectory into Ti and 72 where Ti $1, 01 $2 02 Sc, ag and T3 Se+1, ac+1. then
n (o g) Ag(se, (ts
Lg(0,1) E
E m 9)
E pampa(talm m1
(19)
For that 7a ma(aa))s.g) (g(02191.9)
Ag az g) is independent from 72 conditioned on Tı,
Troaalls, 9) TT(as (1.g)
Ag(se, a..
tg(0,t)
E Ti 7 (Tilg)
E
Te(an 15c, g) TT(aa sc, 9) Ag(se, ac.
T2 TL (Talg)
7 Taa 9)
Ta(arlle Ag(se, de, 9)
E Pa
(20)
Thus,
ye To( g) Ag (se, ac 0c.9)
Lg(0)= =
E an
(21)
1=
12

---

Under review as a conference paper at ICLR 2020
Following the techniques of importance sampling, the objective function be rewritten in the form of new goal g'
Pa(Si: ( no (o(aa)) 9) Pa (1 a1:rlg) TT (T(
Ag azs 9')
Lg(0)
E
(22)
Sa: Pale a la)
Furthermore, given that
p(81s, an (nsgg) = p(8) (())))(( (1. 9)
II plax|ox g)p(s)))p()e))) k=1
(23)
after expanding the objective function and cancelling terms,
Ta Ta(ak lsk 9¹)
Tz(ax sk. 9) Ta Tg(aelse.g)) 9¹) Ao Ag(se, ae. []
7 To (a(a)10r 9')
[0(0)
E
II
5hapea Pp an lg)
(24)
A.3 THEOREM 3.2
Theorem 3.2 (Gradient of HTRPO Objective Function). For the original goal and an alternative goal g', the gradient (0Lg(0) of HTRPO object function with respect 0 given by the following expression:
To (sk ge VoTo(arlse:g) Ag(st, a. II A=1
[
VoLg(0) = E of
E (rlp)))
Ta(ak sk,g) Tg(arlsi.g)
in which 7 51, 01, 02 ..... 8t,
Proof. Starting from equation 24, since that To(a St. g') is the only term relavant to the corre- sponding gradient of the objective function is computed by:
To [o(al(sk.g)) g') VOTo(aelse: Ag ar ar, II
|1
VoLg(0) =
E 01: Pa Sas (1 19)
Ta(ak|sk.9) Tg(ails(.g")
t=
(25)
13

---

Under review as a conference paper at ICLR 2020
B PROOF FOR CONSTRAINT FUNCTION
B.1 THEOREM 4.1
Theorem 4.1 (Logarithmic Form of Constraint Function). Given policies To TT(als) and ma(a))),
the expectation of their KI.-divergence over states 8 ~ P8(8) is written as:
E
[
E
log (ogtaasss) - log [T(ass)]
(p
5,ampy (s,0)
Proof. Expand the expectation in equation by the definition of KL-divergence,
E [Dx[((a(als]]<<[((ss]] = sp)p(())
E
E amlo(aass)
log (a(a)s)
pppa(s)
E log (g(a(s))) sxpp(s))a~_g((ls)
log (a]s))
(26)
B.2
THEOREM 4.2
Lemma B.1 Given distibutions p(x) and g(x), g(x) p(x) ((()), in which n(()) is the
variation of (x) at p(x).
B log (()) log (1)) = E (logp(x) - log (x)))
+ + +b[o((a)))]
(27)
Proof. Consider the second order Taylor expansion of log ((x) at p(z),
1
(xx p(z) p(x) (2))
20(x)² (z)) o(n(z)))
(28)
For the left side of equation 27
IE ((ogp(>) log (x))
(1x)) + 2p(r)² (x)) -(((x)))
p(x)
/ p(x)
((x) + 2p(z)2
2p(z)2 (1x) 2 d(((=))))p(()d dr
(xx) p(z))(((<x3)) dr.
(29)
2p(x)
For the first term on the right side of equation
27
1 -(((x)))))
2 (log p(x) - log {x)))
E
(x))
p(x)
E
p(x))² (x))
+ o((()))))
2
p(x)2 (xx) + c(((z)))))le de
2
= (2p(2)) (x)) ,(==)((((=)))) de
(20(z) (x) p(x))(((=)))) f p(x))(((a)3) de
= [log((x) log (x)) [o(](,)))]
(30)
Theorem 4.2 (Approximation of Constraint Function). For policy Ta(a(s) and (a(ss), and for
Te Te (als) Ta(ass)
[log log [a(ss)]]
E a sp
I
(log [((a)s) - log =)({ss)))
s,arpp(s.0)
+
R (p(aa)
[o(n))]]
10
14

---

Under review as a conference paper at ICLR 2020
Proof. Based Lemma A.1, let p(æ)) xz(als) and g(x)) To(als), equation 27 results in equation 10
B.3 THROREM 4.3
Lemma B.2 For any random variable Y E [0,0.5].
Var(Y & Var(Y).
(31)
in which Var(Y) denotes the variance of Y. Proof.
Var(Y) - Var(V2)= = E(Y3) [E(T)] -(++) + [[E(3)]]
|E(Y E(Y*) - [B(Y)] [EEM) [E
+)(1++)) - -XLL--
Cov(Y(1 v),
(32)
Denote Xi(Y) Y) and X2(Y) = Y Y). Then,
Var(Y) - Var(Y)) = Cov(X1,
= = (X2 E(X2))]
(33)
There always exists Yo E [0, 0.5] that saticefies X2 E(X2). When Y Yo, let X1 (Y)) pl which bn is a constant. Then the equation can be converted by the following steps:
Var(Y) - Var(Y²) = E [X1 m1) (X2 - E(x2)) + E B(X2)]
[X1 m4) (X2 E(x)))
(34)
Thus, when Y Yo, the two factors in equation simultaneously.
34 (X1 11) and (X2 E(X2)) equal to
Also, it is easy notice that when Y E 0.,0 X2 and X₂ are strictly increasing with the increase of Y. Thus, (X, - (1) and (X₂ - E(X,)) are either both positive or both negative, if not zero Therefore,
Var(Y) Var(Y 2 0.
(35)
Lemma B.3 For any random variable Y,
Var([Y)) S Var(Y),
(36)
in which Var(Y) denotes the variance of Y.
Proof. Apparently,
1 (1()) dy 2 f f(v) dul-
(37)
Consequently,
E(YD) 2 |EY)
(38)
For that
Var(Y)=B(YY' = - (E(())²
(39)
have
Var([Y & Var(Y).
36
15

---

Under review as a conference paper at ICLR 2020
Theorem 4.3 (Variance of Constraint Function). For policy TT(als) and no(als), let Var denotes the variance of a variable. When log (a(a)s) log To (als) E [-0.5, 0.5], then
(log Ta(als) - log TT(a)s))²]
Var 8,0m:pp(s.0 (s,a)
[log [Log (g(s) log ma(ass) 8,0mpy(s,d
Var
2
Proof. let = (alls) - log <a(a)))]
Given Lemma B.2,
Var (,ar)
log Tg((s) - log To(als) v
Var (,
[l (gll) - log {a(a)))]
(40)
Given Lemma B.3,
Var s,amopps,,) (3,a)
[ggga((ass) - log no (a|s)]
<
Var p,
[log (a(a - log no (a|s)]
(41)
With the transitivity of inequality, combining equation
40 and equation 41 we know that
llg (o((s))) log To(als) s,0-pg(s.a)
Var
Var s,0=pp(s.0)
[log Ta(a)s) - log log
(42)
B.4
THEOREM 4.4
Theorem 4.4 (HTRPO Constraint Function). For the original goal g and an alternative goal g', the constraint between policy Ta(als) () and policy Te (a(a))) is given by:
a (a (log Ta(arlse,))) g') log 91)
E
<e'
13
TopPl (rla)
2
T (0*1)k
k=1
which
Proof. Starting from equation
9 the constraint condition is written as:
[log TT(als) log (a(a]))] S
(43)
s,
Given Theorem 4.2,
E (s
2 (oggg(al))) - log To (als))) Se
(44)
Multiply the constraint function by a constant T-
E
flog Ta (a(as) log (a(ss)))
5 1-7
(45)
2
pp
Denote the constraint function as (0).
7t P(se =)
fa(0) 1-7
(log (g(aas) log To (a(s)))
06
Σ
(log TT(ar(s)) log To (ac(ss)))
t=0
80
7' (log log -
(46)
Tepp(r)
To write the constraint function in goal-conditioned form,
(0) EE E >>pp(=
Hd (log Tz(a se. 9) log To(oolse. g)
(47)
t=
16

---

Under review as a conference paper at ICLR 2020
In similar way with the proof for Theorem 3.1, denote every time step of fū(0) as fa(O, t), in other words,
To(0)
(0(0,))
(48)
2=0
for trajectory = 81, a1, 82, a2
St. and 72 St+1, ap+1...
(log ( st.9) - log (o (t.)))
t) = 9
E
E
(49)
(ImPo(Tals))
2
For that 2/2 (log Ta(aelse g)
log To (a St, g)}² is independent from 72 conditioned on T1.
f0( E
E
? (log Th (aclss g) log log Tam-Po(rala)
Pa
E
E
2
(log ny (ecls) 9) lo 9))²
(50)
Accordingly,
1 (log Ta(a0lses g) log mo(aila.. 9))
(ā(0)
E S1: Po ar
(51)
Furthermore, by importance sampling, for new goal g'. the constraint can be converted to the following form
P5 (1 (log Tg(aelse. log (aglo(a)))
(a)
E
E
(52)
P8 (1
in which = $1, a1 $2, 02 ..., St. at. Denote = - Based on equation 23 by expanding and canceling terms, the constraint condition can be written as
T (ax 9¹) (log Ta(aeloe, - log (oaess...,
E
E
Se'
13
Trupe (r
2
Te (ax 8k,
r=0 A=1
17

---

Under review as a conference paper at ICLR 2020
C SOLVING PROCESS FOR HTRPO
C.I HTRPO ESTIMATORS
Based on the final form of HTRPO optimization problem, this section completes the feasibility of this algorithm with estimators for the objective function and the KL-divergence constraint.
Given a dataset of trajectories and goals = {T((9, (g(0)) 1. Mi=1. each trajectory ((1) is obtained from interacting with the environment under a goal g()) In order to generate hindsight experience, we also need to sample a set of alternative goals G = {g HTRPO optimization problem with dataset D can be derived as follows:
The Monte Carlo estimation of
max #
II
Tala
g') ? VoTo(a (a 1 gg)
Ag(a
( .8 .. (
(53)
Ta(a
To(ak (6)
,9)
93EG
8.1.
Talaz
5k g')
(log TT(a 10 .g') log To (a 106 ,g))²
Se',
2
k=1 TTa ak 1k (())
( (i)
9'Eg i= 1=U
(54)
in which == NN, and g' is supposed to follow uniform distribution. However, in experiments, we follow the alternative goal sampling method of HPG (Rauber et al. 2019). As result, the goals of training data actually follow the distribution of alternative goals instead of uniform distribution, and the objective and KL expectation will be estimated w.r.t. the alternative goal distribution. Therefore, during the learning process, our algorithm is encouraging the agent to achieve the alternative goals. Such mechanism serves as a mutual approach for all hindsight methods (Andrychowicz al. 2017 Rauber et al. 2019] Plappert et al., 2018), which can be seen as a merit, for the intention is to guide the agent to achieve the alternative goals and then generalize the original goals.
However, as discussed in Rauber et al. (2019), this kind of estimation may result in excessive vari- ance, which leads to unstable learning curve. In order avoid instability, we adopt the technique of weighted importance sampling introduced in Bishop (2016) and further convert the optimization problem to the following form:
Tala (0) ( (9.a") Fa 1 (0)
(4) (8) (a
max 8
N mala (2 .0) 1k=1 aa(a 10t (919)
Ta(a 5k 'g)
15 .9) ,8
(2)
(1) Ag(ak
,g'
(55)
00Eg
0
Ne La (6(" 9') (a 18t g(th)
N
s.t.
y'
(log T (s(()'9))) ,') log
T (0) 196" )))
<
2
mata gg)
g'EG
1=0
Ta(a 16 g())
j=1k=1
(56)
We provide an explicit solution method for the optimization problem above Appendix C.2
While introducing weighted importance sampling may cause certain level of bias which is identical to that of HPG (Rauber et al. 2019). the bias is to decrease in inverse ratio with regard to the increase of the data theoretically (Powell & Swann 1966). Given the limited resources, need to tradeoff' between reducing bias and enlarging batch size. By picking a appropriate batch size, the improvement of weighted importance sampling well demonstrated in the experiments.
C.2
SOLUTION METHOD FOR HTRPO
For the HTRPO optimization problem, briefly denote the optimization problem in expression 55 and 56 as:
max f(0)
#
18

---

Under review as a conference paper at ICLR 2020
s.t. g(t) sc
(57)
For any policy parameter 0 in the neighborhood of the parameter 0, approximate the optimization problem with linear objective function and quadratic constraint:
max f(0) (e]( 0)
s.t. g(0) + Veg(0) - 0- s
(58)
Noticeably, g(0) and Vog(0) 0. which further simplifies the optimization problem to the following form:
max 1(0) (0) 0)
s.l. 5 (0 V Se.
(59)
Given convex optimzation problem with a linear objective function under quadratic constraint, many well-practiced approaches can be taken to solve the problem analytically, among which we Tucker(KKT)
adopt the Karush-Kuhh Karust-KKun-TTTckeKKKKK conditions (Boyd al., 2004).
For Lagrangian multiplier
X,
2 -0- (00-0)V39(0016 0) -650
A
A1 00 - -) -
vof((0) A
(60)
Expressions in 60 form the KKT conditions of the optimization problem. Solving the KKT condi- tions,
2c'
0 = + V V
-
(61)
The policies, however, in this paper are in the form of a neural network, which makes it extremely time-comsuming to compute the Hessian matrix. Thus, we compute (o) with con- jugate gradient algorithm by solving the following equation:
[))( = (o)
(62)
in which [V]g(0)] can be practically calculated through the following expansion:
[V)g(0]]= = Vol(Veg(())=a)
(63)
19

---

Under review as a conference paper at ICLR 2020
D ALGORITHM
Algorithm Hindsight Trust Region Policy Optimization
Input:
Training batch size batchsize
Max number training time steps T mar
Policy /
Q-function Φ
Data Buffer Borigin with its max size equal to batchsize
Initialization
Output:
Optimized Policy l"
1: for iteration = to T /wttchssczeeeeeee do
2: while Borigin is not full do
3: Sample trajectory
{st, Oes Tt, Tt, 8141,9 TTe (ax g))}{= using current policy 0;
4: Borigin Borigin NT;
5:
end while
Collecting data
6: Sample alternative goals G (1) N from achieved goals in B Borigini
7: Birain Ø
8:
for g((() in G do
9:
for in Borigin do for 0 to T do
11:
Compute No (arlst,,, (+)); Modify reward relg - rz (r);
12:
end for T|g = {(st, at, ri rr(g S141, no gf
14:
15:
Brroin = Beroin :
end for end for
17:
Generating training data
Use Btrain optimize policy 9 with objective 55 and constraint 56 following Section C.2
19: Borigin = Ø;
end for
21:
return optimized policy 8";
20

---

Under review as a conference paper at ICLR 2020
E
HYPERPARAMETERS
E.1
HYPERPARAMETERS OF DISCRETE ENVIRONMENTS
Table I: Hyperparameters of Discrete Environments
Flip
FlipBit16
EmptyMaze
training time steps
x 104
5 x 104
5 x 104
batch size eg damping
128
256
256
le-2
le-2
1e-3
reward decay
0.9
0.9
0.95
max KI. step
1e-3
1e-3
1e-3
critic optimizer
Adam
Adam
Adam
critic learning rate
5e-4
5e-4
5e-4
critic updates per iteration
10
10
sampled goal number
FourRoom
FetchReach
FetchPush
training time steps
104
5 x 10a
x 10
batch size cg damping
256
800
800
le-3
1e-3
1e-3
reward decay
0.95
0.98
0.98
max KI. step critic optimizer
1e-3
3e-6
3e-6
Adam
Adam
Adam
critic learning rate
5e-4
le-4
le-4
critic updates per iteration
10
10
10
sampled goal number
00
30
30
E.2 HYPERPARAMETERS CONTINUOUS ENVIRONMENTS
Table 2: Hyperparameters of Continuous Environments
FetchReach
FetchPush
FetchSlide
training time steps
10a
10
2 x 10
batch size
800
1600
3200
cg damping
le-3
1e-3
1e-3
reward decay
0.98
0.98
0.98
max KL step entropy weight
le-5
le-5
le-5
0
le-4
le-4
critic optimizer
Adam
Adam
Adam
critic learning rate critic updates per iteration sampled goal number
5e-4
5e-4
5e-4
20
20
20
100
100
100
21

---

Under review as a conference paper at ICLR 2020
F EXPERIMENTS
In this section, we provide a more comprehensive demonstration for the experiments of HTRPO. In detail, section F.I narrates full introduction to each environment; section F2 gives out the sen- sitivity analysis of HTRPO including the performance under different network architectures and different numbers of alternative goals, in which we strictly adopt the control variable method and only the studied parameter is altered; section E.3 shows the supplementary materials of the experi- mental data including learning curves and success rates during the training process. We finetune the hyperparameters according to experience without hyperparameter search due to limited computing resources.
F.1 ENVIRONMENTS
k-Bit Flipping. In each episode of this experiment, two arrays of length k are generated. The first array is initialized with all 0's while the second one, usually regarded as the target array, is generated randomly. At each time step, the agent able to flip one bit of the first array from 0 to O. Once the first array is exactly the same with the target array, the agent reaches the goal state and is then rewarded. The maximum number of time steps is In this experiment, we observe the performance of HTRPO under conditions that & 8 and that k = 16 respectively. The general process of an 8-Bit Flipping task demonstrated Figure (a).
from
Grid World. In this experiment, the agent starts a position of an 11 x 11 grid with intransitable obstacles, and trying to reach another randomly chosen position this grid. The agent allowed to move up, down, left and right at each time step. Moving into obstacles makes the agent remain in its current position. States this environment is represented by 2-dimensional integral coordinates and the maximum number of time steps is 32. In Empty Maze environment, there is impassable obstacles other than the outer boundary, and the agent starts at the upper-left corner of the grid. In Four Rooms environment (Sutton al. 1999). walls separate the grid into 4 rooms, each with access to its adjacent rooms through single openings. Example cases of Empty Maze and Four Rooms environments adopted in this paper are demonstrated in Figure (b) and (c).
Fetch. Fetch environment contains 7-DoF Fetch robotic arm with a two-fingered parallel gripper in (amulationPPlapertt et al.) 2018). In Fetch Reach environment, target position is randomly chosen and the gripper of Fetch robotic arm needs to be moved upon it. In Fetch Push, the task for the robotic arm is push a randomly placed block towards the goal state, anther randomly picked position, which is represented by 3-dimensional Cartesian coordinate. In Fetch Slide, the robotic arm needs to exert force on the block for it slide towrds chosen goal certain distance. A pictorial demonstration of this environment is shown in Figure in which the red represents the goal position. For the discrete Fetch environment, detailed settings follow that in Rauber et al. (2019; for the continuous version, the configurations of legal actions and states follow that in Plappert et al. (2018). The maximum number time steps is 50. As in (lappert et al. (2018), endure 5cm target scope centered around the goal position for Fetch Reach and Fetch Push, and the tolerance is set as 20cm for Fetch Slide as in Andrychowicz et al. (2017).
22

---

Under review as a conference paper at ICLR 2020
F.2 SENSITIVITY ANALYSIS
F.2.1
DIFFERENT NETWORK ARCHITECTURES
HTRPO
our HPG
official HPG
Average Return
2
2
2
Time stops (1144)
Timp stops (1o4)
Time stops (1c4)
(a) 8-Bit Flipping x 16
(b) 8-Bit Flipping 2
(c) 8-Bit Flipping 2
10
10
10
Average Reaurn
4
2
2
Time steps (194)
Time staps (164)
Time steps (1e4)
(d) 16- Bit Flipping 16
(c) 16- Bit Flipping 2x64
(f) 16-Bit Flipping 256
25
25
Average Rieurn
15
10
10
5
5
2
2
Time steps (1e4)
Time
(g) Empty Maze 1 16
(h) Empty Maze 2 64
(1) Empty Maze 2
Average 10
10
10
steps (1e4)
steps (1e4)
Time steps (1e4)
Four Rooms 1x 16
(k) Four Rooms 2x64
(I) Four Rooms 2x656
Figure 4: Evaluation curves with other network structures. Horizontally, figures in each line illustrate the perfomances in one environment with different network architectures. Vertically, each column illustrate the performances of one kind of network architecture in different environments.
In this experiment, we observe the performance of HTRPO with different network architectures. Specially, we implement the proposed algorithm under 3 different network settings, i.e. networks with a 16-unit layer, two 64-unit layers and two 256-unit layers respectively. For the record, all parameters and other settings remain the same aside from the network architecture. As demonstrated in Figure 4 each shows the performance under different network architecture settings for each environment. A general conclusion can be drawn that networks with more hidden layers and more neurons help to speed up the convergence. However, one difference is that for HTRPO, it converges
23

---

Under review as a conference paper at ICLR 2020
quickly in all the settings, while for HPG, it converges much slower especially when the network architecture is simple. We believe that the iteratively searching of optimal solution in the trust region helps the network converge rapidly and is more robust to different network architecture.
F.2.2 DIFFERENT NUMBER OF ALTERNATIVE GOALS
In this experiment, how the number of alternative goals, as a key parameter, affects the performance of HTRPO is studied. We conduct all the experiments, both discrete and continuous with different number alternative goals. For discrete environments, we set the number of alternative goals to be 10, 30, 100 and oo in turn. For continuous environments, we compare the performance under 10, 30, 100 alternative goals respectively. The evaluation curves are shown in [ From the results, we see that in simple discrete environments, alternative goals produce the fastest convergence. In complex and continuous environments, 30 and 100 alternative goals lead to comparatively good performance. It is not hard see that Hindishgt TRPO with more alternative goals achieves better converging speed, which may be credited to the corresponding increase on training samples. This is, some extent, similar data augmentation.
10 alternative goals
30 alternative goals
100 alternative goals
DC alternative goals
10
25
Average Neturn
20
15
2
3
Time steps (1e4)
Time steps (164)
Time steps (1e4)
(a) 8-Bit Flipping
(b) 16-Bit Flipping
(c) Empty Maze
Average Fieturn
15
12
18
3
0
8
16
20
steps (1e4)
Time steps (1e5)
steps (1e5)
(d) Four Rooms
(e) Discrete Fetch Reach
Discrete Fetch Push
60
35
30
Avarage Return
-40
25
20
20
15
10
5
Time stops (1c5)
Time stops (te5)
Timo stops (1e5)
(g) Countinuous Fetch Reach
(h) Countinuous Fetch Push
(i) Countinuous Fetch Slide
Figure 5: Evaluation curves for different number of alternative goals: 8-Bit Flipping, 16-Bit Flip- ping. Empty Maze, Four Rooms, Discrete Fetch Reach, Discrete Fetch Push, Continuous Fetch Reach, Continuous Fetch Push and Contiuous Fetch Slide. The full lines represent the average evaluation over 10 trails and the shaded regions represent the corresponding standard deviation.
24

---

Under review as a conference paper at ICLR 2020
F.3
COMPREHENSIVE TRAINING CURVES
F.3.1
TRAINING CURVES
HTRPO
our HPG
HTRPO with KL1
HTRPO without WIS
TRPO
25
Average Relum
8
20
15
Time steps (1 e1)
Time steps (164)
Time steps (104)
(a) 8-Bit Flipping
(b) 16-Bit Flipping
(c) Empty Maze
15
Awerage Return
24
12
18
0
12
16
20
Time steps (1e4)
Time steps (1e5)
Time steps (1e5)
(d) Four Rooms
(c) Discrete Fetch Reach
(f) Discrete Fetch Push
Average Return
40
300
20
10
5
5
0
2
3
8
12
12
steps (1e5)
steps (1e5)
steps (1e5)
Countinuous Fetch Reach
(h) Countinuous Fetch Push
(i) Countinuous Fetch Slide
Figure 6: Training curves for all environments: 8-Bit Flipping, 16-Bit Flipping, Empty Maze, Four
Rooms, Discrete Fetch Reach, Discrete Fetch Push, Continuous Fetch Reach, Continuous Fetch Push and Contiuous Fetch Slide. The full lines represent the average evaluation over 10 trails and the shaded regions represent the corresponding standard deviation.
25

---

Under review as a conference paper at ICLR 2020
F.3.2 SUCCESS RATE
In this section, we demonstrate the success rates of HTRPO during both evaluation and training. For the record, the actions during the training process are sampled by the distribution oulput by the network while during the evaluation process, we adopt a greedy strategy to choose the action by the mean value of the distribution. 3ists the success rates of Fetch Push and Fetch Slide during evaluation, in which the ultimate values reflect the mean computed with 1000 test results in each iteration. They are the only two environments listed for they are the most complex ones. Figure illustrates the success rate curves during the training process.
Table Evaluation success rate for Fetch Push and Fetch Slide
Fetch Push
Fetch Slide
Time step
our HPG (%)
HTRPO (%)
our HPG (%)
HTRPO (%)
480k
56.4
63.2
36.1
46.4
960k
65.6
91.9
59.5
79.9
1920k
87.2
89.7
61.2
82.5
HTRPO
our HPG
HTRPO with KL1
HTRPO without WIS
TRPO
Success Rale
0.8
0.8
0.8
0.4
0.4
0.4
0
Time steps (1e4)
Time steps (1e4)
Time steps (1e4)
(a) 8-Bit Flipping
(b) 16 Bit Flipping
(c) Empty Maze
1.0
1.0
1.0
Success Pale
0.8
0.8
0.6
0.2
0.2
0.2
2
3
0
3
4
5
0
12
16
20
Time steps (164)
Time steps (165)
Time steps (185)
Four Rooms
(c) Discrete Fetch Reach
(f) Discrete Fetch Push
1.0
1.0
Rate
0.8
0.8
0.8
0.6
0.8
0.4
0.4
0.2
0.2
6
20
0
12
16
20
Time steps (1e5)
Time steps (1e5)
Time staps (165)
(g) Countinuous Fetch Reach
(h) Countinuous Fetch Push
(i) Countinuous Fetch Slide
Figure 7: Success Rate for all environments:8 B-t Flipping, 16-Bit Flipping, Empty Maze, Four Rooms, Discrete Fetch Reach, Discrete Fetch Push, Continuous Fetch Reach, Continuous Fetch Push and Contiuous Fetch Slide. The full lines represent the average evaluation over 10 trails and the shaded regions represent the corresponding standard deviation.
26

---

Under review as a conference paper at ICLR 2020
F.4 COMPARISON WITH DENSE REWARD TRPO
Figure g demonstrates the success rate for HTRPO, TRPO with sparse reward and TRPO with dense reward for Continuous Fetch Reach, Continuous Fetch Push and Contiuous Fetch Slide. The performance of TRPO with dense rewards similar to that of TRPO with sparse rewards: for FetchReach, converges much more slowly than HTRPO while for FetchPush and FetchSlide, doesnt work in the whole training process (2 million time steps). Similar conclusions can also be found in some other literatures of Hindsight (Plappert et al. 2018). Therefore, it can be concluded that HTRPO outperforms it significantly.
HTRPO
TRPO with Sparse Reward
TRPO with Dense Reward
1.0
1.0
1.0
Succass Rate
0.6
0.6
0.8
0.2
0.2
0.2
5
4
8
12
16
20
0
12
16
20
steps (1e55)
stops (1e5)
stops (1c5)
(a) Countinuous Fetch Reach
(h) Countinuous Fetch Push
(c) Countinuous Fetch Slide
Figure 8: Success Rate for HTRPO, TRPO with Sparse Reward and TRPO with Dense Reward for Continuous Fetch Reach, Continuous Fetch Push and Contiuous Fetch Slide. The full lines represent the average evaluation 10 trails and the shaded regions represent the corresponding standard deviation.
27

---

Under review as a conference paper at ICLR 2020
F.5 COMPARISON OF DIFFERENT KL EXPECTATION ESTIMATORS
Figure 9 demonstrates the estimation results of KL divergence expectation in the training process. From these data can that in the experiments, the approximation of equation 13 can signifi- cantly reduce the variance of KI. expectation estimation. Besides, the comparison of performance between HTRPO and HTRPO with KLI also shows the efficiency of this approximation, which helps improve the final performance significantly, Both "HTRPO" and "HTRPO without WIS" use the estimation method in equation 13 with one difference being that "HTRPO without WIS" doesn't adopt weighted importance sampling. Thus, from Figure we can see that "HTRPO" demonstrates the least variance.
HTRPO
HTRPO with KL1
HTRPO without WIS
KL Divergance (1e-8)
2
3
8
Time steps (1e5)
Time steps (1e5)
Time steps (1e5)
(a) Countinuous Fetch Reach
(b) Countinuous Fetch Push
(c) Countinuous Fetch Slide
Figure 9: Estimation of KI. Divergence Expectation for different variants of HTRPO Continuous Fetch Reach, Continuous Fetch Push and Contiuous Fetch Slide. The full lines represent the average evaluation over trails and the shaded regions represent the corresponding standard deviation. The curves for KLI are comparatively lower than those of equation 13 Note that in TRPO, the linear search mechanism adjust the updating step size according to the estimation of KL divergence expectation. It sets threshold to constrain the KL divergence. For those above the threshold, the updating step size will reduced ensure that the estimation of KL divergence estimation falls within the threshold. This explains why the curves for KI.1 comparatively lower. However, since the estimation of KI. divergence expectation HTRPO falls near the expected value, such step size adjustment rarely triggered. This benefits from the much lower variance of equation 13
28


Reviews:
[{"review_id": "rklIz1_4tH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary* : This paper augments the TRPO policy optimization objective with hindsight data, where the hindsight data is generated from goals based on trajectories. The key contribution of the paper is based on deriving an on-policy adaptation of hindsight based TRPO, that can be useful for sparse reward environments. The paper draws ideas from existing papers such as HPG and considers the IS based variant of HPG for on-policy, similar to TRPO, that can achieve monotonic performance improvements. Furthermore, the authors introduce a logarithmic form of constraint, by re-deriving the KL constraint and leading to a f-divergence based constraint, which is argued to have useful effects in terms of lowering the variance. Experimental results are compared with baselines including HPG and its variants on standard sparse reward benchmark tasks. \n\n*Comments* : \n\n\t- The core contribution of the paper is to introduce hindsight based TRPO where end states from the trajectory treated as goals can be useful for generating pseudo-rewards, such that existing TRPO based approaches can be better suited for sparse reward environments. The claim is that existing policy gradient methods cannot sufficiently explore in sparse reward environments. \n\t- Since incorporating hindsight data can make the approach off-policy in nature, leading to higher variance and instability, the authors propose to augment hindsight approach into on-policy based methods such as TRPO. The key contribution is to develop a theoretically sound approach for hindsight based policy optimization. \n\t- The hindsight TRPO objective is derived based on goal-conditioned policies with TRPO, based on existing recent work on Hindsight Policy Gradient (HPG) (Rauber et al., 2019). The key difficulty that needs to be tackled is when introducing hindsight experience makes the approach off-policy in nature. The authors derive the HTRPO objective based on an importance sampling approach that can also incorporate the hindsight data. \n\t- Equation 7 writes out the key quantity, an IS based approach considering the current goal and alternative goals to derive a similar TRPO objective based on IS based Monte-Carlo estimators, while maintaining the trust region guarantees. Equation 8 further shows the gradient of the objective. Theorem 3.1 and 3.2 follows naturally where the key trick in Thm 3.1 is going from equation 19 to 20 to derive the IS based estimator with goal conditioned trajectories. I am not sure why Thm 3.2 needs to be written out explicitly, given that it follows naturally for gradient of the expectation? Is there any key takeaway from it? \n\t- In TRPO, the expectation is based on states sampled from a state distribution. The authors argue that for hindsight based data, this state distribution in fact can change due to changes in the generated goals (ending states), and hence the KL needs to be in expectation w.r.t to the state occupancy measure. Furthermore, the authors change the KL based constraint into a logarithmic form of a constraint such as to reduce variance and introduce more stability. To achieve this, the paper uses an approximation to the logarithmic form of the constraint, by using an expectation of the square instead of plain differences between the log of policies. The key is that instead of using the KL divergence, the authors introduce f-divergence where the function is convex allowing smooth optimization. \n\t- The overall contribution of the paper can be summarized from equation 15 - introducing a IS based correction, while remaining on-policy for hindsight based TRPO objective. And since hindsight can change the underlying state distribution, leading to more instability, the paper introduces a different form of constraint (based on the f-divergence) which can have lower variance than the KL form of constraint. \n\t- Experimental results are demonstrated for few sparse reward benchmarks, comparing to the standard HPG, TRPO and several variants of the proposed HTRPO approach with weighted IS and with the KL constraint. The advantages of HTRPO on these tasks seems clear, mainly in the Fetch Reach and Fetch Push tasks, significantly outperforming the baselines. Even in the continuous tasks, HTRPO seems to outperform the baselines consistently. \n\n*Feedback/Questions* : \n\n\t- I am not sure of the significance of Theorem 3.2 - it seems obvious that the gradient of the objective spans out naturally from equation 7? \n\t- The authors mention about the state occupancy measure instead of considering the state distribution for the KL term. However, the discussion of state occupancy measure seemed to have faded away? What was the significance of mentioning it, or why should it be considered even? There are no assumptions being made on whether the state distribution should be the discounted state occupancy measure or the stationary distribution (if infinite horizon is considered). \n\t- The introduction of the logarithmic form of constraint, even though shows theoretically to reduce variance, is not well motivated or demonstrated from the suite of experimental results? From the results, it is not obvious whether this form of constraint is indeed having useful effects in terms of reduced variance? \n\t- The paper seems to adapt from the HPG objective, and does indeed a great job comparing to HPG throughout the suite of experiments. However, in the results, the paper mainly compares to other off-policy based methods including HPG and its variants (official and re-implemented one). I find the comparison of results a bit odd in that sense, since it is comparing the on-policy adaptation of HPG (ie, HTRPO) and the off-policy variants? If run for long enough, does all converge to the same results? If so, then the benefits is mainly in faster learning (assumably due to better exploration in the sparse reward tasks). But then again, these benefits may be because of the on-policy approach compared to the off-policy one? \n\t- I would encourage the authors to compare to more standard goal-conditioned or reward shaping based baselines for TRPO. For example, does the proposed HTRPO approach perform better compared to other goal-conditioned approaches of TRPO, or for example if a form of reward shaping (based on goals) are used in TRPO? It would then be a more likewise comparison? The current results seem to show benefits of HTRPO, but I think there is a need for stronger baselines where TRPO + exploraton (reward shaping or goal conditioning) performs better?\n\t- I am not convinced about the arguments with sensitivity of HTRPO with network architecture and parameters. How is this demonstrated from the suite of results?\n\u000b\u000b\n*Summary of Review and Score* : \n\nOverall, I think the paper has merits, mainly in terms of deriving the on-policy adaptation with hindsight data. The key objectives are derived nicely in the write-up and easy to follow, although there are some confusions that need to be clarified (example : the discussion on state occupancy measure and the significance of it). The paper motivates exploration for TRPO in sparse reward tasks, and considers the hindsight adaptation of existing TRPO. But related works such as HPG have already taken a similar approach for the off-policy case, and this paper's key contribution is in terms of theoretically deriving the objectives for on-policy adaptation. However, I am not convinced about the overall merits and contributions of the paper, especially in terms of demonstrated results and proper comparisons with baselines. I think while the objectives and derivations follow naturally, the contributions of the paper is somewhat marginal. \n\nI would therefore vote to marginally reject this paper - mainly in light of the core novel contribution and due to lack of sufficient results demonstrating the usefulness of the approach. The paper combines several things together - especially discussions of the logarithmic form of the constraint. I doubt whether all these introduced together led to the improvements shown in the experimental results or not. It would be useful to clarify the contributions from each of the components. \n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Bye56P4fFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an extension to TRPO that makes use of hindsight experience replay.  The technique follows previous work on HER for policy gradient methods, augmenting these previous approaches with a constraint based on the average squared difference of log-probabilities. Experiments show that the method can provide significant improvements over previous work.\n\nOverall, the paper provides a convincing demonstration of trust region principles to goal-conditioned HER learning, although I think the paper could be improved in the following ways:\n-- Bounding KL by the squared difference of log-probabilities seems loose.  The original TRPO objective is based on a TV-divergence (and before that, based on a state-occupancy divergence). Is it possible to directly bound the TV-divergence (either of actions or state occupancies) by a squared difference of log-probabilities? \n-- The use of WIS greatly biases the objective. Is there a way to quantify or motivate this bias?\n-- What is the method in which the alternative goal g' is proposed? I believe this can have a great effect on the bias and variance of the proposed method.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1xQBe0E5r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe paper builds on top of prior work in hindsight policy gradients (Rauber et.al.) and trust region policy optimization (Schulman et.al.), proposing a hindsight trust region policy optimization. Conceptually this direction makes a lot of sense, since hindsight is in general shown to be useful when training goal conditioned policies. The formulation generally  appears to be sound and is a straightforward extension of the importance sampling techniques from Rabuer et.al. to the TRPO setting. Experimental results show that the proposed changes bring significant improvements over baselines on sparse reward settings.\n\nStrengths:\n+ The paper appears well formulated, and well motivated\n+ The experimental results appear quite strong.\n+ Description of the experimental details is quite clear.\n\nWeaknesses:\n- Most of the weaknesses that I can find are in terms of the presentation and writing which could be improved. Specifically, it would be good to clarify when writing the HTRPO equation (Eqn. 7) that this is still a constrained optimization problem with a KL divergence between polcies. Further, it should be better clarified and justified which KL divergence the constraint should be between, since there are two choices \\pi(a| s, g) or \\pi(a| s, g’). \n- It would be make the results section flow much better if the paper were to adopt g as the original goal, and g’ as the alternative goal (which makes a much better flow from the non-hindsight case to the hindsight case).\n- It seems a bit wasteful to mention Theorem 4.1 as a theorem, since it does not feel like a major result and is a straightforward monte carlo estimation of the KL divergence. \n- Missing baseline: it would be nice to check if the method of estimating the KL divergence using difference of squares of log probs (Eqn. 12) improves TRPO (and a clarification on whether this is would be a novel application in the first place). This might be a result of independent interest outside of the hindsight context. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper pursues an interesting approach, but requires additional maturation.  The experienced reviewers raise several concerns about the current version of the paper.  The significance of the contribution was questioned.  The paper missed key opportunities to evaluate and justify critical aspects of the proposed approach, via targeted ablation and baseline studies.  The quality and clarity of the technical exposition was also criticized.  The comments submitted by the reviewers should help the authors strengthen the paper. 
Decision: Reject