Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Summary

The paper builds on top of prior work in hindsight policy gradients (Rauber et.al.) and trust region policy optimization (Schulman et.al.), proposing a hindsight trust region policy optimization. Conceptually this direction makes a lot of sense, since hindsight is in general shown to be useful when training goal conditioned policies. The formulation generally  appears to be sound and is a straightforward extension of the importance sampling techniques from Rabuer et.al. to the TRPO setting. Experimental results show that the proposed changes bring significant improvements over baselines on sparse reward settings.

Strengths:
+ The paper appears well formulated, and well motivated
+ The experimental results appear quite strong.
+ Description of the experimental details is quite clear.

Weaknesses:
- Most of the weaknesses that I can find are in terms of the presentation and writing which could be improved. Specifically, it would be good to clarify when writing the HTRPO equation (Eqn. 7) that this is still a constrained optimization problem with a KL divergence between polcies. Further, it should be better clarified and justified which KL divergence the constraint should be between, since there are two choices \pi(a| s, g) or \pi(a| s, g’). 
- It would be make the results section flow much better if the paper were to adopt g as the original goal, and g’ as the alternative goal (which makes a much better flow from the non-hindsight case to the hindsight case).
- It seems a bit wasteful to mention Theorem 4.1 as a theorem, since it does not feel like a major result and is a straightforward monte carlo estimation of the KL divergence. 
- Missing baseline: it would be nice to check if the method of estimating the KL divergence using difference of squares of log probs (Eqn. 12) improves TRPO (and a clarification on whether this is would be a novel application in the first place). This might be a result of independent interest outside of the hindsight context. 

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: *Summary* : This paper augments the TRPO policy optimization objective with hindsight data, where the hindsight data is generated from goals based on trajectories. The key contribution of the paper is based on deriving an on-policy adaptation of hindsight based TRPO, that can be useful for sparse reward environments. The paper draws ideas from existing papers such as HPG and considers the IS based variant of HPG for on-policy, similar to TRPO, that can achieve monotonic performance improvements. Furthermore, the authors introduce a logarithmic form of constraint, by re-deriving the KL constraint and leading to a f-divergence based constraint, which is argued to have useful effects in terms of lowering the variance. Experimental results are compared with baselines including HPG and its variants on standard sparse reward benchmark tasks. 

*Comments* : 

	- The core contribution of the paper is to introduce hindsight based TRPO where end states from the trajectory treated as goals can be useful for generating pseudo-rewards, such that existing TRPO based approaches can be better suited for sparse reward environments. The claim is that existing policy gradient methods cannot sufficiently explore in sparse reward environments. 
	- Since incorporating hindsight data can make the approach off-policy in nature, leading to higher variance and instability, the authors propose to augment hindsight approach into on-policy based methods such as TRPO. The key contribution is to develop a theoretically sound approach for hindsight based policy optimization. 
	- The hindsight TRPO objective is derived based on goal-conditioned policies with TRPO, based on existing recent work on Hindsight Policy Gradient (HPG) (Rauber et al., 2019). The key difficulty that needs to be tackled is when introducing hindsight experience makes the approach off-policy in nature. The authors derive the HTRPO objective based on an importance sampling approach that can also incorporate the hindsight data. 
	- Equation 7 writes out the key quantity, an IS based approach considering the current goal and alternative goals to derive a similar TRPO objective based on IS based Monte-Carlo estimators, while maintaining the trust region guarantees. Equation 8 further shows the gradient of the objective. Theorem 3.1 and 3.2 follows naturally where the key trick in Thm 3.1 is going from equation 19 to 20 to derive the IS based estimator with goal conditioned trajectories. I am not sure why Thm 3.2 needs to be written out explicitly, given that it follows naturally for gradient of the expectation? Is there any key takeaway from it? 
	- In TRPO, the expectation is based on states sampled from a state distribution. The authors argue that for hindsight based data, this state distribution in fact can change due to changes in the generated goals (ending states), and hence the KL needs to be in expectation w.r.t to the state occupancy measure. Furthermore, the authors change the KL based constraint into a logarithmic form of a constraint such as to reduce variance and introduce more stability. To achieve this, the paper uses an approximation to the logarithmic form of the constraint, by using an expectation of the square instead of plain differences between the log of policies. The key is that instead of using the KL divergence, the authors introduce f-divergence where the function is convex allowing smooth optimization. 
	- The overall contribution of the paper can be summarized from equation 15 - introducing a IS based correction, while remaining on-policy for hindsight based TRPO objective. And since hindsight can change the underlying state distribution, leading to more instability, the paper introduces a different form of constraint (based on the f-divergence) which can have lower variance than the KL form of constraint. 
	- Experimental results are demonstrated for few sparse reward benchmarks, comparing to the standard HPG, TRPO and several variants of the proposed HTRPO approach with weighted IS and with the KL constraint. The advantages of HTRPO on these tasks seems clear, mainly in the Fetch Reach and Fetch Push tasks, significantly outperforming the baselines. Even in the continuous tasks, HTRPO seems to outperform the baselines consistently. 

*Feedback/Questions* : 

	- I am not sure of the significance of Theorem 3.2 - it seems obvious that the gradient of the objective spans out naturally from equation 7? 
	- The authors mention about the state occupancy measure instead of considering the state distribution for the KL term. However, the discussion of state occupancy measure seemed to have faded away? What was the significance of mentioning it, or why should it be considered even? There are no assumptions being made on whether the state distribution should be the discounted state occupancy measure or the stationary distribution (if infinite horizon is considered). 
	- The introduction of the logarithmic form of constraint, even though shows theoretically to reduce variance, is not well motivated or demonstrated from the suite of experimental results? From the results, it is not obvious whether this form of constraint is indeed having useful effects in terms of reduced variance? 
	- The paper seems to adapt from the HPG objective, and does indeed a great job comparing to HPG throughout the suite of experiments. However, in the results, the paper mainly compares to other off-policy based methods including HPG and its variants (official and re-implemented one). I find the comparison of results a bit odd in that sense, since it is comparing the on-policy adaptation of HPG (ie, HTRPO) and the off-policy variants? If run for long enough, does all converge to the same results? If so, then the benefits is mainly in faster learning (assumably due to better exploration in the sparse reward tasks). But then again, these benefits may be because of the on-policy approach compared to the off-policy one? 
	- I would encourage the authors to compare to more standard goal-conditioned or reward shaping based baselines for TRPO. For example, does the proposed HTRPO approach perform better compared to other goal-conditioned approaches of TRPO, or for example if a form of reward shaping (based on goals) are used in TRPO? It would then be a more likewise comparison? The current results seem to show benefits of HTRPO, but I think there is a need for stronger baselines where TRPO + exploraton (reward shaping or goal conditioning) performs better?
	- I am not convinced about the arguments with sensitivity of HTRPO with network architecture and parameters. How is this demonstrated from the suite of results?

*Summary of Review and Score* : 

Overall, I think the paper has merits, mainly in terms of deriving the on-policy adaptation with hindsight data. The key objectives are derived nicely in the write-up and easy to follow, although there are some confusions that need to be clarified (example : the discussion on state occupancy measure and the significance of it). The paper motivates exploration for TRPO in sparse reward tasks, and considers the hindsight adaptation of existing TRPO. But related works such as HPG have already taken a similar approach for the off-policy case, and this paper's key contribution is in terms of theoretically deriving the objectives for on-policy adaptation. However, I am not convinced about the overall merits and contributions of the paper, especially in terms of demonstrated results and proper comparisons with baselines. I think while the objectives and derivations follow naturally, the contributions of the paper is somewhat marginal. 

I would therefore vote to marginally reject this paper - mainly in light of the core novel contribution and due to lack of sufficient results demonstrating the usefulness of the approach. The paper combines several things together - especially discussions of the logarithmic form of the constraint. I doubt whether all these introduced together led to the improvements shown in the experimental results or not. It would be useful to clarify the contributions from each of the components. 


review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper proposes an extension to TRPO that makes use of hindsight experience replay.  The technique follows previous work on HER for policy gradient methods, augmenting these previous approaches with a constraint based on the average squared difference of log-probabilities. Experiments show that the method can provide significant improvements over previous work.

Overall, the paper provides a convincing demonstration of trust region principles to goal-conditioned HER learning, although I think the paper could be improved in the following ways:
-- Bounding KL by the squared difference of log-probabilities seems loose.  The original TRPO objective is based on a TV-divergence (and before that, based on a state-occupancy divergence). Is it possible to directly bound the TV-divergence (either of actions or state occupancies) by a squared difference of log-probabilities? 
-- The use of WIS greatly biases the objective. Is there a way to quantify or motivate this bias?
-- What is the method in which the alternative goal g' is proposed? I believe this can have a great effect on the bias and variance of the proposed method.
review_title: Official Blind Review #1

Meta-review: The paper pursues an interesting approach, but requires additional maturation.  The experienced reviewers raise several concerns about the current version of the paper.  The significance of the contribution was questioned.  The paper missed key opportunities to evaluate and justify critical aspects of the proposed approach, via targeted ablation and baseline studies.  The quality and clarity of the technical exposition was also criticized.  The comments submitted by the reviewers should help the authors strengthen the paper. 
Decision: Reject