Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work addresses the “survivorship” bias present in Hindsight Experience Replay when in stochastic environments. In particular, the proposed method USHER estimates an importance weight to de-bias HER. USHER is then evaluated on various simulated robot environments, and the policy learned in one simulated environment is deployed on a physical robot.
strengths_and_weaknesses: Strengths: This work tackles an issue of HER in stochastic environments, and fixes it with importance sampling. To minimize the variance often associated with importance sampling, they simply clip the weights. The proposed method USHER is evaluated on a diverse set of experimental tasks, including discrete/continuous action spaces, deterministic/stochastic environments, and deployment on a physical robot.

Weaknesses: My main concerns are on the experimental validation. Some of the comparisons are missing in some of the performance plots. For example, delta-DDPG and DDPG are missing from the Fetch Push/Slide and Throwing tasks. Also, in these plots, the x-axis is “Epochs,” but it’s not clear how this translates to the number of environment steps. From the experiments performed by Blier and Ollivier, it appears that delta-DQN and delta-PPO are able to achieve a near perfect success rate on FetchReach after about 250 and 1500 trajectories respectively. How do these translate into epochs?

There also appears to be multiple relevant algorithms that attempt to address bias in HER mentioned in the related work, which should be compared to, such as ARCHER, Bias-reduced Multi-step HER, and Bias-corrected HER. Even if these are tackling other forms of bias or incorrectly address bias, I believe they are relevant to compare to.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This work addresses the bias of HER in stochastic environments through importance sampling. This method in theory should address the bias, but from the experimental evaluation, it’s difficult to gauge the advantages of USHER over existing methods. In particular, I believe the work can be improved in the following areas: clarifying the x-axis in the plots, inclusion of all comparisons in all of the experimental tasks, comparisons to additional methods, and sensitivity analysis for hyperparameters.
issues: There are a few hyperparameters for USHER like $\alpha_Q$, $\alpha_f$, and the clipping parameter. How are these chosen? How sensitive is USHER to these parameters? Does a clipping parameter of 0.3 work for all environments presented? These details appear to be missing from the main text and appendix, except for the Mechanum robot experiment which appears to use a clipping parameter of 100.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents an algorithm to correct for the bias that’s introduced by the Hindsight Experience Replay (HER) algorithm for multi-goal / goal-conditioned RL. The proposed strategy is to separate out the goal used for the reward function from the goal passed to the policy. This allows the derivation of an unbiased variant of HER, using importance sampling. Practical application requires learning the future goal distribution, which is predicted as a separate head off of the existing Q network. Experiments show that this method performs significantly better than HER, as well as other previous unbiased variants of HER, specifically in stochastic environments crafted with HER’s weaknesses in mind. The experiments also show that the importance sampling doesn’t affect HER’s performance in deterministic environments, where bias isn’t a problem.

strengths_and_weaknesses: Strengths:
- Paper is clear and the language is easy to read, although it looks like a few passages of text and equations may have compressed a bit due to space limitations. 
- The theory is original and novel, and experimental results back up the claims well.

Weaknesses:
- The “transfer” method used for real-robot application is to roll out trajectories in simulation, and then use the same sequence of actions on the real robot applied open-loop. Since the scenario being tested is fairly straightforward (i.e. it’s about picking one risky lane vs another safe one), it’s probably fine, but it doesn’t reflect how a policy might be used in practice.
- How often does one encounter truly stochastic environments in robot applications? (I don’t claim to know the answer to this). Do you think this method will be equally helpful in cases of partial observability?

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper proposes a novel method for unbiased HER in stochastic environments using importance sampling, and experimental evaluations attest to its effectiveness. It’s unclear how often this method will show a benefit in situations encountered in real-world policy learning, but the theory and algorithm are nonetheless a useful contribution to robot learning. I would accept this paper.

issues: It would be nice if you used a consistent color for HER across all graphs (I think it’s different just for Fig 3a and 3b).
In some of the graphs where two of the curves are coincident, it just looks like one of the curves is completely missing. Some correction for this would be useful.
review_title: null

final_score: rating: Strong Reject: I recommend rejecting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper proposes a novel method to alleviate bias in hindsight experience replay (HER) using importance sampling. The authors present experiments on a range of environments which suggest that this method boosts performance beyond the original HER algorithm.
strengths_and_weaknesses: # Strengths
 * The method seems interesting, and has potential to be impactful in the research community upon further experimental evaluation.
 * I don't have the theoretical expertise necessary to verify the authors' proofs, but the method seems to be grounded well in theoretical analysis.
 * The authors present results on robot hardware.
 * The abstract and introduction do a good job motivating the problem and introducing the authors' solution.

# Weaknesses
 * A key weakness of the paper is presentation
     * Algorithm 1 is far too crowded to be useful. The authors should replace some of the math with references to the rest of the text.
     * For some reason when the authors show the environments in Figure 2, they only show a subset of them. For example, what is 4-Torus with Freeze? All environments included in the paper should be either explained in the main text or supplement.
     * The results summary section is very disorganized. The authors present all of their results in a gigantic Figure 3 without any helpful captions or references to them in the text. The authors should reorganize this section to be more clear.
 * The authors' core claim is that their proposed method does a better job alleviating HER bias than other work (ARCHER, BHER) that has attempted to do the same in the past, but they never compare to any of this work in experiments. In order to contextualize this work with respect to other literature, these comparisons are critical.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, I think the method has promise and encourage the authors to address reviewer concerns during the rebuttal period. I feel that my main concerns about presentation issues and lacking comparisons make the paper unpublishable in its current state, but will happily increase my score if the authors can fully address these issues during the rebuttal period.

The theoretical results seem interesting but I don't have the expertise to evaluate them so I have not considered them while writing my review.
issues: ### Larger issues
 * The authors should spend some time reorganizing Section 6 to be easier to follow and give more details about the environments they use for evaluation.
 * The authors should modify Algorithm 1 to be more concise and useful.
 * The authors should add the comparisons to other algorithms they mention in the related works section.

### Small Issues
 * Missing space on line 184
 * Should line 195 say $C < 1.0$?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposed an algorithm that leverages importance sampling to unbias hindsight experience replay in stochastic environments. This is achieved by involving separate goals for rewards function and for the policy correspondingly. By conditioning both goals to the value function, a successor representation can be learned for importance sampling, where Bellman error using the representation provides unbiased error estimation. Concrete mathematical proof is provided to support the claim the HER is biased and how the algorithm could reach unbiased HER. Sufficient experiments were conducted in simulation to support the algorithm.
strengths_and_weaknesses: Strengths:
- The developed algorithm is novel, challenging and interesting. It involves importance sampling to unbias - HER with concrete mathematical proof to verify the efficacy of the algorithm. 
- The paper is well written and easy to follow.
- The design choices are reasonable and has been reflected in the ablation study.
- Comprehensive simulation experiments were conducted to support the paper.

Weaknesses:
Although the simulation experiments are convincing. The Real robot experiment is a bit weak. Only 10 episodes were executed for each of the methods. Given the complexity, noise, uncertainty and domain gap posed by the real environment, 6/10 USHER vs 4/10 HER is far from enough to claim USHER outperforms HER. 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: It is a solid paper that addresses bias in HER. The algorithm is backed by math and the simulation experiments are convinving. The only concern is the real world experiments. The sample size is inadequate to make any conclusions.
issues: See above
review_title: null

Meta-review: metareview: The paper proposes an important sampling-based technique to correct the over-estimation bias of hindsight experience replay (HER) in stochastic environments. The paper addresses an important problem, and the reviewers have appreciated the contribution. The method mitigates over-estimation in stochastic environments while preserving performance in deterministic environments. 

Some of the concerns are: 

- Fit to CoRL: The paper is not tackling a robotics problem, but a RL problem that has been applied to a robot to show that the RL algorithm works. The work does not meaningfully advance a robotics problem.  

- Missing basline comparison pointed out by the reviewers:  ARCHER, Bias-reduced Multi-step HER, and Bias-corrected HER. delta-DDPG and DDPG are missing from the Fetch Push/Slide and Throwing tasks. 

- Paper presentation needs substantial improvement: Algorithm 1 is hard to parse, Figure 3 is confusing -- I would suggest making row/columns consistent and other issues pointed out by Z2DH. 

I hope authors address these along with other concerns of the reviewers in the rebuttal. 

**Post Rebuttal Update**
The authors have satisfactorily addressed the concerns raised by the reviewers. I believe this paper is a useful addition to goal-conditioned learning that has been demonstrated with real-world robotic experiments. 
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)