Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper analyzes hindsight experience replay (HER) through divergence minimization with energy based models. With this analysis, it presents an approach that combines HER with behavioral cloning (BC) by only imitating actions in the dataset if they move the agent a certain amount towards the goal.

The paper concludes by comparing this adaptive HER + BC technique (termed hindsight divergence minimization, HDM) to HER, behavioral cloning and a more straightforward implementation of HER +BC.
The paper concludes with a hypothesis that BC-based techniques work better if the dataset used for learning has a higher proportion of achieved goals in its trajectories, and validates this hypothesis via experiments.
strength_and_weaknesses: ### Strengths
* This paper conducts a study of why hindsight replay helps when learning in goal-conditioned RL from the perspective of divergence minimization.
* This study leads the paper to propose an adaptive algorithm that can take advantage of high quality demonstrations.
* The perspective of goal-conditioned RL as a form of divergence minimization could be interesting.

### Questions and Weaknesses:
* The paper does not have a consolidated related works section. Are there other papers that aim to address goal-conditioned RL via a divergence minimization perspective? For example, it seems that the paper should compare to [1,2] in their exposition as well as their experiments.
* While the rigor in section 4 is to be appreciated, it takes away from ease of understanding and clarity.
* Section 5.3 was unclear on what _achieved-goal_ means. If that section essentially points out that GCSL works better in situations where the trajectories in the dataset get to the goal efficiently, then it is unclear why this is an important insight. BC with expert trajectories is more likely to work better. A better comparison would have been to also plot HER + HBC to show that these sorts of trajectories do not hard HER learning as much.
* How do the results in Table 1 differ from the results in Figure 7?

### References:
[1] Ma, Y.J., Yan, J., Jayaraman, D. and Bastani, O., 2022. How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via $ f $-Advantage Regression. arXiv preprint arXiv:2206.03023.

[2] Durugkar, I., Tec, M., Niekum, S. and Stone, P., 2021. Adversarial intrinsic motivation for reinforcement learning. Advances in Neural Information Processing Systems, 34, pp.8622-8636.
clarity,_quality,_novelty_and_reproducibility: * Clarity and Quality: Fair. As pointed out above, the mathiness retracts from ease of following the exposition, but the notation and math seems clear enough.
* Novelty: Weak. It is unclear how this approach compares and contrasts with prior work. There is no dedicated related work comparison, and significant recent work has not been addressed.
* Reproducibility: Good: Setup and hyper-parameters have been provided.
summary_of_the_review: The paper offers an interesting perspective on HER, and presents an approach to adaptively improve performance by incorporating BC into the objective if it is helping the agent learn. However, there might be related work that is not addressed in the paper, and might perhaps need to be compared to.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper casts the problem of hindsight goal relabeling into a divergence minimization problem.
The authors propose a simple loss form to optimize and provide an algorithm to efficiently mix behavior cloning and RL in the goal relabelling procedure. 

To do so, the authors first summarize the many mathematical tools they used in the paper (Rl, Divergence Minimization, EBM). They en combine those concepts to finally explore how to maximize this objective.  Their derivations also allow the author to see when BC may help improve the policy (as some trajectories may be sub-optimal).

Finally, the authors perform a short empirical study over classic goal relabelling tasks and empirically show that their approach consistently outperforms other goal relabelling methods.
strength_and_weaknesses: The main strength of this paper is to formalize many algorithms into a single consistent framework and draw links between the methods. It also results in a clear objective and a practical algorithm. Unfortunately, I have difficulty grasping clear intuitions by following mathematical explanations. Too many mathematical references are interleaved in a few steps, and the notation sometimes feels too complex with too many nuances. For instance, Section 3 is a good example; there are many mental jumps to reach the final objective. On a single page, there are 4 figures; the authors refer to 4 different algorithms and introduce 5 equations with hard-to-parse notations $p_\nu$ vs. $\rho_\nu$). I believe the authors wanted to be too comprehensive and highly rigorous, but it sometimes makes things too dense.

The final empirical results are also quite promising. However, I would recommend the authors provide more experimental details to be fully convincing (The appendix is a bit short on the matter.) First, I am not sure whether the number of seeds was provided. Then, training curves would be a nice add-on. Similarly, it is unclear whether the authors reproduced the results from other algorithms or whether they reported the scores (hence having potentially different training hyperparameters). Finally, I would have enjoyed more discussions on the hyperparameters (not only that they outperform classic methods on an extensive range). Idem testing the algorithm with different r and beta, for example, (-1,0) vs. (0, 1). What are the extreme case, how should they be set, etc.? I understand that the paper's primary focus is not on the empirical sections, but the paper could be greatly strengthened. The comparisons between the different baselines could be made more comprehensive. A simple table to highlight core elements (reward, clipping, etc.) would be helpful

Then, I have a few other remarks:
 - In the algorithm, one step-q-learning seems like a significant constraint (l1), especially with a long horizon. Why not have a lambda/n-step return
- How n-step return would behave with $\gamma_{hdm}$ 
- Would it make sense to use DQfD[1] instead of pure BC? 
- There is no Bellman residual error here, it is only a TD error

[1] Hester, Todd, et al. "Deep q-learning from demonstrations." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.
clarity,_quality,_novelty_and_reproducibility: As discussed above, clarity may be among the most crucial problem. If I have a guess, the authors tried to be highly comprehensive and rigorous by referring to the equations and the related literature while providing intuitions simultaneously.
Unfortunately, it was too much for me (Maybe, I am not strong enough in the field, and a more advanced reader may have a different perspective). Anyway, I think that better decoupling the concepts (link to other algorithms, proof, intuitions) would greatly ease the reading.
Another unfortunate consequence is that it also becomes less clear what are the actual contributions and core differences between the methods and the literature. The writing sometimes gives the (misleading) feeling that the authors did a patchwork of many algorithms in their method. For instance, it would be easier to show that other methods may be derivate from their training objectives. 
In a few words, I would encourage the author to make the paper more digest by: better disentangling the concepts and moving some parts in the appendix to make the paper less dense.

On a different note, I found the introduction a bit abrupt, and there is a few over-statement from my perspective. Reward is not enough because they are manually created is quite a shortcut. Why is divergence minimization the de facto way to describe imitation? 
summary_of_the_review: I think this paper can be an exciting piece of work that completes the goal of relabelling the framework.
However, the lack of clarity and some issues in the experimental results prevent me from being confident in my assessment. There are still many points that may remain obscure to me. Besides, the cluttered notation was really painful and prevented me from digging much into the mathematical details. Therefore, I am leaning to a borderline accept with little confidence
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this work, the authors present a unification of goal-conditioned behavior cloning/supervised learning and hindsight relabeling methods (e.g., HER) under the framework of divergence minimization. They show that HER is a special case of a divergence minimization when considered under a goal-conditioned Q-learning approach (when Q is treated as an EBM).

To this end, they propose a novel objective that aims to account for and balance the key elements required in goal-conditioned RL, and show promising results in a suite of standard tasks.

They further provide intuition about how excluding some of these elements (e.g., lack of goal-likelihood in the optimization) results in sub-optimal performance, and use this framework as a means to explain the poorer performance of pre-existing approaches.
strength_and_weaknesses: Strengths:
* The unifying framework appears useful, and I found the unification of HER under a goal-conditioned RL framework interesting, and appears to provide insight into how these methods may work on a more theoretical level.
* The authors provide empirical evidence that their approach HDM, which balances goal reaching and state-action matching, outperforms prior methods which don't explicitly manage this effectively.

Weaknesses:
* While the paper is easy to follow in sections, I found the overall narrative a bit unclear. For instance, Section 3 was quite hard to follow, as quantities and PGMs/distributions are introduced with not much intuition, for instance Eqs 10,11 and 12. Explaining the quantities in Eq 10 and 11, and why 12 gives HBC/GCSL would help here. Similarly, when the authors present their approach in Eq 14, the divergence term is flipped, but no explanation as to why this is done appears to be provided. Another case is when deriving the HER objective under their f-divergence formulation; they seem to arbitrarily combine the 1-step part of the EBM formulation with part of the f-divergence term, but don't give explanation why replacing the multi-step part of the EBM objective like this is fine? I also was a bit confused by the inclusion of 4.3 (HER derivation) and 4.4 (BC analysis) in section 4; it might make more sense to separate them (e.g., Section 4 being more focused on just the creation of the optimizable objective, and another section showing how their framework can unify these approaches).
* I wasn't entirely convinced by some of the ablations. For instance, in the appendix the authors ablate values of Beta and Gamma; wouldn't we expect some extreme settings to recover the performance of HBC/GCSL? If not, why?
* Nits:
   * The term Bellman Residual is somewhat confusing, as it is overloaded with Bellman Residual learning [1]; perhaps using Bellman Error would be sufficient here, as I believe the authors explicitly state they put a stop gradient on the target values.
   * How many seeds are used in the experiments?

[1] Residual algorithms: Reinforcement learning with function approximation, Baird, ICML 1995
clarity,_quality,_novelty_and_reproducibility: I found the work clear in sections, but as listed above, there are some areas in the communication that were not entirely obvious to me. Having said this, although I didn't spend significant time re-deriving the formulae, as far as I can see the equations seem to follow. Regarding originality, I am not an expert on GCRL, but the framework presented seems novel enough to merit interest. For reproducibility, pseudo-code and hyperparameters are given, and code is provided.
summary_of_the_review: Overall, I believe this paper merits an acceptance. I think the work sheds interesting light on GCRL, particularly where approaches like HER fit in, and the empirical results seem to largely back up the intuitions presented. As a result, I believe the paper is of interest to the community, and is novel enough in its analysis to merit acceptance at a major conference.

I have some issues with the presentation and experiments that I have listed in the Weaknesses section, and would be grateful to hear from the authors about addressing these.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a new method for goal-conditioned RL that combines three terms: (1) goal-conditioned behavioral cloning, (2) maximizing the likelihood of the goal, and (3) an exploration term. The paper proposes to use a different f-divergence for optimizing the first term, which ends up being done using a DICE-like TD update. Empirically, the proposed method outperforms baselines on a goal-reaching benchmark from prior work.
strength_and_weaknesses: Strengths
* The empirical results seem quite strong.

Weaknesses
* I am unsure about the motivation for the paper. The introduction seems to argue that there aren't any good ways of defining the goal-conditioned RL problem without reward engineering, but I'm not sure this is true (see below). The proposed objective is a combination of a "goal likelihood term" (which is the same as or quite similar to uncited prior work [5, 6, 8]) and an f-divergence term (from prior work, and cited). Most of the paper's contributions seem to be about analyzing the f-divergence term, but it is unclear why that term is needed at all: directly maximizing the goal likelihood term would yield the optimal goal reaching policy. Note that prior work has even considered maximizing the combination of the goal likelihood term and the f-divergence term, albeit with a different choice of f-divergence [9].
* I am unsure about some of the claims in the introduction (see below)
* I am unsure about certain parts of the math. It's possible they are correct and I was just confused by the notation (see details below).
* Many parts of the paper were difficult to follow (see below).
* The proposed method seems quite complex.

Detailed comments
* "this connection between imitation and hindsight relabeling is not well understood" -- I'm not sure this is true. There's been a fair bit of analysis of this in prior work (e.g., [1, 2, 3]).
* "reward function in hindsight experience replay" -- Where is this shown? The HER paper uses a sparse reward on human-specified coordinates.
* "without reward engineering" -- All four of these papers *do* require reward engineering, in the form of specifying a distance metric and (for some) specifying coordinates of interest. Consider referring to prior work that actually does learning without reward engineering [5, 6, 7].
* "it is unclear how to write down a well-defined objective for goal-conditioned policies" -- I don't think this is true. See, e.g., [5, 6].
* "known as contrastive divergence" -- I would recommend *not* referring to this as contrastive divergence, which typically refers to a specific approximate algorithm for fitting EBMs (instead of generating true samples from the model, it approximates the samples using a single gradient step).
* "NCE gives EBM a more tractable way to maximize data likelihood" -- I didn't understand this. Would it be possible to elaborate?
* Eq 10, Eq 11 -- I found these equations and the surrounding discussion confusing. Are these distributions supposed to be the same?
* Eq 12 -- There seems to be a "slight-of-hand" here, where the future state is reinterpreted as the commanded goal. I'd recommend clarifying this by explicitly writing down the distributions for the relevant random variables ($g$, $s^+$).
* Eq. 15 -- I'd recommend making sure the arguments to the f-divergence have the same arguments; currently, one has $g$ and one has $s^+$.
* Eq 15 -- How does this relate to Eq 12? It seems like the direction of the KL has been silently reversed
* Eq 15 -- Is this f-divergence being optimized? If so, w.r.t. what components?
* Eq 16 -- How is $\rho_\pi(s, a, \mid g)$ defined? Is this equal to $\rho_{\pi(a \mid s)}(s, a \mid s^+=g)$?
* Lemma 4.1 -- What reward function is used for defining $Q^\pi(s, a, g)$?
* "now define a Q-function as ... separate from the one defined in Eq. 18" -- I found this confusing. I'd recommend using the standard convention of defining Q-function as expected discounted returns, and using a different variable for this.
* Lemma 4.2 -- I found this hard to parse. I'd recommend adding scaffolding: what are the main aims of the result, what will this result be used to show?
* Eq. 25: How is this similar to and different from [6]?
* Eq 25 -- Can this be re-expressed without the Q-functions, directly in terms of the environment dynamics?
* Eq. 28 -- I'm not sure $s' = s^+$ is well defined for continuous settings.
* "EBM training equation 53" -- Broken reference?
* Fig 8 -- I was unable to read the text in this figure.
* Table 1 -- How are the numbers for GCSL computed? [10] reports much higher numbers for a GCSL-style method on the door task (This paper reports GCSL = 19% and the proposed method gets 87%, but [10] reports that a GCSL-style method gets 95.8%).
* code, scripts folder -- Why are different random seeds used for different environments.
* logger.py, L480 -- I'd recommend not including expletives in open source code.

[1] https://proceedings.neurips.cc/paper/2019/file/3891b14b5d8cce2fdd8dcdb4ded28f6d-Paper.pdf

[2] http://proceedings.mlr.press/v130/tang21b/tang21b.pdf

[3] https://proceedings.neurips.cc/paper/2020/file/a97da629b098b75c294dffdc3e463904-Paper.pdf

[4] https://arxiv.org/pdf/1912.06088.pdf

[5] https://arxiv.org/pdf/2101.07123.pdf

[6] https://arxiv.org/pdf/2011.08909.pdf

[7] https://arxiv.org/pdf/1905.07866.pdf

[8] https://arxiv.org/abs/2104.10190

[9] https://arxiv.org/pdf/2206.07568.pdf

[10] https://arxiv.org/pdf/2112.10751.pdf
clarity,_quality,_novelty_and_reproducibility: **Clarity** -- I have a number of concerns about the clarity of the paper (see above).

**Quality** -- I have a number of concerns about the correctness of the paper (see above).

**Novelty** -- The precise form of the method is novel, to the best of my knowledge. It is very similar to a number of prior methods, and I don't think these similarities are adequately discussed.

**Reproducibility** -- The paper includes code, along with launch scripts to reproduce one of the experiments.
summary_of_the_review: My large number of concerns about the paper compel me to vote to reject it. I do think there might be some really interesting ideas in the TD-like method for minimizing the f-divergence. For example, might that allow a GCSL/HBC-style method to perform off-policy learning?
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper explores the connection between hindsight goal relabeling and imitation learning. The original submission strongly divided reviewers leading to extensive discussion. However, after further discussion of the authors' responses, all reviewers concluded that the paper was not yet ready for publication. The consensus being that there is merit to the ideas, but the current paper does not fairly present the ideas in relation to prior work, clarify important experimental choices (including outstanding questions on the differences in behaviors learnt and reward mechanism chosen) or sufficiently justify theoretical choices made (e.g. flipping the divergence for Lemma 4.1.)
justification_for_why_not_higher_score: Too many issues compounding across literature review, experimental setup and theoretical justifications.
Decision: Reject