Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The proposed method proposes a method for goal-conditioned RL that can be interpreted as a weighted version of prior work.These weights resemble a combination of discounting and advantage weighting. The paper provides theory arguing that these weights cause the proposed method to optimize a tighter lower bound than prior work. Experiments show that it outperforms the prior work, which does not include weights on each training example.
main_review: **Significance**: On the one hand, the empirical results of the paper are quite strong. On the other hand, by introducing a value function, the paper significantly increases the complexity of the method, and simplicity is arguably the most important feature of goal-conditioned behavior cloning methods.

**Originality**: The proposed method is novel, to the best of my knowledge. While both $\gamma$-discounting and advantage weighting have been studied in a number of prior works (as noted in this paper), I am unaware of prior work that applies them to goal-conditioned behavior cloning methods.

**Clarity**: The paper is clear and readable.
* I was confused about the difference between GoalBC, Goal BCQ, and Goal CQL. For example, is Goal-CQL designed so that Q function reflects the *marginal* data collection policy or the *goal-conditioned* data collection policy? (i.e., how is hindsight relabeling combined with CQL)?

**Correctness**: I have a few concerns about the correctness of the paper.
* In the second to last line of the proof, if we replaced the weights $\gamma^{i-t}$ with 0, we'd get a useless objective, but would the bound be made *even* tighter?
* I'm unsure if the main result is actually useful. In particular, Theorem 1 shows that the proposed method optimizes a tighter bound on $J_{surr}$, but we actually want a tighter bound on $J$ (defined in Sec. 2).
* Proposition 1, "good as or better than" -- According to what criterion?
* Proposition 2: I think this result is a bit misleading, because it is really a result about the dataset, not the policy; it is not a "policy improvement" result, as indicated by the subsection header.

**Minor comments**

* Abstract: It's not clear what problem is being solved. I'd recommend making this more clear in the first 1-3 sentences.
* "can we learn goal-conditioned policies from offline data?" -- [1] already does this.
* "hindsight relabeling would generate non-optimal experiences" -- I don't this this is correct: [2] prove that hindsight relabeling is equivalent to inferring a task assuming that the experience were optimal.
* "extremely sparse" -> "sparse"
* "last step reward" -- Clarify what this means.
* Eq. 2, $w_{i,t}$: This should be defined before the equation.
* I'd recommend including advantage weighting methods (e.g., [3]) as an additional baseline. This should be trivial to implement because the proposed method already includes a form of advantage weighting.
* Ethics Statement, "there are concerns that AGI might go out of human control in the future" -- The ethics statement lacks nuance. I'd recommend that it be revised to more carefully address the specific ethical implications of this paper.


[1] Lynch, Corey, et al. "Learning latent plans from play." Conference on Robot Learning. PMLR, 2020.

[2] Eysenbach, Benjamin, et al. "Rewriting history with inverse rl: Hindsight inference for policy improvement." arXiv preprint arXiv:2002.11089 (2020).

[3] Peng, Xue Bin, et al. "Advantage-weighted regression: Simple and scalable off-policy reinforcement learning." arXiv preprint arXiv:1910.00177 (2019).
summary_of_the_review: My main concerns are about the clarity and correctness of the paper (the empirical results are already great!). If the paper were revised to address these concerns, I'd be inclined to change my vote.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an extension of GCSL where the goal-conditioned BC loss is weighted by a variable that correlates with the number of steps necessary to achieve the desired goal.  The advantage of this approach is that sub-optimal trajectories to a particular goal will get downweighted to the benefit of more direct trajectories.  This effectively adds a policy improvement step to GCSL with the underlying objective function being that of arriving at goal states as fast as possible.  

Results on goal-conditioned tasks show that the proposed approach (WGCSL) performs consistently better than GCSL and in some cases significantly better (HandReach-expert).
main_review: Pros:
The paper is well written and the notation very easy to follow.  There are a couple of typos/weird phrase constructions in the paper that could warrant a bit of proofreading.  I put some examples at the end of this section.
The idea, although somewhat an obvious followup to GCSL, is well explained and analyzed, and experimental results are convincing.  

Cons:
One of the advantages of GCSL is that it is a purely supervised method, which greatly enhances the simplicity and stability.  The proposed approach uses a Q-function to derive the advantage function A, which requires a bootstrap-based method to be applied to an offline dataset.  I would greatly appreciate a bit of commenting on this in the paper, as this is well-known to not be trivial to get working.  Is the method relying on the goal-conditioned aspect to compensate for over-estimation issues?  

Another topic I would have like to see more discussion on is the difference between HER and WGCSL.  If I understand correctly, HER is able to train in the offline setting due to goal-conditioned aspects of the problem, but why would WGCSL fundamentally work better than a purely value-based method?  

Overall I vote to accept this paper, but would appreciate responses from the authors on my questions.

Section: 'Expert Datasets': 'Tabel 1' -> 'Table 1'.  'in offline' -> 'in the offline'
'Random Datasets' section: 'relatively well policy'  -> 'relatively good policy'.  'results of the random' -> 'results on the random'.
summary_of_the_review: A somewhat obvious extension to GCSL (online goal-conditioned BC), but that is well-explained and shows convincing experimental results.  I vote for acceptance, and believe this provides a potentially new baseline for offline goal-based methods.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes weighted goal-conditioned supervised learning (WGCSL) as a tighter lower bound to optimize than the typical GCSL objective. Then the paper combines a bunch of prior approaches / heuristics to set these weights and shows improvements on a new offline RL benchmark containing various simulated robotic manipulation tasks. 
main_review: Strengths
- The paper proposes a weighted GCSL approach that appears to be motivated from optimizing a better lower bound (Theorem 1) and results in large empirical gains (although statistical uncertainty in results isn’t reported).
- The empirical experiments are thorough with ablations and show the benefit of the proposed WCGSL.
- The paper is generally positioned well wrt prior work but more discussion of some closely related work is needed for clarity.

Weaknesses:

- I found the paper to be somewhat unclear to read. Some examples:
  - The main contributions are presented very succinctly in Section 3.2. This makes the (1) goal-conditioned advantage re-weighting and (2) best-advantage weight hard to comprehend and understand the motivation behind such schemes.
  - Notations are overloaded heavily, for example, the advantage and value functions are used without any superscript for the policy which was confusing. For example, I didn’t understand what value function is being talked about in Proposition 2.
  - Results from Wang et. al (2018) and Haarnoja et al (2017) are directly used without clearly explaining them to the reader. 
Multi-modality challenge is used in intro and section 3.2 without introducing what is the challenge (it only becomes clear in the last part of sec 3.2) 
  - The motivating example was confusing – if we only have a sparse 0/1 reward function for goal reaching (as defined in the preliminaries), how are we claiming one trajectory to be better than another? Under this reward function, both the trajectories achieve the same return.

- Theoretical presentation of some results don’t add much value. Examples:
  - Theorem 2  trivially follows from Theorem 1 following a 2 line proof and calling it a theorem is unwarranted. 
  - Proposition 2 is presented in the paper but empirical experiments simply avoid its relabeling strategy and implications. This made me wonder about the relevance of that proposition for this paper.


- The empirical reporting and rigor of the results should be improved.
   - No error bars are reported in Table 1 but as shown by Agarwal et al. (2021), it seems critical to report statistical uncertainty in deep RL results. Furthermore, the large number of entries in Table 1 makes it slightly overwhelming to read, maybe presenting it visually using performance profiles might be more informative.  
  - The learning curves are only shown selectively for tasks where WGCSL outperforms other methods. Instead, curves showing aggregate scores across all tasks such as mean scores or interquartile mean (less prone to outliers)  for random and expert datasets with confidence intervals (CIs) might be a better way to summarize performance across tasks.  
  - Adding average probability of improvement of WGCSL over other methods as done by Cong et al (2021) would further make the results more statistically reliable. The rliable library would be quite useful for reporting results with CIs and aggregate metrics: https://github.com/google-research/rliable
  - No std are shown for some of the methods in Figure 5.
  - Why are the default hyper parameters tuned for behavior regularization methods such as CQL and BCQ – it seems unlikely that the default values from D4RL/Mujoco tasks would work for the tasks presented here. Thus, are the comparisons fair?

Other minor comments:
 - Typo: GSCL → GCSL (under expert datasets in experimental section)


References:

[1] Haarnoja, T., Tang, H., Abbeel, P. and Levine, S., 2017, July. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning (pp. 1352-1361). PMLR.

[2] Wang, Q., Xiong, J., Han, L., Sun, P., Liu, H. and Zhang, T., 2018, December. Exponentially Weighted Imitation Learning for Batched Historical Data. In NeurIPS.

[3] Agarwal, R., Schwarzer, M., Castro, P.S., Courville, A. and Bellemare, M.G., 2021. Deep reinforcement learning at the edge of the statistical precipice. In NeurIPS.

summary_of_the_review: While the empirical results of the proposed approach looks promising, as pointed in the weaknesses, I have concerns about clarity of the paper, presentation of the theory and empirical rigor. If the authors can address these issues, I'd be willing to re-assess the paper.

--- After rebuttal --- Updated my score to 6.
review_title: null

Meta-review: The authors introduce a method that improves goal-conditioned supervised learning (GCSL) by iteratively re-weighting the experience by a variable that correlates with the number of steps till the desired goal. The reviewers mention that the authors focus on an important problem, their method is simple and the empirical results are significant. However, they do point several flaws of the paper, the main ones being questionable theoretical claims and the clarity of the presentation. After an extensive discussion, most reviewers agree that the paper should be accepted but I do encourage the authors to take into account the comments by the reviewers for the final version of the paper and make the theory more clear.
Decision: Accept (Poster)