Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work presents a method for *offline* goal-conditioned policy learning, from offline data that *does not contain actions or rewards*, using only a specified target goal distribution p(g). The approach, named GoFAR, is formulated as a state-occupancy matching objective, and draws upon a similar derivation as SMODICE [1], with additional modifications to be applicable to the goal-conditioned setting. As a result, GoFAR inherits the advantages of SMODICE. Experiments study various components of the GoFAR method, provide real-world robotic results, and demonstrate cross-embodiment transfer.

References:

[1] Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching

strengths_and_weaknesses: Originality:
- I would not necessarily consider this work very original. As the authors point out, goal-conditioned reinforcement / imitation learning (as well as their state-only variants) are important well-studied problems. Furthermore, the GoFAR approach appears to be an extension of SMODICE [1] to the goal-conditioned setting.

Quality:
- I believe the quality of the paper is good. The writing is largely clear, and the experiments try to cover various aspects of the approach. However, there are a number of questions / concerns noted below.

Clarity:
- The paper is generally well-written

Significance:
- I think the significance can be better understood after authors response to my questions / concerns below.

References:

[1] Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching
questions: - Why is this called goal-conditioned RL? The objective being discussed throughout the paper, i.e. objective (1), is a state-occupancy matching objective that does not have any rewards involved. Also the title does not reflect that the method is directed towards the setting where the offline dataset contains states only (no rewards or actions).
- I don't see how the teleportation described in the abstract is relevant to this work? Do you mean because p(g) is a distribution of goal states?
- I think in this work the authors need to discuss SMODICE [1] more thoroughly as the method is very closely related. Furthermore, a number of "advantages" of GoFAR that are listed are due to the use of SMODICE-like methodology, rather than a direct contribution of this work. Such aspects should be more clearly discussed. Also in Section 4.1, it would be good if the manuscript noted that the derivations draw upon similar derivations in prior DICE lines of work.
- For state-occupancy matching related work it would be good to cite [2] as they also present results with state-marginal/occupancy matching, and present results for target state distributions that were not generated by an agent (e.g. hand designed target state distributions).
- Equation (7): It is not clear to me why this is a looser bound than (5). Do you have a derivation for why this would be a lowerbound to (5)?
- Equation (11): You should explain why you are proposing to learn the policy using equation 11. Namely the fact that the "f-advantage" is equal (at least theoretically) to the density ratio $\frac{d^*}{d^O}$.
- Line 192: Can you please elaborate on how Bayes rule leads to that equality? Also, I don't have an intuition for what p(g | s, a) is? You explained initially that p(g) is a "desired goal distribution", but I don't understand what p(g | s, a) would mean?
- Section 5.1: It would be good to note which methods are state-only, which methods are state and action, and which methods use rewards. Despite the appendix, it would also be good to at least briefly mention what are the task rewards (e.g. dense or sparse), how the goal distributions are defined, etc.
- Section 5.1: Instead of DDPG, why did you not opt for an offline RL method, given that online RL algorithms tend to perform poorly in offline settings?
- Section 5.1: Please report standard deviation of results, even if it's only 3 random seeds. It is very challenging to understand the significance of your results without error metrics.
- Section 5.1: Can you elaborate on how the sparse binary reward is used with equation (7) to implement GoFAR (binary)?
- Baselines: I think a potentially unfair advantage that GoFAR has compared to the other approaches is that it is being given $p(g)$, the distribution of desired goal states. In contrast for example, HER is relabeling any future state as a goal state and performing updates using all those possible relabeld "goals". I think important alternative baselines would be: 1) use $p(g)$ in HER as well. For example, only do HER relabeling using goals that are in the support of $p(g)$. 2) Perform goal-conditioned offline RL, e.g. using CQL or any favorite offline RL method, where the input to the policy / value function is (s, g) / (s, a, g), where g ~ p(g), and the reward is relabeled as r(s, a, g).
- In the appendix G you don't describe how the $\tilde{O}$ relabeled distribution is obtained for each method, e.g. GCSL.
- Section 5.2: Without errorbars, it is not possible to evaluate the significance of the results.

With regards to aspects that would directly help me better evaluate and improve my score:
- Addressing my concern regarding baselines, either verbally convincing me that I am mistaken in my thought process / assumptions, or implementing the two suggested baselines and demonstrating that they would not resolve the poorer performance of the baselines you tried.
- Highlighting exactly what you believe are the core contributions of *your work*, not aspects of your work that are due to the use of SMODICE, etc. What can GoFAR do that prior methods cannot (e.g. type of data, type of available information, etc.)? What is the first impressive result you would show a person? If I am choosing an algorithm for my use-case, why would I opt for GoFAR as opposed to the other methods (currently I am not convinced solely by numerical results since there is a lack of reporting of standard deviation / error bars across random seeds, as well as my concern regarding baselines discussed above)?

References:

[1] Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching

[2] A Divergence Minimization Perspective on Imitation Learning Methods


limitations: Authors have not highlighted the limitations / potential failure cases / future work for GoFAR.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents an approach for offline goal-conditioned RL that is based on state-occupancy matching. In particular, the paper outlines a procedure for a goal-conditioned state occupancy optimization in an offline setting by deriving a lower bound on the optimization objective such that it does not require online sampling. While doing so, the algorithm incorporates f-divergence based dual optimization and decouples value function from actor policy training, which improves stability of training. In contrast to prior works, the method does not require hindsight relabeling, which further stabilizes the training. The algorithm is compared to multiple recent GCRL baselines and it is shown to outperform them on several simulated robotic manipulation tasks as well as in a dexterous real-world D’Claw environment.
strengths_and_weaknesses: The paper presents an approach for offline goal-conditioned RL that is based on state-occupancy matching. In particular, the paper outlines a procedure for a goal-conditioned state occupancy optimization in an offline setting by deriving a lower bound on the optimization objective such that it does not require online sampling. While doing so, the algorithm incorporates f-divergence based dual optimization and decouples value function from actor policy training, which improves stability of training. In contrast to prior works, the method does not require hindsight relabeling, which further stabilizes the training. The algorithm is compared to multiple recent GCRL baselines and it is shown to outperform them on several simulated robotic manipulation tasks as well as in a dexterous real-world D’Claw environment.

Strengths
- Efficiently training goal-conditioned policies from offline data promises to enable training on large and diverse robotic datasets.
- The paper is well-written, and straightforward to understand and follow. Mathematical derivations are coherent.
- The method is shown to work on a variety of interesting manipulation robotic tasks, including a real world dexterous manipulation task.
- The paper provides source code of the method implementation.

Criticism
- It would be interesting to see an evaluation of the method on higher-dimensional and especially articulated visual goal tasks to better understand the scalability of the approach. 
- It would be interesting to see an evaluation on tasks that require stitching of multiple episodes to accomplish a task, to better understand the ability of the method to combine experiences from different trajectories.
questions: Please refer to Strengths And Weaknesses for a list of questions.
limitations: The authors discussed limitations of their work. The proposed approach does not lead to any obvious negative societal impacts.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new regression-based offline GCRL algorithm from the perspective of state-occupancy matching. Compared to previous methods, GoFAR does not require any hindsight relabeling. GoFAR also decouples the training process of value and policy updating. These features make GoFAR enjoy the benefits of better performance, training stability, and zero-shot transfer to new target domains.  
strengths_and_weaknesses: The biggest contribution of this paper is casting offline goal-conditioned RL as a state-occupancy matching problem. After this formulation, GoFAR reuses techniques from existing offline imitation work [1][2] (see below) to solve the state-occupancy matching problem by regarding the data that accomplish the desired goal as expert data, and the whole offline data as supplemental data.
However, I do have several concerns about this reformulation of offline goal-conditioned RL.

1. The objective of goal-conditioned RL is to find a policy that maximizes the cumulative reward, while state-occupancy matching is only mimicking the expert. Suppose: 1) the reward is not sparse reward (not the case that only gets 1 reward when accomplishing the goal) and 2) the dataset contains successful but not optimal trajectories that accomplish the goal g (i.e., the cumulative reward is low). In this case, mimicking these trajectories will only get suboptimal policies.
2. In Appendix E.1, the authors give a theoretical connection between Goal-Conditioned State-Matching and Probabilistic GCRL, however, this builds on the premise that we can choose $r(s; g) = p(s; g)$. Why is it a valid choice under RL (i.e., have the freedom to choose $r$ as $p$)? It is reasonable under the imitation learning setting but is problematic under the RL setting. As the reward function is given by the problem and $p(s;g)$ is given by the offline data.

Aside from the concerns, the additional strengths and weaknesses of the paper are listed as follows

**Strengths:**

* This paper provides a novel view to solve the offline GCRL problem by adopting state-occupancy matching. It avoids hindsight relabeling, improves training stability, and owns the zero-transfer ability.
* The authors connect well with probabilistic GCRL and HER.
* The proposed method performs well on offline GCRL tasks and the authors conducted comprehensive experiments to show the advantage of GoFAR.

**Weaknesses:**
* The learning objective of GoFAR, Eq. (5), is a lower bound of $KL(d^{\pi}, p)$, rather than the exact KL divergence, and objective (7) is an even looser lower bound.
* The benchmark experiments are not sufficient. Could you test your algorithm on the offline GCRL benchmark introduced in [3]?
	
[1] Kim et al, DemoDICE: Offline Imitation Learning with Supplementary Imperfect Demonstrations. ICLR, 2022.

[2] Ma et al, Smodice: Versatile offline imitation learning via state occupancy matching. ICML, 2022.

[3] Yang et al, Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL. ICLR, 2022. 

questions: 1. Regarding the first concern listed above, why we can choose $r(s; g) = p(s; g)$ in the general offline RL setting? If not, can we still cast offline goal-conditioned RL as a state-occupancy matching problem?
2. Can Proposition 1 be derived from the aspect of variational inference? It seems that Proposition 1 can be derived from variational inference with fewer lemmas and assumptions. 
3. Rewards signal: What’s the difference between discriminator-based reward and dataset-based reward? Could you add some ablations experiments using these two types of reward in the model?

limitations: The main limitation of this work is stated in my concerns regarding the reformulation of offline goal-conditioned RL as an imitation learning problem. If the authors can provide convincing responses to the above questions (especially Question 1), I’d like to raise my score.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a mechanism for performing offline multi-goal reinforcement learning without hindsight goal relabelling, by attempting to mimic the state-occupancy distribution of an agent that teleports directly to the goal. This avoids HER's well-known hindsight bias, and allows GoFAR to perform better on stochastic environments that are challenging for methods that employ hindsight goal relabelling. They find that this method outperforms offline versions of DDPG with HER and several other offline GCRL algorithms on several experiments, including real robot experiments. Additionally, they use the same method to derive a goal-based planner that allows for zero shot transfer to systems with new dynamics, as long as they share the same goal space and reward structure. They additionally prove several finite-sample guarantees for the method.
strengths_and_weaknesses: $\textbf{Originality: }$
The method is fairly original. It uses similar algorithm to SMODICE, but adapts it to a goal-conditioned setting. The two most unique insights here seem to be 
1) given a goal, the "optimal" teleporting agent's behavior is known $\textit{a priori}$, which means no expert demonstration is needed -- the learning objective can be derived analytically, while still retaining the strong theoretical guarantees from imitation learning. 
2) the derived algorithm can be transformed into a planner over intermediate goals. This allows a formulation that does not depend on actions, but instead plans over the goal space, allowing zero-shot generalization to novel dynamics, as long as the goal-space and task remain the same. 
These are both interesting and creative insights. 

$\textbf{Quality: }$
The math results generally look solid, and provide some nice guarantees on convergence. 

The experimental results are slightly more mixed. For several experiments, including the standard fetch experiments, no measures of statistical significance are reported in the main paper, and are only shown in the appendix. Additionally, success rates are not reported for these experiments. Broadly, the results for the Fetch environments  indicate that the method compares favorably against other offline methods, but still falls significantly short of the performance achieved by online methods. A real-robot experiment shows that the method is able to learn a task in which it manipulates a knob, while other methods fail. 

Most interesting is the goal planner experiment, in which the authors are able to significantly improve the performance of a weak policy by learning a sequence of goals that may be zero-shot transferred across different domains. They find that using the planer to guide the weak policy significantly improves the success rate.

$\textbf{Clarity: }$
Although the math can be a tad difficult to get through in places, I found it easier to follow than many other DICE-based papers. Broadly speaking, the core ideas are clear and well-articulated. 

$\textbf{Significance: }$
The proposed method solves the bias problem in the offline setting and provides valuable convergence guarantees. Additionally it provides good experimental results and proposes an interesting new approach to zero shot cross-domain transfer. The one downside is that it's a bit niche, by virtue of sitting at the intersection of two subfields, goal conditioned RL and offline RL. 
questions: 
For the fetch robot results, I'd like to see training curves shown with error bars for statistical significance. Statistical significance is reported for these experiments, but only in the appendix, which hides that fact that the improvement over other methods is not statistically significant for several tasks. The lack of improvement on these tasks is not in and of itself a problem, but it should be easier for a casual reader to see what is significant. I'd also like to see success rates for these methods -- it's more informative and intuitive than reward alone. 

I'd also like to see the goal-conditioned planner elaborated on a bit more, if space allows. For instance, what does a policy over goals mean exactly, and what does it look like? How do the formal guarantees given earlier in the paper translate to a policy that outputs goals
limitations: There seem to be no significant negative social impacts, and no obvious limitations other than those that are inherited from GCRL and offline RL.
review_title: null

Meta-review: metareview: The paper presents a novel method for offline goal-conditional RL that is based on reformulating offline Goal-Contitional RL as a state-occupancy matching problem. From this observation, the author are able to leverage and adapt previous work (in particular SMODICE) to their setting. Reviewers were in agreement this was a strong paper, with excellent writing, good mathematical rigor and thorough experimental work. There were some concerns regarding similarity to SMODICE which the authors addressed in their rebuttal.
confidence: Certain
Decision: Accept