Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a novel method to learn from offline datasets. The paper proposes to first learn an RNet, and then distill the RNet predictions onto a DAG to obtain self supervised rewards from the datasets in a self-supervised manner, which does not require reward specification on any sort, at any stage of the pipeline. The paper evaluated the methods on two continous control tasks and improves upon prior self-supervised reward learning methods like HER. 
strengths_and_weaknesses: Strengths:
- The employment of both Rnet and the graph distance seems like a novel and exciting approach. 
- The subgoal and edge augmentation are very interesting and practical solutions during policy training, since the
- Results on YogaWalker are good.
- The ablation is thorough and discusses all different variants of their proposed algorithm.
- Fig 1 is extremely well done. 

Weaknesses:
- The way the paper is presented, it seems to be mostly for discrete state spaces to construct the graph. But the results shown on YogaWalker is obviously a cts state space. I am confused how the paper constructed the graph for a cts space.
- I would ideally like to see more experiments with cts state space tasks.
- I would like to see the paper explore real robotics tasks, if possible. 
- RNet predicted distance in fig 3 a seem very wrong. Rnet is seemingly overfitting to the xy position distance not the actual map of how "reachable" the far ends of the maps are. I am not convinced the RNet is learning anything meaningfully correct, and the graph is doing the heavy lifting. To me it seems like R is not learning anything particularly meaningful, and the distillation to the DAG is somewhat lucky due to the hill-climbing way of learning.
- Using DAGs are a sensible choice, but the paper doesn't empirically show it in a convincing manner. 
- "we leverage the temporal direction because a state can be reachable from another without the converse being true", i would be interested in a short analysis of if the fwd-bwd models are consistent, and some intuition about if it makes sense for them to be consistent. For most robotic tasks, I can imagine the A->B and B->A is equally reachable, but I am not sure if that holds for some of the very specific tasks considered. 
- Intuitively, the method would perform well on similar tasks as the original HER paper. I would like to see some results on manipulation tasks.
- NIT: typically ablation should follow the main experiments / comparisons with the SOTA


robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: The paper presents a novel, and very interesting approach to learn from offline datasets, using a self-supervised approach, which employs learning a graph to measure the reachability between two states. Overall, the method is extremely interesting, but the experimental evaluation is not sufficiently thorough. 
issues: Please see weaknesses above
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper addresses the problem of learning long-horizon tasks from offline (pre-collected) datasets of behaviors. The paper proposes to learn the structure of the task by learning a global distance in the state space of the MDP by building a graph prior to the offline training. Similar to prior works, using hindsight experience replay, the method constructs a dense reward signal to be used for goal reaching tasks with sparse rewards. 
strengths_and_weaknesses: Strength: 
* The paper proposes a novel method that can solve long-horizon RL tasks provided offline datasets of state-action transitions. 

Weaknesses:
* The paper does not compare to most recent self-supervised offline RL algorithms such as "RETHINKING GOAL-CONDITIONED SUPERVISED LEARNING AND ITS CONNECTION TO OFFLINE RL" (reference [3]). 
* The experimental results are limited to only two tasks. 
* The method does not seem to scale to vision-based policy training tasks since the graph cannot be constructed over image space. 
* Please also see the issues. 
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: The paper introduces a novel method to solve challenging long-horizon tasks in offline RL. The experimental results and the ablations show benefits of the method on two long-horizon offline RL tasks. However, one important baseline is missing and, since the experiments are done in simulation, one can expect results on more than two tasks. 
issues: Line 164-165: "we can use a trick to connect the graph if necessary by adding ...": This is very unclear to understand what the problem is and what the solution is. 

Line 186 (Edge augmentation): It is  unclear where the action a_t^i comes from. A new transition (augmented one) is found by finding s being an incoming state to the node n_{i-1}, and s' being an outgoing state for the node n_i. Therefore, to connect s to s' we need more than one action, most likely a sequence of actions. 

Line 229 (Importance of graph directness): It is obvious that we need asymmetric graph for the given task. A symmetric graph would work for tasks like 2D maze in which actions are reversible. Considering this, I don't understand the purpose of the ablation other than answering a very obvious question.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper is concerned with unsupervised goal-conditioned policy learning. Manually designing reward functions for every single desired skill is very expensive. The authors propose a novel self-supervised learning phase on the pre-collected dataset to understand the structure and the dynamics of the model, and shape a dense reward function for learning policies offline. The proposed algorithm is benchmarked on a gridworld environment UMaze and RoboYoga walker environment from DMControl. 
strengths_and_weaknesses: **Strengths**:

* This paper is well-written and easy to follow. The motivation of goal-conditioned offline policy learning is well explained. 
* To my knowledge, the proposed method of combining Reachability Network (RNet) and a graph-based distance (reward) metric is novel. The method and details are also well-explained. 

**Weaknesses**:

The proposed algorithm is benchmarked on 2 environments, UMaze (a toy gridworld) and RoboYoga (DMControl, another artificially constructed environment). While the experimental results are informative, neither environments have practical value or concrete applications beyond a simple demonstration. For example, the Walker bot in RoboYoga does not have a real counterpart and cannot be realized on available hardware. In prior works such as LEXA [1], empirical results are shown on simulated robotics task like RoboBins and RoboKitchen that involve realistic robot grippers. Since this is a robot learning conference, it is appropriate to show more meaningful and useful robotic tasks. 

Furthermore, limitations are not well discussed in Sec. 6. The current statement is insufficient. 

[1] Discovering and Achieving Goals via World Models. Mendonca et al. 2021. 

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: I am willing to consider raising score if the proposed method can achieve convincing results on simulated robotic tasks that are more meaningful and practically useful. 
issues: My main issue is with the toy experiment settings. Please see "Weakness" section. 

In addition, the limitation section is not really there in Sec. 6, requires a lot more expansion.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a method for learning goal-conditioned policies from reward-free offline datasets. To this end, the method first learns a reachability network, constructs a graph over a state space, and re-label and augment the dataset using the reward defined as a sum of distance measured using the graph and the reachability network. Then the off-policy goal-conditioned RL agent is trained using the dataset. The method is evaluated on UMaze and RoboYoga environments.
strengths_and_weaknesses: Strengths
- Method is a bit complicated but makes sense and shown to be effective on various tasks
- Experiments are well conducted to show the benefit of the proposed techniques with ablations

Weaknesses
- Experiments are conducted on relatively simple tasks -- Performance on more complex tasks, especially manipulation tasks, could further support the claim of the proposed method for learning distance in state space. As a baseline (i.e., Actionable Models) shows its effectiveness on various robotic experiments, I would expect this could also be the case for the proposed method, but explcitly showing this would be required for making the paper strong and more relevant to robotics.
- In terms of presentation, Figure 1 is helpful for understanding high-level concept but more figures for detailed methods would help understanding in detail, especially for augmentations, because notations are a bit complex. For notation, introducing superscripts as a notation for trajectory index and also introducing subscript differently for augmentations makes it a bit difficult to understand what's going on.
- I would not say that this is necessary; but it would be interesting to see how the proposed method performs with high-dimensional observations (pixels) by training a reachability network on pixel observations. At least including the discussion on this extension in the paper could be interesting and helpful for future researches.

Minor
- It's awkward to say to call a robot used in DMC Walker as 'humanoid'
- Figure 3: information about what does each color mean is missing
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations section not present
summary_of_recommendation: I would recommend this paper to be accepted because the method is intuitive and results are strong. But it's difficult to argue strong acceptance because there is a room for improvement, e.g., including additional robotic experiments and improving presentation.
issues: - Additional robotic experiments, especially manipulation experiments that would enable direct comparison with the baseline
- Improved presentation
- Experiments with high-dimensional observations
review_title: null

Meta-review: metareview: # Strengths
- The approach is interesting and the idea of combining reachability networks with graph-based distances is novel and promising

# Weaknesses
- The evaluation is fairly limited wrt the chosen scenarios.
- There are some concerns regarding practicability and scalability to real-world robotic tasks.
- The paper would strongly benefit from additional evaluations on manipulations tasks comparable to the tasks conducted by prior work.
## Limitations
The paper does not discuss the limitations of the proposed approach. Such as discussion is however required by the conference.

# Post-Rebuttal Update
The authors did address the concerns of the reviewers and even evaluated the approach on an additional experimental task.
While I appreciate the work done in the rebuttal phase I expect that the additional experiments in the rebuttal document as well as the clarifications provided in the discussion with the reviewers will be added to the camera-ready version of the paper.
Furthermore, the limitations in the rebuttal document are still very superficial and don't provide any valuable insight into the constraints and downsides of the method. I strongly encourage the authors to provide an insightful limitation section for the camera-ready version.
confidence: 5: The area chair is absolutely certain
Decision: Accept (Poster)