Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper looks at a combination of offline reinforcement learning and planning, in order to learn from offline data while being able to relabel the reward. Specifically, implicit q-learning is used to learn a Q/value function estimator, from purely offline data, and a topological map is built where nodes are sampled observations and edge costs come from the value function. Results are shown on a robotics domain where the value functions are learned on observed robot trajectories and task-relabeled (e.g. preferring particular types of terrain such as grass) with results then shown on an urban environment. 
strengths_and_weaknesses: Strengths

 ++ Overall, methods utilizing offline data are extremely relevant, as abundant data can be gathered or crawled rather than the robot having to perform exploration or roll-outs, which takes significant time and could be unsafe. 

  ++ I really like the idea of combining learning (value functions) and planning

  ++ The paper is understandable and well-written, though could use significant polish (see below)

  ++ The results shown seem to show significant gains 

Weaknesses

 -- Overall, the paper could use significant polish including:
   * Figures 1 and 4 are very blurry and should use vector graphics (or at least higher resolution images)
   * Comments are left in, e.g. L43 "scaling law papers?", missing citations L86/L87 "[]", etc.
   * Several grammatical issues:
     * L4: "utilize previously collect data"
     * L105: "we specifically exploring"
  * The "Discussion" is not actually a discussion - it's a typical conclusion that just reiterates what's in the paper, not a synthesis

 -- The motivation for proposition 3.1 is both natural but also against the stated goal of tackling long-horizon situations which violate assumptions like the Markovian property; yes, planning along the value function-based graph will obviously recover the optimal policy but 1) that requires discretization (the topological map), 2) requires a lot of assumptions, and 3) doesn't happen when things are not Markovian. It doesn't seem to answer anything about why this method might be better than just getting the policy itself through RL (I'm not saying it's not better, just that it recovers the "optimal solution" under unlimited data). 

-- "But this assumption becomes increasingly violated for long horizon tasks with first-person images: while goals that within line of sight of the robot are relatively simple, goals that require navigating around obstacles tend to fail if using the policy directly". This is a (pretty specific) hypothesis yet no evidence for this claim is made. Overall the state transition still seems Markovian here, but even if not it's not clear that this matters and is the reason that the proposed method is better. 

 -- Overall, while some citations are made to the goal-conditioned learning+planning literature, there is not a significant positioning of this paper and that field (see [A] and many papers that cite it). It is a large literature, and merely saying a few short sentences about how related work differs (actually there is not a strong statement I can see in how this paper differs from the goal-conditioned RL-based planning literature). Similarly there is a rich history of topological maps in robotics [B] and while obviously those were pre-deep learning/RL resurgence, were there any prior topological mapping papers that derived their edge costs from some learned functions (RL or otherwise)?

  -- Similarly, there are a few experiments on a custom platform/experiments and the authors show that their method is better than some sampling of the literature; it's not clear *why* the method works better in this case nor *why* those other methods fail. Indeed, the setup is not that complicated in terms of task, so it's not clear to me why the other methods fail. It is not sufficient to just show the results without any significant analysis (ideally tying back to why the authors hypothesized it would do better). 

  -- Overall, there is significant downside in that the topological map has to be pre-built in the new environment - this is akin to exploring the new environment. While no learning is done with this data, it still has access to it, so it's not clear that it's a fair comparison with other methods that don't utilize such data. For example there are a lot of works in the vision-language navigation literature that do pre-exploration [C], and perhaps such works should be compared to. 

[A] Nasiriany, Soroush, et al. "Planning with goal-conditioned policies." Advances in Neural Information Processing Systems 32 (2019).
[B] Kuipers, Benjamin, et al. "Local metrical and global topological maps in the hybrid spatial semantic hierarchy." IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA'04. 2004. Vol. 5. IEEE, 2004.
[C] Tan, Hao, Licheng Yu, and Mohit Bansal. "Learning to navigate unseen environments: Back translation with environmental dropout." arXiv preprint arXiv:1904.04195 (2019).
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Overall, I think this paper is promising in terms of an interesting idea for carrying out this task, using only offline data to train on; however, as it reads currently it presents a (pretty small set) of hypotheses with what's wrong with existing methods, how this paper is situated with respect to them, and why it works better. Since this is done on a completely custom environment, the paper needs significant additions in terms of why it works well, linking it back to claims made in the beginning. 

=== Post-rebuttal

Thank you to the authors for providing additional explanations and experiments. Overall, the authors do address some of my concerns well. However, I am still not fully convinced by the situation of this work w.r.t. prior works on toplogical mapping, or by the experiments which use only limited custom datasets. On the other hand, I believe that the paper does provide an interesting perspective that combines topological mapping with offline RL; therefore, while the paper is not perfect, I think it is high-quality and valuable to be published as a poster. 
issues: - Fix polish issues method
- Provide a much better positioning of how this method compares to the prior state of art (including goal-conditioned and topological mapping literature)
- Provide analysis/explanation as to why the method works better and prior methods fail -- this is presently unclear
- Explain why the comparison is fair even though pre-exploration is done in the new environment to build the topological map
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a system, ReViND, enabling a robot to learn how to navigate based on existing data and incorporate various types of utility beyond path length. The claimed benefits of ReViND are that it is lightweight with respect to additional data collection, able to incorporate utility other than minimum path length (e.g. need for solar power), and can generate viable trajectories at longer horizons than prior work. 

ReViND is as follows: existing data is relabeled with custom reward, in this case procedurally, and a policy and Q function are trained on this data using IQL. A topological graph is then constructed by doing "drive-throughs" of the environment and creating a graph with drive-through observations as nodes and Q function values to weight edges. After training, a robot will receive a goal, find the shortest path using Dijkstra's on the graph, then follow the path using the policy.

The data is tested in five environments with two additional utilities. 

strengths_and_weaknesses: **Strengths** 
- Thoughtfully-designed system that aims for and incorporates many important requirements and constraints, including impracticality of custom data collection and real-time responsiveness (as needed in online RL), various needs from a single algorithm, and fragility of purely independent offline RL
- Simple and clear solutions to the problems outlined
- Benefits of the topological graph in particular are demonstrated convincingly through simple but targeted results. More broadly, the approach to evaluation is reasonable and intuitive. 
- The paper is very well-written. After the initial readthrough, the structure and details are clear. The paper structure makes sense and the writing is easy to follow. 
- Overall, a great project setup

**Weaknesses**
- Paper claims that users can use ReViND "without collecting any of their own data". I agree that the use of offline RL is a huge benefit that probably makes this system usable whereas an online approach would not be. However, the paper's language strongly suggests that not needing to collect *any* data is a primary benefit, but the drive-throughs and relabeling are still required. It's possible that drive-throughs are feasible and negligible compared to the data needed for offline RL, so this might be a nitpick, but it's not so clear with relabeling. The procedural relabeling seems useful, but anything more sophisticated and tunable for the user's needs would require either hand relabeling (very expensive for a large offline dataset) or a reward model, which would require data of its own that may need to be custom to be domain-adapted. Given that adaptability is another main claimed benefit, there seems to be a trade-off without a solution. 
- Paper claims that ReViND is highly adaptable beyond minimum-length path optimization. I'm convinced that adding in another loss term procedurally based on an additional reward annotation works, but that's not just because of the experimental results, it's also because that's very well understood. Adding loss terms together that come from ground-truth reward annotations is very standard. Therefore, the claimed flexibility of the system doesn't strike me as novel. 
- Furthermore, utility in the sense of balancing sun for solar power and least distance for efficiency, or grassyness and least distance, or any other such example, presents a tradeoff. How the degree of tradeoff should be decided depends on circumstances specific to the situation, which the training data may or may not contain. Even in the best case where it does, the single offline RL-trained algorithm will make decisions in a particular way, and it's important for the user to at least understand how tradeoffs are being made. I would therefore appreciate examples in goal cases where there are different degrees of tradeoff between e.g. sun exposure and least distance, to understand how the trained algorithm fares. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, I think that this paper is clean, elegant, and useful. My main concerns are about significance, because it has three main claims of uniqueness - no need for user data, which seems to be false (but it's certainly true that the offline approach drastically reduces the need for training data), long horizon capability (useful and convincingly presented), and ability to include custom utility (unclear how much this is in the user's control, and the method is not only unsurprising but completely standard machine learning). However, since it is a low-data useful system that can handle real-world-scale navigation problems, I do think it's a solid applied paper. 
issues: More thorough experimentation on utility, and clarification of the data claims.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper studies how offline RL can be applied to long-horizon navigation tasks with different user preferences. The method leverages graph-based methods to search for distant goals and combines it with offline RL to navigate locally. The authors claim that the proposed method can learn from public offline datasets and generalize to novel (unseen) environments in zero-shot.
strengths_and_weaknesses: Strengths: 
* The paper tries to address a very important and challenging navigation problem. 
* The experimental results seem very promising in that the method could learn in zero-shot from publicly available datasets. However, there are some concerns regarding this (please see the weaknesses below). 

Weaknesses: 
* The paper claims that by relabeling large-scale offline datasets, one can expect effective generalization in real world even without the need for any task specific data collection. The provided experimental results also support this claim. However, the paper does not study what conditions are required to make this generalization feasible. In general, one cannot expect such zero-shot generalization to happen, especially using public datasets collected by someone else in completely different environments. The paper needs to provide more details why this generalization is possible. Then, explicitly describe the limitation of the method, i.e., conditions in which the method would fail.  
* One contribution of the paper is to learn different behaviors from an offline dataset. However, this is done by re-labeling the transitions in the offline dataset by defining a reward function. There are two issues here: (1) defining a reward function, especially to label image observations, is challenging and sometimes require a lot of manual work, and (2) relabeling is a standard approach and cannot be claimed as a contribution here. 
* The theoretical contribution of this paper is very limited. The method resembles SoRB with two differences: (1) formulating the problem as an offline RL, and (2) re-labeling the offline data with different reward functions. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The most important concern is that the paper shows zero-shot generalization for a very challenging problem without clearly describing what makes this generalization feasible. It is possible that the method is heavily engineered to work only on this particular setting, and does not scale well to other similar problem settings. 
issues: * Some references are missing. See lines 43, 86, 87. 
* Some details of the IQL method are missing. As an example, $\tau$ is not defined. Besides, since the method is not constructed on top of IQL (it is only used as a tool for offline training), there is no need to describe IQL. I suggest to move section 3.2 to appendix.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: he authors consider the problem of autonomous robot visual navigation over long horizons while obeying hand-specified reward functions to achieve additional behaviors like "stay in sunny areas while navigating." To achieve this, the authors leverage recent methods in offline reinforcement learning to learn value functions for short-horizon goals, and use a graph search algorithm to reason over the remaining portion of the planning horizon. Impressive experimental results showing the efficacy of the approach are presented. 
strengths_and_weaknesses: (+) Broadly speaking, this is an extremely interesting problem space, and it's very interesting to see the authors bring offline RL tools to bear on the particular problem of long-horizon visual navigation. 

(+) The experimental work presented here is extremely impressive, with the authors having used the proposed method to train a robot to perform the desired behaviors in outdoor deployments over distances on the order of kilometers. 

(+) The discussion about the authors experience with visual navigation relating to non-Markovian state and the short-horizon limitation of value function learning was very much appreciated. 

 

(-) Some discussion that might be critical to reproducibility is missing. In particular, because the problem being considered is one of combining two objectives (ie, the navigation task and the secondary task that the authors refer to as maximizing a "utility measure"), it strikes me that the problem of how to "balance" these two objectives at deployment time is of critical importance. The paper is careful to specify that the utility measure must be specified manually by a human, but I didn't find any discussion of how these two objectives are, eg, weighted relative to one another. 

(-) To my read, it seems as though the authors are implying that the value functions learned using ORL with the manually-specified utility measure are potentially transferable to new environments (eg, the discussion at the end of 3.1). But it doesn't seem to me that the experiments actually support this claim as it appears that each of the experiments is done in the same environment as the offline data set was collected. I do note that there is some discussion related to this in the "limitations" section, but it doesn't seem to address this concern in particular.  

(-) As a minor comment, the paper in its submitted form still requires a few polishing tasks such as removing comments between authors (blue text on L43) and missing references (L85-86). 

(-) As a minor comment, the text/numbers on the left hand side of Figure 3 were not readable for me. 

(-) As a minor comment, L246 mentions something about driving "on loans," which seems like a typo but I'm not sure what was meant ("roads?"). 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper synthesizes several important topics in the robot learning literature, including offline reinforcement learning and long-horizon visual navigation and describes an interesting and potentially impactful method for incorporating manually-specified utility measures into the navigation task using only pre-collected data. That said, there are some shortcomings in the current draft including important but missing discussion regarding algorithmic details and limitations.

POST-DISCUSSION UPDATE: The authors adequately addressed the issues I raised during the discussion phase, and therefore I'm raising the score of my recommendation.
issues: (see weaknesses above)
review_title: null

Meta-review: metareview: Scores: FZSz: Weak Accept, V1xj: Weak Reject, qBaA: Strong Accept, 6zkp: Weak Reject

Quality: The paper is of high quality enabling long-horizon navigation of a real robot with 
 user-specified navigational preferences by only using an offline dataset. 
   
Clarity: The paper has been improved in the revision. Better contrast to prior work and flat RL were provided. A clarification regarding reward relabeling was added, and zero-shot capabilities were shown in visually similar environments. Additional experiments for analyzing the tradeoff between utility and goal-reaching were performed.

Originality: This seems to be the first algorithm that combines topological graphs with offline RL for long-horizon planning capabilities and testing it on real navigation tasks.

Significance: Provides a solid contribution to robot learning by combining offline RL with topological graphs for long-horizon planning and incorporating utility functions.

Pros: 
-  Combination of offline RL methods with topological maps. 
- Experiments on a real-robot platform. 
- Long-horizon capabilities and ability to include custom utility.

Cons: 
-  Extending the algorithm to more complex utility measures is challenging and may require tedious manual work.
- The conditions under which circumstances (utility measures, environments, etc.) the algorithm will fail/work are not entirely clear.








confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Oral)