Offline Reinforcement Learning for Visual Navigation
Authors: Dhruv Shah
Arjun Bhorkar
Hrishit Leen
Ilya Kostrikov
Nicholas Rhinehart
Sergey Levine
Venue: CoRL 2022 Oral (2022)
Keywords: offline reinforcement learning
visual navigation
motion planning

Abstract: Reinforcement learning can enable robots to navigate to distant goals while optimizing user-specified reward functions, including preferences for following lanes, staying on paved paths, or avoiding freshly mowed grass. However, online learning from trial-and-error for real-world robots is logistically challenging, and methods that instead can utilize existing datasets of robotic navigation data could be significantly more scalable and enable broader generalization. In this paper, we present ReViND, the first offline RL system for robotic navigation that can leverage previously collected data to optimize user-specified reward functions in the real-world. We evaluate our system for off-road navigation without any additional data collection or fine-tuning, and show that it can navigate to distant goals using only offline training from this dataset, and exhibit behaviors that qualitatively differ based on the user-specified reward function.

Offline Reinforcement Learning for Visual Navigation
Dhruv Shah', Arjun Bhorkar', Hrish Leen, Ilya Kostrikoy, Nick Rhinehart, Sergey Levine UC Berkeley
Abstract: Reinforcement learning can enable robots to navigate to distant goals while optimizing user-specified reward functions, including preferences for fol- lowing lanes, staying on paved paths, or avoiding freshly mowed grass. However, online learning from trial-and-error for real-world robots logistically challeng- ing, and methods that instead can utilize existing datasets of robotic navigation data could be significantly more scalable and enable broader generalization. In this paper, we present ReViND, the first offline RL system for robotic naviga- tion that can leverage previously collected data to optimize user-specified reward functions in the real-world. We evaluate our system for off-road navigation with- out any additional data collection or fine-tuning, and show that can navigate to distant goals using only offline training from this dataset, and exhibit behaviors that qualitatively differ based on the user-specified reward function.
Introduction
Robotic navigation approaches aim to enable robots to navigate to user-specified goals in known and unknown environments. The geometric approach to this problem involves using a geometric map of the enviromment to plan a collision collision-frre path towards the goal. The learning-based approach to this problem involves training policies by associating new inputs with prior navigational expe- rience, typically through imitation learning (IL) or reinforcement learning (RL). In many practical applications, the goal is not merely to reach particular destination, but to do so while maximizing some desired utility measure, which could include obeying the rules of the road, staying in bike lane, maintaining safety, or even more esoteric goals such as remaining in direct sunlight for solar- powered vehicle. In these cases, neither IL nor geometric approaches alone would suffice without accurate reconstructions of the environment or task-specific expert demonstrations, which may be difficult to obtain. RL can address these challenges, but prior work on applying RL to robotic navi- gation relies on infeasible amounts of online data collection, or requires high-fidelity simulators for simulation to real world transfer [1, 2]. Is there practical RL paradigm that can solve this challenge directly from real-world data?
RL from offline datasets [3] can address this challenge by learning policies from a prior dataset of trajectories and associated reward labels. Given a previously collected diverse dataset of navigational trajectories, it is possible to relabel that dataset post-hoe with reward labels as desired, train a policy that maximizes this reward function, and deploy it in the real world. Since this approach can leverage large datasets, it may lead to significantly better generalization [4] than methods that require much more tightly curated data, such imitation learning methods. However, end-to-end trained "flat" RL policies tend to perform poorly for long-horizon tasks [5-7]. How can we design a system for learning control policies from large datasets that can be immediately deployed onto a mobile robot?
In this paper, we describe a robotic learning system that performs visual navigation to distant goals (e.g. 100s meters away) while also incorporating user-specified reward objectives. Our system con- sists of two parts: (i) an offline Q-leaming algorithm [8] that can incorporate the desired preferences in the learned Q-function and trains policy operating directly on raw visual observations, and (ii) topological representation of the environment for planning, where nodes are represented by the raw
These authors contributed equally. Project page: sites www.se.oooog.oooo.o......... com/vies/revind. This research
was partly supported by DARPA RACER and DARPA Assured Autonomy.
6th Conference Robot Learning (CoRL. 2022), Auckland, New Zealand.

---

Hoading
Offline Q-Learning
Veloogg
Planning
STOTE
Flo g)
High
Current Obsercation
Figure Long-range RL with ReViND: it's corresponding value function V, from an offline dataset of interactions and user preferences, encoded as rewards. We then create topological graph using - V as the pairwise "distance function"". The minimum-com cost path to the goal in this graph is the desired reward- maxidiinxiiizin path to the goal, resulting in varied behaviors such goal-reaching while driving on the grass, or following bike lane.
We use Implicit Q-learning learn goal-conditioned policy and
visual observations and the connectivity between them is described by the learned value function (see system overview Fig. 1). While the Q-function alone may only be sufficient to learn accu- rate navigational strategies over short horizons, composing with planning allows scaling to large environments by searching for a plan that maximizes the desired objective coarse level. The low-level policy derived from the Q-function is subsequently used navigate between the nodes, maximizing the desired objective at both levels.
The primary contribution of this work is ReViND, a robotic system for Reinforcement learning for Visual Navigation from prior Data that can act in real-world environments, adopt behavior that maximizes the user-specified reward functions, and reach distant goals, by combining planning and Q-learning. We demonstrate that ReViND can incorporate high-level rewards, such staying pavements driving sunlight and reach goals in complex environments over hundreds of meters. ReviND is pre-trained on 30 hours of publicly available data [9] and is deployed in novel, visually similar environment without any on-policy data collection fine-tuning, To the best of our knowl- edge, this is the first demonstration of offline RL for real-world navigation utilizing only publicly available datasets. Our experiments show that ReViND demonstrates diverse qualitative behaviors by tweaking the reward objectives while outperforming policies trained with IL and model-free RL.
Related Work
Robotic navigation has been studied from the perspective of mapping. planning, imitation learning, and reinforcement learning. Classical navigation methods first acquire a geometric map and then use this map to plan collision-free paths [10]. The map can be built up incrementally, including with intelligent exploration strategies that maximize information gain or otherwise optimize for faster map acquisition [11-22]. Such methods often aim map out the entire environment first and then execute specific navigational tasks, though it is also possible to perform target-driven exploration, where the map is built in the course of navigating to specific goal [23, 24]. However, all such methods are fundamentally geometric: the task is defined as reaching some destination rather than in terms of semantic reward functions that we consider in this work.
Similarly, many soaming-based approaches navigation also focus on largely geometric tasks. such the "PointGoal" task [25], though they often utilize reinforcement learning methods that, in prin- ciple, can accommodate any reward function. The more severe issue with such methods is that RI. algorithms can require a large amount of online experience (e.g., millions even billions of trials) [1]. A method that requires a million 1-minute episodes would take more than 1.5 years of nonstop real-world collection, making it poorly suited for learning from scratch directly the real world. Therefore, such methods typically require simulation, transfer, and other additional compo- nents [26-28]. An alternative for encoding specific user preferences into learning- bbaed method employ imitation learning [7]. While imitation learning can enable user define their desired behavior through the demonstrations, such demonstrations are time-consuming to gather, and must be recollected for each new reward function. In contrast, our offline RL method utilizes previously

---

collected datasets, which we show is practical for real-world robots, and relabels the same dataset with different reward functions, which means no reward-specific data collection is needed.
Prior offline RL work has proposed a number of algorithms that can utilize previously collected data [3, 8, 29, 29-33]. Our goal is not to develop new offline RL algorithm, but rather explore their application to robotic navigation tasks. Most prior robotics applications of such methods in- clude multi-task learning for manipulation [34-38]. Unlike these works, our focus is specifically on how a single dataset can be reused to enable long-horizon navigation with different user pref- erences. To that end, we combine our approach with graph-based search to reach distant goals, which we show significantly improves over direct use of the learned policy, and utilize the same exact data to optimize different reward functions. While prior work has also explored the use of offline data with varying reward functions [39, 40], address significantly longer horizon tasks by incorporating model-free RL. and graph search.
Our use of graph search in combination with RL parallels prior work that integrates planning into supervised skill learning methods [7, 41] and goal-conditioned reinforcement learning [6, 42, 43]. However, our method differs from these works in two ways. First, while these prior works use the value function to estimate the temporal distance between pairs of nodes in the graph, we specifically explore using divese objectives. More importantly, we use offline RL, whereas prior work uses either supervised regression for distances, or online RL. Our goal is not to develop a new offline RL algorithm, but to explore it's application to robotic navigation tasks by building a learning-base -based system for long-horizon planning. To our knowledge, our work is the first to combine topological graphs with RL for arbitrary reward functions, and the first to combine them with offine RI..
3 Offline Reinforcement Learning for Long-Horizon Robotic Navigation
Our system combines offline learning of reward-specific value functions with topological planning over a graph constructed from prior experience in a given environment, so as to enable robot to navigate to distant goals while maximizing user-specified rewards. The learned value function is used not only to supervise a local policy that chooses reward-maximizing actions, but also to evaluate edge costs on a graph constructed from past experience. The graph is then used to plan a path, and the policy used to execute the action to reach the first subgoal on that path. Structurally, this resembles SoRB [6], but with two critical changes: learning "offline" value functions from prior data, and the ability to handle diverse reward functions beyond simple goal-reaching.
3.1 Problem Statement and Assumptions
The robot's task is defined in the context of a goal-conditioned Markov decision process, with state observations S, actions A, and goals g G. The robot receives a reward r(x:) each time step f, which depends on the degree to which it is satisfying user preferences (e.g., staying the graph). The objective can be expressed as maximizing the total reward of the robot's executed path, since the reward accounts for both the desired utility and goal reaching. The state observations consist of RGB images from the robot's 's forward-facing camera and a 2D GPS coordinate, the actions are 2D steering and throttle commands, the goal is a 2D GPS coordinate expressed in the robot's frame of reference. In this setting, reinforcement learning methods will learn policies of the form (arlst, St, It), though our approach will not command the final task goal Sr directly, but instead will use planning method to determine intermediate subgoals, which in practice makes it significantly easier to reach distant goals. To enable this sort of planning, we make an additional assumption that parallels prior work on combining RL with graph search [6. 7]: we assume that the robot has access to prior experience from the current environment that can use to build a topological graph that describes its connectivity. Intuitively, this corresponds to kind of "mental map" that describes which landmarks are reachable from which other landmarks. Importantly, we do not assume that this graph is manually constructed or provided: the algorithm constructs the graph automatically using an uncurated set of observations recorded from prior trive-throughs of the environment. In our experiment, these traversals are done via teleoperation, though they could also be performed
3

---

via autonomous exploration, and our method could be extended to handle unseen environments by integrating the exploration procedures discussed in prior work [9].
3.2 Reinforcement Learning from Offline Data
Offline RL algorithms learn policies from static datasets. In our implementation we use implicit Q-learning (QQL) [8], though approach is compatible with any value-hased offline RL algo- rithm. We summarize offline RL in general and IQL specifically in this section. Given dataset D [(8;, ai, 8()
1....... Nj. the goal of offline RL is to learn policy that optimizes the sum of discounted future rewards without any additional interactions with the environment. IQL involves fitting two neural networks, Qo and Vy. where Qo(s, g) approximates the Q-function of an implicit policy that maximizes the previous Q-function, and Vi represents the corresponding value function. The Q-function updated by minimizing squared error against the next time step value function, with the objective
L(0)= = )' gppp:="") (TVV + (((,,, Qo(s,a,)))))
where p(gls) is a goal distribution, which we will discuss later. The value function Vp (s, g) should be trained to correspond to Q0(s, a, for the optimal action a that maximizes the value at s, but directly computing max, Qo(s, a. g) is likely to select an "adversarial" out-o--distribution action that leads erroneously large values, since the static dataset does not permit Qo a, g) to be trained on all possible actions [3, 8, 32]. Therefore, IQL employs an implicit expectile update, with a loss function given by
L(V) =
a. - (=('''g)))
where Lī(u) |- 1(u 0)]u². This can be shown to approximate the maximum over in- distribution actions [8], but does not require ever querying out-of- sample actions during training. To instantiate this method, it remains only to select p(g)s) s).
Goal relabeling. The IQL algorithm not goal-conditioned [8], and the dataset was not collected with a goal-reaching policy, so the goals must selected post-hoc with some sort of relabeling strategy. While a variety of relabeling strategies have been proposed in prior work [5, 6, 38, 44, 45], follow prior work offline RL for goal-reaching [38] and simply the goal states that are observed in the dataset in the same trajectory at time steps subsequent implementation, we select this time step at random between 10 and 70 time steps after (the total trajectory lengths are typically around 80 steps). Algorithm I outlines pseudocode for training the Q-function with IQL.
given sample 8,- In our
Long-horizon control. Instead of directly using the policy learned with IQL, in this paper we use the IQL value function to obtain edge costs for graph used for topological planing. The standard IQL method directly extracts reactive policy from the Q-function. However, we found that in the real world, this approach was unable to reach goals farther than 20m, 80 time steps. A deeper analysis of the system revealed that, while the policy and values learned by the IQL agent are valid over shorter horizons, they degrade rapidly as the horizon increases. This is not surprising, because like all value-based methods, IQL, assumes that & represents a Markovian state. But this assumption becomes increasingly violated for long-horizon tasks with first-person images: while goals that are within line of sight of the robot are relatively simple, goals that require navigating around obstacles tend to fail if using the policy directly. In the next subsection, will discuss how we can use a topological graph as a sort of "nonparametric memory" of the environment to alleviate this challenge, enabling our method to reach distant goals.
3.3 Long-Horizon Reward Maximization with a Topological Graph
To enable long-horizon navigation, we combine the value function learned via offline RL with topological graph built from prior observations in given environment. As discussed previously, assume that the robot has limited amount of prior experience in the test environment that can be used to build "mental map." corresponding to a graph where nodes are observations and edges

---

represent the cumulative reward the robot will accumulate as it travels from one node to another. Note that this graph is topological rather than geometric: the nodes are image observations, and the connectivity is determined by the learned value function. We do not use the data from the test environment to finetune the value functions, only to construct the graph.
The graph G is constructed in the same way as prior work on graph- based navigation (see Shah et al. [7] for the closest prior method): each state observation s; in the test environment corresponds to We further
a node n4, and each edge Eij receives a cost corresponding to C(es)) = (s; filter these edges based on the GPS coordinates of the nodes to eliminate wormholes arising due to optimistic value estimates. For more details on how the graph constructed, please see Appendix C. Given an overall task goal, we add it the graph as an additional node ng. along with node representing the robot's current state, and then Dijkstra's 's algorithm compute the shortest path with these edge costs. We then use the policy learned via offline RL navigate to the first node along this path. Algorithm 2 outlines pseudocode for this procedure.
In our implementation, we use goal-conditioned reward functions of this form:
R(st, Oz 9) {
(sx)
Vs, *
otherwise.
where (s:) > 0 is always positive ensure that the planner actually reaches the goal.
Proposition 3.1 we we recover the optimal value function (s ss' for short-horizon goals s' (rela- tive to s), and G S (all states exist in the graph), and the MDP is deterministic with 7 = 1, then finding the minimum-cos. path the graph with edge-weights -V*(8, 8') recovers the optimal path, that is, a policy that maximizes V*(sgg).
Proof (sketch): The Bellman equation can be used to write the cost of the minimal-cost path in the graph with edge-weights - (s, J' = m +')'
= max [V s') J'(s', g)]- We can further expand (ss, into a sum rewards induced by the policy и and then rearrange the terms to obtain a similar optimality equation for V* that demonstrates that J (= = -v) While the above proposition makes strong assumptions, provides some degree of confidence that our proposed method is correct and consistent. We present further analysis of this proposition in Appendix A.
Algorithm Training Re
Algorithm 2 Deploying ReViND
1: Initialize parameters v/. 0. d. for each gradient step do
Inputs:
current observation obs := {img, 2). set
2:
of past observations N := {πı.-... nm }. IQL
3: Sample mini-batch (1,,s()
agent (Q, V, goal node ng E N
4: for each sample do
G ConstructGraph(//
5: T + T D si E 9i SampleCoal(
while not IsClose YáG:) do
6:
4:
(oodaleGaaphhobs))
7:
8;, 5 Relabel( $1, s1.
5:
Wi...., W/k <- DijkestaaSaardh(dbs, ni)
Ti <- Reward(s, gx)
6:
for t 1..... H do
+ Av (v)
goal vector Relatiiv,,, wx)
10:
+ Ao (8)
8:
RunPolicy (img. goal) runs on robot obs + next observation
Ô <- (1 a)d + al
9:
4
System Evaluation
We now describe our system and experiments that evaluate ReViND in woaldworld environ- ments with variety of utility functions. Our experiments evaluate ReViND's ability to incorporate diverse objectives and learn customizable behavior for long-horizon navigation, and compare it al- ternative methods for learning navigational skills from offiine datasets.
In our implementation, goals are detined only in terms of GPS coordinates, so technically, the second argument is only the GPS coordinate of sy. which we found to be sufficient. Extending the method to use the full image observation is simple modification.
5

---

Start
Goal
Agent Paths
Min-Dist
Grussy
Sunny
40m
Figure Comparison of policies for different reward functions learned by ReViND. Left: overhead map (not available to the method). with grassy areas indicated with green shading. Note that the policy for the "sunny" reward chooses a significantly different path through a concrete parking lot without tree cover, while the policy for the "grassy" reward takes frequent detours to drive on lawns. Right: first person images during each traversal, with the chosen path indicated with colored lines.
4.1 Mobile Robot Platform
We implement ReViND on a Clearpath Jackal UGV platform (see Fig. 1). The sensor suite consists of 6-DoF IMU, GPS for approximate localization, and wheel encoders to estimate local odometry. The robot observes the environment using a forward-facing 170" field-of-view RGB camera. Com- pute is provided by an NVIDIA Jetson TX2 computer, with the RL controller running on-board. Our method uses only the images from the on-board camera and unfiltered GPS measurements.
4.2 Offline Trajectory Dataset and Reward Labeling
The ability to utilize offline datasets enables ReViND to learn navigation behavior directly from existing datasets - which may be expert tele-operated or collected via an autonomous exploration policy- without collecting any new data. We demonstrate that ReViND can learn behaviors from a small offline dataset and generalize to a variety of previously unseen, visually similar environments including grasslands, forests and suburban neighborhoods. To emphasize this, we train ReViND using 30 hours of publicly available robot trajectories collected using a randomized data collection procedure in an office park [9]. Expanding this training dataset include more diverse scenes can help extend these results to alternate applications (e.g. indoors).
To utilize this data with our method, we "relabel" it with several different reward labels corre- sponding to diverse behaviors: simple shortest-path goal-reaching, driving in the sun (to emulate a solar-poweree vehicle that needs sunlight), driving on grass (to stay off the road), and driving on the pavement (to stay off the grass). We generate these labels by different mechanisms either by manually labeling them, by using learned reward classifier network, or automatically, by exploit- ing pixel-level patterns (e.g., in the color space). We implement these rewards via additive bonus to the negative rewards which corresponds to reducing the penalty for traversing these areas. For more details, see Appendix B. As discussed Sec. 3.2, we use IQL to learn the value functions and policies for each task.
4.3 Learning Varied Behaviors with ReViND
We now evaluate our method both in terms of its ability tailor the navigational strategy to the pro- vided reward, and in terms of how it compares to prior approaches and baselines. We test Re ViND in five suburban environments for a large number of goal-reaching tasks (see Appendix G). While these environments are visually similar to the offline training data, they exhibit dynamic elements such as moving obstacles, automobiles, and changes in the appearance of the environment across the seasons. In each evaluation environment, we construct a topological graph by manually driv- ing the robot and collecting visual and GPS observations. The nodes of this graph are obtained by sub-sampling these observations, such that they are 10-30m apart, and the edge connectivity is determined by the corresponding value estimates. Note that the Q-function is not updated with this data, it is only used to build the graph. Once the graph is constructed, the robot is tasked with reach- ing a goal location, where it follows Alg. 2 to search for path through the graph, and then executes

---

Start
Goal
Agent Paths
Min-Dist
Grassy
BC
MB
IQL.
Pavement
VING
Figure 3: Qualitatively, only ReViND reaches the
Figure 4: ReViND takes different paths through environment for different reward functions.
goal while prioritizing grassy terrain (shaded green).
Method
Uses Graph?
Easy (<50m)
Medium (50-150m)
Hard (150-500m)
Success
E1l grass
Success
E1 grass
Success
E1 grass
Behavior Cloning
No
1/5
0.08
0/5
0.04
0/5
0.12
Filtered BC
No
3/5
0.29
0/5
0.08
0/5
0.12
IQL [8]
No
3/5
0.37
0.29
0/5
0.16
VING [7]
Yes
5/5
0.07
4/5
0.09
3/5
0.14
Filtered BC Graph
Yes
5/5
0.24
4/5
0.15
3/5
0.19
ReViND (Ours)
Yes
5/5
0.47
4/5
0.84
4/5
0.78
Table Success rates and utility maximization the task of navigation in grassy regions (Rgrass).
via the learned policy. Fig. 2 shows the paths taken by different policies for a specific start-goal pair. The overhead image is not available ReViND and is only provided for illustration.
Our results show that utilizing value functions for different rewards from ReViND leads to signif- icantly different paths through the environment. For example, the "sunny" reward function causes a large detour through a parking lot without tree cover, while the "grassy" reward causes frequent detours to drive on lawns. All the policies successfully avoid obstacles and collisions and suc- cessfully reach the goal. Table we provide a quantitative summary of the behavior of the method for each reward function, showing success weighted by path length (SPI., which corresponds to an optimality measure that awards higher scores to successful runs with the shortest route length), the average value of the grass reward, and the average value of sun reward for trials corresponding to each reward function (note that these rewards are normalized to maximum of 1). As expected, we see that the values of these metrics strongly covary with the commanded reward.
Next, we compare ReViND to four baselines, each trained on the same offline dataset. These ap- proaches represent natural points of comparison for our method, and include prior imitation learning and RL methods, well prior graph-based method that does not use RL. Since our approach is (to our knowledge) the first to combine RL with arbitrary rewards topological graph search, no prior approach supports both graphs and arbitrary rewards. All methods have access to egocentric images and GPS, and command future waypoints to the robot.
Agent Utility
SPL
ERgmss
ER
Rdist
0.87
0.16
0.61
Raist + R Reruss
0.84
0.86
0.39
Raist Rsun
0.64
0.05
0.68
Table 1: ReVIND learns diverse behaviors that maximize the desired utility.
Behavioral Cloning (BC): goal-conditioned imitation policy that maps images and goals to con- trol actions [46]. This baseline does not incorporate reward information.
Filtered BC (fBC): A similar goal-conditioned BC policy that incorporates reward information by filtering the training data, picking only trajectories with the top 50% aggregate rewards [47].
VING: A graph-based navigation system that combines a goal-conditioned BC policy and distance function with topological graph [7]. This baseline does not incorporate reward information.
IQL: baseline that uses only the learned Q-function, without a topological graph [8].
Fig. 3 shows the qualitative behavior exhibited by the different systems for maximizing the "grassy" reward function. IQL and Filtered BC can incorporate the reward function into the policy, but
7

---

since they rely entirely on a reactive policy for navigation, they are unable to determine how to navigate toward the goal, and exhibit meainingless bee-lining behavior. Using a graph search to find a minimum distance path, ViNG can reach the goal, but does not satisfy the reward function. Only ReViND is successful in navigating to the goal while taking a short detour that maximizes the desired objective, demonstrating affinity to grassy terrains.
We provide quantitative evaluation of these methods in Table 2 and Appendix D, showing the average distance traveled by each method over all test trials prior to disengagement, as well as the average value of the utilities. We see that non-RI. methods are unable to take into account the task reward, and simply aim to reach the task goal, which leads to suboptimal utility. We can take reward into account either using RL, or by filtering BC imitate only the high-reward trajectories. In easier environments, we see that both fBC and TQL. can learn reward-maximizing behavior. However, both the RL and BC flat policies suffer sharp drops in performance as the distance to goal increases. The addition of graph greatly helps improve performance. Here again, we notice that offline RL (ReViND), which uses Q-learning to optimize the reward, consistently outperforms filtering- based approaches (fBC-graph) - this confirms that reward information is important for respecting the user's preferences, and that offline RL is more effective at this than filtering.
The biggest failure mode for current offline RL and IL methods in our task is their inability reach distant goals. BC, IBC and IQL consistently fail to reach goals beyond 15-20m away, due to chal- lenges in learning a useful policy from offline data- - these flat baseline policies often demonstrate bee-lining behavior, driving straight to the goal, which often leads to collisions.
5 Discussion
We presented ReViND, a robotic navigation system that uses offline reinforcement learning in com- bination with graph search to reach distant goals while optimizing user preferences. We showed that ReViND can be trained on navigational dataset collected prior work, in combination with reward labeling. to exhibit qualitatively distinct behaviors. Our experiments show that Re ViND can generalize to novel, visually similar environments, and is responsive the user preferences, signif- icantly outperforming prior methods that either do not utilize high-level planning, utilize graphs without RL and therefore do not support reward specification. We hope that our work will provide a step towards robotic learning methods that routinely reuse existing data, while still accomplishing new tasks and optimizing user preferences. Such methods can exhibit effective generalization in the real world through their ability to incorporate existing diverse datasets, while also flexibly solving new tasks, so long as the specified reward functions are valid in the novel environments and tasks.
Limitations: While ViND supports variety of reward functions, it has number of limitations. First, all of the data must be labeled with the specified reward function. While this can even be done manually (since the method is fully offline), in practice, it is much more practical to implement the reward function programmatically and extending our system to incorporate autonomous labeling with a diverse family of rewards would be beneficial. should also be noted that the current system relies heavily on the specified reward function being valid across the environments, which can be challenging to achieve in practice when scaling up more diverse environments (e.g. "grassy" may mean different things in different places, and the level of "sunniness" varies with the season). In future work, accommodating preferences, textual commands, or other more natural modes of specifying the utility function would make the method more practical. Second, the behavior learned by ReViND does depend critically on the data that is provided. For example, if all provided data drives on roads, then it is impossible for the policy to learn how to drive off-road, since the relevant experience is simply unavailable, Combining Re ViND with online exploration and finetuning may address this limitation in the future. Lastly, we evaluate ReViND in settings where the graph is built from previously seen experience. An exciting direction for future work would be to combine ReViND with method for exploring a new environment to search for goal, while maximizing the user-specified reward, for example by constructing the graph on-the-fly [9].
8

---

References
[1] E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa, D. Parikh, M. Savva, and D. Batra. DD- PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames. In International Conference on Learning Representations (ICLR), 2020. 1,
[2] A. Kadian, J. Truong, A. Gokaslan, A. Clegg, E. Wijmans, S. Lee, M. Savva, S. Chernova, and
D. Batra. Sim2Real Predictivity: Does Evaluation in Simulation Predict Real-Worke Perfor- mance? IEEE Robotics and Automation Letters, 2020.
[3] Levine, A. Kumar, G. Tucker, and J. Fu. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems, 2020. 1,3,4
[4] Kaplan, McCandlish, T. Henighan, B. Brown, B. Chess, R. Child, S. Gray, A. Radford, Wu, and Amodei. Scaling laws for neural language models. CoRR, abs/2001.086 008361, 2020. URL https: //txxvv/./iivvv/aa org/ aba/2001.1100 08361. 1
[5] M. Andrychowicz, Wolski, Ray, J. Schneider, R. Fong, P. Welinder, McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gareett, editors, Advances in Neural Information Processing Systems, 2017. 1,4
[6] B. Eysenbach, R. R. Salakhutdinov, and S. Levine. Search the Replay Buffer: Bridging Planning and RL. In Advances in Neural Information Processing Systems (NeurIPS). 2019. 3,
4
[7] D. Shah, B. Eysenbach, G. Kahn, N. Rhinehart, and S. Levine. ViNG:
Learning Open- Open-World Navigation with Visual Goals. In IEEE International Conference on Robotics and Automation (ICRA), 2021. 1, 2, 3, 7,4
[8] I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. arXiv preprini 1 2021. 1, 4,7
[9] D. Shah, B. Eysenbach, N. Rhinehart, and S. Levine. Rapid exploration for open-worid navi- gation with latent goal models. In 5th Annual Conference on Robot Learning, 2021. 2, 4, 6, 8,
[10] S. Thrun, W. Burgard, and D. Fox. Probabilistic Robotics. The MIT Press, 2005. ISBN 0262201623. 2
[11] Bourgault, A. A. Makarenko, S. B. Williams, B. Grocholsky, and H. Durrant- Whyte. Information based adaptive robotic exploration. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), volume pages 540-545 vol.1, 2002. doi:10.1119/ 2
[12] Kollar and N. Roy. Efficient optimization of information-theoreti exploration in slam. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2008
[13] W. Tabib, K. Goel, J. Yao, M. Dabhi, C. Boirum, and N. Michael. Real-time information- theoretic exploration with gaussian mixture model maps. In Robotics: Science and Systems, 2019.
[14] B. Yamauchi. A frontier-based approach for autonomous exploration. In IEEE International Symposiun Computational Intelligence in Robotics and Automation (CIRA), 1997.
[15] D. Holz, N. Basilico, F. Amigoni, and S. Behnke. A comparative evaluation of exploration strategies and heuristics to improve them, 2011.
[16] B. Charrow, S. Liu, V. Kumar, and N. Michael. Information-thhorret mapping using cauchy- schwarz quadratic mutual information. In IEEE International Conference Roboties and Automation (ICRA), 2015.

---

[17] B. Charrow, G. Kahn, S. Patil, S. Liu. K. Goldberg, P. Abbeel, N. Michael, and V. Kumar. Information-theoneiicc planning with trajectory optimization for dense 3d mapping. In Robotics: Science and Systems (RSS), 2015.
[18] McCormac, A. Handa, A. Davison, and S. Leutenegger. Semanticfusion: Dense 3d seman- tic mapping with convolutional neural networks. In 2017 IEEE International Conference on Robotics and Automation (ICRA), 2017.
[19] K. M. Jatavallabhula, G. Iyer, and L. Paull. gradslam: Dense SLAM meets automatic differen- tiation. CoRR, 2019.
[20] C. Connolly. The determination of next best views. Proceedings. 1985 IEEE International Conference Robotics and Automation, 1985.
[21] C. Papachristos, S. Khattak, and K. Alexis. Uncertainty-aware receding horizon exploration and mapping using aerial robots. In 2017 IEEE International Conference Roboties and Automation (ICRA), 2017.
[22] M. Selin, M. Tiger, D. Duberg, F. Heintz, and P. Jensfelt. Efficient autonomous exploration planning of lcage-scale 3-d environments. IEEE Robotics and Automation Letters, 2019. 2
[23] J. Delmerico, E. Mueggler, J. Nitsch, and D. Scaramuzza. Active autonomous aerial explo-
ration for ground robot path planning. IEEE Robotics and Automation Letters, 2017. 2
[24] I. Lluvia, E. Lazkano, and A. Ansuategi. Active mapping and robot exploration: A survey. Sensors, 2021. 2
[25] Manolis Savva*, Abhishek Kadian*, Oleksandr Maksymets*, Y. Zhao, E. Wijmans, B. Jain, Straub, J. Liu, Koltun, J. Malik, D. Parikh, and D. Batra. Habitat: Platform for Embod- ied AI Research, In IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
2
[26] D. Chaplot, Gandhi, S. Gupta, A. Gupta, and R. Salakhutdinov. Learning Explore using Active Neural SLAM. In International Conference Learning Representations (ICLR), 2020.
2
[27] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi. Al2-THOR: An Interactive 3D Environment for Visual AL. ArXiv, 2019.
[28] P. Anderson, A. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, J. Kosecka, J. Ma- lik, R. Mottaghi, M. Savva, and A. R. Zamir. On Evaluation of Embodied Navigation Agents, 2018.
[29] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. In International Conference on Machine Learning. pages
PMLR, 2019.
3
[30] Lange, Gabel, and M. Riedmiller. Batch reinforcement learning. In Reinforcement learn- ing. pages 45-73. Springer, 2012.
[31] A. Nair, A. Gupta, M. Dalal, and S. Levine, Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint 200::20006003399 2020.
[32] A. Kumar, Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforce- ment learning. Advances in Neural Information Processing Systems, 33:1179-1191, 2020.
4
10

---

[33] I. Kostrikov, R. Fergus, J. Tompson, and O. Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pages 5774-5783. PMLR, 2021. 3
[34] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXiv:2104.082 /2, 2021.
[35] A. Singh, A. Yu, J. Yang. J. Zhang, A. Kumar, and Levine. Cog: Connecting new skills past experience with offline reinforcement learning. arXiv preprint arXiv:2010.14500, 2020.
[36] A. Mandlekar, F. Ramos, B. Boots, Savarese, L. Fei-Fei, Garg, and D. Fox. Iris: Implicit reinforcement without interaction scale for learning control from offline robot manipulation data. In 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020.
[37] A. X. Lee, C. M. Devin, Y. Zhou, T. Lampe, K. Bousmalis, J. T. Springenberg. A. Byravan, A. Abdolmalcki, N. Gileadi, D. Khosid, et al. Beyond pick-and-place: Tackling robotic stack- ing of diverse shapes. In 5th Annual Conference on Robot Learning, 2021.
[38] Y. Chebotar, K. Hausman, Y. Lu, T. Xiao, D. Kalashnikov, J. Varley, A. Irpan, B. Eysenbach,
R. Julian, C. Finn, and S. Levine. Actionable models: Unsupervised offline reinforcement learning of robotic skills. arXiv preprint arXiv:2104.07749, 2021. 3,
[39] G. Kahn, A. Villaflor, B. Ding, P. Abbeel, and S. Levine. Self-Superviser Deep RL with Generalized Computation Graphs for Robot Navigation. In IEEE International Conference on Robotics and Automation (ICRA), 2018. 3
[40] G. Kahn, A. Villaflor, P. Abbeel, and S. Levine. Composable action-conditioned predictors: Flexible off-policy learning for robot navigation. In Conference on Robot Learning, 2018. 3
[41] N. Savinov, Dosovitskiy, and Koltun. Semi-Parametric Topological Memory for Naviga- tion. In International Conference Learning Representations, 2018. 3
[42] A. Faust, K. Oslund, O. Ramirez, A. Francis, L. Tapia, M. Fiser, and J. Davidson. PRM-RL: Long-range Robotic Navigation Tasks by Combining Reinforcement Learning and Sampling- Based Plamning. In IEEE Int'l Conference on Robotics and Automation (ICRA), 2018. 3
[43] S. Nasiriany, V. Pong, S. Lin, and S. Levine. Planning with goal-conditioned policies. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2019.
[44] L. P. Kaelbling, Learning to achieve goals. In IJCAI, pages 1094-1099. Citeseer, 1993. 4
[45] B. Eysenbach, Geng, Levine, and Salakhutdinov. Rewriting history with inverse rl: Hindsight inference for policy improvement. Advances neural information processing systems, 33:14783-14795, 2020.
[46] Codevilla, M. Müller, A. López, V. Koltun, and A. Dosovitskiy. End-to-End Driving Via Conditional Imitation Learning. IEEE International Conference on Robotics and Automa- tion (ICRA), 2018.7 7
[47] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Ad- vances in Neural Information Processing Systems, 2021. 7
[48] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, S. Legg, and K. Kavukcuoglu. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In ICMI, 2018.


Reviews:
[{"review_id": "iVOkX1X_vYE", "reviewer": "Reviewer_qBaA", "summary": "he authors consider the problem of autonomous robot visual navigation over long horizons while obeying hand-specified reward functions to achieve additional behaviors like \"stay in sunny areas while navigating.\" To achieve this, the authors leverage recent methods in offline reinforcement learning to learn value functions for short-horizon goals, and use a graph search algorithm to reason over the remaining portion of the planning horizon. Impressive experimental results showing the efficacy of the approach are presented. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "(+) Broadly speaking, this is an extremely interesting problem space, and it's very interesting to see the authors bring offline RL tools to bear on the particular problem of long-horizon visual navigation. \n\n(+) The experimental work presented here is extremely impressive, with the authors having used the proposed method to train a robot to perform the desired behaviors in outdoor deployments over distances on the order of kilometers. \n\n(+) The discussion about the authors experience with visual navigation relating to non-Markovian state and the short-horizon limitation of value function learning was very much appreciated. \n\n \n\n(-) Some discussion that might be critical to reproducibility is missing. In particular, because the problem being considered is one of combining two objectives (ie, the navigation task and the secondary task that the authors refer to as maximizing a \"utility measure\"), it strikes me that the problem of how to \"balance\" these two objectives at deployment time is of critical importance. The paper is careful to specify that the utility measure must be specified manually by a human, but I didn't find any discussion of how these two objectives are, eg, weighted relative to one another. \n\n(-) To my read, it seems as though the authors are implying that the value functions learned using ORL with the manually-specified utility measure are potentially transferable to new environments (eg, the discussion at the end of 3.1). But it doesn't seem to me that the experiments actually support this claim as it appears that each of the experiments is done in the same environment as the offline data set was collected. I do note that there is some discussion related to this in the \"limitations\" section, but it doesn't seem to address this concern in particular.  \n\n(-) As a minor comment, the paper in its submitted form still requires a few polishing tasks such as removing comments between authors (blue text on L43) and missing references (L85-86). \n\n(-) As a minor comment, the text/numbers on the left hand side of Figure 3 were not readable for me. \n\n(-) As a minor comment, L246 mentions something about driving \"on loans,\" which seems like a typo but I'm not sure what was meant (\"roads?\"). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper synthesizes several important topics in the robot learning literature, including offline reinforcement learning and long-horizon visual navigation and describes an interesting and potentially impactful method for incorporating manually-specified utility measures into the navigation task using only pre-collected data. That said, there are some shortcomings in the current draft including important but missing discussion regarding algorithmic details and limitations.\n\nPOST-DISCUSSION UPDATE: The authors adequately addressed the issues I raised during the discussion phase, and therefore I'm raising the score of my recommendation.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "(see weaknesses above)", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "JH9BQZeQLk9", "reviewer": "Reviewer_6zkp", "summary": "The paper studies how offline RL can be applied to long-horizon navigation tasks with different user preferences. The method leverages graph-based methods to search for distant goals and combines it with offline RL to navigate locally. The authors claim that the proposed method can learn from public offline datasets and generalize to novel (unseen) environments in zero-shot.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n* The paper tries to address a very important and challenging navigation problem. \n* The experimental results seem very promising in that the method could learn in zero-shot from publicly available datasets. However, there are some concerns regarding this (please see the weaknesses below). \n\nWeaknesses: \n* The paper claims that by relabeling large-scale offline datasets, one can expect effective generalization in real world even without the need for any task specific data collection. The provided experimental results also support this claim. However, the paper does not study what conditions are required to make this generalization feasible. In general, one cannot expect such zero-shot generalization to happen, especially using public datasets collected by someone else in completely different environments. The paper needs to provide more details why this generalization is possible. Then, explicitly describe the limitation of the method, i.e., conditions in which the method would fail.  \n* One contribution of the paper is to learn different behaviors from an offline dataset. However, this is done by re-labeling the transitions in the offline dataset by defining a reward function. There are two issues here: (1) defining a reward function, especially to label image observations, is challenging and sometimes require a lot of manual work, and (2) relabeling is a standard approach and cannot be claimed as a contribution here. \n* The theoretical contribution of this paper is very limited. The method resembles SoRB with two differences: (1) formulating the problem as an offline RL, and (2) re-labeling the offline data with different reward functions. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The most important concern is that the paper shows zero-shot generalization for a very challenging problem without clearly describing what makes this generalization feasible. It is possible that the method is heavily engineered to work only on this particular setting, and does not scale well to other similar problem settings. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "* Some references are missing. See lines 43, 86, 87. \n* Some details of the IQL method are missing. As an example, $\\tau$ is not defined. Besides, since the method is not constructed on top of IQL (it is only used as a tool for offline training), there is no need to describe IQL. I suggest to move section 3.2 to appendix. "}}, {"review_id": "AYEjE42FK-S", "reviewer": "Reviewer_FZSz", "summary": "This paper looks at a combination of offline reinforcement learning and planning, in order to learn from offline data while being able to relabel the reward. Specifically, implicit q-learning is used to learn a Q/value function estimator, from purely offline data, and a topological map is built where nodes are sampled observations and edge costs come from the value function. Results are shown on a robotics domain where the value functions are learned on observed robot trajectories and task-relabeled (e.g. preferring particular types of terrain such as grass) with results then shown on an urban environment. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n ++ Overall, methods utilizing offline data are extremely relevant, as abundant data can be gathered or crawled rather than the robot having to perform exploration or roll-outs, which takes significant time and could be unsafe. \n\n  ++ I really like the idea of combining learning (value functions) and planning\n\n  ++ The paper is understandable and well-written, though could use significant polish (see below)\n\n  ++ The results shown seem to show significant gains \n\nWeaknesses\n\n -- Overall, the paper could use significant polish including:\n   * Figures 1 and 4 are very blurry and should use vector graphics (or at least higher resolution images)\n   * Comments are left in, e.g. L43 \"scaling law papers?\", missing citations L86/L87 \"[]\", etc.\n   * Several grammatical issues:\n     * L4: \"utilize previously collect data\"\n     * L105: \"we specifically exploring\"\n  * The \"Discussion\" is not actually a discussion - it's a typical conclusion that just reiterates what's in the paper, not a synthesis\n\n -- The motivation for proposition 3.1 is both natural but also against the stated goal of tackling long-horizon situations which violate assumptions like the Markovian property; yes, planning along the value function-based graph will obviously recover the optimal policy but 1) that requires discretization (the topological map), 2) requires a lot of assumptions, and 3) doesn't happen when things are not Markovian. It doesn't seem to answer anything about why this method might be better than just getting the policy itself through RL (I'm not saying it's not better, just that it recovers the \"optimal solution\" under unlimited data). \n\n-- \"But this assumption becomes increasingly violated for long horizon tasks with first-person images: while goals that within line of sight of the robot are relatively simple, goals that require navigating around obstacles tend to fail if using the policy directly\". This is a (pretty specific) hypothesis yet no evidence for this claim is made. Overall the state transition still seems Markovian here, but even if not it's not clear that this matters and is the reason that the proposed method is better. \n\n -- Overall, while some citations are made to the goal-conditioned learning+planning literature, there is not a significant positioning of this paper and that field (see [A] and many papers that cite it). It is a large literature, and merely saying a few short sentences about how related work differs (actually there is not a strong statement I can see in how this paper differs from the goal-conditioned RL-based planning literature). Similarly there is a rich history of topological maps in robotics [B] and while obviously those were pre-deep learning/RL resurgence, were there any prior topological mapping papers that derived their edge costs from some learned functions (RL or otherwise)?\n\n  -- Similarly, there are a few experiments on a custom platform/experiments and the authors show that their method is better than some sampling of the literature; it's not clear *why* the method works better in this case nor *why* those other methods fail. Indeed, the setup is not that complicated in terms of task, so it's not clear to me why the other methods fail. It is not sufficient to just show the results without any significant analysis (ideally tying back to why the authors hypothesized it would do better). \n\n  -- Overall, there is significant downside in that the topological map has to be pre-built in the new environment - this is akin to exploring the new environment. While no learning is done with this data, it still has access to it, so it's not clear that it's a fair comparison with other methods that don't utilize such data. For example there are a lot of works in the vision-language navigation literature that do pre-exploration [C], and perhaps such works should be compared to. \n\n[A] Nasiriany, Soroush, et al. \"Planning with goal-conditioned policies.\" Advances in Neural Information Processing Systems 32 (2019).\n[B] Kuipers, Benjamin, et al. \"Local metrical and global topological maps in the hybrid spatial semantic hierarchy.\" IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA'04. 2004. Vol. 5. IEEE, 2004.\n[C] Tan, Hao, Licheng Yu, and Mohit Bansal. \"Learning to navigate unseen environments: Back translation with environmental dropout.\" arXiv preprint arXiv:1904.04195 (2019).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, I think this paper is promising in terms of an interesting idea for carrying out this task, using only offline data to train on; however, as it reads currently it presents a (pretty small set) of hypotheses with what's wrong with existing methods, how this paper is situated with respect to them, and why it works better. Since this is done on a completely custom environment, the paper needs significant additions in terms of why it works well, linking it back to claims made in the beginning. \n\n=== Post-rebuttal\n\nThank you to the authors for providing additional explanations and experiments. Overall, the authors do address some of my concerns well. However, I am still not fully convinced by the situation of this work w.r.t. prior works on toplogical mapping, or by the experiments which use only limited custom datasets. On the other hand, I believe that the paper does provide an interesting perspective that combines topological mapping with offline RL; therefore, while the paper is not perfect, I think it is high-quality and valuable to be published as a poster. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- Fix polish issues method\n- Provide a much better positioning of how this method compares to the prior state of art (including goal-conditioned and topological mapping literature)\n- Provide analysis/explanation as to why the method works better and prior methods fail -- this is presently unclear\n- Explain why the comparison is fair even though pre-exploration is done in the new environment to build the topological map"}}, {"review_id": "9e5_aUYbR6Zv", "reviewer": "Reviewer_V1xj", "summary": "This paper presents a system, ReViND, enabling a robot to learn how to navigate based on existing data and incorporate various types of utility beyond path length. The claimed benefits of ReViND are that it is lightweight with respect to additional data collection, able to incorporate utility other than minimum path length (e.g. need for solar power), and can generate viable trajectories at longer horizons than prior work. \n\nReViND is as follows: existing data is relabeled with custom reward, in this case procedurally, and a policy and Q function are trained on this data using IQL. A topological graph is then constructed by doing \"drive-throughs\" of the environment and creating a graph with drive-through observations as nodes and Q function values to weight edges. After training, a robot will receive a goal, find the shortest path using Dijkstra's on the graph, then follow the path using the policy.\n\nThe data is tested in five environments with two additional utilities. \n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths** \n- Thoughtfully-designed system that aims for and incorporates many important requirements and constraints, including impracticality of custom data collection and real-time responsiveness (as needed in online RL), various needs from a single algorithm, and fragility of purely independent offline RL\n- Simple and clear solutions to the problems outlined\n- Benefits of the topological graph in particular are demonstrated convincingly through simple but targeted results. More broadly, the approach to evaluation is reasonable and intuitive. \n- The paper is very well-written. After the initial readthrough, the structure and details are clear. The paper structure makes sense and the writing is easy to follow. \n- Overall, a great project setup\n\n**Weaknesses**\n- Paper claims that users can use ReViND \"without collecting any of their own data\". I agree that the use of offline RL is a huge benefit that probably makes this system usable whereas an online approach would not be. However, the paper's language strongly suggests that not needing to collect *any* data is a primary benefit, but the drive-throughs and relabeling are still required. It's possible that drive-throughs are feasible and negligible compared to the data needed for offline RL, so this might be a nitpick, but it's not so clear with relabeling. The procedural relabeling seems useful, but anything more sophisticated and tunable for the user's needs would require either hand relabeling (very expensive for a large offline dataset) or a reward model, which would require data of its own that may need to be custom to be domain-adapted. Given that adaptability is another main claimed benefit, there seems to be a trade-off without a solution. \n- Paper claims that ReViND is highly adaptable beyond minimum-length path optimization. I'm convinced that adding in another loss term procedurally based on an additional reward annotation works, but that's not just because of the experimental results, it's also because that's very well understood. Adding loss terms together that come from ground-truth reward annotations is very standard. Therefore, the claimed flexibility of the system doesn't strike me as novel. \n- Furthermore, utility in the sense of balancing sun for solar power and least distance for efficiency, or grassyness and least distance, or any other such example, presents a tradeoff. How the degree of tradeoff should be decided depends on circumstances specific to the situation, which the training data may or may not contain. Even in the best case where it does, the single offline RL-trained algorithm will make decisions in a particular way, and it's important for the user to at least understand how tradeoffs are being made. I would therefore appreciate examples in goal cases where there are different degrees of tradeoff between e.g. sun exposure and least distance, to understand how the trained algorithm fares. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, I think that this paper is clean, elegant, and useful. My main concerns are about significance, because it has three main claims of uniqueness - no need for user data, which seems to be false (but it's certainly true that the offline approach drastically reduces the need for training data), long horizon capability (useful and convincingly presented), and ability to include custom utility (unclear how much this is in the user's control, and the method is not only unsurprising but completely standard machine learning). However, since it is a low-data useful system that can handle real-world-scale navigation problems, I do think it's a solid applied paper. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "More thorough experimentation on utility, and clarification of the data claims. ", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (oral)