Model-Based Visual Planning with Self-Supervised Functional Distances
Authors: Stephen Tian
Suraj Nair
Frederik Ebert
Sudeep Dasari
Benjamin Eysenbach
Chelsea Finn
Sergey Levine
Venue: ICLR 2021 Spotlight (2021)
Keywords: planning
model learning
distance learning
reinforcement learning
robotics

Abstract: A generalist robot must be able to complete a variety of tasks in its environment. One appealing way to specify each task is in terms of a goal observation. However, learning goal-reaching policies with reinforcement learning remains a challenging problem, particularly when hand-engineered reward functions are not available. Learned dynamics models are a promising approach for learning about the environment without rewards or task-directed data, but planning to reach goals with such a model requires a notion of functional similarity between observations and goal states. We present a self-supervised method for model-based visual goal reaching, which uses both a visual dynamics model as well as a dynamical distance function learned using model-free reinforcement learning. Our approach learns entirely using offline, unlabeled data, making it practical to scale to large and diverse datasets. In our experiments, we find that our method can successfully learn models that perform a variety of tasks at test-time, moving objects amid distractors with a simulated robotic arm and even learning to open and close a drawer using a real-world robot. In comparisons, we find that this approach substantially outperforms both model-free and model-based prior methods.

Published as a conference paper at ICLR 2021
MODEL-BASED VISUAL PLANNING WITH SELF-SUPERVISED FUNCTIONAL DISTANCES
Stephen Tian', Suraj Nair?, Frederik Ebert', Sudeep Dasari', Benjamin Eysenbach",
Chelsea Finn", Sergey Levine'
'University of California, Berkeley
Stanford University
CCamegie Mellon University
ABSTRACT
A generalist robot must be able to complete a variety of tasks in its environ- ment. One appealing way to specify each task is in terms of a goal observa- tion. However, learning goal-reaching policies with reinforcement learning re- mains a challenging problem, particularly when hand-engineered reward fune- tions are not available. Learned dynamics models are promising approach for learning about the environment without rewards or task-directed data, but plan- ning to reach goals with such model requires notion of functional similarity between observations and goal states. We present a self-supervised method for model-based visual goal reaching, which uses both a visual dynamics model well as a dynamical distance function learned using model-free reinforcement learning. Our approach learns entirely using offline, unlabeled data, making it practical to scale to large and diverse datasets. In our experiments, we find that our method can successfully learn models that perform variety of tasks at test-time, moving objects amid distractors with a simulated robotic arm and even learning open and close drawer using a real-world robot. In com- parisons, we find that this approach substantially outperforms both model-fie free and model-based prior methods. Videos and visualizations are available here: https:/ // s tes. goog e. www/borkeeeeeee..o www.rreeeee.edemmeeeeeee
INTRODUCTION
Designing general-purpose robots that can perform wide range of tasks remains an open problem in Al and robotics. Reinforcement learning (RL.) represents a particularly promising tool for learn- ing robotic behaviors when skills can be learned one at time from user-defined reward functionn. However, general-purposs robots will likely require large and diverse repertoires of skills, and learn- ing individual tasks one at a time from manually-specifed rewards is onerous and conssminnn How can we design learning systems that can autonomously acquire general-purposs knowledge that allows them to solve many different downstream tasks?
To address this problem, we must resolve three questions. (1) How can the robot be commanded to perform specific downstream tasks? A simple and versatile choice to define tasks terms of desired outcomes, such
example observation of the completed task. (2) What types of data should this robot learn from? In settings where modern machine learning attains the best general- ization results (Deng et al.) 2009, Rajpurkar et al. 2016: Devlin et al. 2018). common theme is that excellent generalization is achieved by learning from large and diverse mask-agnostic datasets. In the context of RL, this means need offtine methods that can all sources of prior data, in the absence of reward labels. As collecting new experience on a physical robot is often expen- sive, offline data is often more practical use in real-world settings (Levine et al., 2020). (3) What should the robot learn from this data to enable goal-reaching? Similar to prior work (Botvinick & Weinstein 2014 Watter et 2015 Finn & Levine 2017 [bert et al., 20188), we note that policies and value functions are specific to particular task, while predictive model captures the physics of the environment independently of the task, and thus can be used for solving almost any task. This makes model learning particularly effective for learning from large and diverse datasets, which do not necessarily contain successful behaviors.

---

Published as a conference paper at ICLR 2021
While model-based approaches have demonstrated promising results, including for vision-based tasks in real-world robotic systems (Ebert et al.) 2018a; Finn & Levine, 2017), such methods face two major challenges. First, predictive models on raw images tive over short horizons, as uncertainty accumulates far into the future (Denton & Fergus 2018;; Finn et al., 2 Hafner et al. 2019b] Babaeizadeh et al. (017]. Second, using such models for planning toward goals requires a notion of similarity between images. While prior methods have utilized latent variable models (Watter et al. 2015 Nair et al. 2018), f2 pixel-space distance (Nair & Finn 2020), and other heuristic measures of similarity (Ebert ct al. 2018b), these metries only capture visual similar- ity. To enable reliable control with predictive models, we instead need distances that aware of dynamics.
Legend:
SteriCosd Slewss
Prededed Dyseent
a Flarning States
Shorluui Puth is Goal
only effec-
Start
MBOLD leams Gynanics and divtancen
Figure I: The robot must find actions that quickly achieve the desired goal. State tran- sitions and the true optimal distances be- tween states are unknown, so method learns approximate shortest distance function and dynamics model directly on images. These models allow the robot to find the shortest path to the goal at test-time.
In this paper, we propose Model-Based RL with Of- fline Learned Distances (MBOLD), which aims ad- dress both of these challenges by learning predictive models together with image-based distance functions that reflect functionality, from offline, unlabeled data. The learned distance function es- timates of the number of steps that the optimal policy would take to transition from one state to another, incorporating not just visual appearance, but also an understanding of dynamics. However, to learn dynamical distances from task-agnostic data, supervised regression will lead to overestima- tion, since the paths in the data are not all optimal for any task. Instead, we utilize approximate dynamic programming for distance estimation. While prior work has studied such methods to learn goal-conditioned policies in online model-free RL settings (Eysenbach et al. 2019, Florensa et al.) 2019), we extend it to the offline setting and show that approximate dynamic programming tech- niques derived from Q-learning style Bellman updates can learn effective shortest path dynamical distances. Although this procedure resembles model-free reinforcement learning, we find empiri- cally that it does not by itself produce useful policies. Instead, our method (Fig. [ combines the strengths of dynamics models and distance functions, using the predictive model to plan over short horizons, and using the learned distances to provide a global cost that captures progress toward distant goals.
The primary contribution of this work is an offline, self-supervised approach for solving arbitrary goal-reaching tasks by combining planning with predictive models and learned dynamical distances. To our knowledge, our method is the first to directly combine predictive models on images with dy- namical distance estimators on images, entirely from random, offline data without reward labels. Through our experimental evaluation on challenging robotic object manipulation tasks, including simulated object relocation and real-world drawer manipulation, we find that our method can out- perform previously introduced reward specification methods for visual model-based control with relative performance improvement of at least 50% across tasks, and compares favorably to prior work in model-basee and model-free RL. We also find that combining Q-functions with planning improves dramatically over policies directly learned with model-frec RL.
RELATED WORK
Offline and Model-based RL: A number of prior works have studied the problem of learning behav-
iors from existing offline datasets. While recent progress has been made in applying model-free RL techniques to this problem of offline or batch RL (Fujimoto et al., 2019 Wu et al.) 2019 Kumar
af. 2019: 2020 Nair et al. (2020b), one approach that has shown promise is offline model-based RL (Lowrey et al. 2019; Kidambi et al.) 2020 Yu et al.) 2020 Argenson & Dulac-Arnold 2020).
where the agent learns a predictive model of the world from data. Such model-based methods have seen success both in the offline and online RL settings. and have rich history of being effective
for planning (Deisenroth & Rasmussen 2011] Watter et al. al.] 2013) McAllister & Rasmussen 2016
Chua et al. 2018 Amos et al. [0188] Hafner et al. [009bb] Nagabandi et al. |018| Kahn al. 2020
Dong et al. (020) or policy optimization (Sutton 1991 Weber et al. 2017. Ha & Schmidhuber
2018 Janner et al. 2019 Wang & Ba 2019
Hafner et al. 2019a). However, the vast majority of
2

---

Published as a conference paper at ICLR 2021
these prior works consider the single task setting where the agent aims to maximize single task reward. In contrast, in this work we circumvent the need for task rewards by adopting a self- supervised multi-task approach, where a single learned model is used to perform variety of tasks, specified in a flexible and general way by desired outcomes - i.e., goal images.
Self-supervised goal reaching: While the standard RL problem involves optimizing for task- specific reward, an alternative and potentially more general formulation involves learning a generic goal reaching policy, without task-specific reward labels. In fact, a number of prior works learn goal-conditioned policies using model-free RL (Kaelbling, 1993 Nair et al., 2018 Mandlekar et al.
2019 Nair et al. 2020a), or variants of goal-conditioned behavioral cloning (GCBC) (Ghosh al,, Ding et al. [ Lynch et al. 22220). In our experiments, we show that our method outper- forms both mrode--free approaches and goal-conditioned behavioral cloning. A number of methods combine model-free and model-basee elements by planning over a graph representation (Eysenbach) al.
2019
2019: Nasiriany et al., 2019] [Savinov et al., 2018: Liu et al. 2020). Such methods can strug- gle in higher dimensions, where constructing graphs that adequately cover the space may require an excessive number of samples. We compare to these methods in our experiments. Similarly to
Finn & Levine (2017); Ebert et al. (2018b); Nair & Finn (020); Yen-Chen et al. (2019): /Suh & video prediction model to generate plans. However, these prior methods generally utilize hand-crafted image similarity reward measures such as l2 pixel-error (Ebert et al.) 201 Sa Nair & Finn, 2020) and pixel-flow prediction (Finn & Levine] 2017). In complex scenes, this can become a major bottleneck: predictions degrade rapidly further in the future, making an informative image similarity metric critical for effective planning. We pro- pose learn functional similarity metries in terms of dynamical distances, which we find can be combined with predictive models to attain significantly improved results.
Tedrake (2020), our method uses an cectin-connitiions
Dynamical distance learning: Our method learns dynamical distances distances that represent shortest paths from offline data. In the literature, dynamical distances have been learned via direct regression using online data (Hartikainen et al. 2019), representation learning (Warde-Farley 2018 Yu et al. 2019b), or via Q-learning by relabeling goals (Eysenbach et al.) 2019] et al., 2019 J. While these last two works are most similar to ours, in that they also employ approximate dynamic programming to learn distances, our method directly combines these dynamical distances with visual predictive models and planning. Lastly, while prior work has also explored combining model-based planning with value functions (Zhong et al. 2013) Lowrey et al. 2019 Hafner et al., 2019a] Schrittwieser et al. (019:/rreesoo & Dulac-Arnold] (0200), these works consider the single
al,,
task domain with reward function, while our learned value function considers the multi-task goal reaching domain from entirely random, offline data without reward labels.
3 THE SELE-SUPERVISED OFFLINE RL PROBLEM STATEMENT
In this section, we introduce notation and define the problem setting. We will employ a Markov decision process (MDP) with state observations s; E and actions a, E A. both indexed by time t E 0, H. where H denotes the maximum episode length. The initial state is sampled from an initial state distribution So ~ Po(ss). and subsequent states are sampled according to Markovian dynamics: St+1 ~ p(st+1 | st, 0t). Actions are sampled a ~ <<a St, from a policy that is conditioned on both the current state and goal state 8, E S. In our experiments, both the state and goal are images (i.e.. S 3xxx33).
We tackle offline learning in this setting, assuming access to a fixed dataset DD consisting of tra- jectories {so, dg 81, ST of the agent interacting with the environment. This data can include any environment interactions, from expert demonstrations to trajectories which are not particularly successful at any task. In our experiments, we use data collected using a random policy, which is inexpensive to obtain. The agent does not have access to the environment to collect additional training data. Given this dataset, the objective is to determine the optimal goal-conditioned policy r" ra(a $2 Sg). under which the agent is able to transition to any goal state sy from any starting state s, in the minimum number of time steps possible. Note that unlike in the standard formulation of the RL problem, the agent does not receive any reward signal from its environment.
3

---

Published as a conference paper at ICLR 2021
Offline Learning
2. Sampling Based Planning
On
TeLL
Distance
Lowest Distanoel
Predictin
Locs
GlLeaming cois
Rres
Forward
Distance Learning
Legand:
Modelling
ph trspcttry' Action at time
Formard Modal
Figure 2: Model-based visual goal reaching: (Left) During offline learning, we train an image- based predictive model and distance function on the same random dataset. (Right) At test time, we use the learned distance model for MPC, plugging in the learned distance as a cost function.
MODEL-BASED VISUAL GOAL-REACHING
In this section we will introduce our method, MBOLD, for offline, goal-conditioned reinforcement learning. MBOLD, illustrated in Fig. 2 is composed of two neural networks: a predictive model and a learned distance function. The video-predictive dynamics model allows the agent to predict the result of hypothetical sequences of actions. However, this model cannot accurately predict far into the future, and has no notion of whether the predicted outcomes are desirable. Thus, we also learn distance function, corresponding to a value function with self-supervised goal-reaching reward, which will estimate the timestep length of the shortest path between predicted state and given goal. Both networks are trained on the same offline dataset.
At test-time, we use the learned dynamics model and distance function for model preeiccpre control (MPC). MBOLD predicts future states for candidate action sequences using the learned dynamics model, and uses the learned distance function to determine which action sequence will lead the agent closest to the goal. The first of the actions is then executed, and planning repeats upon receiving the subsequent observation from the environment. The remainder of this section describes how learn the dynamics model and distance function, and use them to perform control.
Dynamics learning. Our method learns environment dynamics in order to solve for actions during test time, without an explicit task reward signal during training. MBOLD can use arbitrary image- based forward models, including latent variable models (Hafner al.) 2019b Lee al.) 2019). The particular choice of model a design decision when implementing our method. In our implemen- tation,
convolutional video prediction model adapted from SAVP (Lee et al.) 2018). The network takes as input the current observation and a sequence of h actions ar.+h-1 and returns prediction for the next image observations, fo(st, dma4h-1) {8r+1 Se+A). ). We train this model minimize the & image reconstruction loss:
1+h Σ
min Ep
fl (e((s)))ec - -t]- - srr
(I)
Distance learning. Our method also learns dynamical distance function, so that it can evaluate functional notion of distance from the predicted states to the goal state, for use as a planning cost. However, the environment does not provide reward signal that might be used to deduce these distances. Indeed, the offline dataset is typically composed of highly suboptimal trajectories, so our method may not even have access to examples of shortest path trajectories between states. Our key observation is that goal-conditioned Q-function trained on a modified MDP with an indicator cost function yields values that correspond to shortest path distances in the original environment. Thus, Q-learning-like like methods can recover optimal distance functions even from sub-optimal data.
We therefore formulate MDP by augmenting environment trajectories with the reward function r(st, a, $ 1 1 mag. adding a discount factor of 7, and considering episodes terminated once they reach the goal state. Note that 8t, 8141, and y all represent images, and the reward is
4

---

Published as a conference paper at ICLR 2021
only given when the next state and goal images exactly match. During training, goals are sampled according to a distribution on S, which we will discuss later. If y < 1, the Q-values for a policy that maximizes expected discounted returns in this MDP can be directly mapped to shortest path distances. Specifically, in discrete state environments, the optimal Q-function can be written as Q(s, a, a.g) = pd(s,..)), where d(ss, a. is shortest path distance between s and after taking action Similarly, we can recover d(ss, a. g) log, Q(s, a, 9). Ultimately, our Q-learning approach corresponds to the following Bellman error optimization objective:
min au
( (1m11=g + 7 max Q (0(4411.441000) 0241
(2)
In practice, we use a deep network to represent the Q-function. During training, we sample
The first three components (st, aes st+1) can be sampled randomly from the dataset. However, trajectories in the offline dataset may not be directed towards any particular goals, so a key challenge lies in selecting which goals to choose. The next section describes our approach to sampling these goals.
transitions (s,, at S(+1: 9) to optimize the objective in Equation 2
Selecting goals for relabeling transitions. Naïvely choosing 9, say by sampling random states uniformly from the dataset, will provide an extremely sparse reward signal, as two random state images will almost never be exactly identical. The sparse reward problem can be mitigated by selectively sampling as goals the states that were actually reached future time steps along the same trajectory St [Kaelbling] 1993] Andrychowicz et al. (017). More precisely, to sample goals for a transition at time step f, we sample a discrete time offset A ~ Geom(p). where p E [0, 1] is a hyper-parameter, and use the state at time + A as the goal. Note that if A 1, the reward for this transition is 1, avoiding the sparsity issue.
However, relabeling all transitions in this way creates major issue: since the distance function would only be trained on goals that were actually reached, it would systematically underestimate the distance to unreachable goals. Put another way, goals that were not reached from 8, would be out-of-distribution goals for the resulting Q-function. We found this to result in poor performance. In practice. prior work (Kaelbling) 1993] Andrychowicz et al., (017) actually relabels with a mixture of reached goals and commanded but not necessarily reached goals.
These prior methods can obtain such "negative" goals based on the goals that were commanded during online data collection. This is impossible in our setting, since our offline data may not even have been collected with a goal-directed policy. We therefore need a procedure to select such "negative" goals that are distant yet relevant. Randomly selecting dataset states will lead to pairs of images that are clearly distant with high probability (e.g., pairs which all objects and the robot have been moved), but not necessarily relevant. We would like a goal sampling procedure that produces less obvious examples of distant states, which are more informative for training. Hard negative mining is one example of such a procedure, where pairs are selected based on the model's predictions, but is computationally expensive with large datasets.
Instead, we build upon the intuition that distance functions are likely to pay excessive attention to fully actuated factors in the state, such as the position of the robot's 's arm, because they are strongly predictive of distances. We propose sampling "negative" goal states g which have similar actuated components to reached states. When randomly sampling pairs of states under this constraint, the un- deractuated dimensions (e.g. the objects), which are generally not known, are likely to have distinct positions. Hence, these data points can serve as informative hard negatives that encourage the model to pay more attention to the difficult, underactuated parts of the state. Unlike hard negative mining, this sampling approach is computationally inexpensive, as it does not rely on the current distance function, and practical, as actuated components of the state can typically be measured through en- coders on the actuator. In practice, we sample these "negative" goals from observations across all dataset trajectories via nearest-neighhors search, using arm joint l2 distance as the similarity key. Note that this does assume proprioceptive state information from the agent (e.g. robot joint angles), which is almost always available in real-world robotics settings, but does not require knowledge about object positions or other ground-truth environment information. While we use actuator infor- mation for generating training examples, the distance function and dynamics model use only image observations and actions as inputs. See Appendix I. for details.
Control via MBOLD. At fest-time, the learned distance function and dynamics model are used together to solve control tasks via MPC. In other words, the dynamics model predicts how candidate
5

---

Published as a conference paper at ICLR 2021
Manipulation tasks
Irikal mage
Goal image
MBOLD (Au5))
Sorie (Boannbach, 2019) PGG (loar 2018)
Rundon Drearrer (1abber 2019)- - MSS:
cajects
Succerss Rate
0.8
Goal Concitooned Behavior Gloning
Vieue Fonwight (foeet, 2010))
0.6
Three oojects
(hard)
clding
0.0
Ona objoct, ragular
Ona objoct, hard
oojects, regular Three oojects, Tesk
Door sliding
Figure 3: Comparative evaluation results: (Left) Example initial states and task definitions for Sawyer object pushing and Franka door sliding simulated environments, as well as the real-world drawer closing task. Note that "hard" tasks require the arm to take detours from moving the final arm position in order relocate the object. Arrows indicate successful trajectories. (Right) MBOLD is consistently able to outperform prior methods these harder manipulation tasks, and by larger margin the most difficult tasks ("hard" variants of object pushing and door sliding). Error bars show standard deviations over seeds.
actions will affect the environment, and the distance model rates predicted sequences based on which bring the agent closest to the user-defined goal state. This mechanism works follows: given the current state 8t, goal state 8y. candidate actions a+:++A-11 and predicted future states fo(se, a1:++h1) from the learned dynamics model, the learned distance function calculates
V (a = max Qo(f a
+ h], a, sy))
(3)
In practice, the maximization over is performed by an actor network learned simultaneously with the Q-function. V ((aut+h-1) acts as an objective function for MPC. Plainly, the controller's goal is to find candidate actions Gct+h-1 which minimize the dynamical distance to the goal h steps into the future. After this process completes, the best action executed by the agent. Note that this controller re-plans after every action taken in the environment (i.e every timestep), in order to prevent errors in dynamics prediction from compounding.
MPC Algorithm. MBOLD uses the CEM algorithm (De Boer et al. 2005) to optimize the objective in Equation 3 It begins by sampling N random trajectories from prior multi-variate Gaussian distribution. Then, the top K actions which score highest according to ((rrtt+--) 1) are selected as candidates. A new Gaussian distribution is on these candidates, and the loop starts over again by sampling fresh actions from this distribution. After I iterations, the loop finishes and returns the best action found so far. See Appendix A.2 for full CEM implementation details.
5 EXPERIMENTS
Our experiments aim to answer three questions: (1) How does MBOLD compare to prior model- based and model-free methods when learning to reach goals from aask-agnostic offline data? (2) Can our method perform visual robotic manipulation in real-world settings? dynamical distance learning methods compare to MBOLD in terms of providing effective distance functions for planning?
(3) How do different
We first evaluate our method, prior methods, and baselines on three simulated tasks with visual observations: (1) simple reaching task that requires moving Sawyer 7-DoF arm to goal location, which provides a way to validate implementations of all methods, (2) object pushing, in which Sawyer arm must relocate an object
particular goal location, in environments with I or objects, and (3) door sliding, which requires repositioning a sliding door with Franka 7-DoF arm. These tasks are challenging because they require long-horizon planning without access to intermediate rewards.
For each task, we define the action space such that actions control the Cartesian position of the robot's end-effector, as well as the robot's gripper. We randomly generate set of 100 test goals, consisting goal image and starting state, for each task, on which all methods are tested. trial is considered successful if the final distance to the goal of each relevant object, e.g. slide position for the door sliding task, ends below a given threshold. For the object relocation task, we evaluate each method two scenes, containing move one of the objects, with the others serving as distractors. We also study two levels of difficulty: "regular," where goals generated from random trajectories in which the object moves certain
and three objects. All evaluation goals require the robot to

---

Published as a conference paper at ICLR 2021
Initial image
Final image
MBOLD
Visual Foresight
(ours)
(l2 pixel error)
Drawer open
8/10
5/10
Drawer close
7 7/10
0/10
Figure 4: Real-world robot evaluation: (Left) Third-person view of example task setting and (Right) results. Success rates are computed using 10 trials for each task. Each task specified by goal image, and as in previous experiments, the same trained models are used across tasks. Task success is determined by the final position of the drawer only.
minimum distance, and "hard," where the arm is additionally enforced to be distant from the object in the goal observation, requiring the robot to push the object and then withdraw the arm, We depict the tasks in Fig. (left) and provide full experimental details in Appendix A.3
For all tasks, we generate an offline dataset by running random policies for le4 episodes of 30 timesteps each. We provide only this offline dataset to all methods, with online training. At test time, the agent only receives the goal image and current observation at each step, and no intermediate rewards besides those that it computes itself.
Comparative evaluation. We compare MBOLD to prior work in model-based and model-free RL. As MBOLD uses purely offline data and does not require rewards from the environment, ods where necessary to provide a fair comparison. Many of these prior methods (though not all) require the envi- ronment to provide a ground truth reward signal. In this case, we provide these methods with simple "uninforma- tive" rewards, following prior work (Nair et al.] (2018), which consist of the MSE between the current and goal image. Many of these methods were initially presented in the online setting. The offline setting is harder for RL methods (Fujimoto et al. 2019: Wu et al. 2019 Kumar al. 2019), partially explaining their poor performance.
Reaching task
Fowaight 20121
fardon
DCB0
Dreamet (hatees ImsgeMCE
Senti Jouetbcch
2010 11 citarcs
make modifications these meth-
Mithod
Figure 5: Comparisons on the simple reaching task, where most methods attain good performance.
See Appendix B for details all baselines. We compare MBOLD the following methods:
Reinforcement Learning with Imagined Goals (RIG) (Nair et al., 2018): RIG is a model-free RL method for visual goal-reaching. Unlike the other methods, we still allow RIG to collect additional online data to train its policy.
Dreamer (Hafner et al. 2019a): Dreamer, a model-based method for image-based tasks, also uses combination of value functions and planning, but uses online data collection and, crucially, ground truth reward signals. We adapt Dreamer for the offline, reward-free setting.
Dreamer l arm distance: We additionally compare with an "oracle" version of Dreamer that uses privileged information about the ground-truth position of the arm.
Search the Replay Buffer (SoRB) (Eysenbach et al. 2019): SoRB performs planning graph constructed using learned distances, learned without reward function.
Conl-Conditioned Behavior Cloning: We train behavior cloning model using goals sampled from observations achieved further in given trajectory. This can be viewed as an offline variant of GCSL. (Ghosh al. 2019) or a non-recurrent version Lynch al. (2020).
Visual Foresight (Ebert al.) 2018b): Visual Foresight also plans with an aetion-conditioned video prediction model, but uses (among other choices) Eg pixel error as a cost function.
Since methods are trained from offline data with no additional environment interaction, we present final performance the test goals as a bar graph, rather than learning curves. The com- parison on the simple reaching task is shown in Figure 5 and suggests that on this task, many of the methods perform quite well. However, on the substantially more complex tasks, shown in Fig- ure 3, we see clearer differentiation between the different algorithms. On harder object pushing tasks, MBOLD attains the best performance, by considerable margin. Interestingly, simple goal- conditioned behavioral cloning actually represents one of the strongest baselines on this task. On the hardest simulated door sliding task, our method attains the best performance by large margin.

---

Published as a conference paper at ICLR 2021
Goal Image
MBOLD (ours)
Fixel-wise MSC
Tempora distance regression
Driect CONNDIA
Dijec popprio
28000 cwam Brr
Figure 6: Hcatmap visualizations of our distance functions. Each pixel in every heatmap represents the distance between a generated starting image containing the object at that (z, y) coordinate and the fixed goal image (pictured on left). All three distance functions show a minimum when the object position is near the goal position of (0.1, 00.05. However, our Q-function produces a better-shaped signal than the direct regression model, and avoids occlusion errors- like the local minimum at high y-values, which plague pixel-wise MSE.
Real-world evaluation. We additionally evaluate MBOLD in a real-worid drawer manipulation task using a 7-DoF Franka arm. We train the dynamics model and distance function on a preexisting dataset of 1000 trajectories collected by weakly supervised batch exploration algorithm in prior work (Chen et al., 2020). As shown in Figure MBOLD outperforms visual foresight on both ma- nipulation tasks with visual inputs, particularly arm position in the goal image does not solve the task. The success of our method in this domain highlights that our method can be applied to offline datasets collected using different exploration strategies. While MBOLD performs well on manipulation tasks even with complex real-world vi- suals, we find that the negative sampling procedure we adopt limits precision in matching highly actuated components such as the arm position. We perform additional analysis through simulated experiments detailed in Appendix E. Videos of both simulated and werldworld task execution can be found at the project website: https://stt http:/ssttsssssss google. com/ www.eeeeeeeeee/ooolll.
drawer closing, for which simply matching the
Qualitative analysis. In this section, we examine the distance functions learned by MBOLD, and show qualitatively that our learned distances better model the dependence of func- tional separation between two states on the rela- tive positions of objects in their scenes. Figure presents heatmaps of predicted distances for a fixed goal image on the object pushing task, as the initial observation is varied based on object position. The robot arm is set to the same po- sition in each initial image. We see that the Q- function is able to learn relatively well well-shaped distance which accounts for the object position.
Planning cost ablation
MBOLD (ours)
Disancs Random wark desancc regnessioo
MBOLD D, regative miving
Posl MSE
Sucooss Rats
Gne chjeet repuier
Dnn object hard Thres ohjecta, 1og.
These chjects, hard
Task
Figure 7: Our learned distance function yields higher success rates than alternative approaches from prior work, such as the la distance of VAE latent space (Nair et al. 2018) and temporal distance regres- sion (Flartikainen al. 2019). We also see consistent improvements from using negative transition mining, especially "hard" tasks.
We additionally visualize baseline distance mod- els for comparison. First, we look at an ablation of our distance model, which is trained via re- gression map pairs of states randomly sampled from given dataset trajectory to the number of timesteps separating them in that trajectory, and can be viewed as an offline variant of DDL (Har- tikainen et al. 2019). We call this scheme that effectively predicts random walk distances "temporal distance regression." The second baseline we compare to is pixel-wise mean-squared error.
We find that the temporal distance regression model produces more sharply peaked distances than the Q-function, and performed worse as a reward signal during planning, as we find through our ablation experiments. The pixel-wise MSE metric produces low distances near the goal object position, but is impacted by occlusions of the objects as well as the position of the visually pronounced arm. While this analysis does not necessarily directly correspond to control performance, as it ignores the movement of the robot, it demonstrates that similarity of nearby object positions, despite the fact that they are learned entirely from images with actions corresponding to the movement of the arm, not the object.
learned distances are aware of the functional
Ablations. Our ablation studies aim to answer three questions: (1) How does Q-learning for learning dynamical distances compare to alternative distance metrics, such distance in the latent space of VAE, or dynamical distances learned using direct regression on temporal distances found random data? (2) How important is mining negative transitions to the performance of our method? (3) How
8

---

Published as a conference paper at ICLR 2021
beneficial is it to combine the learned distance function with planning through predictive model, as compared to directly acting using the learned policy, as in standard model-free offline RL?
To answer the first two questions, perform experiments in the object pushing domain. We evaluate alternative distance metrics for visual planning, by duplicating the planning setup, us- ing the same dynamics model, and only modifying the metric used for scoring candidate trajec- tories. The first distance we consider is Euclidean distance in the latent space a VAE, that is, d(s,gg) le(s) - ((g)]l2. where e is learned encoder, which resembles the reward function used in prior work on image-based goal reaching (Nair et al. 2018). The second is the direct tempo- ral distance regression model described previously. As shown Figure 7 Q-function distances outperform alternative distances on all of the object pushing tasks. While the temporal distance re- gression scheme provides competitive performance in some settings, it often provides overestimates of distances between states rather than shortest paths, as shown qualitatively in Figure 6
We also find that the negative transition mining scheme also consistently improves performance, and is particularly important for the "hard" tasks. We hypothesize this is because augmenting the training data in this way causes learned distance functions to better take into account the positions of objects in the scene, rather than just visually prominent components such as position.
Table I: Comparison of success rates + standard de- viation across 5 random training seeds for our method, which combines Q-functions and planning with model, to a baseline that uses the Q-function to choose actions directly without planning.
Q- o-function
Q-function only
To address the third question, we compare our method, which uses learned distances for plan- ning, to the policy discovered when perform- ing Q-learning to learn dynamical distances. As shown in Table | the policy learned directly
+ plamning
object push 3 object push
55.2 = 4.3%
19.2 + 3.6%
44.8 + 2.9%
3.6%
Reach
94.4 + 3.3%
31.8 + 5.2%
from offline RL alone greatly outperformed by MBOLD, We hypothesize that this is due to chal- lenges in advantage learning from offline data with extremely sparse rewards.
6 CONCLUSION
We presented self-supervised approach to tackling goal-reaching tasks, which learns to reach un- seen visual goals given only an offline, random dataset without reward labels. Our method combines the strengths of predictive models and learned dynamical distances, where predictive model can provide effective predictions for planning actions over short horizons, while dynamical distances can provide a useful planning cost that captures distance to goals over longer horizons. By perform- ing visual model predictive control with a learned visual dynamics model and a goal conditioned Q-function as the planning cost, find that our method is able to perform goal reaching tasks more effectively than model-based planning approaches that utilize other reward specification tech- niques, as well as purely model-free methods. We show that MBOLD can also scale to real-world manipulation settings and learn from offline datasets collected with various exploration strategies, outperforming visual foresight on a drawer manipulation task. By leveraging offline data collected without specific goal in mind, our method may make it possible to utilize large, unstructured, open- world robotic manipulation datasets. Scaling up this method to more complex real-world systems and large data sources therefore represents particularly exciting direction for future work, which may broaden the capabilities and generality of robotic systems.
Acknowledgements We thank students from the Robotic AI and Learning Lab for insightful feed- back on earlier drafts of this paper and Aurick Zhou and Danijar Hafner for helpful discussions. This work was supported in part by Schmidt Futures, the Fannie and John Hertz Foundation, the Office &
of Naval Research (grants N00014-20- 1-2675, 100014-16-1-2400 National Science Foundation (DGE-1745016 and through an NSF GRFP (GRFP 201822 2018259676).. This research used the Savio computational cluster resource provided by the Berkeley Research Computing program the University of California, Berkeley.
and the

---

Published as a conference paper at ICLR 2021
REFERENCES
Brandon Amos, Laurent Dinh, Serkan Cabi, Thomas Rothörl, Sergio Gómez Colmenarejo, Alistair Muldal, Tom Erez, Tassa, Nando de Freitas, and Misha Denil. Learning awareness models, 2018.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience re- play. In Advances in neural information processing systems, pp. 5048-50885 2017.
Arthur Argenson and Gabriel Dulac-Amold. Model-based offline planning. arXiv preprint arXiv:2008.000 05556, 2020.
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy Campbell, and Sergey Levine. Stochastic variational video prediction. arXiv preprint arXiv: /252, 2017.
Matthew Botvinick and Ari Weinstein. Model-based hierarchical reinforcement learning and hu- man action control. Philosophical Transactions of the Royal Society B: Biological Sciences, 369 (1655):20130480 2014.
Annic S. Chen, HyunJi Nam, Suraj Nair, and Chelsea Finn. Batch exploration with examples for scalable robotic reinforcement learning, 2020.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement leamn- ing in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pp. 4754-44544444 4765, 2018.
Pieter- Pieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Rubinstein. tutorial on the cross-entropy method. Annals of operations research, 134(1):19-67, 2005.
M. Deisenroth and C. Rasmussen. Pilco: model-based and data-efficient approach to policy search. In ICML, 2011.
Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Emily Denton and Rob Fergus. Stochastic video generation with learned prior. arXiv preprint arXiv:1802.07687, 2018
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800, 2018.
Yiming Ding, Carlos Florensa, Picter Abbeel, and Mariano Phiclipp. Goal-conditioned imitation 2019.
learning. In Advances in Neural Information Processing Systems, pp.
Kefan Dong, Yuping Luo, Tianhe Yu, Chelsea Finn, and Tengyu Ma. On the expressivity of neural networks for deep reinforcement learning. In International Conference on Machine Learning, pp- 2627-2637. PMLR, 2020.
Frederik Ehert, Sudeep Dasari, Alex X. Lee, S. Levine, and Chelsea Finn. Robustness via retry- ing: Closed-loop robotic manipulation with self-supervised learning. In Conference on Robot Learning. 2018a.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual fore- sight: Model-based deep reinforcement learning for Dision-based robotic control. arXiv preprint 1rXiv:1812.00568, 2018b
Ben Eysenbach, Russ Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. In Advances in Neural Information Processing Systems, pp. 15246-15257. 2019.
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In 2017 IEEE International Conference Roboties and Automation (ICRA), pp. 2786-2793. IEEF, 2017.
10

---

Published as a conference paper at ICLR 2021
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Advances in neural information processing systems, pp. 64-72, 2016.
Carlos Florensa, Jonas Degrave, Nicolas Heess, Jost Tobias Springenberg, and Martin Riedmiller. Self-supervised learning of image embedding for continuous control, 2019.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. International Conference on Machine Learning, pp. 1582-1591, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp.
2019.
Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning, 2019.
David Ha and Jürgen Schmidhuber. World models, 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603, 2019a.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learming, pp. 2555-2565, PMLR, 2019b.
Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja, and Sergey Levine. Dynamical distance learning for semi-supervised and unsupervised skill discovery. In International Conference on Learning Representations, 2019.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. In Advances in Neural Information Processing Systems, pp. 12519- 12530, 2019.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. arXiv preprint arXiv:170..08/ 2017.
Leslie Pack Kaelbling. Learning to achieve goals. In IN PROC. OF IJCAI-93, pp. 1094-1098. Morgan Kaufmann, 1993.
Gregory Kahn, Pieter Abbeel, and Sergey Levine. Badgr: An autonomous self-supervised learning- based navigation system. arXiv preprint arXivv2002.05700, 2020.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. In Advances in neural information processing systems, 2020.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Sys- tems, pp. 11794, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint WxXiv:2006.04779 2020.
Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv: \xXiv:1804001522, 2018.
Alex X. Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with latent variable model, 2019.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0164. 2020.
Kara Liu, Thanard Kurutach, Christine Tung, Pieter Abbeel, and Aviv Tamar. Hallucinative topo- logical memory for zero-shot visual planning. 2020.

---

Published as a conference paper at ICLR 2021
Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control. In Interna- tional Conference on Learning Representations (ICLR), 2019.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on Robot Learning, pp. 1113- 1132, 2020.
Ajay Mandlekar, Fabio Ramos, Byron Boots, Li Fei-Fei, Animesh Garg, and Dieter Fox. Iris: Implicit reinforcement without interaction at scale for learning control from offline robot manip- ulation data. arXiv preprint arXiv:1911.0532) 2019.
Rowan McAllister and C. Rasmussen. Improving pilco with bayesian neural network dynamics models. 2016.
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing. and Sergey Levine, Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning In 2018 JEEE Interna- tional Conference on Robotics and Automation (ICRA), pp. 7559-7566. IEEE, 2018.
Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep dynamics models for learning dexterous manipulation. In Conference Robot Learning, pp. 1101-1112, 2020.
Ashvin Nair, Shikhar Bahl, Alexander Khazatsky, Vitchyr Pong, Glen Berseth, and Sergey Levine. Contextual imagined goals for self-supervised robotic learning, In Conference on Robot Learning, pp. 530-539, 2020a.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.0955 2020b.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Vi- sual reinforcement learning with imagined goals. In Advances in Neural Information Processing Systems, pp. 9191-9200 2018.
Suraj Nair and Chelsea Finn. Hierarchical foresight: superviue learning of long- long-horizon tasks via visual subgoal generation. In International Conference on Learning Representations (ICLR), 2020.
Soroush Nasiriany, Vitchyr Pong, Steven Lin, and Sergey Levine. Planning with goal-conditioned policies. In Advances Neural Information Processing Systems, pp. 14843-14854, 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv: /60.005500 2016.
Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi-parametric topological memory for navigation. In International Conference on Learning Representations (ICLR), 2018.
Julian Schrittwieser, Toannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with a learned model, 2019.
H. J. Terry Suh and Russ Tedrake. The surprising effectiveness of linear models for visual foresight in object pile manipulation, 2020.
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 2(4):160-163, 1991.
Tingwu Wang and Jimmy Ba. Exploring model-based planning with policy networks, 2019.
David Farde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through parammetet discriminative rewards, 2018.
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances neural information processing systems, pp. 2746-2754, 2015.
12

---

Published as a conference paper at ICLR 2021
Théophane Weber, Sébastien Racanière, David P. Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adria Puigdomenech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, Razvan Pascanu, Peter Battaglia, Demis Hassabis, David Silver, and Daan Wierstra. Imagination- augmented agents for deep reinforcement learning, 2017.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning.
arXiv preprint arXiv: (rXXv:9911.11361. 2019.
Lin Yen-Chen, Maria Bauza, and Phillip Isola. Experience-embedded visual foresight. In Confer- ence on Robot Learning, 2019.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcemeett learning. In Conference Robot Learning (CoRL), 2019a.
10897
Tianhe Yu, Gleb Shevchuk, Dorsa Sadigh, and Chelsea Finn. through distributional planning networks. Roboties: Science and Systems XV, Jun 2019b. doi: 10. 10.15607/rss.201 200607/rs..01.xxv00000 xv.020.
Unsupervised visuomotor control
doi /bt. 2019 9. XV 020
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In Advances neural infor- mation processing systems, 2020.
Mingyuan Zhong, Mikala Johnson, Yuval Tassa, Tom Erez, and Emanuel Todorov. Value function approximation and model predictive control. In 2013 IEEE symposium on adaptive dynamic programming and reinforcement learning (ADPRL), pp. 100-107. IEEE, 2013.
13

---

Published as a conference paper at ICLR 2021
A MBOLD IMPLEMENTATION DETAILS
A.1 DISTANCE FUNCTION
This section explains the implementation details for our distance function. Following prior work (Fujimoto et al. 2018), we learn two independent Q-functions and use the minimum for per- forming Bellman backups. Recall that sampled goals from two distributions: future states in the same trajectories, and states from different trajectories where the robot arm was in a similar posi- tion. To implement the second strategy, fit k-nearest neighhors graph 200000 (about 60% of total) dataset observations, and use the f2 arm joint distance as the similarity key. Each batch con- tains equal numbers of transitions generated from each goal sampling method. For computational efficiency, we implement the k-NN search using the GPU-enabled FAISS library (Johnson et al., 2017).
We relabel half of the transitions in each training batch with reached goals and the other half with "negative" goals with similar actuated components, finding through ablation experiments that this combination achieves stronger performance compared to using just reached goals in our evaluation environments. In other domains, more careful consideration required to determine if the assump- tions which motivate this "negative" goal sampling strategy are satisfied.
We also modify the reward specification scheme by providing a small positive reward at each step where the goal is not reached, and then a large positive reward upon reaching the goal. Specifically, we choose to give a reward
by default and 10 when the goal reached (compared to and respectively as presented in the discussion in Section 4). although parameter. We find that it does not affect performance in a statistically significant way (results for each reward choice within standard deviation of one another) to choose this reward over the (0, 1) rewards. Note that this does not change the interpretation of the Q-function as shortest path distance, merely slightly complicating the conversion calculations from Q-values distances in timesteps
do not extensively tune this
Finally, add additional loss term perform conservative Q-learning (CQL) (Kumar al., 2020). a method for offline model-free RL,, which penalizes Q-values of randomly selected actions and increases Q-values of in-dataset actions. We use the Lagrangian version of CQL to automatically tune the weighting term, and detail the parameters below. We find using CQL improves performance on the door sliding task from mean success rate of 41 % to 58%, but does not significantly impact performance on the others.
The Q-function network architecture consists of convolutional and fully connected layers. We define network called the convolutional encoder, which will be used throughout the appendix. This takes as input an image of shape 64 x 64 x containing the starting and goal images concatenated channel- wise, and consists of 2D convolutional layers, with [8, 16, 32, 64] filters, respectively, with all with kernel size (4, 4) and strides of (2, We use Leaky ReLU activations after each intermediate convolutional layer, and batch-norm layers after the second and third Leaky ReLUs.
We flatten the output the convolutional encoder, concatenate the inputted actions, and feed the features through 6 fully-connected linear layers of 128 units each, with the final layer outputting a single value. Each intermediate fully-connected layer is followed by a ReLU activation and a batch-norm layer.
The actor network architecture first contains the above "convolutional encoder", whose outputs are flattened and input into a 10 layer MLP with 128 fully connected units each, and ReLUU activations and batch-norm layers in between. The final output, of dimension 4, is passed through a tanh activation to constrain the normalized action space [-1,
Additional training hyperparameters are detailed in Table 2
A.2 MODEL-PREDICTIO CONTROL
In Table 3 we describe the parameters for model-based planning in our experiments. These pa- rameters shared across all tasks and planning costs (in ablation experiments). Most values selected based on prior work (Ebert et al.) 2018b). We find that replanning every 6 steps produces slightly better performance than replanning every 13 steps, but not by large margin, and we do
14

---

Published as a conference paper at ICLR 2021
Parameter
Value 10000 trajectories
Dataset size
Train/test/val split
0.9/ 0..9/0.05/0.00
Trajectory length
30 steps
Observation dimensions
64 x 64 x 3
State observations in kNN graph
200000
Goal relabeling sampling parameter (p)
0.3 (tuned over [0.2, 0.3])
Discount factor (7)
0.8
Learning rate
3e-4
Target network update Polyak factor Batch size
0.995
64
Actor network noise o
0.1
Actor network maximum noise magniuude
0.2
Training iterations
93750 (300 epochs)
Optimizer
Adam
CQL Lagrange multiplier learning rate
lc-3
CQL slack parameter 7 (object pushing)
3.0
CQL slack parameter T (reaching)
3.0
CQL slack parameter т (door sliding)
10.0
CQL number of randomly selected actions
10
Table 2: Hyperparameters for distance learning
not tune this further due to computation constraints. We sample actions using the filtering scheme described in Nagabandi et al.] (2020) make sequences smoother in time. We initialize sampling distributions using each environment's 's data collection parameters, as shown in Table
To compute the planning cost described in Equation we maximize over a by feeding in the final predicted state to the policy network learned by TD3, and using the outputted action as the maxi- mizer.
Parameter
Value
Planning horizon (h)
13 steps
Actions executed per planning step (k)
actions
CEM Iterations
3 iterations
Elite sample fraction
0.05 (10 samples)
Samples per CEM iteration
200 samples
Table 3: Hyperparameters for model-based planning
A.3 ENVIRONMENTS
The Sawyer environments are adapted from the Meta-World benchmark YYu al. 2019a). and the door sliding environment is based off of the environment presented by Lynch et al. (2020). For each task, we define the 4-dimensional action space such that actions control the Cartesian position of the robot's end-effector, as well as the robot's gripper.
We randomly generate set of 100 different test goals for each setting. Each task is defined by goal image and starting state, on which all methods are tested. We define success for each task in terms of the final distance to the goal of each relevant object, e.g. object position for the object repositioning task. A trial is considered successful if the final distance is below certain threshold manually chosen for each task, listed in the table below. We evaluate the success rate of each method over different random training seeds.
We generate offline datasets for each task by running random policies for 1e4 episodes of 30 timesteps each. In the beginning of each episode, object positions are reset uniformly randomly over the range of possible positions across each joint. The random policy actions are drawn using a filtering technique, which smooths random zero-mean Gaussian samples across time. We apply the
15

---

Published as a conference paper at ICLR 2021
correlated noise scheme described by Nagabandi al. (2020), setting the hyperparameter 3 0.5. The parameters of the multi- vartiivariatt Gaussian samples in each dimension are listed in Table 4
Reaching
Object pushing
Door sliding
Data colln. stdev (2))
[0.6, 0.6, 0.3, 0.3]
[0.6, 0.6, 0.3, 0.3]
[0.3, 0.3, 0.3, 0.15] Slide
Object compared in success threshold Success distance threshold
Arm end effector
Block
0.05m
0.05m
0.075m
Table 4: Environment and task details
B COMPARATIVE EVALUATION IMPLEMENTATION DETAILS
B. REINFORCEMENT LEARNING WITH IMAGINED GOALS (NAIR ET AL.. 2018
In this section, we will discuss implementation details of our adaptation of Reinforcement Learning with Imagined Goals (RIG). We begin by training a 8-VAE with latent dimension 8. The VAE trained on randomly sampled states from the entire offline dataset. For the loss, we use a combination of maximum likelihood term and a KL divergence term which constrains the latent space to a unit Gaussian. In particular, we compute the mean pixel error, that is, HW |s all2. where is the original image, and is the reconstruction, both normalized to be in [0.1]. We add this to the KL divergence between the latent distribution and the unit Gaussian, with weighting factor of Le on the KL penalty.
The architecture of the VAE encoder consists of the "convolutional encoder" described in section A. whose features are passed through two FC layers with 128 units with a ReLU activation and batch-norm layer in between. The VAE decoder takes as input latent states into two FC layers with 128 units with a batch-norm layer and ReLUU activation after each. This is followed by the inverted architecture of the encoder, consisting of transposed 2D convolutions.
Then, we perform model-free RL in a modified MDP, using encoded observations as a substitute for environment observations, and computing rewards as negative E distances in latent space. We sam- ple random goals from the multivariate Gaussian prior (A(0, I)) at the beginning of every episode. We use the open-source implementation of soft actor-critic (SAC) in RLKit, and use the default SAC parameters and architecture found in the implementation, making the following modifications: We increase the number of layers of all MI.P networks from to We use a maximum path length of 30 steps for consistency with our other experiments, and discount factor of 0.95. Along with the goal sampled from the prior at the beginning of each episode, we find that relabeling goals with the achieved observation at the end of the trajectory improves performance, and add these transitions to the replay buffer as well. Note that unlike in the original RIG formulation, we do not update the weights of the learned VAE using data collected online. We evaluate the learned policy after 600 epochs of training, long after environment returns plateau.
B.2 DREAMER HAFNER ET AL.. 2019A)
Dreamer, a model-based method for image-based tasks, also uses combination of value functions and planning. We adapt Dreamer from its original single-task setting to learn a goal-conditioned policy, reward predictor, and value function; however, we do not condition the dynamics model on the goal. Dreamer has been previously demonstrated only in settings where the environment provides rewards to the agent, so we modify the method to learn from unlabeled, offline data by using experience replay. We find that using an indicator reward function as in our method or a heuristically defined reward function, image MSE, causes Dreamer to struggle to learn. We thus additionally demonstrate the performance of Dreamer using manually specified arm distance reward for the Sawyer reaching task.
We build off of the open source implementation of Dreamer by the original authors, written in com/danjjar/dddeeeeer Specifically, to modify the networks TensorFlow2 and found at add independent convolutional encoders which take the goal image to support conditioon,,,,,, as input to each network. Each encoder consists of 2D convolution layers with [32, 64, 128, 256] filters and kernel sizes of to each network, and we concatenate the flattened features the inputs
16

---

Published as a conference paper at ICLR 2021
of each network. We additionally increase the number of fully-connected layers for the value and actor networks from 3 and 2 respectively to 10. We use a discount factor of = 0.93. All other hyperparameter values are defaults from the public implementation.
For training, we relabel trajectories sampled from the fixed, offline dataset with uniformly ran- domly selected observation from the trajectory the goal. In most of our experiments, we compute the negative pixel-wise MSE as the reward, but in one reaching experiment, we use the negative & Euclidean distance between the arm end-effector position and the goal end-effector position. We train for 2000 iterations for each experiment, although initial experiments in which we trained for 20x longer did not yield improved results.
B.3 GOAL-CONDITIONED BEIIVIIRR CLONING
To train condiiinie behavior cloning policy, begin by relabeling random transitions from the dataset with goals which are later achieved those trajectories. Specifically, we sample state- goal pairs from trajectories in the dataset by first selecting the initial state index 4; uniformly from all timesteps, and then selecting the goal state index 'g uniformly from timesteps greater than 4, We then train a neural network to predict the transition action a; given the state s; and the relabeled goal Sy. using mean-squared error loss.
The network architecture is the same as that of the actor network used in Q-learning for MBOLD, described in Appendix A. We train the model for 3125000 iterations (1000 epochs) using a batch size of 32, and use the same optimizer and learning rate as the distance learned for MBOLD.
B.4 SEARCH ON THE REPLAY BUFFER (EYSENBACHI ET AL., 2019
For Search on the Replay Buffer (SoRB), train distributional Q-function to represent distances as in the original paper. Distributional RL discretizes possible value estimates into set of bins 10 for all of experiments. We train this distributional Q-function for 300 epochs, in the distance function training for MBOLD. We also use the same architecture and training scheme, altering the number of outputs to 10 bins and using the KL-divergence loss for the distributional Q-function as in Eysenbach et al. (2019). However, unlike in Eysenbach et al. (2019), we train on just the fixed, offline dataset. We then perform the planning portion of SoRB with the "maxdist" parameter set to 4, after manual tuning. We use graph size of 2000 states for all experiments, due to computational constraints.
We find that the policy learned through Q-learning performs very poorly reaching subgoals, instead substitute the goal-conditioned behavior cloning policy this purpose. We find that this greatly improves performance across all tasks.
B.5 VISUAL FORESIGHT (EBERT ET AL.. 2018B
To compare MBOLD to visual foresight, we use the same dynamics model and planning setup as in MBOLD, however, we substitute the learned dynamical distance function with the lg pixel error cost used in visual foresight.
C ABLATION EXPERIMENTS IMPLEMENTATION DETAILS
C.1 VAE DISTANCE
We the same architecture as the VAE used in the RIG comparison described in Appendix B We set the latent space dimension to 256 and weight the KL divergence term using a factor of -5 We train the model for 3125000 iterations (1000 epochs) using a batch size of 32, and use the same optimizer and learning rate as the distance learned for MBOLD.
C.2 TEMPORAL DISTANCE REGRESSION
To train the temporal distance regression model, we sample state-goal pairs from trajectories in the dataset by first selecting the initial state index uniformly from all timesteps, and then selecting the
17

---

Published as a conference paper at ICLR 2021
goal state index iy uniformly from timesteps greater than &.. We compute the label for this pair as min(ty - t, mazdist). where mardist is a hyperparameter we set to 10. The mardist parameter helps to improve the optimality of distances target label using an f2 error loss. We train the network for 3125000 iterations (1000 epochs) with a batch size of 32, and use the same optimizer and learning rate as the distance learned for MBOLD.
average. We train the neural network to regress this
The architecture for the temporal distance regression model begins with the convolutional encoder described in Appendix B Its flattened outputs are fed into 5 fully-connected layers of 256 units each, with batch-norm and ReLU activations after each intermediate layer.
C.3 Q-FUNCTION POLICY
We find that the policy directly learned by our method when learning distances performs extremely poorly. However, performing Q-learning using random shooting over 100 uniformly random actions selected from [-1, 1] optimize over actions to compute target values produces much better results when used directly as a policy, compared to using an actor network to perform this optimization as in our method. Therefore, we report results from acting according to this random shooting method. At test time, we estimate the optimal action a' arg max, Q(8t, a, by again sampling 100 uniformly random actions, and selecting the best one.
D COMPUTATIONAL COMPLEXITY ANALYSIS
In this section, we discuss the computation complexity of training and acting using MBOLD.
Training: Training the dynamics model takes about 30 hr while training the distance function takes about 5 hr. These training times are dwarfed by the cost of collecting data in the real world, which could take on the order of 3-4 days in the real world (but can be reused for various tasks). In contrast, a single RL approach only requires learning the distance function. While this means that it takes MBOLD significantly longer to train than the single RL approach. note that the dynamics model can be shared across many tasks. We train the dynamics model for 200k and distance function for 94k training steps. A training step for the dynamics model involves one forward and backward pass through the dynamics model. A training step for the distance function requires sampling positive and negative goals, two Q-function forward passes and a policy network forward pass to compute target values and current Q-values, and backward pass to update model parameters. In contrast, a single RL approach would just learn the distance function, not the dynamics model. From the above estimates, this means that training steps for the dynamics model are around 3 times slower than training steps for the distance model. Because the dynamics model can be used perform many tasks, this cost is amortized over these tasks, as compared to single RL. approach.
Acting: Selecting a sequence of actions (6 actions in our experiments) using MBOLD requires one forward pass of the dynamics model for each CEM iteration (3 total in our experiments), and one forward pass through the distance function and policy network. Amortized over a trajectory, this amounts to about 2 seconds wall clock time per action, which can be sped up by around 2x with similar performance by replanning less frequently. For a single RL approach, each action would require just one forward pass through the policy network.
E
ABLATION EXPERIMENTS
E. NEGATIVE MINING ACTUATED STATE COMPONENTS
The ablation experiments presented in Section demonstrate that the negative mining technique can improve performance on manipulation tasks, as evaluated by the final position of the object being manipulated. However, in experiments performed in the real-world Franka drawer setting which only required the robot arm to "reach" to a particular location to match the goal, we found that MBOLD achieved mean final Euclidean distance to goal of 0.14m. while Visual Foresight achieved 0.066m over 10 trials. Here, we conduct additional experiments in simulation to investigate the effect of negative mining reaching goals based accuracy of matching the highly actuated components, for example, the robot arm. In the single-object block pushing setting, we evaluate the performance of distance functions trained with and without negative mining on reaching the desired
18

---

Published as a conference paper at ICLR 2021
goal position. We perform the evaluation using (1) the set of test goals used in original experiments, which include object movement, and (2) an additional set of test goals which only require robot movement. We present the results in Table We find that training without negative mining improves the planner's ability to reach goal arm positions when goals also require object movement, but note that this results in weaker performance in actually relocating those objects, establishing a trade-off. When goals are selected to require just arm movement, performance is comparable with and without the negative sampling scheme.
E.2 PLANNING HORIZON ABLATIONS
In this section, we investigate the effect of the planning horizon on control performance. After training distance functions according to Appendix AA.1| we perform planning with three different settings for h on the simulated block pushing tasks. We present the results in Figure We find that longer planning horizon is beneficial, especially for solving more difficult tasks. We hypothesize that this is because longer planning horizons allow the planner and distance function to better distinguish promising predicted states, while the fidelity of state predictions remains relatively high.
E.3 RANDOM OBJECT RESET ABLATIONS
In this section, perform experiments to evaluate the impact of the distribution of initial object position on task performance. In particular, we look at the single-object Sawyer pushing task. We collect additional dataset with the same policy and other parameters that used in the main parative evaluations, but restrict the random object initialization position to be within [-0.05, 0.05)² opposed to [-0.2, 0.2|2. This represents a 16x reduction in the area of possible initializations. We then train a new dynamics model and distance function from scratch and compare the control performance on the same benchmark tasks from the main comparisons. We present the results in 5 We find that the control performance on these tasks remain within one standard deviation despite the restriction in reset position.
Planning horizon ablation
Success
0.2
0.0
Taxe
Figure 8: Results for planning horizon ablations.
Table 5: Comparison success rates for our method when trained using a dataset where object positions the start of each episode greatly restricted, compared to uniform over the entire space. Standard deviations are over random seeds.
Uniform reset
Restricted reset
object push (regular) object push (hard)
55.2 + 4.3%
54.5 + 3.9%
40.2 + 7.2%
43.2 4: 7.2%
Table 6: Effect of training using negative mining on final arm position matching performance. final & distance to goal arm position of 0. 0.05m less is considered success. Standard deviations success rates are computed over 5 random seods.
Test goals
MBOLD
MBOLD (no
negative mining)
No object movement
89.2 + 1.9%
91.6 + 2.3%
Object movement
64.4 + 5.0%
83.4 4.0%
19


Reviews:
[{"review_id": "jrKZFGoWj17", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# ICLR 2021 Review - Model-based Visual Planning\n\n## Summary\n\nThe work introduces a method that allows learning a forward model and a distance function (for visual similarity) from offline data to later use this in planning. The work compares to sensible baselines and achieves competitive results.\n\n## Strengths & Weaknesses\n\n#### Strengths:\n\n- The method is surprisingly simple (in a good way)\n- The method can leverage random motion without any task-specific reward assignment to learn a model that's suitable for planning solely based on a goal image.\n\n#### Weaknesses:\n\n- It took me the longest time to figure out what's happening in this method because the paper isn't written terribly clearly. I think this can be resolved by reformulating the \"distance learning\" part of section 4. The high-level idea is clear from the beginning but I was struggling to see how exactly the distance function is learned. What you're saying is (a) that you assume that for all recorded image observations, you're also storing the robot's proprioceptive joint states, (b) that you're randomly sampling goals in the recorded trajectories, and that (c.) you're learning the distance function based on how similar the joints are, not how similar the images are. Maybe just add to the formula $r(s_t,a_t,s_{t+1},g) = 1_{s_{t+1}=g}$ that this is true if $x_{t+1} \\approx g \\pm \\epsilon$, where $x_{t+1}$ are the joint states of the robot in state $s_{t+1}$ and $\\epsilon$ is a small tolerance value. Maybe another diagram to illustrate this would be nice. Figure 2 is nice as a high-level overview but the crucial part is definitely the distance function and I felt that wasn't well-explained.\n- I think another important weakness of this method is the scope, i.e. data generation wrt. hard-to-reach areas of the state space. Imagine that your robot arm can't reach an object directly but has to pick up a tool to reach the target object. A random policy may touch the tool but is surely not going to take the tool and figure out that it can be used to reach the target object. This problem can be solved with dense reward (i.e. distance to tool, then distance tool-object) but in this case, this would be incredibly tricky. This is a fundamental problem of using offline data without any annotations and I acknowledge that it wasn't the intention of the method to overcome this exact problem but I'm saying this limits the usefulness of this method in any practical setting where certain areas of the state space are harder to reach.\n- No code was included, making this harder to reproduce.\n\n**TL;DR to improve my rating,** it'd be great if you could rewrite the method section to be a bit more clear and maybe add a sentence to the discussion section on how you'd deal with hard-to-reach areas of the state-space. Also please add an anonymized Github repo containing the project code or link a ZIP file - this has become a common practice in many ML conferences.\n\n## Impact & Recommendation\n\nThere's a handful of works that deal with offline policy learning but none to my knowledge that learn an image-based distance function that can be used to reach a target configuration. There's VICE-RAQ [1], but they need a lot of positive/negative human labels and there's PlaNet/Dreamer [2] but they do need the reward during training to shape their latent representation. One of my students experimented in the past where they removed the reward prediction loss from PlaNet and the performance of the forward model dropped significantly, indicating that the reward-conditioning is very important there.\n\nI'd recommend accepting this paper and I have high faith that the authors can incorporate my suggestions above. \n\n\n## Nitpicks and Comments\n\n- Intro paragraph 2 can be shortened significantly. I don't think a lot in there is relevant to set the stage for the paper.\n- Make fonts in Fig. 1 bigger. Also in Fig. 1, it looks like the image on the top left is one step before the image at the bottom center. Slightly confusing.\n- Fig. 3 fonts on the left are terribly small, also please remove the grey background on the right. Plots don't need to have backgrounds (same for Fig. 4 and 5).\n- Please specify the number of seeds for all experiments in the main paper.\n- The second paragraph in section 5-\"Qualitative Analysis\" is also a bit ambiguous and could use some rephrasing.\n- Fig. 6 is good.\n\n## References\n\n[1] https://arxiv.org/abs/1904.07854\n[2] https://arxiv.org/abs/1912.01603", "extra_scores": {}, "extra_text": {}}, {"review_id": "_sBZB6FwsE", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper combines optimal control and reinforcement learning (RL) for the execution of robotic manipulation tasks with variable goals. The approach learns, on the one hand, an image-based predictive model using a deep neural network and, on the other hand, a distance cost function using Q-Learning. These model and cost function are used to define the next action to execute using model predictive control (MPC). The cost function is learned from task-agnostic data by randomly generating goal states from the state space and including these goals as input variables in a Q-learning approach. This permits defining cost functions for MPC for variable goals. \n\nThe approach is appealing since it exploits the synergies between MPC for planning over short horizons using approximated predictive models and RL for learning longer horizon goals reflected in the cost function.\n \nThe approach is an extension of already existing methods that use approximate dynamic programming for distance estimation. One of the claimed contributions with respect to them is the offline nature of the proposed method, in contrast with the online one of these other approaches. This offline nature is highlighted across the paper. In my opinion, going from online to offline is more a downgrade than an upgrade since offline approaches rely on limited and possibly biased data that might be insufficient to guarantee optimality an adaptability to variable scenarios. I would suggest the authors to be more conservative with this particular claim.\n\nThe distance learning is based on adding the goal as input variables in a Q-learning setting, where these goals are randomly selected. I am wondering what would be the result if they use a single RL method to learn the policy considering the goal as part of the input space. The authors indeed provide some evidence in the Experiment section (Table I) in this vein by comparing the results of their approach with that of the policy learned in the distance learning method. However, this comparison is still not conclusive. Experiments with more advanced RL approaches (e.g. deep RL) should be provided to fully assess the benefits of combining MPC with RL.\n\nThe authors suggest in the Abstract that the approach is suitable for applications where distances in the observation space are not meaningful. However, in the experiments, they use the final distance to the goal of each relevant object to evaluate success and, in distance learning, they use a notion of distance to the goal to define the reward. I would rephrase the statement in the Abstract for consistency.\n\nProvided the method uses limited data in an offline approach, how can the authors guarantee optimality and how can they quantify how different is the learned behavior with respect to the optimal one?\n\nI am also missing an analysis of the computational effort required to learn the predictive model and the distance cost function as well as to define the next action to execute using MPC. I suspect that the computational effort would significantly surpass the one needed by using a single RL approach. This establishes a trade-off between performance and computational effort that needs to be discussed.\n\nThe motivation of the proposed approach is very similar to that of inverse reinforcement learning (Fu et al., 2017). The authors should explain the advantages of using their approach with respect to it.\n\nIn the second page, the authors refer to their approach as a model-based RL one. I disagree, their approach uses RL only to learn the distance function while the decision-making is carried out using MPC without explicitly representing a policy. \n\nMinor comments:\n- The difference between positive and negative goals is not clearly explained (Page 5).\n- The bullet points in Page 7 are not properly introduced.\n\n\nFu, J., Luo, K., & Levine, S. (2017). Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710.11248.", "extra_scores": {}, "extra_text": {}}, {"review_id": "SqJCjef9Klz", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary:\n\nThis paper proposes to learn functional distances to varying goals concurrently with a latent dynamics model from images. A network is trained to predict the Q-value of state action pairs for a sparse reward at the goal. This way the Q-function represents the shortest path distance to the goal. The paper also proposes a specific learning scheme for training the model from random rollouts in the environment. The learned distance function is used with the dynamics model for planning from start to goal images using model-predictive control (CEM). The approach is evaluated on simulated reaching and object pushing tasks. It is compared with state-of-the-art methods and demonstrates improved performance in some of the tasks.\n\n#### Strengths:\n* The proposed approach for learning functional distance via Q-learning seems novel and effective.\n* The approach is evaluated on simulation benchmark tasks and compared with state-of-the-art method. The proposed approach outperforms previous methods in some of the tasks.\n\n#### Weaknesses:\n* Training data in the environment is collected using random policies, however, the paper does not provide sufficient detail about the design of these policies. Does data collection require random initial states in the environment and is this feasible in real systems?\n* The data collection scheme seems to be important for the performance of the method. Can this be applied to the other state-of-the-art approaches as well, and what is the performance gain for those?\n* The paper states that \"in theory\" MBOLD could use arbitrary image-based forward models. Please detail why and which theory.\n* p. 5, the explanation on how state transitions with positive goals from the sample trajectories are generated is not comprehensive (\"To sample positive goals...\"). Why should it make sense to use goals by images with a \\Delta time in between a transition (t -> t+1) and how to obtain this from the data with a given sampling rate? Please clarify.\n* The main paper does not give details on how the Q function network is implemented. It only appears in the supplementary material which suggests that the implementation significantly differs from the presented method: The Q function as in eq 3 is actually not implemented by the method, but an actor network directly predicts the best action in a state. Please clarify.\n* What is the importance of choice of the prediction horizon h in eq 3 on model performance?\n* The training data and goal selection scheme should also be used to train and evaluate the other state-of-the-art approaches to analyze the contributions of training data generation vs. distance learning. \n* The training data selection and goal specification encode the tasks in the evaluated simulation environments. How does the method perform with random trajectories and goal states which require less domain knowledge? The claims in the conclusion appear misleading (\"By leveraging offline data collected without a specific goal in mind,...\"). Please clarify the claim.\n* p. 8, Table I, please compare performance with different planning horizons. Does \"Q-function only\" correspond to a planning horizon of 1 ?\n\n#### Recommendation:\nThe paper proposes an interesting new approach that learns distances to goal states for model-based visual planning. \nThe paper should address the points raised in sec \"Weaknesses\".\n\n#### Question for Rebuttal:\nPlease address points raised on weaknesses above in the rebuttal.\n\n#### Post-rebuttal comments:\nThe author response has well addressed most of my concerns on technical details and experiments.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5kYM5AZaEU", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "###########################################################################################\nSummary:\n\nThis paper proposes a novel distance function that considers task dynamics for visual MPC applications.\n\n###########################################################################################\n\nPros:\n1.\tThe dynamics distance function idea is useful and it can be useful for many existing tasks or visual foresight frameworks.\n2.\tThis work builds on previous works on model-based reinforcement learning and visual foresight. The simple idea of dynamics distance function works pretty well from the results.\n3.\tThis work leverages offline datasets obtained from a random policy instead of online interaction.\n4.\tThe realization of the dynamic distance function idea is interesting by referring to the robot proprioceptive state information and the arm movements. From the experiment, this seems to be a simple idea but works quite well.\n\n###########################################################################################\n\nCons:\nMy concern about the generality of this work actually comes from my last point in the Pros section. Will this assumption always true? Or is it just true when the robot takes more pixels in the observation image? Since there is no such analysis or a more detailed explanation about why this helps, it is hard to tell whether this assumption can generalize to other tasks.\n\n###########################################################################################\n\nRecommendation and explanation:\n\nOverall, I feel like this paper presents a simple idea that works quite well for the presented task. I thus recommend acceptance. Please also refer to my Pros section for more details.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The rebuttal (revisions, and released code) very successfully addressed all the major concerns the reviewers had.

Pros: The dynamics distance function is a very neat, simple (which is good in this case) idea that is theoretically sound, has proven to perform well in thorough experimental results, and that can be broadly applied.

Cons: None
Decision: Accept (Spotlight)