Latent Plans for Task-Agnostic Offline Reinforcement Learning
Authors: Erick Rosete-Beas
Oier Mees
Gabriel Kalweit
Joschka Boedecker
Wolfram Burgard
Venue: CoRL 2022 Poster (2022)
Keywords: Offline Reinforcement Learning
Imitation Learning
Robot Learning

Abstract: Everyday tasks of long-horizon and comprising a sequence of multiple implicit subtasks still impose a major challenge in offline robot control. While a number of prior methods aimed to address this setting with variants of imitation and offline reinforcement learning, the learned behavior is typically narrow and often struggles to reach configurable long-horizon goals. As both paradigms have complementary strengths and weaknesses, we propose a novel hierarchical approach that combines the strengths of both methods to learn task-agnostic long-horizon policies from high-dimensional camera observations. Concretely, we combine a low-level policy that learns latent skills via imitation learning and a high-level policy learned from offline reinforcement learning for skill-chaining the latent behavior priors. Experiments in various simulated and real robot control tasks show that our formulation enables producing previously unseen combinations of skills to reach temporally extended goals by ``stitching'' together latent skills through goal chaining with an order-of-magnitude improvement in performance upon state-of-the-art baselines. We even learn one multi-task visuomotor policy for 25 distinct manipulation tasks in the real world which outperforms both imitation learning and offline reinforcement learning techniques.

Latent Plans for Task-Agnostic Offline Reinforcement Learning
Erick Rosete-Beas Oier Mees" Gabriel , Joschka Boedecker', Wolfram Burgard
Wooffram
UUniversity of Freiburg Uniiversity of Technology Nuremberg
http://tacorl. cs. uni-ffeiburgg htt.:/acoo..c.uuuifffeebbrrrr de
Abstract: Everyday tasks of long-horizon and comprising sequence of multiple implicit subtasks still impose major challenge in offline robot control. While a number of prior methods aimed to address this setting with variants of imita- tion and offline reinforcement learning, the learned behavior is typically narrow and often struggles to reach configurable long-horizon goals. As both paradigms have complementary strengths and weaknesses, we propose a novel hierarchi- cal approach that combines the strengths of both methods to learn task-agnostic long-horizon policies from high-dimensional camera observations. Concretely, we combine low-level policy that learns latent skills via imitation learning and high-level policy learned from offline reinforcement learning for skill-channing the latent behavior priors. Experiments in various simulated and real robot control tasks show that formulation enables producing previously unseen combina- tions of skills to reach temporally extended goals by "stitching" together latent skills through goal chaining with order-of-man maggitdde improvement in perfor- mance upon state-of-the-art baselines. We even learn one multi-task visuomotor policy for 25 distinct manipulation tasks in the real world which outperforms both imitation learning and offline reinforcement learning techniques.
Keywords: Offline Reinforcement Learning, Imitation Learning, Robot Learning
1 Introduction
In recent years, reinforcement learning (RL) has achieved tremendous successes in a variety of do- mains [1. 2, 3, 4]. Especially offline RL [5, 6, 7, 8, 9, 10] with its appealing property to estimate (close-to) optimal policies from previously collected and fixed datasets yielded a strong current in robot control research. However, despite the exceptional progress this fast-moving field, cur- rent offline RI methods are often evaluated on highly specific and artificial benchmarks lacking the complexity and long-term dependencies of everyday tasks, which inherently entail a sequential relationship of multiple implicit subtasks. It lies in the nature of such composite tasks that this trans- lates to estimating optimal actions for a significant amount of consecutive decision steps, making learning of such optimal policies very difficult. In fact, Kidambi et al. [11] discovered quadratic relationship between the horizon of a task and the worst-case accumulated error of any offline RL method. This poses major challenge especially in case of raw and unstructured sensory inputs, as robots must be capable of learning a large repertoire of skills and combine them to perform everyday tasks acting on long time scales.
One way to alleviate the problem of long horizons the hierarchical subdivision of task into high- and lew-level policies, where a high-level policy is chaining executions of multiple low-level policies over primitives. Most commonly, such hierarchical structures are rather rigid and act upon a fixed number of low-level policies, thus lacking the flexibility and extendability required for most real-life settings. In addition, the a priori definition of useful low-level skills or their discovery from data is highly non-trivial task. Related prior work attempted to solve this via coal-condiiooeed reformu- lation [12] of Conservative Q-learning [6]. However, it was shown that on short and distinct robot manipulation tasks, self-supervised learning on unlabeled play can significantly surpass the perfor- mance of individual expert-trained behavioral-cloning policies [13] which, on the flip-side, can be on par with computationally expensive offline RL methods in such settings [14, 15]. In this work, we thus propose to leverage play data, i.e,, non-goal-directed collections of trajectories grounded
Equal Contribution
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Latent Plan
Low level Policy
High-level Policy
Figure I. TACO-RL learns single 7-DoF hicrarchical visuomator policy from offline data. It solve long-horizon robot manipulation tasks by using high-lovel policy that dividos a task into sequence of latent behaviors that are exccuted by low-level policy that interacts environment. reduces effective borizon high-level policy learns
skills through dynamic programaning.
in human action execution, to estimate short-horizon expert policies via imitation learning chained via a coarse-grained high-level policy optimized by offline RL to account for optimal solutions over long horizons. By stitching latent plans extracted from unstructured data, our formulation offers the simplicity of imitation from collected play data while offering long-term optimality for sequential multi-tier tasks. Specifically, we use the collected data to learn our hierarchical policy as acquiring data with good state coverage and visual variety important for successful applications of offline RL. Play data assumes access to an unsegmented teleoperated dataset of semantically meaningful behaviors provided by users, without a set of predefined tasks in mind. Unlike previous hierarchi- cal approaches, that learn long-horizon tasks by performing discrete set of tasks with a low-level policy, we are motivated by the idea of an agent capable of task-agnostic control. In this setting, we endowing the agent with the ability reach any possible target state from given initial state. Specifically, we learn a low-level policy that decodes motor control actions from latent skills learned through imitation learning in self-supervised manner. This low-level policy can perform various behaviors when present in a state by conditioning the policy with latent plan. Consequently, we also learn high-level policy that will order these behaviors achieve long-horizon tasks. This high-level policy is modeled as a goal-conditioned policy that outputs latent plans to be decoded by the low-level policy. The skill-chaining of the behavior priors is learned through offline RL by aug- menting plan transitions with hindsight relabeling. This hierarchical approach constitutes a practical solution by decomposing whole task into smaller chunks of sub-tasks. The high-level policy can learn long-horizon tasks as the effective episode horizon reduced and it does not need to capture in detail the physics of the world, simplifying the underlying dynamics of the RL agent. The model formulation allows to design a general-purpos training objective by considering every possible state reached in the data as potential task. Additionally, our approach is effective for learning policies from large and diverse datasets which do not necessarily contain optimal behaviors.
The primary contribution of this work is hierarchical sule-supe approach learning task- agnostic control policies from high-dimension. observations by combining model-free RL methods with imitation learning. To our knowledge, our method is the first learning system explicitly aiming to solve long-horizon multi-tier tasks from purely offline and unstructured play data without access to model. We integrate components in unified framework, called Task-AgnostiC Offline Reinforcement Learning (TACO-RL). See Figure for an overview. We show that our model obtains the highest success rate when tested against other -fate-of-the-art baselines on various long-horizon tasks of the challenging CALVIN environment [16] and that it is able to learn a single visuomotor 7-DoF policy that can perform wide range of long-horizon manipulation tasks in both simulated and a real-world tabletop environment. At test time, the real world system is capable of solving a challenging suit of 25 manipulation tasks 10 Hz that involve more than 300 decisions per task.
2
Related Work
Offline Reinforcement Learning. Offline RL [5], i.e., RL from fixed and possibly mixed transition sets, constitutes recent trend in RL and robot control research. Generally, at least in model-free

---

settings, these techniques put a regularization on out-of -iioo--dittriuttonn actions, so as to enforce the learned policy to remain in the coverage of the dataset, since offline RL methods tend suffer strongly from the problem of value overestimation. The simplest, yet competitive, attempt to solve this issue a behavioral cloning addendum classical actor-critic framework [8]. Conservative Q-learning [6], on the other hand, imposes a penalty for actions not covered in the dataset making cut-of-distribution actions non-optimal. Fisher-BRC parameterizes the critic as the log-behavior- policy [17] extended by weighted offset term. Implicit Q-learning [10] modifies the Bellman optimality update towards a SARSA-like update, maximizing only over actions in the data-set. How- ever, the rationale behind most current offline RL methods remains rather similar. While we make use of Conservative Q-learning in our experiments which recently emerged as one of the most widely used benchmarks in offline RL we want to point out that any other sophisticated improve- ment upon the classical offline RI. objective is orthogonal to our work and could in principle be incorporated in our framework.
Hierarchical policy learning. Hierarchical policy learning involves learning a hierarchy of policies where a low-level policy performs motor control actions and a high-level policy directs the lowelevel policy to solve task. While some works [18, 19, 20J learn a discrete set of lower-level policies, each behaving primitive skill, this is not appropriate for a general-purpose robot that accomplishes a continuum of behaviors. A large body of hierarchical policy optimization approaches following similar rationale to method use planning in latent state spaces [21, 22, 23, 24, 25, 26, 27, 28] and hence require a model covering the complex dynamics of multistep skills, which is an active field of research on its own. We alleviate the necessity of a model by estimating the high- level policy via model-free RI. as opposed to model-based planning and thus keep the optimization over the continuous set of skills completely at training time. In contrast to plethora of prior work (29, 30, 31, 32, 33, 34, 35], our approach acts in the offline paradigm as it yields a lot of appealing properties for learning robot control policies. En route solve long-horizon problems, our approach exploits unstructured play data to estimate the latent skill embeddings as opposed to other work that relies on expert data or predefined skills [36, 37, 38). Whilst in principle also other latent skill representations could be considered [39, 40, 41, 42, 43), we build upon Play-LMP [13] as it has already shown great performance and robustness in this very setting. Our design choices are directed towards a general-purposs visuomotor agent that has a high zero-shot generalization even for complex long-horizon tasks, a setting more complex than in previous offline methods [13, 44]. In summary, our approach is aiming to solve temporally extended tasks without the necessity of model or planning from entirely offlin,, unstructured, unlabeled and suboptimal data. This unique combination of properties thus adds scalable and extendable optimization method to the toolbox of robot learning.
skill-chaining policy able to
3 Mathematical Foundation
In this section, introduce notation and define the problem setting. We model the interaction be- tween and environment and a goal-conditioned policy as goal-augmented Markov decision process M (S, A. p. G, ?) where S represents the state-observation space, A represents the action space, (ss'ls,, a) is a state-transition probability function, r(s, a, s') represents the reward function, G S specifies the goal space, (s)) an initial state distribution, and y E (0. represents the discount factor. We note that the agent does not have access to the true state of the environment, but visual observations. We learn an offline manner by assuming to use a large, unlabeled, and undirected fixed dataset DD = {(11, 01), (82, 02),.., (ST, a.)). We then relabel this long temporal state-action stream to produce a dataset of trajectories D) = {(T: = (st, ) IN that can be used to learn both the low-level and high-level policy without access to a model.
Offline goal-conditioned RL with TACO-RL
In this section, we elaborate on TACO-RL. Our proposed method considers the bottom-up approach; we start by training lowelevel policy and we use to provide a higher-level action space for a high- level policy that, due to this task division, is ideally facing an easier learning problem. First, we describe our unsupervised objective, which learns a continuous space of conditiond behav- iors n.. (a)ss z) from D, where 8, represents the current state. Afterward, we detail how to learn the high-level policy with offline RL by hindsight relabeling sub-trajectories with the aid of the pre- viously learned low-level policy. This reduces our effective task horizon, making it easier to learn long-horizon tasks. Additionally, the lowelevel policy will predict actions close to the offline data distribution, bringing stability to the whole learning pipeline. See Figure for an overview.
3

---

Objeclive
Action Ikalhoad
Offine Training
iss
Lou-llevll
KL divergence minimization
High-level policy 1
Soowr
action
Dog
Low-lew level policy
KHz
Labent space
Relabeled transition
Mttnt
Biy Fky Rteae- 7 (c, Boy Aosiell
65 lertz
plan
pann
gual
High-lival poiity
Fixed
pnor
Encoder de
pre-trained
TTg
enpoder 9o
ourrnt
pool
Intr
Endre
acsion
action
Ofline dalaset D
Offine dalaset D
(1) Low-level pallcy learning
(2) High-level policy leamning
(3) Policy inference
2. TACO-RI Overview. TACO-RI. RI. is self supervised general purposs madel learned from offline datnset mhot interactions, it generalizes to a wide variety of long-horizon monipulation tasks. (1) Low-level policy: Recognizes and organizes repertoire of behaviors unlabeled. undirected dataset lotent plan space. High-level policy: Hindsight selabeling sampled windows of experience reward augmented latent plan transitions. Leurned with oflline RL. this allows the high level policy to stitch plans logetber to achaeve complex. long horizon tasks, (3) Inference: hierarchical model to perform goal conditioned rollouts rooo manipulation tasks.
4.1 Learning the low-level policy
We would like to extract a continuous space of primitives that propose meaningful behaviors for an agent to take within a given state. We Iearn low-level policy To (a]s,,2) from the offline, unstructured dataset D that is able decode latent plan its respective motor-control actions a. After training, we can use the latent plans as an action space for the high-level policy to learn reaching temporally extended goals by "stitching" together latent skills through goal chaining.
In our fixed static dataset D, it is expected to find different valid behaviors achieving the same out- come in scene, e.g. closing drawer quickly or slowly. We address this inherent multi coltiiimdality by auto-encoding contextual data through a latent plan space with a sequence-to-sequene variational auto-encoder (seq2seq CVAE) [13, 45]. Conditioning the action decoder the latent plan allows the policy to use the entirety of its capacity for learning uni-modal behavior. Conse- quently, propose the following objective for learning the low-level policy T (alse)
conditional
1r] log( (aelst,
min E D, wid
where E indicates empirical expectation and (2(2))) may be interpreted as the latent plan encoder. As additional component of the algorithm, we enforce consistency in the latent variables pre- dicted by encoder [=(2]T] and prior Ts ((]])t,,)) Since our goal is to obtain a latent plan that captures temporal sequence of actions for given trajectory
(so, ao. Sk, ax), we utilize regularization that enforces the distribution go (2/T) to be close to just predicting the primitive or the latent variable given the initial and last state of this sub-trajectory, i.c. (2(2|se. sg)- The Evidence
Lower Bound (ELBO) [46] for the CVAE can be written as:
log logp(z(s) 2 KL (g(2|ær,
|| p(2(s)) + Eg(ala.,)) [log p(x]2,
The conditioning of the prior Ta(2|Se, 8g) on the initial and final state regularizes the distribution (2(2)) to not overfit to the complete sub-trajectory т. In practice, rather than solving the con- strained optimization directly, we implement the KL--contrrintt as penalty, weighted by an ap- propriately chosen coefficient 3. Thus, one may interpret our objective as using a sequential 3- VAE [47]. Finally, we use balancing terms within the KL loss [48, 49], see Appendix D.I.
4.2 Offline RL with Hindsight relabeling
After distilling learned behaviors from D terms an encoder (2(7), latent behavior policy
Tu (a)se, 2), and a prior TT Tx(2|8n, 8g), TACO-RL then applies these behaviors to learn a general- purpose agent with offline RI. We formulate a goal augmented MDP by augmenting environment
trajectories with reward function. Thereby, we sample trajectory from the dataset =~ D.

---

Positive examples:
r=
Negative examples:
==
Current slate
Samples from goal distribution
relabel sampled trajectories reward augmented transitions by sampling goal states that can reached after executing sequence of hehaviors. With green border, have the frame found at the end of the sampled trajectory. As this state will be reached after executing the behavior, the reward for transition
Figure 3.
With border, find future states occur These goals are necessary for chaining behaviors. The reward for these transitions is 0, Finally, with red border, we present images with similar proprioceptive information to the final state in the sampled trajectory, but different scene arrangement. The reward for these transitions O.
sampled sequence.
Then, we use this trajectory to represent high-level policy transition by using the pre-trained fixed encoder. For this, we sample a latent plan from the policy encoder 2₁ ~ 4s (2(rr and we generate an interaction transition using the sampled latent plan (st, Rt, St+k-1). To augment this transition with a reward, we use hindsight relabeling with a sparse reward as follows r(st, 2t, $1+8-1, 8y) = can decode the latent plan
1 -14x-1=55- In this formulation, we assume that during inference using the low-level policy and we will reach the final trajectory state St+k-1- Note that 8t, k and 5y all represent images, and the reward is only given when the high-level transition reached state and goal state exactly match. During training, goals are sampled according to distribution 8y E S, which we will discuss later. Our Q-learning approach corresponds the following Bellman error optimization objective:
min Ey D.--)i--aaslliissss [Qa(s 2,9) Q8., 2t, D)
where: Qs,,, 21) (I + 71 - max 14
We can then learn high-level policy (2(2s 8g) with an off-the-shelf offline actor-critic method. In TACO-RL, we Conservative Q-Learning (CQL) [6] where we initialize the actor weights with the previously learned prior policy Ta a (2|s2, $g). as the prior policy is already a good starting point that is able solve short-horizon tasks.
Selecting goals for relabeling transitions. Our high-level policy is learned via offline RL as we want to learn through dynamic programming how to chain skills to reach long-horizon goals. For this we need to create a formulation that sample goal states 8g that can be reached after executing a sequence of plans. Naively choosing $g. say by sampling random states uniformly from the dataset, will provide an extremely sparse reward signal, as two random state images will rarely be identical. The sparse reward problem can be mitigated by selectively sampling as goals the states that were reached in future time steps along the same trajectory as 8, [50]. As we want to sample states that are reached after executing plan, we assume an arbitrary window size k. Concretely, to sample goals for a transition at time step f, we sample a discrete- time offset A ~ Geom with e [0, 1], and use the state at time + A (k- 1) as the goal. Note that we assume consistent transitions from latent plan decoding, then if A = 1, the reward for this transition is I, as the low-level policy will reach the specified state after executing the plan, avoiding the sparsity issue.
However, relabeling all transitions in this manner introduces a problem: because the distance func- tion is only trained on goals that have been achieved, will systematically underestimate the distance to unreachable goals. We require a method for selecting "negative" goals that are distant, but still relevant. Randomly selecting states will produce pairs of images that likely be distant, but necessarily relevant (c.g., pairs in which all objects and the robot have been moved). We want goal sampling procedure that generates less obvious examples of distant states that are more informative. Similar to Tian et al. [51]. we sample negative goal states Sy which have a similar proprioceptive state. This constraint enables the Q-function learn focus on the scene's under-actuated parts (e.g., objects), which are likely to have distinct positions. As result, these timesteps act as hard negatives, encouraging the model to pay closer attention to the scene. This sampling approach is computationally inexpensive, as we can query a precomputed k-nearest neighbors structure.
5

---

5
Experimental Results
We evaluate TACO-RL for learning a general-purpon robot in both simulated and real-world envi- ronments. The goals of these experiments are to investigate: (i) whether our hierarchical model is effective in performing complex long-horizon skills, (ii) how TACO-RI. compares with alternative goal-conditioned policies, (iii) if our model scales to be used in real-world robotics.
5.1 Experimental Setup
We evaluate our approach in both simulated and real-world environments. We first investigate learn- ing 7-DoF visuomotor rohot skills in the CALVIN environment [16]. We train on the environment I) of CALVIN, which contains 6 hours of unstructured play data collected via teleoperating a Franka Emika Panda robot arm to manipulate objects in a 3D tabletop environment.
Baselines Methods. As we aim to combine the complementary strengths of both paradigms, imita- tion and offline RL, we compare TACO-RL to representatives of the two extrema of the spectrum: the offline RL method Conservative Q-learming [6] extended by hindsight relabeling (CQL+HER) and the imitation learning method Play- supervisd Latent Motor Plan (LMP) [13]. CQL+HER is trained the derived reward explained in Section 4.2 and introduces a penalty for out-of- distribution actions to limit the respective values of unseen actions. To make a fair comparison, this baseline is also trained with the negative mining trick. LMP, on the other hand, trains goal conditioned imitation agent and resembles the low-level policy in TACO-RL, however, without any long-term optimality guarantees which TACO-RL accounts for by offline RL of a higher-level plan- selective policy. Additionally, we also compare against Relay Imitation Learning (RIL) [32]. This algorithm is the most related hierarchical algorithm, it also learns from offline play data. RIL represents the family of methods that learns to predict latent subgoals for a low level policy.
5.2 Simulation Results
We start by evaluating our approach in the CALVIN environment [16]. This is a challenging en- vironment as the scene changes through time and we act by using only RGB images of static camera as input. As there is no predefined reward signal in this dataset, we relabel the transitions analogously as do with TACO-RL. We investigate if our method is capable of performing com- plex long-horizon tasks in a robot control setting. We first attempt to solve 500 unique chains of image-based goals queried in a row. For each subtask in a row the policy is conditioned on the current sub-goal image instruction and transitions to the next sub-goal only if the agent successfully completes the current task if 180 timesteps have passed without reaching a success. We call this evaluation of performing multiple tasks on a row, long-horizon multitask with visual observations LH-MTVVs.. This setting is very challenging as it requires agents to be able to transition between different subgoals. Additionally, we ablate our model by increasing the negative goals ratio to 50% and removing the negative mining.
Method
LIV-MVVis
No. Instructions in a Row (500 chains)
Avg. Len.
Ours
95.4% :-2
82% 233
17.8%
11.7%
6.9% #3.1
2.7:0.33
No neg. goals 50% neg, goals
94.9%
69.7%:1114.
31.19:115.7
5.5%
0.9%
2.02=0.44
71.2
2..1%
6.2%.11..6
0.23
09:10
1.05.0.11
RIL | LMP [13]
70.3%
32.9%
10.5::4.01
2410077
0.1::0.1
L17::0.15
91.4% 12.3
63.3%
23.1
3.6%
0.20-10.08
1.810.08
CQL-HER
65.5%
25.3%
5.6%
0.6%::50.2
0%
0.940.2
Table Success rates models running chains per three different random seeds, using intermediate sub goal images.
In Table 1. we can see that TACO-RL is able to outperform all baselines. This experiment demon- strates that our agent is able to transition between different sub-goal images more naturally, enabling chaining more tasks in a row. Through our ablations, we observe that the performance of our model drops significantly when increasing the negative goal ratio to 50%. This result is to be expected as reducing the number of positive examples leads to a less informative reward indicating which are the useful behaviors to reach transition. If we remove the negative goals from the goal distribution, the agent is still able to perform more sequential tasks than the baselines, but underestimates the distance to the goals. This results in decreased performance compared to our full approach.

---

Figure Real-world Manipulation Tasks, Examples shown door left, moving the sliding door right, lifting the block, rotating the block, pushing the block, tumingg the green LED on, placing the block top of the drawer and placing the block in the container.
right closing drawer, opening drawer, moving sliding
We then evaluate the capacity of our model to perform 1000 rollouts of two sequential tasks using single goal image. For this, we allow the agent to perform actions until 300 timesteps has passed. We record the success rate of all models in Table 2. TACO-RI. successfully performs long-horizon tasks that require reasoning over sequential behaviors with an final success rate of 27% which corresponds to order of magnitude improvement upon the LMP and CQL+HER baselines. RIL also exploits a hierarchical structure which allows it to reason over longer horizons than the other baselines, but TACO-RL still obtains more than two times its accuracy when performing both sequential tasks using a single image, proving its effectiveness chaining skills through dynamic programming.
We further test the capacity of the models to perform single task when the goal image does not contain the robot performing the task, but the end effector appears another position af- ter the task was performed (cf. Table 3). We run 50 rollouts for each task. These harder tasks re- quire reasoning about changes in the scene and additionally evaluates generalization, as each rollout uses different goal image not seen dur- ing training. With our ablations, we can see that including the negative goals in our framework contributes greatly to obtain good performance in this scenario alleviating to not only imitate the final end effector position, but reach the entire scene configuration. We noticed that TACO-RI. was able to outperform the baselines, which can be explained through the skill-chaining abilities of our model and the negative goal mining used to train the critic network of the high-level policy. On the other hand, CQL+HER is not able to achieve the desired goal image and LMP has a strong bias towards the end-effector position ignoring the changes in the environment. RIL can imagine intermediate latent sub-goals required to achieve the task, which reduces the bias towards the end effector position, but it stills obtain a lower accuracy rate than our method.
Method
LH-MTVis
No. Instractions in Row (1000 chains)
Avg. Len.
Ours
67.9% 13.99
27%
0.94-0..6
neg. goals
29.7
4.2%
0.44=0.04
50% neg. goals
43.4%
6.3%
0.52=0.05
RIL [32] LMP [13]
66.2%
13.3%-22.3
0.79=0.00
34.3%:43.2
2.7%=+0.1
0.3+0.03
CQLAHER
35.2% 1:3.4
0.4%
0.37=0.04
Table 2. Success rates of models running 1000 chains per three different random conditioned only on the last goal image.
Method Task
Place block in drawer
Open drawer
Move slider left
Turn on lightbulh
Ours
94% #8.7 7..3%
87.3% ::5
111.7
94% 444 9.3 92%
No neg, goals
37.3% 37.3% 133.3
13.3%
50% neg, goals
88%-17.2
181.6
29.3
[32] LMP [13]
74.67% 1:26.6
7.4
92.79
78.6%=66.
12%=2
12%
10%
CQL+TER
67.6%%:1:0.88
8.6%
17.3%
4%
Table 3. The average porforming the task. Three models trained from diffcrent random soeds were used to perform the rollouts.
of coldiinnitooned models running 50 rollouts where the goal image does contain the end effector
5.3 Real-Robot Experiments
For the real-world experiments, we investigate learning single policy capable of performing mul- tiple goal conditioned tasks. Examples of the tasks are shown in Figure 4. To generate the training dataset we collected nine hours of play data recorded via teleoperating Franka Emika Panda robot arm with a VR controller. To avoid self-occlusions in the scene, these models also receive RGB
7

---

images of gripper camera as an additional input. After training the models with the offline dataset, we performed 20 rollouts for each task using multiple goal images and start positions. TACO-RL was able to outperform the baselines consistently, especially when evaluated from a start position far away from the goal image or that required extended reasoning. We recorded the success rate of each model in Table 4.
Task \Method
Ours
LMP [13]
CQL+HER
Lift the block top of the drawer
60%
60%
20%
Lif the block inside the drawer
65%
50SS
1.5%
the black from the
60%
30%
10%
Lift the block from the container
65%
60%
20%
Lift the block from the table
70%
70%
30%
Place the black on top of the drawer
60%
50%
30%
Place the block inside the drawer
70%
40%
20%
Place the block the slider
30%
0%
Place the block in the container
65%
30%
1.5%
Stack blocks
30%
0%
Unstack the blocks
30%
10%
0%
Rotate block left
70%
40%
10%
Rotate block right
70%
50%
15%%
Push block left
60%
50%
20%
Push block right
60%
50%
10%
Close drawer
90%
70%
20%
Open drawer
70%
50%
10%
Move slider left
75%
30%
0%
Move slider right
70%
10%
0%
Turn red light
60%
30%
0%
Tur red light off
50%
20%
0%
Turn green light
70%
10%
Turn green light off
65%
50%
10%
Tom blue light on
50%
50%
5%
Turn blue light ofl
50%
30%
10%
Average over tasks
61%
40%
11%
Table 4. The average success rate of the multi task goal conditioned models rumning roll-outs in the real world.
We also tested our approach to perform sequential tasks with the real robot to verify that our ap- proach can be scaled for long-horizon tasks. For this experiment we use a goal image for each task that the robot executes. See the supplementary video for qualitative results that showcase the diver- sity of tasks and the long-horizon capabilities of the different methods. Our agent trained completely from unlabeled play data is able successfully perform most of these sequential tasks, by inferring how to transition between tasks and reach the state depicted by the goal image. More details in Appendix F.3.
Finally, we evaluate TACO-RL performing complex tasks with a single goal image, such as lifting the block and moving it to desired position, and stacking a block on top of another. These tasks requires the agent to reason in a long-horizg position, the objects would not be arranged correctly in the scene. By stitching together the learned latent behaviors, our model was able to perform these tasks consistently.
manner, as if the robot imitates only the end effector
6 Conclusion and Limitations
In this paper, we introduced Task-Agnostic Offline Reinforcement Learning (TACO-RL), which ex- ploits a latent plan representation estimated from unstructured play data effectively limit the hori- zon of a high-level offline RL policy acting upon this latent plan space. By dividing sequential multi-tier tasks into chunks of implicit subtasks solved by imitation learning. TACO-RL showed up to an order-of titer-o--manntudee improvement in performance compared to state-of-the-arr both imitation learning and offline reinforcement learning baselines in both, simulated and real robot control tasks.
While TACO-RL is quite capable, does have a number of limitations. Specifying a task to the requires providing suitable goal image at test-time, which should be consistent with the current scene. Besides, tracking task progress might be useful when sequencing skills of different time horizons. We discuss limitations in more detail in Appendix H. But overall, are excited about the confluence of imitation and offline RI. methods towards scaling robot learning.
8

---

Acknowledgments
This work has been supported partly by the German Federal Ministry of Education and Research under contract 011S MIIS8804B-OMM
References
[1] V. Mnih, K. Kavukeuoglu, D. Silver, A. Rusu, Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning. Nature, (1aee,,518755444 518(7540):52-533,, 2015.
[2] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, et al. general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 144, 2018.
[3] M. Riedmiller, R. Hafner, T. Lampe, M. Neunert, J. Degrave, T. van de Wiele, V. Mnih, N. Heess, and J. T. Springenberg. Learning by playing solving sparse reward tasks from scratch. In J. Dy and A. Krause, editors, Proceedingy of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4344 4353. PMLR, 10-15 Jul 2018.
[4] T. Haarnoja, S. Ha, A. Zhou, Tan, G. Tucker, and S. Levine. Learning walk via deep reinforcement learning. In Proceedings of Robotics: Science and Systems, 2019.
[5] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
[6] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforce- ment learning. Advances Neural Information Processing Systems, 33:1179-1199 2020.
[7] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. In International Conference on Machine Learning, pages 2052-2062. PMLR, 2019.
[8] S. Fujimoto and S. Gu. minimalist approach to offline reinforcement learning. In Advances in Neural Information Processing Systems, 2021.
[9] T. Yu, G. Thomas. L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model- based offline policy optimization. In Advances in Neural Information Processing Systems, volume 33, pages 14129-14144 Curran Associates, Inc., 2020.
[10] Kostrikov, A. Nair, and S. Levine. Offline reinforeement learning with implicit q-learning. In International Conference on Learning Representations, 2022.
[11] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel: Model-based offline rein- forcement learning. In NeurIPS, 2020.
[12] Y. Chebotar, K. Hausman, Y. Lu, T. Xiao, D. Kalashnikov, J. Varley, A. Irpan, B. Eysenbach, R. Julian, C. Finn, al. Actionable models: Unsupervised offline reinforcement learning of robotic skills. In International Conference on Machine Learning, pages 2021.
PMLR,
[13] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. In Conference on robot learning, pages 1113-1132. PMLR, 2020.
[14] A. Kumar, J. Hong, A. Singh, and S. Levine. Should i run offline reinforcement learning or behavioral cloning? In International Conference on Learning Representations, 2022.
[15] P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mor- datch, and J. Tompson. Implicit behavioral cloning. Conference on Robot Learning, 2021.
[16] O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard. Calvin: A benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and 7(3 7(37377777 2022.
Automation Letters

---

[17] I. Kostrikov, R. Fergus, J. Tompson, and O. Nachum, Offline reinforcement learning with fisher divergence critic regularization. In ICML, volume 139 of Proceedings of Machine Learning Research, pages 5774-5783. PMLR, 2021.
[18] A. Gupta, C. Lynch, B. Kinman, G. Peake, S. Levine, and K. Hausman. Demonstration- bootstrapped autonomous practicing via multi-task reinforcement learning. arXiv, 2022.
[19] X. B. Peng, M. Chang, G. Zhang, P. Abbeel, and S. Levine. Mcp:
Learning composable hi- erarchical control with multiplicative compositional policies. Advances in Neural Information Processing Systems, 32, 2019.
[20] P.-L. Bacon, J. Harb, and D. Precup. The option-critic architecture. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[21] O. Rybkin, C. Zhu, A. Nagabandi, K. Daniilidis, I. Mordatch, and S. Levine. Model-based reinforcement learning via latent-space collocation. In ICML,, volume 139 of Proceedings of Machine Learning Research, pages 9190-9201. PMLR, 2021.
[22] K. Pertsch, O. Rybkin, F. Ebert, S. Zhou, D. Jayaraman, C. Finn, and S. Levine. Long-horizon visual planning with goal-conditioned hierarchical predictors. In NeurIPS, 2020.
[23] K. Pertsch, O. Rybkin. J. Yang. S. Zhou, K. G. Derpanis, K. Daniilidis, J. J. Lim, and A. Jaegle. Keyframing the future: Keyframe discovery for visual prediction and planning. In L4DC, volume 120 of Proceedings of Machine Learning Research, pages 969-979. PMLR, 2020.
[24] Nair and C. Finn. Hierarchical foresight: Self-supervised learning of mong-horizon tasks via visual subgoal generation. International Conference on Learning Representations, 2020.
[25] D. Jayaraman, F. Ebert, A. A. Efros, and S. Levine. Time-agnostic prediction: Predicting predictable video frames. In International Conference Learning Representations, 2019.
[26] K. Fang, Y. Zhu, A. Garg, S. Savarese, and L. Fei-Fei. Dynamics learning with cascaded variational inference for multi-step manipulation. In CoRL, volume 100 of Proceedings of Machine Learning Research, pages 42-52. PMLR, 2019.
[27] B. Ichter, Sermanet, and C. Lynch. Broadly-exploring, local-policy trees for long-horizon task planning. In CoRL, volume 164 of Proceedings of Machine Learning Research, pages 59-69. PMLR, 2021.
[28] L. X. Shi, J. J. Lim, and Y. Lee. Skill-based model-based reinforcement learning. arXiv preprint (xiv:2200.007660. 2022.
[29] K. Pertsch, Y. Lee, and J. Lim, Accelerating reinforcement learning with learned skill priors. In CoRL, volume 155 of Proceedings of Machine Learning Research, pages 188-204. PMLR, 2020.
[30] K. Fang, P. Yin, A. Nair, and Levine. Planning to practice: Efficient online fine-tuning by composing goals in latent space. arXiv preprint arXiv:2205.08122, 2022.
[31] S. Nasiriany, V. Pong, S. Lin, and S. Levine. Planning with goal-conditioned policies. In NeurlPS, pages 14814-14825 2019.
[32] A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. CoRL, volume 100 of Proceed- ings of Muchine Learning Reseurch, pages 1025-1037. PMLR, 2019.
[33] O. Nachum, S. Gu, H. Lee, and S. Levine. Data-efficient hierarchical reinforcement learning. In NeurlPS, pages 3307-3317, 2018.
[34] T. Shankar and A. Gupta. Learning robot skills with temporal variational inference. In Inter- national Conference on Machine Learning, pages 8624-8633. PMLR, 2020.
[35] Borja-Diaz, O. Mees, G. Kalweit Hermann, Boedecker, and W. Burgard. Affordance learning from play for sample-efficient policy learning. In ICRA, Philadelphia, USA, 2022.
10

---

[36] A. Mandlekar, D. Xu, R.
S. Savarese, and L. Fei-Fei. GTI: learning to general- ize across long-horizon tasks from human demonstrations. Roboties: Science and Systems, 2020.
[37] D. Xu, S. Nair, Y. Zhu, J. Gao, A. Garg, L. Fei-Fei, and S. Savarese. Neural task programming: Learning to generalize across hierarchical tasks. In ICRA, pages 1-8. IEEE, 2018.
[38] D. Shah, P. Xu, Y. Lu, T. Xiao, A. T. Toshev, S. Levine, et al. Value function spaces: Skill- centric state abstractions for long-horizon reasoning. In International Conference on Learning Representations, 2021.
[39] A. Singh, H. Liu, G. Zhou, A. Yu, N. Rhinchart, and S. Levine. Parrot: Data-driven behavioral priors for reinforcement learning. In International Conference on Learning Representations, 2021.
[40] D. Ghosh, A. Gupta, and S. Levine. Learning actionable representations with goal conditioned policies. In ICLR (Poster). OpenReview.net 2019.
[41] O. Nachum, M. Ahn, H. Ponte, S. S. Gu, and V. Kumar. Multi-agent manipulation via locomo- tion using hierarchical sim2real, In Conference Robot Learning, pages 110-121. PMLR, 2020.
[42] K. Hausman, J. T. Springenberg, Z. Wang, N. Heess, and M. Riedmiller. Learning an embed- ding space for transferable robot skills. International Conference on Learning Representa- tions, 2018.
[43] O. Mees, M. Merklinger, G. Kalweit, and W. Burgard. Adversarial skill networks: Unsuper- vised robot skill learning from videos. In ICRA, Paris, France, 2020.
[44] A. Ajay, A. Kumar, P. Agrawal, S. Levine, and O. Nachum. OPAL: offline primitive discov- ery for accelerating offline reinforcement learning. In International Conference on Learning Representations, 2021.
[45] L. Wang, X. Meng, Y. Xiang, and D. Fox. Hierarchical policies for cluttered-scene grasping with latent plans. IEEE Robotics and Automation Letters, 2022.
[46] D. P. Kingma and M. Welling. 13/26113 14, 2013.
Auto-encoding variational bayes.
arXiv preprint
[47] I. Higgins, L. Matthey, A. Pal, C. P. Burgess, X. Glorot, M. M. Botvinick, S. Mohamed, and
A. Lerehner. beta-vae: Learning basic visual concepts with a constrained variational frame- work. In International Conference on Learning Representations, 2017.
[48] D. Hafner, T. P. Lillicrap, M. Norouzi, and J. Ba. Mastering atari with discrete world models. In International Conference on Learning Representations, 2020.
[49] O. Mees, L. Hermann, and W. Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Roboties and Automation Letters (RA-L). 7(4):1 1205- 11212, 2022.
[50] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. To- bin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[51] S. Tian, S. Nair, F. Ebert, S. Dasari, B. Eysenbach, C. Finn, and S. Levine. Model-based visual planning with self-supervised functional distances. In International Conference on Learning Representations, 2021.
[52] T. Salimans, A. Karpathy, X. Chen, and D. P. Kingma. Pixelenn++: Improving the pix- elenn with discretized logistic mixture likelihood and other modifications. arXiv preprint 2017.
MrXiv:1701.055
[53] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- tation for robot manipulation. arXiv preprint 22060 2022.
11

---

[54] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint #rXiv:204.0022/1 2020.
[55] O. Mees and W. Burgard. Composing pick-and-place tasks by grounding language. In Pro- ceedings of the International Symposium on Experimental Robotics (ISER), La Valletta, Malta, 2021.
12


Reviews:
[{"review_id": "ix1yOBnC85P", "reviewer": "Reviewer_nPLz", "summary": "The authors propose an approach for learning hierarchical policies from demonstration data. TACO-RL first learns a goal-conditioned, imitation learned policy from play data, similarly to [13]. Then TACO-RL uses offline reinforcement learning to train a high-level policy to propose goals for the low level, facilitated by a goal relabeling and negative mining approach to improve performance. This is then shown on a maze and robot desk environment in real and sim to outperform LMP and CQL baselines in both short-horizon and long-horizon tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "he paper is generally clear, as are the figures. The approach is interesting and developing scalable policies that leverage unstructered demonstrations is important. The results show it performs well, particularly for image space, interestingly it also seems to stabilize short-horizon policies as well as long. Including the code is appreciated.\n\nMy primary concern with TACO-RL is its framing with HRL and prior work that makes it difficult to judge the novelty. \n- The authors minimally discuss other HRL work and frame this approach as a “planner” which I do not believe it fits into the class of. Particularly many recent works have focused on offline HRL and goal-conditioning. See:\n  - [1] Data-Efficient Hierarchical Reinforcement Learning (https://proceedings.neurips.cc/paper/2018/file/e6384711491713d29bc63fc5eeb5ba4f-Paper.pdf) \n  - [2] Actionable goal-conditioned policies (https://openreview.net/pdf?id=Hye9lnCct7)\n  - and many more related to these works\n- The authors claim “To our knowledge, our method is the first learning system for a generalist-purpose robot that uses the benefits of hierarchical policies skill-chaining latent behaviors to reason over long-horizon tasks.” This is a significant overclaim to my understanding, based on prior work such as below (but also many others in HRL):\n  - [3] Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (https://arxiv.org/pdf/1910.11956.pdf) which chains goals\n  - [4] Value function spaces (https://arxiv.org/pdf/2111.03189.pdf) which chains skills\n  - [5] Neural Task Programming (https://arxiv.org/pdf/1710.01813.pdf)\n  - [6] GTI (https://arxiv.org/pdf/2003.06085.pdf)\n  - Even the original play paper does this to an extent.\n- These approaches do skill chaining particularly for robots, the first of which proposes a very similar methodology to TACO-RL.\n- The authors need to discuss and compare to works such as above, as the comparison to LMP and CQL is nice, but not informative as they are not comparable methods in this class.\n- With a more HRL framing the authors should compare to some of these prior works and possibly on some standard benchmarks.\n\nA few major questions/clarifications:\n- The authors of [3] discuss issues around training a high-level policy on demonstration data, when the low-level policy may not mirror it. Is this an issue at all here?\n- The authors should clarify in Section 4.1 what is the standard LMP formulation and what is theirs. And highlight any changes.\n- The authors should ablate over their algorithmic choices, such as the negative mining.\n- It would be useful to see long-horizon tasks in real.\n\nThere are significant typos and grammatical errors. The paper should be reviewed for clarity:\n- L18- reinforcement learning (RL) achieved -> reinforcement learning (RL) has achieved \n- L20- fixed datasets yield a strong current in -> rephrase\n- Split up paragraph 2, its hard to get through\n- “generalist-purpose robot” -> “general purpose\n- Many more\n\nMinor:\n- Images of every environment would be useful\n- Figure 1 could be clearer\n- Define LH-VisMTLC\n- It would be useful to have one video with the an overview and then rollouts from each environment. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The approach is reasonable and the performance is good compared to baselines of LMP and CQL. But these are not the baselines I would have expected, as this is a type of HRL planner. The framing of TACO-RL as a planner versus HRL, leads to missing these ablations and baselines, and generally discussing and framing the paper within existing literature.\n\nAfter rebuttals I have raised my score to weak accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See weaknesses above.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "EAB8vPsuq36", "reviewer": "Reviewer_jnvs", "summary": "This paper presents an approach that learns a hierarchical policy from offline play data. On the low level, the propose approach imitates a latent-conditioned policy using a CVAE. On the high level, a goal-conditioned policy is trained using CQL. The proposed approach is evaluated in both simulated and real-world table manipulation tasks and achieves better performance than LMP and CQL baselines. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n- The proposed method consistently achieves a noticeable performance improvement compared to LMP and CQL+HER. \n\n- The real-world task seems to be challenging and the proposed method achieves good success rates in these tasks.\n\n\nWeaknesses:\n\n- The paper does not clearly justify why the proposed hierarchical model is necessary. Usually, hierarchical reinforcement learning enables sharing knowledge about low-level skills across multiple tasks. The alternative and more common hierarchical RL frameworks either learn a goal-conditioned policy on the low-level and use a planner on the high level (e.g. Planning with Goal-Conditioned Policy (https://arxiv.org/abs/1911.08453), Planning-to-Practice (https://arxiv.org/pdf/2205.08129.pdf)), or learns a latent-conditioned policy on the low-level and use a non-goal-conditioned high-level policy. The proposed method is a hybrid of both and seems to be redundant to me. Could the authors explain why this design is better and include comparisons with these approaches in the experiments?\n\n- The proposed low-level learning algorithm is reasonable but similar approaches have been proposed in prior work, e.g. Pertsch et al. (https://arxiv.org/abs/2010.11944), Fang et al. (https://arxiv.org/pdf/1910.13395.pdf), Singh at al. (https://arxiv.org/abs/2011.10024). I would recommend the authors discuss the differences with these works and include comparisons in the experiment section. \n\n- It is not clear to me why the authors regularize q_{\\phi} to a prior conditioned on both the s_c and s_g instead of only the current state. The sentence on line 143 (“Since our goal is to obtain a latent plan……” is a bit confusing to me.\n\n- Only “three different random seeds” are used in each experiment, which is a bit insufficient.\n\n- Is there any specific reason for using CQL instead of AWAC or IQL, which demonstrates better performance in many experiments. \n\n\n- The precisions in each of Table 1-4 are not unified. Sometimes the number is rounded to the first digit sometimes the second. \n\n- s_c is not defined in the paper. \n\n- The limitations discussed in the paper is about goal-conditioned RL and behavioral cloning in general instead of being about the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am leaning towards rejection for now since the paper needs to better justify its design choices and include more thorough comparisons with prior work. I would consider raising my scores if the authors can address the above issues. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "- The design choices need to be better explained and fully justified. \n\n- More prior work needs to be cited and discussed.\n\n- More baselines need to be compared in the experiment section. ", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "CPs4IlzF5RJq", "reviewer": "Reviewer_mMCb", "summary": "The paper presents a method (TACO-RL), to tackle self-supervised, goal-conditioned RL from offline datasets with just visual observations. Specifically, given \"play\" data (unstructured human demonstrations), the method learns a policy to reach goal images, with an emphasis on reaching long horizon goals. \n\nThe proposed method learns two models:\n- First, a low-level skill conditioned model trained with imitation learning. This part of the model is same as the LMP method (Lynch et al) where relabeled goals are mapped to a latent skill space which the policy is conditioned on.\n- Second, a high level goal conditioned policy whose actions are the space of skills which gets fed to the low level policy. This policy is trained with offline RL with relabeled goals and a binary relabeled reward. \n\nThe proposed method outperforms each of the parts of the method (i.e. LMP and CQL+HER individually). Results look strong on a D4RL ant task, CALVIN manipulation, and a real robot setup. \n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n- The problem tackled is well motivated, learning to reach long-horizon goals from reward-free offline dataset is important in getting robots to learn from large datasets.\n- The proposed method is straightforward and well motivated. It makes sense that the goal-conditioned imitation is reasonable on shorter horizons, but wont reach distant goals. Meanwhile, offline RL with goal-relabeling can stitch together trajectories and makes sense as a choice for a high level policy enabling reaching distant goals. Also unlike many HRL algorithms, since the low-level policy is simple imitation and decoupled from the training of the high level policy, the method seems stable and scalable.\n- The simulated experiments on CALVIN show pretty solid improvements over LMP and CQL+HER individually. \n- The paper has robot results.\n\n*Weaknesses*\n\n- First, the coverage of related work could be improved. There are several prior works [1,2,3,4,5] which also aim to reach long-horizon goal images in a self-supervised way via subgoals/hierarchy that aren't mentioned. [5] could also be good include as an additional point of comparison. \n\n[1] Jayaraman et al. Time-Agnostic Prediction: Predicting Predictable Video Frames. 2018.\n[2] Nair et al. Hierarchical Foresight. Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation. 2019.\n[3] Pertsch et al. Keyframing the Future: Keyframe Discovery for Visual Prediction and Planning. 2019.\n[4] Pertsch et al. Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors. 2020. \n[5] Rybkin et al. Model-Based Reinforcement Learning via Latent-Space Collocation. 2021.\n\n- Second, I'd be interested in seeing some more exhaustive ablations in the experiments. While LMP and CQL+HER are effectively the main two ablations, I'm also curious what would happen if the hard negative mining trick from Tian et al. is removed from TACO-RL. From what I can tell from the text, the baseline CQL+HER does not get this same trick, in which case the comparison isn't totally fair. \n\n- Third, while it is great the paper has real robot results, the real world tasks look pretty short horizon, especially compared the skill chaining highlighted in the simulation results. Some longer-horizon real world tasks would make the results more compelling. \n\nFinally, I had a few more questions about the method:\n- From Figure 2 it seems like the high level policy is executed once for every 1000 steps of the low level policy. I'm curious how this parameter was chosen, and if performance varies depending on the choice of how frequently to query the high level policy. \n- How important is the LMP latent plan/skill learning component of the low-level policy? Could the low-level policy be replaced with a simpler GCBC policy? \n\nSome other minor comments:\n- Table 4 caption has some formatting issues.\n- It would be nice to see videos of the task execution (especially the simulated skill chaining). An anonymized website which highlights the sim and real rollouts of TACO-RL and baselines would be a nice addition to the paper. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall a solid paper which combines LMP and CQL+HER for good results on long-horizon goal reaching from offline datasets and visual observations. Method is clear, has strong improvement over baselines in sim, and has real robot results. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See Strengths and Weaknesses.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "AZujrpsClE4", "reviewer": "Reviewer_3aE6", "summary": "This paper proposes a hierarchical policy and skill learning approach that can learn from human collected play data. The goal is to be able to achieve longer horizon tasks while training the hierarchical policy in a task agnostic manner. Firstly, a latent action model is learnt to encode play data. Specifically, this consists of learning a mapping sub trajectories: $s_t, s_{t+1}..., s_{t + k}$ to a latent $z_t$ which encodes the \"skill\". A low level policy is used to decode this latent into actions $a_t$. A policy prior is also trained to output the right $z$ to reach between state $s_t$ and goal state $s_g$. \n\nUsing the learnt encoder and policy prior, an offline RL goal-conditioned policy is learnt to produce the right $z$ given current state, achieved goal state and desired goal state. This is trained via a standard offline RL procedure and a designed goal state relabeling scheme. Experiments show good performance on CALVIN benchmark and standard offline RL benchmarks. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\nI think this paper tackles an important problem of learning a hierarchical policy from play data. Play data can provide useful signal and it is non-trivial to try to build a policy using this. This method, to my knowledge, is a novel way to learn a goal conditioned policy based on play data -- but the novelty could be made cleared (see weaknesses paragraph). In general, this method is technically sound and well motivated, and the presentation quality of the paper is good. Overall,  I appreciate experiments on both the offline RL benchmark, CALVIN benchmark and real world settings. In all of these, the method shows a good performance. \n\nWeaknesses: \n\nI think the main weakness for this paper is that it is unclear how it situates with previous work done in this area, such as LMP [13]. I think there could be more baseline comparisons, especially with other skill learning and hierarchical policy learning approaches. It is currently now quite clear how the CQL + HER baseline works - in my understanding this is just doing relabeling on the raw data. Overall the experiments could be strengthened with a more rigorous ablation section, for example understanding the role of the relabeling scheme (and comparison to others), they type of imitation learning done on the play data, which skills are actually represented by $z_t$. The real world experiments are interesting, but it would be good to see chaining of the skills for long horizon tasks -- as this is one the stated benefits of this approach. This could include for example opening the drawer, picking up the object, putting it inside the drawer and closing it. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "My recommendation for this paper is a \"weak reject\" but I am willing to increase my rating if the authors can clarify exactly where this work falls with respect to prior work such as LMP and others. Overall, I think the ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "The main issues that should be addressed: \n\n- More comparisons to hierarchical policy learning methods \n- More ablations analyzing the work\n- Possible long horizon tasks, especially combining the skills seen in the real world (open drawer, grasp object etc) "}}]

Author rebuttal:

Decision: Accept (poster)