Generalization with Lossy Affordances: Leveraging Broad Offline Data for Learning Visuomotor Tasks
Authors: Kuan Fang
Patrick Yin
Ashvin Nair
Homer Rich Walke
Gengchen Yan
Sergey Levine
Venue: CoRL 2022 Oral (2022)
Keywords: Reinforcement Learning
Representation Learning
Planning

Abstract: The use of broad datasets has proven to be crucial for generalization for a wide range of fields. However, how to effectively make use of diverse multi-task data for novel downstream tasks still remains a grand challenge in reinforcement learning and robotics. To tackle this challenge, we introduce a framework that acquires goal-conditioned policies for unseen temporally extended tasks via offline reinforcement learning on broad data, in combination with online fine-tuning guided by subgoals in a learned lossy representation space. When faced with a novel task goal, our framework uses an affordance model to plan a sequence of lossy representations as subgoals that decomposes the original task into easier problems. Learned from the broad prior data, the lossy representation emphasizes task-relevant information about states and goals while abstracting away redundant contexts that hinder generalization. It thus enables subgoal planning for unseen tasks, provides a compact input to the policy, and facilitates reward shaping during fine-tuning. We show that our framework can be pre-trained on large-scale datasets of robot experience from prior work and efficiently fine-tuned for novel tasks, entirely from visual inputs without any manual reward engineering.

Generalization with Lossy Affordances: Leveraging Broad Offline Data for Learning Visuomotor Tasks
Kuan Fang, Patrick Yin, Ashvin Nair, Homer Walke, Gengchen Yan, Sergey Levine
University of California, Berkeley
Abstract: The use of broad datasets has proven to be crucial for generalization for wide range of fields. However, how effectively make use of diverse multi-task data for novel downstream tasks still remains grand challenge in reinforcement learning and robotics. To tackle this challenge, we introduce a framework that acquires goal-conditioned policies for unseen temporally extended tasks via of- fline reinforcement learning on broad data, combination with online fine-tuning guided by subgoals in learned lossy representation space. When faced with a novel task goal, our framework uses an affordance model to plan sequence of lossy representations problems. Learned from the broad prior data, the lossy representation emphasizes task-relevant information about states and goals while abstracting away redundant contexts that hinder generalization. tasks, provides a compact input to the policy, and facilitates reward shaping dur- ing fine-tuning. We show that our framework can be pre-trained on large-scale datasets of robot experience from prior work and efficiently fine-tuned for novel tasks, entirely from visual inputs without any manual reward engineering.
subgoals that decomposes the original task into casier
thus enables subgoal planning for unseen
Keywords: Reinforcement Learning, Representation Learning, Planning
Broad Offline Data
Target Task
Lossy Representation Space
state encoder
actions
affordance model
planned subgosis
Pree-ti
initial state
goal-oondilloned policy
final gos!
Fine-T Tine-Tuning with Lossy Affordance Planner (FLAP). Our framework leverages broad offline data for new temporarily extended tasks using learned lossy representations of states and goals. We pre-train a state encoder, and affordance model, and a goal- coal-conditioned policy on the offline data collected from diverse environments and fine-tune the policy to solve the target tasks without any explicit reward signals. To provide guidance for the policy, subgoals are planned in the lossy representation space given visual inputs.
Figure I:
Introduction
Learning-based methods can enable robotic systems automatically acquire large repertoires of behaviors that can potentially generalize to diverse real-world scenarios. However, attaining such generalizability requires the ability to leverage large-scale datasets, which presents a conundrum when building general-purpose robotic systems: How can we tractably endow robots with the de- sired skills if each behavior requires a laborious data collection and lengthy learning process to master? In much the same way that humans use their past experience and expertise to acquire new
Project webpage: bitesggoog..comm/vee/jjjeetttt
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

skills more quickly, the answer for robots may be to effectively leverage prior data. But in the setting of robotic learning, this raises a number of complex questions: What sort of knowledge should we derive from this prior data? How do we break up diverse and uncurated prior datasets into distinct skills? And how do we then use and adapt these skills for solving novel tasks?
To learn useful behaviors from data of wide range of tasks, we can employ goal-conditioned reinforcement learning (GCRL) [1, 2], where a policy is trained to attain specified goal (e.g., indicated as an image) from the current state. This makes it practical to learn from previously collected large datasets without explicit reward annotations and to share knowledge across tasks that exhibit similar physical phenomena. However, it is usually difficult to learn goal-conditioned policies that can solve temporally extended tasks in zero shot, as such policies are typically only effective for short-horizon goals [3]. To solve the target tasks, we would need to transfer the learned knowledge to the testing environment and efficiently fine-tune the policy in an online manner.
Our proposed solution combines representation learning, planning, and online fine-tuning. The key intuition is that, if we can learn a suitable representation of states and goals that generalizes effectively across environments, then we can plan subgoals in this representation space to solve long-horizon tasks, and also leverage it help finetune the goal-conditioned policies on the new task online. Core to this planning procedure is the use of affordance models [4, 5], which predict potentially reachable states that can serve as subgoal proposals for the planning process. Good state representations are necessary for this process: (1) as inputs and outputs for the affordance model, which must generalize effectively across tasks and domains (since if the affordance model doesn't 't generalize, it can't provide guidance for policy fine-tuning); (2) as inputs into the policy, so to facilitate rapid adaptation; (3) as measures of state proximity for use as reward functions for fine- tuning the policy, since well-shaped rewards are essential for rapid online training but notoriously difficult to obtain without manual engineering.
To this end, we propose Fine-Tuning with Lossy Affordance Planner (FLAP), framework that leverages diverse offline data for learning representations, goal-conditioned policies, and affordance models that enable rapid fine-tuning to new tasks in target scenes. As shown in Fig. 1, our lossy representations are acquired during the offline pre-training process for the goal-conditioned policy using a variational information bottleneck (VIB) [6]. Intuitively, this representation learns to capture the minimal information necessary to determine if and how a particular goal can be reached from a particular state, making it ideally suited for planning over subgoals and providing reward shaping during fine-tuning When new task is specified to the robot with a goal image, we use an affordance model that predicts reachable states in this learned lossy representation space to plan subgoals for prospectively accomplishing this goal. During the fine-tuning process, the goal-conditioned policy is fine-tuned on each subgoal given the informative reward signal computed from the learned represen- tations. Both the offline and online stage in this process operates entirely from images and does not require any hand-designed reward functions beyond those extracted automatically from the learned representation. Building the components based on the prior work [5, 7], we demonstrate that the par- ticular combination of lossy representation learning, coul-conditioned policies, and planning-drive fine-tuning can enable performance that significantly exceeds that of prior methods. We evaluate our method in both real-world and simulated environments using previously collected offline data [8, 7] for learning novel robotic manipulation tasks. Compared to baselines, the proposed method achieves higher success rates with fewer online fine-tuning iterations.
2 Preliminaries
Goal-conditioned reinforcement learning. We consider a goal-conditioned reinforcement learning (GCRL) setting with states and goals as images. The goal-reaching tasks can be represented by a Markov Decision Process (MDP) denoted by a tuple M (S,A, p, P. G 7) with state space S, action space A, initial state probability
transition probability P, a goal space G, and discount factor 7. Each goal-reaching task can be defined by a pair of initial state su E S and desired goal 8g E G. We assume states and goals are defindd in the same space, i.e. g = S. By selecting the action at E A at each time step I, the goal-conditioned policy π (ur(st, 8n) aims to reach a state s such that d(s 8g) S e, where d is a distance metric and is the threshold for reaching the goal. We use the sparse reward function ri r 8g) which outputs 0 when the goal reached and -1 otherwise and the objective is defined as the average expected discounted return E[E,)'r.).
2

---

Variaionall Infommation Bolleneck
iffoddnne
canditene plans
culinel plan
pollcy
mit' a.
aryiza of
HL loGs
(R00
(a) Pre-Training on Offline Data
(b) Fine-Tuning with Lossy Affordance Plans
Figure 2: FLAP Architecture. (a) We pre-train the representation with RL on the offline data. We first encode the initial state and the goal and then compute the RL loss with policy the value function V. and the Q-function Q. (b) We use planned subgoals to guide the policy during online fine-tuning (right). Given the encodings of the initial state and the final goal, subgoal sequences are recursively generated by the affordance model the learned representation space with latent codes sampled from the prior p(u). The optimal subgoal sequence a is then selected according to Eq. 6 for guiding the goal goal-conditioned policy.
Planning with affordance models. When learning to solve long-horizon tasks, subgoals can sig- nificantly improve performance by breaking down the original problem into easier short-horizon tasks. We can use sampling-bar planning to find a suitable sequence of K subgoals 81.K. which samples multiple candidate sequences and chooses the optimal plan based on a cost function. In high-dimensional state spaces, such planming process most sampled candidates will be unlikely to form a reasonable plan. To facilitate planning, we would need to focus sampling subgoal candidates that are realistic and reachable from the cur- rent state. Following the prior work [5. 7]. we use affordance models to capture the distribution of reachable future states and recursively propose subgoals conditioned on the initial state of each task. The affordance model can be defined as a generative model m(s'|s ((ss'ss, u), where u is the latent representation that captures the information of the transition from the current state s the goal s'. It can be trained in a conditional variational autoencoder (CVAE) [9] paradigm using the encoder q(u),, (') estimate the evidence lower bound (ELBO) [10].
be computationally intractable, since
Offline pre-training and online fine-tuning. Offline reinforcement learning pre-trains the policy and value function
dataset Doftinee consisting of previously collected experiences (s), aj, ri, 4141), where i and are the index of the trajectory and time step. In this work, we assume Domline does not include data of the target task. To solve the target task, the pre-trained policy is then fine-tuned by exploring the environment and collecting online data D Douline. Following Fang et al. [7], we first train both the policy and the affordance model m on Dottine and then use the subgoals planned using the affordance model to guide the online exploration of т. In the online phase, We freeze the weight of m and fine-tune π a combination of Domine and Dontine- While our contribution is orthogonal to the specific choice of the offiine RL algorithm, we use implicit Q-learning [11] which is well-suited online fine-tuning using an expectile loss to construct value estimates and advantage weighted regression to extract policy.
3 Pre- Pre-Training and Fine- Tine-Tuning with a Lossy Affordance Planner
The objective of our method is to enable a robot to leverage diverse offline data to efficiently learn how to achieve new goals, potentially in new scenes. While we do not require the offline dataset to contain data of the target tasks, two assumptions are made for enabling effective possible generaliza- tion. First, we will consider tasks that require the robot to compose previously seen behaviors (e.g., pushing, grasping, and opening) in order to perform a temporally extended task that sequences these behaviors in a new way (e.g., pushing an obstruction out of the way and then opening a drawer). Second, we will consider tasks that require the robot to perform behaviors that resemble those in the prior data, but with new objects that had not seen before, which presents a generalization challenge for the policies and the affordance models.
Our method is based on offline goal-conditioned reinforcement learning and subgoal planning with affordance models. To solve temporarily extended tasks with novel objects, we construct the goal- conditioned policy and the affordance model in learned representation space of states and goals that picks up on task-relevant information (e.g., object identity and location), while abstracting away unnecessary visual distractors (e.g., camera pose, illumination, background). In this section, we first describe paradigm for jointly pre-training the goal-conditioned policy and the lossy representation through offline reinforcement learning- Then, we describe how to construct an affordance model in this lossy representation space for guiding the goal-conditioned policy with planned subgoals.
3

---

Finally, we describe how we utilize the lossy affordance model in a complete system for vision- based robotic control set goals and finetune policies online.
3.1 Goal-Conditioned Reinforcement Learning with Lossy Representations
To capture the task-relevant information of states and goals, we learn parametric state encoder ((]|s) to project both states and goals from the high-dimensional space (e.g., RGB images) S into a learned representation space Z. We use 2 and 2g to denote the representation of the state 8; and the goal g, respectively. In the learned representation space, we construct the goul-conditioned policy Tal zy), well the value function V(2,, =g) and Q-function Q ²g. a). Both V and Q are used for training the policy via the offline RL algorithm, which in our prototype is IQL [11], and V is also used for selecting subgoals in the planner, which will be discussed in Sec. 3.2. When training Φ. т, V, and Q to optimize the goal-reaching objective described in Sec. 2, á would need extract sufficient information for selecting the action transit from 8t to g and estimating the required discounted number of steps.
We jointly pre-train the representation with reinforcement learning on the offline data shown in Fig. 2. The original IQL objective optimizes LRL = Cx + Lq + Ly as described in Kostrikov et al. [11]. To facilitate generalization of the pre-trained policy and the affordance model, we would like to abstract away redundant domain-specific information from the learned representation. For this purpose, we add variational information bottleneck (VIB) [6] the reinforcement learning objective, which constrains the mutual information I [ss;; 2t) and [(g: 2y) between the state space and the representation space by constant C. The joint training of é, n, V and Q can be written as an optimization problem using 0 to denote the model parameters:
max
LRL. s.t. Tgg S C, T(8)2) S C, for = 0,...,
(1)
Following Alemi et al. [6], we convert this into unconstrained optimization problem by applying the bottleneck on the Q-function representation and directly selecting the Lagrange multiplier a, resulting in the full VIB objective:
C CRL aDwz Il p(z)) (DDz(o(zzlo)) ll p(=))
(2)
where we use the normal distribution p(z), the prior distribution of
By optimizing this objective, we obtain lossy representations that disentangle relevant factors such as object poses from irrelevant factors such as scene background. We exploit this property of the learned representations in several ways: learning policies and affordance models that generalize from prior data, having prior for optimization, and as a distance metric for rewards during finetun- ing. Next, we will describe how we generate subgoals utilizing the bottleneck representation.
3.2 Composing Subgoals using Lossy Affordances
The effectiveness of using subgoals to guide the goal-conditioned policy relies on two conditions. First, each pair of adjacent subgoals needs to be in the distribution of Dolline so that the pre-trained goal-conditioned policy will have sufficient success rate for the transition. Second, the subgoals should break down the original task into subtasks of reasonable difficulties. As shown in Fig. 2, we devise an affordance model and cost function to plan subgoals that satisfy these conditions and use the learned lossy representation to facilitate generalization to novel target tasks.
Instead of sampling goals from the original high-dimensional state space, we propose subgoals ě in the learned lossy representation space. For this purpose, we learn a lossy affordance model to capture the distribution p(z'lz), where z is the representation of a state and a' corresponns to the future state s' that is reachable within At steps. The lossy affordance model can be constructed as parametric model (2='|z,, u), where is latent code that represents the transition from z to z'. Given a sequence of sampled latent codes U1:K, we can recursively sample the k-th subgoal εn in the sequence ě1:M with by taking ê and 2h inputs, where we denote En- 2g.
The lossy affordance model is pre-trained on the offline dataset Domine Given each pair (s, 8') sam- pled from Dottine the pre-trained encoder produces distribution of lossy representation ((='|s"). Given the sampled representation 2, we would like the affordance model to propose representations that follow the distribution o(2'|s'). Using Pmo (z'lz) to denote the marginalized distribution of the

---

lossy affordance model, the learning objective can be defined as the KL-divergence:
(P (z'|z) 6(z'ls'))
(3)
Under the conditional variational autoencoder (CVAE) [9] paradigm, we define the CVAE encoder q(ulz 2 2') to infer the latent code u. The ELBO [10] of this objective can be written
Ez- (mxx(ma==))), u) II (==1|6))]] = Pillluulz, z²) p(u))]
(4)
Given the trained affordance model, the sampling- based planning can be conducted in the lossy rep- resentation space. Given the initial state sn and the final goal g. first project them into latent representations 2g and bg using db. Then, we would like to find the optimal sequence of subgoals in the representation space that (1) reaches the final goal Zg. (2) consists of a sequence of subgoals that the RL. policy believes are easy to reach, and (3) where each subsequent subgoal has a high proba- bility under the affordance model given the previous one, indicating that the sequence of subgoals is physically plausible. We use the Euclidean distance in the lossy representation space to measure if the goal is reached. We use the value function estimate the difficulty of the transition from one subgoal to another and require the estimated value of the transition between adjacent subgoals fall into a specified range [Vmin V.ma The probability of the transition can be measured by the prior probability p(u) of the latent code u used for generating the subgoal from the affordance model, with low www-robbbilltyy transition latent corresponding implausible transitions. To summarize, the constrained optimization problem that our planning algorithm aims to solve is given by minimize subject to where we denote V&= V(λx-1, 2x).
|| Ex|1
(5)
Vmiin -
(1 2x) S Vmax, Pmin S p(ux), for As = I,..,,KK K
The cost function can thus be written
K
c(z0, Rg. ul:k) = ||اg - 2x|1 +|
Σ
(Vmin- Va)* + + 72 Vmmax) ) + ma log p(un))
(6)
where 7/, 72 and i7s are the Lagrange multipliers and we use (.)+ to denote max(-; 0) which trun- cates the negative values to zero. After sampling candidate subgoal sequences in the lossy repre- sentation space, we evaluate each candidate using the cost function. Model predictive path integral (MPPI) [12] is used to iteratively optimize the plan through importance sampling.
3.3 Fine-Tuning with Planned Subgoals
Finally, when the task is different from the training distribution, the learned offline policy may not generalize perfectly,
fine-tune with online interaction on target task specified by a goal image The affordance model is used to generate subgoals during this phase. Given the encoding of the goal image Zg. we optimize subgoals 2 according to Eq. 6. During trajectory, the subgoals are fed into the policy sequentially, executing action G₁ ~ 7 (als,,, âs) switching to new subgoal representation žą from the plan every h steps. During online fine-tuning, we freeze the weights of the state encoder and only update the policy, the value function, and the Q-function.
For online fine-tuning, we need to evaluate rewards. While GCRL algorithms can operate from only relabeled sparse rewards, shaped reward functions can assist in improving performance. But simple rewards such as the Euclidean distance between and g as in Sec. 2 can often be misleading for determining the goal-reaching success condition, since a small change in the environment can often result in huge difference in the corresponding image observation. On the other hand, the learned lossy representation is trained to make distances in the latent space meaningful, and can thus be more robust to such changes. We leverage this property during fine-tuning: in addition serving as the input to the policy and value functions, the learned representation is used to define a more informative reward function. Therefore, once the encoder & is pre-trained and fixed, we re-define the reward function to be the thresholded cosine similarity ((,g) = 1((2g =1)((I=gllllal)) < e} instead, which provides more informative reward signal during fine-tuning.
4 Experiments
In our experiments, we aim to answer the following questions: (1) Can the proposed method pro- duce meaningful subgoals in the learned latent space? (2) Can the subgoals generated by the lossy affordances facilitate learning new tasks in new scenes? (3) How much does offline data from other environments and tasks actually aid in learning the new task using FLAP?
5

---

4.1 Experimental Setup
Environments and tasks. Our experiments are con- ducted in table-top manipulation environments using low-level visuomotor control, with observations and goals corresponding to RGB images captured by a third-person camera. We use a real-world WidowX robot operating in kitchen-themed environments a frequency of 5 Hz through 7-DoF control. The action space is defined as the translation, rotation, and finger position of the gripper, without using any hard-coded motion primitive. In each target task, the
Reall World
Simulation
A
Initial Stale
fitekgool
initial
tinal goal
Tats B
Initial state
final goal
Initial state
final goal
Task C
inttal state
final poal
robot is asked to reach the goal in 200 steps by inter- acting with multiple objects unseen in the environ- ment. The target tasks include moving a metal pot from one stove another (Task A), moving colan- der onto the stove and picking and placing a stuffed toy into the pot (Task B), and placing sushi on a plate and then dropping a knife into pan in the sink (Task C). We also use a simulated environment for additional ablations and experiments based on prior work [13]. The simulated target tasks require sequential interactions with the environment such as opening/closing the drawer, sliding the cylinder, and picking and placing the block. To complete these tasks, the robot needs to reason about the objects' affordances and spatial relations from the visual inputs and strategically interact with the environment a feasible sequential order.
initial state
finall goal
Figure 3: Target tasks. Three multi-stage tasks are evaluated in the real world and the simulation. In each target task, the robot needs to strategically interact with the environment. The initial state and the desired goal state shown for each task.
Offlin Datasets. In the real world, we use the Bridge Dataset [8] which consists of 11 11,980 demon- stration trajectories in 10 different environments with a variety of objects. Note that this dataset was not collected or designed specifically for our method, but is sourced directly from prior work (where it was used for imitation learning). In simulation, the offline data consists of 12,000 trials in 6 scenes, which differ in terms of object texture and camera pose.
Prior methods and comparisons. We compare FLAP four other approaches. Model-Free Free is broadly representative of prior work that learns a goal-conditionel policy by directly feeding the final goal without using any subgoals [5, 13, 14]. LEAP [15], GCP [16]. and PTP [7] learn to plan subgoals in the original high-dimensional state space, instead of a learned lossy representation space as in our method. We also compare full model Ours (Broad Data), which is pre-trained on the previously discussed diverse prior datasets, with an ablation Ours (Target Data), which is trained only on subset of that data from our target domain. This baseline allows us to evaluate the importance of pre-training on a large and diverse dataset. All methods use the same neural network architectures, and all methods except for the ablation are pre-trained on the same prior data.
4.2 Comparative Results
Real-world evaluation. The results for a real-world evaluation with fine-tuning on three new tasks are presented in Table 1. The results show that across all three tasks, our full system consistently leads to significant improvement from fine-tuning, and indeed is the only method to produce im- provement across all tasks. Note that these tasks involve multiple distinct stages, presenting major challenge to standard goal-conditioned RL methods, as indicated by the poor performance of the Model-Free baseline both before and after fine-tuning. The PTP [7] baseline generally struggles in these domains due to their visual complexity, and from a cursory examination, the generated image- space subgoals are of low quality and blurry, which likely leads to poor subgoal guidance for the goal-conditioned policy. Without prior data pre-training both the policy and the affordance model generalize poorly and generally fail to make progress, which indicates that much of the benefit of our approach comes from being able to effectively leverage the diverse prior data pre-training.
Simulation. We report the average success rate during the online fine-tuning phase in Fig. 4. In all the three tasks, our full system consistently outperforms the prior methods and baselines. At the beginning of fine-tuning, the success rate close to zero for all the methods. Using the subgoals planned with the lossy affordance models, FLAP efficiently improves the success rate to 65.6%, 62.9%, and 89.5%. Without the planned subgoals (Model-Free), the robot often ends up performing the individual steps of the task in the wrong order, or is foreed to rely on random action exploration

---

Task A
Task
Task
3.0
0.6
0.6
0.3
Durm (limed Dana) Ourt (Tarpet Datai
9.5
0.4
2.4
07
0.2
0.0
0.8
0.0
20
120
240
200
20
spoch 140
200
180
1000
140
180
Figure 4: Quantitative comparison in simulation. The success rate averaged across rums during fine-tuning shown with the shaded region indicating the standard deviation. Each epoch consists of 5 trajectories.
Table 1: The real-world success rates after offline pre-training and then after online fine-tuning.
Model-Free
PTP
Ours (Target Data)
Ours (Broad Data)
Task
Offline
Online
Offline
Online
Offline - Online
Offline Online
Task
0.0% + 0.0%
0.0% + 25.0%
0.0% + 12.5%
12.5%
75.0%
Task B
0.0% 0.0%
0.0%
0.0%
25.0%
25.0%
25.0% 62.5 %
Task C
0.0% 0.0%
0.0% + 0.0%
12.5% 12.5%
25.0% + 50.0% %
7
VQ- VAE
FLAP (Ours)
Figure 5: Visualization of Learned Lossy Representations. compare our learned lossy representations with VQ- (AE using t-E. We sample 9 diverse trajectories from 3 different tasks (indicated by different col- ors) and plot the encoded trajectories. The trajectories and their initial states are indicted by the corresponding colors and number. FLA demonstrates to have more informative structure, as discussed in Sec. 4.3.
and fails to make progress. Especially in Task C, the baselines always attempt to open the drawer before moving the obstacle out of the way, which leads to failure. Prior methods such as PTP that generate goals in the high-dimensional image space, rarely find suitable subgoals for the novel tasks. When trained on only the target data (i.e., Ours (Target Data)), both the policy and the affordance model generalize poorly and generally fail to make progress, which indicates that much of the benefit of our approach comes from being able to effectively leverage the diverse prior data pre-training.
4.3 Learned Lossy Representations
We visualize the learned lossy representations in the real world using t-SNE [17] in Fig. 5, comparing them to a representation learned via a VQ- VAE model [18], as used by PTP [7]. Image observations in the target scene are sampled from the offline dataset and encoded using our state encoder and VQ-VAE, respectively. Each encoded representation is plotted we sample 9 trajectories with varied camera poses, lighting conditions, objects, and motions. We subsample the trajectories every 5 time steps, then plot the encoded trajectories overlaid on the SNE plots, with the number showing the start state of each one. Different colors indicate different trajectories and trajectories of the same semantic meaning are marked by similar colors.
grey dot. From 3 different tasks,
The embedding illustrates several interesting differences between our learned lossy embedding and the task task-agnostic VQ-VAE embedding. Although both embeddings exhibit some grouping or clumps (corresponding to visually and semantically distinct scenes), our representations appear to dedicate more representational power to functionally relevant degrees of freedom as demonstrated in in Fig. 5. Specifically, when using the VQ-VAE encodings, each trajectory collapses into different clique regardless whether they are from the same type or how much the scene changes each trajectory. However, when in our learned lossy representation space, each trajectory traverses a larger region. The second half of trajectories 1-3 and the first half of trajectories 4-6 are both moving objects from the right stove to the left stove, and interestingly we see that they intersect in the t-SNE plot.
7

---

5
Related Work
There is a growing body of work on learning policies from broad and diverse datasets in control and robotics using offline RL [19. 20, 21, 22, 23, 24, 25, 26, 27, 28]. Learned offline policies can also be fine-tuned with additional online interaction [29, 30, 31, 5, 32, 33]. While most of these methods focus on learning from offline data of the same task, this paper aims to utilize data of diverse environments and tasks to facilitate fine-tuning for novel tasks. A wide range of methods have been proposed for multi-task learning [34, 35] and meta learning [36, 37]. In contrast to these methods, we do not assume the task labels or explicit reward functions are provided in the offline data, instead adopting a goal-conditioned approach that can utilize diverse datasets without any explicit task labels or rewards. conl-conditioned RL [1. 2] aims to learn policies that can flexibly solve multiple tasks by taking different goals as inputs. Through goal relabeling [38]. goal- conditioned policy can be trained with self self-supervision in absence of meticulously designed reward functions [39, 40, 41, 42, 43, 44, 45, 46, 47, 48]. To guide the exploration of conlicondi policy, several methods use goals produced by a goal generator or affordance model that learns to capture the distribution of reachable goals [14, 49, 13, 5, 50]. Compared to this prior work, we show that can utilize broad datasets by utilizing representation learning within goal-conditioned RL. Moreover, by utilizing planning, we can significantly extend the capabilities of goal-conditioned RL. Recent works have also proposed to plan sequences of subgoals for learning long-horizon tasks [15, 3, 16, 7]. These methods successfully improve the performance of goal-conditioned RL, but have significant shortcomings in utilizing broad datasets to learn novel tasks. Nasiriany et al. [15] plans in the latent space of a variational autoencoder [10] for goal-conditioned RL in single environment, but does not handle diverse prior data. Nonparametric methods [3] utilize graph- based planning, which makes them difficult to adapt for learning unseen tasks from data where states visited for the new task have not been traversed. Instead of generating and planning subgoals in the original state space, our method learns a representation of goals to facilitate generalization across domains. A number of methods have been proposed for learning latent representations of states for control and robotics [51, 52, 53, 54, 55, 56, 57, 58, 59]. While most focus on learning compact representations for learning the policy or the dynamics model, our method learns representations in an offlin goal-conditioned reinforcement learning setting and uses the learned representations to facilitate subgoal planning across domains. Shah et al. [60]. Shah and Levine [61] use mutual information objective learns representation of the transition for selecting the goals to guide exploration. Instead of proposing subgoals from previously visited states prior distribution, our method learns a generative model of the latent goals to plan subgoals for target tasks do not exist in the offline data.
Conclusion
We presented FLAP, framework that leverages broad offline data to more effectively learn new temporally extended tasks. Our framework uses an affordance model to provide planned subgoals to guide a goal-conditioned policy, which is then finetuned for the new task. Our method trains lossy representation of states and goals from the offline data, which then provides compact inputs to the policy, facilitates subgoal planning. and shapes reward signals during the online fine-tuning process. Our framework can effectively utilize offline data collected from diverse environment and tasks, and because of the use of GCRL, it can be pre-trained on this offline data without any hand-specified reward signal. Our results show that FLAP significantly improves over alternative designs for such a system, and attains good results even in visually complex real-world settings.
Limitations. The main limitation of our method is that the offline data and the target task need to share enough similarity for effective knowledge transfer. First, the state spaces need to have the same dimension, otherwise the state encoder pre-trained on the offline dataset cannot be applied for the target task. Second, the offline data needs contain trajectories with structurally similar behaviors to those that are required to complete the new task, so that the affordance model can sample meaningful subgoals after being pre-trained on the offline data.
Acknowledgement. We acknowledge the support by the Office of Naval Research, ARL DCIST and AFOSR FA9550-22-
17-2-0181, CRA 191 ARO W911NFF21-1-00 like to thank Frederik Ebert, Dibya Ghosh, and Dhruv Shah for their constructive feedbacks. And would like to thank Abraham Lee for helping with the data collection.
We would
8

---

References
[T] L. Kaelbling. Learning to Achieve Goals. In ICCA,, 1993.
[2] T. Schaul, D. Horgan, K. Gregor, and D. Silver. Universal Value Function Approximators. In International Conference on Machine Learning (ICML), 2015.
[3] B. Eysenbach, R. Salakhutdinov, and S. Levine. Search on the Replay Buffer: Bridging Plan- ning and Reinforcement Learning. In Advances in Neural Information Processing Systems, 2019.
[4] J. Gibson. The Theory of Affordances. 1977.
[5] A. Khazatsky, A. Nair, D. Jing, and S. Levine. What Can Do Here? Learning New Skills by Imagining Visual Affordances. ICRA, 2021.
[6] A. A. Alemi, I. S. Fischer, J. V. Dillon, and K. P. Murphy. Deep variational information bottleneck. ICLR, 2016.
[7] K. Fang, P. Yin, A. Nair, and S. Levine. Planning to practice: Efficient online fine-tuning by composing goals in latent space. JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022.
[8] Ebert, Y. Yang. K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and S. Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109.13399 2021.
[9] K. Sohn, X. Yan, and H. Lee. Learning Structured Output Representation using Deep Condi- tional Generative Models. In Advances in Neural Information Processing Systems, 2015.
[10] D. Kingma and M. Welling. Auto-Encoding Variational Bayes. In International Conference on Learning Representations (ICLR), 2014.
[11] I. Kostrikov, A. Nair, and S. Levine. Offline Reinforcement Learning With Implicit Q- Learning. arXiv preprint a 10.06119, 2021.
[12] M. S. Gandhi, B. Vlahov, J. Gibson, G. Williams, and E. A. Theodorou. Robust Model Pre- dictive Path Integral Control: Analysis and Performance Guarantees. IEEE Robotics and Au- tomation Letters, 6:1423-1433 2021.
[13] A. Nair, Bahl, A. Khazatsky, V. Pong, G. Berseth, and Levine. Contextual Imagined Goals for Self-Super Robotic Learning. In Conference on Robot Learning (CoRL), 2019.
[14] A. Nair, V. Pong. M. Dalal, S. Bahl, S. Lin, and S. Levine. Visual Reinforcement Learning with Imagined Goals. In Advances in Neural Information Processing Systems, 2018
[15] S. Nasiriany, V. Pong, S. Lin, and S. Levine. Planning with Goal-Conditioned Policies. In Advances in Neural Information Processing Systems, 2019.
[16] K. Pertsch, O. Rybkin, F. Ebert, C. Finn, D. Jayaraman, and S. Levine. Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors. ArXiv, abs/2006. 13205, 2020.
[17] L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(11), 2008.
[18] A. van den Oord, O. Vinyals, and K. Kavukcuoglu. Neural Discrete Representation Learning. In Advances in Neural Information Processing Systems, 2017.
[19] S. Lange, T. Gabel, and M. Riedmiller. Batch Reinforcement Learning. In Reinforcement learning, pages 45-73. Springer, 2012.
[20] S. Fujimoto, D. Meger, and D. Precup. Off-Policy Deep Reinforcement Learning Without Exploration. In International Conference on Machine Learning (ICML), pages 2052-2062, 2019.

---

[21] A. Kumar, J. Fu, G. Tucker, and S. Levine. Stabilizing Off- Off-Policy Q-Learnigg via Bootstrap- ping Error Reduction. In Advances Neural Information Processing Systems, 2019.
[22] A. Gupta, A. Murali, D. Gandhi, and L. Pinto. Robot learning in homes: Improving general- ization and reducing dataset bias. ArXiv, 2018.
[23] C. Zhang, R. Kuppannagari, and Prasanna. BRAC+: Going deeper with behavior regu- larized offline reinforcement learning, 2021.
[24] A. Kumar, A. Zhou, G. Tucker, and Levine. Conservative Q-Learning for Offline Reinforce- ment Learning. arXiv preprint arXiv:2006.04779, 2020.
[25] S. Fujimoto and S. S. Gu. Minimalist Approach Offline Reinforcement Learning. arXiv preprint axXiv:2166006666 2021.
[26] A. Singh, A. Yu, J. Yang, J. Zhang, A. Kumar, and S. Levine. COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning. In Conference on Robot Learning (CoRL), 2020.
[27] A. X. Lee, C. Devin, Y. Zhou, T. Lampe, K. Bousmalis, T. Springenberg. A. Byravan, A. Abdolmaleki. N. Gileadi, D. Khosid, C. Fantacci, J. E. Chen, A. Raju, R. Jeong, M. Neunert, A. Laurens, S. Saliceti, F. Casarini, M. Riedmiller, R. Hadsell, and F, Nori. Beyond pick-and- place: Tackling robotic stacking of diverse shapes. In Conference Robot Learning (CoRL), 2021.
[28] A. X. Lee, C. Devin, J. T. Springenberg. Y. Zhou, T. Lampe, A. Abdolmaleki, and K. Bous- malis. How to spend your robot time: Bridging kickstarting and offline reinforcement learning for vision-based robotic manipulation. ArXiv, 2022.
[29] A. Nair, M. Dalal, A. Gupta, and Levine. Accelerating online reinforcement learning with offline datasets. ArXiv, (bs/2000.09339, 2020.
[30] A. Villaflor, J. Dolan, and J. Schneider. Fine-tuning Offline Reinforcement Learning With Model-Based Policy Optimization. Offline Reinforcement Learning Workshop at Neural Infor- mation Processing Systems, 2020., 2020.
[31] Y. Lu, Hausman, Y. Chebotar, M. Yan, E. Jang, Herzog, T. Xiao, A. Irpan, M. Khansari, D. Kalashnikov, and S. Levine. Aw-opt: Learning robotic skills with imitation and reinforce- ment at scale. ArXiv, abs/2111.05424,, 2021.
[32] S. Lee, Y. Seo, K. Lee, P. Abbeel, and J. Shin. Offline-to-nlliee Reinforcement Learning via Balanced Replay and Pessimistic E-Esemble. Conference on Robot Learning (CoRL), 2021.
[33] L. Meng, M. Wen, Y. Yang, C. Le, X. Li, W. Zhang, Y. Wen, H. Zhang. J. Wang, and B. Xu. Offline pre-trained multi-agent decision transformer: One big sequence model tackles all smac tasks. ArXiv, abs/21 12.02845, 2021.
[34] L. Pinto and A. Gupta. Learning to Push by Grasping: Using multiple tasks for effective learning. IEEE International Conference on Robotics and Automation (ICRA), 2017.
[35] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and Hausman. MT-Opt:: Continuous Multi-Task Robotic Reinforcement Learning at Scale. Conference on Robot Learning (CoRL), 2021.
[36] R. Dorfman and A. Tamar. Offline meta reinforcement learning. ArXiv, abs/2008.02598, 2020.
[37] E. Mitchell, R. Rafailov, X. B. Peng. S. Levine, and C. Finn. Offline meta-reinffrrcement learning with advantage weighting. In International Conference on Machine Learning (ICML), pages 7780-7791. PMLR, 2021.
[38] M. Andrychowicz, D. Crow, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, Abbeel, and W. Zaremba. Hindsight Experience Replay. In Advances Neural Information Processing Systems, 2017.
10

---

[39] P. Rauber, F. Mutz, and J. J. Schmidhuber. Hindsight policy gradients. In CoRR, volume abs/1711.0, 2017. URL http:p:://axxv.orgg//o
wew1////00000066..... pdf.
[40] Pong, M. Dalal, Lin, A. Nair, S. Bahl, and Levine. Skew-Fit: State-Covering Self- Supervised Reinforcement Learning- ArXiv, abs/ fbss/1903.0369. 2020.
[41] M. Fang, T. Zhou, Y. Du, L. Han, and Z. Zhang. Curriculum-guided Hindsight Experience Replay. In Advances in Neural Information Processing Systems, 2019.
[42] Y. Ding, C. Florensa, M. Phielipp, and P. Abbeel. Goal-conditioned Imitation Learning. In Advances in Neural Information Processing Systems, 2019.
[43] A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman. Relay Policy Learning: Solv- ing Long-Horizon Tasks via Imitation and Reinforcement Learning. In Conference Robot Learning (CoRL), 2019.
[44] H. Sun, Z. Li, X. Liu. D. Lin, and B. Zhou. Policy Continuation with Hindsight Inverse Dynamies. In Advances in Neural Information Processing Systems, 2019.
[45] B. Eysenbach, X. Geng, Levine, and R. Salakhutdinov. Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement. ArXiv, (bs//202..1189,, 2020.
[46] D. Ghosh, Gupta, A. Reddy, J. Fu, C. Devin, B. Eysenbach, and S. Levine. Learning to Reach Goals via Iterated Supervised Learning, arXiv: Learning, 2021.
[47] B. Eysenbach, R. Salakhutdinov, and S. Levine. C-Learning: Learning to Achieve Goals via Recursive Classification. ArXiv, abs/201 1.08909, 2021.
[48] Y. Chebotar, K. Hausman, Y. Lu, T. Xiao, D. Kalashnikov, J. Varley, A. Irpan, B. Eysen- bach, R. C. Julian, C. Finn, and S. Levine. Actionable models: Unsupervised offline rein- forcement learning of robotic skills. International Conference on Machine Learning (ICML), abs/2104.07749, 2021.
[49] K. Fang. Y. Zhu, Garg, S. Savarese, and L. Fei-Fei. Dynamies Learning with Cascaded Variational Inference for Multi-Step Manipulation. Conference on Robot Learning (CoRL), 2019.
[50] E. Chane-Sane, C. Schmid, and Laptev. Goal-Conditioned Reinforcement Learning with Imagined Subgoals. In International Conference Machine Learning (ICML), 2021.
[51] T. D. Kulkarni, A. Saeedi, S. Gautam, and S. J. Gershman. Deep successor reinforcement learning. ArXiv, abs/1606.02396 2016.
[52] C. Gelada, S. Kumar, J. Buckman, O. Nachum, and M. G. Bellemare. Deepmdp: tinuous latent space models for representation learning. International Conference on Machine Learning (ICML), 2019.
Learning con-
[53] A. Srinivas, M. Laskin, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning (ICML), 2020.
[54] A. X. Lee, A. Nagabandi, P. Abbeel, and S. Levine. Stochastic latent actor-critic: Deep rein- forcement learning with a latent variable model. Advances in Neural Information Processing Systems, 2019.
[55] W. Dabney, A. Barreto, M. Rowland, R. Dadashi, J. Quan, M. G. Bellemare, and D. Silver. The value-improvement path: Towards better representations for reinforcement learning. AAAI Conference Artificial Intelligence, 2020.
[56] A. Stooke, K. Lee, P. Abbeel, and M. Laskin. Decoupling representation learning from rein- forcement learning. International Conference on Machine Learning (ICML), 2020.
[57] O. Nachum, S. S. Gu, H. Lee, and S. Levine. Near-optimal representation learning for hierar- chical reinforcement learning. International Conference on Learning Representations (ICLR), 2018.

---

[58] J. Choi, A. Sharma, H. Lee. S. Levine, and S. S. Gu. Variational empowerment as representa- tion learning for goal-base boal-based reinforcement learning. ArXiv, abs/2106.0140k 2021.
[59] B. Eysenbach, R. Salakhutdinov, and S. Levine. Robust predictable control. In Advances in Neural Information Processing Systems, 2021.
[60] D. Shah, B. Eysenbach, G. Kahn, N. Rhinehart, and S. Levine. Recon: Rapid exploration for open-world navigation with latent goal models. Conference on Robot Learning (CoRL), 2021.
[61] D. Shah and S. Levine. Viking: Vision-basce kilometer-scale navigation with geographic hints. Robotics: Science and Systems (RSS), 2022.
[62] E. Coumans and Y. Bai. Pybullet: A python module for physics simulation for games, robotics and machine learning. http://pybullet.. org, 2016-2021.
12


Reviews:
[{"review_id": "xVNLnHfO1M", "reviewer": "Reviewer_jYk6", "summary": "This work uses a big video dataset to train an affordance model, which can be used to rollout virtual trajectories on a learned feature space. In addition, they design a cost function to evaluate the virtual trajectories' quality. With the help of above-mentioned tools, they can plan a high-quality path from the current state to the desired target on the learned feature space, and use the future states on the path as the subgoals to guide a goal-conditioned policy to control the robot. Their experiments show that their method can be viewed as a good offline-pretrained model. After online finetuning in new tasks, their model outperforms baselines by a large margin.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think the research direction that utilizes a large offline dataset to boost robotic performance is very promising. This paper introduces a novel and interesting method in this direction. Their theory looks reasonable. Their experiments are done on real robots and look solid and significant. The paper is well written and easy to follow.\n\nIn the limitation discussion, they briefly describe the importance of the similarity between the offline data and the online tasks. I think it is better to have some experiments to show how the similarity affects the performance. For example, create some target tasks with different similarity levels w.r.t. offline data, and analyze the failure cases.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think this paper proposes a novel and effective method in a very important research direction without obvious issues. Their experiment looks convincing. Therefore, I tend to accept this paper.\n\n**After Authors' Replies**\nThe authors addressed my questions reasonably and I'm satisfied with the research topic they target at, the novelty of their method, and their experiments. Therefore, I think this paper should be accepted and I increased my score.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "I’m curious about the required computational resources. As the model is trained with a big video dataset, and usually training with video is costly, it is important to describe the computational resource they use to train the model. For example, the number of GPUs, the training time, etc.\n\nIt will also be definitely interesting (but not necessary) to see if their method can be applied to the setting where the test robot arm and the robot arm in the offline dataset are different, as other labs or companies might not have the same robot arm as the dataset, and collecting new datasets are costly.\n\n"}}, {"review_id": "bBeelnW-IK", "reviewer": "Reviewer_xMTo", "summary": "This paper deploys a reinforcement learning (GCRL) framework, where a policy is trained to attain a specified goal (indicated as an image) from the current state. This is a very practical setting, where the agent can learn from offline dataset of images without explicit reward annotations. The proposed framework, Fine-Tuning with Lossy Affordance Planner (FLAP), leverages diverse offline data for learning representations, goal-conditioned policies, and core to the planning procedure, the affordances. Affordance models predict potentially reachable states that can serve as subgoal proposals for the planning process. This is useful when planning long-horizon tasks, subgoals can assist the learning by breaking down the original problem into easier short-horizon tasks. This also enables sharing the same dataset for learning various tasks that are sub-skills of the original task.\n\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- A framework for pre-training behaviors that can be composed to learn longer horizon tasks is a very promising approach and important to the community.\n- The results and sim2real experiments look promising.\n- The method itself, is a combination of existing work, but the overall approach is novel.\n\n## Weaknesses\n- The core contribution, specially with respect to existing work is not explicitly highlighted. There exists baselines, it is shown that they either fail to solve the tasks or perform poorly. However I had to briefly check those methods to decipher how they differ.\n- Certain assumptions is made with regards to the availability of the sub-task trajectories, and how *exactly* might match to the final task scenario. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The core underlying assumptions using this method is that they consider  tasks that require the robot to compose previously seen behaviors (e.g., pushing, grasping, and opening) in order to perform a temporally extended task that sequences these\nbehaviors in a new way (by mix and matching these skills).\n\nThis method performs pre-training in order to learn a lossy representations during the offline pre-training process for the goal-conditioned policy using a variational information bottleneck (VIB). Such representation learns to capture the minimal information needed to determine how a particular goal can be reached.\nThe learned affordance models, captures the distribution of reachable future states and recursively propose subgoals conditioned on the initial state of each task. The affordance model is a generative model $m(s'|s.u)$ where $u$ is the latent representation that captures information of the transition from current state to the goal s'. Its trained as a conditional variational decoder (CVAE) using evidence lower bound (ELBO) objective function. By adding a a variational information bottleneck (VIB) to the reinforcement learning objective, which constrains the mutual information $I(s_t; z_t)$ and $I(g; z_g)$ between the state space and the representation space. This is converted to an an unconstrained optimization problem by applying the bottleneck on the Q-function representation and directly selecting the Lagrange multiplier $\\alpha$\nresulting to the VIB objective.\n\n\n\n\n", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "I have the following questions/comments for the authors:\n\n- Missing reference for implicit Q-learning  (IQL) for training offline RL algorithm. Perhaps you can provide a brief overview. I had to look it up.\n- Figure 5: Visualization of Learned Lossy Representations. How does one know which region corresponds to what task and how were the sub-task trajectories followed? I wasn't sure if these trajectories were determined by hand or automatically somehow.\n- Perhaps some discussion about the baselines and how this method is expanding on them, or what is missing from existing methods would be useful to a reader unfamiliar with these methods. They are only referenced and the final results are shown (maybe you removed this due to lack of space, but for sure to be included in the full paper version).\n- Why is there such a vast difference between your method Ours(Broad data) and (Target Data) ? I understand this was trained on a a subset of that data from our target domain, but the Target Data almost does not learn at all. Which for me raises the question of generalizability of the (Broad data) version. I understand you run real robot experiments on real RGB images, but as far as I understood, there is no zero-shot generalization here, but rather fine-tuning on the real data?\n- For the limitation, the authors mention \"the state spaces need to have the same dimension, otherwise the state encoder pre-trained on the offline dataset cannot be applied for the target task.\"\nI dont see this as a huge problem, am I wrong? Since you are using images as inputs. Does the input also include proprioceptive states of the robot/task too?\n- I like this observation/motivation made by the authors \"By optimizing this objective, we obtain lossy representations that disentangle relevant factors such as object poses from irrelevant factors such as scene background.\" But is there a way to support this claim? Just by the results presented here, its not immediately clear to me that this is the case.\n"}}, {"review_id": "MC8zzNLVDmL", "reviewer": "Reviewer_qYWX", "summary": "This work investigates how to make use of large offline data for fast online adaptation in a goal-conditioned RL setting.\n\nThe main idea consists of:\n1) Learn a low dimensional representation for state/goal via an encoder (e.g. $z_s = \\phi(s)$ and $z_g=\\phi(g)$). To learn a robust representation, the authors propose to use information bottleneck, that minimize the mutual info between $z$ and $s$ while optimizing the RL objective.\n2) Learn an \"affordance\" model, basically an inverse dynamics model that given a pair $(s, s')$, predicts what \"action\" leads to $s'$ given $s$. The model is formulated as a conditional variational autoencoder (CVAE), where the latent $u$ plays the role of \"action\".\n3) Given an initial state $s$ and goal $g$, project them to latent space learned in 1), then using the learned affordance model in 2) to solve a planning problem. Additionally, some online data and reward shaping help fine-tune the performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "***Strengths***:\n1. The paper is well-written and the main idea is easy to follow.\n2. Each proposed step is straight-forward and reasonable to my opinion.\n\n***Weaknesses***:\n1. Decomposing everything to 3 separate steps plus online fine-tuning is a bit cumbersome, especially for every new task, the model needs to fine-tune everything except the encoder.\n2. While I have no doubt about step 1 and 2, I am a bit concerned about how practical is step 3. First, step 3 involves hyper-parameters like $V_\\text{min}$, $V_\\text{max}$, $p_\\text{min}$, $\\eta_{1,2,3}$, all of which have to be manually picked, possibly even for every new task. Also, I am a bit confused why in the constraint, we need both constraint on $V$ and constraint on $u$. It seems to me that both are playing a similar role.\n3. Moreover, as already mentioned by the authors, the assumption of generalization is strict, the new environment has to be very similar to the old ones for meaningful generalization. Given the heavy procedure, I am not sure whether the \"model-based\" approach here is worthy.\n\n***Other points***:\n1. Line 34, you mention that the goal is to solve temporal extended task in zero-shot fashion. But isn't the proposed work in a few-shot setting?\n2. Should CVAE be conditional variational **auto**encoder (line 89/167)? autoencoder is very different from encoder alone.\n3. Equation 4 seems problematic to me (please correct me if I am wrong). Here, $z_g$ should be $z'$? Also, if you take expectation of $z' \\sim \\phi(s')$ outside, then it should not be a KL inside?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think this paper proposes a reasonable pipeline for solving temporal extended tasks in a model-based fashion. The part I like is that it assumes GCRL setting, where manually designed reward is not necessary. But as I mentioned in the weakness section, I think there is some concern about how practical the method is. In general, I think the pros outweigh the cons so I recommend weak accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "1. Please go through equations again and answer my question regarding the ELBO.\n2. In Line 192, the author mentioned switching goals every $h$ steps. Goal switching is an important problem in option based learning works, but here it is assumed to using a heuristics. Why a constant $h$ can work here?\n3. Maybe I miss this part, but can the author provide some descriptions about how they choose hyperparameters and how hard to tune them?"}}]

Author rebuttal:

Decision: Accept (oral)