Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work investigates how to make use of large offline data for fast online adaptation in a goal-conditioned RL setting.

The main idea consists of:
1) Learn a low dimensional representation for state/goal via an encoder (e.g. $z_s = \phi(s)$ and $z_g=\phi(g)$). To learn a robust representation, the authors propose to use information bottleneck, that minimize the mutual info between $z$ and $s$ while optimizing the RL objective.
2) Learn an "affordance" model, basically an inverse dynamics model that given a pair $(s, s')$, predicts what "action" leads to $s'$ given $s$. The model is formulated as a conditional variational autoencoder (CVAE), where the latent $u$ plays the role of "action".
3) Given an initial state $s$ and goal $g$, project them to latent space learned in 1), then using the learned affordance model in 2) to solve a planning problem. Additionally, some online data and reward shaping help fine-tune the performance.
strengths_and_weaknesses: ***Strengths***:
1. The paper is well-written and the main idea is easy to follow.
2. Each proposed step is straight-forward and reasonable to my opinion.

***Weaknesses***:
1. Decomposing everything to 3 separate steps plus online fine-tuning is a bit cumbersome, especially for every new task, the model needs to fine-tune everything except the encoder.
2. While I have no doubt about step 1 and 2, I am a bit concerned about how practical is step 3. First, step 3 involves hyper-parameters like $V_\text{min}$, $V_\text{max}$, $p_\text{min}$, $\eta_{1,2,3}$, all of which have to be manually picked, possibly even for every new task. Also, I am a bit confused why in the constraint, we need both constraint on $V$ and constraint on $u$. It seems to me that both are playing a similar role.
3. Moreover, as already mentioned by the authors, the assumption of generalization is strict, the new environment has to be very similar to the old ones for meaningful generalization. Given the heavy procedure, I am not sure whether the "model-based" approach here is worthy.

***Other points***:
1. Line 34, you mention that the goal is to solve temporal extended task in zero-shot fashion. But isn't the proposed work in a few-shot setting?
2. Should CVAE be conditional variational **auto**encoder (line 89/167)? autoencoder is very different from encoder alone.
3. Equation 4 seems problematic to me (please correct me if I am wrong). Here, $z_g$ should be $z'$? Also, if you take expectation of $z' \sim \phi(s')$ outside, then it should not be a KL inside?
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I think this paper proposes a reasonable pipeline for solving temporal extended tasks in a model-based fashion. The part I like is that it assumes GCRL setting, where manually designed reward is not necessary. But as I mentioned in the weakness section, I think there is some concern about how practical the method is. In general, I think the pros outweigh the cons so I recommend weak accept.
issues: 1. Please go through equations again and answer my question regarding the ELBO.
2. In Line 192, the author mentioned switching goals every $h$ steps. Goal switching is an important problem in option based learning works, but here it is assumed to using a heuristics. Why a constant $h$ can work here?
3. Maybe I miss this part, but can the author provide some descriptions about how they choose hyperparameters and how hard to tune them?
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This work uses a big video dataset to train an affordance model, which can be used to rollout virtual trajectories on a learned feature space. In addition, they design a cost function to evaluate the virtual trajectories' quality. With the help of above-mentioned tools, they can plan a high-quality path from the current state to the desired target on the learned feature space, and use the future states on the path as the subgoals to guide a goal-conditioned policy to control the robot. Their experiments show that their method can be viewed as a good offline-pretrained model. After online finetuning in new tasks, their model outperforms baselines by a large margin.
strengths_and_weaknesses: I think the research direction that utilizes a large offline dataset to boost robotic performance is very promising. This paper introduces a novel and interesting method in this direction. Their theory looks reasonable. Their experiments are done on real robots and look solid and significant. The paper is well written and easy to follow.

In the limitation discussion, they briefly describe the importance of the similarity between the offline data and the online tasks. I think it is better to have some experiments to show how the similarity affects the performance. For example, create some target tasks with different similarity levels w.r.t. offline data, and analyze the failure cases.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I think this paper proposes a novel and effective method in a very important research direction without obvious issues. Their experiment looks convincing. Therefore, I tend to accept this paper.

**After Authors' Replies**
The authors addressed my questions reasonably and I'm satisfied with the research topic they target at, the novelty of their method, and their experiments. Therefore, I think this paper should be accepted and I increased my score.

issues: I’m curious about the required computational resources. As the model is trained with a big video dataset, and usually training with video is costly, it is important to describe the computational resource they use to train the model. For example, the number of GPUs, the training time, etc.

It will also be definitely interesting (but not necessary) to see if their method can be applied to the setting where the test robot arm and the robot arm in the offline dataset are different, as other labs or companies might not have the same robot arm as the dataset, and collecting new datasets are costly.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper deploys a reinforcement learning (GCRL) framework, where a policy is trained to attain a specified goal (indicated as an image) from the current state. This is a very practical setting, where the agent can learn from offline dataset of images without explicit reward annotations. The proposed framework, Fine-Tuning with Lossy Affordance Planner (FLAP), leverages diverse offline data for learning representations, goal-conditioned policies, and core to the planning procedure, the affordances. Affordance models predict potentially reachable states that can serve as subgoal proposals for the planning process. This is useful when planning long-horizon tasks, subgoals can assist the learning by breaking down the original problem into easier short-horizon tasks. This also enables sharing the same dataset for learning various tasks that are sub-skills of the original task.


strengths_and_weaknesses: ## Strengths
- A framework for pre-training behaviors that can be composed to learn longer horizon tasks is a very promising approach and important to the community.
- The results and sim2real experiments look promising.
- The method itself, is a combination of existing work, but the overall approach is novel.

## Weaknesses
- The core contribution, specially with respect to existing work is not explicitly highlighted. There exists baselines, it is shown that they either fail to solve the tasks or perform poorly. However I had to briefly check those methods to decipher how they differ.
- Certain assumptions is made with regards to the availability of the sub-task trajectories, and how *exactly* might match to the final task scenario. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The core underlying assumptions using this method is that they consider  tasks that require the robot to compose previously seen behaviors (e.g., pushing, grasping, and opening) in order to perform a temporally extended task that sequences these
behaviors in a new way (by mix and matching these skills).

This method performs pre-training in order to learn a lossy representations during the offline pre-training process for the goal-conditioned policy using a variational information bottleneck (VIB). Such representation learns to capture the minimal information needed to determine how a particular goal can be reached.
The learned affordance models, captures the distribution of reachable future states and recursively propose subgoals conditioned on the initial state of each task. The affordance model is a generative model $m(s'|s.u)$ where $u$ is the latent representation that captures information of the transition from current state to the goal s'. Its trained as a conditional variational decoder (CVAE) using evidence lower bound (ELBO) objective function. By adding a a variational information bottleneck (VIB) to the reinforcement learning objective, which constrains the mutual information $I(s_t; z_t)$ and $I(g; z_g)$ between the state space and the representation space. This is converted to an an unconstrained optimization problem by applying the bottleneck on the Q-function representation and directly selecting the Lagrange multiplier $\alpha$
resulting to the VIB objective.





issues: I have the following questions/comments for the authors:

- Missing reference for implicit Q-learning  (IQL) for training offline RL algorithm. Perhaps you can provide a brief overview. I had to look it up.
- Figure 5: Visualization of Learned Lossy Representations. How does one know which region corresponds to what task and how were the sub-task trajectories followed? I wasn't sure if these trajectories were determined by hand or automatically somehow.
- Perhaps some discussion about the baselines and how this method is expanding on them, or what is missing from existing methods would be useful to a reader unfamiliar with these methods. They are only referenced and the final results are shown (maybe you removed this due to lack of space, but for sure to be included in the full paper version).
- Why is there such a vast difference between your method Ours(Broad data) and (Target Data) ? I understand this was trained on a a subset of that data from our target domain, but the Target Data almost does not learn at all. Which for me raises the question of generalizability of the (Broad data) version. I understand you run real robot experiments on real RGB images, but as far as I understood, there is no zero-shot generalization here, but rather fine-tuning on the real data?
- For the limitation, the authors mention "the state spaces need to have the same dimension, otherwise the state encoder pre-trained on the offline dataset cannot be applied for the target task."
I dont see this as a huge problem, am I wrong? Since you are using images as inputs. Does the input also include proprioceptive states of the robot/task too?
- I like this observation/motivation made by the authors "By optimizing this objective, we obtain lossy representations that disentangle relevant factors such as object poses from irrelevant factors such as scene background." But is there a way to support this claim? Just by the results presented here, its not immediately clear to me that this is the case.
review_title: null

Meta-review: metareview: Below is a summary of the strengths and weaknesses of the paper, according to the reviewers.

Strengths:
- The proposed method is novel and interesting.
- The proposed method performs well in experiments.
- Real-world experiments are conducted.
- The paper is well written and easy to follow.

Weaknesses:
- In Step 3, there are a number of hyper-parameters. If these hyper-parameters must be tuned for every new task, then the method could be very impractical.
- The offline data and data for the target task must come from very similar tasks. This is already highlighted by the authors in the paper's limitations section, so it is not a major weakness in the paper itself. However, it would have been helpful to see experiments exploring how different levels of similarity affect performance.
- It is not always clear what the core difference is between the method and the baselines in the evaluation.

In your rebuttal, please address the above weaknesses, as well as any other concerns or questions raised by the reviewers.

---------

After rebuttal:

Reviewers retained their strong opinion of the paper following the rebuttal. This is topical paper focussing on offline reinforcement learning, with a novel technical contribution and interesting real-world experiments for evaluation. One key limitation -- that the offline data needs to be from a similar distribution to data in the target domain -- is a general current limitation of the field of offline reinforcement learning, rather than a specific limitation of this paper, and is an open problem. Therefore, there are no major weaknesses in this paper, and it clearly provides an important contribution to the field.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Oral)