Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes to use a multi-task RL policy, trained offline with (i) data (state-action-state-reward tuples) for a collection of tasks, as a starting point to fine-tune, online, 2 new policies for a new, similar task (one policy to solve the task itself and one policy to autonomously reset the environment).
In case the new task is not sufficiently similar to the tasks for which the multi-task policy was pre-trained, (ii) additional data from operator-guided demonstrations are also used.
The approach is demonstrated both in real-world experiments and simulation for 6 pick-and-place tasks with a robotic manipulator.
The results in real-world experiments show how online fine-tuning improves the success rate of the multi-task policy on new tasks.
The paper interestingly brings together 3 ideas (offline learning, multi-task transfer learning, and learned automated reset policies) into a single approach.
strengths_and_weaknesses: The paper makes simultaneous use of three very interesting and promising methods (offline learning, multi-task transfer learning, and learned automated reset policies), and treats the one-hot task representation of the multi-agent policy as a continuous, learnable parameter that also represents the task embedding of new tasks (and the reset policy), although not entirely new, is a neat approach to address the paper's problem.
Besides the marginal/incremental novelty, one possible weakness of the paper is that it seems that the new tasks need to be such (similar enough) that the old multi-task policy can already achieve success. The role of human demonstrations (initially presented as optional, then used for so-called out-of-distribution new tasks) should be better introduced and more formally justified.
The way data and results are presented in the tables should be improved for clarity and stand-alone readability
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper is easy to read, enjoyable, and summarizes very interesting ideas for real-world robotics RL. My main concerns are the lack of novelty and the not overly clear presentation of the real-world experimental results.

issues: - I would recommend being very clear from early in the paper (e.g. Fig 1) and what is the motivation for the use of human demos and how they play into the learning pipeline, for what tasks, etc. (the explanation comes into multiple parts in sections 4 and 5)
- If extra space is needed (for clarification, additional results), Section 3 is very textbook-like and could be removed
- I would add a clarifying statement about frequent vs unfrequent and automated vs manual resets
- Provide some numerical figure when saying that the scripted policy fails "very frequently"
- Tables 1 and 2 should be improved, e.g. clarifying in the captions what the ratios stand for, separating in and out of distributions tasks, etc.
- line 300, should it be 360 instead of 600? what was the "offline only" zero-shot generalization performance?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Real-world robotic reinforcement learning (RL) typically requires time-consuming data collection and frequent human intervention to reset the environment. The offline-RL approach provides a way to utilize previously collected data to bootstrap the learning process so that a new task can be learned with a small number of interactions. In this work, the authors study a problem where such offline data is available from several tasks (along with their task labels). The data may be collected with optimal or sub-optimal policies.  The authors propose a complete system for extracting useful skills from prior data and applying them to learn new tasks autonomously. When faced with a new task, the proposed system adapts previously learned skills to quickly learn to both perform the new task and return the environment to an initial state, effectively performing its own environment reset. The authors show the effectiveness of their system with extensive physical and simulated experiments.
strengths_and_weaknesses: Strength:

- The paper addresses an important problem of data efficiency and reset-free learning for real robotic applications.
- The paper clearly mentions the prior works on which the proposed system is based.
- Extensive experimentation to assess the effectiveness of the system both in simulated and physical environments.

Weakness:

The main weakness of the paper is that it is a bit difficult to see what the actual contribution of the paper is and how different the approach is compared to the previously published work. Several prior works already demonstrated how online RL can be accelerated with prior offline data. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: in my opinion, the work is not significantly different from the prior work. However, the paper focuses on an important problem and demonstrated the effectiveness of their proposed approach through experimentation.
issues: Maybe the contribution of the paper could have been clearly mentioned instead of "demonstrating already known" facts as a contribution. For instance, we already know using prior data makes RL samples efficient, there is nothing to demonstrate here again in my opinion.

Line 65: "we adopt the strategy learning a single policy conditional on a space of tasks" => we adopt the strategy *of* learning a single policy condition*ed* on a space of tasks.

There are several citations that are now published. But the arxiv versions are still cited on the paper. Please use the published versions. For instance, [31] should be ICRA 2021. 

line 246: Typo here "...policy to collect to collect data for...."
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper discusses a system to leverage prior data from related but distinct tasks to increase learning efficiency and generalization in new task. The system works in two steps: 1. Using offline data with consisting of different tasks with task ids to learn a task-conditional policy using offline RL. 2. Optimize task-conditioning embedding during deployment for the new task. To reduce the human supervision involved in resets the method proposes to extract two different embeddings one that accomplishes the tasks and the other that resets the task. The experiments are shown in both real world robot and simulation.
strengths_and_weaknesses: Strengths:

1. The paper presents a relatively novel combination of existing known modules for learning with offline data and learning reset policies.
2. The paper evaluates the results on real world and simulated domains and provide comprehensive results to show the validity of the method. The experiments demonstrate the superior generalization ability of the system arising as a result of learning from multiple tasks and increased learning efficiency.

Weaknesses:

1. Limited baselines: The paper evaluates ARIEL on the real world tasks but does not present a comparison with any baseline. Potential baselines to be compared to are: [2,29,31,32,33](references from paper) and common RL baselines which are more suited to sparse rewards like HER[1,2,3] (references below).
2. The experiments use a short horizon of M=20 which is small for a number of locomotion and navigation tasks. It is not clear how this method would scale with horizon since the start-state distribution for learning the reset policy can grow exponentially.

[1] :Hindsight Experience Replay [Marcin Andrychowicz] et al

[2]: [Replacing rewards with examples: Example-based policy search via recursive classification],Eysenbach et al

[3] [C-Learning: Learning to Achieve Goals via Recursive Classification], Eysenbach et al
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper presents a novel combination of existing modules for autonomous robot learning and demonstrates that near-autonomous learning is possible with similar tasks in the short-horizon manipulation domains. The paper can be strengthened by more extensive comparisons to baselines to properly situate how their work progresses the field.
issues: 1. Limited baselines: Can the authors present baselines for their real-world and simulated robot experiments. If the comparisons are not feasible, can the authors describe in the paper what makes such a comparison not possible.
2. Technical novelty: The work seems to borrow heavily from previous works [8,31] (references from paper) on reset free learning and offline multitask learnings. Can the authors comment on the significance of the new proposed methods in comparison to previous work?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Pre-training with large offline data and fine-tuning on a small amount of online data is a sample-efficient way of learning a new task. However, most recent approaches require explicit environment resetting for online data collection. This work proposes to minimize manual resetting of the environment when learning a new task in the real world leveraging offline multi-task data. First, from the offline multi-task data, the proposed method, ARIEL, learns a multi-task policy conditioned on a task embedding $\mathbf{z}$ using offline RL. Then, during the online fine-tuning phase, ARIEL searches for a task embedding for the forward task as well as another task embedding for the backward task (environment reset). ARIEL iterates the forward and backward tasks, which ideally resets the environment and performs the target task without human intervention. The real-world experiments show that ARIEL achieves more than 70% success rates with only 5 to 30 manual resets.
strengths_and_weaknesses: ### Strenghts

* The idea of utilizing a multi-task policy for both forward and backward (reset) policies makes sense and the paper empirically shows its effectiveness.

* The real-world experiments are exhaustively conducted with many different objects and in three different environment setups.

### Weaknesses

* In RL, covering diverse initial states is important for robustness and generalization of the learned policy. However, without explicit randomized resetting, the proposed backward controller may learn to reset to a few (easy to reset) initial states, which maximizes its own reward. It would be great if the authors discuss this potential issue.

* The proposed alternating scheme is appealing in automating online reinforcement learning with less human intervention. However, one of the critical assumptions in this work is that both forward and backward controllers can be achieved with a few random shootings. The reviewer is wondering whether this is a practical assumption for robot learning.

* To show the sufficient difference between training tasks and testing tasks, another interesting point could be freezing the policy after pre-training and only finding appropriate $\mathbf{z}$ using CEM. If this works as well as the proposed method, it could mean the prior data is too similar to the testing tasks, which means the experimental setup might be too easy.

* The proposed method assumes a task embedding corresponds to a task. However, in many cases, the offline data trajectories can be compositional (combination of multiple behaviors) and the target task may require stitching fo multiple of these trajectories. The proposed method does not seem trivial to scale to this common scenario.

### Questions and suggestions

* Figure 2b is too small, compact, and not descriptive without enough explanation. It would be better to include only two examples (one from in-distribution and another one from out-of-distribution tasks) and provide sufficient descriptions.

* L242-243: placing the ring on the peg does not seem to require "greater precision" than the prior tasks.

* In L244-245, while the paper claims that the proposed method can utilize data from any reasonable source, the proposed method requires reward annotation, which is not available in reward-free demonstrations (e.g. play data, multi-task expert demonstrations). Given that the proposed method can potentially work with sparsely-labeled data using offline RL, it may not be a big issue, but it should be clarified.

* In L253, what does it mean that rewards are specified at the end of the trajectory?. The reviewer thinks rewards for the target demonstrations should be all 1 at the end. Or, does it mean that the reward is labeled depending on the forward or backward tasks?

* What is the definition of "trial" in Section 5 and 6? Does this mean $M$ steps or $2M$ steps?

* In Table 1, why do in-distribution tasks perform worse than out-of-distribution tasks?

* In Table 1, including a baseline with episodic reset as an upper bound can be helpful to understand how much "reset" information is important for learning these tasks. 

* In Table 1 and Table 2, which experiment uses 40 downstream task demonstrations? This is not clear.

* The explanation in L291-L293 about the experiments for Table 2 is a bit confusing. Are the testing tasks unseen during online fine-tuning? More specifically, "Put X in Container" tasks are evaluated using a policy pre-trained from multi-task prior data and fine-tuned for "Put Tiger on Lid"? 

* In Table 2, how many trials were used for training "Single Task Data Only"?

* In A.2.1, there are three different environments, "Tray Container", "Tray Drawer", and "Kitchen". For offline training of ARIEL, are data from all three environments used together? or data from the same environment with the target task is used for offline training?

* In Algorithm 1, "steps" should be reset to 0 and new $\mathbf{z}$ should be sampled, like L9, when the task direction $d$ is reversed in L14. 

* In Algorithm 1, L2, what does it mean by "fit $q_f(\mathbf{z})$ and $q_b(\mathbf{z})$ to offline task indices?

* In Algorithm 1, L5, $d$ should be also reset to $f$ as the environment state is reset. Simply moving L3 to L5 can work.

* How much randomness in object poses and robot poses for each trial of evaluation in Table 1 and 2?

* In Table 1, how is "Offline Only" evaluated? More specifically, which $\mathbf{z}$ is used for this experiment? And, how can out-of-distribution tasks have such high success rates only with offline only?

* A bit more detailed description of the task indices $\mathbf{z}$ can be helpful to understand the proposed method. The paper mentions that $\mathbf{z}$ is a task embedding from Hausman et al. in L159, but more details about how it is trained can be elaborated in the paper.

* It could be interesting to see how success rates of the forward and backward controllers over the online training. 

* Many references are from arXiv. Please cite their conference or journal versions.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This paper proposes a (semi) autonomous reinforcement learning leveraging offline multi-task data, and the empirical results demonstrate its effectiveness and autonomy (less than 30 manual resets required). Although the experimental tasks look a bit simple and the presentation should be improved, the proposed idea is novel and the results show some potential for autonomous RL.


**After rebuttal**

Thank the authors for the response. The rebuttal clarified all my confusion, but the weaknesses are still there and a more in-depth discussion about the reviewers' concerns would make the paper stronger. Overall, I believe the proposed method shows an interesting research direction and empirically demonstrated its potential in real-world scenarios; thus, I would like to accept this paper.
issues: Please refer to the above weaknesses and questions.
review_title: null

Meta-review: metareview: Please check the comments of the reviewers in detail.

### Strengths
- interesting combination of existing ideas
- experiments with a real system (low-cost arm)

### Weaknesses
- not highly novel: combination of existing ideas
- comparison with the state-of-the-art could be stronger: using prior data has been explored several times in the past
- statistics need to be clarified (number of replicates, including for the real-world experiments, evaluation of variance, etc.)
- assumption that both forward and backward policies can be learned with a few random shoots, which seems unlikely in most scenarios

#### Post-rebuttal remarks
I would like to thank the authors for their efforts in answering the comments. The main concern left is the novelty & potential impact, but, overall, the reviewers liked the paper.
confidence: 3: The area chair is somewhat confident
Decision: Accept (Poster)