Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper, titled REBOOT, proposes a method for reusing data for bootstrapping real-world dexterous manipulation, and creates an autonomous pipeline to reset and collect data that can be used across multiple tasks for training task-specific policies. This includes skills such as in-hand manipulation of a triangular valve, a T-shaped pipe, and a toy football. Their method trains a reset policy (via imitation learning) and an RL policy in tandem to train a system that can autonomously collect data in the real world.
review: Overall, the real world demonstrations of this robot platform and approach are one of the strongest points of this method. They seem to show fairly strong results in the four tasks they propose, learning within a few hours. However, for a method like this, training in simulation for speed of testing and verifying that their method can scale to the chosen tasks is fairly critical. This is especially true since some of the experimental runs require hours of real-time training to evaluate a single model, which is bound to be bottle-necked by robot hours of operation. It is highly possible that the unusual drops in performance as more training time progresses In Figures 5 and 6 is an artifact of the object resetting approach used, but this goes unmentioned in the experimental section.
Additionally, the link does not seem to work.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: After the rebuttal period, the authors produced more results indicating that their approach is not handicapped by the amount of training they performed on the real robot. After adding simulated experiments, they confirmed that their method's results were not limited by the robot hours, and that REBOOT has a stronger effect on the learned policy success rate than policy transfer and fine-tuning. For this reason, I recommend the paper as a weak accept, as it produces some interesting insights for reset-free methods and fine-tuning for dexterous grasping tasks.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This practical study explores the role of replay buffer initialization in accelerating RL skill learning, particularly by leveraging prior data from other tasks.

More specifically, this work also proposed a modified visual reward learning algorithm considering the random replay buffer initialization, which is to keep the VICE reward from other tasks static. 

Real-world experiments validate the effectiveness of this simple yet powerful technique, demonstrating significant improvements in sample efficiency when learning new skills, even across different object domains.
review: This work presents an interesting and practical exploration, confirming that leveraging prior non-task specific data can significantly enhance skill-learning efficiency in real-world dexterous manipulation tasks.

One notable limitation is its confinement to image-based domains, where different tasks share the same observation space (dimension). This commonality enables the utilization of prior data effectively, possibly due to the properties of image signals. Additionally, the existence of shared visual features in the same dexterous manipulator allows for the transfer of visual-based control skills across various tasks.

However, to further enhance the work's credibility, a more comprehensive analysis could be beneficial. Additional details regarding this aspect can be found in the questions section. Addressing these points would strengthen the overall contribution of the study.

For details please refer to the question part.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I really appreaciate the idea of using prior non-tasl related data to accelerate the skill learning. It also surprisingly accelerate the skill learning practically. However, more ablation studies are need to make this work complete.

-----------------------------------------

The authors added the additional ablation studies and comparison experiments. After the Rebuttal I would change my rating to accept
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper proposes a novel system for learning real-world dexterous in-hand manipulation policies in an end-to-end fashion (REBOOT). By leveraging a combination of buffer initialization and reset-free learning techniques, the proposed method is able to achieve desired target goals with greater sample efficiency compared to learning policies from scratch. The method is evaluated against three different objects and is shown to have success when transferred across the different objects.
review: Strengths:

- The paper is well written and easy to understand.
- The quantitative results showcase the sample efficiency of the method when transferred to new goals and objects.
- The end-to-end, reset free training paradigm is exciting to see, and promising for learning real-world robot dexterous manipulation at scale.
- The method requires minimal human intervention and engineering, and does not require hand-engineered rewards.

Weaknesses:

- It is not clear what is the paper’s most significant contribution. It appears that minor variations of multiple pre-established methods have been compiled into a single end-to-end method for learning dexterous manipulation, which is impressive, but for such a systems paper it would be more ideal if there were stronger empirical results showcasing the strength of the proposed system. 
- One of the paper’s major focuses is the relative sample efficiency compared to learning from scratch; however, this has already been shown in the prior work cited, and while the proposed method utilizes a novel implementation (images as input applied to robot manipulation domain), that alone does not seem to warrant a strong technical contribution.
- A clear limitation is that the proposed method still requires at least 1 task to be learned from scratch (i.e.: the task that will then seed the resulting data buffer). It is unclear how much this limits the scope of the method – given that even for relatively simple geometries (e.g.: the football), both learning from scratch and even applying REBOOT results in relatively poor success, it is unclear whether the method can be expected to perform well across a wide range of object geometries (especially given that the other two experimental objects share very similar geometries).
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The proposed method clearly exhibits greater sample efficiency compared to learning individual in-hand policies from scratch by leveraging prior task data. Moreover, the method requires minimal assumptions, and does not require manual resets or rewards. However, the overall novelty of the system remains unclear, with more analysis and discussion needed to better justify the technical contributions of the overall system.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper introduces an approach to enhance the efficiency of reinforcement learning (RL) in tackling complex in-hand reorientation tasks. It addresses various challenges associated with deploying RL methods in real-world scenarios, such as sample efficiency, environment resets, and reward signals. To address these issues, the approach reuses prior data to initialize the replay buffer to improve sample efficiency. It also leverages a behavior cloning (BC) policy to reset the environment for continuous real-world training. Additionally, an adversarial discriminator is used to obtain rewards for online RL training in the real world. The authors conduct real-world experiments on in-hand manipulation of a 3-prong object, a T-shaped pipe, and a football to demonstrate the effectiveness of the approach.
review: Strengths:
* This paper introduces a new system that learns in-hand reorientation skills autonomously with RL.
* Extensive, convincing hardware experiments are shown.
* The paper is well-structured and written.

Weaknesses:
* In the experimental results section, the emphasis is mainly on showcasing the improvement in sample efficiency resulting from the replay buffer initialization with prior data, which is already known in prior works. It would be interesting to see if a learned policy can handle unseen objects or reorient different objects.
* No state-of-the-art comparison.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: After careful consideration of the points raised, I suggest a weak reject. The main concern is the focus of the experiments which primarily demonstrate the benefits of using replay buffer initialization with prior data in terms of sample efficiency, a known insight from prior works. Moreover, the paper lacks insights on how to enhance in-hand reorientation performance. Additionally, it is essential to include a comparison with state-of-the-art approaches to evaluate the proposed method's advancements. Without such a comparison, it is difficult to determine the method's competitiveness and potential impact in the field.
review_title: null

Meta-review: This paper presents an efficient system for learning dexterous manipulation skills through sample-efficient RL and replay buffer bootstrapping. After discussion and clarification during the rebuttal phase by the authors, all the reviewers have issued 3 weak accepts and 1 accept.
Decision: Accept (Poster)