Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The authors present a new way of collecting robot demonstrations for manipulation using an augmented reality (AR) framework, that requires no real-world robots and can be done conveniently by users in their personalized setups. The proposed methodology is compared to alternatives (such as using VR, kinesthetic demos etc.) in a user study, and is shown to be as good as expensive kinesthetic demos while being much faster to collect data, with the added benefits of customization and not requiring a real robot.

review: The paper is very well-written and the findings are crisply and convincingly demonstrated. Overall, I really like the idea of generating cheap simulation-esque data for training policies and transferring to the real setup with a small amount of real data, and this seems like a great way to get realistic data without having to construct complicated sim scenarios of all tables/objects etc. The user study is very informative and answers any doubts I had getting into this paper, and I wish more robot learning papers did this.

I do have a few concerns that I hope the authors can address:
- A fundamental limitation of the paper is the simplistic nature of tasks considered, commonly known as “pose matching”. While the findings and proposal are definitely interesting, I find the task to be of fairly limited utility, and this may be the biggest deterrent to more research labs/groups adopting this setup. What would it take to use this methodology for a more realistic task, say pick-and-place, or something more challenging, such as dexterous manipulation? While demonstrating this may be out of scope for the current paper, it is fundamental to at least address these limitations and discuss how this can be extended to more interesting/useful tasks. Otherwise, for the simple task of “reach object X”, much simpler approaches can work (e.g. have a few-shot detector or SAM pick the object and servo to it). This does not take away from the merits of the paper, but somehow feels incomplete.
- Another fundamental assumption to challenge is the dependency on end-effector control, even if the demos could be made more expressive and useful. Fundamentally defining demo trajectories with end effector positions, followed by motion planning, limits the variety of behaviors that can be demonstrated, and hence learned, by the system. This really needs to be addressed better.
- Is there any plan to open source the system design components and the iOS application developed for this purpose? Assuming the primary contribution of the paper is the validity of such an approach for data collection, it would be very beneficial for other researchers to use this directly. Otherwise, its a heavily engineered system with many moving parts, and may be a reproducibility nightmare (not more than actually dealing with robots, granted).
- Could the authors share more information on the demographics of the users studied? I can imagine the SUS scores varying greatly between CS grad students v/s STEM undergrads v/s random turkers etc. using the system, and having access to this variable seems important for readers to interpret the message.
- While not fully related, it may be good to acknowledge work in the area of third-person imitation for human-to-robot transfer of demonstrations. This may seem like a fairly related area of research (at least at the surface) and discussing the differences in assumptions and objectives would be useful.
- Small typo: L171-172 seems to reference Figure 4, rather than Figure 3.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This is a great paper with a simple and convincing proposal. However, the authors should address the limitations of regarding the simplicity of the task and end-effector specification limiting the variety of learnable behaviors, for the system to be of use to the broader community. I really like this idea and am willing to upgrade my recommendation after further discussions in the rebuttal period.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper introduces AR2-D2, a system for collecting human demonstrations in robot learning. Unlike existing methods that require specialized training and real robots, AR2-D2 is an iOS app that allows users to record videos of themselves manipulating objects while capturing essential data for training real robots. The key contributions of AR2-D2 are threefold: it eliminates the need for specialized training, it does not require real robots during data collection, and it enables the manipulation of diverse objects. The experiments demonstrate that training with AR2-D2 data is as effective as using real-world robot demonstrations. Additionally, a user study shows that users find AR2-D2 intuitive to use without requiring prior training, distinguishing it from other common methods. Overall, the paper presents AR2-D2 as a practical and accessible approach to collect demonstrations for robot learning, expanding the possibilities for training robots with diverse object manipulation tasks.

review: Quality: ​​This paper presents a novel setup for learning from demonstration, which significantly improves the accessibility of robot training using real-world data. The authors effectively demonstrate the practicality of their approach through a working case study involving a smartphone application in real-world robot training tasks. The integration of these innovative ideas and technologies in this paper highlights its contribution to advancements in robotic applications.

Clarity: The paper is well-written and easy to follow. The authors have done a good job of explaining the motivation for the work, the technical details of the system, and the experimental results. They have mostly avoided using jargon and acronyms, which makes the paper accessible to a wider audience. The figures and tables are also mostly well-labeled and easy to understand.

Originality: The paper is original. The authors have developed a novel robot demonstration collection system that has the potential to make robot training more accessible. The paper also makes a good contribution by demonstrating that AR2-D2 can be used to train a real-world robot to manipulate personalized objects with comparable performance to training using real-world demonstrations.

Significance: The impact of the paper is above average. The development of AR2-D2 has the potential to significantly facilitate the way that robots are trained. It could be used to train robots to manipulate a wider variety of objects, which would advance the deployment of robots in the real world.

Strengths: The main strengths of the paper are:

- The development of a novel robot demonstration collection system that is easy to use and can be used to collect data for a wide variety of objects.
- The demonstration that AR2-D2 can be used to train a real-world robot to manipulate personalized objects with comparable performance to training using real-world demonstrations.
- The user study, which showed that users found AR2-D2 to be intuitive and easy to use.

Weaknesses: The main weaknesses of the paper are:

- The need for fine-tuning to match the performance of real data.
- Unclear explanation of the manipulation behaviors.
- The inability to validate experimental outcomes via simulation.
- The small participant pool in the user study. 
- Insufficient details about the implementation of the system. This would be helpful for other researchers who want to use/replicate the approach.

Overall: I think the paper is good. The authors have made a good contribution to the field of robot learning by developing AR2-D2, a novel robot demonstration collection system that has the potential to facilitate robot training. I am excited to see how AR2-D2 is used in the future to advance the field of robot learning.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I believe that the paper is well-written, technically sound, and has the potential to have a good impact on the field of robot learning. The ability to collect demonstrations for a wide variety of objects without access to a real robot will make it possible to train robots to manipulate a wider variety of objects. I think that the paper is well-written and technically sound. The authors have done a good job of explaining the motivation for the work and the overall process. Having said that, some further clarifications, especially on the details of the AR data collection, how manipulation skills are defined, and the experimental procedures, will make the paper even better.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper presents a system which enables robot manipulation data collection using an iOS app. The authors leveraged the built-in hand gesture detection, combined with inpainting to remove the hand image, and canonical AR support to project an arm into the scene. The AR robot arm is then controlled using motion planning to track the 6DOF hand gesture. Thus, a complete demonstration including RGB, depth, arm action can be generated and collected. The authors demonstrated the effectiveness of the proposed system by training BC models using AR2D2 data and data collected with baseline methods, and evaluating in the real world. Comparable performances were shown between the tested collection methods, while the proposed method was quantitatively shown to be faster, as well as being more flexible and applicable to more environments as it requires an iOS device rather than a physical robot.
review: Strengths
- The proposed approach is attractive, as the authors have highlighted, in scaling up robot learning by removing the requirement of a real robot for data collection 

- The authors have demonstrated competitive model performance using the proposed method, while also presented convincing results in terms of usability and collection speed.

Weaknesses
- During training, the model still requires fine tuning using teleop data, since the data presents a hybrid of a real environment with a simulated robot.

- The fidelity of the physical interaction between the object and the simulated arm depends on the accuracy of end effector rotation/position estimation. It is somewhat unclear whether interactions with certain objects may be less accurate when compared to using a real robot for collection, and whether the fine tune step can work as well for any arbitrary objects

- The state-of-art may solve canonical tabletop manipulation tasks soon. A more diverse task domain, e.g. mobile-manipulation for tasks beyond tabletop, would be more data hungry, and also involve novel objects which could not be easily brought into the lab (e.g. open a unique door). The proposed method will demonstrate larger impact if efficacy can be shown along those lines.


robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper presented an effective way to collect data in real world without requiring a real robot. The method removes one major cost for robot data collection, and therefore can contribute to significantly scaling up data collection both in quantity and environment diversity. The paper is overall well presented. However, this paper will benefit if some of the unique aspects of this setup, e.g. collision detection and its potential limitations, is discussed in more details. I would also appreciate more insight into applications in mobile-manipulation settings.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: The manuscript proposes an approach to scale data in robotics, designed to collect human demonstrations for robot learning using augmented reality. It allows users to record videos of themselves manipulating objects and then modify the captured keyframes, replacing human hand manipulation with robot manipulation of the same object. The manuscript presents experimental results that utilize the system to train an imitation learning agent to manipulate real objects and benchmark the system's usability against traditional methods of collecting human demonstrations (such as kinesthetic teaching or AR controllers).
review: **Strength**:

1. The proposed approach utilizes augmented reality to project an AR Franka robot into a physical environment, freeing human demonstrators from the need for teleoperation of the robot. This has the potential to create a scalable method for generating robotics data in real-world settings.
2. The manuscript provides evidence of comparable results between the proposed AR-based method and human teleoperation using VR and kinesthetic teaching of the robot.
3. The core method, which involves AR data collection without a physical robot, fine-tuning using a few real robot demonstrations, and deployment, is straightforward and easy to understand.

**Weakness**:

1. The requirement for fine-tuning data still necessitates physical robot demonstrations, indicating a significant AR-to-real gap. However, the reviewer believes that this issue can be addressed in future work.
2. Some details are missing, as mentioned in the questions for the rebuttal section, which the reviewer finds confusing in terms of the methodology.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The reviewer recommends strong acceptance. Thanks reviewer again for the timely response to the review. The manuscript introduces a novel approach for collecting robot demonstrations in real-world environments using augmented reality. Users can record demonstrations through human hand movements, and an augmented reality Franka robot mimics the recorded motions. It should be noted that although the title of the manuscript suggests training a robot without a physical robot, there is still an AR-to-real gap that necessitates fine-tuning the policy model with some actual robot demonstrations
review_title: null

Meta-review: Summary:
The paper presents a novel method of data collection for robot manipulation using augmented reality through an iOS app. The approach cleverly utilizes hand gestures and AR to simulate robot actions in real environments. The method presents an effective, scalable solution for collecting robotic data, reducing the need for physical robots.

Strength:
- Scalability: The proposed system offers a scalable way to collect robotic data without physical robots. This can enhance robot learning by providing varied real-world scenarios, without the constraints of lab environments.
- Usability and Speed: The paper demonstrates competitive model performance, and has presented convincing evidence regarding the efficiency and speed of data collection using the proposed method.
- Novelty: The usage of AR for robot data collection in real-world settings, while bypassing the need for teleoperation, is innovative and commendable.

Weakness:
- Fine-tuning with Real Robot Data: Despite the AR-based approach, the system still requires fine-tuning using real robot data. This presents a hybrid data scenario, potentially complicating the learning process.
- Fidelity and Accuracy: Concerns about the accuracy of the simulated interactions have been raised, especially when considering certain objects or situations. The readers would be keen to understand the limitations of such scenarios, and whether the gap might affect the system's performance.
- Collision Detection: Reviewers have mentioned concerns about collision detection and its potential limitations, citing a specific scenario for further clarification.
- Methodology Details: Reviewer has highlighted some missing details in the paper, particularly regarding the parametrization of gripper actions, in-painting of user’s hand in depth data, and assumptions regarding camera pose and hand occlusion.
- Small number of subjects and lack of information on whether subjects have basic knowledge of robotics

Rebuttal and Discussion:
Through the discussion in the rebuttal phase, the authors and reviewers discussed the following concerns, and the authors appropriately replied to reviewers’ questions:
- The benefit of enabling diverse objects and the limitations in traditional teleoperation settings.
- Insight into the contributors of the sim2real gap.
- How collision detection is managed and if specific scenarios are handled.
- Gripper action parametrization and depth data in-painting.
- Assumptions regarding camera pose, hand occlusion, and if bimanual motions can be accommodated.

Recommendation:
Based on the review comments and discussion, it's clear that the paper presents a promising approach. However, some concerns and questions need addressing in the final manuscript. AC recommends and believes that the authors provide an appropriate revision addressing the questions and concerns raised by the reviewers. If these issues are addressed satisfactorily, the paper holds great potential for acceptance.
Decision: Accept (Poster)