Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper presents an imitation learning system OPTIMUS, which trains large-scale visuomotor transformer policies. The key idea is to obtain large-scale demonstrations through task and motion planning. Then visuomotor transformers are used to imitate the TAMP demonstrations directly from visual input to avoid the requirements of the privileged state information. The authors perform an extensive study of imitating TAMP solutions and demonstrate the generalizability of the transformer policy. The method can solve a wide range of multi-stage tasks with a high success rate.
review: Strength: 
- The proposed method utilizes TAMP (Task and Motion Planning) and a procedure generator to synthesize large-scale demonstrations in four different environments, facilitating robot learning. The authors introduce several useful techniques, such as cost minimization, two-stage motion planning, and improved IK, to enhance TAMP solutions for imitators. While not entirely novel, these techniques offer valuable guidance for future research. The authors also ensure the data is designed to support the imitator, including multiview images with a wrist view, task-space control, and the removal of subpar demonstrations.
- The utilization of a large and diverse dataset allows for the learning of a generalizable visuomotor policy. The authors demonstrate that employing a transformer with a vision backbone and GMM action head yields excellent performance. The proposed method surpasses both the behavior transformer and BC-RNN baselines.
- The experimental results showcase that OPTIMUS successfully learns to solve long-horizon manipulation multi-stage tasks and even outperforms the TAMP supervisor.

Weaknesses:
- The primary limitation of this method lies in its reliance on a TAMP solver. This approach may necessitate additional human effort to solve new tasks, and the range of solvable tasks is somewhat limited without significant exertion. For instance, considering dynamics or tasks involving contact-rich manipulation proves challenging. While the paper demonstrates that the learned policy can outperform human experts, the framework remains constrained to the tasks specifically designed for TAMP.
- Another notable aspect is the absence of evaluation in real-world scenarios, despite the method being specifically designed for such applications (due to the unavailability of privileged states in the real world). It remains unclear whether the learned policy can seamlessly transfer to real robot tasks or if the visuomotor approach can be readily generalized.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper proposes a good framework for demonstration generation and imitation in robot learning. However, it does not provide real-world justification.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes a system which distills a Transformer-based motion control policy from data generated by Task and Motion Planning (TAMP). The paper provided the following key observations. 
1) TAMP can be used to generate sufficient manipulation data, while requiring access to privileged state info
2) the generated data can be used to train a transformer model, which can successfully perform in-distribution tasks. The paper also discussed approaches to deal with multi-modality in the context of behavior cloning
3) the paper observed that for output actions, it is easier to train using task-space compared to joint-space
4) by filtering using various criteria like episode length, end effector location, trained model can perform better compared to the TAMP policy used for data generation.

review: Strengths

- The paper clearly demonstrated the following points. 1) it is possible to learn from TAMP-generated data through IL; 2) the proposed model architecture performs better than baseline models on the same dataset.

- The authors provided detailed insights into 1) characteristics of TAMP-generated data, 2) effective techniques in order to learn from the data to obtain good performance in sim evals 

- The paper includes an informative ablation study into the choices of control schemes and data filtration strategies.

Weaknesses

- The proposed method requires access to privileged state info, which means successful applications in the real world would require either a sim-to-real transfer, or other means of state estimation. This means the comparison between the proposed method and manual data collection (in the real world) is not apples to apples comparison. The authors did mention these options as proposed future work. However as mentioned in line 42,  `TAMP on its own requires accurate estimation of the scene geometry and its state`. Therefore, there may be a significant gap between the demonstrated results and a successful demonstration in the real world.

- Related to the point above, if one of the benefits of distilling TAMP into a model is so that state estimation is no longer needed, then this benefit apply to the entire approach end-to-end. As the authors mentioned in line 368, state estimation (through methods like apriltags) is still needed for applications in the real world.

- The paper proposes OPTIMUS as a system which consists of a data generation component as well as a learning model component. The paper rightfully emphasize the importance the data generation component as a key component. However, the paper did not provide a more detailed assessment on the benefit over other data generation methods.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper proposes a reasonable approach towards learning robot control policies in sim. The results are good at solving single task categories (with varying object types/numbers). The overall idea of procedure generating data for behavior cloning, as well as the transformer based model architecture work well in the proposed setting, while incremental compared to prior results. The main limitations of the results in this paper include lack of real world results (with likely technical difficulties when applied in the real world), and lack of demonstrated multi-task category (or one model does all) capabilities.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper outlines a workflow to collect datasets for complex robotic tasks using a Task and Motion Planning (TAMP) agent. It proposes different data collection strategies, such as filtering out outlier trajectories and restricting trajectories to a workspace bound. The curated dataset is used in a behavior cloning framework to train a visuomotor transformer network to output low-level task-space control of the robot. The authors show that the trained transformer model improves task performance and execution times compared to the TAMP agent and learns to solve long-horizon tasks using multi-view camera observations.
review: The following are the main strengths of this paper:
- Improved task accuracy compared to baseline TAMP agent.
- Empirical evidence that using task space control improves the performance of the transformer agent compared to using joint space control.
- The trained policy adapts to manipulating diverse objects in the shelf and microwave task.

The authors mention the difficulty in collecting datasets for training reinforcement learning (RL) policies. But with methods such as MoDeM [1], which trains RL policy on complex robotic tasks with few interactions and expert demonstrations, it is difficult to gauge the significance of this work. Perhaps comparisons could be made to understand better if data curation using TAMP is more efficient than RL policies.

Although the authors conduct extensive experiments to evaluate their method, it is hard to follow the flow of the paper. Breaking down long sentences (e.g., ln 164-166) might make it easier to follow the paper. Extensive use of the bold font is also quite distracting in the results section and should only be used sparsely. I also observed the following smaller errors:
- In line 82, missing ‘t’ in ‘o’.
- Line 145, missing verb
- In line 174, the word “policy” is repeated.

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The idea proposed in this paper will help train agents using behavior cloning for long-horizon robotic tasks. The data efficiency of this method to other RL algorithms cannot be verified from the results.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper presents a novel imitation learning system called OPTIMUS that trains large-scale visuomotor Transformer policies by imitating a TAMP agent, and shows that the combination of large-scale datasets generated by TAMP supervisors and flexible Transformer models to fit them is a powerful paradigm for a wide variety of challenging vision-based robot manipulation tasks with 70 to 80% success rates.
review: strengths:  
1. Clear writing. The methods and experiments are easy to follow.  
2. Large-scale experimens.The proposed methods can solve a wide variety of challenging vision-based manipulation tasks with over 70 different objects, ranging from long-horizon pick-and-place tasks, to shelf and articulated object manipulation, achieving 70 to 80% success rates.  


weaknesses:  
1. no hardware experiments.  
2. Very simple combination of existing methods: dataset is generated from TAMP, and policy is trained by BC. Besides BC, give more results about RL or more advanced Transformer-based network will enhance the technique contribution of the paper. 
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper is clearly written and the proposed methods can solve a wide variety of challenging vision-based manipulation tasks with over 70 different objects, ranging from long-horizon pick-and-place tasks, to shelf and articulated object manipulation, achieving 70 to 80% success rates.  However, there are no hardware experiments, and most importantly, the proposed method seems to be a very simple combination of existing approaches: dataset is generated from TAMP, and policy is trained by BC. If the authors can prove or at least discuss whether their methods are also suitable for RL or more advanced Transformer-based network, I will give more support for the acceptance of this paper.
review_title: null

Meta-review: The authors present two main system components: (1) a set of {environments, expert TAMP planner for those environments, data generated from that TAMP planner}, and (2) a set of results studying behavior-cloning, i.e. distilling, the TAMP-generated data into an observable, visuomotor policy.

The reviewers had quite consistent opinions: in terms of strengths, they thought there was value in studying how TAMP data can best be distilled and thought it was interesting to see that the distilled BC policy, when filtered, could outperform the original TAMP planner.  In terms of weaknesses, many reviewers pointed to the lack of real-world validation, and also the unclear path of how such results could affect the real-world -- both the two paths offered by the authors in the rebuttal (either using expert perception systems, or trying sim2real) have considerable challenges.  The dissenting reviewer still leaned to reject based on the lack of connection to the real world.  While this is a valid concern, I don't think this alone prevents acceptance for the work, as the work is clearly aligned with the interests of the robot learning community, with high-quality simulations of realistic scenarios (including, using only observable information to learn the derived distilled policy). There is also considerable overlap with the overall idea of studying imitation of TAMP from CoRL 2021's paper by McDonald et al (https://openreview.net/pdf?id=-JwmfQC6IRt).  Accordingly reviewers, and the AC, believe there is little novelty.   

However, the majority of the reviewers still lean positively on this paper, and I agree.  For one, I think there is a dearth of good datasets which offer interesting, diverse data for studying imitation learning.  Most datasets might either fall into the category of (i) obtained with an expert RL policy, or (ii) obtained with human demonstrations, where (i) can often give wonky behavior and (ii) is often limited in size and adaptability.  The presented environments, expert planner, and data can provide valuable assets to the community, if they are made available, which the authors in their response say they will do.
Decision: Accept (Poster)