Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The authors present MimicGen - al algorithm for generating diverse robot demonstrations from a small set of initial human teleoperated demonstrations. Then they show how such a synthetically-generated dataset can be useful in a behavioural cloning scenarion where a robot learns from the dataset but only uses images as inputs to its policy.
review: Overall the paper is well-written and presented and outlines relevant issues for the field of robot learning. Experiments on a real robot are appreciated and so is the supplementary video. The breadth of different tasks is impressive. However, I have a major issue with the methodology overall and it's actual usability - I think the 3 assumptions listed in section 2 are rather strong - i.e. we assume we have very good scene and object observability and we stay in the delta end effector pose action space, which neatly allows us to depend heavily on good motion planning. But that means we can alleviate human demonstrators and generate useful datasets *only* for tasks that are already easy to segment into parts that are each solvable by just knowing where the manipulated objects are at the beginning, where they have to be at the end and motion planning between those two. What's the value of learning a visual policy here? What about tasks where this part segmentation is not that obvious because object state is not observable or maybe is even hard to represent - e.g. cloth folding tasks and manipulating deformable objects in general. 

One option would have been to demonstrate that even though we have perfect observability and state information in the simulator, we can generate visual synthetic datasets and train a policy that generalised well to the real world where such perfect information in not available. This would have made MimicGen a valuable contribution to the sim2real community but as the authors note, such transfer is not trivial and does not achieve very good success scores.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I recommend weak reject but would be happy to change my mind depending on the rebuttal discussion.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes a method for generating a diverse set of manipulation trajectories from a few demonstrations which could be of the same task or some variant. Their algorithm, MimicGen, takes in a few human demonstrations and parses them into object-centric subtasks. These source segments are then transformed to generate novel trajectories while preserving end-effector poses with respect to the object for the subtask.  As part of the evaluation, a large dataset is produced on various tasks from a small number of human demonstrations, both in simulation and hardware. Authors validate their method by showing that similar policy performance was achieved using a MimicGen generated dataset from a small number of human demonstrations versus the full amount from only human demonstrations. Additional experiments show that policies trained with the generated datasets achieve higher success than those using only the human demonstrations, from which the dataset was generated from, alone.
review: Quality/ Clarity
--------------------
The paper is well written; however, the authors refer to the appendix in multiple instances which hampers the readability. The paper should be more self-contained. The related works covered are moved entirely onto the appendix, the parts that remain in the paper are vague. The authors should at least, move some of the most relevant works in the paper. Not clear how 50k demonstrations were produced from the ~200 human demonstrations.

Originality
--------------
Though trajectory generation algorithms exist, the proposed method is meant to be more general purpose and removes assumptions about the tasks and algorithms that utilise the data. Furthermore, MimicGen generates a more diverse dataset, specifically for manipulation tasks, by varying various parameters in the scenes and objects associated. However, the method itself is quite straight forward and lacks any significant novelty.

Significance
-----------------
The proposed MimicGen method generates a diverse dataset from a small number of human demonstrations which reduces the amount of expensive data collection required. Removing assumptions on the task and algorithm using the data also makes their method more widely applicable to various problem settings, such as in sim-to-real frameworks.

Strengths
--------------
* The proposed approach addresses limitations in the current literature well.
* The authors perform a thorough experimentation with different manipulators across different tasks.
* The results are presented well and summarised effectively.
* Method shown to work well on real world experiments.

Weaknesses
------------------
* No comparison with other trajectory generation methods.
* Lacks key information regarding experimental setup such as computational resources used and formulation details of the MDP.
* The naive interpolation scheme may heavily reduce the practicality of the proposed method, particularly in cluttered environments.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The proposed data generation method is well motivated; however, the novelty is rather weak. The main strength of the paper is the experiments which demonstrate the wide applicability of the method. There are exciting avenues for future work and utilising MimicGen in other problem settings such as sim-to-real transfer. The paper itself could be more self contained with more thorough description of methods and less reliance on appendices.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper introduces a methodology for generating a large number of diverse demonstrations from a small initial set for use in imitation learning, under the assumption that the initial demonstrations are time-consuming or difficult to collect, e.g., human teleoperation. The method is based on online data generation, in which initial trajectories are transformed to new scenes and replayed in order to collect new demonstrations. A comprehensive empirical analysis is performed, in which task success for a learned behavior cloning policy is evaluated given a) datasets with different initial state distributions, b) datasets with different target hardware, c) datasets with different simulators, and d) datasets consisting of real-world demonstrations.
review: Strengths:
* This paper is well-written and it is easy to follow the methodology; intuitively the method is straightforward and seems reasonable. The related work, limitations, and empirical analysis are very comprehensive (albeit, much of it is located in the appendix due to space limitations).

* The experiments test for many interesting conditions, including changes in number of source/target demonstrations, demonstration distributions, demonstrator quality, and target hardware, in addition to data generation in real-world settings. Tested conditions also include different forms of manipulation, e.g., mobile vs tabletop. In general, the results seem to indicate strong performance gains when the generated trajectories are utilized in training.

* The limitations and assumptions are thorough and clearly laid out.

Weaknesses:
* The actual novelty of the individual components is questionable. Generalization to different objects and scenes is essentially just domain randomization, and replaying prior trajectories is also well-explored (coarse-to-fine, etc). While the proposed method applies a simple transformation to the source demonstration to map it to the target scene, this is fairly simple in nature and only seems to enable variations in starting/ending locations, with transformation failures being captured through a rejection sampling scheme. I do appreciate that the limitations are thoroughly and explicitly stated, however.

* While it is argued that online data generation of trajectories as proposed in this work is advantageous to offline data augmentation due to physics-consistency (Appendix B5), it is unclear if this true in practice since augmentation methods are not evaluated. It has previously been shown that training over a distribution of physics models results in more robust policies [1,2]. I wonder whether similar performance gains could not be realized through simple trajectory replay with domain randomization methods and/or perturbations, which seem like they would be significantly easier to implement in practice since trajectory transformations are not required. This is also hinted at given applying action noise to the generated trajectories yields a non-significant increase in performance (Table M.1).
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: In general, I find the novelty of the methodology a little questionable and am unsure if this approach yields actual improvements as compared to standard domain randomization and augmentation techniques. However, the paper is very thorough I think many of the findings could be useful to the community going forward, so I am leaning towards accept.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes a data generation system MimicGen, which can synthesize successful demonstrations for randomized environment initializations. The system can handle randomizations of a number of aspects in the env, including object position and arm type. The authors then used the generated data to train a BC policy to demonstrate the efficacy of this method. Experiments are conducted in sim and real. Key ideas implemented in MimicGen include
- Collect a small set of human demos for each task
- Randomize the scene configuration, segment the sequence by key points on the time axis
- Interpolate end effector movement based on object position/orientation
- Execute the modified trajectory, and keep as a demo if successful]

~50k demos were generated from ~200 human demos, a scale-up factor of 250x 
review: Strengths
- This is a well presented paper. I appreciated the concise descriptions of the method, its strength and its weaknesses.
- The proposed method was tested on and shown to be effective on a representative set of diverse tasks in sim. The experiment results and analysis are clearly presented.

Weaknesses
- Limitations to the applicability of the proposed method in the real world: the interpolation step may cause safety concern when applied in the real world. The authors implemented mitigations which resulted in performance degradation. This calls for further investigations on how to address this so the performance of the proposed method is more in line with sim results. Can MimicGen still match the performance of an equal amount of human-collected data?
- It was unclear how success of each subtask in a trajectory is detected. How is it done in sim and real respectively? 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper provides a simple recipe for scaling up data collection, which if applied successfully can have a large impact on accelerating the acquisition of visuomotor skills using BC. The robustness of the recipe is demonstrated well in a nice array of evals in sim. The weakness of the results mainly comes from the real robot results, where the significant drop of performance compared to sim results for the select 2 evals raises questions for the applicability of the method in real. Considering the more readily available selection of data generation methods exclusive to sim, as well as the significant value from removing the need of sim2real transfer, the impact of this paper will be greatly increased if stronger real world results can be demonstrated.
review_title: null

Meta-review: Thank you for your submission to CoRL 2023. The reviewers have spent considerable time working with this paper and the current version is recommended for acceptance.

Please follow up with promised improvements, experiments and additional suggestions from the discussion phase. These include in particular clarity about the assumptions, added comparisons and further requested details.
Decision: Accept (Poster)