Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This work introduces an iterative procedure for data augmentation where task plans are generated, executed, and success verified.  The approach is simple but builds on several trained components to ensure the accuracy of the resulting data.  I am in agreement with the philosophical claims of collecting and learning alignments to motion.


Note: There are some rendering issues with figures that requires they be opened in adobe reader, other (Mac preview, Browser, etc) are not showing Fig 3 or lines in Fig 5. The figures seem to be very heavy so CPU jumped to 100% trying to render them.
review: Fundamentally, the insight is that the data an agent can get via imitation is limited and expensive to collect but that sampling from existing components allows for testing boundaries and compositions that are still valid. This is nicely emphasized in the catapult case where the motion planner allows for atypical but successful placements.  

See questions before for better understanding the effects of the composition of components

This is tied to the limitations of this work -- where do we already have robust components? is there a way to use the proposed verification loop to learn new skills? 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Fundamentally, the idea is simple, appears to work well, but I am having trouble understanding the likelihood that other researchers would be able to leverage the approach or what the boundaries are of where it is most useful (e.g. Sim vs Real assumptions, MP quality, ...).  The single biggest gain seems to be related to distillation which is mentioned throughout but details are sparse, so it's difficult to understand what's unique versus retraining.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper aims to learn language-conditioned control policies by collecting data in simulation. Using Large Language Models (LLMs) and predefined motion primitives, the proposed method can generate diverse tasks in simulation to train a language-conditioned diffusion policy to solve instruction following tasks. The learned policy is compared with a variant of Code-as-Policies on a set of simulated tasks and achieves better performance. 
review: Strengths: 

- The paper studies how to use LLMs to guide data collection, which is interesting and novel. This work can potentially pave the way for exciting new directions. 

- An ablation study is included in Table 2 with detailed discussions, which provides useful insights about how each component contributes to the performance. 

- The paper is mostly well-written and easy to follow. The data generation pipeline is clearly explained.

- I like the design of the testing tasks. These tasks are sufficiently complex while grounded in real-world applications. 

Weaknesses:

- It is hard to justify the contributions regarding the diffusion policy (distillation party) since no comparisons or ablations are provided in that regard. I would suggest the authors run ablations using policy networks adapted from BC-Z (Jang et al.), Language-Play (Lynch et al.), and other prior work. 

- It would be better to include discussions about prior work on unsupervised task generation such as POET (Wang et al.), MAP-Elites (Fontaine and Nikolaidis), ATR(Fang and Migimatsu et al.), PLR (Jiang and Dennis et al.); and language-guided exploration such as ELLA (Mirchandani et al.) and ELLM (Du and Watkins et al.).

- It is unclear how such methods can be applied to the real world. 


robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper proposes a reasonable method for learning language-conditioned policies by generating data in simulation. My main concerns are the lack of justification of the contributions regarding the diffusion policy part and insufficient discussions about prior work. If these concerns are addressed, I will consider raising my rating.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: The authors propose a robust framework for using LLMs to call planners to execute long-horizon manipulation tasks to generate data for training a policy.
review: Overall, I enjoyed reading this paper and I think it should be accepted. There are a few issues that I found while reading that I think the authors could help address in the rebuttal.

### Strengths

**************************************Writing and Clarity**************************************: This paper is very well written and clear. The method, at a high level, is also very intuitive and simply (though pretty complicated at a lower-level with respect to the LLM) and the authors did a good job separating details from intuition. Figures are also excellent throughout the paper.

**************************Experiments:************************** Overall, experiments are solid and the baselines are sensible. Results demonstrate the method is necessary under this problem formulation compared to the baselines. The environment is also very diverse in task distribution and demonstrates many axes of comparison.

************************Motivation:************************ Upon reading this paper at first, I was thinking “why not just use the planners if they are already good enough to *************generate data************* by using the LLMs to call the planners when a new task arrives?*************”************* However, the authors resolve this problem by using difficult environments in which even good planners struggle to solve the task. This does lead to the question of “can these failed trajectories be used to improve the planners?” but that is likely less scalable than just training a policy.

### Weaknesses

************************Motivation:************************ This work claims to be very general, but does seem to require quite a bit of domain-specific tuning. Robot pick-and-place is the task that the authors focus on but to make this work generally (e.g., navigation + manipulation for general robots) requires difficult design of the robot utility primitives and likely also difficult design of the “verify + retry” mechanism. Of course, to be fair, the authors don’t claim that this will work on those more general cases so this is somewhat of a nitpick.

Also given how big the LLM prompt is in the supplementary, it likely requires careful tuning of the prompt mechanism along with a very expensive/expressive/large language model, AND extensive scene descriptions to work. Is this approach very scalable to real-world environments? How robust can these success detectors be without perfect object-state information and the like? The authors experiment only in sim.

****************************************Method explanation:**************************************** How do the authors use the failed trajectories? L182: “In the output replay buffer, the only failed trajectories are ones which timed-out or led to invalid states (e.g. object dropped on the floor).” Are these failed trajectories used to train the policy? If so then how does behavior cloning differentiate between successful and failed trajectories? If not, then please clarify this in the text.

************************Experiments:************************

- Baseline explanations: L245 "ground truth segmentation,” segmentation of what exactly?
- How much data is generated for each task? This is included in the supp (500 successful trajs) but should be in the main paper for context.
- It would be beneficial to the academic community, especially among those who want to reproduce this type of work without the ability to work with industry-level language models or paying for GPT-3/3.5/4/4++ credits, to have an ablation study with some other open-source LLMs. VICUNA, LLAMA, ALPACA, and many other animal-inspired open-source LLMs are now flourishing while sometimes claiming to match GPT-3.5/4 performance on certain tasks. Please include some ablation study of this — this is important to the academic community and is the main reason for my “weak” accept.
- 50 seeds per task but no error bars on anything?
- Not asking for this to be included because this is asking too much, but did the authors try VLMs that can caption images? E.g., LLAVA-13B, GPT4, PALM (Bard), or more traditional captioning models? These models could help detect success conditions *********possibly********* more robustly than generating success checkers.
- No robot experiments, but at this point these simulated environments are very similar to real-world robot tabletop manipulation so the application is clear for CoRL.

### Minor Points

- Related work: you can add this recent work to the **************scaling visuo-linguo-motor data************** paragraph for scaling up pre-annotated data: https://arxiv.org/abs/2306.11886
- Related work: please add this work to the same section for work very similar to code-as-policies: https://arxiv.org/abs/2209.11302 (L78)
- L142: function → functions
- L238: an → a
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I enjoyed this paper. It is a weak accept mainly due to the lack of ablation studies that would be useful for the research community (LLM ablations) and some questions I have regarding the motivations.

If these are addressed during the rebuttal, I would easily recommend this paper with a strong accept.
review_title: null

Meta-review: The paper combines ideas to scale up the collection of vision+language+motor data with novel approaches to train language conditioned policies (based on Diffusion Policy). The general approach is highly relevant to bringing strongly generalizing LLM to robotic manipulation.

The reviewers appreciate the novel ideas. The rebuttal addressed many technical points raised by the reviewers. (The 2nd reviewer indicated potentially raising the score.) A limitation is the evaluation in simulation only. The authors mention 'the Sim2Real route as a good way for our method to be applied to the real world'.
Decision: Accept (Poster)