GenSim: Generating Robotic Simulation Tasks via Large Language Models
Authors: Lirui Wang
Yiyang Ling
Zhecheng Yuan
Mohit Shridhar
Chen Bao
Yuzhe Qin
Bailin Wang
Huazhe Xu
Xiaolong Wang
Venue: ICLR 2024 spotlight (2024)
Keywords: LLM Code Generation
Robotic Simulation
Multi-task Policy Learning
Primary area: applications to robotics, autonomy, planning
TL;DR: We investigated LLM's capability to generate over 100 simulation tasks for training language-conditioned multitask robotic manipulation policy, which demonstrates task-level generalization in both simulation and the real world.

Abstract: Collecting large amounts of real-world interaction data to train general robotic policies is often prohibitively expensive, thus motivating the use of simulation data. However, existing methods for data generation have generally focused on scene-level diversity (e.g., object instances and poses) rather than task-level diversity, due to the human effort required to come up with and verify novel tasks. This has made it challenging for policies trained on simulation data to demonstrate significant task-level generalization. In this paper, we propose to automatically generate rich simulation environments and expert demonstrations by exploiting a large language models' (LLM) grounding and coding ability. Our approach, dubbed GenSim, has two modes: goal-directed generation, wherein a target task is given to the LLM and the LLM proposes a task curriculum to solve the target task, and exploratory generation, wherein the LLM  bootstraps from previous tasks and iteratively proposes novel tasks that would be helpful in solving more complex tasks. We use GPT4 to expand the existing benchmark by ten times to over 100 tasks, on which we conduct supervised finetuning and evaluate several LLMs including finetuned GPTs and Code Llama on code generation for robotic simulation tasks. Furthermore, we observe that LLMs-generated simulation programs can enhance task-level generalization significantly when used for multitask policy training. We further find that with minimal sim-to-real adaptation, the multitask policies pretrained on GPT4-generated simulation tasks exhibit stronger transfer to unseen long-horizon tasks in the real world and outperform baselines by 25%. See our project website (https://gen-sim.github.io) and demo (https://huggingface.co/spaces/Gen-Sim/Gen-Sim) for visualizations and open-source models and datasets.

Published as a conference paper at ICLR 2024
GENSIM: GENERATING ROBOTIC SIMULATION TASKS VIA LARGE LANGUAGE MODELS
Lirui Wang', Viyang Ling"
2,3 Zhecheng Yuan", Mohit Shridhar', Chen Bao", Yuzhe Qin', Bailin Wang', Huazhe Xut, Xiaolong Wang* MIT CSAIL, UC San Diego, "Shanghai Jiao Tong University, *Tiinghua University, Univeesity of Washington, CCMU https:://iimt g thub. io/ geggsssmm
ABSTRACT
Collecting large amounts of real-world interaction data to train general robotic policies is often prohibitively expensive, thus motivating the use of simulation data. However, existing methods for data generation have generally focused on scene- level diversity (e.g., object instances and poses) rather than task-level diversity, due to the human effort required to come up with and verify novel tasks. This has made it challenging for policies trained on simulation data demonstrate significant task-leve generalization. In this paper, we propose to automatically generate rich simulation environments and expert demonstrations by exploiting a large language models' (LLM) grounding and coding ability. Our approach, dubbed GENSIM, has two modes: goal-directed generation, wherein target task is given to the LLM and the LLM proposes task curriculum to solve the target task, and exploratory generation, wherein the LLM bootstraps from previous tasks and iteratively proposes novel tasks that would be helpful in solving more complex tasks. We use GPT4 to expand the existing benchmark by ten times to over 100 tasks, which we conduct supervised finetuning and evaluate several LLMSs including finetuned GPTs and Code Llama on code generation for robotic simulation tasks. Furthermore, we observe that LLMs-generated simulation programs can enhance task-level generalization significantly when used for multitask policy training. We further find that with minimal sim-to-real adaptation, the multitask policies pretrained on GPT4-generated simulation tasks exhibit stronger transfer to unseen Lang-horizon tasks in the real world and outperform baselines by 25%.
INTRODUCTION
Achieving general-purpose robotic policies necessitates significant amounts of data, which is labor- intensive to collect in the real world. Although simulation provides an economical solution for generating diverse amounts of data at the scene level and instance level (Akkaya al., 2019; Kauf- mann et al., 2023; Fang et al., 2022; Deitke et al., 2022), increasing task-level diversity in simulation remains challenging due to the significant human effort required, especially for complex tasks. For example, creating new tasks involves specifying new asset relationships and task progression, as well as ensuring achievability and transferability other contexts such the real world. Due the challenges. typical human-curated simulation benchmarks usually have only tens to hundreds of tasks (Zeng et al., 2021: Yu et al.. 2020; James et al., 2020).
Recent years have witnessed significant progress in Large Language Models (LLMs) (OpenAI, 2023; Bubeck et al., 2023; Anil et al., 2023) in natural language processing and further in code generation (Chen et al., 2021; Rozière et al., 2023) for various tasks. In robotics, LLMs have been applied multiple aspects, ranging from user interface (Ahn et al., 2022: Shridhar et al., 2022; Lynch et al., 2023; Driess et al., 2023), to task and motion planning (Lin et al., 2023; Huang et al., 2022), summarizing robot logs (Liu et al., 2023), and cost and reward designs (Yu et al., 2023b; Ha et al., 2023), revealing impressive capabilities on physical grounding and code generation. In this work, we take a step further to investigate whether LLMs can be used to create diverse simulation tasks, tapping further into these capabilities. Our M-aee framework, GenSim, provides

---

Published as a conference paper at ICLR 2024
A
J
Figure 1: Task gallery of over 100 tasks generated by GPT4. GenSim leverages LLM code generation pipeline to scale up simulation tasks for policy training and task-level generalization.
automatic mechanism to design and validate task asset arrangement and task progression. More importantly, the generated tasks exhibit great diversity, fostering task-level generalization of robotic policies. Conceptually in GenSim, the reasoning and coding capabilities of LLMs are distilled into lingo-visuo-action policies via the intermediately synthesized simulation data.
The framework is structured around three components: (1) prompting mechanisms that propose new tasks in natural language instruction and their corresponding implementations in code; this provides an automatic mechanism to design and validate task asset arrangement and task progression. (2) task library that caches previously generated high-quality instruction code for validation and language model finetuning, returned as a comprehensive task dataset. (3) language- conugge-ondiiooeedd multi-task policy training procedare that leverages the generated data to enhance task-level generalization. The framework operates in two distinct modes. In the goal-directed setting, where the user has a specific task or desires to design a task curriculum, the framework adopts a "top-down" approach. It takes the desired task input and iteratively generates related tasks to attain the targeted objective. Conversely, in the exploratory setting, where no prior knowledge of the target task is available, the framework gradually explores beyond existing tasks and aims to establish a task-agnostic foundational policy.
By initializing the task library with 10 human- curated tasks (Shridhar et al., 2022), we use GenSim to scale it up and generate over 100 tasks (Figure 1). We propose several tailored metrics to progressively measure the quality of generated simulation tasks, and evaluate several LLMs in both goal- goal-directed and exploratory settings. Based on the generated task library from GPT4 we conduct supervised finetuming on, including GPT3.5 and Code-Llama to further improve task generation performance of LLMs. Additionally, we quantitatively measure task achievability by policy training, and present task statistics across different properties, and code comparisons among different models.
Importantly, we have trained multitask robotic policies that generalize well on the generated tasks altogether and improve sero generalization performance compared to models that train on only human- curated tasks. Furthermore, we also show that training jointly with GPT4-generated tasks can improve the generalization performance by 50% and have around 40% zero-shot transfer to new tasks in simulation. Finally, we consider sim-to-real transfer and show that pretraining on diverse simulation tasks achieves better real-world generalization capabilities by 25%. Overall, policies that trained on diverse LLM- L.M-generated tasks have better task-level generalization to new tasks, which indicates the potential of training foundational policies by scaling simulation tasks with LLM. The contributions of this work can be summarized as follows:
We propose a novel simulation task generation pipeline through LLMs to generate over 100 tasks. We observe that LLMs are capable of generating high-quality, achievable, and diverse tasks by bootstrapping from existing human-curated tasks.
We benchmark tfate-of-the-an LLM models such as GPTs and Code-Llama on simulated manipulation task creations. We find that prompting and finetuning based on the task library can significantly improve the capability of LLLMs generating higher quality tasks.
See our project website htttps://fiuiiwgiihhu.. htwpssiii.../g/smmmmmm.... demo (ttp:://hnggigggiecccoooooCCCCnn Sim), and code http:://gihhhh.cm//iiww///aesss for more details.
2

---

Published as a conference paper at ICLR 2024
on Retrieval
Rerlection
Can you generate the task "buildicar??
class melikaritaco:
"Gostrict stracture staske cylinders.
Took Library
A43 sleetic
veet stse 18.2, 6.E,, 9.821
Large
Language Model
wett srdf www.eeeeeeeeeeeeeeoe wwweeetetttts
construct corner blocks
color arde-ordered- insertion
Todk- Geneiilzaaioo
Program Synthwois
Simulstion Engine
oojanwmell,,
Task
natcheverg cmeaii,, 401,
Lanuaggaanaae analleta the steelsy place a black caliader each side of the base atocks.")
you generate new different from the existing anes?
ball-on-bax-in
pyramid-on pallat
container
Sina to Real Adagtation
GenSim is an LLM framework to scale up simulation task diversity for robotic policy training. We investigate goal-directed mode (top prompt) and exploratory mode (bottom prompt) that generates robotic simulation task codes. The generated task codes are cached in task library which can be used for policy training to achieve better task-level generalization and sim-to-real adaptations.
Figure 2:
We illustrate the effectiveness of the GPT4- -ereereeedd tasks for Dinguage-conditinn visuo- motor policies by improving in-domain generalization by over 50% as well as zero-shot generalization unseen tasks and instructions in both simulation and the real world.
AUTOMATED TASK GENERATION FOR POLICY LEARNING
We present GenSim (Figure 2), LLM framework to generate simulation environments, tasks, and demonstrations through program synthesis. The GenSim pipeline starts with a task creator (Sec. 2.1) with prompt chain operating in modes, namely goal-directed mode and exploratory mode, depending on the knowledge of the target task. As a memory component, the task library (Sec. 2.2) used store previously generated high-quality tasks. Finally, the stored tasks from the library can be used for multitask policy training (Sec. 2.3) or finetuning LLMs for better task generation.
2.1 TASK CREATOR
The goal of the task creator to propose novel task descriptions and corresponding code implementa- tions, which can be further broken down into scene generations and demonstration generations. In particular, we use the Ravens benchmark (Zeng et al., 2021; Shridhar et al., 2022) which focuses on motion primitives such as pushing and pick-and-place that can be parameterized by two end-effector poses at each timestep. From the example in Figure 3, the reset function in the simulation environment code, efficiently initializes the assets and their attributes and poses, well as the spatial and language goals that parameterize the action in each step?. In the exploratory task generation setting, the pipeline is prompted to generate novel task that is sufficiently different from the existing tasks. In the goal-directed setting, the pipeline aims to fill in the task descriptions and implementations of a specified task name. The exploratory approach requires creativity and reasoning capability come up with new tasks while goal-directed approach focuses on simulation coding as a specific task.
In both settings (Figure 3), the language chain first generates task description and associated implementations afterward. The task description includes the task name, assets, and task summary. We adopt few-shot prompting for code generation the pipeline. LLM is prompted to retrieve reference tasks and codes from existing tasks in the task library introduced in the next section. This process is critical for LLM to know exactly how implement task class (such as the procedure of sampling asset URDFs and building scene first, and then adding spatial goal and language goals). In contrast to other LLM coding tasks, there are various feedback forms robotic simulations including the execution pipeline, simulators, policy training, and humans. See Appendix $D for more details on the prompts and generated example code.
2.2 TASK LIBRARY
In the GenSim framework, leverage external memory, dubbed task library, retrieve the generated tasks by the task creator to come up with better new tasks and to train multitask policies.
Note that in this work, task is defined by its code (and the associated language template) rather than specific scene configuration, object relations, demonstration trajectory.
3

---

Published as a conference paper at ICLR 2024
Example
Feedbackk
Description
eotatin Tvey, rower, yetffe
Coce doo rut compric
3- 2099 20101/200 1959
salt runsel
TESAM ctatris Eirauasant cacra rates hower then NPS
Task Library
Sanro a0500
Sell Reree
The bid aloudy reus in toi
saauety Tiak inplasestaiinn nut slyped descrptone
Model
Elacks
Code
Pulay Trgeep
Stigletnk palky maaiin
victe soe terte
The balk rurching the baggage intrrccion
silt www.e//eeeeeeeeeeeeeeeeee
cneitO, HD.
Execation
Dota
Figure 3: Our automatic simulation task generation pipeline (top left) generates a task code that can be used to generate scenes, simulations, and expert demonstrations for imitation learning, In addition to common inccution-basee feedback in LLM program synthesis tasks, the LLM critic and the task library provide task quality feedback. Finally, humans and single-policy training can provide the final the eynert and learner rollente withont anv evtencive codino eynerience Annendiv 6) extrapolation
composition
paran oogallle
hild-trrge
beld-ar
color anderd Issetion
on Boo Dww.HOoRELeee
beild Introrddss
Figure 4: GenSim demonstrates interesting task-level composition and extrapolation behaviors in code generation for simulation tasks, which are distilled to policy learning through demonstrations.
The task library is initialized from the tasks in the human-curated benchmark. provides the task creator with list of past task descriptions to condition on in the description generation stage and list of past codes the code generation stage. Then, the task creator prompted retrieve reference tasks from the task library as examples for coding new tasks, i.c. retrieval augmented generation (RAG). After the task implementation we then prompt the L.M to reflect on the new task and the task library and form an ensembled decision for whether the newly generated task should be added to the library. In Figure 4, we observe interesting composition and extrapolation behaviors in the tasks generated by GenSim. These saved task codes can be used offline to generate demonstration trajectory data for multitask policy training.
finished and can successfully generate demonstrations,
The task library, generated in exploratory mode, be used as bootstrapping data for iteratively training task creators to generate better simulation tasks in goal-directed mode. This is important to scale up task generations and incorporate human feedback as finetuned models more economical to use as task creators. In the next section, we discuss how to distill the task-level generalizations in LLM code generation into policy learning, by expanding the corpus of the training tasks.
2.3 LLM SUPERVISED MULTITASK POLICY
Once the tasks are generated, we can use these task implementations to generate demonstration data and train manipulation policies. We use similar two-stream transporter network architecture as in Shridhar et al. (2022) to parametrize the policy with affordance predictions. The code generation to language-conditioned behavior cloning process can be viewed as the distilling process from LLMs into the low-level control and affordance of robot policies. Treating the program as an efficient representation of the task and associated demonstration data (Figure 5), can define the embedding space among tasks, whose distance metric is more robust to varying factors from perception such as object poses and shapes and yet more informative than the language instructions.
4

---

Published as a conference paper at ICLR 2024
GPT Task Embedding
Polayy Intresement en Reated Tools
ge mal
glec
drew
3m to Roal Pollay Acapoatio
Policy Gone-ulaation Novel Tosts
ullaaiion to
Figure 5: The task library be used for retrieval and finetuning in GenSim Pipeline. Moreover, task code embedding can be used to create an embedding space in the task library (visualized as a T-SNE plot), which be used for clustering tasks and policy training. For example, the purple represents the tasks involving rope, and blue denotes tasks that involve building structures.
3 EXPERIMENTS
In this section, we aim to use experiments to validate our framework, through the following specific questions: (1) How well can LLM design and implement simulation tasks? Can we improve the performance of LLMs on task generation? (2) Does training on LLM-generated tasks improve policy generalizations? Does policy training benefit more if more generated tasks given? (3) Can pretraining on LLM-gencrated simulation tasks benefit real-world robot policy deployment?
3.1 EVALUATING LLM ROBOTIC SIMULATION TASK GENERATION
In this section, we ablated the design of the exploratory LLM pipeline with simulation task-driven metric. Specifically, we measure a sequence of pass rates on "syntax-correct" which measures basic coding syntax issues as well answer formatting problems, "runtime-verifie"" which tests asset hallucinations and code reasoning capability, and finally "task completed" which measures the designs of task demonstrations (the success in pick-place motions). These metrics are shown in the x-axis of Figure 6. Our metric has an incremental structure from syntax to runtime to successfully generate demonstrations where failing the former metrics implies failing on the latter metrics. Note that some problems such as misaligned language instruction and task behavior will not be captured by these metrics. To distill these task generation capabilities into more economic and scalable language models and to potentially conduct self-improvementt we use the 100 GPT4-gemerated tasks in the task library as a dataset for finetuning. We use OpenAI API to finetune GPT models and achieve better performance through this finetuning process. Moreover, such as Code-Llama (Rozière et al., 2023) with LoRA (Hu et al., 2021). We use the task names with short prompt as input tokens and the task code output tokens for autoregressive training.
also finetune open-sourced LLMs
Exploratory task generation. In total, the pipeline generates 120 diverse tasks that can be used for task demonstrations. We compare our task description and code separation prompt with a single prompt that requests both together as well as zero-shot prompt that does not provide as many reference codes. Figure 6 shows that a two-stage prompt chain with few-shot examples and task library can effectively improve the code generation success rates. The exploratory task generation requires the LLM to have language reasoning capabilities understand the prompts and creativity for new tasks.
Goal-directed task generation. We also experiment with the goal-diree goal-directed procedure to measure the coding capabilities of different language models. Specifically, we pick 10 held-out tasks and prompt each model to generate three trials (multiple implementations of the same task) for evaluating the metric. We observe that GPT4 is still outperforming other models in the specific robot simulation coding tasks. More specifically, strong closed-source models such as GPT-3.5 and GPT4 with in-context learning (prompting) can generate creative tasks and yet are still prone to hallucinations in code. We observe that the finetuned open-source models can achieve closer performance as state-of-the-a LLMs. The finetuned open-source models can generate the correct code flow and syntax but they occasionally exhibit misalignment between high-level objectives and implementation,
5

---

Published as a conference paper at ICLR 2024
Exploratory Task Generation
Goal-Directed Task Generation
0.07
08-
Zate
8.--
0.10
0,3
0.45
5-43
04
5.30
A.22
0.20
20
00
sysil at-ett
runtirmecae
Lawbsctmmllete
Funtimme--ee Lask-cormplelle
Metrics
codelama 1ib-fnetuund
codelama--bemneturee
Figure 6: Left) The prompt chain with few-shot examples and the task library are helpful for LLM simulation task generation. Right) Finetuning on GPT4's generated tasks can improve simulation coding capabilities, for both closed-source GPT3.5 and open-source Code-LLama LLama-Instuutt- 13B.
Few-shot Improvement by Adding Tasks
Zemo-shot Generaliztion by Pretmining
Few-shot Improvement by Adding tasks on Different Sources
GPT
Sarb
single
Tekks
52
milii tnclind tack
32
Saacees
multi tasti-- tasks
tastlced tanks
28
15
12
Saiits
20 bara
20 cat
50 lasta
laka
CLPer: Task Source
CoAGLAMA Too
Added Tasks
Task Number
Figure 7: Joint training with augmented data generated from the GenSim tasks improve policy performance, in both single-task and multi-task settings. Left) the x-axis denotes single CLIPort task + N GPT tasks where N=0,2,4 and multiple CLIPort tasks N GPT tasks where N=0,4,8. Middle) We also showed that when training on more and more tasks, the policy exhibits stronger and stronger zero-shot generalization capabilities. Right) The performance of few-shot joint training varies for different sources of task generation including open-source LLMs.
due to the complexities of long simulation code. This motivates increasing better language models in simulation task creations such as self-instruct (Wang et al., 2022) well distillation from more capable models. See Appendix $A, $D for more experiment details and examples.
3.2 TASK-LEVEL GENERALIZATION
In this section, study how the generated tasks from LLM can help with tabletop language- conditioned visuomotor policy learning for generalizations and adaptation. We adopt the 0 (fail) to 100 (success) scores proposed in the Ravens benchmark (Zeng al., 2021) which considers partial credits for completing tasks. The simulation robot setup is a Universal Robot UR5e with suction gripper. The policy input top-down RGB-D reconstruction, and the output is affordance map that is then transformed to pick and place actions. We use the CLIPort (Shridhar et al., 2022) architecture but the framework independent of which policy parametrization we use. The set of testing tasks is not particularly selected. We have released a language-conditions model weights and task lists) generated by GPT, ranging from 10 tasks 100 tasks, study scaling policy learning with affordance prediction.
benchmark with
Few-shot policy generalization to related tasks. In particular, from Fig. left, we show that jointly training I.LM-generated tasks can improve the policy performance on the original CL.Port (Shridhar et al., 2022) tasks by over 50%, especially under low data regime such as demos. The result is averaged the overfitting problem on a few demos. On the right, we also experimented with pretraining with different task sources of the human written tasks, closed-source LLM, and open-source finetuned LLM, and observed similar improved performance.
two different task splits. This is expected adding related tasks reduces
Zero-shot policy generalization to unseen tasks. generated by LLM, our model can generalize better to tasks in the original Ravens benchmark. The task-level generalization is surprising considering that the task and language instructions have
From Fig. 7 middle, by pretraining more tasks

---

Published as a conference paper at ICLR 2024
Label
No Adaptation
No Pretrain
CLiPort
Ours
Pair-Blocks-Cn-a "put the other blue block next to the second red block on the line"
Stack-Blook: "stack the blue block the yellow block
t+O
1-T
Orgenize-Kiss: "pack pens end spoons Into the box"
Build-Wheel: "place the purple and green blocks around the white cylinder. to form wheel shape"
Figure 8: (a) The labels and the affordance heatmap of different real-world policies (green denotes placing and red denotes picking). (b) Executions of long-hooe horizon tasks by GPT4 GPT4 pretrained models.
Task
Episodes Sample)
No Adaptation
Success
No Pretrain
Success
CLIPoct Suceess
GenSim (50 Tisks)
Success
GenSim (70 Tasks)
Success
put-black-in-tone
20(20)
(110
stack-block
25(25)
0/10
2/10
6/10
9/10
10/10
block-towe towek-twwer-eee corner
20(80)
0/10
0/10
2/10
3/10
4/10
blook-ii--
10(20)
(110
pon-hlock-ad zone-comer
10(40)
pait-blecks-on-
20(80)
0/10
0/10
0/10
2/10
4/10
align block
20/60)
0/10
0/10
2/10
3/10
6/10
build wheel
pack spherss
20/30)
0/10
6/10
7/10
4/10
7/10
organize. kits
15(60)
0/10
5/10
5/10
3/10
6/10
sat-ggarage
place-brea
15(60)
0/10
2/10
2/10
1/10
3/10
ovemage
205(665)
085
27.3%%
39.25
46.7%
62.5%
Table 1: Success rates (%) of multi-task policies that finetuned on the base models from simulation. The test performance measured across different scenes and tasks.
not been seen in the training datasets. Note that these tasks are selected using the distance metric on the task embedding (Figure 5). One intuition is that LLM code generation expands the task data, and thus training on these related tasks leads to more robust generalizable representations.
Specifically, shown in Fig. 4, the color-ordered, innertton be thought of as compositional task between block- insertion and place-bloce in-bowl from CLIPort original task tasks. The former task involves placing block into some colored bowl, while the goal of the latter task is to insert specific block into fixtures. In the left of Figure 7, our experiment suggests that learning jointly with related and more complex task nolor-ordered-ineee insertion has the potential contribute to enhanced generalization capabilities of put-block-in-bow for novel task generalization and even tasks as possible, corresponding to exploratory task generation. See Appendix $B for more details.
On the other hand, when we try to learn a base policy domains, is more beneficial learn from as diverse
3.3 ADAPTING PRETRAINED MODEL TO THE REAL WORLD
In this section, we conduct experiments to transfer the policy trained in simulation the real environment. We hypothesize that by expanding the diversity of training tasks generated by LLM in the simulation, the trained policy would exhibit enhanced adaptability in real-world scenarios. To further enhance the sim-to-real transition, we incorporate an adaptation process for the real world. This process includes collecting small set of real-world data for each task, followed by data augmentations and the fine-tuning of the simulation-pretrained model over 50 epochs. We perform our real-world experiments using an XArm-7 robot equipped with a suction gripper. A bird's "od's-eye-view camera is installed facing downward to capture RGB-D observations. In Table the model pretrained on 70 GPT4 -eT--rereateed tasks achieves an average success rate of 62.5% over 10 trials of tasks, over 20% increase compared to baselines that only pretrained CLiPort tasks
7

---

Published as a conference paper at ICLR 2024
(a)
Color
ASSET
AcOih
60
100
73
25
10
goen
YORN oringe
1.61
Llae-Llama (Finetuned)
GPT-4
Class Blcckascy/nnrrroealleeeeee
class (y dellee (Tokk)
serpuck up a bioce and place at in top of LyLdee
mrrucc theee catarod blocks ene place each of them en top of cylunder of the same color:
which is located as pallee.
Afte serts, the resot needs pack each cylareer lucts the black on top) and slace
11 on pallet of the same coler. The onder the cotere coould go red, blus, and then grear,
Adi pallit.
Add cylinder.
befire colers for blocks, cylinders and pollets
Aod biecks.
cylancers.
pot.oll (d), punesls ill,
pallacs.
tang (yrearrttyiirrrrrrreeee rppaaceratae,
Goal: stack cyttnder sad euth cytinder pallet talar.
netatiassiiree, neteice ppee', parascitinn, stousac_rearde, tengeepest tonplati
rarge(shi
htt.:atlopttlee-lbbckkllo cyLinders
23),
"Laeasss tylandera (al.gee.eestaatl
(ware-1/3)
wtt.:et..oalloosssllllee anchecorp.eesttt 111,
tong poses-2allattttll.o pesitiantl,, replacesfalse,
retetderaetraat ectrice 'poce, paranseline, step_uu_rewrree/
NETT vetr.iang_gatbb.epccc lang, tww.tetet.eeeeeeeeee ctw.e//mmttttttttttttttoooo
Figure 9: (a) Task statistics across different properties in the generated benchmark. (b) Example failure modes of the Code LLama after finetuning, and GPT4 through prompting.
and 15% improvement on models that pretrained on only 50 tasks. Qualitatively, baseline models without sufficient pretraining often pick or place the wrong objects, and baselines without adaptations have diffused affordance prediction. We hypothesize that GPT4-generated task pretraining exposes policy with broad language instructions and task compositions, which allows it to generalize better to a diverse set of tasks after sim-to-real adaptation. Moreover, we qualitatively observe that pretraining on diverse simulation tasks improves robustness example, the GPT4-pretrainedd model has a more robust performance on build-wheel task in the real world. Additionally, we show that the simulation pretrained weights can also be useful for tasks such as place- bread and sort-garbage. These are distinct from the simulation tasks with different assets and semantic information. We defer the task, training, and evaluation details the Appendix &C.
honghhorzoon complex tasks (Figure 8). For
3.4 ADDITIONAL. ANALYSIS
Simulation Training Success Rates. On Table 2, we showed single-task and multi-task policy training success rates subset of our generated tasks with 200 demos. The average task success rate for policy training on GPT4-generated tasks is 75.8% for single-task and 74. % for multi-task which is similar 76.6% for single task and 76. 7% for multi-task from human-curated tasks in CLIPort (Shridhar et al., 2022). See Appendix $B for more training details.
Generated Task Statisties. In Figure 9 (a), showed task statistics different features of the 120 generated tasks by LLM. We observe an interesting balance of colors, assets, actions, and the number of instances generated by the LLM models. For instance, the generated code contains many scenes with over 7 object instances and many primitive actions of pick-and-place and assets such as blocks.
Code Generation Comparison. In Figure (b), we qualitatively evaluate failure cases in the top-down experiments for both the Code Llama and GPT4. The coding objective is to implement the multi-step logic of placing a cylinder on a pallet first and then placing the block on the pallet. Based on the add goal step, we observed that Code LLaama has ignored the pallet completely, and GPT4 got the order of placement wrong, while being a bit noisy in text descriptions. Other common LLM modes include misaligned language descriptions and task objectives as well as imbalanced task distribution when scaling the LLM generation pipeline. More details can be found in $A.5.
Human Effort Evaluation. In Section $D.6, we discuss and evaluate the human verification time spent and pass rates on the LLM-generater tasks. The average human time is around 10 seconds (if each task is checked) and the success rates are above 50%.
4 RELATED WORK
Reasoning and Coding via LLMs. There have been impressive emerging abilities of LLMs. including zero-shot prompting and intricate reasoning (Wei et al., 2022; Chowdhery et al., 2022) in the past few years. For example, Park et al. (2023); Zhang et al. (2023) use language models in
8

---

Published as a conference paper at ICLR 2024
Task
Single-Task Task Multi-Tass
Task
Single-Tane Task Multi-Tans
build-car
84.1
85.4
mnsert-blocks--ngee
84.1
80.0
color-coordinated-
sove-piles-along-
99.2
93.0
27.7
32.7
sphere-
line
cylinder-ring-
manipulating-wv
stack
82.0
81.0
ropes
30.8
40.2
multi-level-block-
color-ordered-
construction
67.8
49.0
insortion
93.0
94.0
stack-blocks.
align-cylinders-
in-container 99.0
98.8
91.2
87.8
in-zones
Table 2: Success rates (%) of the trained policy example GPT4-generated tasks.
explorations of new tasks. Our work, which uses separate LLM critics to evaluate the generated programs and a memory for storing and reflecting previous outputs, is also related the rich line of work on parameterizing agents with LLMs (Li et al., 2023; Wang et al., 2023). On the other hand, program synthesis has a long line of research in NLP (Manna & Waldinger, 1971; 1980; Chen et al., 2021; Wang et al., 2019; Chen et al., 2018). LLM models, such as Codex (Chen et al., 2021) make program synthesis accessible by only requiring specification of docstring or incomplete code. Execution-basse code generation (Ellis et al., 2019; Chen et al., 2018) uses execution outcomes at the code compiler level to iterate code generation. Skreta et al. (2023) uses a rule-based verifier to improve program generation and Wang et al. (2023) incorporates environment feedback and self-verification in Minecraft into the pipeline. Our work benchmarks simulation task creation with both open-souree and closed-source LLM code models. In addition, method also leverages multiple verification methods as feedback for simulation task scripts in robotic simulations.
Task and Scene Generation for Simulation. In robotic simulations, recent works have explored domain randomizations (Tobin et al., 2017; Fang et al., 2022; Chen et al., 2023; Ramos et al., 2019) and procedural asset generations (Deitke al., 2022; Makatura et al., 2023) and text to 3D (Jun & Nichol, 2023; Nichol et al., 2022; Poole et al., 2022; Yu et al., 2023a). In this work, we treat each task as the code implementation and expand the original 10 tasks in the Ravens benchmark (Zeng et al., 2021; Shridhar et al., 2022) to over 100 novel tasks and associated expert demonstrations through LLM. Consequently, different from previous works that study compositional generalization (Jiang et al., 2022) or object-level (Shridhar et al., 2022) generalization, we study the task-level generalization of policy learning.
Language Models in Robotics. In robotics, large language models have been applied to policy learning (Driess al., 2023), task and motion planning(Lin et al., 2023; Huang al., 2022), summarizing logs (Liu et al., 2023), well as synthesizing policy programs (Liang et al., 2022) and optimization programs (Yu et al., 2023b). Past work has also explored LLM's physical grounded capability (Liu et al., 2022) and concurrent work explores using LLM together with task and motion planners to create expert demonstrations (Ha et al., 2023). Ahn et al. (2022); Lynch et al. (2023) attempted to collect huge-scale real-world interactions but only focused on specific task families. Instead of interacting in expensive real-world settings, we explored how to create increasingly complex simulation tasks and demonstrations jointly with LLM, and study the scaling law on policy generalizations through training on GPT-generated tasks.
CONCLUSION AND FUTURE WORKS
In this work, we present GenSim, a scalable LLM framework to augment diverse simulation tasks for robotic policy, which aims to distill LLM grounding and coding capabilities into low-level policies. We investigate LLM prompting, retrieval-augmented generation, and finetuning in both goal-directed and exploratory approaches to generate new simulation task codes. We leverage the generated tasks for training multitask policies that show generalization capabilities both to new tasks in simulation and the real world. We have seen some limitations in enabling L.LM to design tasks and demonstrations in robotic simulation. The generated code still contains basic syntax errors and suffers from ballucinations and a lack of grounding in physical and geometric details. Another problem is that the code generation evaluation metric is imperfect (such as misaligned language description) and therefore the generated tasks can require some manual filtering before policy training. Finally, we have only explored table-top pick-and-place task generation, and generating dexterous and complex robotic tasks could be more challenging. We hope these limitations shed some light on future attempts at simulation code generations with L.Mss.

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGEMENT
We thank MIT Supercloud for providing computing resources. The authors would like to thank many helpful discussions from Yoon Kim and Russ Tedrake at MIT. This work is supported in part by the Amazon Greater Boston Tech Initiative and Amazon PO No. 2D>-0631022 This project was supported, in part, by the Amazon Research Award, the Qualcomm Innovation Fellowship, the Intel Rising Star Faculty Award, and the CISCO Faculty Award.
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as can, not as i say: Grounding language in robotic affordances. arXiv preprint wrXiv:2204.0169/ 2022.
Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's 's cube with robot hand. arXiv preprint arXiv:1910.07113 2019.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303. 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint (ttp::2200003374,, 2021.
Xinyun Chen, Chang Liu, and Dawn Song. Execution Execution-guided neural program synthesis. In Interna- tional Conference on Learning Representations, 2018.
Zoey Chen, Sho Kiami, Abhishek Gupta, and Vikash Kumar. Genaug: Retargeting behaviors to unseen situations via generative augmentation. arXiv preprint arXiv:2302.06671 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied ai using procedural generation. Advances in Neural Information Processing Systems, 0599982-5994, 2022.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.0333 0303.03333 2023.
Kevin Ellis, Maxwell Nye, Yewen Pu, Felix Sosa, Josh Tenenbaum, and Armando Write, execute, assess: Program synthesis with repl. Advances in Neural Information Processing Systems, 32, 2019.
Kuan Fang. Toki Migimatsu, Ajay Mandlekar, Li Fei-Fei, and Jeannette Bohg. Active task random- ization: Learning visuomotor skills for sequential manipulation by proposing feasible and novel tasks. arXiv preprint arXiv:2211.06134 2022.
Huy Ha, Pete Florence, and Shuran Song. Scaling up and distilling down: Language-guided robot skill acquisition. arXiv preprint 2.ti:2337.1153355 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhun Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 204v:2/06009660 2021.
10

---

Published as a conference paper at ICLR 2024
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.0560
2022.
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019- 3026, 2020.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:221000300 2022.
Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02463, 2023.
Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 620 (7976):922-887, 2023.
Belinda Li, William Chen, Pratyusha Sharma, and Jacob Andreas. Lampp: Language models as probabilistic priors for perception and action. arXiv e-prints, pp. arXiv-2302, 2023.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07753. 2022.
Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Text2motion: From natural language instructions to feasible plans. arXiv preprint arXiv:2303. /2/53, 2023.
Ruibo Liu, Jason Wei, Shixiang Shane Gu, Te- Yen Wu, Soroush Vosoughi, Claire Cui, Denny Zhou, and Andrew M Dai. Mind's eye: Grounded language model reasoning through simulation. arXiv preprint arXiv:2210.053:2 2022.
Zeyi Liu, Arpit Bahety, and Shuran Song. Reflect: Summarizing robot experiences for failure explanation and correction. arXiv preprint arXiv:2306.15724 2023.
Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots real time. IEEE Robotics and Automation Letters, 2023.
Liane Makatura, Michael Foshey, Bohan Wang, Felix HähnL Pingchuan Ma, Bolei Deng, Megan Tjandrasuwita, Andrew Spielberg, Crystal Elaine Owens, Peter Yichen Chen, et al. How can large language models help humans design and manufacturing? arXiv preprint arXiv:2307.14377 2023.
Zohar Manna and Richard Waldinger. A deductive approach to program synthesis. ACM Transactions on Programming Languages and Systems (TOPLAS), 121, 1980.
Zohar Manna and Richard J Waldinger. Toward automatic program synthesis. Communications of the ACM, 14(3):111-16,, 1971.
Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.0875:1 2022.
OpenAl. Gpt-4 technical report, 2023.
Joon Sung Park, Joseph C O' O'Brien, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra human behavior. arXiv preprint arX1v:2304.03342, 2023.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint 2209 /4988, 2022.
Fabio Ramos, Rafacl Carvalhaes Possas, and Dieter Fox. Bayessim: adaptive domain randomization via probabilistic inference for robotics simulators. arXiv preprint arXiv: 100600900000/72 2019.
11

---

Published as a conference paper at ICLR 2024
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308 2023.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning, pp. 894-906. PMLR, 2022.
Marta Skreta, Naruki Yoshikawa, Sebastian Arellano-Rubache Zhi Ji, Lasse Bjørn Kristensen, Kourosh Darvish, Alán Sapuru-Guzik, Florian Shkurti, and Animesh Garg. Errors are useful prompts: Instruction guided task programming with verifier-assisted iterative prompting. arXiv preprint arXiv:2303. 14100, 2023.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 20/7 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang. Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305. 16291, 2023.
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poct): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv: 2019.
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212. 10.60000 2022.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities large language models. arXiv preprint arXiv:2206.0723 07682, 2022.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100. PMLR, 2020.
Tianhe Yu, Ted Xiao, Austin Stone, Jonathan Tompson, Anthony Brohan, Su Wang, Jaspiar Singh, Clayton Tan, Jodilyn Peralta, Brian lchter, et al. Scaling robot learning with semantically imagined experience. arXiv preprint arXiv:2302. 1550, 2023a.
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huce Lee, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, et al. Language rewards for robotic skill synthesis. arXiv preprint arXiv:2306. 08647, 2023b.
Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong. Ivan Krasin, Dan Duong, Vikas Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference on Robot Learning, pp. 726-747 PMLR, 2021.
Jenny Zhang, Joel Lehman, Kenneth Stanley, and Jeff Clune, Omni: Open-endedness via models of human notions of interestingness. urXiv preprint 21
2023. 2023.
12

---

Published as a conference paper at ICLR 2024
CONTENTS
Introduction
2
Automated Task Generation for Policy Learning
2.1
Task Creator
2.2
Task Library
2.3
LLM Supervised Multitask Policy
3
Experiments
3.1
Evaluating LLM Robotic Simulation Task Generation
3.2
Task-Level Generalization
6
3.3
Adapting Pretrained Model to the Real world
3.4
Additional Analysis
8
4
Related Work
5
Conclusion and Future Works
Acknowledgement
10
LLM Experiment Details
14
A.1 Simulation Task Generation Details
14
A.2 Embedding Analysis
14
A.3 Task Statistics
14
A.4 Training Details
14
A.5 Task Creation Limitations
15
B Simulation Training Experiment
15
C Real World Experiment
16
Language Qualitative Results
18
D.1 Task Description Example
18
D.2 Prompt Details
19
D.2.1 Task Proposal Prompt Details
19
D.2.2
Task Memory Prompt Details
20
D.3
Example Task Library Conversations
21
D.4
Example Implementation Errors
22
D.5
GenSim Feedbacks
.
23
D.6
Human Verification
23
D.7
Framework Scalability Discussions
24
13

---

Published as a conference paper at ICLR 2024
D.8
Task Code Example
24
D.9
Top-Down Evaluation Example
A
LLM EXPERIMENT DETAILS
A.1
SIMULATION TASK GENERATION DETAILS
The held-out simulation task list used to compare and evaluate LLM models is salign-riinbow-lloe line", "cylinder-in-colorffe ior-ii--oooofuulootaneeee" "splitting-pile piiiigggpplls..... sal--yylinde-ppyramii "build -araaid-yramid-ii zone", "block-on-" "onn""cliidde-oo-ppala
construct-symmetriid block-walll"" Ainsert-blueoo cere--onrr-cllddeer" "rainbow-pyramid' We will prompt different LLMs with these task names and run targeted task generation evaluation experiments on code generation three times.
"align-ellinder-inzzoe
We have also experimented with a longer LLM pipeline with components such API reviews and common error reviews. In these two stages, we provide the base task definition of the Ravens benchmark and summarize several syntax errors that GPT tends make when designing and implementing tasks. We empirically observed these to generate more stable codes for challenging tasks, require more budgets.
A.2 EMBEDDING ANALYSIS
In Figure 5, we visualize the GPT embedding of the generated task codes using PCA to project to 50 dimensions and then use TSNE for visualization on 2D. We then apply KNN clustering discover clusters of the generated tasks. As shown in the plot, different colors of clusters represent different groups of tasks such as specific objects or specific types of motions. For instance, the purple color represents mostly the tasks involving rope, and the blue color represents mostly the tasks that involve building some structures using primitive shapes. Inspecting, expanding, and cleaning this task library is also straightforward because of the nature of the code. This embedding space is useful both for retrieving tasks as retrieval augmented generation (RAG) and finding related tasks for joint policy training.
A.3 TASK STATISTICS
Although we focus on the task-level diversity and generalization of simulation LLM, in this section, we investigate some task statistics of the 120 generated tasks by LLM. Shown in Figure (a), we observe an interesting balance of colors, assets, actions, and number of instances generated by the LLM models. For example, common colors such "red, blue, yellow, green" show up in many tasks. Due to our prompt, overall GPT generates multiple objects in the scene. In Figure 13, we also showed gallery of the generated tasks. We can easily generate and scale more of the simulation tasks if the budget permits. See project website for animated rendering of the tasks.
A.4 TRAINING DETAILS
In addition the prompting pipeline, we explored different training and finetuning schemes LLMs. Different from designing a full prompting techniques to improve GPT4 base model, finetuning focuses on domain-specific tasks such given the task name, and output its implementation. has several benefits such as cheaper costs and faster inference speed. We use the 112 tasks as the dataset for finetuning experiments. We use the task names with short prompt input tokens and the task code output tokens for autoregressive training. For closed-source LLM models such as GPT-3 (davinci), GPT-3.5, we use OpenAl's finetuning API for training and inferencing. For Code-LLama experiments (Rozière et al., 2023), we use the open-source Code-I.LaMA- A-Istrucct 7B and Code-LLaMAAIInsinee
13B models. We use Huggingface transformers for LoRA (Hu al., 2021) finetuning with quantization and parameter-efffcient finetuning for 10 epochs on V-100 GPUs. The training on GPT4 generated dataset with batch size=1 takes around hours to finish 3000 iterations. Generated examples for these open-sourced models can be found in $D.
In evaluating the open-sourced code-llama model, we were not able to get sensible results with just prompting. Specifically, it generates the example code in the prompt for all prompts in the few-shot case or random answers in the zero-shot case. With finetuning, the model writes much more structured code and can achieve similar performance as GPT-3.5. Overall we improved
14

---

Published as a conference paper at ICLR 2024
code generation with the finetuning models and leave self bolf-bootstrapping and training more code examples for future work.
A.5 TASK CREATION LIMITATIONS
There are several common failure modes in the task description and implementations generated by LLM that need to be aware of. These raise attention for inspections before potential downstream applications and also hopefully shed light future directions. We recommend users to play with to get a sense of the generation process and limitations. Here, we provide detailed descriptions and examples below.
1. Repeated Tasks: GPT can generate tasks that have different names (and even different im- plementations) but represent the same tasks, such as "color-sequenc" serooo-sqqennee-bblee sawenced-blookiineee block-insertion" "put- incc--in-owwl" and noolor-corddintte-Bee the robot to put some colored blocks into colored bowls. Similarly, nolor-linked-ball-bowe ordering", "color-coordinated--pherr whw.aaddd--ppee-----oooooo//eee -owttmmmmc-ooww--mchh" and soolored-bwwi-ball-sorttte also same tasks.
catraated--all-issttion------i pnvertion-in-ooll.
which all aim instruct
2. Mismatched Language Instructions: GPT can generate mismatched language descriptions for tasks. For instance, for task poolor-coordinated-yylleeeee -an-dder-ttan-----nnenee the language instructions try to set goals for each step to put the corresponding colored cylinder on the stand, such "place the red cylinder on the red stand". However, the task implementation tries to stack as many cylinders as possible onto the same block. This problem is also not captured by the metric used in the benchmark.
3. Ungrounded Motion Sequence: For instance, the task cetor--oooddinnddbboooooo bowl-stackinng" has the description "put each block in the bowl the same color and stack the bowls from largest to smallest in the sequence of red, blue, green, and yellow", and yet the action to stack bowls, is neither implemented nor achievable in the benchmark. Similarly in task Dball-in-containee
with language instruction "Use the spatula to lift a ball over a wall of boxes and drop it into a container", the action of using spatula to lift an object is not waassh--ooc-int--conttee
grounded well. Finally some other objects are unlikely to succeed.
the motion of pushing a block into
4. Noisy Descriptions: There are usually some noises in the task names, task descriptions, and language instructions. For instance which is not capturing the task goal. The task descriptions also contain sentences such as "The task requires the robot to pick up four balls of different colors (red, blue, green, yellow) and place each of them into the corresponding colored bowls strategically positioned at different corners of the maze, without knocking over any blocks, demanding careful navigation and color coordination". which is not in its most concise form. Similarly, 'rearrange, sort, insert, place" are usually used interchangeably by LLM.
has "obstacle-course"
5. Imbalanced Tasks: Due to the bootstrapping process, LLMs has a certain bias towards the majority of the tasks in the task library, especially if the L.M reffection/fitterng is turned off. For instance, we usually see "color-coordinated" and pick-place motions in the task generation whereas tasks involving rope and piles are more rarely. For improved success rates, we also do not consider tasks that are overly complicated such as palletizing boxes and tower of haoni. Moreover, the definition of task is still subjective in robotics. Different tasks such as "arrange, insertion.pur" end up using the pick and place motions, and the hope is LLM can help distinguish among different tasks. We limit the benchmark to 100 tasks for a balance of diversity and control of quality.
6. Task Complexity: Overall this work only explores top-down pick and place tasks where the demonstration action can be parametrized by pick and place actions. More dexterous tasks in robotics (higher degrees of freedom, more contact-rich motions etc) and reward designs for automatically figuring out how solve these tasks will be interesting future works.
B SIMULATION TRAINING EXPERIMENT
We follow the original CSrPortSh et al., 2022) setup for training. All simulated experiments performed using Universal Robot UR5e equipped with suction gripper. This setup offers
15

---

Published as a conference paper at ICLR 2024
Repeated Tasks
Mismatched Languages
Ungrounded Actions
Put-block-in-howll
Color-coodinnted- cylinder-stand saaaaa--tadd--iiii
Puth-kacl-int--cottallet
Figure 10: Visualization of common failure cases.
systematic and replicable environment for evaluation, particularly for benchmarking the capacity to ground semantic concepts like colors and object categories. The input observation derived from top-down RGB-D images captured via RealSense camera. We downscale the original image dimensions from 640 480 to 320 x 160 for policy input. To enhance the transition from simulation to reality, we incorporate data augmentation techniques such color jittering.
Note that our implementation supports multiple samples in a batch (the original version only supports batch size 1) which speeds up training. We train on 4/8 GPUs for 2 days for the multitask setup and train for 10000 iterations for the single-task setup. For larger-scale pretraining experiments, 100 tasks can take up to 5 days with 8 GPUs.
When training language-conditionn sidered in the joint dataset. For instance, the tasks "put-block-in inrck-ii-bowl"" and pplace-red-in-greet both involve putting colored blocks into colored bowls with similar language instructions. This could potentially lead conflicting data points that instruct placing the red block into the green bowl or into bowls of other colors. Surprisingly, we find the policies pretrained on all 10 CLiPort tasks only achieve lower than 5% success rate in GPT tasks. Conversely, there could be many tasks that involve the term "pyramid" in instructions, but this could refer to varying layers of the block pyramid. This ambiguity could pose challenge when training single policy for multiple tasks. The question of whether training with certain tasks can help generalization can be best addressed by the bias-variance trade-off fundamental in machine learning. Tasks that are semantically closed have a higher likelihood of being beneficial, and vice versa. Therefore, we leverage the task code embedding to pick related tasks for joint training, and find that to be robust in finding tasks. During the evaluation of our GPT-gemerated tasks, exclude specific tasks that fall outside of the task generation domains, such as the tower of Hanoi and inserting kittings. addition to the consideration of compute budgets, these tasks also help focus generalization experiments have been evaluated with 1-shot generalization performance on tasks such "mixing miixisg-pilees" and 5-shot generalization performance on
multitask policies, the relations among tasks need to con-
task-level generalization. The few-shot
C REAL WORLD EXPERIMENT
For real-world experiments, employ XArm-7 robot equipped with suction gripper. A bird's sird''-eye-view camera, mounted overhead and oriented downwards, is used to capture RGB-D observations. We leverage workspace boundaries and background subtraction generate a mask on the depth images. See Figure II for detailed setup. In the real world, we observe the following failure cases: (1) manipulate with the wrong color of objects. (2) imprecise picking and placing. To tackle the challenge of scene-level generalization, we use data augmentation such as color jittering, and also data relabeling where we increase the data multiple times and relabel the language prompt based on the objects on the table. Our labeling tools on pick and place motion do not require real robot and can efficiently label up to 100 images within minutes. Real-world training takes 50 epochs of the augmented data and usually takes less than 3 hours to complete. The tasks are selected with a focus on long-horizon tasks and task-level generalization, ie. how well the policy generalizes to tasks rather than different colors or object instances in compositional sense. We have experimented with providing both the per-step instruction or high-level goal of the tasks, and did not find much performance difference in the single-task setting. We thus share similar limitations
16

---

Published as a conference paper at ICLR 2024
camera
RGB-D Input
F/T sensor
Figure 11: Realworld Setup. We have overhead camera for observing top-down rgb-d images similar the simulation setup.
Start
End
Start
End
carresponding
"place purpic green cylinderr. to form wheal shape"
"first place the blue bowl in the yallow zone, then placa the block the blue bow/
"out the blocks en the four corners of the zone
"place green bow!"
along green line side
by side"
"stack red block on the blue block"
"move the spheres from the zone to the bow!
"stack block tower at the corner"
"pack pons and spoons into the box"
500
gerbage different
"place the pizza and bread from the plate the bowl"
Figure 12: We show the start and goal of the 12 tasks the real-world experiments. The involve long-horizon tasks and tasks with very distinct assets from the simulation.
on the policy side as in Shridhar et al. (2022). We note that the main limitation for this choice of pick-and-place task is due to hardware (only suction cup) at the time of experiments.
17

---

Published as a conference paper at ICLR 2024
D LANGUAGE QUALITATIVE RESULTS
D.1 TASK DESCRIPTION EXAMPIEE
We attach few generated task description examples from GPT-4.
Task: Build-Car
task-name: build-car
task-description: Construct simple car structure using blocks and cylinders.
assets-used: blockk/lockkurdf, bal//ball-emplaae.uddd
Task: Cono--Cooodinated-Bll---nneiiiie In-Boxes
task-name: color-cowrdiinated--lll---iiii -oaall-inetttiiin-bboe
-essreesciption: Pick up four balls different colors (red, blue, green, and yellow) insert them into four separate boxes of matching colors. The boxes are located inside corners of square structure and the task requires precise insertion, color coordination, and careful navigation around the square structure.
assets-used taal/ball-template... box/box- tettoo--tempatt.urff square/square weapguuuarreteplttt.tuu urdf
Task: Color- Soarddinted--onn---raaggeeee
task-name:
task-description: On the tabletop, there are nine blocks of three different colors (three red, three blue, and three green) and three pallets of matching colors (one red, one blue, one green). The task is pick up each block and place it the pallet of the same color, arranging the blocks each pallet line. However, there are a few small blocks randomly scattered on the tabletop, which rohot has to navigate around without knocking them over while transporting the blocks to the corresponding pallets. The challenge lies the precise navigation, placement the blocks, color matching, and maintaining the balance the pallets.
usec: block/block.urdh pallet/pallet.urff block/small.urdf
Task: Place- Blue--O-Liine Onn-Line-Ends
lask-name: place-blue blue-on-line-enee
task dexeriesrrition Pick up each blue box and accurately place at the end of green line.
assets-used: fine/line-templte.uudde te//box-emmllate.urd]]
Task: Move-Kit-Froni Kit-From-Zone- Toe---To--ylinn Cylinder
task-name: fove--kiffrom-zznett--gyye
dessdescri Place the specific from zone to cylinder.
assets-used: Cyylinderryylnnee-temmaaaee.c zone/zone.urdf kiting//bjecttiemmllaee
Task: Put-Blues-Around-P
task-name: put-blues-around-red
task description: Place the blue blocks around the red block.
assets-used:
18

---

Published as a conference paper at ICLR 2024
Task: Multicolor Mitticolor-BBock- Block-Bridge
task-name: multicolor-block--ridde
task-description: Build bridge by stacking three red, three blue, and three green blocks pallet. Arrange a sequence from left right: red, blue, and green. Then, place three cylinders of corresponding colors on top of the stacked blocks, forming a bridge. The cylinders should roll from the block to the pallet, creating a challenge of precision and control.
assets-used:
| block/block.un urdf, pallet/pallet.ure urdf, htt.paalcc...d..lliiiie//eeeeeer eylinderleylidder.eenpaaac
D.2
PROMPT DETAILS
D.2.1 TASK PROPOSAL PROMPT DETAILS
To reach this level of lengthy code generation, we break the prompt of the agent into several steps enforce its logical structure: task description generation, API and common mistake summary, few-shot reference code selection, and code generation. The input prompt to GPT-4 task generation stage consists of several components:
(1) available assets that are in the codebase
samples of reference tasks from the task library to act as few-shot examples
(3) past task names to make sure the agent does not provide overlapped tasks.
(4) some examples of bad tasks and the reasons such not physically feasible.
(5) some additional rules (such as do not use unavailable assets ) and the output format.
This stage has temperature=1 to encourage diversity and the rest components would have tempera- ture=0 to have some robustness. The simplified task description prompt be found below.
Bottom Up Task Creation Prompt
You are an AI in robot simulation code and task design. I will provide you some example tasks, code implementation, and some guidelines for how to generate tasks and then you will help me generate new task. My goal is design diverse and feasible tasks for tabletop manipulation. I will first ask you describe it.
in natural languages and then
you write the code for
Here are all the assets. Use only these assets in the task and code design.
Here some examples of good tasks. Try to learn from these structures but avoid overlapping with them.
Here are some tasks that you have come up with before. Try to learn from these structures but avoid overlapping with these tasks. For instance, 'bowl-ball 'bowl-ball-placemee placemetatmmntt and sort-balls-in-bowll are the same task. "ple-boxee-in-commer' and Macck-blckksinttopplleee are similar tasks, 'align- cylinder-in-corner" and ealign-ylinderrcommer similar. PAST-TASKNAM--TEMML...
Here are some example task instances with explanations.
reasons: not interesting because it overlaps with the current task "put-block-inhowl:
Now please describe the new task in natural languages and explain novelty and challenges. Format answer python dictionary with keys "task-name" value type string, "task- description" (one specific sentence) and value type string, and "assets-used" and value type list of strings. Note that
Do not assets that not in the list above.
Tasks that have more colors and shapes are interesting.
Be as specific as possible about the number, shape, and color of each asset in the descriptions.
The task need to ohcy physics and remain feasible.
19

---

Published as a conference paper at ICLR 2024
Top Down Task Creation Prompt
You are an robot simulation code task design. I will provide you some example tasks, code implementation, and some guidelines for how to generate tasks and then you will help me generate new task. My goal is to design creative and feasible simpler tasks to eventually help solve the task "TARGET-TASK-NAMEE languages and then will you write the code for
I will first ask you to describe the task in natural
Here are the assets. Use only these assets in the task and code design.
Here are some examples of good tasks. Try to learn from these structures but avoid overlapping with them.
Here are some tasks that you have come up with before. Try learn from these structures but avoid overlapping with these tasks. For instance, howl-ball-placeneee and bort-balls-in-bowls are the same task. "ple-boxes-in-commer and baack-blokk-intt-pallee are similar tasks, 'align- cylinder-in-corner' and nlign-cylinder-commr' are similar. WASTTASKNAMM-/TEMPLAT
Here bad example task instances with explanations.
reasons: not interesting because it overlaps with the current task 'put-block-in-bowl:
The goal is to solve the task TTAGGE-TASK-NAMA eventually. Due to its complexity, let's think step-by-step about what simpler task be useful achieve this goal. Please describe the new task, which alize torwards it, in natural languages in a clear and detailed way. Think step by step how this task can help contribute to the skills that are quired to solve TARGET-TASK-AAM Then format the answer in python dictionary with keys "task-name" and value type string with lower-case and separated by hyphens, "lask-description" (one sentence and do mention urdf paths) and value type string, and "assets-used" and value type
"TA "TARGET-TASK-ANAE
but can help training a policy gener-
strings, Note that
Do not use assets that are not in the list above.
Tasks that have more colors and shapes are interesting.
Be specific as possible about the number, shape, and color of each asset in the descriptions.
The task need obey physics and remain fcasible.
The simplified task code prompt can be found below.
Task Implementation Prompt
Now will provide you some reference and you can write the code for task "TASK- NAME-TEMPLATER
.... Code Ignored 1...
Do not use libraries, functions, and assets that you don't know, For each object, try to describe its color, size, category in the task first hefore you write the code. You do not need extra helper functions. Comment the code liberally explain what each piece does and why it's written that way.
Now write the code for the task "TASK-NAME TEMPLATE" in python code block starting with "python. Reminder: BEEK-STRING-TEMPEE
D.2.2 TASK MEMORY PROMPT DETAILS
In addition to storing existing generated tasks in the memory, the task reflection stage prompt has the following component:
(I) the generated task description and code
the current tasks in the library
20

---

Published as a conference paper at ICLR 2024
(3) examples of accepting and rejecting new task
Then LLM prompted answer whether to accept this new task and this improvement will also be in the context window in the next round of the agent task design. Note that to improve the robustness of this stage, we prompt GPT three times in parallel to get diverse answers with temperature 0.5, and only accept the task if there is agreement. The simplified task reflection prompt can be found below.
Task Reflection Prompt
Do you think your task is sufficiently interesting to added task for future tasks given that we already have the following task name and descriptions? Moreover, does simulation code achieve the goal and the language descriptions in the task? Be as rigorous and high-standard as possible.
Reminder: your task: TASK-STRING MASKSTRIIGGTEMPPATT TEMPLATE BASK-COD-TTEMNN current task list: CUREEN-TAKK-AMM-TTEETT
Reply explain your reasons and then say True or False, formatted python dictionary, do miss commas or add extra spaces. Here are some examples.
Please incorporate these feedbacks when you design new task!
The finetuned model usually only requires less tokens and complex prompt as input. Here is an example of the prompt for the finetuned model.
Finetuned Model Task Implementation Prompt
Write the pybullet simulation task class NTASK.NAM..TEMPP Provide answers in a python code block starting with "'python
D.3 EXAMPLE TASK LIBRARY CONVERSATIONS
We have observed interesting reference task selection in the task memory to show how the memory can benefit the task creation process.
Task Reference Conversations for Catio--Coooniiaeee-h-------tiitn Constructtntt"""""""
Question: Please pick 4 task python files that you would like to use as reference. Format them a python list.
Answer: Sure, based on the task description, the following tasks seem to be the most relevant:
httpr-ooooddiaadd--ll----e-eeeee coordinated-inseetion.yy
[
staackoolrrcooniinncccbbleessssp construct-ccolarfulaarchh color-
We also have observed interesting task descriptions and code critics from the task memory. This shows that the critic has looked into the code implementations and reaching an agreement indeed requires the task to be sufficiently high-standard. In this example, the task described as new task that would rearrange the blocks into a different fixture. but the code and its purpose is exactly the same the coolor-cordiintte-insee repeated with minor differences, while the second critic catches the coding error and refuses to add to the task library.
innrrooooiiateed--eetttooo-----e rearraange"
The first critic thinks that the task is
21

---

Published as a conference paper at ICLR 2024
Task Reflection Conversations for "Sphere-
"Sphere-Insertion" Insertion"
Input:
task-name: calor-cooddinate--nnseeee -nsertionnreearaagee rearrange
There are four colored ell objects (red, blue, green, yellow) inserted in randomly ordered fixtures on the tabletop. The task is to remove each ell object from its current fixture and insert it into new fixture in specific color order: from left to right, insert
-skkdescription:
red, bluc, green, and finally yellow.
assets-used. jinsertion/ell. urdf, insertion/fixture.urd
Critic 0:
Add to task library?: No
Reasons: The task is very similar objects into fixtures in a specific order. The only difference is that in the cxisting task, the ell objects are matched with fixtures of the same color, while in the proposed task, the ell objects are rearranged fixtures of different colors. However, this difference is minor and does not add significant novelty complexity to the task. Therefore, it may not he necessary add the proposed task to the task list.
cecoor-oddrrd-insertttie which also involves inserting colored ell
Critic
Add to task library?: ?: No
Reasons: The task similar to "color-onderrdd-ineettoon but adds a new challenge of rearranging the ell objects. This task would require more complex manipulation skills and could be interesting for developing advanced robotic skills. However. the simulation code does not order from to right described the task. randomly places the fixtures and ell objects, and the goals are added in the order of the colors array, not according to their positions on the tabletop
ensure the color
D.4 EXAMPLE IMPLEMENTATION ERRORS
We include additional misaligned task and language instruction examples as well as error messages in this section. Specifically, we have collected an "error book" to be used as reflection on LLM's failures and also include into prompts to improve task generation success rates. Here we show selected list of concrete examples and our analysis.
1. using assets (urdfs) that do not exist
2. use ambiguous language descriptions as goals. For instance, "place the colored blocks into the matching colored bowls" with one goal and sparse reward the task instead of adding subgoal "place blue block into blue bowl" and give continuous reward.
3. 'matches' in the goal has wrong dimensions. should have the same dimensions as number of objects (N) multiplied by the number of goal poses (M). Usually is N by M.
4. have vector dimension problem such as www.ddommccoicebbo.ccoo 'box size' a tuple and not an int
or "box_size / 2 where
5. make too large object for stacking or make the task objects invisible for picking.
6. accessing index out of bound 'pallet.pose(2]' for 'pallet.pose ret_pse=get_randommpp
'pal- returns a tuple (translation, rotation). It does not have 3rd component. Similarly accessing 'container.pose22 or "box_pose[2]" would cause errors as well. Since it's a tuple, try to modify it in-place will also trigger errors.
7. forget to replace str using 'fill template()' for urdfs with template such as 'cylinder- template.urde urdf". 'hall-template.ur urdf", "iinettemllate.urf"
8. use "self.ee = Spatula() as function when doing pushing tasks, which is incorrect. should be 'self.ee Spatula'.
9. mistakenly use 'random pose" for target pose. Design target poses based on task objectives.
10. add only one or fewer language goals which cause language-motion inconsistency. Note that the language goals usually are the same number as the pick and place goals.
Some concrete syntax errors listed:
22

---

Published as a conference paper at ICLR 2024
Syntax Errors
"environment. line 195", in reset (self) File "<<string>>, line 38", reset TypeError: can only concatenate str (not "list") to str
"environment.py. line 195", in reset ob object texeuddrcoet shape
nр. random. choice (object uteeconeesco shapes) in numpy. waw.ddmmmtrand.aa.. Randonsttt.ootttt...c...o. choice ValucError: must be 1-dimensional
"task- line 325", in get getlttextuderaccree random)textunderscore pose
pos [pos[0], pos[1], abjlttexuunerrcor size[2] / 2) IndexError: tuple index out of range
"task.py", line 200, \ in reward pts - objttextunderscore [ts[obi]textundersco id] IndexError: arrays used as indices must be of integer (or boolcan) type
"task.py", line 206, W\ in reward for soneNtextunderscore idx, tconeltexttudderscore pose, zomeltextunderscore size) in enumerate (zones) TypeError: 'NoneType' object is not iterable
"task.py" line 252, W ball\textundeerscor pose sel reoor random' fandomnteextunderscorr pose (env, ball ballVieettundrrcoore size)
ballNtextunderscore [0) 0.02
TypeError: 'tuple' object does not support item assignment
D.5 GENSIM FEEDBACKS
We detail the various feedback forms, which have hierarchical structure, in GenSim this section. Specifically, once simulation code is generated by GenSim, it will first be checked syntax errors by "evaluating" the code piece python. Then we run through the code to generate demonstrations. During this runtime, check both the runtime errors (such missing paths or out of bound), and if the code can successfully generate demonstrations. Successfully generating the demonstrations requires having the actual robot execution correctly achieve each subgoal and attain the full reward. This test is unique to robotic simulation, and is critical to filter out bad tasks without any real-world risks. The tasks saved at this stage could already be used for training, but in other ML pipelines, good training performance depends highly on quality. We note that there are three caveats here: (1) task code can successfully generate demonstrations but is also mundane or repeated (2) it has misaligned instructions and actual demonstrations (3) it can be perfect but the task too hard to train policy. Therefore, to ensure the high-quality tasks, our validation stage has LLM critic that automatically reflects on whether the task is interesting. We have single-task policy training evaluate if a task is good for training. Finally, we have the human verification which only requires few seconds on average to look at the generated visualization videos without any technical backgrounds, and can be large-scale deployed with mechanical turks. We detail the human efforts required to scale up simulation tasks in the next section.
D.6 HUMAN VERIFICATION
Despite the impressive performance that have seen in the GenSim system, it still has gaps to automatically generate infinite "meaningful and diverse" tasks to train robotic policies simulation. We note that in other MI. systems, data engineering efforts by humans are also inevitable, and
23

---

Published as a conference paper at ICLR 2024
User 1
User
User
Task
Time
Pass
Time
Pass
Time
Pass
Average
12.83
0.8
5.11
0.63
17.4
0,41
Table 3: Human verification time spent and pass rates on tasks generated. The average human time is around 10 seconds (if each task is checked) and the success rates are above 50%.
currently general judicious LLM evaluation still an open research question. Luckily, the GenSim pipeline, have various feedbacks before a human has to come into inspection (for instance, in order to even be considered to be saved as a task, the task needs to be able to successfully generate demonstrations and be considered by other LLM critics interesting). Moreover, even when human has to come to check if a task is meaningful and interesting, he/she usually only has to look at the generated demo and give binary answer, which is almost effortless and can be done very efficiently. Actual code modification and manual human tasks are sometimes useful, but collaborative coding with GenSim the top-down mode will also be helpful in that case. Overall, generated 30 tasks to benchmark human filtering, and we measured the amounts of human efforts for three different users with varying experiences in robotic simulation and observed that the average time spent on each task was less than 10 seconds. This usually means looking at the visualization videos and checking if the robot actions match the language instructions, rather than actually checking the code. See Table 3 for more details. We note that the human time spent and the pass rates are likely subjective and depend on each person's 's familiarity with the contexts of robot simulations. One interesting future direction is to comprehensively study how the quality of the generated tasks might affect the quality of the trained policies. For instance, what levels of degrades can be attributed to misaligned language instructions. For example, GPT-4 model sometimes hallucinates concepts such "boundary" and "ascending size" without actually implementing them in code. This language annotation problem will not show up in single-task policy training, but it might affect multi-task policy training in ways.
D.7 FRAMEWORK SCALABILITY DISCUSSIONS
Although we have shown scaling the simulation task generation with foundation models, we discuss how to improve the efficiency and scalability of such systems in the future. We first note that such system can support both automatic generation (exploratory) and co-pilot generation (goal-directed) and it reduces the technical backgrounds required for making new simulation task to train robot policies. We usually can let the program run for an hour or a few hours and then check back on the generated tasks. One issue we have observed is that codebase modifications to support cleaned and documented code structures as well as adding fixes tolerate some common hallucinations is important to improve the success rates. Two things that plan explore in the future larger-scale retrieval augmented generation (RAG) and advanced sampling in task references. These important provide contexts to LLM and guide toward generating more balanced tasks. Finally, we note that such a system can already be deployed with Amazon Turk to scale up more tasks.
D.8 TASK CODE EXAMPLE
We attach a few generated code examples from LLM (library imports ignored). The generated task visualizations are also attached.
24

---

Published as a conference paper at ICLR 2024
Task: Color- Colored Insertion
class ColorOrderrdInsertion (Task)
"""Insert differently-coloree ell objects into the matching color fixture in a specific order. uun
def (self)
se' famax_ssteps 20
self feif.langttemplate - "put the (color) I. shape block in the shape hole"
seif.task htt.taaskccompeteed dese "done with insertion."
self.additional reset ()
def reset (self, env)
super reset (env)
# Define colors and their order colors = ['red', 'blue', 'green', 'yellow'] color_order (color: i for i, color in enumerate (colors))
# Add fixtures.
fixture_size (0.12, 6.12, 0.02)
fixture_urdf
bebertion/fixture.urr
fixtures []
for color in colors:
fixture_
Bactiennppsseeeeeeeeeeeeeeeeee (erv,
sixture_size
fixture_id jo (fixture_urdf,
fixcure_pose, color=uti1ss.011sssooo
catogory=' apptures
Add ell objects.
ell eii_size (0.04, 0.04, 0.04)
cil_urds 'Lnsertion/e_l.urde
eils []
for color in colors:
ell_pose self.get__andon_pse (env, ell_size)
ell_id - anv.add (ell_urdf, ell_pose,
color-utils wot.uutiiccoooos_c COLORS [color]) aplssappenddell
Goal: each ell is inserted into the matching color fixture in the correct order.
for i, ell in (ellss)
belf.ada add.fadd__olllll gold_goallobb]] (obbssseell) matches=np. ones ((1, 1)), wrg.ppssss..ggtaaaeaaattttoooooe
[ixtures[i])], replace
rotations-Tne
metric='posee pose', params-None,
step max rewarc=1 (ells),
language
wwwaaaae_oaa=ssel..o gwwl=self. www.gnggeemeeeeeeeee format
color=colors (11)) (il))
25

---

Published as a conference paper at ICLR 2024
Task: Four- Caar--Coeer-eee Comee--yrraid Pyramid-Challenge
class y (Task):
"""onnstruct pyramid of blocks in each with specific color sequence. "**
def _iiit_(selff))
super super()._ini___. self.max_steps 20
deif.lang_template "build pyramid blocks in each zone with the sequence red, blue, green, and yellow from botton to top"
self.task_completeed desc desc "done building pyramids." sel self.addiitional_reee ()
def reset (self, env)
super reset (env)
# Add zones.
cone_size (0.12, 0.12, 0)
zone_urdf "zone/ zone. urdf zone_poses []
for in range
zone_pose self saaff.get_ranom_posetttt
zone_size]
env.add env.add_object (zone_urdf, zone_pose, "fieed')
wea.poses.appenne (zone_pose)
Block colors.
colors -
stils.COLONS red' 1,
['blue']
utils. COLORS green' 1, utils.
vcilO.COLORS
scils.coLORS COLORS yellow')
# Add blocks.
block_size = (0.04, 0.04 0.04)
block_urdf bblock/block.urd
blocks = []
for i in range
for in range
block_poss
sel selif.getranddo__pss (env, block_size)
block_id env object atjjec (block urdf, block_pose, color=colorslil (bber id)
Associate placement locations for goals.
place_pos = [(0, -0.05, 0.03), (0, 0, 0.03),
(0, 0.05, 0.03), (0, -0.025, 0.08),
(0, 0.025, 0.08), (0, 0, 0.13)]
targs apttils.apll (zone_pose, i), zone cone_pose[1] for place_pos for zone_pose zone_poses]
# Goal: blocks are stacked in a pyramid in each zone.
for in range (4):
self.add_goa1 (objs-bl ockks [1x; (1+1)
cocheessnp..nes(((i) 4)),
targ i
replace=False,
rotations=True, metric='pose" parems-Nene,
step ht__maa___ee_e_ reward=w
(ommetrries<(n.p1/22++
Language_gcal-see1...1 hat.://el....aagggmaaaaaaaml
26

---

Published as a conference paper at ICLR 2024
D.9 TOP- Do--DowN EVALUATION EXAMPLE
In this section, we showcase different LLM's output for an example task in the top-down evaluation.
Tnsk (Zero-Shot Prompt Before Code-LLAMA Instructt Fine- tuning)
class ogl.ndeericcloofuoooottaaneeeee Task)
def _init_(seef,, cylinder_radlus, cylinder_height,
container_size, container_color)-
seif.cyllinder_rddiu cylinder_radiie radius
seif.cyllindrr_eeihht cylinder_height
self.container_siee container_size
self.contaiieerccolo: container_color
def create create_cylingte (syiee__ccyinddessss111
cylinder_id inde (self.c//irrerrrradds
self.cylinder__hiieee return cylinder_id
def create_connt (oontainerrself))
container id p.ertte box (self. contssse..onttaerr.sie....
sols.conttineer_color
return container_id
def add add__ch cylinder_id, container_id): p.ce__olcccc_t_oooottttrrrrrr//
container_id)
def (self):
cylinder cylinee_y ()
self sea.f.createccontiinne ()
container_id
container_id) p.step_simulation return cylinder_id, container_id
27

---

Published as a conference paper at ICLR 2024
Tuskeeylinder-in-coooffu container (Few-Shot Prompt Before Code-LLAMA Instruct Fine- tuning)
class CyLimderthhooofffuCCottttaerrr (Task)
def _ (self):
super () in__iit
melf.max_steps
seif.lang_template "put the cylinder in the colorful
container"
selff.task_completeddddd desc "done with insertion." self self.aadditional_eeet(((
def get_ranndon_pose(eelf,, env, obi_a
pose super ()ppeol(_ee_tradommmppo... (env, obi_size) pos, rot pose atdiiis.elerre/._ooouaaavav
np-pi 2)
return pos, rot
def reset (self, env) super reset (env)
"*"Add cylinder. nnn
size (0.1, 0.1, 0.C4)
urdf 'insertion/cylinder urdf'
pose self helf.get_randon, wwseeannoooeeeeeee_eee pose (env, size)
cylinder_iis env.add (urdf, pose)
"*"Add colorful container to place cylinder. "** size (0.1, 0.1, 0.04)
urdt L.aeriion//orttaaee..ddd
targ_pose seaaf.get_ranoo_poo (env, size) env.add_object (urdf, tasg_pose, (fixed')
self goal log_ id), matches=-pp.nnt32 ([(1))), parg__oses=___ppeepsssssssssssss [targ_pose]]]
replace=False,
rotations-False, metric='pose" params-Kone,
stop_max_rexard-1, symmetries=12 np.pil,
language_goal=sclf tongoooo
28

---

Published as a conference paper at ICLR 2024
Task: gelinder-n--coloful-oooo (After
Finetuning)
import numpy
from cliport. import Primitive from cliport tasport.tasss..... tasks. task import Task from cliport. import utils
class tyLindericcooofffllotttttttrrer (Task)
"""Pick up each cylinder and place into the container the same color, in specific color sequence. .**
def (nnit_(self))
super ()__ini)) sei f.max_ateps 20
self. aaf.iangttemmlttee "place the (color) cyl inder in the (coler) container*
telf.task_complette desc "done placing cylinders in containers." self.additional reset ()
def reset (self, env):
super reset (env)
# Define the color sequence color_sequence = ['red', 'blue', 'green', 'yellow')
# Add containers.
container_size (0.12, 0.12, 0.12)
container_un ccontai atttaa/eeeetttaiiie-eeeellaaeee container_poss []
for color in color_sequence:
container_ pose self self.get_ pose (env,
container_siin size)
env object (containez_urdf, container_ pose,
etv.add_object
'fixed', color=coler) container_poses append (container_pose))
# Add cylinders.
cyl cylinder_size = (0.04, 0.04, 0.04)
cylinder_urdf 'cylinderccyl inder-tenplate.
cylinders
for color in color_sequence:
cylinder_poo soceerppsseeee =
randon pwseaanooosseeeeee_eee (env, cylinder_siiee
cylinder_id env anv.add_object (cylinder__rdff cylinder_pose, color=celor) h.liiders.aapendcccc (cylinder_id)
Goal: each cylinder is in the container of the same color, in the specified color sequence. in range (len (cylinders)):
self
(1)
aar__poss_[comtaaieer_essss
replace=False,
rotations= mesric="pose' params-None,
step_max_reward-l1lee ( inders),
language watpeeelooaaaeel.oooee htg. (cl.ooooooooooooooooooooooo sequenceeeeeeeeeee iil))
29

---

Published as a conference paper at ICLR 2024
Task: cylinder-in-conl sylinde-inncoooffuul-cotiiiiiii (GPT 3 (davinci) Finetuned)
import numpy as
import
import pybullet P import random from cliport. import primitives from from claport.tasks.tas. import Task
grippers import Spatula
from cliport.utils import utils import numpy as up
from cliport.tasks import primitives grippers import Spatula
from eliport. tasss...p.e.....s
from cliport. ttiport.tasks.ooo. task import Task from aliport.atils import utils
class Dy (Task)
""*Put a cylinder into a container a specific color"""
def in (self)
super in self.max_steps 10
self.lang_template "put the cylinder into the (color)
container" self.taa__compeeeeeee.ee "done putting cylinder in container... self.additiona__reset ()
def reset (self, env)
super resett (env)
# Add cylinder.
cylinder_size _ize = (0.C2, 0.02, 0.05)
cylinder_urde byylinder/cyLinder.ucee
cylinder_pose solf.gee_random_pssaaeaa cylinder_size)
eylinder_id env. ony.ad Loyddrrrrrd cylinder_pose)
# Add containers.
container_ai size (0.1, 0.1, 0.1)
container_urd: wyy.inder/conaanee..rrr
# Colors of containers.
$ IMPORTANT: RETRIEVE THE ACTUAL COLOR VALUES container_colors lutils.COLORS [c] for utils.COLORS if à !- rrd']
container_poses []
container_ids
in range (3):
container_pose = seif.gge__raddm__poo (env,
container_size)
container_id a (container_urd)
container_pose, ittoorocontaiee__ooolooo
container_poses append serntttiier_______ container ids. acon id)
Goal: cylinder is in sel f. aedfgadd_goal (odss-[oyynddddd_ddd matches-np.ones (1,
container.
3)), targ_pooss-ottaleeeepeees rotations=False,
replace-True,
patric='pss'' params-None,
step_mas max Betep_ma__eewaaeee (aaxx_ewwrd_1)
sel f. seif..angg_oals.apper (self baageeeeesl..ggteeeealteeeeeeee format (color="red"))
30

---

Published as a conference paper at ICLR 2024
Task: cylinder-in- in coiinder-i--cooffuu--cttttieeee (GPT3.5 Prompting)
import numpy
from teporr.ttskk.t.... import Task
from pniport.utill import utils
class @g._mdeertiicooofuuullttttttttt
def _init__ (self)
() se seif.max_atops 10
seif.Lang_templetee = "Flace the (color) cylinder in the
(color) container*
sel sel fe..taaaccooolllseeee.ee "Cylinder" placed in containers."
sei saif.aditiooal_rree ()
def reset (self, env)
super reset (env)
I Define colors and corresponding containers
colors red'' 'blue', 'green', 'yellow'] containers ['red', 'blue', 'green', , 'yellow"]
# Shuffle the colors and containers to avoid always
picking the same combination np. ww..randm.ssuufile...... (colors) www.anddonccufffeeeeootttateeaae
# Add containers
container_size (0.1, 0.1, 0.1)
container_template o urd
container_post []
for color in containers:
container_pose = soiff.et__radon_poss (env,
container_size)
container_urdf
COLOR', coler)
env.add_object (container-urds, container_pose,
"fixed')
container_pooe poses append (container_pose)
# Add cylinders
cylinder_ste size = (0.05, 0.05, 0.1) cylinder_templatee
ssze
cylinders
for 1, color in aumerate
cylinder_pose = self.get
_ sareee_rraddo___ee___e_e (env,
hylinder_sizer
cylinder_urdf
htt.iidee_temppateeeeppleeeeee COLOR',
color)
cylinder_id Jev (cylideerrrdl
cylinder_poor pose) cyl indders (cy' inder_id)
Add goals
for i, color in enumerate (colors): language goal =
sels. template...eeeee..e. format (color=color)
hall:laan_ttemlate
self add_gal (objs= [i]]
matches-np. ones ((1, 1)),
[os_ppsee_(cottaanee__ooo [i]], replace=False, rep.eeefeeel
rotations=True "arric='pose""
params-None, step step_max___________ ma__ma______ max_reward=l,
lagguage_goal-lagguaeeoooo
31

---

Published as a conference paper at ICLR 2024
Task: sglinderii-coloofful-ccttiiiii (GPT3.5 Finetuned)
import numpy as
import
import pybullet P import random
from cliport. tasks import primitives from from cliport.tasks.ter import Task from cliport.utils import utils
import Spatula
class CyLindeenncooofuuooottteeee (Task)
"""Pick up cylinder of specific color and place it into container of the same coler, n.
def iniit_(se11
super () init__)
self self.max_sse steps 20
steps_p_pppp______m___ seif.lang_template "put the (color) cylinder the (color) container*
"done placing cylinders in colors - ['red', 'blu'', 'green', 'yellow"] containers = []
seif.task_complotess containers. seif.additional_resstt
def reset (self, env) super reset (env)
Define colors and their corresponding containers and cylinders.
cylinders
for color in colors:
$ Add container.
container_size = (0.12, 0.12, 0.12)
container_pose self.get_random_pp (env,
container_size)
container_id
env. env.add_object (' (
container_pose poseen, colcr-utils. COLORS (co00 containers..ppeedd
id)
$ Add cylinder.
cylinder_stie size = (0. 04.04, 0.04, 0.04) cylinder_poon sel geef..get_rnddn_ppss (env, cylinder_size)
cylinder_id
unv.add_objeer htypiaider/guinieeeettaaaaatt urdf''
cylinder_poo pose, calcrrutils..cooosscoom ley.eerss..ppenlleeeeeed.eeeee
# Goal: each cylinder is in container of the same color.
for in range (1en (colors)
sel felf.ad falf_add_goal (objs- inders [nfrrt])
srecches-pp.ones 1)), tase_
= iip.seeesssssiiiconnniettttttiit
containers(il)11 rep.aleeaae.......aaee...
rotations=True, http:///eeeemirr/aaaaaaaa metric='pose' params=None,
stog_maa_reewrrrcc//eooooll..ll (colorr)))
bag.gaa.ggggggoooooorrroaaaaaaaa (colo (color=colord (il))
32

---

Published as a conference paper at ICLR 2024
Task: sylinderiin-oooofful-cctttiiiii (GPT 4)
import numpy as
import
import pybullet P import random from cliport. import primitives from from cliport.tasks.cer from cliport.util. import utils
import Spatula
import Task
import numpy as up
from siiport.taskk..te import Task from cliport.utiae import utils
class I
(Task)
""epick up four cylinders of distinct colors (red, blue, green, yellow) and place each of them into a container of the same color. ***
def iseit
super () ()__init___ self self.max_dee deif.lang_template "put the (} cylinder in the () container"
self.max_stops
safffttask_cmlletteddd = "done placing cylinders in containers." self
def reset (self, env) super reeset (env)
# Define colors colors ['red', 'blue', 'gzeen', 'yellow')
# Add containers.
container_size (0.12, 0.12, 0.12)
container_templatoe oonttalherrcootaalorr-oooa...t. urd container_poses []
for color in colors:
container_pose se' seef..eetrrndo_ppssseeee
container_sizer
edV.add_object
container_pose,
category=' fixea', coLor=utils..oOORss (olor)) container Baciiineerrosssseeee append (container_pos
Add cylinders.
cylinder_size = (0.04, 0.04, 0.12)
Bilinder_templast = l
cylinders =
for color in colors:
cylinder_pose sclf.get_randor_psse (env,
cylinder_size)
cylinder_id env otv.add_eebjectccyyylnee templaate,
cylinder_pose, cy1 (
Goal: each cylinder is in container of the same color. for i in range (Ly(iindedrss) self goal (objs= cylinders cy matches-mp.ones 1)),
targ_poses- (onnsaooosocntttaneee__ooooo [i]], replace-False,
rotations-True, mstric=/
params-None,
step_ma ree__max_rewarddl/1en (cylinders)
self. 1lff.langgggals wptp:_ggals..ppenddssee. (se.. atp format formatt
colors (i], colors (11))
33

---

Published as a conference paper at ICLR 2024
LET
34
Figure 13: Gallery generated tasks by GPT-4.

---

Published as a conference paper at ICLR 2024
"Put the (color) L shape block in the L. shape hole"
"Build pyramid of blocks in each zone with the sequence red, blue, green, und yellow from bottom to top"
Figure 14: The simulation task visualizations for crdoor- insertion" and foour-conerrpprrmmt challenge" correspond to the implementations below.
35


Reviews:
[{"review_id": "PVckJUssoG", "reviewer": "Reviewer_XopG", "summary": "This paper proposes a methodology to automatically generate tasks in simulation and corresponding expert policies to increase the task diversity when training robot policies. Existing large language models are utilized for this, and a number of ablations are conducted to study various design choices of the proposed method. Experimental evaluation both in simulation and in the real world reveals that the proposed methodology leads to better generalization than the baseline.", "questions": "- A number of approaches exist in the literature for data augmentation using foundation models (e.g. ROSIE, CACTI, etc.). How do such approaches compare to the proposed methodology when it comes to generalization to unseen tasks? Arguably, some of these are simpler to implement and use.\n- What is done in the cases where the output of the LLM has a syntactical error and cannot be run?\n- How much prompt engineering went into this?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Interesting idea. The proposed idea is very interesting and novel.\n- Well-written. The paper is very well-written and easy to follow. The figures very much aid in understanding the paper.\n- Experimental Evaluation. The experiments reveal that the proposed methodology outperforms baseline methods by non-trivial margins. In particular, the proposed method enables both few-shot policy generalization to related tasks and zero-shot policy generalization to unseen tasks. The experiments are conducted in both simulation and in the real world, which further speak to the strength of the proposed method.\n- Ablations. Ablations are conducted to better understand various design choices of the proposed methodology. Visualizations and qualitative results also aid in this.", "weaknesses": "- Simplistic tasks. The experimentation is limited to table-top pick-and-place task. Furthermore, the use of a suction gripper is a weakness -- this simplifies the already simple task, and limits generalizability to other kinds of tasks. Unclear whether the proposed methodology can be adapted for more complex tasks, and how well it would do. Experiments on more complex tasks would be more convincing.\n- Experimental Evaluation. While a diverse set of experiments are conducted, only one setting (ie environment) is adopted. The results would be more convincing if the presented trends were shown to hold true across a number of settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yc3yaNAF5c", "reviewer": "Reviewer_1vRz", "summary": "This paper addresses the problem of automatically generating robotic tasks in synthetic simulation environments. A method relying on LLMs is presented, where a LLM is used to generate the code that describes a task (assets selection, initial configuration of the assets, and language/robot motion instructions). A task library is initialized using 10 examples written by a human, and the LLM is asked to generate new tasks given carefully designed prompts. The generated tasks are added to the task library if they pass validation tests (e.g. the syntax is correct, the success rate of a policy trained on this task is high enough, it passes some human inspection etc), or the feedback is explicitly given to the LLM as additional instructions in a few-shot prompting scheme. 120 tasks are generated and policies are pre-trained on these tasks. The experimental evaluation shows that pre-training on a large number of tasks increases zero-shot generalization performance on new tasks (held-out test set also generated) in both simulation and real environments after fine-tuning.", "questions": "The paper addresses an important problem and the method is technically sound. The experiments demonstrate that policies pre-trained on the automatically generated tasks can generalize to novel tasks in both synthetic and real environments. I am however concerned about the diversity of the tasks which the method is evaluated on. In particular, the evaluation tasks seem to be less varied compared to the ones used in CLIPort (W2). I am also questioning the “automatic” nature of the method as human intervention is required (W1). I will gladly update my review if the authors can address these concerns and respond to my questions (see weaknesses).", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. The problem addressed in this paper is important. Multi-task robot policies strongly rely on diverse training data (in terms of environments and task) but access to such training data is limited by real collection efforts, or having to manually invent and design new tasks. Automatically generating such training data could have a significant impact.\n\nS2. The overall method is technically sound. The appendix provides many details. The details of the prompts which are provided to the LLM is appreciated. Their design design is key as demonstrated in Fig 6 left). However, additional intuition why the two-stage prompt chain is better than the single-prompt would be appreciated.\n\nS3. The evaluation of Table 1 is convincing. In this experiment, multiple models pre-trained on synthetic data are fine-tuned using few real demonstrations. The models pre-trained on 70 generated tasks performs significantly better than a model pre-trained on 50 generated tasks and on the 10 original tasks from CLipport. This experiment demonstrates the practical applications of the proposed approach.", "weaknesses": "W1. Figure 3 mentions that human inspection of a generated task is required. Was feedback collected for each of the generated tasks ? How many tasks were manually annotated in total. In addition, what is the granularity of feedback provided by the human ?\n\nW2. The experiments presented in Table 1 use a different set of tasks than the real tasks used to evaluate CLIPort. In particular, the tasks do not involve sweeping, or folding of objects. The variety of asset is also smaller than what is shown in CLIPort where scissors or chess pieces are manipulated. A similar observation is made regarding the synthetic experiments, e.g. the held-out simulation tasks used for evaluation mostly use blocks and omit more complex objects like ropes. Additional explanations on how these tasks were selected are needed to convince the readers that the method indeed generalizes to a wide variety of tasks. Experiments presented in Fig 6 left) evaluate models on the same tasks as CLIPort but it is not clear if the models were pre-trained on all generated tasks in this experiment (see W3.)\n\nW3. The “few-shot policy generalization to related tasks” experiment needs additional description to understand the meaning of “single-original, single+2, single+4 etc”. Were the models pre-trained on original clipport tasks, original clipport tasks + 2 generated tasks etc ? Or are the models pre-trained on all CLIPort tasks + all the ones that were generated automatically ? It is important to clarify this experiment as the current version lets the reader think that only a small subset of of the generated tasks were used to pre-train the policy, and the question of how these were selected arises.\n\nW4. Page 6 mentions “When only pretraining on the 10 tasks in clipport, the policy does not generalize well on the GPT4 tasks.” In fig 7 right) the model pre trained on tasks generated with LLMs is better than the one trained on CLipport tasks, but the difference is 15-20% of relative improvements, the claim should therefore be mitigated. It is also suggested to justify how the set of testing tasks (given in Appendix A.1) was selected. See also W2.\n\n W5. Details of the metrics used in Fig 6 would be appreciated. In particular, how is “code reasoning capability” measured in “runtime-verified”, and is the “task completed” an average over several environment resets ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QPc8iZFKCT", "reviewer": "Reviewer_hwME", "summary": "The paper proposes a framework for generating new tasks given a fixed number of existing tasks by leveraging the code generation capability of LLMs for robotics simulation environments. It further uses the generated over 100 tasks for training multi-task policies and evaluates them in both simulation and real world. Stronger generalization is observed by training across more diverse tasks. The idea is novel and profound for breaking the hardness of creating new robot learning tasks in simulation.", "questions": "In Sec. 3.1, for exploration task generation, which LLM is used? Is it fine-tuned or GPT models without fine-tuning? Also for the generation of 120 tasks, are the newly generated tasks used for fine-tuning as well or just as prompt examples? Please provide more explanations.\n\nFor Fig. 6, it seems GPT-4 without fine-tuning outperforms all other fine-tuned models, so is it true that using fine-tuned smaller models is just for consideration of inference cost?\n\nAs mentioned in Sec. 1 the task library is initialized with only 10 tasks, for fine-tuning the LLMs to gain context of robotic simulation tasks, is the overfitting to these 10 tasks a problem? If so, how is it handled?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The task-level diversity is indeed a hard problem due to the required human efforts and this paper proposes a promising approach for solving it, by a novel usage of LLMs. It verifies the effectiveness of this approach and is inspiring.\n\nThe strong generalization of the multi-task policies even with zero-shot transfer on new tasks is impressive. \n\nThe paper is well-written.\n\nThe experiments are thorough for verifying the generalization capability of the policies.", "weaknesses": "Most of the current tasks are within the domain of top-down pick and place, and the generalization of policies within such a domain is relatively easy. It could be more impressive to see more dexterous manipulation tasks and policy generation over those.\n\nFor the failure cases of task generation, perhaps providing more examples will draw a better boundary on the limitation of the current approach, or for each model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HswuhN1oHy", "reviewer": "Reviewer_LF4g", "summary": "In summary, this study introduces GenSim, a scalable framework harnessing large language models (LLMs) to enhance robotic policy simulation tasks. \n\nThe primary goal is to distill LLM capabilities into practical, low-level policies for robots. We delved into LLM finetuning and prompting methods, exploring both goal-directed and exploratory approaches to generate novel simulation task codes. \n\nThe study focused on table-top pick-and-place tasks, and addressing challenges in generating intricate robotic tasks remains an area for future exploration. Moreover, future research could explore algorithms for training multitask policies capable of efficiently adapting to these larger-scale generated task benchmarks, thereby paving the way for more sophisticated and realistic robotic simulations.", "questions": "Could you provide more details on the types of syntax errors and lack of grounding observed in the generated code? Understanding these specific issues would be valuable for refining the framework. Additionally, how do these errors impact the subsequent policy training and task performance?\n\nConsidering the limitations and challenges identified, what are the key areas of future research you are planning to explore? Are there specific aspects of GenSim that you aim to enhance or refine based on the feedback and insights gathered from this study?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors successfully leverage Large Language Models (LLMs) to tackle the issue of limited task-level diversity in robotic simulation. By tapping into the reasoning and coding capabilities of LLMs, the proposed GenSim framework autonomously designs and validates task asset arrangements and progressions, leading to the creation of over 100 tasks.\n\nSecondly, the paper presents a meticulous evaluation of the generated tasks using tailored metrics, ensuring the tasks' quality and achievability. The authors go further to compare different LLM models, including GPT4 and Code-Llama, and demonstrate that prompting and finetuning techniques based on the task library significantly enhance the LLMs' ability to generate tasks of higher quality. \n\nAdditionally, the paper showcases the practical impact of the generated tasks on language-conditioned visuomotor policies. The rigorous evaluation and comparison of different LLM models, and the tangible improvements demonstrated in the generalization capabilities of language-conditioned visuomotor policies.", "weaknesses": "Although the paper discusses sim-to-real transfer briefly, there is a lack of in-depth evaluation regarding the effectiveness of policies trained on LLM-generated tasks in real-world robotic applications. Generating over 100 tasks is a positive step, but the computational resources, time, and effort required for this scale of task generation are crucial aspects, especially when considering real-world applications. Without addressing these scalability issues, the framework's practical utility might be limited.\n\nThe paper acknowledges the imperfections in the evaluation metrics used for code generation, including misaligned language descriptions. However, it does not delve deeply into the potential implications of these imperfections on the quality of the generated tasks. A more comprehensive analysis of how these imperfections affect task quality and subsequent policy training is essential.\n\nCompared with the vision modules, the language planning modules are much more sophisticated. The potential of this model is not fully unlocked.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)