ROBO-INSTRUCT: Simulator-Augmented Instruction Alignment For Finetuning Code LLMs

## Abstract

Abstract Open-weight LLMs are particularly appealing choices to generate training data for fine-tuning Code LLMs on domain-specific service robot applications because they are cost-effective, customizable, and offer better privacy protection. However, unlike proprietary LLMs, open-weight models are more error-prone and often produce programs that violate domain-specific constraints.
A promising solution is to incorporate a robot simulator with a well-defined environment to verify program correctness. Yet, these environments require pre-enumeration of relevant entities and their states, which limits the diversity of programs that can be effectively verified.
In this work, we introduce Robo-Instruct that
preserves the diversity of programs generated by an LLM while providing the correctness of simulator-based checking. Robo-Instruct introduces RoboSim to dynamically synthesize consistent simulation environments for each generated program.
Moreover, Robo-Instruct handles subtler instruction-program inconsistencies that do not result in a constraint violation via InstAlign ,
an LLM-aided instruction-program alignment process.
Given domain-specific APIs and a few seed examples, Robo-Instruct can leverage an 8B Llama3 model to generate a training dataset for fine-tuning a 7B CodeLlama model. Our fine-tuned model achieves a 28.75% improvement in pass@1 over the original base model and a 13.75% improvement compared to its Self-Instruct -finetuned counterparts, even surpassing the performance of a few proprietary LLMs, such as GPT-3.5-Turbo and Gemini-Pro.

## 1 Introduction

Large language models (LLMs) have shown great promise at leveraging domain-specific application programming interfaces (APIs) and generating robot programs from natural language instructions (Hu et al., ; Huang et al., ; Biggie et al., ; Liu et al., ; Wu et al., ; Liang et al., ; Singh et al., ; Huang et al., ).
For instance, by formulating a robot’s navigation and perception skills into APIs, such as go_to(location) and is_in_room(object), an LLM can generate a program for a service mobile robot to complete the task: *“Determine the number of conference rooms without markers”*. However, despite impressive results, the performance gap remains wide between proprietary and open-weight LLMs in generating robot programs from domain-specific APIs (Hu et al., ). To bridge the performance gap, Self-Instruct (Wang et al., ) is a popular method for generating domain-specific data for finetuning LLMs. Given the definition of APIs and a few seed task examples, Self-Instruct prompts an LLM to generate diverse instruction-program pairs as training data.

Figure: Figure 2: Examples of programs violating domain-specific constraints.
Refer to caption: x1.png

This makes Self-Instruct a seemingly appealing way to prompt open-weight LLMs to generate training data, for the fine-tuning of Code LLMs on domain-specific service robot applications (due to their cost-effectiveness, better privacy protection, and customizability).
However,
open-weight LLMs are still prone to errors. As a result, using Self-Instruct naively can produce low-quality data, such as programs that violate domain-specific constraints or instructions that are infeasible for the robot to execute. For instance, as shown in the figure Fig. 2, the program may attempt to pick up an apple that is not present in the current location (example 1); or the instruction might ask the robot to pick up multiple objects simultaneously, which is physically impossible (example 2). To catch these errors, executing the program in a simulator with well-defined environments may appear promising. However, such environments require pre-enumeration of relevant entities and their states, which can result in rejecting
a vast number of programs generated by Self-Instruct even if they are executable, simply because the pre-defined simulation environment fails to capture certain relevant aspects (e.g., the presence of an “apple”); this undermines the diversity of the data generated.

This work introduces Robo-Instruct to bring the best of both worlds — it preserves the diversity of programs generated by an LLM while providing the correctness of simulator-based checking. Robo-Instruct is a new framework based on Self-Instruct, to address the challenges posed by using an open-weight LLM for generating domain-specific robot programs. As shown in Fig. 1, Robo-Instruct introduces two novel components: (1) RoboSim, a task-agnostic simulator that encodes domain-specific constraints and validates robot programs generated from Self-Instruct. Critically, RoboSim *dynamically synthesizes consistent simulation environments* starting from arbitrary programs.
(2) InstAlign, an LLM-aided instruction-program alignment procedure that revises the generated instructions to better reflect the intent of the generated programs.
Robo-Instruct also uses a rejection-sampling mechanism to discard invalid programs detected by RoboSim and query Self-Instruct for a new program based on the same instruction.

We use an 8B Llama3 model (AI, ) to generate instruction-program training data and fine-tune a 7B CodeLlama model (Rozière et al., ) on RoboEval, a domain-specific benchmark for service robot programs. Our Robo-Instruct-fine-tuned model significantly outperforms the base model by 28.75% in pass@1 scores and achieves a 13.75% improvement over its Self-Instruct-fine-tuned variant. Moreover, it surpasses several larger code models, including Deepseek-Coder-33B (Guo et al., ), Starcoder2-15B (Lozhkov et al., ), GPT-3.5-Turbo (OpenAI, ), and Gemini-Pro (Team et al., ).
Finally, in real-world deployment, we demonstrate that Robo-Instruct achieves significantly lower latency compared with GPT-3.5 and GPT-4.

##### Contributions

Our main contributions are as follows (^1^11Project Website: https://amrl.cs.utexas.edu/robo-instruct/)

- 1.
We introduce Robo-Instruct, a new framework for improving open-weight LLMs’ ability to generate training data for finetuning Code LLMs on domain-specific service robot applications. This framework introduces two novel components, RoboSim and InstAlign.
- 2.
We present RoboSim, a method to *synthesize consistent simulation environments dynamically* for verifying the generated programs. This method preserves the diversity of programs generated by an LLM while providing correctness with simulator-based checking.
- 3.
We present InstAlign, a simple *instruction-program alignment procedure* that revises generated instructions better to reflect the actual results of the generated program.
- 4.
We show that the Robo-Instruct finetuned model can significantly outperform the original base model and its finetuned variant using Self-Instruct in generating domain-specific service robot programs. It also surpasses several other larger Code LLMs.
- 5.
We demonstrate that the Robo-Instruct fine-tuned model has significantly faster inference than proprietary models, making it more suitable for real-world robot deployments.

## 2 Robo-Instruct

In this section, we describe how Robo-Instruct generates training datasets for domain-specific service robot programs. Fig. 1 provides a high-level overview of the framework. To add an entry to the training dataset, Self-Instruct first generates an instruction-program pair based on robot APIs and seed tasks (detailed prompts in Appendix A.4.2). Then, RoboSim dynamically synthesizes a consistent simulation environment to validate the program as it executes.

If the program is invalid, Robo-Instruct applies a rejection-sampling method, which generates a new program based on the same instruction and re-evaluates it. This process continues until the program is valid or a predefined maximum resampling limit is reached. If the limit is exceeded, the instruction may be incompatible with the domain-specific APIs or too complex,
so the instruction-program pair is discarded.

Finally, if the program is valid, InstAlign uses an open-weight LLM to refine the instruction to better align with the program’s intent, and the revised instruction-program pair is added to the training dataset. In the following subsections, we delve into the specific design of each component.

### 2.1 RoboSim : Dynamic Synthesis of Simulation Environments for Program Validation

We present a principled approach to designing RoboSim, a system for dynamically synthesizing consistent simulation environments to validate domain-specific robot programs. For service mobile robots, a simulation environment often relies on three concepts:

- 1.
A list of *entities* to reason about, e.g., “apple”, “kitchen”
- 2.
The *type* of the entities, and hence their affordances, e.g., “apple” is an object, you can pick it up; “kitchen” is a location, you can go to it, and it contains objects.
- 3.
The *state* of the entities in the world, e.g., the “apple” is in the “kitchen”.

These concepts are closely related to the domain-specific APIs, where each API invocation during program execution can trigger updates to the simulation environment. To handle the interaction between APIs and simulation environments, we introduce DynamicEval, an algorithm inspired by Angelic Execution (Broy and Wirsing, ), a software engineering technique to infer program properties from incomplete API specifications.
DynamicEval automatically generates a simulation environment for each program and checks its correctness within the inferred environment.

Figure: Algorithm 1 $\textsc{DynamicEval}(\text{api_fn},\text{api_inputs},\mathcal{W})$

As shown in Alg. 1, upon each API invocation, the corresponding inputs and the current simulation environment are passed into DynamicEval. DynamicEval first infers relevant entities when they appear in the program being checked (line 5). For instance, if a program includes the statement pick(”apple”), DynamicEval infers that apple is an entity to consider, even if it is not currently defined in the simulation environment.

For each relevant entity, if it has already been initialized (line 7), DynamicEval infers its type and new state from the API invocation and checks for any inconsistencies with the current simulation environment (lines 8-10). For example, pick requires an object type, while go_to requires a location type (^2^22In this example, type compatibility check is strict (i.e., “apple” is only an object and no further inference is made about its location). Nevertheless, the algorithm is also capable of handling more advanced scenarios.). If a program contains:

By invoking API calls sequentially, DynamicEval first infers that “apple” is an object and then raises an error when go_to(“apple”) is called. If no inconsistency is detected, the simulation environment is updated accordingly based on the API definition (line 11).

On the other hand, if the entity has not been initialized, it will be assigned a random plausible state (line 16). For example, the API is_in_room(object) checks if an object is in the same location as the robot and expects a boolean return. In this case, DynamicEval assigns a 50% probability for the apple to be in the robot’s current location — determining the state of the apple as either present or absent in the robot’s location:

Finally, DynamicEval computes and returns the value based on the API specification and the updated simulation environment. Fig. 3 illustrates how RoboSim leverages DynamicEval to dynamically synthesize the simulation environment and validate the program. In RoboSim, the simulation starts with only the robot at its initial position, and entities are added as the program runs. At execution timestep 2a, after the robot picks up an apple, it becomes unclear whether another apple remains at the location. Therefore, DynamicEval sets the apple’s state to ”Undefined” and removes it in subsequent executions. This method is also related to STRIPS planning, as we demonstrate the connection in Appendix A.2.

Figure: Figure 3: Illustration of RoboSim executing a task program and dynamically updating the simulation environment. Initially, the environment only contains the robot’s starting location. As the program runs, two possible execution paths emerge (purple and blue). The environment is updated by DynamicEval at each step, reflecting the robot’s actions, object states, and location changes along both paths.
Refer to caption: x2.png

In the example program shown in Fig. 3, it’s clear to humans that the program’s logic is flawed because it attempts to pick up the apple, which is not present in the room. But *how would the simulator identify this as a failing state?* RoboSim solves this issue by simulating all possible states of the discovered entities and verifying that none lead to erroneous program execution. In this case, the “apple” can either be present in the room or not. If the apple is not present, executing pick(”apple”) will result in an error. Checking all possible states requires exploring an exponentially growing number of combinations based on the entities discovered. To manage this, RoboSim employs a bounded compute budget to randomly sample from this exponential space, as detailed in the overall RoboSim algorithm in Alg. 2.

Figure: Algorithm 2 $\textsc{RoboSim}(\mathcal{P})$

### 2.2 InstAlign : LLM-aided Instruction-Program Alignment Procedure

Figure: Figure 4: Overview of InstAlign.
Refer to caption: extracted/5904398/figures/instalign_iclr.png

Even after RoboSim verifies that a program doesn’t violate domain-specific constraints, it may still have subtle inconsistencies with the instructions — such as omitting a step implied by the instruction. The proposed rejection-sampling strategy alone cannot resolve this issue, and correcting verified programs to fully reflect the instructions while ensuring they remain valid is a challenging task.

We note that LLMs nowadays have demonstrated impressive code *understanding* capabilities (Rozière et al., ; Nam et al., ; Leinonen et al., ; Li et al., ; Lekshmi-Narayanan et al., ). Instead of correcting the program, it may be more effective for the LLM to revise its generated instructions to better align with the program’s intent.

We propose InstAlign, a procedure that prompts an LLM to revise generated instructions to better match the intent of the program, as shown in Fig. 4. InstAlign follows two steps: first, it uses Chain-of-Thought reasoning (CoT) (Wei et al., ) to generate a revised instruction based on the instruction-program pair; then, it prompts the LLM to compare the original and revised instructions, selecting the one that best aligns with the program’s intent. Detailed prompts are provided in Appendix A.4.3.

## 3 Analysis and Experiments

In this section, we investigate the following two research questions:

- 1.
Is Robo-Instruct effective at generating training data to fine-tune a small language model to generate domain-specific programs for robots?
- 2.
How do RoboSim and InstAlign impact the effectiveness of Robo-Instruct?

We conduct our investigation by fine-tuning the CodeLlama-Python-7B model (Rozière et al., ) on the synthetic dataset generated by Robo-Instruct and evaluate the fine-tuned model using RoboEval (Hu et al., ), a domain-specific code generation benchmark for service mobile robots. In the following subsections, we first provide a brief description of RoboEval. Then we present our experimental results addressing the two main research questions. Finally, we offer more analysis of RoboSim, InstAlign, and the synthetic dataset.

### 3.1 RoboEval : A Domain-Specific Robot Code Generation Benchmark

Figure: Figure 5: RoboEval APIs and benchmark task example.
Refer to caption: x3.png

RoboEval is a domain-specific code generation benchmark, featuring a suite of 16 tasks designed to evaluate the ability of LLMs to understand custom APIs and generate programs for service robots.
In this domain, a service robot can perceive objects, navigate to various locations, manipulate items, and communicate with humans. Furthermore, the robot should be capable of basic commonsense reasoning and executing complex tasks that involve conditional and repetitive actions.
To facilitate these capabilities, RoboEval defines a set of 8 API functions in Python as skill primitives.
Fig. 5 illustrates these function signatures and definitions, alongside an example task instruction and its canonical solution from the benchmark.
In addition, unlike other popular code generation benchmark tasks (Chen et al., ; Austin et al., ; Li et al., ; Liu et al., ; Lai et al., ; Hendrycks et al., ), *the order of the robot’s actions is crucial for successfully completing the specified tasks*. For instance, in the task *”bring me a marker from the classroom that does not have a whiteboard,”* the robot must check each classroom until it finds one without a whiteboard, whereas simply bringing back a marker is insufficient. Hence, RoboEval evaluates the generated program by executing it in a simulator to capture the action traces, which are subsequently validated for sequence correctness using temporal logic.

### 3.2 RQ1: Is Robo-Instruct Effective at Generating Training Data to Fine-Tune a Small Language Model for Generating Domain-Specific Robot Programs?

**Table 1: Pass@1 results of different LLMs on RoboEval computed with greedy decoding $T=0$ and nucleus sampling $T=0.2$.**
|  |  |  | RoboEval pass@1 |  |  |
| --- | --- | --- | --- | --- | --- |
| Fine-tune | Model | # Param | $T=0$ | $T=0.2$ | Licensing |
| - | GPT-4 | - | 83.75% | 85.81% | Proprietary |
| - | GPT-3.5-Turbo | - | 67.5% | 65.56% | Proprietary |
| - | Gemini-Pro | - | 60.00% | 59.88% | Proprietary |
| - | Codellama-Python | 7B | 40.00% | 39.31% | Open |
| - | Codellama-Python | 34B | 46.25% | 48.25% | Open |
| - | Starcoder2 | 15B | 62.5% | 60.94% | Open |
| - | Deepseek-Coder | 33B | 53.75% | 52.13% | Open |
| - | Llama3-Inst | 8B | 48.75% | 48.38% | Open |
| Self-Instruct | CodeLlama-Python | 7B | 55.00% | 52.69% | Open |
| Robo-Instruct (ours) | CodeLlama-Python | 7B | 68.75% | 66.00% | Open |

Experiment Setup.
We use the open-weight LLM, Llama3-8B-Inst, for Robo-Instruct. To generate a diverse dataset, we employ nucleus sampling for creating instruction-program pairs, setting the temperature $T=1$ and top $p=0.95$. The maximum resampling limit is capped at $3$ to accommodate instructions that initially produce invalid programs. For the LLM used in InstAlign, we empirically adjust the generation temperature to $T=0.3$ to optimize performance. Furthermore, we assess the edit similarity between token sequences of each instruction pair in the dataset (Lee et al., ), removing duplicates where the similarity score exceeds 0.6. We use the same setup to generate data via Self-Instruct. Instead of discarding invalid programs, Self-Instruct includes every generated instruction-program pair in the training dataset.
Finally, we create two datasets with 5K instruction-program pairs each using Self-Instruct and Robo-Instruct respectively.
These datasets are then used to fine-tune the CodeLlama-Python-7B model. The learning rate is set to be $3e\text{-}5$ with a warmup ratio of $3\%$ and a constant lr scheduler. We employ the AdamW optimizer (Loshchilov and Hutter, ) with an effective batch size of 8, training each model for 5 epochs using a sequence length of 2048 tokens. We train all our models on a single H-100 GPU using unsloth (Unslothai, ).

Baselines. We divide our baseline models into 2 categories: 1) proprietary LLMs, including GPT4 (OpenAI et al., ), GPT3.5-Turbo (OpenAI, ), Gemino-Pro (Team et al., ), and 2) open-weight LLMs, including Codellama-Python-7B (Rozière et al., ), Codellama-Python-34B, Starcoder2-33B (Lozhkov et al., ), Deepseek-Coder-33B (Guo et al., ), and Llama3-8B-Inst (AI, ).

Tab. 1 presents the average pass@1 results for different LLMs on RoboEval, using two settings:
greedy decoding at temperature $T=0$ and nucleus sampling at temperature $T=0.2$.
The results show that Robo-Instruct-fine-tuned CodeLlama significantly improves upon the base CodeLlama-Python-7B, and outperforms the Self-Instruct-fine-tuned variant (Appendix A.3.3 shows that improvements over Self-Instruct are not the result of distributional biases in the selection process). Notably, Robo-Instruct surpasses all open-weight models, including larger ones like CodeLlama-Python-34B and Deepseek-Coder-33B.
Additionally, although the training dataset was generated using Llama3-8B-Inst, which scores less than 50% pass@1 on RoboEval from Tab. 1, our Robo-Instruct-fine-tuned model still achieves a significant improvement, scoring 68.75%.
Finally, compared to proprietary models, while our Robo-Instruct-fine-tuned model trails the more powerful GPT-4, it outperforms GPT-3.5-Turbo and Gemini-Pro in generating programs for service mobile robots. This result demonstrates the effectiveness of our approach in generating domain-specific robot program data for fine-tuning a much smaller language model. It suggests that the fine-tuned model could potentially replace some proprietary models, providing a more cost-effective and private option for local deployment.

### 3.3 RQ2: How Do RoboSim and InstAlign Impact the Effectiveness of Robo-Instruct ?

**Table 2: Pass@1 results of different methods on RoboEval computed with greedy decoding $T=0$ and nucleus sampling $T=0.2$. The Invalid Programs column indicates the percentage of programs that result in execution errors when tested on RoboEval tasks.**
|  | T=0 | T=0.2 | Invalid |  |  |
| --- | --- | --- | --- | --- | --- |
| Method | pass@1 | Improv. | pass@1 | Improv. | Programs |
| Codellama-7B-Python | 40.00% | +0% | 39.31% | +0% | 38.31% |
| Self-Instruct | 55.00% | +15.00% | 52.69% | +13.38% | 20.94% |
| $+\text{Reject Unsolvable}$ (RU) | 60.00% | +20.00% | 57.62% | +18.31% | 23.38% |
| $+\textsc{RoboSim}+\text{RU}$ | 63.75% | +23.75% | 63.88% | +24.57% | 14.13% |
| $+\textsc{InstAlign}+\text{RU}$ | 58.75% | +18.75% | 59.81% | +20.50% | 23.44% |
| $+\text{Both}$ (Robo-Instruct) | 68.75% | +28.75% | 66.00% | +26.69% | 17.07% |

Using the same setup as in the previous section, we investigate the effectiveness of RoboSim and InstAlign. Since Self-Instruct may generate instructions for which no corresponding valid program can pass in RoboSim, we include Reject Unsolvable (RU) as an additional baseline. Self-Instruct+RU discards instructions for which no valid programs were found to successfully execute in RoboSim, and preserves instructions for which at least one passing program was found.
Tab. 2 shows the average pass@1 results from CodeLlama-7B-Python fine-tuned on different datasets generated by each method. First, results from $\textsc{Self-Instruct}{}+\text{RU}$ indicate that simply discarding invalid instructions improves model performance. Additionally,
fine-tuning with a dataset created from $\textsc{Self-Instruct}{}+\textsc{RoboSim}$ results in the smallest proportion of invalid program errors.
Finally, incorporating either RoboSim or InstAlign individually offers improvements over the baseline $\textsc{Self-Instruct}{}+\text{RU}$ results, incorporating both in Robo-Instruct results in the best pass@1 performance.
We refer the readers to Appendix A.1 for more results.

Figure: Figure 6: Self-Instruct-Generated Program Errors. Examples highlight errors that violate domain-specific constraints.(^4^44Programs have been adapted to succinctly demonstrate the types of errors and fit within the figure.)
Refer to caption: x4.png

### 3.4 Qualitative analysis of the generated program errors

We qualitatively analyze invalid programs identified by RoboSim, as shown in Fig. 4. The first three examples are easily recognizable to humans as flawed. However, the last example is more complex and involves an error when the robot can navigate to more than two rooms. After the robot places a toy in the living room, DynamicEval updates the environment to reflect that a toy is now in the room (line 8). However, when the robot returns to the living room later (line 6), it will not drop the item it’s holding (line 8). As a result, when the robot enters a third room (line 4) and tries to pick up another toy (line 5), an error will occur because the robot is only capable of carrying one item at a time. This example demonstrates that RoboSim can detect invalid programs beyond those easily identifiable through human inspection.

## 4 Real-World Deployment Results

We deployed the Robo-Instruct fine-tuned model to generate and execute mobile robot programs in the real world, as shown in Fig. 7. Compared to GPT-4 and GPT-3.5-turbo (Tab. 3), our model generates programs about 6x faster than GPT-4 and 3x faster than GPT-3.5-turbo, with similar output quality. In Appendix A.5 we showcase more results on long-horizon tasks beyond RoboEval.

Figure: Figure 7: Deployment of the Robo-Instruct fine-tuned model to generate programs based on user-provided instructions and execute them on the robot.
Refer to caption: x5.png

**Table 3: Inference speed of different models.**
| Models | GPT-4 | GPT-3.5-Turbo | Robo-Instruct (Ours) |
| --- | --- | --- | --- |
| Inference Speed | 19 tokens/s | 42 tokens/s | 114 tokens/s |

## 5 Related Work

### 5.1 LLMs for Robot Code Generation

LLMs have shown impressive capabilities in generating robot programs from natural language (Liang et al., ; Singh et al., ; Huang et al., ). One popular approach uses LLMs to generate composable costmaps for robots to plan their motion on. In this approach, Voxposer (Huang et al., ) focuses on the tabletop manipulation setting and NavCon (Biggie et al., ) focuses on creating composable maps for navigation.
Using LLM to create reward functions is also promising. Eureka (Ma et al., , ) and Language to Rewards for Robotic Skill Synthesis (Yu et al., ) both show that LLM can generate good reward functions that allows robots to acquire complex skills.
Finally, LLM can also be used to generate programs for high-level planning. LLM+p (Liu et al., ) outputs a robot plan in the form of the well-defined planning domain definition language (PDDL). Tidybot (Wu et al., ) uses an LLM to generate a rule that captures user preferences from examples and executes a program to sequentially complete the task in order. RoboEval (Hu et al., ) focuses on generating domain-specific programs for service mobile robots. It generates a program that allows the service robot to carry out long-horizon tasks and then validates the correctness of the program.

### 5.2 Generating Datasets For Fine-tuning LLMs

To enhance LLMs’ performance in code generation, numerous studies have explored the creation of specialized datasets (Muennighoff et al., ; Köpf et al., ; Muennighoff et al., ). Self-Instruct (Wang et al., ) is one popular method for generating synthetic datasets using an LLM. Following this methodology, Alpaca (Taori et al., ) generates 52K instruction-following demonstrations and subsequently fine-tunes the LLaMA 7B model (Touvron et al., ) to create Alpaca 7B, which can behave qualitatively similarly to OpenAI’s text-davinci-003. Code Alpaca (Chaudhary, ) extends this approach to generate code instructions using 21 seed tasks, while Gorilla-LM (Patil et al., ) adapts the method to focus on ML domain-specific APIs from Huggingface, TensorFlow Hub, and Torch Hub. To create more complex instructions, Evol-Instruct (Xu et al., ; Luo et al., ) proposes iteratively updating instructions to become more complex through different prompting strategies. In addition to Evol-Instruct, OSS-Instruct (Wei et al., ) uses open-source code snippets to generate 75K high-quality instruction data and fine-tunes the CodeLlama-Python-7B model to create Magicoder, which can match the performance of GPT-3.5-Turbo (OpenAI, ) on HumanEval (Chen et al., ). While these works focus on creating seed instruction sets to generate synthetic data to effectively fine-tune an LLM, our research investigates post-processing methods in addition to Self-Instruct. Specifically, we concentrate on generating domain-specific programs in robotics (Hu et al., ), where we can effectively leverage constraints to filter out erroneous programs.

## 6 Conclusion, Limitation and Future Works

In this work, we introduce Robo-Instruct, a novel framework to generate synthetic training data to fine-tune small language models for domain-specific robot programs.
Robo-Instruct comprises two novel components: 1) RoboSim, a method to synthesize consistent simulation environments dynamically for verifying the generated programs, and 2) InstAlign, an LLM-aided instruction alignment procedure to revise instructions to better align with the generated programs.
The experimental results show that the 7B CodeLlama model fine-tuned on the Robo-Instruct dataset significantly outperforms larger open-weight LLMs and proprietary models like GPT-3.5-Turbo and Gemini-Pro in generating service robot programs. However, a key limitation is that Robo-Instruct relies on Self-Instruct to filter invalid programs, which may introduce biases and affect dataset quality. Another limitation is the use of a simple rejection-sampling method to handle invalid programs, which may not fully address the underlying issues. Future work will focus on improving dataset quality by integrating Robo-Instruct with advanced methods like Evol-Inst and OSS-Inst.

## Appendix A Appendix

### A.1 Overview

In this appendix, we first outline the relationship between DynamicEval and the classic STRIPS planning formulation in subsection A.2, providing a new perspective on the proposed algorithm. In subsection A.3, we present additional ablation experiments to analyze the percentage of invalid programs generated by Self-Instruct and the effectiveness of the rejection-sampling strategy combined with RoboSim. We also explore how the generation temperature in InstAlign impacts final performance and compare the dataset diversity produced by Robo-Instruct and Self-Instruct. Subsection A.4 lists the seed tasks used in RoboEval and the CoT prompts. in subsection A.5, we report real-world experiments that empirically evaluate the performance of our fine-tuned model on two long-horizon tasks, which differ significantly from those in RoboEval, and assess the model’s latency in generating programs.
Although this work focuses on service mobile robots, the proposed framework is adaptable to other domains. In subsection A.6, we offer toy examples showing how DynamicEval can be extended to verify programs by incorporating domain-specific constraints.

### A.2 Relevance to STRIPS planning

The proposed DynamicEval shares significant similarities with the formulation of STRIPS planning. A STRIPS instance is typically represented as a tuple $\langle I,G,A,P\rangle$, where $I$ denotes the initial state of the simulation environment, $G$ represents the desired goal state that the robot aims to achieve, $A$ defines the set of actions available to transition between states, and $P$ is the set of preconditions that must be satisfied before performing actions. Thus, DynamicEval can be reformulated to align with the STRIPS formulation as shown in Alg. 3. Each API invocation corresponds to an action, and its precondition consists of a set of literals, representing specific combinations of entities, types, and states.

To address this, we extend the classic STRIPS formulation by incorporating dynamically discovered literals. Unlike the conventional STRIPS approach, where each literal is binary—True when defined and False when not—we introduce a third value, ”Undefined.” This means a literal must be explicitly defined as either True or False; otherwise, it remains in the Undefined state. When an action requires a literal that is undefined, a random value (True or False) is assigned to it, and the literal is added to the state of the simulation environment (line 7). Once the precondition is fully defined, the action is executed, and domain-specific constraints are checked for any violations (line 10). This extension enables DynamicEval to handle arbitrary programs effectively.

Figure: Algorithm 3 $\textsc{DynamicEval}_\text{STRIPS}(\text{api_fn},\text{params},\mathcal{W})$

### A.3 Ablation Exmperiments

Figure: Figure 8: Ablation Experiment Results
Refer to caption: extracted/5904398/figures/combined_plots.png

#### A.3.1 the Effectiveness of the Rejection-Sampling Strategy

We analyze the percentage of instruction-program pairs discarded by RoboSim at various maximum resampling limits, as shown in Fig. 8. Initially, with the maximum resampling limit set to 0, disabling the rejection-sampling method, approximately 51% of the programs generated by Self-Instruct contain errors. As the limit increases, fewer programs are discarded. However, there is a diminishing return; even with the maximum resampling limit set to 10, about 15% of the instructions still result in invalid programs.

#### A.3.2 Instruction Alignment model temperature

We further investigate how varying LLM temperatures for generating the revised instruction in InstAlign impact the performance of the fine-tuned model. Fig. 8 shows the bar chart of the pass@1 score of the models fine-tuned over datasets generated using different LLM temperatures in InstAlign. The model performs the best when fine-tuned on the dataset generated using LLM temperature $T=0.3$. As the temperature increases, we observe a decrease in performance.

#### A.3.3 Analysis of Generated Dataset

Figure: Figure 9: Cosine similarities between RoboEval and generated data.
Refer to caption: extracted/5904398/figures/RE_similarity_comparison.png

Similar to Magicoder [Wei et al., ], we show the improvements from Robo-Instruct are not merely due to selection bias, i.e., including data more aligned with the distribution of RoboEval tasks than Self-Instruct. We pair each sample from the generated dataset with task instructions and their canonical solutions, then compute cosine similarity using TF-IDF embeddings [Sparck Jones, ]. Fig. 10 shows comparable token similarities between both methods. Additionally, Fig. 10 presents the token length distribution, which also appears similar for both.

**Table 4: Dataset Statistics**
| Method | Size | Ngram=4 Score | # Synth. Loc. | # Synth. Obj. |
| --- | --- | --- | --- | --- |
| Self-Instruct | 5K | 0.581 | 956 | 1060 |
| Robo-Instruct | 5K | 0.587 | 1025 | 928 |

Since RoboSim does not rely on pre-defined simulation environments, we aim to assess the diversity of programs generated by Self-Instruct and whether Robo-Instruct can maintain this diversity. To do so, we measure the number of distinct entities, such as synthetic locations and objects. As shown in Tab. 4, with a dataset of only 5,000 samples, approximately 1,000 unique objects and locations are generated, highlighting that conventional robot simulations with pre-defined environments are insufficient. Additionally, Tab. 4 presents the n-gram diversity scores for each dataset, indicating that both distributions and dataset statistics are highly similar. This suggests that Robo-Instruct not only preserves but enhances the quality of generated data compared to Self-Instruct, rather than simply aligning the dataset with benchmark tasks.

### A.4 Prompts

#### A.4.1 RoboEval Seed Task Example

Seed Task Example 1:

Seed Task Example 2:

Seed Task Example 3:

Seed Task Example 4:

Seed Task Example 5:

Seed Task Example 6:

#### A.4.2 Prompts to Generate Synthetic Dataset Using Self-Instruct

#### A.4.3 CoT Prompts for InstAlign

### A.5 Real-world Experiments

#### A.5.1 Program Generation Results Beyond RoboEval

We conducted a small qualitative experiment to assess the performance of the base model, Self-Instruct fine-tuned models, Robo-Instruct fine-tuned models, and GPT-4 on long-horizon tasks. To showcase performance, we created two new long-horizon tasks that differ significantly from those in the RoboEval benchmark. Using a temperature setting of 0, we generated programs with each model and found that our Robo-Instruct fine-tuned model successfully produced correct programs, while both the base model and Self-Instruct fine-tuned model failed. Additionally, GPT-4 made an error on the second instruction. We show the generation results from Robo-Instruct fine-tuned models and GPT-4 below.

Task instruction 1:

Robo-Instruct fine-tuned model (Ours) [Success]

GPT-4 [Success]

Task instruction 2:

Robo-Instruct fine-tuned model (Ours) [Success]

GPT-4 [Failure]

#### A.5.2 Generation Latency

Figure: (a) Example Program 1 generated by our fine-tuned model.
Refer to caption: x6.png

Figure: (c) Another Example Program generated by our fine-tuned model.
Refer to caption: x8.png

### A.6 Toy Examples Beyond Service Mobile Robots

#### A.6.1 Robot with low-level controls

Consider a tabletop manipulation scenario with a potential API function, is_rotate(robot_gripper_ name, radians), where the robot’s gripper has a physical constraint, allowing rotation only within the range $[-\frac{\pi}{6},\frac{\pi}{6}]$ radians. For the following generated program snippet:

DynamicEval will first infer that ”left hand” is an entity of the robot gripper type. Then, regardless of the initial configuration of the gripper, DynamicEval will throw an error because the program causes the gripper to exceed its allowable range of motion.

#### A.6.2 AI-powered personal digital assistant

Consider a broader application than robotics: code generation for an AI-powered personal digital assistant. This AI assistant could handle scheduling events using an API function like schedule_on_calendar(event, start_time, duration). Given the instruction: *”My schedule is free tomorrow morning. Please create two 1-hour timeslots for office hours for my robotics and deep learning class.”* The assistant could generate a program to create these timeslots:

In this example, DynamicEval needs to reason about the entities “robotics class office hour” and “deep learning class office hour”, which are categorized as event types. The event type indicates that these entities have associated timeslots. The state of these entities is defined by the time they occur: robotics class office hour is set for 9:30-10:30 am, and deep learning class office hour is set for 10:00-11:00 am. During evaluation, DynamicEval can identify a time conflict between these two office hours and thus determine that the generated program is invalid.

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 6
presentation: 4
questions: Could you please fix the formatting of table 5 to remove the overlapping text?
soundness: 4
strengths: Clarity. The authors did a phenomenal job describing their method and experimental process with precision. The specifics of the robosim environments were easy to follow from the method section and the motivation for dynamic environment generation and its unique application to robotic service agents was well presented.

Quality. The approach is simple and sound and I believe there is sufficient information for researchers to reproduce the results. On the RoboEval benchmark their method produces a model that outperforms even proprietary language models.

Significance. The idea of using dynamic environments to evaluate code could have broad impact for robotic code generation. The author present a strong first demonstration of this.
summary: The goal of this paper is to
weaknesses: Presentation. Focusing solely on open models as being error-prone unnecessarily limits the scope and potential impact of the paper's contributions when it could be relevant to any base LLM.

Quality. There was a limited diversity of baselines. The domain specific language looks very similar to the code-as-policies test environments. In the experimental section, the authors should contextualize the results by either explaining the best analogy to code-as-policies that they run or by explicitly discussing a code-as-policies style baseline. The authors could also try simpler variants of their method: for example, taking some data points generated by robosim rejection sampling process and putting them in the prompt instead of doing model fine-tuning and alignment.

Significance. The APIs in the roboeval benchmark are very high-level and I wouldn't be surprised if there are many solid engineering approaches to getting a performant policy that works with high-level APIs. It's unclear how well roboinstruct will scale with more complex APIs and tasks, which limits the significance of the work.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: (1) How does ROBO-INSTRUCT handle edge cases where simulator-based validation cannot capture subtler domain inconsistencies?

(2) relying on data augmentation techniques such as self-instruct or evol-instruct may introduce bias or even hurt the generalization of LLMs, it would be nice to see related evaluation on 

(3) the paper verifies the program correctness, is there any other filtering like ROGUE-L methods as used in self-instruct?
soundness: 3
strengths: Clarity: The framework's purpose, components, and experimental results are presented clearly, though some complex aspects could benefit from additional clarification (e.g., the alignment between ROBOSIM and traditional STRIPS planning). The experiment design is well-articulated, showing comparisons across multiple baselines and careful control of variables.

Novelty: The integration of ROBOSIM for dynamic simulation and the INSTALIGN for instruction alignment introduce a novel approach to overcoming LLM limitations in handling domain-specific instructions. The work holds promise for cost-effective deployment of LLMs in real-world robot applications, especially where open-weight models are prioritized for privacy and customizability.
summary: The paper introduces ROBO-INSTRUCT, a framework designed to improve open-weight Large Language Models (LLMs) for generating domain-specific training data. This framework aims to enhance service robot applications, focusing on Code LLMs that use domain-specific APIs to generate executable robot instructions. The ROBO-INSTRUCT framework comprises two main components: ROBOSIM: A task-agnostic simulator to synthesize simulation environments for verifying program correctness dynamically. INSTALIGN: An alignment tool that adjusts program instructions to reflect the true intent of generated programs, using a large language model (LLM) for instruction revision. Experiments show that models fine-tuned with ROBO-INSTRUCT outperform base models and models fine-tuned with SELF-INSTRUCT, improving pass@1 scores and real-world deployment latency.
weaknesses: (1) The data augmentation approach seems somewhat incremental, given its widespread use in Evol-Instruct, WizardLM, and similar frameworks. It would be valuable to explore more unique challenges and solutions tailored to robotics, which often requires handling more complex tasks. Additionally, an evaluation on scaling performance regarding parameter count, generalization, and related metrics would strengthen the analysis.

(2) Another concern is that the evaluated tasks in the paper appear overly simplified. While I understand that many current studies also rely on simplified environments like VirtualHome, the solution's handling of out-of-distribution scenarios remains insufficiently understood. This is a crucial factor for robotics applications, where the risk of overfitting due to augmentation is particularly high.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. I'm a bit confused about ROBOSIM. It is claimed that ROBOSIM is task-agnostic but it seems that ROBOSIM still requires a lot of expert knowledge for the corresponding task. For example, ROBOSIM cannot work on the "apple" tasks if the "apple" or "kitchen" related properties (i.e., missing any of the entities, type, or state) are absent. Or, to put it another way, even if "apple" and "kitchen" are present and ROBOSIM can complete tasks related to "apple" and "kitchen," it still won't be able to complete tasks related to "apple" and "living room" due to the absence of the "living room."
2. As the above question said, how much expert effort is needed to construct ROBOSIM?
3. A typo in line 375 "Gemino-Pro".
4. In Table 2, why does ROBO-INSTRUCT have a higher invalid program rate than ROBOSIM + RU?
soundness: 2
strengths: 1. Applying code to embodied AI is an important direction, so exploring how to enhance the code generation capabilities of LLMs in the robotics domain is also meaningful.
2. The idea of guiding the synthesized data with verification of the ROBOSIM environment is reasonable.
3. The experimental result looks promising.
summary: The paper introduces ROBO-INSTRUCT, a framework designed to enhance the generation of training data for fine-tuning Code LLMs in domain-specific service robot applications. It consists of ROBOSIM, a task-agnostic simulator that dynamically creates consistent simulation environments to verify program correctness, and INSTALIGN, an LLM-aided instruction-program alignment process. The framework significantly improves the performance of a fine-tuned model, achieving a 28.75% improvement in pass@1 over the base model and surpassing several proprietary models. Additionally, ROBO-INSTRUCT demonstrates faster inference speeds, making it suitable for real-world robot deployments.
weaknesses: I'd be happy to raise my score if my concerns are addressed.
1. There are many instruction tuning methods for code LLMs, but this paper only compares with SELF-INSTRUCT. Could the authors compare with methods like evol-instruct in Wizardcoder as well?
2. The choice of settings is somewhat confusing. For example, why use the Llama3 model to generate the training set and then fine-tune the 7B CodeLlama instead of Llama3 itself? Why not use more powerful closed-source models like GPT-3.5 or GPT-4-Turbo for synthesizing the dataset? I think the authors should either provide corresponding results or at least explain the reason for doing so.
3. I think the author should have decontaminated the test set, but it seems the author did not mention any relevant details in the experimental setup.
4. The real-world deployment results are great but the authors only measured the inference speed. Could the authors measure some accuracy or success-related metrics?

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: - How different are the generated programs from the tasks in RoboEval? Is it a generalized performance or performance on the training set?
- Why use Llama3-8B-Instruct as the generating LLM and CodeLlama as the fine-tuned LLM? If you change them to other LLMs, will the framework still work?
soundness: 1
strengths: - Having a simulation (not a physics engine in this paper) to improve the diversity of the generated programs intuitively increases the performance of an LLM for robotic applications, the paper is therefore addressing problems of importance.
summary: This paper proposes a framework (Robo-Instruct) to generate training data to fine-tune a code LLM for domain-specific service robot applications. Robo-Instruct contains 2 components: (1) RoboSim that dynamically synthesizes consistent simulation environments for each generated program, and (2) InstAlign that handles the instruction-program inconsistencies. In the experiments, the authors use Llama3-8B-Instruct as the LLM to generate training data, and fine-tuned CodeLlama to perform on the RoboEval benchmark.

I think the authors are tackling an important problem that would allow LLMs to be better applied to robotics. However, I find the paper hard to follow, with insufficient experiments to support the potentially over-claimed contributions. Please see my explanation below.
weaknesses: - The contributions of the paper are limited. Of the 5 contributions listed in the introduction, I find 4 of them questionable. (1) RoboSim is said to produce diversity, yet on line 245 the authors explain that checking all possible states is not possible and they resort to random sampling with limited compute budget. This does not necessarily guarantee diversity. (2) InstAlign is just CoT. (3) The authors claim the fine-tuned model is better, but it is not clear how different are the data it has been trained on from the tasks it is tested in. (4) The authors claim the fine-tuned model is faster in inference than proprietary models. This is especially unfair and misleading. Any onboard model with sufficient hardware support is faster than remote API calls. 
- Insufficient experiments. How different are the generated programs from the tasks in RoboEval? Is it a generalized performance or performance on the training set? Why use Llama3-8B-Instruct as the generating LLM and CodeLlama as the fine-tuned LLM? If you change them to other LLMs, will the framework still work?
- Presentation can be significantly improved. The paper is full of details that do not help a reader follow the main thread of the paper. None of the links (e.g., citations, sections, figures, etc) works, and I have to manually search for those entities. Inefficient use of space (e.g., are the 2 python programs on lines 188 and 202 necessary?).

Meta-review: This paper proposes a framework to fine-tune a code LLM for robotics tasks. The paper is well written and motivated. However, at the same time the reviewers have raised several concerns, particularly on the novelty of ideas in the paper. The evaluated tasks and the real world deployments seem to be simplistic and hence are quite limited in its current form from having significant impact in the ML community.

Decision: Reject