Title: Robo-Instruct: Simulator-Augmented Instruction Alignment For Finetuning CodeLLMs
Venue: NeurIPS 2024
Accepted: no
Keywords: Self-Instruct

Code LLMs Fine-tuning

Domain-Specific Program Generation

Code LLMs for Robotics
Abstract: Large language models (LLMs) have shown great promise at generating robot programs from natural language given domain-specific robot application programming interfaces (APIs). However, the performance gap between proprietary LLMs and smaller open-weight LLMs remains wide. This raises a question: Can we fine-tune smaller open-weight LLMs for generating domain-specific robot programs to close the performance gap with proprietary LLMs? While Self-Instruct is a promising solution by generating a diverse set of training data, it cannot verify the correctness of these programs. In contrast, a robot simulator with a well-defined world can identify execution errors but limits the diversity of programs that it can verify. In this work, we introduce Robo-Instruct, which brings the best of both worlds --- it promotes the diversity of Self-Instruct while providing the correctness of simulator-based checking. Robo-Instruct introduces RoboSim to synthesize a consistent world state on the fly by inferring properties relevant to the program being checked, and simulating actions accordingly. Furthermore, the instructions and programs generated by Self-Instruct may be subtly inconsistent --- such as the program missing a step implied by the instruction. Robo-Instruct further addresses this with InstAlign, an instruction-program alignment procedure that revises the task instruction to reflect the actual results of the generated program. Given a few seed task descriptions and the robot APIs, Robo-Instruct is capable of generating a training dataset using only a small open-weight model. This dataset can then be used to fine-tune small open-weight language models, enabling them to match or even exceed the performance of several proprietary LLMs, such as GPT-3.5-Turbo and Gemini-Pro.

Abstract

Large language models (LLMs) have shown great promise at generating robot programs from natural language given domain-specific robot application programming interfaces (APIs). However, the performance gap between proprietary LLMs and smaller open-weight LLMs remains wide. This raises a question: Can we finetune smaller open-weight LLMs for generating domain-specific robot programs to close the performance gap with proprietary LLMs? While SELF-INSTRUCT is a promising solution by generating a diverse set of training data, it cannot verify the correctness of these programs. In contrast, a robot simulator with a well-defined world can identify execution errors but limits the diversity of programs that it can verify. In this work, we introduce ROBO-INSTRUCT, which brings the best of both worlds - it promotes the diversity of SELF-INSTRUCT, while providing correctness of simulator-based checking. ROBO-INSTRUCT introduces ROBOSIM to synthesize a consistent world state on the fly by inferring properties relevant to the program being checked, and simulating actions accordingly. Furthermore, the instructions and programs generated by SELF-INSTRUCT may be subtly inconsistent -such as the program missing a step implied by the instruction. ROBO-INSTRUCT further addresses this with INSTALIGN, an instruction-program alignment procedure that revises the task instruction to reflect actual results of the generated program. Given a few seed task descriptions and the robot APIs, ROBO-INSTRUCT is capable of generating a training dataset using only a small open-weight model. This dataset is then be used to fine-tune small open-weight language models, enabling them to even exceed the performance of several proprietary LLMs including

GPT-3.5-Turbo and Gemini-Pro.

1 Introduction

Large language models (LLMs) have demonstrated great promise at generating robot programs from 25

natural language instructions [3, 10-12, 17, 18, 31, 39]. For example, consider an instruction for

26 a service mobile robot: "Check how many conference rooms have no markers." The robot may 27 be equipped with a domain-specific robot application programming interface (API) that includes 28 skills such as go_to(location) for navigation and is_in_room(object) for perception. Since 29 such domain-specific APIs do not exist in the training dataset of general-purpose LLMs, in-context 30 learning (ICL) via few-shot examples is often employed to describe and use such APIs for performing 31 few-shot inference. However, there is a significant performance gap [10] in the correctness of 32 programs generated by ICL for large proprietary models and smaller open-weight models that can be 33 deployed locally on robots. This raises a question: can we fine-tune small open-weight LLMs for 34 generating domain-specific robot programs to close the performance gap with proprietary LLMs? 35

1.

Instruction

World state

2.

Flow chart

Fail, Resample up to m times

Instruction-Program Pair

Check if there is a key here. If

so, pick it up.

1 def task_program():

2  if

is_in_room

("key"):

3

pick

("key")

Gen.

Program

InstAlign

Program

1 def task_program():

2  if not

is_in_room

("key"):

3

pick

("key")

RobotPickError: 'key' does not

exist in the current location

Instruction

Ask if there is a key here. If so, pick

it up.

Misalignment: The robot doesn't

verbally ask

but

visually check

if a key is present.

Input Prompt

Aligned

Instruction

Instruction

Program

Examples

Success

Fail

Self-Instruct

Gen.

Instruction

Robo-Instruct

Input Prompt

Here is a robot that has the following

skills:

{RoboEval APIs}

{Seed Examples}

Generate an interesting robot task that

can be accomplished using above skills.

Pass@1

RoboSim

Robot World State Init

Program Translation

Exec Program

1

2

3

Query & Update

Return Values w.r.t to

the Current State

Robot APIs

×k

+

→

→

Valid

Program

❓

❓

<1>

</>

<1>

</>

85.81%

80

Robo-Instruct-CL-7B(Ours)

60

68.75%

67.5%

62.5%

60.0%

55.0%

53.75%

OtherModels

Self-Instruct-CL-7B

48.75%

48.25%

40.0%

40

20

0

RI-CL-7B(Ours)

GPT-3.5-Turbo

Starcoder2

Gemini-Pro

SI-CL-7B

DSC

Llama3-8B-Inst

CL-7B-Python

CL-34B-Python

Since training datasets of the domain-specific robot programs are often unavailable, SELF-INSTRUCT 36 might seem like a promising solution [29, 36]. Consider the setting of generating programs for 37 service mobile robots that can perceive objects, navigate to various locations, manipulate items, and 38 communicate with humans. By formulating these robot skills into APIs, we can create a few seed 39 task examples demonstrating their use case and employ SELF-INSTRUCT to generate a diverse set of 40 instruction-program pairs as training data, as illustrated in Fig. 1. However, using SELF-INSTRUCT 41 naïvely may generate infeasible instructions-e.g., asking the robot to pick up multiple objects at once 42 when it cannot due to physical constraints. They can also violate domain-specific constraints. For 43 example, in Fig. 1, after line 2 confirms the absence of a key at the current location, line 3 erroneously 44 attempts to pick up a key. Further, these instructions may not align with the generated programs, even 45 if these programs are valid. For example, Fig. 1 shows an example instruction directing the robot to 46 verbally ask in each room if a key exists, whereas the program instructs the robot to visually check 47 in each room. Finally, the generated programs may have execution errors. These challenges may 48 appear to be solvable using a simulator, but a simulator needs an initial world state to check against 49 programs. A simulator using a hand-curated world state will end up rejecting the wide diversity of 50 programs generated by SELF-INSTRUCT, even if they are executable, just because the world state did 51 not capture some aspect relevant to them ( e.g., the presence of a 'key'). 52

This work introduces ROBO-INSTRUCT, a new framework based on SELF-INSTRUCT, to address these 53 issues and improve the performance of small open-weight language models for generating domain54 specific robot programs. As shown in Fig. 1, ROBO-INSTRUCT introduces two novel components: 55 (1) ROBOSIM, a task-agnostic simulator that encodes domain-specific constraints and validates 56 robot programs generated from SELF-INSTRUCT. Critically, ROBOSIM dynamically synthesizes 57 a consistent world state starting from arbitrary programs. (2) INSTALIGN, an instruction-program 58 alignment procedure that revises the generated instructions to better reflect the intent of the generated 59 programs. ROBO-INSTRUCT also employs a rejection-sampling mechanism that rejects invalid 60 programs detected by ROBOSIM and queries SELF-INSTRUCT for a new program corresponding to 61 the same generated instruction. 62

Wevalidate ROBO-INSTRUCT by fine-tuning Codellama-Python-7B [30] and evaluate on ROBOEVAL, 63 a domain-specific code generation benchmark for service mobile robots. We show that ROBO64 INSTRUCT is capable of improving the performance of the Codellama model by using only a small 65 open-weight model to generate the training dataset. Compared to the base Codellama-Python66 7B model without fine-tuning, our ROBO-INSTRUCT fine-tuned models outperform by 28 . 75% in 67 average pass@1 scores; and, compared to SELF-INSTRUCT fine-tuned model, our model outperform 68 by 13 . 75% .; and the best pass@1 of ROBO-INSTRUCT fine-tuned model achieves a 68.75% match, 69 surpassing the performance of the proprietary GPT-3.5-Turbo and Gemini-1.0-Pro. 70

1 def task_program():

2  if not is_in_room

("key

3

pick

("key")

Robo-Ins

Su cc

ess

Fai lur

e

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

Contributions Our main contributions are as follows:

We introduce ROBO-INSTRUCT, a new framework for improving the code generation performance of small open-weight language models for domain-specific robot programs. This framework introduces two novel components, ROBOSIM and INSTALIGN.

We introduce a dynamic world synthesis and evaluation process for generating relevant world states for automated code checking for diverse, arbitrary tasks in ROBOSIM.

We introduce INSTALIGN, an instruction alignment procedure to refine instruction-code pairs to improve alignment between instructions and code generated by SELF-INSTRUCT.

We fine-tune a small open-weight model, Codellama-Python-7B [30], using ROBOINSTRUCT, and improve its performance to outperform several CodeLLMs, including Deepseek-Coder-33B [8], and Starcoder2-15B [21] and two proprietary LLMs, GPT-3.5Turbo [27] and Gemini-1.0-Pro [33] on the ROBOEVAL benchmark.

Our code and data will be released at URL anonymized .

2 ROBO-INSTRUCT

In this section, we present how ROBO-INSTRUCT generates training datasets of domain-specific robot programs. Alg. 1 shows a broad overview of the framework. To add an entry in the training dataset, SELF-INSTRUCT first generates an instruction-program pair, ( I , P ) , from the robot APIs and seed tasks, shown in Appendix A.4. Then, ROBOSIM dynamically synthesizes a consistent world state on the fly as it executes and validates P . If P is invalid, ROBO-INSTRUCT employs a rejection-sampling method, which generates a new program P given the same I and evaluates the new P again. This process repeats until P becomes valid or a predefined maximum resampling limit is reached. If the limit is reached, the instruction might be invalid given the domain-specific APIs or too complex to generate a program, so the instruction-program pair is discarded. Finally, if P is valid, INSTALIGN takes in ( I , P ) to revise I to better reflect the intent of P and the aligned instruction and program is saved to the training dataset. In the following subsections, we elaborate on the specific design of each component.

Algorithm 1 ROBO-INSTRUCT: Instruction-Program Generation

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

2.1 ROBOSIM: A Task-Agnostic Simulator For Domain-Specific Programs

We present a principled approach to design ROBOSIM for validating domain-specific robot programs. Alg. 2 illustrates the high-level algorithm used to assess the correctness of a robot program. ROBOSIM employs the concept of world state to simulate the robot actions directed by a program, ensuring consistent and reliable evaluation. A world state is a symbolic representation of the environment in which the robot operates, and it keeps track of the high-level changes in the robot state and the surrounding environment as the robot performs actions in order. For example, consider a program instruction that commands a robot to check if an apple is nearby. The world state queries the stored information about the surrounding environment, identifies all objects at the robot's current location, and informs the program whether an apple is present.

However, since SELF-INSTRUCT generates arbitrary programs based on the provided APIs, ROBOSIM does not know what a plausible world state relevant to the program would be a priori e.g., reasoning about the existence of an apple in the example program. Thus, we equip ROBOSIM with the ability to expand the world state as more robot actions are performed. Our approach is inspired by angelic execution [4], which has previously been used for software verification of programs with partially defined library functions. In our case, instead of partially defined library functions, we have unknown plausible world states. ROBOSIM dynamically synthesizes and grows a world state based on domainspecific constraints ( e.g., object permanence, robot skills, etc. ) and the execution trace of the program, which allows it to infer a consistent and relevant world state.

Specifically, ROBOSIM modifies the program to replace all API calls with the DYNAMICEVAL function (Alg. 2 line 4) - when an API function is called during execution, the DYNAMICEVAL function is invoked instead.

DYNAMICEVAL makes an important extension to the formulation of STRIPS [7] to integrate with API functions. DYNAMICEVAL equips each API function with specific pre-conditions, effects, and return values. The pre-conditions are composed of literals tailored to the function's requirements. For instance, the API function is_in_room('apple') , which determines if an object 'apple' is in the same room as the robot, uses two literals for its pre-condition: robot_at(X) and obj_at(X, 'apple') . Generally, STRIPS assigns one of two possible values to each literal: True if the literal is defined, otherwise False . However, prior to program execution, DYNAMICEVAL is unaware of the program-relevant literals. Thus we assign a third value, undefined , to such unknown literals. Literals must thus be explicitly defined as either True or False , or they remain undefined if not specified.

Alg. 3 demonstrates how DYNAMICEVAL executes an API function and updates the world state. First, it calculates the precondition specified for the function. It then checks each literal in the precondition to see if it is defined. If a literal is undefined, DYNAMICEVAL invokes GROWWORLD, a stochastic function that assigns a random truth value to the literal and updates the world state accordingly. Finally, DYNAMICEVAL proceeds to execute the API function using the current world state, retrieves the return values, and applies the function's effects to update the world state.

Fig. 2 illustrates an example of ROBOSIM executing a generated program. Initially, ROBOSIM's world state only specifies the robot's current location, and whether a pie is in the same room as the robot remains undefined (line 2). Therefore, DYNAMICEVAL invokes GROWWORLD to

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

Algorithm 3 DYNAMICEVAL ( api_fn , params , W )

World State robot_at(start_loc): True object_at(start_loc, key): True robot_holding(key): True pick("key") randomly determine a truth value for the obj_at(start_loc, "pie") literal, leading to two distinct execution paths depicted in light purple and blue. Subsequently, as additional API functions are called, more literals are introduced or updated in the world state to ensure consistent evaluations.

is_navigatable('kitchen')

Finally, due to the stochastic nature of DYNAMICEVAL, ROBOSIM must execute the generated program multiple times to validate the program. If all executions are successful, the program is deemed correct (Alg. 2 line 5-11).

2.2 INSTALIGN: Instruction-Program Alignment Procedure

Given that LLMs are extensively trained in code understanding [30], INSTALIGN is a procedure that prompts an LLM to revise I to better reflect the intent of P . This procedure involves two steps: first, given I and P , INSTALIGN leverages Chain-of-Thought reasoning [37] (CoT) to prompt an LLM to generate a revised instruction, I revised; second, INSTALIGN invokes the LLM again to determine whether I or I revised is more aligned with P 's intent and output the chosen instruction as I aligned .

To generate I revised, the prompt to the LLM comprises the robot API function definitions, I , P , and CoT instructions. The CoT asks the LLM to perform the following three steps in order: 1. write down all the robot APIs used in the program; 2. examine these APIs and write down step by step what the program does; 3. combine all the information above to revise the robot instruction. Similarly, to determine I aligned, an LLM is prompted to think step by step about P , I and I revised to arrive at a conclusion. Detailed prompt is shown in Appendix A.6.

3 Analysis and Experiments

In this section, we investigate the following two research questions: 156

False

True

False

True

True

True

157

158

159

160

161

162

163

164

165

Is ROBO-INSTRUCT effective at generating training data to fine-tune a small language model for generating domain-specific robot programs?

How do ROBOSIM and InstAlign impact the effectiveness of ROBO-INSTRUCT?

We conduct our investigation by fine-tuning the Codellama-Python-7B model [30] on the synthetic dataset generated by ROBO-INSTRUCT and evaluate the fine-tuned model using ROBOEVAL [10], a domain-specific code generation benchmark for service mobile robots. In the following subsections, we first provide a brief description of ROBOEVAL. Then we present our experimental results addressing the two main research questions. Finally, we offer more analysis of ROBOSIM, INSTALIGN, and the synthetic dataset.

3.1 ROBOEVAL: A Domain-Specific Robot Code Generation Benchmark 166

Task Instruction

Go to the elevator. Wait until someone shows up and ask them if they are here for the tour. If

yes, welcome them to the university, tell them to follow you, and take them to the main

conference room. If not, wait for the next person. When you get to the conference room, say

you have arrived at the conference room and also say enjoy your visit here!

Canonical Solution

1  def

task_program

():

2

go_to

("elevator")

3   while True:

4     if

is_in_room

("person"):

5       response =

ask

("", "Are you here for the conference?", ["Yes", "No"])

6       if response == "Yes":

7

say

("Welcome to the university. Please follow me.")

8         break

9     time.sleep(1)

10

go_to

("conference room')

11

say

("We have arrived. Enjoy your time here")

# Get the current location of the robot.

def

get_current_location

() -> str

# Get a list of all rooms.

def

get_all_rooms

() -> list[str]

# Check if an object is in the current room.

def

is_in_room

(

object

: str) -> bool

# Go to a specific named location.

def

go_to

(

location

: str) -> None

# Ask a person a question, and offer a set of specific

options for the person to respond. Returns the response

selected by the person.

def

ask

(

person

: str,

question

: str,

options

: list[str]) -> str

# Say the message out loud.

def

say

(

message

: str) -> None

# Pick up an object if you are not already holding one. You

can only hold one object at a time.

def

pick

(

obj

: str) -> None

# Place an object down if you are holding one.

def

place

(

obj

: str) -> None

16 RoboEval Benchmark Tasks

RoboEval Domain-Specific API Definitions

FindBackpack

ElevatorTour

…

GetDrink

WeatherPoll

ROBOEVAL is a domain-specific code generation benchmark, featuring a suite of 16 tasks designed 167 to evaluate the ability of LLMs to understand custom APIs and generate programs for service robots. 168 In this domain, a service robot can perceive objects, navigate to various locations, manipulate items, 169 and communicate with humans. Furthermore, the robot should be capable of basic commonsense 170 reasoning and executing complex tasks that involve conditional and repetitive actions. To facilitate 171 these capabilities, ROBOEVAL defines a set of 8 API functions in Python as skill primitives. Fig. 3 172 illustrates these function signatures and definitions, alongside an example task instruction and its 173 canonical solution from the benchmark. In addition, unlike other popular code generation benchmark 174 tasks [2, 6, 9, 14, 16, 19], the order of the robot's actions is crucial for successfully completing the 175 specified tasks . For instance, in the task "bring me a marker from the classroom that does not have a 176 whiteboard," the robot must check each classroom until it finds one without a whiteboard, whereas 177 simply bringing back a marker is insufficient. Hence, ROBOEVAL evaluates the generated program by 178 executing it in a simulator to capture the action traces, which are subsequently validated for sequence 179 correctness using temporal logic. 180

181

182

3.2 RQ1: Is ROBO-INSTRUCT Effective at Generating Training Data to Fine-Tune a Small Language Model for Generating Domain-Specific Robot Programs?

Experiment Setup. We use the open-weight LLM, Llama3-8B-Inst, for ROBO-INSTRUCT. To 183 generate a diverse dataset, we employ nucleus sampling for creating instruction-program pairs, 184 setting the temperature T = 1 and top p = 0 . 95 . The maximum resampling limit is capped at 3 to 185 accommodate instructions that initially produce invalid programs. For the LLM used in INSTALIGN, 186 we empirically adjust the generation temperature to T = 0 . 3 to optimize performance. Furthermore, 187 we assess the edit similarity between token sequences of each instruction pair in the dataset [15], 188 removing duplicates where the similarity score exceeds 0.6. We use the same setup to generate 189 data via SELF-INSTRUCT. Instead of discarding invalid programs, SELF-INSTRUCT includes every 190 generated instruction-program pair in the training dataset. Finally, we create two datasets with 5K 191 instruction-program pairs each using SELF-INSTRUCT and ROBO-INSTRUCT respectively. These 192 datasets are then used to fine-tune the Codellama-Python-7B model. The learning rate is set to be 193

ElevatorTour

CountSavory

HalloweenList

SetTemperature

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

3 e -5 with a warmup ratio of 3% and a constant lr scheduler. We employ the AdamW optimizer [20] with an effective batch size of 8, training each model for 5 epochs using a sequence length of 2048 tokens. We train all our models on a single H-100 GPU using unsloth [35].

Baselines. We divide our baseline models into 2 categories: 1) proprietary LLMs, including GPT4 [28], GPT3.5-Turbo [27], Gemino-Pro [33], and 2) open-weight LLMs, including CodellamaPython-7B [30], Codellama-Python-34B, Starcoder2-33B [21], Deepseek-Coder-33B [8], and Llama3-8B-Inst [1]. All the results are evaluated using ROBOEVAL and reported in Tab. 1.

Tab. 1 presents the average pass@1 results for different LLMs on ROBOEVAL, using two different temperature settings for generation: greedy decoding at a temperatures of T = 0 and nucleus sampling at a temperature of T = 0 . 2 . The results show that ROBO-INSTRUCT-fine-tuned Codellama significantly improves upon the base Codellama-Python-7B and outperforms the SELF-INSTRUCTfine-tuned variant. Notably, it surpasses all open-weight models, including larger ones like CodellamaPython-34B and Deepseek-Coder-33B. Additionally, although the training dataset was generated using Llama3-8B-Inst, which scores less than 50% pass@1 on ROBOEVAL, our ROBO-INSTRUCTfine-tuned model still achieves a significant improvement, scoring 68.75% under deterministic temperature settings for generation. Finally, compared to proprietary models, while our ROBOINSTRUCT-fine-tuned model trails the more powerful GPT-4, it outperforms GPT-3.5-Turbo and Gemini-1.0-Pro in generating programs for service mobile robots. This result demonstrates the effectiveness of our approach in generating domain-specific robot program data for fine-tuning a small language model. It suggests that the fine-tuned model could potentially replace some proprietary models, providing a more cost-effective and private option for local deployment.

3.3 RQ2: How Do ROBOSIM and InstAlign Impact the Effectiveness of ROBO-INSTRUCT?

Using the same setup as in the previous section, we investigate the effectiveness of ROBOSIM 216 and INSTALIGN. Since SELF-INSTRUCT may generate invalid instructions that no corresponding 217 valid program can pass in ROBOSIM, we propose rejecting these unsolvable instructions (we name 218

Logical Error k_program():

("game room")

_in_room

("Hello Jack")

nse =

ask

("Jack", you want to play a

es", "No"])

rror: 'Jack' is not at room' location

this process RU) to evaluate the upperbound performance of SELF-INSTRUCT. Tab. 2 shows the 219 average pass@1 results from Codellama-7B-Python fine-tuned on different datasets generated by 220 each method. First, findings from SELF-INSTRUCT + RU indicate that simply discarding invalid 221 instructions could also improve model performance. Additionally, fine-tuning with a dataset created 222 from SELF-INSTRUCT + RoboSim results in the smallest proportion of invalid program errors. Finally, 223 while incorporating either ROBOSIM or INSTALIGN individually offers some improvement over the 224 baseline SELF-INSTRUCT + RU results, ROBO-INSTRUCT still results in the best performance. This 225 indicates that the integration of these two components is important to the framework's effectiveness. 226

3.4 Qualitative analysis of the generated program errors 227

Weanalyze invalid programs identified by ROBOSIM, categorizing the errors into two types: languagenative errors and domain-specific constraint violations. Fig. 4 displays eight examples of these programs, with Examples 1 to 4 illustrating errors specific to the Python language, and Examples 5 to 8 highlighting errors rooted in domain-specific constraints. Language-native errors are generally straightforward, such as syntax errors, the use of undefined variables or functions, or improper use of provided APIs.

In contrast, errors related to domain-specific constraints tend to be more complex to detect. For instance, Example 5 illustrates the program incorrectly trying to pick up a watering can (line 3) after establishing that it is not present at the location (line 2). Similarly, Example 6 demonstrates an error where the program inappropriately asks Jack (line 5) after confirming his absence from the room

Violation of Constraints

1 def task_program():

2   current_location =

get_current_location

()

3   all_rooms =

get_all_rooms

4   for room in all_rooms:

5

go_to

(room)

228

229

230

231

232

233

234

235

236

237

()

×

1 def task_program():

2

("David's office")

go_to

3   response =

ask

"Where do you want to meet t

["") +

get_all_rooms

() + [""

4

(response)

go_to

5

say

("Tomorrow's Meeting at 10

SyntaxError: closing parenthesis match opening parenthesis '['

×

1 def task_program():

2   list_of_rooms =

get_all_rooms

3   rooms_with_robots = []

4   for room in list_of_rooms:

5     if "robot" in is_in_room

("r

6

go_to

(room)

TypeError: is_in_room("robot") re

'bool' type and it is not iterabl

×

Logical Error

1 def task_program():

2

("game room")

go_to

3   if

is_in_room

4

("Hello Jack")

say

5   response =

ask

"Do you want to play a game

["Yes", "No"])

RobotAskError: 'Jack' is not at t room' location

Logical Error

1 def task_program():

2   if not is_in_room

can"):

3

pick

("watering

4

go_to

("flower bed")

5   response =

ask

("Joh

("

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

(line 3). Example 7 illustrates a scenario in which ROBOSIM updates the world state by labeling "item storage room" as a location after executing the go_to command (line 2). Subsequently, the robot attempts to pick up this location (line 3), resulting in an error. Example 9 is the most intricate scenario where the world state in the living room is updated to include a toy after the robot places it there (line 7). When the robot returns to the living room for the second time (line 5), it does not place down what it holds (line 7). Hence, in the third room the robot visits (line 3), when it attempts to pick up a toy again (line 4), an error occurs because the robot can only carry one item at a time.

4 Related Work

4.1 LLMs for Robot Code Generation

LLMs have shown impressive capabilities in generating robot programs from natural language [11, 17, 31]. One popular approach uses LLMs to generate composable costmaps for robots to plan their motion on. In this approach, Voxposer [12] focuses on the tabletop manipulation setting and NavCon [3] focuses on creating composable maps for navigation. Using LLM to create reward functions is also promising. Eureka [23, 24] and Language to Rewards for Robotic Skill Synthesis [41] both show that LLM can generate good reward functions that allows robots to acquire complex skills. Finally, LLM can also be used to generate programs for high-level planning. LLM+p [18] outputs a robot plan in the form of the well-defined planning domain definition language (PDDL). Tidybot [39] uses an LLM to generate a rule that captures user preferences from examples and executes a program to sequentially complete the task in order. RoboEval [10] focuses on generating domain-specific programs for service mobile robots. It generates a program that allows the service robot to carry out long-horizon tasks and then validates the correctness of the program.

4.2 Generating Datasets For Fine-tuning LLMs

To enhance LLMs' performance in code generation, numerous studies have explored the creation of specialized datasets [13, 25, 26]. SELF-INSTRUCT [36] is one popular method for generating synthetic datasets using an LLM. Following this methodology, Alpaca [32] generates 52K instructionfollowing demonstrations and subsequently fine-tunes the LLaMA 7B model [34] to create Alpaca 7B, which can behave qualitatively similarly to OpenAI's text-davinci-003. Code Alpaca [5] extends this approach to generate code instructions using 21 seed tasks, while Gorilla-LM [29] adapts the method to focus on ML domain-specific APIs from Huggingface, TensorFlow Hub, and Torch Hub. To create more complex instructions, Evol-Instruct [22, 40] proposes iteratively updating instructions to become more complex through different prompting strategies. In addition to Evol-Instruct, OSS-Instruct [38] uses open-source code snippets to generate 75K high-quality instruction data and fine-tunes the Codelllama-Python-7B model to create Magicoder, which can match the performance of GPT-3.5Turbo [27] on HumanEval [6]. While these works focus on creating seed instruction sets to generate synthetic data for effectively fine-tuning an LLM, our research investigates post-processing methods in addition to SELF-INSTRUCT. Specifically, we concentrate on generating domain-specific programs in robotics [10], where we can effectively leverage constraints to filter out erroneous programs.

5 Conclusion, Limitation and Future Works

In this work, we introduce ROBO-INSTRUCT, a novel framework to generate synthetic training data to fine-tune small language models for domain-specific robot programs. ROBO-INSTRUCT comprises two novel components: 1) ROBOSIM, an angelic-execution-based algorithm to effectively validate SELF-INSTRUCT-generated programs, and 2) INSTALIGN, an instruction alignment procedure to revise instructions to better align with the generated programs. The experimental results demonstrate that the Codellama-Python-7B model fine-tuned on the ROBO-INSTRUCT-generated dataset can significantly outperform many popular open-weight LLMs for generating domain-specific robot programs. It also outperforms two proprietary LLMs, GPT-3.5-Turbo and Gemino-1.0-Pro, as well as the SELF-INSTRUCT-fine-tuned variant. A limitation of this study is that ROBO-INSTRUCT relies on SELF-INSTRUCT to filter invalid programs, making the dataset quality dependent on SELFINSTRUCT's performance. This can introduce biases if SELF-INSTRUCT consistently fails in certain areas. Future work will explore integrating ROBO-INSTRUCT with advanced methods like Evol-Inst and OSS-Inst to enhance dataset quality for domain-specific robot programs.

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

References

Meta AI. Introducing meta llama 3: The most capable openly available llm to date. https: //ai.meta.com/blog/meta-llama-3/ , 2024. Accessed: 2024-05-21.

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021.

Harel Biggie, Ajay Narasimha Mopidevi, Dusty Woods, and Christoffer Heckman. Tell me where to go: A composable framework for context-aware embodied robot navigation, 2023.

Manfred Broy and Martin Wirsing. On the algebraic specification of nondeterministic programming languages. In Proceedings of the 6th Colloquium on Trees in Algebra and Programming , CAAP '81, page 162-179, Berlin, Heidelberg, 1981. Springer-Verlag. ISBN 3540108289.

Sahil Chaudhary. Code alpaca: An instruction-following llama model for code generation. https://github.com/sahil280114/codealpaca , 2023.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. 2021.

Richard E. Fikes and Nils J. Nilsson. Strips: a new approach to the application of theorem proving to problem solving. In Proceedings of the 2nd International Joint Conference on Artificial Intelligence , IJCAI'71, page 608-620, San Francisco, CA, USA, 1971. Morgan Kaufmann Publishers Inc.

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming - the rise of code intelligence, 2024. URL https://arxiv.org/abs/2401.14196 .

Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. NeurIPS , 2021.

Zichao Hu, Francesca Lucchetti, Claire Schlesinger, Yash Saxena, Anders Freeman, Sadanand Modak, Arjun Guha, and Joydeep Biswas. Deploying and evaluating llms to program service mobile robots. IEEE Robotics and Automation Letters , 9(3):2853-2860, 2024. doi: 10.1109/ LRA.2024.3360020.

Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual language maps for robot navigation. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) , London, UK, 2023.

Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. In 7th Annual Conference on Robot Learning , 2023. URL https://openreview.net/forum?id= 9_8LF30mOC .

Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Minh Nguyen, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Alexandrovich Glushkov, Arnav Varma Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Julian Mattick. Openassistant conversations

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

democratizing large language model alignment. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2023. URL https:// openreview.net/forum?id=VSJotgbPHF .

Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, WenTau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: A natural and reliable benchmark for data science code generation. ArXiv , abs/2211.11501, 2022.

Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 8424-8445, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.577. URL https://aclanthology.org/2022.acl-long.577 .

Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. Science , 378(6624):1092-1097, 2022. doi: 10.1126/science. abq1158. URL https://www.science.org/doi/abs/10.1126/science.abq1158 .

Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In arXiv preprint arXiv:2209.07753 , 2022.

Bo Liu, Yuqian Jiang, et al. LLM+P: Empowering Large Language Models with Optimal Planning Proficiency. arXiv:2304.11477 , 2023.

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https: //openreview.net/forum?id=1qvx610Cu7 .

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum? id=Bkg6RiCqY7 .

Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Marc Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder 2 and the stack v2: The next generation, 2024.

Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=UnUwSIgK5W .

Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv: Arxiv-2310.12931 , 2023.

Yecheng Jason Ma, William Liang, Hungju Wang, Sam Wang, Yuke Zhu, Linxi Fan, Osbert Bastani, and Dinesh Jayaraman. Dreureka: Language model guided sim-to-real transfer. 2024.

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786 , 2022.

Niklas Muennighoff, Qian Liu, Armel Randy Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=mw1PWNSWZP .

OpenAI. Chatgpt: Optimizing language models for dialogue. https://openai.com/blog/ chatgpt/ , 2022.

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, et al. Gpt-4 technical report, 2024.

Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334 , 2023.

Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code, 2024.

Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 11523-11530, 2023. doi: 10.1109/ICRA48891.2023.10161317.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca , 2023.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul R. Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, et al. Gemini: A family of highly capable multimodal models, 2024.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.

Unslothai. Unsloth: Finetune llama 3, mistral & gemma llms 2-5x faster with 80 URL https://github.com/unslothai/unsloth . Accessed: 2024-05-22.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions, 2022.

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code is all you need, 2023.

Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song, Jeannette Bohg, Szymon Rusinkiewicz, and Thomas Funkhouser. Tidybot: Personalized robot assistance with large language models. Autonomous Robots , 2023.

Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. WizardLM: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=CfXh93NDgH .

Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, Brian Ichter, Ted Xiao, Peng Xu, Andy Zeng, Tingnan Zhang, Nicolas Heess, Dorsa Sadigh, Jie Tan, Yuval Tassa, and Fei Xia. Language to rewards for robotic skill synthesis. Arxiv preprint arXiv:2306.08647 , 2023.

A Appendix 458

459

A.1 Overview

In this appendix, we first present ablation experiments to investigate the percentage of invalid 460 programs generated by SELF-INSTRUCT and examine how the generation temperature in INSTALIGN 461 affects final performance. Next, we analyze and compare the datasets generated by ROBO-INSTRUCT 462 and SELF-INSTRUCT. Finally, we list the seed tasks used in ROBOEVALand the CoT prompt. 463

A.2 Ablation Exmperiments 464

68

50

66

40

Programs (%)

Discarded

64

30

ass

20

62

10

60

0

1

23

4

5

6

7

8

9

10

58

0.1

0.3

0.5

0.7

0.9

Maximum Resampling Limit

InstAlign LLM Generation Temperature

A.2.1 effectivenss of the simulator

We analyze the percentage of instruction-program pairs discarded by ROBOSIM at various maximum resampling limits, as shown in Fig. 5. Initially, with the maximum resampling limit set to 0, disabling the rejection-sampling method, approximately 51% of the programs generated by SELF-INSTRUCT contain errors. As the limit increases, fewer programs are discarded. However, there is a diminishing return; even with the maximum resampling limit set to 10, about 15% of the instructions still result in invalid programs.

A.2.2 Instruction Alignment model temperature

465

466

467

468

469

470

471

472

We further investigate how varying LLM temperatures for generating I revised in INSTALIGN impact 473 the performance of the fine-tuned model. Fig. 5 shows the bar chart of the pass@1 score of the 474 models fine-tuned over datasets generated using different LLM temperatures in INSTALIGN. The 475 model performs the best when fine-tuned on the dataset generated using LLM temperature T = 0 . 3 . 476 As the temperature increases, we observe a decrease in performance. 477

A.3 Analysis of the Generated Datasets 478

0.14

Self-instruct

Robo-Instruct

0.14

0.12

Self-lnstruct;MedianScore:0.117

0.12

Robo-lnstruct;Median Score:0.120

Percentageof Total

0.10

Percentage

0.10

0.08

0.08

0.06

0.06

0.04

0.04

0.02

0.02

0.00

0.0

0.1

0.2

0.3

0.4

0.5

0.00

0

200

400

600

800

1000

CosineSimilarityScore

Number of Tokens

(a) Token Length Distribution for SELF-INSTRUCT vs. ROBO-INSTRUCT (b) Cosine Similarity with ROBOEVALfor SELFINSTRUCT vs. ROBO-INSTRUCT

We first compute and plot the distribution of token lengths in the SELF-INSTRUCT-generated dataset 479 and the ROBO-INSTRUCT-generated dataset, as shown in Fig. 6(a). Next, we measure the cosine 480 similarity between each dataset and the ROBOEVALbenchmark tasks following the approach in 481 Magicoder [38], as depicted in Fig. 6(b). Finally, Tab. 3 presents the n-gram diversity score of each 482 dataset, along with the number of synthesized locations and objects. Our findings indicate that both 483 distributions and dataset statistics are very similar, suggesting that ROBO-INSTRUCT enhances the 484 quality of the generated data over SELF-INSTRUCT rather than merely aligning the dataset towards 485 the benchmark tasks. 486

487

A.4 ROBOEVALSeed Task Example

A.5 Prompts to Generate Synthetic Dataset Using SELF-INSTRUCT 590

You are a helpful assistant. Here is a robot that has the following capabilities:

def get_current_location() -> str:

def get_all_rooms() -> list[str]:

def is_in_room(object : str) -> bool:

def go_to(location : str) -> None:

def ask(person : str, question : str, options: list[str]) -> str:

def say(message : str) -> None:

def pick(obj: str) -> None:

def place(obj: str) -> None:

Generate an interesting robot task that can be accomplished using the above capabilities.

{{SEED EXAMPLE}}

Generate an interesting robot task that can be accomplished using the above capabilities.

...

A.6 CoT Prompts for INSTALIGN 591

### Role: You are an expert at understanding robot programs. You will be given a task instruction and robot program pair. However, the instruction may not align with the program well. You need to correct the task instruction to match the given robot program.

### Context: The robot only has access to the following 8 APIs and standard Python functions

- ask(person : str, question : str, options: list[str]) -> str:

def place(obj: str) -> None:

### Inputs

Original Instruction: This is a task instruction that may not align with the robot program Robot Program: This is a python function starting with 'def task_program():'

### Task:

1. Write down all the provided APIs used in the program and explain the effect of each API in this program

2. Examine these APIs and write down step by step what the program does

3. Combine all the results above and rewrite the instruction under # Final Corrected Instruction:

You need to be specific and clear in your final corrected instruction.

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

B CheckList

[Claims] Yes. The research questions listed in the evaluation section are formulated so as to directly reflect the claims of the paper.

[Limitations] Yes. This is discussed in Section 5.

[Theory, Assumptions and Proofs] N/A. We do not have any theoretical results.

[Experimental Result Reproducibility] Yes. We provide the training hyperparameters in Section 4. We will also release our model upon acceptance.

[Open Access to Data and Code] Yes. We provide the prompts that are used to generate the training dataset. We will also release our training dataset upon acceptance.

[Experimental Setting/ Details] Yes. We discuss the details of the training scheme in Section 3.2, which follows the standard approach to fine-tuning an LLM.

[Experiment Statistical Significance] Yes. We performed ablation studies to validate our methods in Section 3.3.

[Experiments Compute Resource] Yes. We mention that we train all our models on a single H-100 GPU using unsloth in Section 3.2.

[Code Of Ethics] Yes

[Broader Impacts] N/A: This paper addresses an existing problem (using LLMs to synthesize robot programs [10, 12, 17]), and does not introduce any novel concerns beyond the existing scope.

[Safeguards] N/A: the programs we will generate or release are domain-specific with respect to RoboEval [10], which has existing safeguards in place.

[Licenses] Yes - we build on SELF-INSTRUCT, Llamav3 [30], and RoboEval [10] with attribution, and Table 1 refers to the licenses of the models used in the evaluation.

[Assets] N/A. The code we will release will include details of documentation, training, 615 license, and limitations. The code will be released upon acceptance. 616

[Crowdsourcing and Research with Human Subjects] N/A 617

[IRB Approvals] N/A 618

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces ROBO-INSTRUCT, a novel framework designed to generate synthetic training data for fine-tuning small language models to create domain-specific robot programs. The framework features two main components: ROBOSIM, an algorithm that validates programs using angelic execution, and INSTALIGN, which aligns instructions with the generated programs.\nThe key contributions of this work include the development of ROBO-INSTRUCT to enhance the code generation performance of small open-weight language models for domain-specific robot programs. This framework introduces ROBOSIM, which features a dynamic world synthesis and evaluation process for generating relevant world states and performing automated code checks for diverse tasks. Additionally, it includes INSTALIGN, a procedure that refines instruction-code pairs to improve alignment between instructions and the code generated by SELF-INSTRUCT. By fine-tuning the Codellama-Python-7B model using ROBO-INSTRUCT, the model significantly outperforms several other open-source and most proprietary models.\n\nSTRENGTHS:\n- The paper demonstrates strong clarity and organization, making complex concepts accessible to readers. Each section flows logically, and technical details are explained effectively, ensuring that the methodology and findings are easy to follow.\n- The paper thoroughly reviews and incorporates current literature.\n- The paper meticulously validates all its claims through experimental results and detailed analysis. The effectiveness of ROBO-INSTRUCT, ROBOSIM, and INSTALIGN is demonstrated convincingly through empirical data and comparisons with existing models and benchmarks. This empirical validation ensures that the contributions are not just theoretical but substantiated with practical evidence.\n\nWEAKNESSES:\n- While ROBO-INSTRUCT offers significant advancements for fine-tuning language models in robot programming, there are some weaknesses to consider. Firstly, it heavily relies on SELF-INSTRUCT for generating initial programs, potentially introducing biases from the base model's training data. This could limit the diversity and quality of the generated programs.\n- Moreover, while ROBO-INSTRUCT shows promising results on benchmarks like ROBOEVAL, its application to real-world robot programming tasks requires thorough evaluation and validation. Real-world robot environments often present unpredictable challenges that benchmark datasets may not fully capture, necessitating further testing to assess the framework's robustness and generalizability in practical applications.\n\nQUESTIONS:\nWhat factors do you think contribute to GPT-4's superior performance compared to ROBO-INSTRUCT, despite your method showing competitive results against other models like GPT-3.5-Turbo and Gemini-1.0-Pro\n\nLIMITATIONS:\nThe limitations of the work are clearly stated", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tries to improve the performance of small open-sourced LLMs to generate code that can run successfully on a service robot simulator to solve tasks. The idea is to use another small model to generate program data using SELF-INSTRUCT and fine-tune a 7B model for the robotics domain. The authors note that data generated by SELF-INSTRUCT may have good diversity but lack correctness. To this end, they build a simulator RoboSIM that takes the programs generated by SELF-INSTRUCT and verifies the correctness of the execution in addition to syntax errors given a predefined robot API. They further modify the instructions to align with the verified programs better. Overall they show the 7B LLM fine-tuned on this clean data can outperform a GPT3.5-Turbo in the robotics domain.\n\nSTRENGTHS:\nThe paper is well-written and motivated. Figures are easy to understand and helpful in conveying high-level ideas. The idea to build a pseudo-simulator that tracks world states without running the generated programs through an actual simulator is novel. It is amazing that synthetically generated data filtered by simple heuristics such as requiring programs to pass the world state tracking RoboSim is sufficient to help improve the performance of an actual simulator.\n\nWEAKNESSES:\nWhile the paper is well-written for the scope it sets for itself, I am not sure if the contribution is significant enough. There are many works using LLM to generate data and fine-tune domain-specific models, so the idea behind this paper is not super novel. The performance gain is also limited by the rather heuristic method considering the gap between the best model presented by the paper and GPT-4 it sought out to beat. In fact, given the 17% performance gap, simple baselines could be using GPT-4 to generate the programs and fine-tuning small models or using GPT-4 as the critic to filter programs. These simpler heuristics may yield better results and prove the Robo-Instruct method (which is also rather heuristic) proposed by the paper unnecessary. For reference consider \n[1] Improving Small Language Models on PubMedQA via Generative Data Augmentation\nTherefore, I am not sure the contribution of this paper is all that significant.\n\nQUESTIONS:\n1. Could you explain why InstAlign would improve pass@1 success rate by ~5% on top of RoboSIM+RU in table 2? What's the error bar/variance of these statistics? What is the intuition that InstAlign would help and could you give some examples? \n2. One claim made early on in the paper is that Self-Instruct gives high-diversity data but lacks correctness, and a simulator can give correctness but lacks diversity. The paper is trying to get the best of both worlds. While the current experiment results show that Robo-Instruct does generate data with high correctness, we don't see evidence that Robo-Instruct preserves diversity. Can you show some evidence (through some metrics) on diversity?\n\nLIMITATIONS:\nBuilding RoboSim to track world states might work only for simple pick-n-place or task-planning problems. How to generalize this heuristic of tracking world states to more complex problems is unclear. Some analysis of the scope (suitable for what kind of problem class) of this approach is needed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper introduces a framework for generating paired instruction and robot code for further fine-tuning LLMs for robot-specific tasks. A symbolic simulator is used to check the correctness of the generated code and an LLM is prompted with chain-of-thought reasoning to align generated instruction. The resulted dataset was used to fine-tune a robot specific LLM and later tested on benchmark tasks.

STRENGTHS:
This paper tackles two challenges of automated data generation for robot code synthesis, one is to check the correctness of the code by grounding it to logical state of objects and the other is to align the generated the instructions to provided robot capabilities. By designing principled and general modules that tackle each of these problems, RoboInstruct is shown to generate useful data for finetuning general purpose LLM to robot specific code generation applications.

WEAKNESSES:
RoboSIM can only check for semantically meaningful steps of the code and may not catch lower-level error that requires spatial/geometric reasoning, or even reasoning about physics, including commands that take in numerical parameters e.g. move(0,0.2,0), rotate(0.75). This seem to limit the usefulness of RoboInstruct to certain types of robot APIs.

QUESTIONS:
Can the prompt of InstAlign be adapted to the instruction generation step? What information after the fact is being used by instAlign that cannot be useful at initial generation round?

LIMITATIONS:
see weakness
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ROBO-INSTRUCT, a novel framework designed to improve the code generation capabilities of smaller open-weight language models (LLMs) for domain-specific robotic tasks. ROBO-INSTRUCT leverages two key components:\n\n1. ROBOSIM with DYNAMICEVAL:  A task-agnostic simulator that dynamically synthesizes a consistent world state based on the robot's actions within the program. This allows ROBOSIM to identify execution errors and validate generated programs even for diverse and complex tasks.\n2. INSTALIGN: An instruction-program alignment procedure that utilizes Chain-of-Thought reasoning to refine the generated instructions. This ensures that the instructions better reflect the intent of the generated robot program, improving alignment between the two.\n\nThe paper evaluates ROBO-INSTRUCT by fine-tuning a Codellama-Python-7B model and testing its performance on ROBOEVAL, a benchmark for service mobile robots. The results demonstrate that the ROBO-INSTRUCT fine-tuned model significantly outperforms other open-weight models\n\nSTRENGTHS:\nNovel framework: Introduces ROBO-INSTRUCT, a unique approach to generating training data for fine-tuning smaller LLMs on domain-specific robot tasks.\n\nDynamic world synthesis: ROBOSIM's ability to dynamically create relevant world states allows it to validate diverse programs generated by SELF-INSTRUCT, overcoming the limitations of traditional simulators.\n\nInstruction-program alignment: INSTALIGN effectively refines instructions to better reflect the program's intent, improving the quality of the training dataset.\n\nStrong empirical results: Demonstrates that ROBO-INSTRUCT significantly improves the performance of small open-weight LLMs, enabling them to surpass even some proprietary LLMs.\n\nCost-effective and private:  Provides a potential alternative to deploying proprietary LLMs for local robot deployment, offering cost-effectiveness and privacy benefits.\n\nWEAKNESSES:\nLimited novelty: The idea of using a sim/emulator to verify the generated program has already been explored in previous works such as Chain-of-code, which is not mentioned by this work. \n\nLimited scope: The paper focuses on a specific domain (service mobile robots), and it is unclear how well ROBO-INSTRUCT generalizes to other robot domains.\n\nLack of real-world evaluation:  The paper only evaluates ROBO-INSTRUCT on a synthetic benchmark. Real-world deployment and testing are required to further assess its practicality.\n\nQUESTIONS:\nFigure 1 is confusing as the authors put the overview of the proposed method, counter examples from previous methods, as well as the benchmark results all  in a single figure.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}