ROBO-INSTRUCT: Simulator-Augmented Instruction Alignment For Finetuning Code LLMs

Authors: Zichao Hu, Junyi Jessy Li, Arjun Guha, Joydeep Biswas
Primary area: applications to robotics, autonomy, planning
Keywords: Finetune LLM For Domain Specific Application, Angelic Execution, Self-Instruct, Synthesize Simulation Environment, CodeLLMs for Robotics

Abstract:
Open-weight LLMs are particularly appealing choices to generate training data for fine-tuning Code LLMs on domain-specific service robot applications because they are cost-effective, customizable, and offer better privacy protection. However, unlike proprietary LLMs, open-weight models are more error-prone and often produce programs that violate domain-specific constraints. A promising solution is to incorporate a robot simulator with a well-defined environment to verify program correctness. Yet, these environments require pre-enumeration of relevant entities and their states, which limits the diversity of programs that can be effectively verified. In this work, we introduce ROBO-INSTRUCT that preserves the diversity of programs generated by an LLM while providing the correctness of simulator-based checking. ROBO-INSTRUCT introduces ROBOSIM to dynamically synthesize consistent simulation environments for each generated program. Moreover, ROBO-INSTRUCT handles subtler instruction-program inconsistencies that do not result in a constraint violation via INSTALIGN, an LLM-aided instruction-program alignment process. Given domain-specific APIs and a few seed examples, ROBO-INSTRUCT can leverage an 8B Llama3 model to generate a training dataset for fine-tuning a 7B CodeLlama model. Our fine-tuned model achieves a 28.75% improvement in pass@1 over the original base model and a 13.75% improvement compared to its SELF-INSTRUCT-finetuned counterparts, even surpassing the performance of a few proprietary LLMs, such as GPT-3.5-Turbo and Gemini-Pro.

Decision: reject
Reviewer scores: 6, 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1729224345963

thread_type:
official_review

<thread_object>
creator: reviewer_dwpd

<content>
summary:
This paper proposes a framework (Robo-Instruct) to generate training data to fine-tune a code LLM for domain-specific service robot applications. Robo-Instruct contains 2 components: (1) RoboSim that dynamically synthesizes consistent simulation environments for each generated program, and (2) InstAlign that handles the instruction-program inconsistencies. In the experiments, the authors use Llama3-8B-Instruct as the LLM to generate training data, and fine-tuned CodeLlama to perform on the RoboEval benchmark.

I think the authors are tackling an important problem that would allow LLMs to be better applied to robotics. However, I find the paper hard to follow, with insufficient experiments to support the potentially over-claimed contributions. Please see my explanation below.

strengths:
- Having a simulation (not a physics engine in this paper) to improve the diversity of the generated programs intuitively increases the performance of an LLM for robotic applications, the paper is therefore addressing problems of importance.

weaknesses:
- The contributions of the paper are limited. Of the 5 contributions listed in the introduction, I find 4 of them questionable. (1) RoboSim is said to produce diversity, yet on line 245 the authors explain that checking all possible states is not possible and they resort to random sampling with limited compute budget. This does not necessarily guarantee diversity. (2) InstAlign is just CoT. (3) The authors claim the fine-tuned model is better, but it is not clear how different are the data it has been trained on from the tasks it is tested in. (4) The authors claim the fine-tuned model is faster in inference than proprietary models. This is especially unfair and misleading. Any onboard model with sufficient hardware support is faster than remote API calls. 
- Insufficient experiments. How different are the generated programs from the tasks in RoboEval? Is it a generalized performance or performance on the training set? Why use Llama3-8B-Instruct as the generating LLM and CodeLlama as the fine-tuned LLM? If you change them to other LLMs, will the framework still work?
- Presentation can be significantly improved. The paper is full of details that do not help a reader follow the main thread of the paper. None of the links (e.g., citations, sections, figures, etc) works, and I have to manually search for those entities. Inefficient use of space (e.g., are the 2 python programs on lines 188 and 202 necessary?).

questions:
- How different are the generated programs from the tasks in RoboEval? Is it a generalized performance or performance on the training set?
- Why use Llama3-8B-Instruct as the generating LLM and CodeLlama as the fine-tuned LLM? If you change them to other LLMs, will the framework still work?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730031527972

thread_type:
official_review

<thread_object>
creator: reviewer_nfrk

<content>
summary:
The paper introduces ROBO-INSTRUCT, a framework designed to enhance the generation of training data for fine-tuning Code LLMs in domain-specific service robot applications. It consists of ROBOSIM, a task-agnostic simulator that dynamically creates consistent simulation environments to verify program correctness, and INSTALIGN, an LLM-aided instruction-program alignment process. The framework significantly improves the performance of a fine-tuned model, achieving a 28.75% improvement in pass@1 over the base model and surpassing several proprietary models. Additionally, ROBO-INSTRUCT demonstrates faster inference speeds, making it suitable for real-world robot deployments.

strengths:
1. Applying code to embodied AI is an important direction, so exploring how to enhance the code generation capabilities of LLMs in the robotics domain is also meaningful.
2. The idea of guiding the synthesized data with verification of the ROBOSIM environment is reasonable.
3. The experimental result looks promising.

weaknesses:
I'd be happy to raise my score if my concerns are addressed.
1. There are many instruction tuning methods for code LLMs, but this paper only compares with SELF-INSTRUCT. Could the authors compare with methods like evol-instruct in Wizardcoder as well?
2. The choice of settings is somewhat confusing. For example, why use the Llama3 model to generate the training set and then fine-tune the 7B CodeLlama instead of Llama3 itself? Why not use more powerful closed-source models like GPT-3.5 or GPT-4-Turbo for synthesizing the dataset? I think the authors should either provide corresponding results or at least explain the reason for doing so.
3. I think the author should have decontaminated the test set, but it seems the author did not mention any relevant details in the experimental setup.
4. The real-world deployment results are great but the authors only measured the inference speed. Could the authors measure some accuracy or success-related metrics?

questions:
1. I'm a bit confused about ROBOSIM. It is claimed that ROBOSIM is task-agnostic but it seems that ROBOSIM still requires a lot of expert knowledge for the corresponding task. For example, ROBOSIM cannot work on the "apple" tasks if the "apple" or "kitchen" related properties (i.e., missing any of the entities, type, or state) are absent. Or, to put it another way, even if "apple" and "kitchen" are present and ROBOSIM can complete tasks related to "apple" and "kitchen," it still won't be able to complete tasks related to "apple" and "living room" due to the absence of the "living room."
2. As the above question said, how much expert effort is needed to construct ROBOSIM?
3. A typo in line 375 "Gemino-Pro".
4. In Table 2, why does ROBO-INSTRUCT have a higher invalid program rate than ROBOSIM + RU?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712297021

thread_type:
official_review

<thread_object>
creator: reviewer_d5tg

<content>
summary:
The paper introduces ROBO-INSTRUCT, a framework designed to improve open-weight Large Language Models (LLMs) for generating domain-specific training data. This framework aims to enhance service robot applications, focusing on Code LLMs that use domain-specific APIs to generate executable robot instructions. The ROBO-INSTRUCT framework comprises two main components: ROBOSIM: A task-agnostic simulator to synthesize simulation environments for verifying program correctness dynamically. INSTALIGN: An alignment tool that adjusts program instructions to reflect the true intent of generated programs, using a large language model (LLM) for instruction revision. Experiments show that models fine-tuned with ROBO-INSTRUCT outperform base models and models fine-tuned with SELF-INSTRUCT, improving pass@1 scores and real-world deployment latency.

strengths:
Clarity: The framework's purpose, components, and experimental results are presented clearly, though some complex aspects could benefit from additional clarification (e.g., the alignment between ROBOSIM and traditional STRIPS planning). The experiment design is well-articulated, showing comparisons across multiple baselines and careful control of variables.

Novelty: The integration of ROBOSIM for dynamic simulation and the INSTALIGN for instruction alignment introduce a novel approach to overcoming LLM limitations in handling domain-specific instructions. The work holds promise for cost-effective deployment of LLMs in real-world robot applications, especially where open-weight models are prioritized for privacy and customizability.

weaknesses:
(1) The data augmentation approach seems somewhat incremental, given its widespread use in Evol-Instruct, WizardLM, and similar frameworks. It would be valuable to explore more unique challenges and solutions tailored to robotics, which often requires handling more complex tasks. Additionally, an evaluation on scaling performance regarding parameter count, generalization, and related metrics would strengthen the analysis.

(2) Another concern is that the evaluated tasks in the paper appear overly simplified. While I understand that many current studies also rely on simplified environments like VirtualHome, the solution's handling of out-of-distribution scenarios remains insufficiently understood. This is a crucial factor for robotics applications, where the risk of overfitting due to augmentation is particularly high.

questions:
(1) How does ROBO-INSTRUCT handle edge cases where simulator-based validation cannot capture subtler domain inconsistencies?

(2) relying on data augmentation techniques such as self-instruct or evol-instruct may introduce bias or even hurt the generalization of LLMs, it would be nice to see related evaluation on 

(3) the paper verifies the program correctness, is there any other filtering like ROGUE-L methods as used in self-instruct?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730764189704

thread_type:
official_review

<thread_object>
creator: reviewer_ycyr

<content>
summary:
The goal of this paper is to

strengths:
Clarity. The authors did a phenomenal job describing their method and experimental process with precision. The specifics of the robosim environments were easy to follow from the method section and the motivation for dynamic environment generation and its unique application to robotic service agents was well presented.

Quality. The approach is simple and sound and I believe there is sufficient information for researchers to reproduce the results. On the RoboEval benchmark their method produces a model that outperforms even proprietary language models.

Significance. The idea of using dynamic environments to evaluate code could have broad impact for robotic code generation. The author present a strong first demonstration of this.

weaknesses:
Presentation. Focusing solely on open models as being error-prone unnecessarily limits the scope and potential impact of the paper's contributions when it could be relevant to any base LLM.

Quality. There was a limited diversity of baselines. The domain specific language looks very similar to the code-as-policies test environments. In the experimental section, the authors should contextualize the results by either explaining the best analogy to code-as-policies that they run or by explicitly discussing a code-as-policies style baseline. The authors could also try simpler variants of their method: for example, taking some data points generated by robosim rejection sampling process and putting them in the prompt instead of doing model fine-tuning and alignment.

Significance. The APIs in the roboeval benchmark are very high-level and I wouldn't be surprised if there are many solid engineering approaches to getting a performant policy that works with high-level APIs. It's unclear how well roboinstruct will scale with more complex APIs and tasks, which limits the significance of the work.

questions:
Could you please fix the formatting of table 5 to remove the overlapping text?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a framework to fine-tune a code LLM for robotics tasks. The paper is well written and motivated. However, at the same time the reviewers have raised several concerns, particularly on the novelty of ideas in the paper. The evaluated tasks and the real world deployments seem to be simplistic and hence are quite limited in its current form from having significant impact in the ML community.