VLAS: Vision-Language-Action Model with Speech Instructions for Customized Robot Manipulation

Authors: Wei Zhao, Pengxiang Ding, Zhang Min, Zhefei Gong, Shuanghao Bai, Han Zhao, Donglin Wang
Primary area: applications to robotics, autonomy, planning
Keywords: Vision-Language-Action Model, Speech Instructions, Robot Manipulation

Abstract:
Vision-language-action models (VLAs) have recently become highly prevalent in robot manipulation due to its end-to-end architecture and impressive performance. However, current VLAs are limited to processing human instructions in textual form, neglecting the more natural speech modality for human interaction. A typical approach of incorporating speech modality into VLA necessitates a separate speech recognition system to transcribe spoken instructions into text. Such a cascading pipeline raises two major concerns for robotic systems. First, the entire model grows in size and complexity, potentially resulting in redundant computations and increased memory consumption. Second, the transcription procedure would lose non-semantic information in the raw speech, such as voiceprint, which is crucial for a robot to successfully understand and complete customized tasks. To this end, we propose VLAS, the fisrt end-to-end policy model that seamlessly integrates speech modality for robot manipulation. We present a three-stage speech instruction tuning strategy leveraging multimodal datasets, including our manually curated SQA and CSI datasets. Furthermore, to facilitate personalized operations, we develop a voice retrieval-augmented generation (RAG) approach to enhance the robot's performance in tasks requiring individual-specific knowledge. Experimental results show that the proposed VLAS, following either textual or speech instructions, can achieve performance comparable to traditional VLAs on the CALVIN benchmark. In addition, we created a benchmark consisting of customization tasks, where our VLAS demonstrates absolute superiority by fully leveraging the auxiliary information in speech.

Decision: accept (poster)
Reviewer scores: 8, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730355356487

thread_type:
official_review

<thread_object>
creator: reviewer_5gyv

<content>
summary:
This work proposes VLAS and integrates speech for robot manipulation. It makes communication more natural and presents two new datasets for the community.

strengths:
1. This paper is easy to read and follow.
2. The core idea and motivation are interesting and practical. I believe it is great to have the speech modality in interactions with robots, as it feels natural and intuitive. Using speech to capture human intent is also a good idea. Creating a unified model provides a solid foundation for future work.
3. The illustrations are very helpful.
4. The datasets provided to the community are beneficial for the field.

weaknesses:
Some ablation studies may be needed. Please refer to the "Questions" section.

questions:
1. Are there any statistics from real-world experiment results? I think it's important to understand how the impressive results from simulations can transfer to real-world applications.
2. You use the RAG and the pretrained speech model to obtain the voiceprint, which is then used to retrieve preferences from the database. It seems the voiceprint is derived from the RAG module. You only present the VLA experiment in a customized task, and I wonder about the performance of the VLA combined with speech RAG in this context. I think this need to be given to support your motivation ( why not use the cascading pipeline, you mentioned the cost and the intent capture), I suspect that the speech RAG can capture the intend and give the intend text from the database to VLA model.
3. I think you need some more ablation studies, such as examining the effects of not using the RAG module in the customized task.

I think if these questions are addressed, I will raise my score cause I think the unified large model for speech, vision, text and action is meaningful.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730668078465

thread_type:
official_review

<thread_object>
creator: reviewer_mepm

<content>
summary:
This paper creates a vision-language-action model by incorporating speech to create a VLAS model.  Two new datasets SQA and CSI are introduced.  A RAG setup is used for handling contextual queries.  Section 1 gives background for why they incorporated speech, including contextual queries.  The model is built on top of LLaVA.  A pretrained speech encoder was used.  The model is called, “Speech LLaVA.”  Comparable accuracy can be found on the CALVIN benchmark to traditional VLAs.  For customization tasks, this approach appears to do very well.  Section 2 describes related work.  Section 3 describes the approach, including the LLaVA-based approach, action tokenization / detokenization.  SQA dataset was created by sampling examples from the original LLaVA dataset and running them through a TTS system.  CSI dataset used 389 textual examples x 500 voices to result in 194k examples; training was interleavesd with text and audio.  A three stage process involving audio alignment, speech question answering and then robot manipulation fine tuning was performed.  Section 4 describes experiments and results across CALVIN, customized tasks and LLaVA.

strengths:
This paper presents a natural extension to LLaVA to use speech input. With minor reservations about related work, the confounding nature of speech with and without context (and the lack of a text baseline), the exclusive use of TTS outputs and a few baselines, the paper is well-written and the results appear to be reproducible.

weaknesses:
see above

questions:
Some questions about the paper:

* The RAG pipeline appears to be intimately coupled with speech in this paper, but needn’t be.  Why wasn’t a baseline with language only inputs performed?  What would be the results?
* All the experiments appear to be with TTS output, which will certainly be biased.  Some results need to be with real speech.  Do you have any quantitative numbers with real speakers?  Even on a subset of any of the data you presented?
* The incorporation of RAG into the pipeline makes some parts of the paper confusing.  For example, in Section 4.1 it wasn’t clear if these leveraged the context part of the model (e.g., RAG) (e.g., Table 1).
* Why wasn’t OpenVLA shown in Table 1?


For related work, it would be good to cite at least the following:
* Other multi-modal models that incorporate audio should be included here.  Some missing references include, but there are likely more:
    * Unified-IO 2: https://arxiv.org/abs/2312.17172
    * Gemini (inherently multi-modal)
* Other VLMs, such as IDEFICS and Prismatic:
    * https://huggingface.co/blog/idefics
    * https://github.com/TRI-ML/prismatic-vlms

Other small things:
“pre-cess” -> “process” p2.
Inconsistency between VLAS and SpeechLLaVA terms.
“Absolute superiority” -> probably weaken this term.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702416742

thread_type:
official_review

<thread_object>
creator: reviewer_tjdb

<content>
summary:
The paper introduces VLAS, a novel vision-language-action model designed to facilitate robot manipulation tasks by integrating speech as a primary instruction modality, bypassing traditional speech-to-text conversion methods. This approach enables VLAS to capture essential non-verbal cues, such as voiceprints, which assist in executing personalized tasks based on individual user needs. Built on the LLaVA model, VLAS employs three training stages to process multimodal inputs across text, image, and speech, aligning these representations for seamless action generation. The authors also developed two datasets, SQA and CSI, to enhance the model’s handling of speech instructions, alongside a voice retrieval-augmented generation (RAG) system to access individualized knowledge. Experimental results demonstrate VLAS’s superior performance on complex, customized robot tasks, particularly when handling personalized commands, outperforming other models that rely on separate speech recognition systems.

strengths:
1. The paper introduces VLAS, a vision-language-action model evaluated on a comprehensive set of tasks, including customized tasks that leverage speech input for personalized robot manipulation. 

2. Additionally, the authors contribute two new datasets, SQA and CSI, which enhance the model's ability to process multimodal inputs by including speech instructions alongside visual data.

weaknesses:
1. The necessity of incorporating raw speech, given the availability of video and language (transcripts), remains unclear. Many tasks focus on ownership, and contemporary speech recognition models are capable of discerning individual identities through voice characteristics. Integrating recognized user identity and vocal nuances as metadata in text input might achieve similar effects in current vision-language models (VLMs).

2. While speech transcripts alone may lack certain nuances, the video input could compensate for these through visual cues like gestures. The paper’s limited treatment of emotion in its framework is another concern, especially as language models like ChatGPT already offer emotion detection from text ("Emotion detection GPT"), which could outperform humans [8].

3. Overall, the additional complexity introduced by adding raw speech input to video and transcript data may not significantly enhance model performance in a meaningful way.

4. The paper might benefit from using more recent multimodal VLMs, such as MA-LMM [1], Chat-UniVi [2], ST-LLM [3], LLaVA-NeXT-Video [4], VideoChat2 [5], VideoLLaMA 2 [6], and Video-LLaVA [7], to contextualize the VLAS model's contributions within the latest advancements.

References:

[1] He, Bo, et al. "Ma-lmm: Memory-augmented large multimodal model for long-term video understanding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Jin, Peng, et al. "Chat-univi: Unified visual representation empowers large language models with image and video understanding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[3] Liu, Ruyang, et al. "St-llm: Large language models are effective temporal learners." European Conference on Computer Vision. Springer, Cham, 2025.

[4] https://huggingface.co/llava-hf/LLaVA-NeXT-Video-7B-hf

[5] Li, KunChang, et al. "Videochat: Chat-centric video understanding." arXiv preprint arXiv:2305.06355 (2023).

[6] Cheng, Zesen, et al. "VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs." arXiv preprint arXiv:2406.07476 (2024).

[7] Lin, Bin, et al. "Video-llava: Learning united visual representation by alignment before projection." arXiv preprint arXiv:2311.10122 (2023).

[8] Elyoseph, Zohar, et al. "ChatGPT outperforms humans in emotional awareness evaluations." Frontiers in Psychology 14 (2023): 1199058.

questions:
"The voice retrieval-augmented generation (RAG) (Zhao et al., 2024) is subsequently proposed": 

If this method was previously developed by Zhao et al. (2024), why is it described as "subsequently proposed" here? Is the paper introducing a new variant or application of this technique, or is it simply implementing the existing RAG approach as described in Zhao et al. (2024)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704872581

thread_type:
official_review

<thread_object>
creator: reviewer_pcso

<content>
summary:
This paper recognizes the problems of losing auxiliary semantics and complicating the system in performing transcription on human speech as robot manipulation command. It proposes to include the speech modality into an end-to-end LVA model, resulting in the VLAS model.

The VLAS model directly process textual and speech instructions with vision and proprioception, and generates the robot action in an auto-regressive way. A VQA dataset and a he CALVIN dataset are augmented with the speech instructions to train the VLAS model, and the latter one is further used to tune the model on robotic manipulation tasks.

The experiments demonstrate the promising results of VLAS, and showcases interesting way of commanding robots through the ownership, preference, and compound tasks.

strengths:
1. The paper is well-written and easy to read. It provides efficient background information and detailed illustration of the proposed method.
2. The speech-driven manipulation task is novel. The design of storing user information within the dataset and generate the actions with RAG is smart, demonstrating interesting use cases concerning user ownership and preference.
3. The real-world demonstration further showcase the capability of the model.

weaknesses:
1. Despite the common issue of inference speed in tuning a large VL model as a VLA model, the authors should still provide an analysis on the time required on using the model, especially the bottleneck (is it the audio processing part?) as well as the possible strategies to boost it.
2. The evaluation on robotic manipulation is not efficient. It should be evaluated whether the policy generalizable to novel scenes (like training on CALVIN ABC and test on D, or in other simulations) or tasks. More importantly, if the model requires further training on novel cases, whether the speech processing modules can be transferred.

questions:
1. I wonder if there are additional use case that leverage more semantics information from the speech, like the emotion?
2. How long historical information can the model handle? If there are multiple users in history and their ownership violates each other, will the model be able to handle this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733311139978

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall response

comment:
We sincerely thank all the reviewers for their reviews and suggestions! We have made every effort to address the mentioned typos, clarifications, and concerns. With the discussion phase drawing to a close, the major improvements have been summarized as follows:

1. A more detailed inference efficiency analysis of VLAS has been provided to illustrate how the potential inference speed challenges are addressed when integrating the speech modality into the vision-language-action (VLA) model.
2. We have thoroughly evaluated our model’s ability to generalize to novel scenes, for example, training on CALVIN ABC and testing on D.
3. Our proposed Voice RAG module is employed either in a coupled or decoupled manner with the speech input to facilitate more comprehensive ablation studies.
4. In addition to TTS-generated audio, we have incorporated real speech instructions to further validate the effectiveness of our model.
5. We have presented some real-world experimental results and a comparison between openVLA and our self-implemented VLA baseline, to address the reviewers’ concerns.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes VLAS, a vision-language-action (VLA) model built upon LLaVA for commanding robot manipulation using natural language speech as the primary input modality. This is in contrast to standard approaches that first convert speech to text using an off-the-shelf speech recognizer. Reasoning directly over speech affords the ability to reason over aspects of the commands that are not natively captured by speech recognizers (e.g., voiceprints, which can be important to understanding personalized commands). VLAS employs a three stage training process that includes audio alignment, speech question answering, and action fine-tuning. In addition to VLAS, the paper introduces two new datasets to facilitate training the model to reason over text, images, speech, and actions. Additionally, they introduce a retrieval-augmented generation (RAG) framework to support the model's ability to handle tasks that require reasoning over individual-specific knowledge. The paper evaluates VLAS on a series of custom robot tasks, with results showing that VLAS outperforms existing methods, including those that rely on separate speech recognition tools, particularly in the context of handling personalized tasks.

The paper was reviewed by four referees, and the evaluation involved a significant amount of discussion between the authors, reviewers, and later the AC. All four reviewers emphasized that the paper is well written and easy to read. There was some disagreement regarding the importance of having a model capable of reasoning directly over raw speech as opposed to using off-the-shelf speech processing models. There was a healthy discussion regarding this point between the authors and Reviewer TJDB, both of whom make valid points. As someone who has worked on language understanding in the context of robots for more than a decade, the AC understands the potential benefits of being able to handle raw speech as the primary input modality. Reviewer TJDB's points that some of the information that is conveyed via speech but not captured by a TTS model, could be added as an auxiliary input alongside the recognized text may be reasonable. However, it is not clear that such a two-stage approach would be appropriate in all settings or equally as performant. Of course, more extensive comparisons to such two-stage approaches would go a long way in supporting the importance of reasoning over raw speech (the comparison to the VLA+RAG baseline helps, but there were concerns about the significance of the performance gap and whether this could be attributed to parameter tuning). Other concerns raised by the reviewers include the exclusive use of TTS-based synthetic speech for the initial evaluation and whether the results would generalize to real speech, the lack of a sufficient evaluation of inference speed/efficiency, the coupling of the RAG pipeline with speech, and the absence of relevant baselines (namely, OpenVLA and a text-based baseline). The authors clearly put considerable effort into trying to address these concerns during the rebuttal period, which included performing new experiments involving VLAS' inference efficiency, the model's generalizability given novel scenes, it's performance given real speech, real-world results, and a comparison to OpenVLA.

The AC believes that the paper very much benefited from the discussion period. Updating the paper to incorporate this discussion as well as the additional experiments would go a long way in clarifying the significance of its contributions.


Note: Reviewer PCso is very new to research and has the least experience among the four reviewers. The AC took this into account when making their recommendation.