VLAS: Vision-Language-Action Model with Speech Instructions for Customized Robot Manipulation
Authors: Wei Zhao
Pengxiang Ding
Zhang Min
Zhefei Gong
Shuanghao Bai
Han Zhao
Donglin Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Vision-Language-Action Model
Speech Instructions
Robot Manipulation
Primary area: applications to robotics, autonomy, planning

Abstract: Vision-language-action models (VLAs) have recently become highly prevalent in robot manipulation due to its end-to-end architecture and impressive performance. However, current VLAs are limited to processing human instructions in textual form, neglecting the more natural speech modality for human interaction. A typical approach of incorporating speech modality into VLA necessitates a separate speech recognition system to transcribe spoken instructions into text. Such a cascading pipeline raises two major concerns for robotic systems. First, the entire model grows in size and complexity, potentially resulting in redundant computations and increased memory consumption. Second, the transcription procedure would lose non-semantic information in the raw speech, such as voiceprint, which is crucial for a robot to successfully understand and complete customized tasks. To this end, we propose VLAS, the fisrt end-to-end policy model that seamlessly integrates speech modality for robot manipulation. We present a three-stage speech instruction tuning strategy leveraging multimodal datasets, including our manually curated SQA and CSI datasets. Furthermore, to facilitate personalized operations, we develop a voice retrieval-augmented generation (RAG) approach to enhance the robot's performance in tasks requiring individual-specific knowledge. Experimental results show that the proposed VLAS, following either textual or speech instructions, can achieve performance comparable to traditional VLAs on the CALVIN benchmark. In addition, we created a benchmark consisting of customization tasks, where our VLAS demonstrates absolute superiority by fully leveraging the auxiliary information in speech.

Published as a conference paper at ICLR 2025
VLAS: VISION-LANGUAGE-ACTION MODEL WITH SPEECH INSTRUCTIONS FOR CUSTOMIZED ROBOT MANIPULATION
Wei Zhao' Pengxiang Ding
Min Zhang!
Zhefei Gongb
Shuanghao Bai
Han Zhao
Donglin Wang
Westlake University 2Zhejiang University
X an Jiaotong University
ABSTRACT
Vision-lagguage-actionn models (VLAs) have become increasingly popular robot manipulation for their end-to-end design and remarkable performance. However, existing VLAs rely heavily on vision-language models (VLMs) that only support text-based instructions, neglecting the more natural speech modality for human-robot interaction. Traditional speech integration methods usually in- volves separate speech recognition system, which complicates the model and in- troduces error propagation. Moreover, the transcription procedure would lose non- semantic information in the raw speech, such voiceprint, which may be crucial for robots to successfully complete customized tasks. To overcome above chal- lenges, we propose VLAS,, novel end-to-end VLA that integrates speech recog- nition directly into the robot policy model. VLAS allows the robot to understand spoken commands through inner speech-text alignment and produces correspond- ing actions to fulfill the task. We also present two new datasets, SQA and CSI, to support a three-stage stage tuning process for speech instructions, which empowers VLAS with the ability of multimodal interaction across text, image, speech, and robot actions. Taking step further, voice aarreeall- generation (RAG) paradigm is designed to enable our model to effectively handle tasks that require individual-specifie knowledge. Our extensive experiments show that VLASS can effectively accomplish robot manipulation tasks with diverse speech commands, offering a seamless and customized interaction experience.
INTRODUCTION
With the advent of large vision-language models (VLMs) and the availability of extensive robotic datasets, vision- vision-language-actiio models (VL.As) (Brohan et al. 2022 2023 Kim et al.) 2024) have become a promising approach for learning policies in robotic manipulation. These models demon- strate enhanced generalization to novel objects and semantically diverse instructions, as well as range of emergent capabilities. VLAs, such RT-2 (Brohan et al. 2023), which are fine-tuned from foundation VLMs like PaLM-E (Driess et al., 2023) using robotic trajectory data, can take hu- man instructions and visual observations as inputs to generate robot actions. However, these models primarily focus on textual and visual modalities, leaving the speech modality largely unexplored.
Imagining a scenario where robots provide daily assistance in home care, it is crucial to acknowledge that individuals may exhibit significant variations in physical abilities and subjective preferences. To improve the user experience, robots need to be more accessible and customizable. Speech serves as an ideal modality for achieving this goal, enabling natural and intuitive communication. Given these practical needs and existing technologies, a key question arises: How can we integrate vision- language-acnn action models with speech modality to produce a simpler and better end-user experience?
Based on the above analysis, we propose guiding robot's behavior through speech rather than text. A typical approach involves leveraging an external automatic speech recognition (ASR) sys- tem RRafford et al. 2023 Yu et al.] 2023) to transcribe speech into text for downstream tasks.
"Corresponding author; wangdong1 i n@weat lake. edu.cn

---

Published as a conference paper at ICLR 2025
However, this method presents two significant issues: Firstly, such cascading pipeline leads to larger and more complex robotic system, potentially expanding computational demands and memory consumption. Secondly, the transcription process may lose auxiliary information beyond semanties, such as identity, emotion, and intonation, which are vital for the robot's comprehension of human intent. Many everyday human instructions unstructured and can only be accurately understood with the support of above auxiliary information from speech. For instance, as illustrated in Figure [] (a), when given the task "Please pick up my cup". traditional VLA with text instructions incorporating an ASR system may fail to select the correct cup. Therefore, developing policy model that utilizes raw speech for voice recognition can greatly improve task execution.
VLA
To alleviate these two problems, we present VLASS, an innovative end-to- end policy model that seamlessly inte- grates speech modality for robot ma- nipulation. Notably, VLAS is capa- directly processing both tex- tual and speech instructions alongside visual observations. VLAS built upon the widely adopted open-source vision-language model, LLaVA (Liu al. 2023). and is developed through three distinct training phases. Firstly, employ an established encoder process speech for hidden represen- tations. The multi-layer perceptrons (MLPs) are fine-tuned to transform these representations into the unified language space LLaVA. Secondly, we fine-tune the LLaVA model and above MLPs together with multimodal datasets, including curated Speech Question Answering (SQA) dataset and Visual Question Answering (VQA) datasets. The resulting model, termed VLASS- Base, structions. Finally, we further fine-tune VLAS-Base through behavior cloning (Ross et al., 2011) curated CSI dataset, which encompasses image observations, speech instructions, and robot manipulation trajectories. The voice retrieval-augmennee generation (RAG) is subsequently pro- posed to enable VLAS to perform personalized operations based on individual-specifie knowledge. Experimental results show that the proposed VLAS, following either textual or speech instructions, can achieve performance comparable traditional VLAs on the CALVIN benchmark. In addition, we created benchmark consisting of customization tasks, where our VLAS demonstrates improved performance by fully leveraging the auxiliary information in speech.
don't know which cup you want. Fallure
This voice. He green cup, pick it!
Siccesss
Textual instruction: "Please pick up my cup.
Speech instruction: "Please pick up my cup.
(a)
(b)
Figure I: For personalization tasks, (a) previous VLAs with text instructions fail, while (b) our VLAS with speech in- structions could successfully address them.
effectively generate responses to both text-image and speech-image in-
To sum up, the main contributions of this work are listed as follows: 1) We propose VLAS, the first Iision-language-acttoo model that integrates speech for robot manipulation without needing external speech recognition systems, enabling more natural communication with robots. 2) A Voice RAG paradigm is designed to enable VLAS to effectively address customized tasks that require setiiidull-spcifff knowledge. 3) Besides the robot policy model, introduce ILAS-Base. which extends the widely used vision- -aiiuaee model LLa VA accept speech instructions. This model is also valuable for other downstream tasks involving speech inputs. We also present new datasets, SQA and CSI for community further study. The model, data and code will be publicly available at https: /ttp://ggtttbbbt conc whi wwtp:/wwhhhhhhhhhgggeeeeecocceee
2
RELATED WORK
Languaee Model Large language models (LLMs),, such as FLAN-PPLMM (Chung et al.
2022 InstructGPT (Ouyang et al. 2022). LLaMA (Touvron et al.) (223). and Mamba (Gu & Dao, trained on web-scale
datasets, have demonstrated exceptional effective- ness in performing few-shot and shot natural language processing tasks. This approach has also been rapidly adopted in the field of computer vision. Building on these pretrained LLMss researchers have developed various dision-langusa models (VLMs), including OpenFlamingo (Awadalla al. 2023). BLIP-2 et al. 2023), LLMMA-Adapt (Zhang et al. 2024). IDEFICS (Laurençon et al. Prismatic (Karamcheti al. 2024). LLaVA (Liu al. 2023 and Cobra (Zhao al. 2025
2024
cnarructionfolllowiii
2023
2

---

Published as a conference paper at ICLR 2025
capable of processing inputs from both text and image modalities simultaneously. Many VLMs tai- lored for video modalities have also emerged. such as VideoLLaMA (Zhang et al. 2023). V
LaMA 2 (Cheng et al., 2024). PiTe (Liu et al. 2024), Vidco-LLaVA (Lin et al. 2024), and LLaVA- NeXTI-Interlave (Li et al. 2024a).
It is worth mentioning that the VLMs discussed in this work refer to models that work in a question- answering format, as opposed to models like CLIP (Radford et al.] 2021) and BLIP (Li et al. 2022). which are specifically designed to learn joint representations of linguistic and visual information.
Among the prevalent VLMs, LLaVA stands out as a significant milestone due to its full accessibility, reproducibility, and outstanding performance. The key to LLaVA's success lies in its two-stage visual instruction tuning and the utilization of a carefully curated image-text pair dataset. In the first training stage, I.L.VA VA fine-tunes a multilayer perceptron (MLP) on the image-captioning task, aiming to map the output tokens from the image encoder into the language embedding space. In the second training stage, all network components, except for the pre-trained image encoder, are updated to optimize the model's instruction- question answering (VQA), LLaVA lacks support for instructions in the form of speech. Many studies have also explored the direct integration of audio information processing into multimodal LLMs, such as ImageBind-LLMM (Han et al.) (203) and Unified-IO 2 (Lu et al., 2024). However, there were fewer VLMM capable of supporting raw speech understanding until the recent introduction of GPT-4o (OpenAl 2024), Gemini (Team et al. (024) and VITA (Fu et al. 2024).
capabilities. Despite its strong performance in visual
Vision-laaguaag--cctionn Model A growing body of research has focused on applying VLMs in robotics, aiming to transfer general intelligence from software applications to the physical world. Specifically, two primary approaches have emerged for utilizing vision-language foundation models in the field of robot manipulation. One category of methods employs these foundation models only for high-level task planning. such as PaLM-E (Driess et al. 2023 SayCan (Ahn et al., 2022 and Code as Policies (Liang et al., 2023). In such studies, robots are typically equipped with pre- trained primitive skills, while the VL.M responsible for organizing these low-level skills to accomplish the target task. The other approach, exemplified by models such as RT-2 (Brohan et al.) 2023), Roboflamingo (Li et al. 2024b), and OpenVLA (Kim et al. 2024). seeks to generate robot actions directly by fine-tuning the VLM with robot manipulation data. These models are commonly re- ferred to as vision- aaaioon-nnnuaggaettiiio
(VLA) models (Ding et al.] 2024 Tong et al. 2024 Yue et al.) 2024 Zhang et al.] 2025) However, current VLA models typically focus processing only two input modalities: textual instructions and visual observations (Belkhale et al. 2024). Some studies have also explored integrating additional input modalities, such as hapties and depth information, to further enhance model performance (Cai et al. 2024) Zhhen et al. 2024). Although MUTEX (Shah) ctt al., 2023) provides a unified policy for multimodal task specifications, it does not fully leverage the capabilities of recent vision-language models.
Nevertheless, few studies have investigated how speech modality inputs could be incorporated into VLA models. The most common approach to enabling speech input is to convert speech to text using an external speech recognition tool. However, this approach is not only complex but also results in the loss of auxiliary information present in the speech. To that end, an increasing body of research has recently started to explore the direct integration of speech into large language models in an end- to-end manner (Fu et al. 2024). Thus, our work takes step further by developing a VLA model that supports speech instructions, showcasing how speech modality input enhances performance in scenarios where personalized knowledge is required.
3 METHOD
We present VLAS, a VLA model directly supporting speech instructions for robot manipulation. As illustrated in Figure 2 we first provide an overview of the VLAS architecture (Section 3.I.. Section 3.2 introduces the curated SQA and CSI datasets, which are employed to train the VLAS model. Finally, in Section 3.3. we detail the training paradigm for speech instruction tuning.
3.1 ARCHITECTURE OF VLAS
Overall Framework VLAS takes human speech instructions s and visual observations O as input to directly generate robot actions The input image and speech instruction represented by frequency
3

---

Published as a conference paper at ICLR 2025
VLAS
"lift my block"
Speaker Identification
Fóice 2 Knowledge
Vision Fncoddr
Speech + Encoder
blue block. things drowers.
Voice RAG
Yole?
Personal Knowlaalee
MLP
MLP
Tokenizer
own the redi one
oown the blue one
LLaMA
AT [-0.9,033,00 0.11
De-Tokenize
Deploy
133 18S 113
AGrigper
Figure 2: Overall Framework of VLAS. VLAS encodes visual and speech inputs via encoders and MLP layers obtain respective embeddings. The Voice RAG module retrieves personalized knowledge based on speaker identification and converts it into embeddings using a text tokenizer. All embeddings are then processed by LLaMA generate action tokens, which are subsequently detokenized into continuous values to control the robot's movements.
domain features are each converted into a sequence of embedding tokens through their correspond- ing encoders. During the inference phase, the output RAG(s) of the voice retrieval-augmentee generation module is also tokenized into sequence of embedding tokens. Both visual and speech tokens are transferred separate MLPs map them into the same language space. Subsequently, all the embedding tokens are concatenated as input to the LLM backbone. Formally:
Emb(s, O) concat(MLP, (Emb,,ss) Tok, (RAG(s)) MLP, MLP.,EEb,, (Emb,(O)))
where Emb,, and Emb, denotes speech and vision encoder, respectively; MLP, and MLP, means corresponding projector; Tok; is the text tokenizer. This concatenated embedding is then fed into the LLM backbone to produce the predicted actions in an autoregressive manner as:
N
p (a Emb(s, 0))
II (a. Emb(s, O), azi)
where N denotes the number of dimensions for a single step action, and a is the discretized action tokens, which require a detokenizer be converted into continuous values.
Network Backbone VLAS is built upon the vision-language model LLaVA, as illustrated in Fig- ure In addition the LLaMA LLM backbone, the key components of VA are the Vision Transformer (VIT) (Dosovitskiy et al. (021). which converts input image patches into sequence of embedding tokens, and MLP's that map these tokens the same semantic space as the LLM. When the vision tokens and text tokens are fed in together, the LLM can correlate these inputs and generate a corresponding response. particular, we use the CLIP (Radford al. (02T) model as the visual encoder and Vicuna (Chiang et al.) 2023), fine-tuned variant of LLaMA, the foundation model.
Speech Encoder To equip our model with the ability to process speech modality input, we employ the Whisper (Radford et al. 2023) encoder Emb, convert speech instruction into sequence of hidden states Emb, (s), similar to the visual tokens. Before being fed into the Whisper encoder, the speech signal is first transformed into 80-bin -peettrcro using short-time Fourier transform (STFT) and then padded to a fixed length 3000 frames. The speech encoder processes this mel- spectrogram and produces sequence of 1500 hidden representations. Given that long sequence of
4

---

Published as a conference paper at ICLR 2025
SQA
Is the hus driving down the cut doing image?
M ESPnet
What the color of the tollet in ..??
TTS model trained on LibriTS
I8SK Image- Pairs over 1152 Difforent Voices
I8SK Image- Text Pairs
CSI
slide the pink block to right. grasp the bluee block the drawer the door all the the right.
ESPett
TTS model trained on LibrITT
23K Episodes
23K Episodes with 194K Audio Instractions over 500 Different Voices
with 389 Text Instructions
Figure 3: Data collection process for the SQA and CSI datasets.
speech tokens may impose a significant computational burden when directly input into the LLM, we apply a simple reshape operation along the time dimension, using a reduction factor of An MLP is used to project the speech tokens into the semantic space shared with the text and vision tokens.
Generation (RAG) (Zhao al. 2024) is highly effective method for equipping large language models with the capability to efficiently process dynamic and up-to- date information. Human-spoken instructions frequently exhibit informality and lack of structure, resulting in inadequate semantic content for task completion. To address this issue, propose novel Voice RAG framework to bolster model performance on tasks that require extensive personal knowledge. The Voice RAG module allows model to access additional customized knowledge beyond the original instruction content. As illustrated in Figure 2 the raw speech command is processed by the speaker identification module to extract voiceprint. This voiceprint serves as a key to query an external database, retrieving relevant information. The retrieved data is then integrated background knowledge and passed the LLM, in conjunction with visual and speech tokens. To streamline this process, we utilize pre-trained voiceprint extraction module, avoiding the need for com--scatch training. The integration of the Voice RAG significantly enhances the model's 's ability to comprehend and execute complex spoken commands by supplementing additional contextual information.
Voice RAG Retrieval-Augment
Action Tokenization We discretize continuous action value into 256 uniformly spaced bins and represent them as integer indices. Specifically, we reutilize the 256 least frequently used tokens in the LLM vocabulary to serve action tokens. Then, the robot action tokens across all motion dimensions can be concatenated with a space character to form a textual string, which serves as training label. Consequently, a 7-dimensional action value is formatted as:
Ir, y, 2, d, 0, g].
where y, represent the Cartesian coordinates of the end effector's position, Φ, 0, 1/ denote the rotation angles of the end effector along each axis, and is the gripper state.
3.2 DATA COLLECTION FOR VLAS
Since traditional datasets used for fine-tuning VI.M VIAA models do not include speech instruc- tions, constructed two new datasets train proposed model.
Speech Question Answering (SQA) Dataset The original visual instruction tuning dataset used for LLaVA contains extensive image-text question answering pairs, covering conversations, detailed descriptions, and complex reasoning tasks. Among the three aforementioned task types, the con- versation subset follows multi-turn format, whereas the others single-turn. To construct the SQA dataset, we randomly sampled one round of dialogue from the multi-turn conversation subset and converted the textual questions into corresponding speech shown in Figure3 These speech
5

---

Published as a conference paper at ICLR 2025
+ child sitting?
Answer: arms
Answer: tridge
Answer: [NT, AR, AGripper
Vision
Speech Encoder
Vision Encoder
Speech Encoder
Vision Encoder
Speech Encoder
Encoder
MLP
MLP
MLP
MLP
MLP
MLP
Large Language Model
Large
Language Model
Large
Language Model
Stage I: Speech Aligament
Stage II:
Speech Question
Stage III: Robot Manipulation
Answering
Fine-tuning
Figure 4: Training paradigm VLAS. The training process of VLAS divided into three stages. Stage I: Speech Alignment, where the model aligns speech with text through MLP fine-tuning. Stage II: Speech Question Answering. where the model is trained on both speech and visual question answering tasks facilitate comprehension of multimodal inputs. Stage III: Robot Manipulation Fine-tuning, where the model is further fine-tuned to execute robot manipulation tasks using both speech and text instructions.
instructions, paired with associated images and textual answers, form the SQA dataset. We used the text-to-speech (TTS) tool ESPnet Hayashi et al. (2020) to generate the speech, specifically employ- ing the pre-trained VITS TTS model Kim al. (2021) trained on the LibriTTS dataset Zen et al. (209),, which supports over 2,000 distinct voices. During the conversion of textual questions to speech, the speaker's voice randomly selected. In total, 185K SQA samples were generated, featuring over 1,152 different voices.
CALVIN with Speech Instructions (CSI) Dataset Given that conventional robot manipulation datasets contain only textual task instructions, we utilized the aforementioned TTS model to gen- the corresponding speech instructions. For the CALVIN dataset, which contains 389 textual instructions, we employed 500 different voices to convert each instruction into speech, resulting in approximately 194K audio samples. In the training process, the raw robot manipulation datasets are structured as pairs of (Immage, Instruction Action:). To enable the robot policy model to support both text and speech instructions, we randomly replaced half of the training samples with the synthesized speech instructions.
3.3 TRAINING PARADIGM OF VLAS
The training process of VLAS consists of three stages, as depicted in Figure 4 The details of each stage are outlined below.
Stage I: Speech Alignment The first stage focuses on coarse-grained modality alignment between speech and text, achieved by fine-tuning the model using the LibriSpeech-360 speech recognition dataset Panayotov et al. (2015). During this phase, only the MLP layer between the speech encoder and the LLM backbone updated fulfill speech recognition tasks. is worth mentioning that the speaker identification module for voiceprint extraction can be co-trained during this stage. However, this is optional, as we may directly employ a pre-trained speaker identification model.
Stage II: Speech Question Answering Fine-tuning The second stage focuses on further enhancing the model's capability to process information from multiple input modalities. At this stage, the model fine-tuned using both our curated speech question answering (SQA) dataset and the original visual question answering (VQA) datasets from LLaVA, as well as the LibriSpeech- 100 speech recognition dataset Panayotov et al. (2015). Throughout this phase, all network components the exception of the pre-trained image and speech encoders. Notably, after this stage, we obtain the foundation model, referred VLAAS-asse for the subsequent robot manipulation task. The
updated, with

---

Published as a conference paper at ICLR 2025
VLAS-Bas Base model can also serve valuable resource for the research community in advancing studies on multimodal large language models.
Stage III: Robot Manipulation Fine-tuning In the final training stage. the model is fine-tuned on the CSI robot manipulation dataset in a manner similar to that of stage 2. Each sample this dataset contains complete motion trajectory, represented
sequence of robot actions, along with visual observations from two distinct views and corresponding human instructions in either textual or speech form. For simplicity, the two images each time step are concatenated together.
EXPERIMENTS AND RESULTS
In this section, we conduct a series experiments to assess the effectiveness of the proposed method from multiple perspectives. Section[4.]] first provides quantitative evaluation of the performance of our VLAS model on the CALVIN benchmark. In Section 4.2 we then constructed a new benchmark consisting of various customized tasks further assess method. Finally, in Section to verify whether our foundation model for robot manipulation truly understands speech instructions with- compromising LA's original performance, we evaluate VLAS-Base on general multimodal benchmarks, as well as two benchmarks for speech understanding.
4.1 ROBOT MANIPULATION WITH SPEECH INSTRUCTIONS
To quantitatively assess the performance of our proposed model for robot manipulation tasks, we conduct experiments on the CALVIN benchmark, which comprises 1,000 long-horizon tasks. Each long-horizon task consists of sequence of five successive sub-tasks, accompanied by correspond- ing human command. We trained a traditional VLA model with the same configurations by directly fine-tuning the LLaVA backbone, without support for speech instructions, as the baseline.
Table I: Performance of different robot policy models on the CALVIN benchmark. *: Evaluated with the ground truth textual instructions. ": Evaluated with the speech instructions. On this bench- mark, the Voice RAG module not utilized by VLAS to acquire any customized knowledge.
Models
Splits
LH-1
LH-2
LH-3
LH-4
LH-5
Len
MCIL*
ABCD/D
37.3%
2.7%
0.2%
0.0%
0.0%
0.40
HULC
ABCD/D
89.2%
70.1%
54.8%
42.0%
33.5%
2.90
RT-1*
ABCD/D
84.4%
61.7%
43.8%
32.3%
22.7%
2.45
VLA+
ABCD/D
95.5%
85.0%
74.9%
66.8%
58.2%
3.80
VLAS*
ABCD/D
94.5%
84.4%
73.6%
64.6%
56.6%
3.74
Roboflamingo +ASR
ABCD/D
89.8%
78.6%
68.2%
56.5%
48.3%
3.41
VLA VLA**ASR +ASR
ABCD/D
88.7%
74.1%
61.0%
49.2%
40.2%
3.13
VLAS*
ABCD/D
94.2%
84.0%
73.2%
64.3% %
54.6%
3.70
VLAS* (Rea))
ABCD/D
93.6%
82.8%
71.6%
61.4%
51.3%
3.61
As shown in Table I VLAS, with either textual speech instructions, significantly outper- forms the official MCIL Lynch* & Sermanet* (2021) model and other prevalent models such as HULC al. (022)) and RT-1 Brohan al. (2022). Specifically, VLAS with textual instruc- tions also achieves performance comparable to the baseline VLA model. Moreover, our VLAS compared for speech modality input with the baseline VLA model and another powerful VLA model. Roboflamingo, both similarly derived from the VLM. Since traditional robot policy models do not directly support speech instructions, we employ an external ASR model to transcribe the speech instructions into text. The most powerful ASR model, Whisper large-v2, released by Ope- nAI used in the experiments. To generate the speech instructions for evaluation, the previously discussed TTS model is employed with 39 novel voices not included in the SQA and CSI datasets. For each instruction, corresponding voice is randomly sampled. In addition, we have included real speech instructions recorded from 10 individuals for evaluation. As can be observed, even with real speech instructions, our VLAS still achieves strong performance, only slightly behind the VLA baseline with a gap of 0.19.
7

---

Published as a conference paper at ICLR 2025
We found that VLAS significantly outperforms the other two methods that utilize cascading pipeline for speech understanding. We attribute this to the higher accuracy of our method in recog- nizing speech instructions for robot manipulation, dataset. Conversely, the external ASR model less sensitive to controlling commands for the robot, leading amplified propagation emrors. It is important to highlight method is orthogonal to other VLA models, and thus, can be combined with them to achieve superior performance.
the model has been fine-tuned
specialized
4.2 ROBOT MANIPULAATION FOR CUSTOMIZED TASKS
Table 2:
Performance of three types of customized tasks for robot manipulation. +: Evaluated with the ground truth textual instructions. Evaluated with the speech instructions. On this benchmark, the Voice RAG module is utilized by VLAS to acquire customized knowledge.
Compound-Wy -Multistaags
Models
Ownership
Preference
Compound
Avg.
Stage-1
Stage-2
VLA+
17.9%
30.8%
23.1%
35.9%
5.1 %
19.2%
VLAS*
94.7%
84.6%
100.0%
100.0%
66.7%
86.5 %
VLAS (Real)
89.5%
70.0%
100.0%
90.0%
55.0%
78.6%
VLAS* RRAG
15.4%
12.8%
25.6%
33.3%
10.3%
16.0%
+RA 97.4%
84.6%
97.4%
82. %
48.7%
82.0%
To evaluate our model's capability in executing personalized tasks, we developed new benchmark comprising diverse, unstructured spoken instructions within the simulation environment. All these tasks require the robot to utilize personal knowledge beyond superficial semantic content. Particu- larly, this benchmark includes three task categories: (1) Object Ownership Tasks: The robot must interact with the appropriate objects according to their ownership. When given spoken instruc- tion, the robot needs to identify the person's intention and use the correct object belonging to them. (2) User Preference Tasks: These tasks necessitate the robot comprehend the user's preferences. Given the identical command, the robot is expected to perform different actions depending on the specific user's preferences. (3) Compound Tasks: Tasks in this category require the robot not only to select appropriate objects, but also to perform actions that align with the user's preferences. In particular, this category includes multistage tasks where the robot required to respond to two suc- cessive human instructions. Since the outcome of the previous task can easily impact the execution of the subsequent task, these multistage tasks pose greater challenge. For each task category, there total of 39 unseen voices beyond training datasets.
Table presents a detailed comparison between the VLA baseline and VLAS. Because the VLA baseline relies solely on text instructions and lacks access to background knowledge, its performance severely limited, with an average success rate of below 20%. Such a model can only perform tasks through random attempts by drawing inferences from contextual information. However, VLAS, which directly receives raw speech input, leverages the Voice RAG to access individual-specific knowledge, allowing perform customized operations more effectively. As result, our model demonstrates much better performance on this benchmark, achieving an average success rate of over 86%. Figure and F several concrete case studies showing how the proposed method performs customized operations for different users.
We introduce real speech instructions for evaluation, which also demonstrates acceptable perfor- mance. Ablation studies are conducted to further validate the effectiveness of our proposed Voice RAG module. can be seen from Table that when the RAG module removed, the performance the customized benchmark. Meanwhile, when our RAG module
of VLAS significantly degrades integrated with the VLA, its performance significantly improves. Both the ablation studies above demonstrate the effectiveness of the Voice RAG module.
4.3 EXPERIMENTS WITH A REAL- UR5 ROBOT ARM
We fine-tune our VLAS-Ban Base by utilizing both the Berkeley URS demonstration dataset and our own cup-pick sip-picking dataset. This results in VLAS model that can be deployed real-world robots. As shown in Figure 7 our model can respond to different actions according to the personal information of the speaker, like picking up the specific cup considering the ownership.
8

---

Published as a conference paper at ICLR 2025
Speaker
Ownership
Speaker
Ownership Speaker
Ownership
Meta: have (red; block.
Meta: have (pink) block.
Meta: have (blue) block.
lift my block from the slddnng oahinot.
graup and lift my block.
grosp and lift my block
Speaker 4507
Preference Speaker
Preference
Speaker
Preference
Meta: prefer tright) side.
Meta: prefer (left) side.
Metu: prefer (left) side.
go push the rod block.
go push the ved block
go push red black
Figure 5: Demonstration of object ownership tasks (top row) and preference tasks (bottom row)
for customized robot manipulation.
Speaker id: 7127
Compound Stage
Compound One Stage Meta: have (blue) block, I prefer rotate it (right) side.
Speaker id: 6829
Meta: have (pink) block, prefer rotate it (right) side.
my
my
Speaker id: 1995
Compound Stages Meta: have (pink) block, I prefer store it in [drawer).
Compound Two Slages Meta: have (pink) block. I prefer store it (cabinet).
Speaker id: 260
grusp block store the grasped block.
grasp and block. store the grasped block.
Figure 6: Demonstration of compound tasks for customized robot manipulation.
4.4 ANALYSIS FOR THE VLAS
VLAS-BASE BASE FOUNDATION MODEL
The multimodal understanding capability of the VLAS-Base Base is critical when fine-tum fine-tuning it with robot trajectories to develop the proposed VLAS. Therefore, we quantitatively assess the performance of VLAS-Base from two perspectives. First, the VLASSBase is expected achieve performance comparable to the original LLaVA model, as the ability to comprehend visual and language infor- mation serves as the foundation for intelligent robot manipulation.
Table 3 provides detailed comparison between the VLAS-Base and other prevalent VLMs across general multimodal benchmarks. As can be observed, VLAS-Base obtains nearly the same perfor- mance to LLaVA, while significantly outperforming other VLMs. These results indicate that the introduction of the speech modality does not degrade the performance of the foundation model.
Second, the VLAS-BaL Base model is also expected to have a strong understanding of speech modality input. For this purpose, we conduct experiments on the LibriSpeech automatic speech recognition benchmark and our self-constructed speech question answering benchmark, SGQA. Given the lack of Q&& evaluation benchmarks for image-speech pairs, we converted all textual questions in the GQA benchmark for visual question answering into speech format with external TTS model, resulting in the SGQA benchmark. For the speech recognition benchmark, we employ the state-of- the-art W Whisper large-v2 v2 model the baseline. For the speech question answering benchmark. since prevalent VLMs typically do not support speech input, we use LLaVA and BLIP-2 with ground-truth textual instructions as baselines.

---

Published as a conference paper at ICLR 2025
Pick up my
Speaker ld: 1089
Meta: have a (green) cup
eup
Speaker Id: 6829
Meta: have [red] cup
-...............................
Figure 7: Demonstration of success cases of VLAS on the real-world UR5 robot
Table VLAS-Base achieves comparable performance to Whisper large-v2 on the LibriSpeech test set. Considering that reduction factor is applied to downsample the speech spectrum for VLAS-Buse, its performance could potentially be improved by optimizing this factor or by employ- ing more advanced downsampling module. Moreover, although VLAS-Base falls behind LLaVA with ground-truth textual instructions on the SGQA benchmark, still surpasses BLIP-2.
These results indicate that our foundation model, used for developing VLAS, effectively process diverse speech instructions. We can even utilize co-training with robot trajectories and speech ques- tion answering data to further improve VLAS's capacity to handle more complex human commands.
Table 3: Performance comparison between state-of-the-ar VLMs to VLAS-Ban --Se across diverse
multimodal evaluation benchmarks.
Model
LLM
VQA"
VizWiz
SQA
VQA
POPE
GQA
BLIP-2
Vicuna- 13B
65.0
19.6
61.0
42.5
85.3
41.0
InstructBLIP
Vicuna- 13BB
33.4
63.1
50.7
78.9
49.5
Qwen- VL
Qwen-
78.8
35.2
67.1
63.8
59.3
LLaVA v1.5
Vicuna-77 7B
78.8
50.0
66.8
58.2
85.9
62.0
VLAS- Base
Vicuna-78 7B
78.7
51.1
72.2
58.1
85.5
62.0
Table 4: Performance comparison on LibriSpeech and SGQA benchmark, using word error rate (WER) and accuracy evaluation metrics. L.LVVA and B1.IP-2 employ the ground truth textual insturctions on SGQA.
Model
LibriSpeech (WER)
SGQA
LLaVA v1.5
N/A
62.0
BLIP-2
N/A
41.0
Whisper
2.7%
N/A
VLAS-Base
2.79%
50.8
CONCLUSION
This paper presents an end-to-end VLA model for robot manipulation that is capable of understand- ing speech instructions without relying an external speech recognition system. As the raw speech directly taken as the model's input, auxiliary information in the speech, such as voiceprint, can be fully utilized to more effectively complete the given task. In particular, we introduce Voice RAG method for our model to improve its performance in following spoken instructions that require extensive individual-specine knowledge. Consequently, the integration of speech modality data in VLAS not only simplifies the overall pipeline for robot control but also enables the robot to handle wide range of customized tasks. Our future work may focus on exploring other auxiliary informa- tion human speech or environmental sounds to enable the robot to better understand and complete complex tasks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the National Science and Technology Innovation 2030 - Major Project (Grant No. 20222D0208800), and NSFC General Program (Grant No. 62176215). In addition, we thank Gongsheng Li for his valuable discussions and assistance with data processing.
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang. Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huel Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do i can and not say: Grounding language in robotic affordances. In arXiv preprint arXivv2204.01691 2022.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Korblithh, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308..13300 2023.
Suneel Belkhale, Tianli Ding, Ted Xiao, Pierre Sermanet, Quon Vuong. Jonathan Tompson, Yevgen Chebotar, Debidatta Dwibedi, and Dorsa Sadigh. RT-H: Action Hierarchies Using Language, March 2024. URLhttp: /atr:///axiiv... wwe.iiioor////.. abs 2403. 01823, arXiv:2403.0188 [cs].
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for wolld control at scale. arXiv preprint (txxi::2/2.068//1 2022.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexan- der Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalash- nikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-We Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao. Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu. Tianhe Yu, and Brianna Zitkovich. RI-2: Vision-language-actino models transfer web knowledge to robotic control, 2023. URL detps://arxise www./oggg///ooooooo//ooooo...e// 2200d///233...2 15818
Wenxiao Cai, Iaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, and Bo Zhao. Spatialbot: Precise spatial understanding with vision language models, 2024. URL
https://arxiv. org/ abs/ 2405. 13642
Zesen Cheng, Sicong Leng. Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing, VideoLLaMA 2: Advancing Seatial-Temporal Modeling and Audio Understanding Video-LLMs, October 2024. URL http:://axiv: org/aba/ 2406. 07476 arXiv:2406.07476.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuma: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
mmsys.org//
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu,
11

---

Tomoki Hayashi, Ryuichi Yamamoto, Katsuki Inoue, Takenori Yoshimura, Shinji Watanabe, Tomoki Toda, Kazuya Takeda, Yu Zhang, and Xu Tan. Espnet-TTS: Unified, Reproducible, and In- tegratable Open Source End-to-End Texl-to-Speech Toolkit. In International Conference on Acoustics, Speech and Signal Processing, pp. 7654-7658, Barcelona, Spain, May 2020. doi: 10. 11011///AAAAS4477722000000000000 Published as a conference paper at ICLR 2025
Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pel- lat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling instruction-finetuned language models, 2022. Utt.ttttsss///axii/
abs O 11416
Pengxiang Ding, Han Zhao, Wenjie Zhang, Wenxuan Song. Min Zhang, Siteng Huang, Ningxi Yang, and Donglin Wang. QUAR-VLAA Vision-Language-Acttoon Model for Quadruped Robots. In Computer Vision ECCV 2024: 18th European Conference, Milan, Italy, September 29 October 4, 2024, Proceedings, Part V, pp. 352-367, Berlin, Heidelberg, October 2024. Springer- Verlag. ISBN 978-3-031-7265 doi: 10. 10.11007/77---0311-72000 10. N0.1007/978
21. URL https: /ttpi/oo/.ooo///o/////////////
12211-7265222 21
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URLhtttps.
arxiv. /bw/ 201 11929
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model, 2023. URL/actps://axxiv org/ abs /tbbs22300 3330033337888
Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang. Shaoqi Dong, Xiong Wang, Di Yin, Long Ma, Xiawu Zheng, Ran He, Rongrong Ji, Yunsheng Wu, Caifeng Shan, and Xing Sun. VITA: Towards Open-Source Interactive Omni Multimodal LLM, Septem- ber 2024. http://///axxi wwtp:///.rgg/b//b/aaaaa/.b
05211 arXiv:2408.053 [cs].
Albert Gu and Tri Dao. Mamba: Linear-Tine Sequence Modeling with Selective State Spaces. In First Conference on Language Modeling, August 2024. URL
net/
Forum?Ld=tEYskwi htt.::/tti...l.11.llal...ssssss scusssion
Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, Xudong Lu, Shuai Ren, Yafei Wen, Xiaoxin Chen, Xiangyu Yue, Hongsheng Li, and Yu Qiao. ImageBind-LLMM. Multi-modality Instruction Tuning, September 2023. WRRnttp:: hatt:://arxiv/ www. 2309.
ISSN: 2379-190X.
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic VLMs: Investigating the Design Space of -onalll-Conditionedd Language Models. In Proceedings of the 41st International Conference on Machine Learning, pp. 13123-23144 PMLR, July 2024. URL Intpt::://// ftttp:://proceediine ngs..ggsmm..... ww..mmlr.pp.... press/v235/ .ag..eedd
karamcheti24a. html ISSN: 2640-3498.
Jachyeon Kim, Jungil Kong, and Juhee Son. Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-t--S In Proceedings of the 38th International Conference on Machine Learning, pp. 5530- PMLR, July 2021. httpt/tpppp//ppppceemmme mlr.
press/v139/kinn.
html
ISSN: 2640-3498.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong. Thomas Kollar, Ben- jamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open- vision-language-action model, 2024. URL TRLhttpa https::///// http://aaxvvvooo// labs/ 2406. 09246)
12

---

Published as a conference paper at ICLR 2025
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Docu- ments, August 2023. B.R.http://axxiiv
abs///////////// ablaaaasb/202 22066 6527 arXiv:2306. 1662277...
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. LLaVA-NcXT-Imtercaaee Tackling Multi-image, Video, and 3D in Large Multimodal Models, July 2024a. URL http::ttpa//ii..ooo/o abtp//bb//2////// 07895 arXiv:2407.07895
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image
pre-training for unified vision-language understanding and generation, 2022. URL https:
//arxiv. owg.ivvorg/aaao mwtxii.orgg/bb//22000 absp//as.222.2... 12086
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman pre-
training with frozen image encoders and large language models, 2023. URL https://axxiv
org/abs/22 /2011 12597
Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, Hang Li, and Tao Kong. Vision-language foundation models as effective robot imitators, 2024b. /
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control, 2023. URL https: /apps:///axxvvvi org/ /ttgg//bss.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-LLaVA: Learn- ing United Visual Representation by Alignment Before Projection. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, pp. 5971-5984, Miami, Florida, USA, November 2024. Association for Computational Linguistics. URL
amnnppmmiin..
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. URL
https://arxiv. org/ abs 2304/ 08485
Yang Liu, Pengxiang Ding, Siteng Huang. Min Zhang, Han Zhao, and Donglin Wang. PiTe: Pixel-Temporal Alignment for Large Video-Languaa Model. In Computer Vision ECCV 2024: 18th European Conference, Milan, Italy, September 29-October 2024, Proceed- ings. Part V, pp. 160-176, Berlin, Heidelberg, October 2024. Springer- Verlag. ISBN 978-3- 031-72651-4. doi: 978 3- 026-7265222202 10
URL hitps //coi///oooooo /tt.:////ooo../..../..o org/10, 1007/
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action. In 2024 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 26445, Seattle, WA. USA. June 2024. IEEE. ISBN 9798350353006. doi: D0.1099CCVPR3233322244.2222 leee. aeeee.org//dccmeet//5555
URL https: /ttp:///eeeeeppr....o/
Corey Lynch* and Pierre Sermanet*, Language Conditioned Imitation Learning Over Unstructured Data. In Robotics: Science and Systems XVII. Robotics: Science and Systems Foundation, July 2021. ISBN 978-0-9923747-7-8.
doi: 20.15607/RSS.22221 X URL http:://ww.
Oier Mees, Lukas Hermann, and Wolfram Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters (RA-L), 7(4): 11205-1 1212, 2022.
OpenAl. Gpt-4: Generative pre-trained transformer 4.
he 1o-gpt-40/ 2024.
13

---

Published as a conference paper at ICLR 2025
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. URL https::////... htttps://aaxii....
attgg/bbs/220
02155
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) pp. 2206-5210, April 2015. doi: 10.1109/CCSSPP2200
7178964. URL WtRL:httts////eeeeee www.eeewwep......ee.eee....e..e. ore www.ee/e....oooooeooooeeeeeeeeee /7178964 190X.
ISSN: 2379-
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL
htttps://axiv.. org/ 2ttpp//213..0000
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th Interna- fional Conference Machine Learning, volume 202 of ICML '2, pp. 28492 28518, Honolulu, Hawaii, USA, July 2023. JMLR.org.
Stephane Ross, Geoffrey Gordon, and Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning, 2011. URL
/bs/
1011. 6666
Rutav Shah, Roberto Martín-Martín, and Yuke Zhu. MUTEX: Learning Unified Policies from Multimodal Task Specifications. In Proceedings of The 7th Conference on Robot Learning, pp. 2663 2682. PMLR, December 2023. htLLtttss///ooceeeeete mttp://ssmmmmm..xrx///////2// shah23b html
mlr
ISSN: 2640-3498.
Gemini Team, Rohan Anil, Sebastian Borgeaud, and Alayrac Gemini: A Family of Highly Capable Multimodal Models, June 2024. URL http: www.wwwww..aaaaaaaaaaaa 2312 11805 arXiv:2312.
Xinyang Tong, Pengxiang Ding, Donglin Wang, Wenjie Zhang, Can Cui, Mingyang Sun, Yiguo Fan, Han Zhao, Hongyin Zhang, Yonghao Dang, Siteng Huang, and Shangke Lyu. QUART- Online: Latency-Free Large Multimodal Language Model for Quadruped Robot Learning, De- cember 2024. httl:http://raax/ http://wwwwww..../aaa////a/o abs/ 2212. 22141.57.666... arXiv:2412.15576 [cs].
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. htth:t/:::/ /ttp:://axivvvvve www./bs/////oooooooee/..../...ar 2302 13971
Wenyi Yu, Changli Tang, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Connecting Speech Encoder and Large Language Model for ASR, September 2023.
URL http:://arxiii org/ abs/ 13963 arXiv:2309. 13963 [cs, eess].
Yang Yue, Yulín Wang, Bingyi Kang, Yizeng Han, Shenzhi Wang, Shiji Song, Jiashi Feng, and Gao Huang. Deer-vla: Dynamic inference of multimodal large language models for efficient robot execution. NeurIPS, 2024.
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech. In Interspeech 2019, pp. 1526-1530. ISCA, September 2019. doi: 10224437/ntespeechh2099444
URL https: // www. www.isscaaooo wwbpscaaarcciiee.. wttp://rrgggnnneeeeeeeeeeeee 2ttseeeeeeee___nnrrrereeeeeeeee wtwps_nneeeeehhhtttct.e
2019 /eel99
Hang Zhang, Xin Li, and Lidong Bing. Video-LLaMA: An Instruction-tumed Audio-Visual Lan- guage Model for Video Understanding. In Yansong Feng and Els Lefever (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demon- strations, pp. 543-553 Singapore, December 2023. Association for Computational Linguis-
tics. doi: 10. 18.53863/v 1/202/.222mmm 100003..mllp-dmm..4.. URL httppp////alaaatlloo....oooo 2ttphhhooooyy..//222233.
lemn www.lpp-ee...mm 49)
14

---

Published as a conference paper at ICLR 2025
Hongyin Zhang, Pengxiang Ding, Shangke Lyu, Ying Peng, and Donglin Wang. Gevrm: Goal-expressive video generation model for robust visual manipulation. arXiv preprint arXiv:2502.09268, 2025.
Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient funing language models with zero- init attention, 2024. UtL.tttss://arxttttt
abs 2303. 16 16199.
Han Zhao, Min Zhang, Wei Zhao, Pengxiang Ding, Siteng Huang, and Donglin Wang. Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference, January 2025. URL http:://axixu org/ abs 2ww.s///43.... .1.003114520 14520 arXiv:2403.14520 [cs].
Siyun Zhao, Yuqing Yang, Zilong Wang, Zhiyuan He, Luna K. Qiu, and Lili Qiu. Retrieval aug- mented generation (rag) and beyond: comprehensive survey on how to make your Ilms use external data more wisely, 2024. URL (RRL:ttss///rxiii org/ abs 2409 14924.
Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3D-VLA: A 3D Vision-Language-ACee Generative World Model. In Proceedings of the 41st International Conference Machine Learning, pp. 61229-61245 PMLR, July 2024. URL
mlr. TLL..prees/v235/2ee
html ISSN: 2640-3498.
A TRAINING DETAILS
We perform fine-tuning in Stage I on the train-clean-100 split of the LibriSpeech dataset for 5 epochs, using a learning rate of le-3 and a batch size of 16. Subsequently, the fine-tuning in Stage II is conducted on SQA dataset, along with the released LLaVA 665K instruction-folloviinn dataset and the train-clean-360 split of LibriSpeech, for 1 epoch using learning rate of 2e-5 and batch size of 16. Finally, we fine-tune the model on the CSI robot manipulation dataset for epoch, with a learning rate of 2e-5 and batch size of 16. Specifically, we combined actions from 5 time steps into single training label increase the operating frequency of the robot policy model. The Adam optimizer without weight decay and a cosine learning rate schedule with 3% warmup ratio are used throughout the experiments. Flash Attention 2, BF16, and TF32 are enabled to achieve balance between training speed and precision.
All models are trained using 8x A100 GPUs, except for the fine-tuning Stage We empirically found that employing single GPU for coarse-grained speech alignment yields better performance.
B EXTENDED EXPERIMENTAL RESULTS
B.1 FAILURE CASES OF VLAS AND VLA ON THE CUSTOMIZATION BENCHMARK
Speaker id: 8224 Meta: prefer (right) side.
Speaker ad: 8230 Meta: prefer (right) side.
Speaker id: 8463
Meta: prefer (right) side.
go push
go pusc
go puch
Speaker id: 7176
Compound stage Meta: have (red) block, I prefer store it in (cabinet).
Speaker 7729
Compound 2th stage Meta: have (red) black, I prefer store it in (cubinet).
grasp and block
grasp and Mock. store the grasped block.
2. store the grasped block.
Figure 8: Demonstration of failure cases of VLAS the customization benchmark.
15

---

Published as a conference paper at ICLR 2025
Speaker 7021
Ownership
Speaker
Preference
Speaker Compound ith stage Meta: have (red; block.
Meta: have (pink) block.
Meta: prefer (right) side.
lift my block from the sliding cahinat.
go pash the red blook
greup and lif my block
Speaker
Compound
Speaker Compound stage Metu: prefer in (drawer).
Meta: have (pink) block, I
it [left] side.
take my block and rotate it.
store the grasped bock.
Figure 9: Demonstration of failure cases of VLA the customization benchmark.
We conducted additional analysis on the failure cases of VLAS and VLA on the customization benchmark to better identify the underlying reasons. As observed in the Figure X failure cases of the VLAS model mainly occur in the preference task and the second phase of the compound task. The but fails to execute the actions successfully. We conjecture this issue can be addressed by refining the policy model's architecture and training process. On the contrary, the VLA model exhibits diverse range of сπтоr patterns, as illustrated in Figure [ Since the VLA model has access only to superficial semantic information from human instructions, relies on random attempts complete these personalized tasks, leading to numerous failures.
pattern more consistent, suggesting that the model understands the instructions
B.2 COMPARISON WITH COBOFLAMMIGO ON THE CALVIN BENCHMARK
RoboFlamingo is another prominent VLA model reported on the CALVIN Benchmark. Table provides comparison between VLAS and RoboFlamingo the CALVIN Benchmark using textual instructions. can be seen that VLAS performs slightly behind RoboFlamingo mainly due to lack of historical information when predicting actions. When the historical information, i.e. the LSTM policy head, is removed, the performance of RoboFlamingo significantly deteriorates. Thus, we can leverage similar approaches further enhance the performance of model, these methods are completely orthogonal.
Table 5: Comparison with RoboFlamingo on the CALVIN Benchmark. The performance of RoboFlamingo without historical information is derived from results presented in their original pa- per. Evaluated with the ground truth textual instructions.
Models
Splits
LH-1
LH-2
LH-3
LH-4
LH-5
Len
Roboflamingo*
ABCD/D
96.4%
89.6%
82.4%
74.0%
66.0%
4.09
Roboflamingo Hist)
ABCD/D
~60
~00%
20%
~00
~20%
1.0
VLAS*
ABCD/D
94.5%
84.4%
73.6%
64.6%
56.6%
3.74
B.3 COMPARISON WITH OPENVLA ON THE CALVIN BENCHMARK
We further compare the performance of our model with both a pre-trained Open model and its fine-tuned variant on the CALVIN benchmark. For the pre-trained Open model, we di- rectly utilized the www.lla-bb-inttueeddee
checkpoint, as it has been tailored for the Franka Emika Panda a simulation environment, closely resembling our experimental setup. For the fine- tuned Open VLA model, followed their officially recommended fine-tuning setup. The CALVIN dataset was converted into the unified RLDS format to enable the fine-tuning of OpenVLA as they suggested.
can observed from Table 6 that the pre-trained Open model fails to achieve a zero shot generalization the CALVIN benchmark. In addition, though the fine-tuned OLen model
16

---

Published as a conference paper at ICLR 2025
complete some tasks in the initial stage of the long-horizon sequence, its overall performance remains suboptimal. We hypothesize that the poor performance of OpenVLA due to limitation of supporting only third-person view imput. Visual feedback from the robot's end effector is of vital importance for facilitating precise manipulation tasks, such as object grasping and rotation. However, this functionality is absent in the current OpenVL..
Table 6: Comparison with OpenVL.A on the CALVIN Benchmark. +: Evaluated with the ground
truth textual instructions.
Models
Splits
LH-1
LH-2
LH-3
LH-4
LH-5
Len
Pre-trained Open VLA+ Fine-tuned VLe
ABCD/D
0.0%
0.0%
0.0%
0.0%
0.0%
ABCD/D 30.4%
2.6%
0.4%
0.1%
0.0%
0.34
VLAS*
ABCD/D
94.5%
84.4%
73.6%
64.6%
56.6%
3.74
B.4 EXPERIMENTAL EVALUATION ON THE CALVIN BENCHMARK USING ABC/D SPLITS
To better evaluate model's generalization capability to novel scenes, conducted experiments in which the model was trained on ABC splits and tested on the D split. It can be observed that, despite all models experiencing performance degradation due the domain gap, VLAS achieved performance comparable to RoboFlamingo while outperforming the other models.
Table 7: Performance different robot policy models on the CALVIN benchmark. *: Evaluated with the ground truth textual instructions. ": Evaluated with the speech instructions. On this bench- mark, the Voice RAG module not utilized by VLAS to acquire any customized knowledge.
Models MCIL*
Splits
LH-1
LH-2
LH-3
LH-4
LH-5
Len
ABC/D
30.4%
1.3%
0.2%
0.0%
0.0%
0.31
HULC*
ABC/D
41.8%
16.5%
5.7%
1.9%
1.1%
0.67
RT-1*
ABC/D
53.3%
22.2%
9.4%
3,8%
1.3%
0.9
VLA*
ABC/D
83.1%
58.4%
34.7%
23.1%
15.1%
2.14
Roboflamingo*
ABC/D
82.4%
61.9%
46.6%
33.1%
23.5%
2.48
VLAS*
ABC/D
85.9%
59.2%
38.5%
25.9%
17.6%
2.27
VLA VLA'+ASR +ASR
ABC/D
74.7%
54.1% %
38.4%
24.1%
16.5%
2.04
VLASS
ABC/D
87.2 %
64.2% %
40.9 %
28.1 %
19.6%
2.40
Moreover, conducted similar experiments our personalization benchmark. The results demon- strate that our model is capable of handling novel scenes.
Table Performance of three types of customized tasks for robot manipulation. *: Evaluated with the ground truth textual instructions. ": Evaluated with the speech instructions. On this benchmark, Voice RAG module utilized by VLAS to acquire customized knowledge.
Compound-Multistnn
Models
Ownership
Preference
Compound
Avg.
Stage-1
Stage-2
VLA*
20.5%
5.1%
0.0%
10.3%
0.0%
6.4%
VLAS*
64.1% %
61.5%
87.2%
74.4%
7.7%
55.1%
VLAS RAG
15.4%
23.1%
0.0%
12.8%
0.0%
9.6%
VLA+-RAM
82.1% %
71.8%
84.6%
82.1%
10.3%
62.2%
B.5
INFERENCE EFFICIENCY ANALYSIS
This paper employs two key optimizations to enhance the inference speed of VLAS: downsampling the speech spectrogram and implementing an action update strategy with multi-step prediction and execution. Speech spectrogram downsampling processing, where adjacent x-frame spectrograms are aggregated into a single-frame feature through reshaping operation, effectively reducing the time dimension length. In experiments, used
widely used strategy accelerate speech signal
17

---

Published as a conference paper at ICLR 2025
x: Since the effectiveness of this approach has been validated in numerous speech recognition and generation tasks, we did not perform additional related analyses. Given that the state of the environment typically does not change significantly effective multi-step prediction and execution policy. Specifically, we set the number of steps for both when r=5, both the VLA and VLAS models achieve
short period, our work adopts simple yet
VLA and VLAS to r=5. As shown in Table ! significant speedups while also demonstrating improved performance on the CALVIN benchmark.
Table 9: Inference efficiency of different models and their average performance on the CALVIN benchmark. *: Evaluated with the ground truth textual instructions. Evaluated with the speech instructions.
Models VLA*(r=I)
Actions See (Hz)
Len
1.89
2.30
VLAS (1=)
1.17
2.02
VLA*(r=5)
3.60
3.80
VL=S((==5)
2.50
3.70
Table 10: Inference efficiency of different models and their average performance on the CALVIN benchmark. * Evaluated with the ground truth textual instructions. Evaluated with the speech instructions.
Models
Actions / Sec (Hz)
Len
VLAS (r=l)
1.17
2.02
VLAS (r=5)
2.50
3.70
VLAS VLAS(((=12) (r=12)
2.88
3.35
VLAS VL=S"((=20)
3.80
0.70
We supplemented our results with an analysis of the inference speed and performance VLAS across different values of Table 10 indicates that r=5 achieves optimal balance between infer- ence efficiency and manipulation performance.
18


Reviews:
[{"review_id": "2xvXqiddeI", "reviewer": "Reviewer_PCso", "summary": "This paper recognizes the problems of losing auxiliary semantics and complicating the system in performing transcription on human speech as robot manipulation command. It proposes to include the speech modality into an end-to-end LVA model, resulting in the VLAS model.\n\nThe VLAS model directly process textual and speech instructions with vision and proprioception, and generates the robot action in an auto-regressive way. A VQA dataset and a he CALVIN dataset are augmented with the speech instructions to train the VLAS model, and the latter one is further used to tune the model on robotic manipulation tasks.\n\nThe experiments demonstrate the promising results of VLAS, and showcases interesting way of commanding robots through the ownership, preference, and compound tasks.", "questions": "1. I wonder if there are additional use case that leverage more semantics information from the speech, like the emotion?\n2. How long historical information can the model handle? If there are multiple users in history and their ownership violates each other, will the model be able to handle this?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to read. It provides efficient background information and detailed illustration of the proposed method.\n2. The speech-driven manipulation task is novel. The design of storing user information within the dataset and generate the actions with RAG is smart, demonstrating interesting use cases concerning user ownership and preference.\n3. The real-world demonstration further showcase the capability of the model.", "weaknesses": "1. Despite the common issue of inference speed in tuning a large VL model as a VLA model, the authors should still provide an analysis on the time required on using the model, especially the bottleneck (is it the audio processing part?) as well as the possible strategies to boost it.\n2. The evaluation on robotic manipulation is not efficient. It should be evaluated whether the policy generalizable to novel scenes (like training on CALVIN ABC and test on D, or in other simulations) or tasks. More importantly, if the model requires further training on novel cases, whether the speech processing modules can be transferred.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RgRLihH6sf", "reviewer": "Reviewer_TJDB", "summary": "The paper introduces VLAS, a novel vision-language-action model designed to facilitate robot manipulation tasks by integrating speech as a primary instruction modality, bypassing traditional speech-to-text conversion methods. This approach enables VLAS to capture essential non-verbal cues, such as voiceprints, which assist in executing personalized tasks based on individual user needs. Built on the LLaVA model, VLAS employs three training stages to process multimodal inputs across text, image, and speech, aligning these representations for seamless action generation. The authors also developed two datasets, SQA and CSI, to enhance the model’s handling of speech instructions, alongside a voice retrieval-augmented generation (RAG) system to access individualized knowledge. Experimental results demonstrate VLAS’s superior performance on complex, customized robot tasks, particularly when handling personalized commands, outperforming other models that rely on separate speech recognition systems.", "questions": "\"The voice retrieval-augmented generation (RAG) (Zhao et al., 2024) is subsequently proposed\": \n\nIf this method was previously developed by Zhao et al. (2024), why is it described as \"subsequently proposed\" here? Is the paper introducing a new variant or application of this technique, or is it simply implementing the existing RAG approach as described in Zhao et al. (2024)?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 1, "strengths": "1. The paper introduces VLAS, a vision-language-action model evaluated on a comprehensive set of tasks, including customized tasks that leverage speech input for personalized robot manipulation. \n\n2. Additionally, the authors contribute two new datasets, SQA and CSI, which enhance the model's ability to process multimodal inputs by including speech instructions alongside visual data.", "weaknesses": "1. The necessity of incorporating raw speech, given the availability of video and language (transcripts), remains unclear. Many tasks focus on ownership, and contemporary speech recognition models are capable of discerning individual identities through voice characteristics. Integrating recognized user identity and vocal nuances as metadata in text input might achieve similar effects in current vision-language models (VLMs).\n\n2. While speech transcripts alone may lack certain nuances, the video input could compensate for these through visual cues like gestures. The paper’s limited treatment of emotion in its framework is another concern, especially as language models like ChatGPT already offer emotion detection from text (\"Emotion detection GPT\"), which could outperform humans [8].\n\n3. Overall, the additional complexity introduced by adding raw speech input to video and transcript data may not significantly enhance model performance in a meaningful way.\n\n4. The paper might benefit from using more recent multimodal VLMs, such as MA-LMM [1], Chat-UniVi [2], ST-LLM [3], LLaVA-NeXT-Video [4], VideoChat2 [5], VideoLLaMA 2 [6], and Video-LLaVA [7], to contextualize the VLAS model's contributions within the latest advancements.\n\nReferences:\n\n[1] He, Bo, et al. \"Ma-lmm: Memory-augmented large multimodal model for long-term video understanding.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Jin, Peng, et al. \"Chat-univi: Unified visual representation empowers large language models with image and video understanding.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[3] Liu, Ruyang, et al. \"St-llm: Large language models are effective temporal learners.\" European Conference on Computer Vision. Springer, Cham, 2025.\n\n[4] https://huggingface.co/llava-hf/LLaVA-NeXT-Video-7B-hf\n\n[5] Li, KunChang, et al. \"Videochat: Chat-centric video understanding.\" arXiv preprint arXiv:2305.06355 (2023).\n\n[6] Cheng, Zesen, et al. \"VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs.\" arXiv preprint arXiv:2406.07476 (2024).\n\n[7] Lin, Bin, et al. \"Video-llava: Learning united visual representation by alignment before projection.\" arXiv preprint arXiv:2311.10122 (2023).\n\n[8] Elyoseph, Zohar, et al. \"ChatGPT outperforms humans in emotional awareness evaluations.\" Frontiers in Psychology 14 (2023): 1199058.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hWwnIUm54M", "reviewer": "Reviewer_mePM", "summary": "This paper creates a vision-language-action model by incorporating speech to create a VLAS model.  Two new datasets SQA and CSI are introduced.  A RAG setup is used for handling contextual queries.  Section 1 gives background for why they incorporated speech, including contextual queries.  The model is built on top of LLaVA.  A pretrained speech encoder was used.  The model is called, “Speech LLaVA.”  Comparable accuracy can be found on the CALVIN benchmark to traditional VLAs.  For customization tasks, this approach appears to do very well.  Section 2 describes related work.  Section 3 describes the approach, including the LLaVA-based approach, action tokenization / detokenization.  SQA dataset was created by sampling examples from the original LLaVA dataset and running them through a TTS system.  CSI dataset used 389 textual examples x 500 voices to result in 194k examples; training was interleavesd with text and audio.  A three stage process involving audio alignment, speech question answering and then robot manipulation fine tuning was performed.  Section 4 describes experiments and results across CALVIN, customized tasks and LLaVA.", "questions": "Some questions about the paper:\n\n* The RAG pipeline appears to be intimately coupled with speech in this paper, but needn’t be.  Why wasn’t a baseline with language only inputs performed?  What would be the results?\n* All the experiments appear to be with TTS output, which will certainly be biased.  Some results need to be with real speech.  Do you have any quantitative numbers with real speakers?  Even on a subset of any of the data you presented?\n* The incorporation of RAG into the pipeline makes some parts of the paper confusing.  For example, in Section 4.1 it wasn’t clear if these leveraged the context part of the model (e.g., RAG) (e.g., Table 1).\n* Why wasn’t OpenVLA shown in Table 1?\n\n\nFor related work, it would be good to cite at least the following:\n* Other multi-modal models that incorporate audio should be included here.  Some missing references include, but there are likely more:\n    * Unified-IO 2: https://arxiv.org/abs/2312.17172\n    * Gemini (inherently multi-modal)\n* Other VLMs, such as IDEFICS and Prismatic:\n    * https://huggingface.co/blog/idefics\n    * https://github.com/TRI-ML/prismatic-vlms\n\nOther small things:\n“pre-cess” -> “process” p2.\nInconsistency between VLAS and SpeechLLaVA terms.\n“Absolute superiority” -> probably weaken this term.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper presents a natural extension to LLaVA to use speech input. With minor reservations about related work, the confounding nature of speech with and without context (and the lack of a text baseline), the exclusive use of TTS outputs and a few baselines, the paper is well-written and the results appear to be reproducible.", "weaknesses": "see above", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rCkgilBNjf", "reviewer": "Reviewer_5GYV", "summary": "This work proposes VLAS and integrates speech for robot manipulation. It makes communication more natural and presents two new datasets for the community.", "questions": "1. Are there any statistics from real-world experiment results? I think it's important to understand how the impressive results from simulations can transfer to real-world applications.\n2. You use the RAG and the pretrained speech model to obtain the voiceprint, which is then used to retrieve preferences from the database. It seems the voiceprint is derived from the RAG module. You only present the VLA experiment in a customized task, and I wonder about the performance of the VLA combined with speech RAG in this context. I think this need to be given to support your motivation ( why not use the cascading pipeline, you mentioned the cost and the intent capture), I suspect that the speech RAG can capture the intend and give the intend text from the database to VLA model.\n3. I think you need some more ablation studies, such as examining the effects of not using the RAG module in the customized task.\n\nI think if these questions are addressed, I will raise my score cause I think the unified large model for speech, vision, text and action is meaningful.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is easy to read and follow.\n2. The core idea and motivation are interesting and practical. I believe it is great to have the speech modality in interactions with robots, as it feels natural and intuitive. Using speech to capture human intent is also a good idea. Creating a unified model provides a solid foundation for future work.\n3. The illustrations are very helpful.\n4. The datasets provided to the community are beneficial for the field.", "weaknesses": "Some ablation studies may be needed. Please refer to the \"Questions\" section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)