Grounding Multimodal Large Language Models in Actions
Authors: Andrew Szot
Bogdan Mazoure
Harsh Agrawal
R Devon Hjelm
Zsolt Kira
Alexander T Toshev
Venue: NeurIPS 2024 poster (2024)
Keywords: Embodied AI
Multimodal Large Language Models
Reinforcement Learning
Imitation Learning
Primary area: generative_models

Abstract: Multimodal Large Language Models (MLLMs) have demonstrated a wide range of capabilities across many domains including Embodied AI. In this work, we study how to best ground a MLLM into different embodiments and their associated action spaces, including both continuous and discrete actions. For continuous actions, a set of learned tokenizations that capture an action at various resolutions allows for sufficient modeling precision, yielding the best performance on downstream tasks. For discrete actions, semantically aligning these actions with the native output token space of the MLLM leads to the strongest performance. We arrive at these lessons via a thorough study of seven action grounding approaches on five different environments, encompassing over 114 embodied tasks.

Grounding Multimodal Large Language Models in Actions
Andrew Szot
Bogdan Mazoure' Harsh Agrawal' 7solt Kira² Alexander Toshev!
Devon Hjelm
Apple, 2 Georgia Tech, Mila
a.szot@appler toshev@ apple.comm
Abstract
Multimodal Large Language Models (MLLMs) have demonstrated a wide range of capabilities across many domains, including Embodied AI. In this work, we study how to best ground MILM into different embodiments and their associated action spaces, with the goal of leveraging the multimodal world knowledge of the MLLM. We first generalize a number of methods through unified architecture and the lens of action space adaptors. For continuous actions, we show that a learned tokenization allows for sufficient modeling precision, yielding the best performance on downstream tasks. For discrete actions, we demonstrate that semantically aligning these actions with the native output token space of the MLLM leads the strongest performance. We arrive at these lessons via a thorough study of seven action space adapters on five different environments, encompassing over 1 14 embodied tasks.
Introduction
Multimodal Large Language Models (MLLMss) defined Large Foundation Models that take as input text and images and generate text, have recently seen rapid progress and impressive performance [1- 13]. These models are important as they solve large range of useful yet difficult natural language and image tasks, such as describing images, answering visual and textual questions, reasoning, and learning from small number of examples. They have only recently improved the point of being usable enough for general deployment with human non-experts [14-16].
While MLLMs are capable of describing real-world embodied concepts, their capabilities in embodied tasks are limited to using text for actions through generating code [17, 18], representing actions text [19], extracting actions from internal representations [20, 21J. Grounding [22] MLLMs to generate actions extends their capabilities to embodied tasks, such as robot manipulation and navigation, and is of tremendous value for practical problems, potentially overcoming the high cost of training tabula rasa. Extending MLLMs to multimodal image generation enables object detection and segmentation, and image and video generation [3, 23-27]. In embodied settings, grounding MLLMs via predicting agent affordances and generating actions yields effective policies capable of generalizing new tasks [19, 21, 28, 29].
key and open challenge in grounding MLLMs, which limits their capabilities in embodied tasks, is the gap between the native output space, natural language, and the action space of embodied agents. This problem is particularly acute continuous action spaces, where low-level controllers may require high degree of precision. Across the literature, number of architectures and ways of handling action spaces have been proposed, but there has not been a systematic study of these designs. Our contributions generalize prior attempts adapt MLLMs to generate actions through an empirical study on which principles and strategies are necessary to effectively close the gap between the action spaces of MLLMs and embodied agents. We study various grounding re-parameterization
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

CALVIN (34 tasiej
Meta- (45 baoiinl
Action Space Adapter Comparison
Overal Success (99)
Multimodal Large
Residual VQ
Language Model
Gontinute
Prediction
Semantic
VQ
Untor
Tok.
Voll
Environments
Continuous Actions
Discrete Actions
Action Space Adapter
BabyAl
Habitat (30
[0.72, 0.24, -..2 0..2
"Pick apple"
Figure I: We empirically analyze how to ground MLLMs in actions across 114 tasks in continuous and discrete action spaces. In each environment, we train a multi-task policy with different Action Space Adapters (ASAs) to re-parameterize the MLLM output actions. For continuous actions, learning a tokenization with several tokens per-action performs best (Residual VQ). For discrete actions, mapping actions to semantically related language tokens performs best (Semantic Tokenization).
strategies, which we refer to as Action Space Adapter (ASA), across a range of embodiments, action spaces, and environments. In particular, we explore the following types of ASAs: (1) ASAs that directly generate actions from a new prediction policy using the MLLM hidden representations as input; (2) ASAs that reuse the native token space of the MILM to encode actions; (3) and ASAs that introduce a new token space to encode the actions of the agent while adapting the MLLMs to predict these new tokens.
Further, we empirically identify important principles for designing ASAs. For continuous action spaces, learned tokenization with several vocabularies that residually model continuous actions gives the right modeling precision while using vocabularies of manageable sizes and, as a result, yields the best performance across all continuous control environments. This learned tokenization outperforms direct action prediction, indicating this approach allows the model to effectively learn a multimodal distribution over action spaces. In addition, the above tokenization strategy boosts performance when the policy is a MILM,, compared to other standard non-LLMMbassed policies, indicating that it manages to better tap into the knowledge of the model.
For discrete action spaces, we study ASAs that better align the embodied actions with the output space of the MILM. We demonstrate that semantic alignment between these mapping discrete actions to semantically related tokens the MILMM vocabulary yields the best strategy compared to other adapters that either reuse define new vocabulary. The superiority of this strategy is evident in performance on environments with discrete action spaces and also in RL sample efficiency.
Finally, the above principles are thoroughly validated across five embodied AI environments, three of which are robotic continuous control and two with discrete actions as illustrated Figure 1. Altogether, consider 1 14 language specified tasks. In the continuous case, the best tokenization achieves 72% on CALVIN [30], up from 68% for direct action regression and 28% for uniform action tokenization; and 84% on Meta-World [31], I], up from 61% for direct action regression and 75% for uniform tokenization. Similarly, in the case discrete actions, the proposed semantically aligned action tokens yield 51 % on LangR [21]. up from 42% for direct action prediction.
2 Related Work
Prior works propose different Action Space Adapters (ASAs) adapt MLLMs into policies. Some works use LLMs or MLLMs as zero-shot policies by prompting them to output text code that can be executed as actions [18, 32-38J. The ASA in this case is given executor or low-level controller that takes text as input and outputs actions in the environment. Other works investigate adapting MILMs for actions, but focus on single ASA and environment. For example, RT-2 [19] uniformly discretizes continuous actions and predicts tokens corresponding to each of the action dimensions. RohoFlamingo [20], Lamo [39], and L.RRP [21] use an MI.P to predict an environment action from an LLM hidden state. GFlan [40] treats discrete actions text and ranks actions by the LLM log

---

Continuous ASA
Discrete
Adapter Decoder
Action Tokens
Regression
Classification
Unifom Tokerizztion
B E
Semantio
"al0 upple
Adapter Head
Tokermaation
p430 153
Tokenization
Non-Sarmentcc Tokerization
350220
LLM Final Hidden
A
Action for Environment
Multimodal LLM
Ot
Language Embed
Downsampler
Adapter Embed
Pretrained MLLM
Task Goal Prompt
Action Space Adapter
Visual Encoder
Figure 2: Generic architecture studied here for adapting MILMs for action-specific decision making. The MLLM takes the embedding of the task instruction, prompt, and visual tokens as input. The MLLM then autoregressively predicts a sequence of m action tokens. These action tokens are then decoded into an environment--pecifee action.
probability to form a distribution over actions. At a high level, our work is distinct in that we study variety of methods across multiple environments for learning ASAs. We focus tasks with low zero-shot VLM performance, such as low-level control or long-horizon planning tasks. We summarize the differences between our investigation and prior work adapting VLMs for action in Appendix A.
Investigating action representations in embodied settings is not new. Some works learn representations of actions to help generalization to new actions or operating in large action spaces [41, 42] in the context of Reinforcement learning (RL.). Our study proposes ASAs for tokenizing continuous actions, and other works use different types of discretization or tokenization strategies on continuous action spaces. [43, 44] use k-means to discretize continuous actions help learn from multimodal behavior datasets, such as from play data data from different experts. VQ-BeT BeT [45] finds learning residual VQA (RVQ) codebook for continuous actions works best but does not apply this idea to MLLMs. [46] predicts actions text. [47] learns multi-task transformer policy and models actions with diffusion head.
More broadly, prior works have adapted MLLMs for modalities other than actions, such as object bounding boxes and image generation, both being continuous nature while the latter of high dimension. For example, [27, 48] train MILMs to output spatial reference tokens to ground text responses in image regions. For image generation, [49] adapt MILMs to generate image patches; [50, 51] tokenize images using a VQ-VAE model and adapt MILMMs to generate images by decoding these image tokens, which has inspired us to use the same learned tokenization; [52] uses an RVQ model [53] generate images, similarly our best performing tokenization scheme.
Method
In order to solve an embodied task, an agent learning in an interactive environment must select a decision from set of valid actions. For example, action space could be set of keyboard presses for a video game or a real-valued vector that controls a robotic manipulator. Our work studies how to best adapt MILM,, which is originally trained output text tokens, to instead model actions from given environment. We refer to the module that bridges a MLLM with a certain action space as an Action Space Adapter (ASA) (see Figure 2).
3.1 Problem Setting
Our analysis focuses on language-spedde specified tasks with visual observations. Specifically, we consider goal-specified Partially-Obeeralt Observable Markov Decision Process (POMDP) [54] that has an observation space O, action space A, and goal space G. For brevity, we omit other elements the MDP. In our setting, is a textual description of the task to solve. O consists of RGB visual perception and agent proprioception. We consider a range of different action spaces A that broadly fall into two categories discrete and continuous. The primary objective is learn language-con conditoonoo policy that maps
3

---

observations and the instruction text to an action T(a As later described in Section 3.3, we learn this policy through supervised fine tuning from expert demonstrations or reinforcement learning that maximizes the expected discounted cumulative reward of the POMDP.
3.2 From Vision and Language to Action
The process studied here for adapting MLLMs for decision making is illustrated in Figure 2. The MLLM policy takes as input textual instruction describing the downstream task, a sequence of past observations in the task and outputs an action in the agent's ' action space. In the bottom left of Fig. 2, the task description, as well as the environment description, embeddings. To these embeddings, the MI.LM then appends a sequence of visual embeddings from the current observation O₂- Since visual embeddings can often be comprised of large number of tokens (the popular VA-155 model [6] has 556), we introduce a downsampling layer to enable the MLLM to attend over a longer history of observations. In practice, we take the downsampling layer to be a Perceiver model [55]. a learnable transformation that reduces the number of tokens from the visual encoder before being used as input the MLLM.
first encoded to produce language
The sequence of language and visual embeddings is passed through the MLLM, whose final hidden state h encodes the entire input. The ASA, whose trainable parameters are denoted 6, is comprised of three parts: (1) an adapter head, (2) an adapter embedding, and (3) an adapter decoder. The hidden state is first passed through the adapter head to produce action tokens u, Ao (ht)). The action tokens are then embedded using the action embedding into Eg(u)), ). and passed autoregressively through the MILM to produce further hidden embeddings n.? ......... u," and associated action tokens ui,,.. u²", resulting in total m tokens per time step. The predicted action tokens are then decoded into the final action oe by the adapter decoder, which produces the final action a, Du(u "."). As ag E A, it is then executed in the environment to produce ot+1, and the process continues.
Next, we describe possible ASA implementations for discrete and continuous action spaces.
3.2.1 Discrete Action Spaces
We define the following action spaces adapters for a discrete action space A::
Categorical Prediction (Pred): Implement the action space adapter MLP network, which predicts the logits of a categorical distribution over environment actions from the MLLM hidden state. The adapter head is an MLP that maps the hidden state h' directly an action a E A. This amounts to producing a single action token w², which directly corresponds to the action a, with the action decoder being an identity map. Both the adapter head and token embeddings initialized from scratch. This type of ASA is used by [21].
Semantic Language (SemLang): The action space adapter predicts natural language text that maps to discrete action. First, each action a E AA is described with freeform text tokenized as (4,..... ... 4m The MILMM then autoregressively predicts sequence of tokens, which are then decoded by the adapter decoder to the corresponding action. For example, in an action space choosing a high-level skill a could be described as "pick apple", which is tokenized as [5839, 26163] with the LLaMA tokenizer. The MLLM then must sequentially predict token 5839, then token 26163 to call this action. Sequences of tokens corresponding invalid actions are either avoided entirely with the token filter described in Section 3.3 or treated as a no-op. Both the adapter head and the token embeddings are re-used to be the pretrained L.M''s language head and embedding layer, respectively, meaning no additional parameters over the pretrained MLLM are added. This type of ASA is used by [29].
Non-Semantic Language (Lang): Actions mapped to language tokens, but instead of semantically meaningful descriptions of the actions as with SemLang, the actions are mapped to sequences of numbers. For example, "pick apple" is represented with the string "5 3". The policy must then output the tokens corresponding to this text to call this pick action. Note that we can pick any text for this mapping and the choice of integers is arbitrary. However, the selected text is not semantically representative of the action.
3.2.2 Continuous Action Space Adaptors
We define the following four ASAs for a continuous D-dimensional action space A: the first ASA predicts in the original action space while the other three use tokenization. At training time, we learn

---

policy to predict these action tokens from the ASA. At test time, we employ an action decoder that maps these action tokens actions in the original space A.
Continuous Regression (Pred): Regress to the original continuous action from the MILM hidden state ht- This is achieved via single-layer MLP network, which is trained using MSE loss. This ASA is used by [20,39] 39].
Uniform Action Tokenization (Uniform): The simplest approach is to use uniform binning of the action space, In particular, we express each action as a sequence of D tokens by quantizing each of the D action dimensions into one out of K uniform bins:
Uniform(a) (A1 (k) such that ddd bin d)
kD)
where bin(k: d) denotes the hthh bin along the dhh action dimension. If my and Ma denote the lower and upper bounds respectively of the dh action dimension, then its definition reads bin(k; d) = [ma+ + k M ma + (k + 1) _ At test time, we decode predicted action tokens to the center of the corresponding bins for each dimension. This type of ASA is used by [19].
x
Vector Quantized Tokenization (VQ): To adapt the tokenization the particular action space, we propose to use learned tokenization. In particular, we express each action as single token that corresponds to the closest action code from learned codebook V. Using encoder network fo that maps actions to a latent embedding space:
vQ(a) = (ki) where k = arg min [Ife(a) vs 11
where up V. The codebook V of size K is learned over offline dataset D of actions using VQ-VAE [56] trained with the mean-squared error for action reconstruction and commitment loss. We overwrite K infrequently used tokens from the LLM vocabulary represent V. We defer the full details of this tokenization process to Appendix C.2.
Residual Vector Quantized Tokenization (RVQ): Precise control requires precise action modeling that can suffer after tokenization. To increase the precision of a learned tokenization, we further investigate the use of sequence of several action tokens as in Uniform. Similar VQ, these tokens are from M action codebooks Vn, m E {l..... M). However, each codebook models the residual space obtained after modeling the action using preceding codebooks, thus each subsequent token captures increasingly finer action information:
RVQ(a) = (Ai..... kM) where
kim = arg min &
(ooaa)
12
Uk
k
where of E 1/ the ttth code from the 7th codebook. Such tokenization can be learned using Residual VQ-VAE IRVQ-VAE, 52] on an offline dataset of actions. The actual number of token sequences can represent is KM Hence, RVQ presents the opportunity to exponentially increase the action space quantization without having to drastically increase the size of the learned individual codebooks.
3.3 Training
We use VA-VA-15--7 [6] the base MLLM. We finetune the MLLM with interactive (i.e., action- labeled) data to make it more suited for interacting with embodied and interactive environment.
Supervised Fine Tuning (SFT) with Expert Demonstrations: We finetune the MILM for inter- active tasks using a dataset of expert demonstrations. Each demonstration contains (1) a language description of the task, (2) sequence observations, and (3) sequence of actions that successfully solve the task. Note that in this work, we are primarily interested in learning imitation policies from offline data, which can be extended to offline reinforcement learning if per-timestep rewards are included in the dataset. Specifically, we train the MLLM with supervised learning to predict the expert actions from the observations and language description in the data. While the pre-trained LLM and the visual encoder remain frozen, we finetune the ASA, the visual downsampler, and parts of the LLM with LoRA [57]. In total, the model has R 100M learnable LLM parameters and 40M learnable downsampler and ASA parameters. The learned tokenization schemes (RVQ and VQ) have an additional pre-training phase, where the VAE models are first trained on actions from the offline dataset and then frozen to prevent further updates in later stages.
5

---

84
Residual VQ
Semantic Tokenization
80-
Categorical
68 56
Continuous Prod. Uniform
Non-Semantic Tokenization
61
80
51
20- 28
29
32
17
19
Habilat Pick
LangReerrange
BabyAI
Continuous Actions
Discrete Actions
Figure 3: Comparing ASAs for continuous and discrete action spaces across environments. For continuous actions, the RVQ tokenization performs best. For discrete actions, SemLang performs best. Each bar gives the average over all tasks in the environment with the full breakdown in Appendix E.
Reinforcement Learning (RL) from Environment Feedback We also optionally finetune the MLLM to optimize an environment reward using RL. However, predicting actions in the MLLM token space dramatically increases the number of possible action predictions, with many possible predictions corresponding to no valid action. For example, there are 32,000 tokens in the LLaMA text tokenizer, giving 32, 000" possible predictions by the model with tokens per action. This makes exploration difficult RL as only a small fraction of the possible actions are valid. We therefore use token filter restrict the autoregressive sampling only be from token sequences corresponding valid actions. The token filter is function M ( :..... ((= that produces binary mask over all
tokens to represent valid tokens for the jth decoding step.
4 Experiments
4.1
Experimental Settings
We study adapting MLLMs for action across a variety of environments with different embodiments and action spaces. All environments provide RGB visual observations and natural language instruction specifying the goal to achieve. We provide the important environment details below and defer complete details to Appendix B.
CALVIN [30]: This manipulation benchmark tests the ability of a tabletop robot to interact with an object to complete natural language instruction. The continuous actions specify 6DoF end-effector control and the binary gripper state. The observation 200 x 200 RGB image from fixed-position camera. We use the ABC + D split of the benchmark with 34 tasks, and the agent is evaluated on unseen instruction phrasings and table background.
Meta-World [58]: We use the ML-45 version of this tabletop manipulation benchmark which has 45 tasks. The action space is continuous control specifying 3DoF end-effector translation and the continuous gripper state. The observations are 200 x 200 RGB images from fixed camera. The agent is evaluated on unseen object and robot starting states.
Habitat Pick (HabPick) [59]: A mobile manipulation robot must pick up an object specified by name from receptacle. The continuous actions specify the 7DoF relative joint positions of the arm, the 2D base velocity, and the gripper state. The observations are 336 x 336 RGB images from the robot's egocentric head camera. The instruction specifies the name of the object type to pick up. The evaluation distribution is on unseen houses and new arrangements of objects.
BabyAl [60]: BabyAI is grid world task where agent navigates and interacts with objects to complete an instruction. The discrete action space consists of navigation and interaction actions. The observation is 200 200 RGB top-down view. We the five tasks from [40], and we report generalization to instructions rephrased with synonyms.
Language Rearrangement (LangR) [21]: A mobile manipulation robot must rearrange objects to complete instructions like "store all the fruit in the fridge". The discrete actions are 70 high-level skills to interact with objects and navigate. The observation is 336 336 RGB head camera.

---

Caderbook Corde books (VVQ)
64
6464
Codebroz (VG)
Codeboncs (8V0)
Codebook Size
Turte C'Conoo
Horber Cr
(d) # Codes: Recon. Figure 4: (a.b) show the effect of the number of codes in the codebook for RVQ and VQ on final policy success rate (see (a)) and reconstruction on unseen action trajectories in Meta- World (see (b)). (c,d) show the effect of number of codebooks on final policy success rate (see (c)) and action reconstruction (see (d)). All metrics are computed on Meta- World.
(a) # Codes: Success
(b) # Codes: Recon.
(c) # Codebooks: Success
Evaluation instructions test generalization to unseen houses and 10 unseen instruction datasets measuring paraphrastic robustness and behavior generalization.
In all environments, we report the success rate as the fraction of episodes in which the agent completed the language instruction. We use the success criteria provided by each environment. We train a policy per action adapter for each environment and report the generalization performance in the main text. When reporting a single success rate per environment, it is the success averaged between all evaluation episodes containing all tasks. We give the full per-task breakdown for results in Appendix E. CALVIN, Meta-World HabPick, and BabyAl Al provide expert demonstrations succeeding the task. CALVIN has 17.9k from humans, Meta-Worl 22.5k from scripted policy, HabPick 6.7k generated from an RL policy, and Baby AI 5k from a scripted policy. Full details on the train and evaluation setups per environment are Appendix B.
We train with supervised finetuning for CALVIN, Meta- Veta-World, HabPick, and BabyAL. We train with reinforcement learning on Language Rearrangement. As described in Section 3.3 train s 140M parameters with LoRA A [57]- We use the Adam optimizer [61] with a learning rate of 3e-1, a warmup period of 10% of the total number of training steps, and cosine learning rate decay to by the end of training. For RI, we use PPO [62]. For the learned tokenization action space adapters, we, by default, use codebook size of 512 with 512 dimensions per codebook element. Complete hyperparameter and policy details are in Appendix C.
4.2 Continuous Action Space Adapter Comparison
We first study adapting MLLMs through Uniform, Pred, VQ. and RVQ action space adapters for the continuous action environments CALVIN, Meta-World and HabPick.
RVQ the best performing continuous action ASA. The results in Figure 3 show that the RVQ action adapter consistently outperforms all other ASA approaches across all environments. While Pred is the second best performing method on all tasks, except on Meta- World, RVQ outperforms it by 12% average absolute difference. One hypothesized reason for this is that Pred only learns unimodal distributions of actions, which hurts performance when learning from diverse demonstrations [43-45]. Another potential reason is the tokenization from RVQ allows the MLLM to better leverage its existing knowledge, whereas the Pred ASA requires training a new MLP network from scratch.
Uniform performs poorly on the majority of the tasks, where RVQ outperforms average by a 27% absolute increase. A reason for this that the Uniform discretization can fail to accurately represent the continuous actions. The performance of Uniform is also closely related to the action dimension. In Meta- World with 4 action dimensions, Uniform performs well. However, Uniform suffers with the 7 action dimensions in CALVIN and the 10 action dimensions in HabPick.
RVQ also outperforms VQ by a 18% absolute difference averaged over all environments. This is due to VQ having worse action reconstructions than RVQ. In Meta- World, both RVQ and VQ policies reach a similar cross-entropy loss on holdout trajectories during finetuning. However, on this same data, RVQ has reconstruction mean squared error (MSE) of 0.005 while VQ has 10x higher reconstruction MSE of 0.05. Increasing the VQ codebook size does not close this gap. We vary the VQ codebook size in powers of 2 from 2 to 211, Figure 4b shows the VQ reconstruction loss decreases with larger codebooks but does not even close the gap to the 27 RVQ codebook size. This poor reconstruction manifests in poor downstream policy performance demonstrated by Figure 4a
7

---

100
100
RVQ
Success Rake (80)
80
48
59
55
48
47
30
Articulated
Press
Push
Pull
Articulated
Press
Lift
Push
Rotate
Figure 5: RVQ and VQ success per-task grouping (defined in Supp. B.6) CALVIN and Meta World.
where policies trained with the VQ ASA plateau in success codebook size 2 VQ policies decrease in performance at codebook size 2 potentially due to overfitting to the large codebook.
We further characterize the performance of RVQ and VQ in Figure 5 by breaking down the per- formance per task group in Meta- World and CALVIN. The task groups, which are fully listed in Appendix B.6, correspond tasks with related required behaviors. Both RVQ and VQ do similarly on "articulated" object interactions (like opening drawers or doors). These tasks require less precise control since many contact points on the articulated link and broad pushing or pulling behavior can achieve the desired behavior. On the other hand, RVQ outperforms VQ on "pressing" tasks that require pushing a button. These tasks require button all the way to a desired state. VQ often reaches the button but fails to press it all the way. The same is also true of other precise control tasks like picking, pulling, and rotating.
precise control since the agent needs push the
potential explanation of RVQ's success can be attributed to adaptive localization of the model's errors, similar prior work in residual reinforcement learning [63] and Bellman error bases [64].
sufficient codebook size and number of codebooks are necessary for RVQ. Figure 4a, we show that RVQ policy performance improves in performance with larger codebook size in Meta-World. Notably, RVQ performs poorly at 29% success rate with codebook size compared to 84% success at codebook size 512. These observations also align with the codebook size decreasing reconstruction error Figure 4b. Figure 4c, we compare the effect the number of codebooks on performance. As carlier discussed with the performance of VQ, one codebook results in poor action reconstruction and, thus, bad policy performance. However, increasing the number of codebooks too much to 6 also hurts performance despite decreasing reconstruction loss. Likewise to the finding that Uniform performs poorly with larger action dimension since there are more tokens per action, increasing the number of codebooks also hurts policy learning.
RVQ tokens transfer to new tasks. We take the model trained on the 45 Meta- Sorad tasks and finetune it on 5 unseen tasks. We collect 50 demonstrations for per task and finetune the policy on all task data. We use the same RVQ ASA trained only on data from the tasks. Figure 6a shows success rate of adapting RVQ compared to an Pred ASA. RVQ outperforms Pred across all tasks, achieving a 50% vs. 20% overall success rate. This demonstrates the RVQ tokens are flexible enough to be applied to new tasks.
The gains from RVQ are unique to MILMs. Next, we analyze the unique interaction between the RVQ tokens and the MILM policy. While we demonstrated that the RVQ ASA performs best, this improvement due to the MLLM being able to leverage these new tokens or the added action representation ability from the separately trained RVQ decoder? To test this, we compare to two policy architectures that do not use LLMs:
Scratch: This the same architecture as the MLLM-basse policy, but with a smaller 300M parameter non-pretrained transformer.
RT-Inspired: This method uses ResNet visual encoder, pretrained Flan [65] language embedding and decoder transformer- based policy. The entire policy is trained from scratch. This method is inspired by RT-1 [66], which does not have publicly released code.
Table compares the effect of Pred versus RVQ ASAs on CALVIN, deta-World and HabPick for these three policy policy architectures. As already established for the MILM,, RVQ is consistently better than VQ. However, for the same policy architecture trained from scratch, RVQ can hurt the performance over Pred. In CALVIN the success drops -7% and in Meta-War World the performance

---

MLLM: Pred~RVV
Seratch: Pred -RQ
RI-Inspired: Pred RVQ
Calvin
68 + 72 (+4)
50 + 43, (-7)
35 + 36, (+1)
Metaworld 61 84 (423)
71 - 56, (-15)
27 + 38, (+11)
Habitat Pick
19 + 29 (+10)
+ 25, (+4)
- 20, (+2)
Table 1: Comparing the effect of the RVQ action space adapter on the success rate of non-LLM based policies. Red indicates RVQ hurts over Pred and green indicates RVQ helps over Pred. RVQ typically has a negative impact on the Scratch policy, and helps the smaller RTT-Inppire policy.
drops 15%. This highlights that MLLM can leverage its existing knowledge about sequencing language tokens to sequencing action tokens. However, we find that for the smaller RT-Inspired policy network, the RVQ ASA consistently helps, which we hypothesize is because the added RVQ network and separate training help compensate for the lack policy network capacity. We also note that RVQ may more consistently outperform Pred demonstrations that explicitly contain multimodal action sequences [43-45].
4.3 Diserete Action Adapter Comparison
SemLang performs the best. In Figure 3, SemLang outperforms the next best ASA (Pred), by 9% on Language Rearrangement and 8% on BabyAL. SemLang performs especially well on tasks with explicit high-level language actions in Language Rearrangement (e.g., "pick apple") where prior work has shown text-only LLM policies achieve non-zero success [21]. SemLang also does well on the BabyAI tasks with discrete low-level actions like "move left". Additionally, Lang performs the worst in both environments, achieving 14% lower success on Language Rearrangement and 11% lower AlbyAI than SemLang. We hypothesize this is because the MLLM has to repurpose its knowledge to leverage these newly assigned action tokens, whereas a newly initialized Pred allows extracting this knowledge from the MILM hidden state.
SemLang enables sample efficient RL. In Figure 6b, we compare the RL training curves for the ASAs in Language Rearrangement. In addition to helping with better generalization, SemLang also enables sample efficient RL training. SemLang converges training performance after just 20M training samples, whereas Pred requires up to 70M steps to fully converge.
Token filter crucial for language- based action spaces. Figure 6b. we show the training of SemLang without the token filter, which restricts policy outputs to only valid action token sequences. Without the token filter, SemLang is unable learn in the large text action space.
80
Success Rate (%)
30
20
Prod
Lang
Semtang
10-
Semtang Riter
Overall
Unlock
Door
Hand Insert
Bax Close
Door
Lock
Bin Picking
Environment Steps
107
(a) Finctuning on Holdout Tasks
(b) Language Rearrangement RL.
Figure 6: (a) Adapting to 5 holdout tasks from Meta- World MI.-45 with demos per task using the fixed RVQ tokenization. (b) RL training curves in Language Rearrangement comparing the ASAs and utility the token filter. Displayed are averages over 2 seeds with the shaded area as the standard deviation between seeds. SemLang learns faster than other ASAs and the token filter is crucial.

---

4.4 Empirical Comparison to Prior Work
The contributions of this work are empirical analysis of ASAs under controlled settings various embodied environments. Direct comparisons to prior work are challenging due to different training algorithms, policy architectures, assumptions about input modalities. Regardless, in this section, we seek to contextualize our RVQ and SemLang MLLM results against prior work. In Meta- Meta-World, to the best of our knowledge, RVQ at 84% success on MLL-45 sets new -ffe-- result, compared to 79% from DualMind [67]. In CALVIN, RVQ at 72% success underperforms a similar work RoboFlamingo which achieves 82% success on the ABC - D split. However, RoboFlamingo uses a different MILM and uses an additional gripper camera input. In I Language Rearrangement, SemLang sets state-of-the-art result with 51% success compared to 42% from LLaRP [21]. In BabyAl, AI, Seml Lang at 40% success rate underperforms GFlan [40], which achieves 55% success. However, we use RGB visual observations, while GFlan operates from a compact, ground truth language state description. In Appendix A.I, we compare these differences in more detail.
Limitations and Conclusion
In this work, we studied various action space adapters (ASAs) across a variety of embodiments. action spaces, and environments. We provide generalization of prior works through the lens of action space adaptors, and for both discrete and continuous action spaces demonstrate designs that we show can leverage the knowledge within the MI.LM. Our findings conclude that for continuous actions, it best to learn action tokens that accurately model the action distribution, while for discrete actions, it is best to reason over semantic language descriptions of actions. We verify these ideas across 1 114 embodied AI tasks in 5 diverse environments.
A limitation of our work is all our analysis is under a single MLLM (LLaVA). Another limitation is that RVQ, the best performing ASA in continuous action spaces, requires collecting demonstrations to train the VQ model. Our analyses are also under only single LoRA training setting. Future analyses can explore different base MILMs under different training regimes like full LLM finetuning. While our investigation of ASAs enables connecting MILM to various action spaces, the performance of these methods is still subpar for real-robot deployment where high success and safety are critical. MILMS with the best ASA still struggle on simple environments like Baby AI, only achieving 40% success rate. Further work is needed to improve the performance of these methods for real-world usage. Our investigation also only studies adapting MLLMs through behavioral cloning on-policy RL. Future work can investigate the choice of ASA varies when adapting the MLLM with other learning algorithms such off- off-policy RL offline RL.
References
[1] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966. 2023.
[2] Shaohan Huang. Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma. Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei. Language is not all you need: Aligning perception with language models, 2023.
[3] Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023. [4] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman pre- pre-training
with frozen image encoders and large language models. 2023.
[5] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards gencral-purpose lisionnlanguage models with
instruction tuning, 2023.
[6] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
[7] Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, and Jianfeng Gao. Multi- modal foundation models: From specialists general-purpose assistants. arXiv preprint 23X9v:2309.1000
2023.
[8] Deyao Zhu, Jun Chen, Xiaogian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint 204542234411059. 2023.
10

---

[9] Qinghao Ye, Haiyang Xu, Guohai Xu. Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, al. mplug-owl: Modularization empowers large language models with multimodality, arXiv preprint arXiv:2304.144 14178, 2023.
[10] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Jingkang Yang. and Ziwei Liu. Otter: A imulti-mode model with co-tontext instruction tuning. arXiv preprint arXiv:2305.0377
2023.
[11] Bo Li, Yuanhan Zhang. Liangyu Chen, Jinghao Wang. Fanyi Pu, Jingkang Yang. Chunyuan Li. and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint NNN:23306.05425 2023.
[12] Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mm1: Methods, amalysis & insights from multimodal
I1m pre-training arXiv preprint arXiv:2403, 09611. 2024.
[13] IDEFICS. Introducing idefics: An open reproduction of state-of-the-an visual language model. https: //huggingface go/blog/ iddfiis. 2023.
[14] Gemini Team, Rohan Anil, Schastian Borgeaud, Yonghui Wu, Jean--aattitt Baptistttstt Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, ct al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 11805, 2023.
[15] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08777 08774, 2023.
[16] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[17] Jacky Liang. Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian lchter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. arXiv preprint 2rX 2022.
[18] Andy Zeng. Maria Attarian. Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo. Vikas Sindhwani, et al. Socratic models: Composing zero-shot
multimodal reasoning with language. arXiv preprint (rxXiv2204.005900 2022.
[19] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-languago-action models transfer web knowledge robotic control. arXiv preprint arXiv:2307. 15818, 2023.
[20] Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing. Weinan Zhang. Huaping Liu, et al. Vision-language foundation models as effective robot imitators. arXiv
preprint arXiv:2311.0 1.01378, 2023.
[21] Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Walter Talbott, Katherine Metcalf, Natalic Mackraz, Devon Hjelm, and Alexander Toshev. Large language models as generalizable policies
for embodied tasks, arXiv preprint arXiv:2310.1722. 2023,
[22] Stefanie Tellex, Nakul Gopalan, Hadas Kress-Gazit, and Cynthia Matuszek. Robots that use language.
Annual Review of Control, Robotics, and Autonomous Systems, 3:25-55, 2020.
[23] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal ilm's referential dialogue magic. arXiv preprint arXiv:2306. 15196,1599..21 2023.
[24] Нaoxua You, Haotian Zhang, Zhe Gan, Xianzhi Du Bowen Zhang. Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything any anywhere at any granularity. In ICLR, 2024. [25] Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-
decoder for vision-centric
tasks, arXiv preprint arXiv:2305. 1175, 2023.
[26] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint arXiv:2308..066 00692, 2023.
[27] Hao Zhang, Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Lei Zhang, Chunyuan Li, et al. .lava-grounding: Grounded visual chat with large multimodal models. arXiv preprint arXiv:2312.0294 212.222999922222.
[28] Michael Ahn, Anthony Brohan. Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu. Keerthana Gopalakrishnan. Karol Hausman, et al. Do can, not say: Grounding language in robotic affordances. arXiv preprint
(tt.:222440169// 2022.
[29] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. PaLM-E: An embodicd multimodal language model. arXiv preprint 2xXivv2303.03378 2023.
[30] Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. Calvin: A benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, ((3y:337-73332 2022.
[31] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094-1100 PMLR, 2020.
[32] Dhruy Shah, Blažej Osiński, Sergey Levine, et al. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. In Conference on Robot Learning, pages 492-504. PMLR, 2023.
11

---

[33]
Wenlong Huang. Fei Xia. Ted Xiao, Harris Chan, Jacky Liang, Pete Florence. Andy Zeng. Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207, 00008, 2022.
[34] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian lehter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA). pages 9493-9500 IEEE, 2023.
[35] Wenlong Huang, Fei Xia, Dhruv Shah, Danny Driess, Andy Zeng. Yao Lu, Pete Florence, Igor Mordatch, Sergey Levine, Karol Hausman, et al. Grounded decoding: Guiding text generation with grounded models for robot control. arXiv preprint 1rXXx:2333300855 20233
[36] Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng. Shuran Song. Jeannette Bohg, Szymon Rusinkiewicz, and Thomas Funkhouser. Tidybot: Personalized robot assistance with large language models. arXiv preprint arXiv:2305.055 005588.5.. 2023.
[37] Tom Silver, Soham Dan, Kavitha Srinivas, Joshua Tenenbaum, Lestie Pack Kaclbling, and Michael Katz. Generalized planning in pddl domains with pretrained large language models. arXiv preprint arXiv:2305. /014, 2023.
[38] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima
Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305. 162229 2023.
[39] Ruizhe Shi, Yuyao Liu, Yanjic Ze, Simon S Du, and Huazhe Xu. Unleashing the power of pre-trained language models for offline reinforcement learning. arXiv preprint arXiv:2310.20588 2025887 2023.
[40] Thomas Carta, Clément Romac. Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Y Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. arXiv preprint arXiv:2302. 026.2,2666222222222 2023.
[41] Ayush Jain, Andrew Szot, and Joseph J Lim. Generalization to new actions in reinforcement learning. arXiv preprint arXiv:2011.00 200011988. 2020.
[42] Gabriel Dulac-Arold. Richard Evans, Hado van Hasselt. Peter Sunehag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann. Theophane Weber, Thomas Degris, and Ben Coppin. Deep reinforcement learning in large discrete action spaces. arXiv preprint arXiv: 12.00769, 2015.
[43] Nur Muhammad Shafiullah, Zichen Cui, Ariuntuya Arty Altanzaya, and Lerrel Pinto. Behavior transform-
ers: Cloning modes with one stone. Advances in neural information processing systems, 35:22955-2222 -226688, 2022.
[44] Zichen Jeff' Cui, Yibin Wang, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. From play to policy: Conditional behavior generation from uncurated robot data. arXiv preprint arXiv:2210. 10.000270 2022.
[45] Seungjae Lee, Yibin Wang. Haritheja Etukuru, H Jin Kim, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Behavior generation with latent actions. arXiv preprint arXiv:2403. 0311,0 2024.
[46] Georgios Pantazopoulos, Malvina Nikandrou, Amit Parekh, Bhathiya Hemanthage, Arash Eshghi, loannis Konstas, Verena Rieser, Oliver I Lemon, and Alessandro Suglia. Multitask multimodal prompted training for interactive embodied task completion. arXiv preprint 2r 1.04067, 2023.
[47] Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprini arXivv2405. 12213, 2024.
[48] Zhiliang Peng. Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang. Shuming Ma. and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023. [49] Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, ct al. Generative multimodal models are in-context learners. arXiv preprint 2rxv
2023.
[50] Tianhe Yu, Ted Xiao, Austin Stone, Jonathan Tompson, Anthony Brohan, Su Wang, Jaspiar Singh, Clayton Tan, Jodilyn Peralta, Brian Ichter, et al. Scaling robot learning with semantically imagined experience. arXiv preprint arXiv:2302. /5500, 2023.
[51] Emanuele Aiello. Lili Yu. Yixin Nie. Armen Aghajanyan, and Barlas Oguz. Jointly training large autoregressive multimodal models. arXiv preprint 2t0i:::300.15666.. 2023.
[52] Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Partern Recognition, pages 11523-15532, 2022.
[53] Neil Zeghidour, Alejandro Lucbs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing,
30:495-507, 2021.
[54] Richard Bellman. A markovian decision process. Indiana Univ. Math. J., 6:679-684, 1957. ISSN
0022-2518.
[55] Andrew Jaegle, Sebastian Borgeaud, Jean- Baattttt Alayrac, Carl Doersch, Catalin Ionescu, David Ding. Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2 2021.
[56] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning, Advances in neural information proessing systems, 30, 2017.
12

---

[57] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 106.:2100.09685..
2021.
[58] Tianhe Yu, Deirdre Quillen, Zhanpeng Hc, R. Julian, Karol Hausman, Chelsea Finn, and S. Levine. Mcta-worid: A benchmark and evaluation for multi-task and meta reinforcement learning. In CoRL, 2019. [59] Andrew Szot, Alexander Clegg, Eric Undersander. Erik Wijmans, Yili Zhao, John Turner, Noah Macstre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, al. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in Neural Information Processing Systems, 34, 2021.
[60] Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. BabyAl: First steps towards grounded language learning with human in the loop. In ICLR, 2019. URL attps:://pott //tenreviee/ee
wwwfooooddddddeooooo [61] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.551 2017.
[62]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: prXiv:/707.000 (000000777777 2017.
[63] Tobias Johamnink, Shikhar Bahl, Ashvin Nair, Jianlan Luo, Avinash Kumar, Matthias Loskyll, Juan Aparicio Ojea, Eugen Solowjow, and Sergey Levine. Residual reinforcement learning for robot control. 2019
international conference on robotics and automation (ICRA), pages 6023-6029. IEEE, 2019.
[64] Ronald Parr, Lihong Li, Gavin Taylor, Christopher of linear models, linear value-functin approximation. and feature selection for reinforcement learning. In Proceedings of the 25th international conference on Machine learning, pages 752-759, 2008
Waketf--,,,,,,,,,, and Michael L Littman. An analysis
[65] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot leamers. arXiv preprint arXiv:2109.006
2021.
[66] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint #rXiv:2212.06817 2022.
[67] Yao Wei, Yanchao Sun, Ruijic Zheng, Sai Vemprala, Rogerio Bonatti, Shuhang Chen, Ratnesh Madaan, Zhongjie Ba, Ashish Kapoor, and Shuang Ma. Is imitation all you need? generalized decision-making with pual- training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16221-16231, 2023.
[68]
Ankesh Anand, Jacob Walker, Yazhe Li, Eszter Vértes, Julian Schrittwieser, Sherjil Ozair, Théophane Weber, and Jessica B Hamrick. Procedural generalization by planning with self-supervised world models.
arXiv preprint arXiv:2111.0155: 1222.....
[69]
Julian Schrittwieser, loannis Antonoglou, Thomas Hubert, Karen Simonyan. Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart. Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7833):604-669 2020.
[70] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenherg, al. A generalist agent. arXiv preprint arXiv:2205.0611 06.155..5 2022."
[71] Tsung- Wei Ke, Nikolaos Gkanatsios, and Katerina Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations. arXiv preprint arXiv:2402.1088 2024.
[72] Andrew Szot, Karmesh Yadav, Alex Clegg, Vincent- Pierre Berges, Aaron Gokaslan, Angel Chang, Manolis Savva, Zsolt Kira, and Dhruv Batra. Habitat rearrangement challenge 2022. https:/ hatee://ahhhbbbs htt///aaaaaaattttaoooooo org/ challenge, Laaggee022__aae2e
2022_rearrange.. 2022.
[73] Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transformers: State-of-the-art
natural language processing. arXiv preprint 1rXi::1910.0377 2019.
[74] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- perfforrrre deep learning library. Advances in neural information processing systems, 32, 2019.
[75] Jeff Rasley, Samyam Rajbhandari. Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep lcarning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Dota Mining, pages 3505-3506, 2020.
13

---

Work
Pastiemeeess
Best ASA
Onre ASAx Statied
Arrin Spece Type
Toannng
Using TENNEENNET
Roboi [20]
CAIVIN
MUP
Contiranes
Oilize
CiPla (41)
IsbyyA
Sem Lang bootnng
Uiarte
Onlmne
RT-3 [ig]
Interee
Unitenn
Continous
BC
LLaRP 1011
Langaago Reamongment
MLP
Diccue
Oudine
[65]
Fronica
RNQQ-MLP
Kitchen Sooress Pla Klycheehen
Ouzs
Lasgunge Resmanerrens Bats Al Mes-
RSGG Lans
MLP VQ. Unifors, Noz Son, Now
Costizuna, Discrte
Oniine BC
World CALVIN Habii Skills
Seat Cori
Table 2: Comparing our investigation to prior work. Prior work typically analyzes a single action adapter in a single environment. We study a variety of action adapters across variety of environments.
A Prior Work Comparison
In this section expand the differences between the prior work in action space adaptation mentioned in Section 2 and our investigation. Table compares our investigation to prior work along several key dimensions. We emphasize that unlike prior works, ours studies variety of action space adapters under a greater diversity of environments.
A.1 Empirical Comparison to Prior Work
We report performance on standard benchmarks which prior work has also extensively studied. However, even within the benchmarks there are differences in training algorithms and sensor input assumptions that make direct comparison to prior work difficult. Regardless of these differences, we study different ASAs for MLLMs in a consistent experimental setting. We also describe differences between the empirical setups of ours and prior works that perform well on these benchmarks.
Meta-Worte (MILM + +RQQ 84% success rate on MI.-55): To the best of our knowledge, our 84% is the highest reported on Meta-World ML-45 so far. Anand et al. [68] operates under similar sensor assumptions and achieves 77% success with MuZero [69]. DualMind [67] achieves 79% success rate on ML-45 and outperforms other generalist agents like Gato [70]. However, DualMind uses privileged simulator information about the joint states and object positions while we only use RGB visual observations.
CALVIN (MLLM +RVQ 72% success rate): RoboFlamingo achieves a higher 82% success rate on the same ABC + D task. However, RoboFlamingo uses the OpenFlamingo VLM while use LLaVA. RoboFlamingo use the gripper and fixed camera while we only use the fixed camera. More recent work like 3D Diffuser Actor [71] practically solves the ABC - D task, achieving 96% success rate. However, this work uses depth inputs, and a diffusion model policy that predicts keypoints for the end-effector rather than underlying actions. Our work uses only RGB visuals, uses a MLLM policy and predicts relative end-effector poses rather than keypoints.
Language Rearrangement (SemLang 51% success rate): This outperforms the prior highest reported number 42% on the overall evaluation set from LLaRP [21].
Baby (SemLang 40% success rate): GFlan [40] achieves 55% success on the same evaluation split. However, the GFlan policy takes as input a ground truth language description of the state, while our policies take as input 200 x 200 RGB top down rendering of the environment. GFlan also trains the policy with reinforcement learning while we train with supervised learning.
B Environment Details
An overview of the environments is visualized in Figure 7. This figure visualizes the training observations input to the agent. We run experiments on 5 environments, and each environment in turn consists of multiple tasks. We arrive at the task count of 114 in the main paper through 45 tasks in World, 34 CALVIN, 20 HabPick where we count each object goal as a different task, 10 in Language Rearrangement for each of the evaluation splits, and 5 in AI. The task count for Language Rearrangement is conservative since technically consists of 282 instruction templates, each of which corresponds to distinct task and goal.
14

---

CALVIN
Meta-World
HabPick
BabyAl
LangR
Rotate the pink block rowards the right
Pushh button on the coffee machine
Pick lemon
Open the blue door
Briig sornetikng to pour hot coffee in the stand
Figure 7: Visualizations of the environments study. The top row shows observation in the environment. The bottom row shows the associated instruction in that episode.
B.1 Meta-
Tasks: We use the ML-45 benchmark from Meta-World [58]. Each of the 45 tasks are specified with a fixed language instruction. We use the task descriptions from Appendix Section A of Yu et al. [58].
Observation Space: 200 200 RGB images from fixed camera position. To render the visual observations, we only use the "corner4" camera position as this gives an unobstructed view of the robot in most of the tasks.
Action Space: 4DoF continuous control of the arm and gripper. The first 3 dimensions specify the relative end-effector translation. The last dimension specifies the desired gripper state.
Training: We use 40 start and goal configurations for each the tasks. We generate 500 demonstra- tions for each of the 45 tasks. We use the scripted policies from Yu et al. [58]. At each step we add Gaussian noise (0, 0.1) to the actions produced by the scripted policy before executing it in the environment. We generate 500 successful trajectories per task, resulting in 45 500 22.5k total trajectories.
Evaluation: We evaluate performance on 10 unseen start and goal configurations for each of the 45 tasks. So in total, we evaluate 450 unseen configurations and report the average performance over these 450 episodes.
B.2 CALVIN
Tasks: We use the CALVIN ABC' - D dataset split.
Observation Space: 200 x 200 RGB observations from the fixed camera view.
Action Space: 7DoF continuous control of the arm and gripper. The first 6 dimensions specify the relative position and rotation of the end-effector. The final dimension is binary indicator for if the gripper should be open or closed. We hold out 1024 subsequences the policy context length from these trajectories for reporting validation performance during the SFT process.
Training: We use the 17,871 demonstrations provided in the CALVIN ABC dataset. These demonstrations are in different table backgrounds. This also includes 1,088 demonstrations for validation.
We report performance on the D split. This evaluation scene is different color than that encountered during training. All the start positions and goals are also different. Many of the language instructions are also unseen from training. We report the average performance over the 1,000 evaluation sequences. We report the success of the first task completed in the sequence.
Evaluation:
B.3 Habitat Pick
Tasks:
We use same Pick task as in Habitat 2.0 Geometric Goal object rearrangement [59, 72], except we provide the agent the name of the object rearrange rather than the starting coordinates of the object and increase the observation resolution. The task successful when the agent picks up the object and returns the end-effector within a fixed offset to a "resting position" in front of the robot.
15

---

The task ends in failure if the agent excessively collides with the scene, drops the object, picks up the wrong object. The agent starts within 2 meters of the object and facing towards the receptacle but with random noise . 1.57) applied to the direction of facing directly at the receptacle. The maximum number of steps per episode is 300 steps.
Observation Space: A 336 x 336 head-mounted RGB camera.
Action Space:
The action space is 10DoF control of the arm, base and gripper. The first dimensions control the linear and angular velocity of the base. The next 7 dimensions control the relative joint offsets of the arm. The final dimension controls whether the suction gripper is engaged not.
Training: We first train privileged policy with RL to complete the task. This policy takes input the egocentric depth image and the ground truth position of the target object to pick up. We collect 20k successful trajectories.
Evaluation: We evaluate on the test episodes from Szot et al. [72] which are 1, 000 episodes in unseen home layouts.
B.4 BabyAI
Tasks:
The tasks all occur in 6 x 6 grid populated with interactable objects. use the task definitions from Carta et al. [40]. This consists of the following 5 instruction templates: "Go to <object>". ", "Pick up <object>", " "Put <object A> next to <object B>,", ", "Pick up <object A> then go to <object B> and Go to <object B> after pick up <object A>", "Unlock <door>". The maximum number of steps per episode is 50 steps.
200 200 RGB observation as a top down of the 6 x 6 scene. Note this is more challenging observation space than prior gridworld navigation tasks that provide the current view as compact entity specific array [60] or by language description [40].
Observation Space:
The action space consists of actions consisting of: turn left, turn right, move forward, pick, drop and toggle.
Action Space:
Training: We collect 1,000 demonstrations for each of the 5 templates. We randomly sample an instruction and starting state configuration for every demonstration. We use the expert planner from Chevalier- Chevalier-Boisvert et al. [60] to generate the demonstrations.
Evaluation: We report performance on the unseen synonyms generalization test, described in Section 4.2 of Carta et al. [40]. We evaluate on 200 episodes per template type, giving 1000 total evaluation episodes.
B.5 Language Rearrangement
Tasks: An agent starts in an unseen house and must complete a rearrangement task from a language instruction.
Observation Space: The agent has 336 336 head-mounted RGB camera. We increase the camera resolution from 256 x 256 in the original Language Rearrangement task to match the input resolution of the LLaVA CLIP encoder.
Action Space: We use the same action space as from the original I Language Rearrangement henchmark Szot et [21]. The agent can select between 70 high-level skills that include picking up objects by name, navigating to receptacles, placing on receptacles by name, and opening and closing receptacles by name.
Training: Since Language Rearrangement does not provide any demonstrations and due to the emphasis on exploration in the problem, they not readily obtainable, even with oracle planners. Therefore, we opt to train policies with reinforcement learning from the environment reward provided by the Language Rearrangement task.
Evaluation: We evaluate on all evaluation datasets from Language Rearrangement consisting of 1,000 evaluation episodes unseen scenes.
16

---

B.6 Task Groupings
In Section 4 we breakdown the performance on CALVIN and MetaWorld for task groupings. Each of the task groupings consists of multiple tasks from the benchmark. We grouped tasks in the following way;
Meta World:
Articulated: "door-close", "door-open", "drawer-close", 'ddrawe-open", "faucet-open", "faucet-
close", "handle- "handle-press--Se press-side", side", "handle-press praliee-ppes,,,,,, "window-open", "window-close"
Press: Doutton-press-oodoone "coffee-button"
"button- praeoon-rresstooooow---eeee
"button-press"" "button- peennn-pesswwwee wall",
Push: "plate-slide,, "plate-slide-side"" "plate-slide-back" "plate-slide-back-sidte "push-back", "push", "push-wall", "stick-push", "sweep-into", "sweep". "soccer", "cooffe-push"
Pick: "assembly", "baskethall", "dial-turn", "disassemble", "hammer", "peg-insert-side". "peg- unplug-side", "pick-out-of-hole",, "pick-place", "pick-place-wall" "reach", "reach-wall", "shelf- place"
Pull: "coffee-pull", "handle-pull-side"" "handle-pull", "lever-pull", "stick-pull"
CALVIN:
Articulated: "move slider left", "open drawer", "close drawer", "move slider right"
Press: "turn off led", "turn led", "turn on lightbulb", "turn off lightbulb"
Lift: "lift blue block slider", "lift pink block table", "lift red block slider", "lift red block table",
"lift pink block slider", "lift blue block table"
Push: "push pink block right", "push blue block right", "push red block left", "push pink block
left", "push red block right", "push blue block left", "push into drawer"
Rotate: "rotate red block right", "rotate red block left", "rotate pink block left", "rotate pink block right", "rotate blue block right", "rotate blue block left"
C Further Policy Details
C.1 Prompt Details
In addition inputting the task instruction the LLM, we also format the instruction with prompt. We base our prompt off the prompt used in LLaVA. For all continuous control tasks, we use the prompt template "Prompt: control the robot. USER: <INSTRUCTION> ASSISTANT: discrete action space tasks, we describe the available actions to the agent in the prompt well. For Aa,yAL, this is the prompt template "Prompt: Control the red triangle complete the instruction using left, right, forward, pick, drop and toggle. USER: <INSTRUCTION= ASSISTANT: ". For Language Rearrangement, this is the prompt template "Prompt: You are a home robot assistant. Your possible actions are: pick object, place receptacle, nav receptacle, open receptacle, close receptacle, STOP. Objects: ball, clamp, hammer, screwdriver, padlock, scissors, block, drill, spatula, spoon, plate, sponge, cleanser, plum, pear, peach, apple, lemon, can, box, banana, strawberry, lego, cube, book, bowl, cup, mug, orange, lid, toy, wrench. Receptacles: chair, black table, brown table, TV stand, sink, right counter, left counter, sofa, fridge, left drawer, right drawer, middle drawer. USER: <INSTRUCTONN> ASSISTANT:
For
knife,
C.2 Action Space Adapter Details
We use the same ASA details between all environments. We detail the architecture and training decisions for the different ASAs when applicable.
VQ: Use codebook size of 512 with 512 dimensions per codebook element. These 512 tokens mapped to token indices 31000 31512 from the LLaMA language modeling head. The encoder and decoder networks for predicting the latent and decoding from the latent 4 layer MLP networks with hidden size 2048 using ReLU activations. The VQ network is trained on the actions in the same dataset used train the policy. The network is trained with MSE loss reconstruct the original actions. We VQ network for epochs over the dataset.
RVQ: Use all the same details VQ, but with Residual-Ve VQ that uses 2 codebooks.
17

---

Hyperparameter
CALVIN
Meta- World
BabyAI
HabPick
LR
3e
3e
3e
3e
Optimizer
AdamW
AdamW
AdamW
AdamW
Number of Epochs
3
20
20
Batch Size Per GPU
32
32
8
32
Context Length
12
3
32
3
Max Gradient Norm
Table 3: Hyperparameters for all imitation learning experiments. Most hyperparameters are the same between environments but the number of training epochs, context length and batch size per GPU are adjusted to fit the need for history, environment dataset size and task complexity.
Pred:
We use 2 layer MLP network with hidden size of 2048 and ReLU activations. We use this same MLP network architecture for discrete and continuous action space tasks. In the robot manipulation tasks, we also found it useful to include the robot proprioception as input to the MLP network and included this as input to the network layer. The robot proprioception consists of the robot robot joint angles and the gripper state. This ASA requires no separate training.
Uniform: In the tasks we consider, the actions are already normalized be in [-,, We then create 512 evenly spaced bins within this interval and assign each action dimension based on which bin it is within. Like with VQ, we assign the 512 tokens indices 31000 31512 from the LLaMA language modeling head. This ASA requires no separate training.
Lang: Starting from the same semantic tokenization with SemLang, remap each token the token corresponding to digit "0" to "9". Therefore, the token count per action is the same between Lang and SemLLagg but the Lang action tokens have semantic meaning being just digits.
C.3 Training and Architecture Details
use all pretrained components from LLaVA. For the visual token downsampler, we use a layer Perceiver network [55] with 4 output latents and hidden size 4096.
We detail the hyperparameters used for imitation learning in Table 3. We trained with the HuggingFace Transformers library [73]. PyTorch [74], DeepSpeed [75]. For reinforcement learning, we use learning rate 3e -4, 32 steps per rollout. parallel environment workers per GPU, an entropy coefficient of ()01, 2 epochs over the data batch per rollout, 6 PPO minibatches, a maximum gradient norm 0.2 and y = 0.99.
We train the CALVIN, Meta-World and HabPick imitation learning results on a 4xA40 GPU setup. We train the Language Rearrangement and BabyAl experiments on 8xA100-SOGB GPU setup.
We train the LLM weights with LoRA and fine tune the entire ASA and downsampler module. For LoRA we use rank value 128, alpha parameter 32 and dropout 0.1.
D Qualitative Results
See Figure 8 for qualitative results of results from Figure 3. The RVQ ASA is visualized for Meta-World, CALVIN and Habitat Pick. SemLang is visualized for Language Rearrangement.
E Per-Task Breakdown
In this section, we show results for each environment by task type. Table 4 shows performance on Language Rearrangement for each of the evaluation datasets. Table shows performance on CALVIN for each of the CALVIN tasks. Table 6 shows performance on BabyAl for each of the BabyAl instruction types. Table shows performance on Meta- World for each of the 45 Meta- Meta-World task types.
18

---

Meta-Worid Success: GGasp stok and pul box with the stick."
Meta World Falure:" "Pul handle
(a) The robot successfully picks the stick and pushes (b) The robot only partially lifts the handle and fails the box to the goal position.
lift it up all the way.
GALVIN Sucress: "Lift the çink block"
CALVIN
(c) The robot grasps the pink block and lifts goal height.
(d) The robot attempts grasp blue block but grasps too high, failing to pick the block.
Habiist Pick Success: "pick cleener"
Habxtat Pick Faluuw: picck lemon""
(e) The rohot moves closer cleaner bottle with (f) The rohot correctly finds the lemon the sink, but
its base and moves the arm to grasp the cleamer. It the tight sink receptacle results in the arm colliding with then returns the end-effector the resting position to the sink and the episode terminating due excessive collisions.
successfully end the task.
Langit "Bring pour
LangRt Falure:
"Move a strawberrty from the TV stand to the right of the left counter"
(g) The robot searches the house, eventually finds the (h) The robot picks the strawberry navigates the mug and then brings the blue table.
counter area, but puts the strawberry on the right counter as opposed the correct receptacle of sink.
Figure 8: Qualitative visualizations of successes and failures from the results in Figure of the main paper. The RVQ action space adapter is visualized for Meta- World, CALVIN and Habitat Pick. SemLang is visualized for Language Rearrangement.
Astresstod Generoiizaionn
Per thaae Brokdans
Pecepheesti
Multiple
Irrdevunt
Spotial
Conditional
Ofjects
Expresiene
Sentlang
06:s
Lang
37
31
24
72
58
74
76.39
21..4
10 10
12
30.00
0.e
2
26
Pied
38
28
95
92 12
47
26
3622
32
0.1
8
39
Table 4: Evaluation results at 20M steps of RL training for all results in Language Rearrangement
We show averages and standard deviations over 2 random seeds of full policy training.
19

---

RVQ
Pred
VQ
Uniform
CALVIN
72
68
56
28
turn off led
50
96
36
16
move slider left
99
100
100
15
rotate red block right
54
17
35
17
open drawer
100
100
56
100
rotate red block left
31
14
14
14
push pink block right
31
100
51
14
push blue block right
42
27
35
20
push red block left
68
36
61
push pink block left
47
50
86
14
push red block right
35
35
push blue block left
56
27
47
14
push into drawer
49
34
rotate pink block left
76
73
73
16
turn on lightbulb
80
34
rotate pink block right
30
73
19
10
rotate blue block right
28
13
13
turn off lightbulb
76
19
19
12
lift blue block table
34
25
34
16
close drawer
100
100
100
70
rotate blue block left
32
11
38
20
move slider right
100
100
100
19
turn on led
31
100
42
14
lift blue block slider
32
22
51
15
lift pink block table
66
68
82
11
lift red block slider
56
22
41
lift red block table
45
53
15
15
lift pink block slider
75
12
62
12
Table 5: Breakdown on every CALVIN task. Note there are not an equal proportion of all tasks in the evaluation dataset.
SemLang
Lang
Pred
goto
90
90
pickup
60
35
35
open
26
21
putnext
8
5
7
pick up seq go to
12
Table 6: Breakdown on every BabyAl task.
20

---

RVQ
Pred
VQ
Uniform
Meta- World
84
61
58
75
assembly
100
70
10
basketball
90
70
60
100
button-press-topdown
100
90
40
100
Watton-press-topdownwaall
100
100
60
90
button-press
100
100
70
100
button-press-wal
100
100
100
100
coffee-button button
100
100
100
100
coffee-puil
100
40
30
50
coffee-push
80
20
30
80
dial-turn
100
50
40
100
disassemble
60
30
30
50
door-close
100
100
100
100
door-open
100
100
100
100
drawer-close
100
100
100
100
drawer-open
100
100
60
100
faucet-open
100
100
100
100
faucet-close
100
90
100
60
hammer
100
40
handle-presd handle-press-side
100
100
100
100
handle-press
100
100
100
handle-pull
40
10
10
10
handle-pull
70
20
50
lever-pull
60
40
50
40
peg-insert-side
60
70
0
40
peg-unplug-side
50
30
100
90
pick-out-of-hole
50
90
40
30
pick-place
80
20
40
60
pick-place-war wall
80
20
40
30
plate-slide
100
60
40
100
plate-slide-side
100
100
100
90
hate-slide-backe
100
90
100
-late-slide-bak--eee
100
20
100
100
push-back
50
30
20
20
push
60
20
70
90
push-wall
80
40
60
100
reach
30
20
10
70
reach-wall
80
80
shelf-place
50
20
10
10
soccer
40
stick-push
100
60
10
100
stick-pull
90
50
30
100
sweep-into
70
40
60
70
sweep
90
30
50
window-open
100
100
100
100
window-close
100
100
100
100
Table 7: Breakdown every Meta-World task.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, claims in the abstract and introduce experimentally verified in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, mention limitations in Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, we describe all experimental settings detail in Appendix C.3, we build open-source models and benchmarks and we plan to open-source the code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification: We train and evaluate our method only on open benchmarks and train from an open source model. We also plan to release our code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (uttps://nips.. co/
http:/guddee//ooeuuuiiisssssssse onoooicooollyyoooooooo for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/nips. wwtpppcwcceeeeeeeccceccccceeeeee wewggueeeoeesuieeeeeeee weweeuummssiioPoooi piionnolcyy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Details provided in Appendix C.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In our reinforcement learning results in Section 4.3 we show results over 2 seeds and display the standard deviation between seeds as shaded region in the plot. For other experiments we only report results on I seed since we are finetuning 7B parameter models, making computationally intensive run multiple seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, complete details are proivded in Appendix C.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https pestps//nnett /ttp::///......eeeeeeeeeee cwttrrisscccppppeep
Answer: [Yes]
Justification: Yes, the paper adheres to the ethics code.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper on foundational research not tied to any particular application and we do not feel it is important to highlight any negative societal impacts of our work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
25

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all datasets used.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwode.hho....c. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
26

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "uWwV7cu1Jp", "reviewer": "Reviewer_f4rS", "summary": "The paper empirically studies how to properly ground MLLMs into embodiments, with a particular focus on the action representations, including the continuous and discrete solutions. The authors conduct a thorough study on 7 methods, encompassing over 114 tasks. The research indicates that for continuous actions, optimal results are achieved by learning action tokens that precisely represent the action distribution (RVQ). For discrete actions, superior outcomes are obtained by aligning actions with the original semantic token space.", "questions": "Please see the Weaknesses.", "limitations": "The authors discussed limitations in Sec. 5.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. How to properly ground MLLMs in embodied tasks is important and under-explored.\n2. The paper is the first to systematically and comprehensively study the optimal recipe for action tokens.\n3. The conclusions drawn from the empirical studies could provide guidance for subsequent research.", "weaknesses": "1. All experiments are performed on LLaVA and LoRA. Conclusions may not be applicable to other MLLM architectures or scales.\n2. Based on existing conclusions, the quality of the paper could be further enhanced if the authors could suggest whether to use continuous or discrete methods, how to break through the accuracy limit of current methods, or where future work should focus on improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and suggestions. We address the reviewer’s points below.\n\n**1. Conclusions may not be applicable to other MLLM architectures or scales.**\n\nIn the main rebuttal PDF, we show results with a different MLLM (PaliGemma [1]) and a different finetuning method (full finetuning as opposed to LoRA). Firstly, we swap the LLaVA MLLM with the recently released PaliGemma model, which uses a different LLM and visual encoder architecture and a higher visual resolution at 448 pixels per dimension. We finetune all 3B parameters of the PaliGemma MLLM and achieve 92% success rate in Meta-World, higher than 84% success rate with LLaVA and LoRA finetuning. Secondly, we finetune all 7 billion parameters of the LLaVA MLLM with the best performing RVQ ASA in the same Meta-World setup as from Section 4. This full finetuning achieves 85% success rate, which is slightly higher than the 84% success rate when finetuning 140 million parameters with LoRA. \n\nThese initial results in Meta-World indicate that our conclusions around RVQ being a strong action space adapter extend to other MLLMs and finetuning methods.\n\n**2. Where should future work focus on improvements?**\n\nOne improvement of the results in the current analysis is to improve the generalization performance to unseen instructions through the core MLLM understanding. For example, even the best action space adapters struggle with generalizing to new spatial instructions (see the low “putnext” task performance in BabyAI in Table 6 and the low “Spatial” split for LangR in Table 4). Another area of improvement is the ability to handle very precise action sequences. As seen in Figure 4, which breaks down the performance by interaction type, MLLMs struggle to do precise manipulation tasks like rotating or pushing. We will add these to the discussion at the end.\n\n**References:**\n\n[1] Beyer, Lucas, et al. \"PaliGemma: A versatile 3B VLM for transfer.\" arXiv preprint arXiv:2407.07726 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WqCsdlFzAL", "reviewer": "Reviewer_YrxP", "summary": "In this paper the authors present a way to adapt a Vision and Language model to perform action execution tasks in embodied environments. Specifically, systematically evaluate different ways of predicting actions both on task having both discrete and continuous action spaces. Thanks to this evaluation, it is possible to assess which are the most performant action prediction losses to use for different use cases. According to the authors' results, approaches based on VQ-VAEs are able to obtain the best performance in many tasks.", "questions": "1. I believe that a reader would appreciate the details of how you have arranged the dataset to train the VQ-VAE variants. The appendix contains some information but it doesn't fully address the nature of the data (e.g., \"What is each example?\").\n\n2. Why did you decide to follow the work from Carta et al. considering that it uses only a subset of the trajectory instead of the BabyAI benchmark which contains a range of different tasks of different complexity? Additionally, when you evaluate, are you sure that 100 episodes are enough to experience different configurations from the ones you used at training time?\n\n3. Could you please clarify what is the language instructions for environments that do not have one such as Meta-World?", "limitations": "The authors acknowledge the limitations of their experimental setup rather than the limitations of their work more broadly. Specifically, I would recommend the authors consider the societal impact of developing embodied AI models. For instance, it would be important to acknowledge that these models are still far from being deployed in real-world scenarios considering that their performance, even on simple grid-world-like environments, is barely over the 30% success rate bar.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. One of the first papers that finally sheds light on the different approaches to performing action prediction in embodied environments. This is the most important contribution of this paper and I believe it will be really useful to refer to this set of experiments for future research. \n\n2. They propose a VQ-VAE variant to generate latent codes for encoding actions. These codes can be intended as a way to learn \"latent bins\" to cluster the action space. Additionally, they propose a variant of this model based on the RVQ-VAE architecture to model a set of codebooks that are used to generate more precise actions.", "weaknesses": "1. The VQ-VAE variants are indeed really interesting and novel. However, I find the description of this method a bit unsatisfactory because it omits some details regarding \"how\" you train these models. See my question below for details. \n\n2. The authors chose a good set of tasks for their evaluation however I believe that a benchmark that would have been perfect for this work is VIMA-Bench because of its focus on systematic generalisation. CALVIN somehow offers this but I don't think it's as systematic as VIMA-Bench.\n\n3. For discrete action spaces they use BabyAI however they only test with a very limited size of the grids and with few tasks. Please see my questions related to this point as well. \n\n4. Some related work missing: \n\n- Team, Octo Model, et al. \"Octo: An open-source generalist robot policy.\" arXiv preprint arXiv:2405.12213 (2024).\n- Pantazopoulos, G., Nikandrou, M., Parekh, A., Hemanthage, B., Eshghi, A., Konstas, I., ... & Suglia, A. (2023, December). Multitask Multimodal Prompted Training for Interactive Embodied Task Completion. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 768-789).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and suggestions. We address the reviewer’s points below.\n\n**1. Details on VQ-VAE training.**\n\nThank you for the suggestion; we will add these details to the paper. We train the VQ-VAE learned tokenizers to reconstruct actions from the same dataset used for supervised finetuning. Specifically, we randomly sample a batch of actions from the supervised finetuning dataset. A batch of single actions is then encoded by the VQ-VAE into the codebook elements. These discretized codes are then used to predict the original continuous action. Thus, the “examples” the VQ-VAE reconstructs are single continuous actions. The VQ-VAE is trained with the mean-squared error over the action reconstruction and commitment error. We train for 3 epochs over the dataset with a batch size of 32 and use the same AdamW optimizer and learning rate schedule as described in L248-250 of the paper. We will update Section 4.2 with all these details.\n\n**2. Using VIMA-Bench as a benchmark.**\n\nWe don’t benchmark on VIMA-Bench because its task specifications are multimodal, meaning they include videos and images, whereas our study focuses on language instruction-specified tasks. Future work can investigate how MLLMs can interpret multimodal instructions with text and images to complete tasks.\n\n**3. Clarify what are the language instructions for environments that do not have one such as Meta-World.**\n\nFor Habitat Pick, the instruction is “pick a” followed by the name of the instruction, e.g. “pick a mug” or “pick a plate” (L637). For Meta-World, we use the task descriptions provided by the Meta-World authors in Appendix Section A of the original Meta-World paper, e.g. “Sweep a puck off the table” is the instruction for the sweep task (L607). For CALVIN, LangR and BabyAI, we use the instructions exactly as provided by the benchmark implementation.\n\n**4. Acknowledge broader limitations of work.**\n\nWe thank the reviewer for pointing this out and will add the following the limitations in Section 6: “While our investigation of ASAs enables connecting a MLLM to various action spaces, the performance of these methods is still subpar for real-robot deployment where high success and safety are critical. MLLMs with the best ASA still struggle on simple environments like BabyAI, only achieving 40% success rate. Further work is needed to improve the performance of these methods for real-world usage.” \n\n**5. Why evaluate on Carta et al. BabyAI variant?**\n\nWe evaluate with this BabyAI variant [1] because it is built for generalization to unseen language instructions. The standard BabyAI benchmark has no separate train and test split for language instructions. The Carta et al. variant evaluates generalization to unseen synonyms. We use this version to better test the language understanding capabilities of MLLMs.\n\n**6. Are you sure that 100 episodes are enough to experience different configurations from training ones?**\n\nYes, we believe 100 episodes are sufficiently covering differences to test generalization since these episodes all have unseen instructions from training by replacing words with synonyms. We also change the environment random seed from that used to generate the training demonstrations. \n\n**7. Missing related work.**\n\nWe thank the reviewer for pointing out these relevant works and will update our related work to include them.\n\n**References:**\n\n[1] Carta, Thomas, et al. \"Grounding large language models in interactive environments with online reinforcement learning.\" International Conference on Machine Learning. PMLR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aZCSkFCyur", "reviewer": "Reviewer_eGot", "summary": "This paper studies how to \"ground\" multimodal LLMs (MLLMs) to the action spaces of agents with various embodiments. The authors examine \"Action Space Adapters\" (parameterization strategies; ASAs) for various embodiments and MLLMs and identify principles for constructing ASAs based on the target action space. The authors consider 5 embodied AI environments (3 continuous and 2 discrete) and over 100 tasks.", "questions": "1) Can you be clear about the difference between SemLang and Lang? The way it is written right now it reads as if the only difference between the two is that there are different numbers in the sequence being predicted. But since SemLang is predicting tokens that correspond to words, I assume that there is some underlying embedding that when softmaxed produced the token index. In that case, where to the numbers in the vocabulary of Lang come from and how are they being predicted if it's not using an underlying semantic (embedding) representation like SemLang?\n\n2) Section 3.3: \"In the complex environments we consider, the zero-shot performance of MLLM is poor across all ASAs, even with detailed prompts.\" Is this demonstrated anywhere, either in this paper (in the appendix perhaps?) or elsewhere (if elsewhere, please provide the citation)?\n\n3) \"We train with supervised finetuning for CALVIN, Meta-World, HabPick, and BabyAI. We train with reinforcement learning on Language Rearrangement.\" Can you please explain why this does not result in an invalid comparison between the environments, if they are trained for each environment using a) different methods and b) to the specifications of the environment themselves? Does this not risk overfitting the ASA to the environment and the training method? Or am I misunderstanding the goal here?\n\n4) How precisely do the differences in embodiments affect the results? Are the embodiments those provided out of the box in each environment or do you do any adjustment or controlling for the effects of the differing embodiments?", "limitations": "The only limitations the authors mention is that they use only a single MLLM and the data collection requirements on RVQ and SemLang. These are good to mention but does not constitute a serious discussion of limitations IMO. There are differences in the way the models are trained/fine-tuned (see Q3 above) that complicate the axes of comparison, as do the wide variety of experiments conducted. Some discussion on the limitations of the analysis would also be welcome.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper addresses an important issue for the future generalization of MLLMs to embodied environments and different action spaces. The authors conduct a variety of experiments in different environments and action spaces.  The motivation for the method is given a theoretical grounding, and the experimental results provide concrete recommendations for adapting MLLMs to novel action spaces and environments. The combination of theoretical motivation and actionable takeaways is a strong contribution.", "weaknesses": "Overall the paper is very dense. This is a complex topic so understandably there is a log packed into 9 pages and as a result there are many parts of the paper that are hard to follow. There are a fair number of parts of the paper that are not clearly written, which impacts my scores. If the authors' rebuttal clarifies these points satisfactorily, I would be willing to reconsider my evaluation. The major points are given in Questions.\n\nSome other issues are:\nSome terms are not clearly defined for the reader. For example, \"codes\" as used in Figure 3. (Again, if I simply missed where this was provided, please point it out). Similarly for \"codebook\".\n\nNo specific example outputs are provided. Some examples of environments are provided in the appendix but there's no clear discussion of, for example, adaptations that the best performing methods get right or wrong. Providing these would help clarify the definitions issue and make the problem much more grounded (ironically) than the plot and charts in the paper currently allow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating the technical motivation, importance of the problem, and the exhaustive experimentation. The reviewer's clarification questions are addressed below and will be added to the final manuscript, which we believe will greatly improve it.\n\n**1. No specific example outputs are provided.**\n\nIn the rebuttal PDF, we added visualizations of success and failure examples for the best performing action space adapter in the environments. We will include these visualizations in the paper. \n\nWe also highlight the existing Figure 4 which analyzes adapting to different interaction types in CALVIN and Meta-World. This plot shows that RVQ struggles to adapt to rotating and pushing tasks in CALVIN.  Additionally, in Appendix E.1, we provide a per-task breakdown of performance, illustrating which tasks models struggle on. For example, Table 4 shows that SemLang struggles with generalizing to new instructions with spatial relationships. We will update Section 4 of the paper to summarize these qualitative analyses.\n\n**2. Some terms are not clearly defined for the reader. For example, \"codes\" and \"codebook\".**\n\nThank you for pointing this out. We will update L164-179 to clarify the definition of \"codes\" and \"codebook\".  In the VQ-VAE architecture [3], the \"codebook\" is defined as a matrix $V \\in \\mathbb{R}^{N \\times d}$ where $N$ is the size of the codebook and $d$ is the dimension of each codebook element. The \"codes\" are the $d$-dimensional row vectors of $V$. The VQ-VAE encoder network $f_\\theta$ maps a continuous action $a$ to a vector in $\\mathbb{R}^d$ and then quantizes this vector to the closest of the $N$ row vectors in $V$. Thus, each action is mapped to a single \"code\" from the \"codebook\" by the VQ-VAE encoder (or several codes in the case of Residual VQ-VAE). \n\n**3. Can you be clear about the difference between SemLang and Lang?**\n\nWe will incorporate in L136-L149 the following clarification: \"For Lang and SemLang, we represent each action $a\\in A$ as a short text $w(a)$. The difference between the two options is how this text is selected. In the case of SemLang, we pick a text that is representative of the action itself. For example, if we work with high-level actions and action $a$ represents picking an apple, then the corresponding text is $w(a)=\\text{\"pick apple\"}$. In the case of Lang, we map each action to a randomly chosen text. In our implementation, this is a sequence of integers. Note that we can pick any text for this mapping and the choice of integers is arbitrary. However, the selected text is not semantically representative of the action. To connect it with the underlying MLLM, we tokenize the action text with the MLLM text tokenizer.\"\n\n**4. Is poor zero-shot MLLM performance demonstrated?**\n\nThe reviewer is correct that our paper does not empirically validate this claim, so we will remove it from the paper. We note that prior works [1,2] that adapt MLLMs for continuous control also only show results of finetuning MLLMs without considering zero-shot performance. \n\n**5. Why does training with supervised and reinforcement learning not result in an invalid comparison between environments?**\n\nThe reviewer is correct that we do not compare across environments. We only compare across different ASAs while keeping environments, learning settings, and learning algorithms fixed for these comparisons. However, we provide a large number of environments (6 altogether) and two training schemes (RL and supervised learning) for which the different ASA exhibit similar comparative properties, which reinforces the correctness of our conclusions. For example, when we compare which method performs best on LangR, all methods are trained with reinforcement learning, and on BabyAI, all methods are trained with supervised learning. We do not compare LangR to BabyAI success rates; we only compare success rates in the same setting. Our goal is to analyze the relative rankings of which ASA performs best across a variety of environments to derive a conclusion about which ASA is best. \n\n**6. How do the differences in embodiments affect the results?**\n\nThe embodiments are those provided out of the box for each environment. These affect the results by affecting how the agent interacts with the environment via the action space. The three continuous control environments have different robots with different degrees of freedom (DoF) in the action space which affects how methods perform. For example, the \"Uniform\" ASA is greatly affected by the embodiment's action dimension, with it performing well with MetaWorld with 4D actions but performing worse as the action dimension increases on CALVIN and Habitat Pick, respectively. RVQ performs the best consistently, regardless of the action dimension. We will explicitly add these embodiments properties to Section 4.1.\n\n**7. Lacking serious discussion of limitations.**\n\nWe thank the reviewer for pointing this out and will update our Limitations in Section 5 with the following text: \"While our investigation of ASAs enables connecting a MLLM to various action spaces, the performance of these methods is still subpar for real-robot deployment where high success and safety are critical. MLLMs with the best ASA still struggle on simple environments like BabyAI, only achieving 40% success rate. Further work is needed to improve the performance of these methods for real-world usage. Our investigation also only studies adapting MLLMs through behavioral cloning or on-policy RL. Future work can investigate if the choice of ASA varies when adapting the MLLM with other learning algorithms such as off-policy RL or offline RL.\"\n\n**References:**\n\n[1] Li, Xinghang, et al. \"Vision-language foundation models as effective robot imitators.\" 2023. [2] Brohan, Anthony, et al. \"Rt-2: Vision-language-action models transfer web knowledge to robotic control.\" 2023. [3] Van Den Oord, Aaron, and Oriol Vinyals. \"Neural discrete representation learning.\" 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their useful and insightful feedback. Reviewers highlighted that our study is one of the first on how to properly ground MLLMs in embodied action (f4rS, YrxP) and provides concrete and actionable takeaways that will be useful for future research (YrxP, eGot). We highlight that our rebuttal added:

1. New results showing our RVQ tokenization works for PaliGemma in addition to the LLaVA results in the paper and full finetuning in addition to the LoRA results in the paper in Table 1 of the rebuttal PDF (f4rS). 

2. Expanded on the limitations section in regards to the learning algorithm (eGot), performance for real-world deployment (YrxP), and how future work can improve performance (f4rS).

3. Included qualitative examples of successes and failures in Figure 1 of the rebuttal PDF (eGot).

4. Clarified the details of the VQ-VAE training and architecture (eGot,YrxP). 

We will add these new results and details to the main paper, and believe these strongly address all of the major concerns of the reviewers. We again thank the reviewers for their great suggestions.

Meta-review: This paper provides a comparative analysis of various action space adaptors that enables the use of VLMs for embodied agents. The reviewers have agreed that this work will be valuable for future research in the field, leading to the decision to accept this paper.

However, the reviewers have raised several concerns about the current form of the paper. The authors are strongly encouraged to address these issues as outlined in their official comment. Addressing these concerns will enhance the paper's clarity, rigor, and overall impact on the research community.
Decision: Accept (poster)