RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulation
Authors: Jiaming Liu
Mengzhen Liu
Zhenyu Wang
Pengju An
Xiaoqi Li
Kaichen Zhou
Senqiao Yang
Renrui Zhang
Yandong Guo
Shanghang Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Robot Manipulation
Multimodal Large Language Model
State Space Model
Primary area: robotics

Abstract: A fundamental objective in robot manipulation is to enable models to comprehend visual scenes and execute actions. Although existing Vision-Language-Action (VLA) models for robots can handle a range of basic tasks, they still face challenges in two areas: (1) insufficient reasoning ability to tackle complex tasks, and (2) high computational costs for VLA model fine-tuning and inference. The recently proposed state space model (SSM) known as Mamba demonstrates promising capabilities in non-trivial sequence modeling with linear inference complexity. Inspired by this, we introduce RoboMamba, an end-to-end robotic VLA model that leverages Mamba to deliver both robotic reasoning and action capabilities, while maintaining efficient fine-tuning and inference. Specifically, we first integrate the vision encoder with Mamba, aligning visual tokens with language embedding through co-training, empowering our model with visual common sense and robotic-related reasoning. To further equip RoboMamba with SE(3) pose prediction abilities, we explore an efficient fine-tuning strategy with a simple policy head. We find that once RoboMamba possesses sufficient reasoning capability, it can acquire manipulation skills with minimal fine-tuning parameters (0.1\% of the model) and time. In experiments, RoboMamba demonstrates outstanding reasoning capabilities on general and robotic evaluation benchmarks. Meanwhile, our model showcases impressive pose prediction results in both simulation and real-world experiments, achieving inference speeds 3 times faster than existing VLA models.

RoboMamba: Efficient Vision-Language-Actionn Model for Robotic Reasoning and Manipulation
Jiaming Liu', Mengzhen Liu', Zhenyu Wang Pengju An', Xiaoqi Li
Kaichen Zhou', Senqiao Yang', Renrui Zhang Yandong Guo?, Shanghang Zhang
State Key Laboratory Multimedia Information Processing.
School of Computer Science, Peking University; RAI'Robo;ss;
'Beijing Academy of Artificial Intelligence (BAAI)
wtwpiniiiissuuuuu.euuu.u 2125 182 @ bjtu.edu.cn, shanghang @ pku.eduu.cc
Abstract
A fundamental objective in robot manipulation is to enable models to comprehend visual scenes and execute actions. Although existing Vision-Language-Aeee Action (VLA) models for robots can handle range of basic tasks, they still face challenges in two areas: (I) insufficient reasoning ability to tackle complex tasks, and (2) high computational costs for VLA model fine-tuning and inference. The recently proposed state space model (SSM) known as Mamba demonstrates promising capabilities in non-trivial sequence modeling with linear inference complexity. Inspired by this, we introduce RoboMamba, an end-to-end robotic VLA model that leverages Mamba deliver both robotic reasoning and action capabilities, while maintaining efficient fine-tuning and inference. Specifically, we first integrate the vision encoder with Mamba, aligning visual tokens with language embedding through co-training, empowering our model with visual common sense and robotic- related reasoning. To further equip Robo with SE(3) pose prediction abilities, we explore an efficient fine-tuning strategy with simple policy head. We find that once RoboMamba possesses sufficient reasoning capability, it acquire manipulation skills with minimal fine-tuning parameters (0. 1 % of the model) and time. In experiments, RoboMamba demonstrates outstanding reasoning capabilities general and robotic evaluation benchmarks. Mcanwhile, our model showcases impressive pose prediction results in both simulation and real-world experiments, achieving inference speeds 3 times faster than existing VLA models. Our project web page: httss///tte..oooll..ooo/////////
Introduction
The scaling up of data has significantly propelled research on Large Language Models (LLMs) [+33. showcasing notable advancements in reasoning and generalization abilities within Natural Language Processing (NLP). To comprehend multimodal information, Multimodal Large Language Models (MLLMs) 48 g have been introduced, empowering LLMs with the capability of visual instruction- following and scene understanding. Inspired by the strong capabilities of MLLMs in general settings, recent research aims to incorporate MLLMs into robot manipulation. On the one hand, some works 12 enable robots comprehend natural language and visual scenes, automatically generating task plans. On the other hand, Aision-Language Action (VLA) models [33155 leverage the inherent capabilities of MILMs, empowering them with the ability predict low-level SE(3) poses.
'Project Lead, Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Vision
Language
Inference Speed
RoboMamba
9.0 Hz
P
Tokenizer
Question: Predict the
Mamba Language Model
contact point...
3,4 Hz
t
+
Hz
Vision
Open
Encoder
Fine-tuned Parameters
7.0B
Reasoning
Policy Head
9.0 Hz
Reasoning
Manipulation
41.3MI
Q: current goul ist Take out fruits from the bowl. last 20 steps: put pear
Q: Given <image> Predict the contact point and orientation pulling the [object]
table uoseddr put apple on the table next steps?
Robo HooMammmaaheee 2D
OpenTA
MasplEM
I-place the lid on the jar 2-done
LLaMA- A-dapperr
RoboMamba-A: I- place li the pistuchios cup 3- dome
arrange
Contact point: [x. z/
LKF. sh. of (xf, sf. IfI
Rotation matrix: Hon, yu,
OpenVL.A (7B)
RoboMamba (2.8B)
ManipLLM (7B)
Figure 1: Overview of RoboMamba. RoboMamba is efficient robotic VLA model that combines reasoning and manipulation capabilities. First, we integrate and align a vision encoder with the Mamba LLM, endowing our model with common sense and robotic-related reasoning abilities. Subsequently, we introduce an efficient fine-tuning strategy to equip RoboMamba with pose prediction abilities, requiring few dozen minutes to fine-tune a simple policy head (3.7M parameters). In terms of inference speed, RoboMamba achieves the highest control frequency, surpassing other VLA models, running on an NVIDIA A A100 GPU without any quantization or inference acceleration techniques. More real-world downstream tasks are displayed in Figure 4 and Figure 5
Robot manipulation involves interacting with objects in dynamic environments, requiring human- like reasoning abilities to comprehend the semantic information of scenes [T]] [6]. alongside a robust low-level action prediction ability [17 18]. While existing MLLM-based policies can handle a range
of basic tasks, they still face challenges in two aspects. First, the reasoning capabilities of pre-trained MLLMs 6 [9] in robotic scenarios are found to be insufficient.
shown Figure [ (reasoning
example), this deficiency presents challenges for fine-tuned robot MLLMs when they encounter complex reasoning tasks. Second, fine-tuning MLLMs and using them to generate robot manipulation actions incurs higher computational costs due to their expensive attention-based LLMs (20 21). To balance the reasoning ability and efficiency, several studies 24 have emerged in the field of NLP. Notably, Mamba (25) introduces the innovative selective State Space Model (SSM), promoting context-award reasoning while maintaining linear complexity. Drawing inspiration from this, we raise question: "Can develop efficient robotic VLA model that possesses strong reasoning capabilities while also acquiring robot manipulation skills in a cost-effective manner?"
To address this, we propose RoboMamba, an efficient robotic VLA that empowers the Mamba LLM to achieve robust robotic reasoming and action capabilities. As shown in Figure [ we initially integrate vision encoder (e.g., CLIP (26) with Mamba to empower RoboMamba with visual common sense and robotic-related reasoning. Specifically, we proceed with alignment pre-training. activating the cross-modal connector 4 [9 to convert visual information into Mamba's token embeddings.
We then unfreeze Mamba for instructions co-training, utilizing its powerful sequence modeling to comprehend high-level robotic and general instruction data. On top of this, to equip Moboa with SE(3) pose prediction abilities, we explore an efficient fine-tuning strategy with a simple policy head. Notably, we discover that once RoboMamba possesses sufficient reasoning capabilities, can acquire pose prediction skills with minimal parameter fine-tuning. The fine-tuned policy head constitutes only 0.1% of the model parameters, which is 10 times smaller than existing robotic VLA approaches [S] 14] this way, RoboMamba can simultaneously generate robot reasoning using language responses and predict end-effector poses via the policy head.
To systematically evaluate our proposed RoboMamba, we conduct extensive experiments both simulation and real-world scenarios. First, we assess reasoning abilities on general and robotic

---

evaluation benchmarks. RoboMamba, with only 3.2B parameters, achieves competitive performance on several MLLM benchmarks and also delivers promising results on RoboVQA (42.8 BLEU-4) (27). With its strong reasoning abilities, RoboMamba achieves state-of-the-ar (SOTA) manipulation per- formance in the SAPIEN simulation [28], requiring only a 7MB policy head and few dozen minutes single A 100 GPU. Moreover, RoboMamba achieves an inference speed that is 3 times faster than previous robotic VLA models [29] [S]. Additionally, we evaluate RoboMamba in real-world scenarios, where it can generate long-horizon planning and predict the end-effector pose for each atomic task. In summary, our contributions are as follows:
of fine-tuning
We introduce RoboMamba, an efficient VLA model that integrates vision encoder with the linear-complexity Mamba LLM, which possesses visual common sense and robotic-related reasoning abilities.
To equip RoboMamba with action pose prediction abilities, we explore an efficient fine- tuning strategy using simple policy head. We find that once RoboMamba achieves sufficient reasoning capabilities, can acquire pose prediction skills with minimal cost.
In our extensive experiments, RoboMamba excels in reasoning on general and robotic evaluation benchmarks, and showcases impressive pose prediction results in both simulation and real-world experiments.
2 Related work
State Space Models (SSMs). SSMs have become effective substitutes for transformers and CNNs due to their linear scalability with sequence length 30 23
Recent works [22 31 32 use the
state space robustly establish dependencies across long sequences. Especially, Mamba [25] designs the SSM matrices to be functions of the input, creating a learnable selection mechanism that improves adaptability and reasoning capabilities. 333/38] expand selective SSMs vision and video tasks. Furthermore, MambalR [39] focuses on image restoration, and PanMamba [40] addresses pan-sharpening. while DiS 44 integrates SSMs into diffusion models. These findings demonstrate that Mamba exhibits promising performance and efficiency in various visual downstream tasks. With the emergence of SSMs, we make the first attempt introduce Mamba to address critical challenges in robotics, which demands efficient action capabilities.
Multimodal Large Language Models. Large language models (LLMs) have exhibited remark- able reasoning capabilities across various downstream tasks [99 42 2 When addressing complex multimodal reasoning challenges, multimodal large language models (MLLMs) have shown excep- tional visual understanding, i.e., BLIP-2 43 OpenFlamingo [4] LLaMA-Adapter 19 45), and
LLaVA 46 Additionally, the introduction of 3D MLLMS 112 47 48 seeks expand the reasoning and conversational capabilities of LLMs to include the 3D modality. However, deploying LMMs expensive due to their significant computational overhead, primarily caused by their billions of parameters. To mitigate these challenges, recent small-scale models [49 [50] demonstrate impressive performance while maintaining manageable computational costs. VA-Phh 49) empowers the re- cently developed smaller LLM, Phi-2, for visual instruction tuning. TinyLLa [50] and Mobile VLM V2 51 demonstrate that high-quality training data and schemes can effectively compensate for the reasoning abilities of smaller LMMss. Furthermore, Cobra [52] innovatively utilizes an SSM-baser Mamba LLM to reduce complexity and improve inference speed on common sense reasoning tasks. Different from previous works, our goal is to develop an efficient Robotic VLA model using the SSM-based language model. This model not only possesses common sense understanding but also has the capability to complete manipulation tasks effectively.
Robot Manipulation. Traditional robotic manipulation employs state-based reinforcement Iearn- ing (53456). In contrast, [I]] (8--60) use state with visual observation as input and then make predictions. Specifically, Where2Act [1] takes visual observations and predicts on actionable pixels and movable regions in objects. Flowbot3d [57] predicts woint-wise motion flow on 3D objects. Anygrasp 17 employs point cloud data learn grasp poses on a large scale datasets. Inspired by the success of MLLMS in general scenarios 43 19 45 46), several VLA models [13] [6] explore utilizing their common sense reasoning capabilities to address manipulation problems. Palm- E [10 integrates multimodal encodings with LLMs, training them end-to-end for manipulation planning. VoxPoser [TT] extracts affordances and constraints from MI.LMS to further zero-shot predict trajectories. RoboFlamingo [4] fine-tunes MLLM on vision language manipulation dataset
3

---

to complete bunguage-conditioned manipulation tasks. ManipLLM [15] introduces specific training scheme for manipulation tasks that equips MLLMs with the ability predict end-effector poses. Van [22] enhancing robotic manipulation with physically grounded information processed by MILM. In this paper, instead of fine-tuning pre-trained MILLM, we introduce novel efficient VLA model that possesses both robotic-related reasoning and low-level pose prediction capabilities.
3 RoboMamma
In Section 3.1 wc introduce the preliminaries of our proposed RoboMamba, including the problem statement and description of the language model. Subsequently, in Section 3.2 and 3.3 we describe the architecture of RoboMammb and how we empower with common sense and robotic-related reasoning. In Section 3.4, we outline our proposed robot manipulation fine-tuning strategy, which equips our model with pose prediction skills by minimal tinettuning parameters and time.
3.1 Preliminaries
Problem statement. For robot visual reasoning. our RoboMamba generates language answer La based on the image R.W' NoWxxx' and the language question Tug denoted as L. R.g, The reasoning answer usually contains individual sub-tasks (L, + (L, L ........ LX) for one problem Lg. For example, when faced with planning question like 'How to clean the table?', the response typically includes steps such as 'Step Pick up the object' and 'Step 2: Place the object in the box'. For action prediction, we utilize an efficient and simple policy head x to predict an action a = (T((T, I. Following previous works 63 15),
we use 6-DoF to express the end-effector pose of the Franka Emika Panda robot arm. The 6-DoF includes the end-effector position Opos R representing 3D coordinate and direction adir E Rax3 representing rotation matrix. If training for a grasping task, we add gripper status to the pose prediction, resulting in a 7-DoF control.
State Space Models (SSMs). In this paper, we select Mamba 25 as our language model. Mamba consists numerous Mamba blocks, with the most crucial component being the SSM. SSMs (2T) are designed based on continuous systems, projecting the 1D input sequence xx(() E R_ into a 1D output sequence yt) E _² through hidden state (t) E R An SSM consists three key parameters: the state matrix A & the input matrix B E IR Nx1, and the output matrix C E Rx11 The SSM can represented as follows:
1'(t) Ah(t) Bx(t): Ch(t).
(I)
Recent SSMs (c.g., Mamba 25 are constructed as discretized continuous systems using a timescale
parameter A. This parameter transforms the continuous parameters A and B into their discrete counterparts A and B. The discretization employs the zero-order hold method, defined as follows:
A exp( AA),
(2)
B (AA) (exp (A) -1) AB
(3)
fr Ah, BI 1/: Chh.
Mamba introduces the Selective Scan Mechanism (S6) to form its SSM operator in each Mamba
block. The SSM parameters are updated to B E R BxLN C E RBxL and A E BXLLD
achieving better content-aware reasoning. The details the Mamba block are shown in Figure 2
3.2 RoboMamba architecture
To equip RoboMamba with both visual reasoning and manipulation abilities, we start from pre- trained Large Language Models (LLMs) 25 and visual models construct an effective VLA model architecture. As shown in Figure
we utilize the CLIP visual encoder [26] to extract visual features fn E Bx10001 from input images I, where and N represent batch size and tokens, respectively. In contrast to [64 [22] we do not adopt the vision encoder ensemble technique, which employs various backbones (i.e., DINOv2 65], CLIP-ConnNeett (66), CLIP-ViT) for image
feature extraction. The ensemble introduces additional computational costs that severely impact

---

Stage
Mamba Block
Mamha Language Model (n x Mamba Blocks)
Project Layer
SSM
General VQA
VIT
Linear
Activation
Linear
Stage
Robot VQA
Do Position Apos
Language Question
Tokenizer
MLP
Stage 2
Stage 1/ Stage 2
Pooling
|0
Execute
Manipu- lation
Stage Action
Direction Gatr
Frozen
Fine-tune
Stage
Alignment pre-training
Stage Instruction co-training
Stage Robot manipulation fine-tuning
CLIP
Pruject
Mamba
CLIP
Project
Mamba
CLIP
Project
Mamba
Overall framework of RoboMamba. RoboMamba projects images onto Mamba's language embedding using vision encoder and projection layer, which is then concatenated with text tokens and fed into the Mamba model. To predict the position and rotation of the end-effector pose, inject simple MLP policy heads and the global token as input, which generated through pooling operation from the language output tokens. Training strategy of RoboMamba. For model training, we divide our training pipeline into two stages. Stage 1, we introduce alignment tre-training (Stage 1.1) and instruction co-training (Stage 1.2) to equip RoboMamba with both common sense and robotic-related reasoning abilities. In Stage 2, we propose robotic manipulation fine-tuning to efficiently empower RoboMamba with low-level manipulation skills.
Figure 2:
the practicality of VLA model in the real world. Therefore, we demonstrate that a simple and straightforward model design can also achieve strong reasoning abilities when combined with high- quality data and appropriate training strategies. To enable the I.I. to understand visual features, we connect the vision encoder to the LLM using multilayer perceptron (MLP). Through this simple cross-modal connector, RoboMamba can convert visual information into language embedding space ft IR N Note that model efficiency is crucial in the field of robotics, robots need to respond quickly based on human instructions. Therefore, we select Mamba as our language model due to its context-aware reasoning ability and linear computational complexity. Text prompts are encoded into embedding space fe Bx x2660 using the pre-trained tokenizer, then concatenated (cat) with visual tokens and input into Mamba. We leverage Mamba's powerful sequence modeling to comprehend multimodal information and utilize effective training strategies to develop visual reasoning capabilities (as described in the next section). The output tokens T, then detokenized (det) to produce responses in natural language La. To equip our model with both reasoning and manipulation abilities, meticulously design comprehensive training pipeline, which is divided into two stages. We introduce the training recipes Stage in Section 3.3 and present the robot manipulation fine-tuning Section 3.4
3.3 General and robotic-related training
After constructing the RoboMamba architecture, the next goal train our model learn general vision and robotic-related reasoning abilities. As shown in Figure 2 we divide our Stage training into two steps: alignment pre-training (Stage 1.1) and instruction co-training (Stage 1.2). Specifically, unlike previous MLLM training methods [19 67 64], we aim to enable RoboMamba to comprehend both common vision and robotic scenes. Given that the robotics field involves numerous complex and novel tasks, RoboMamba requires enhanced generalization capabilities. Therefore, we adopt co-training strategy in Stage 1. 1,2, combining high-level robotic data (e.g., task planning) with general instruction data. We find that co-training not only Ieads to more generalizable robotic policies but
5

---

also enhances the majority of general scene reasoning abilities due to the complex reasoning tasks embedded in the robotic data (demonstrated in Appendix C). The training details are shown below:
Stage 1.1: Alignment pre-training. We adopt LLaVA [ filtered 558k image-text paired dataset for cross- modal alignment. As shown in Figure 2 we freeze the parameters of the vision encoder and Mamba language model, and only update the project layer. In this way, we can align image features with the pre-trained Mamba word embedding.
State 1.2: Instruction co-training. In this stage, we first follow previous MLLM works 64 52 for
general vision instruction data collection. We adopt the 655K LLaVA VA mixed instruction dataset 44, the ShareGPT4V-SFT dataset [68], or the LLaVA-Next dataset 669 In our dataset selection, mitigating hallucination is crucial in robotic scenarios, the robotic MLLM needs to generate task planning based on real scenes rather than imagined ones. For example, existing MILMs might formulaically answer "open the microwave" with "step 1: find the handle," but many microwaves do not have handles. Next, we incorporate the RoboVQA dataset [27] to learn high-level robotic skills, such as long-horizon planning, success classification, discriminative and generative affordance, past description, and future prediction. During co-training, as shown in F we freeze the parameters of the CLIP encoder and fine-tune the projection layer and Mamba on the combined instruction datasets. All outputs from the Mamba language model are supervised using the cross-entropy loss.
3.4 Robot manipulation fine-tuning
Building upon RoboMamba's ' strong reasoning ability, we introduce our robot manipulation fine- tuning strategy in this section, termed Training Stage 2 in Figure 2 Existing manipulation VLA models [29] 15 [4] require updating the projection layer and the LLM during the manipulation fine-tuning stage. While this paradigm can develop action prediction capabilities, it also breaks the inherent abilities of the pre-trained model and demands significant training resources. To address these challenges, the parameters of RoboMamba and introduce a simple policy head to model Mamba's output tokens. The policy head contains two types of MLPs that separately learn the end-effector 's position apos and direction adir. collectively occupying around 0.1% of the model's total parameters. Following [61]. the position and direction losses formulated follows:
propose an efficient fine-tuning strategy, shown in Figure 2 We freeze all
T.pos
appss
a apoal
(5)
Tr gg T Adir)
N
Lasr
arccos
(6)
N
2
where N represents the number of training samples, Tr(A) means the trace of matrix A. In our open-loop simulation experiments, RoboMamba only predicts the 2D position (x, y) of the contact pixel in the image, which is then translated into 3D space using depth information. We also derive the gripper's 's left direction (gripper f-orwwr from its up and forward orientations based on geometric relationships. To evaluate this fine-tuning strategy, we generate a dataset of 10k end-effector pose predictions using the SAPIEN simulation 28). After manipulation fine-tuning, we find that once RoboMamaa possesses sufficient reasoning capabilities, it can acquire pose prediction skills with extremely efficient fine-tuning. Due to the minimal fine-tuning parameters (7MB) and efficient model design, we need only a few dozen minutes to achieve novel manipulation skills. This finding highlights the importance of reasoning abilities for learning manipulation skills and presents a new perspective: we can efficiently equip VLA model with manipulation abilities without compromising its inherent reasoning capabilities. Finally, RoboMamba can use language responses for common sense and robotic-related reasoning, and the policy head for action pose prediction.
4 Experiment
In Section 4.1
we introduce our experiment settings, including dataset, implementation, and eval- uation benchmark details. Subsequently, we conduct extensive experiments to demonstrate Robo- Mamba's 's reasoning and manipulation abilities in Sections 4.2 and 4.3 respectively. To thoroughly

---

validate the effectiveness of each method design, we perform an ablation study in Section Finally. the qualitative results of real-world experiments are presented in Section 4.5
4.1 Experiment settings
Datasets (Stage 1) In the alignment pre- pre-training stage, we utilize the LLaVA-LCSS 558K dataset [7]. which is a curated subset of the LAION-CC-SDD dataset, supplemented with captions. During the instruction co-training stage, we combine general instruction datasets with the robotic instruction datasets. Specifically, for the general instruction dataset, we selectively adopt the LLaVA mixed instruction dataset [4], the ShareGPT4V-SFT dataset [[8], or the LLaVA-Next dataset [69]. For the robotic instruction dataset, we randomly sample some image-text paired training samples from the Robo VQA (27 dataset. In our main experiments, a mixture of the LLaVA 1.5 instruction dataset and the 300K Robo VQA dataset is used during the co-training stage. Detailed descriptions of these datasets are provided in Appendix B
Datasets (Stage 2) For the dataset used in the robot manipulation fine-tuning stage, we follow the
data collection process of previous works [6] [S]. adopting the SAPIEN engine (28 set up an interactive simulation environment with articulated objects from PartNet-Mobility [58]. The Franka
Panda Robot, equipped with a suction gripper, serves as the robotic actuator. During data collection, we randomly select contact point p on the movable part and orient the end-effector's s z-axis opposite to its normal vector, with a random y-axis direction to interact with the object. Successful operations are categorized as successful samples and integrated into the dataset. In the training set, we collect 10K images across 20 tasks. For evaluation, we generate 1.IK examples for the test set, comprising 20 training (seen) and 10 testing (unseen) tasks. The unseen tasks are used to evaluate the generalization capability of our model. The details of the categories are provided in Appendix B
Implementation details Before training, RoboMamba loads pre-trained CLIP/SigLIP ViT-Large [0] as the visual encoder. and the 2.8/1.4B Mamba [ model as the language model. During the alignment pre-training and instruction co-training, we conduct training for 1 epoch and 2 epochs, respectively. We utilize the A optimizer with (31 822 of 4e-5. The precision of floating-point calculations is set to 16-bit. For manipulation fine-tuning, we train the model for epochs, setting the LR to le-5 and applying a weight decay of 0.1. The floating-point precision is set to 32-bit. All experiments are conducted on NVIDIA A100 GPUs.
(0.9, 0.999) and a learning rate (LR)
Reasoning evaluation benchmarks To evaluate reasoning capabilities, we employ several popu- lar benchmarks. including vQAv2 [7]), OKVQA (72), RoboVQA 27 GQA (73), VizWiz [74), POPE (75). MME (76), MMBench 777. and Vet (78). As detailed in Appendix E we describe the key aspects each benchmark focuses on when assessing models in the field of robotics. Notably, we also directly evaluate RoboMamba's s robotic-related reasoning abilities on the 18k validation dataset of Robo VQA, covering robotic tasks such as long-horizon planning, success classification, discriminative and generative affordance, past description, and future prediction.
Manipulation evaluation benchmarks To evaluate our model's manipulation capabilities, we follow previous works [57] 63 [5] and test open-loop task completion accuracy exclusively in the simulator (28). The predicted contact point and rotation are used to interact with objects. To measure the model's 's performance, we use the classical manipulation success rate, defined as the ratio of successfully manipulated samples to the total test samples. A manipulation action is considered successful if the difference in the object's joint state before and after interaction exceeds a threshold of 0.1 meters. In real-world experiments, we use the Franka Panda robot to manipulate several articulated objects.
4.2 Reasoning quantitative results
General reasoning. As shown in Table I we compare RoboMamba with previous state-of-the-a art (SOTA) MLLMS on general VQA and recent MLLM benchmarks. First, we find that RoboMamba achieves promising results across all VQA benchmarks, using only a 2.7B language model. The results demonstrate that our simple architecture design is effective. The proposed instruction co- training significantly enhance the MLLM's reasoning capabilities. For example, due to the large amount of robot data introduced during the co-training stage, our model's spatial identification performance on the GQA benchmark is improved. Meanwhile, we also test our RoboMamba on recently proposed MLLM benchmarks. Compared to previous MLLMs, we observe that our model
7

---

Table I: Comparison of general reasoning abilities with previous MLLMs across several benchmarks. 'Res.' indicates the resolution of the input image. RoboVQAI to RoboVQA4 represent the BLEU-1 to BLEU-4 scores, respectively. For TinyLLaVA LA and LLaMA-
we evaluate robotic reasoning abilities after fine-tuning the pre-trained MILMS on the RoboVQA dataset.
Method
[LM]es.
Res.
OKVQA VQAV2 GQA
VizWiz POPE MME MMB MM- Ver RoboVQA, RoboVQA,
BLIP-2 [43
7B
224
45.9
41.0
19.6
85.3
1293.8
22.4
InstructB1..0
79
224
49.5
33.4
36
26.2
LLMMA-Adapter
45
7B
336
49.6
70.7
45.1
39.8
1328.4
8.1
27.8
MiniGPT-v2
80
7B
448
$7.8
60.1
53.6
Qwen-VL.
7B
448
58.6
79.5
59.3
35.2
38.2
LLAVA155
67
7B
336
78.5
62.0
50.0
85.9
1510.7
64.3
30,5
SPHINX
64
78
224
62,1
78.1
62.6
39.9
80.7
1476.1
66.9
36.0
LLaVA-Phi
49
2.7B
336
71.4
35,9
85.0
1335.1
59.8
28.9
MobileVLM
2.78
336
59.0
84,9
1288.9
59.6
TinyLLaVA
33]
2.7B
336
77.7
61.0
86.3
1437.3
68.3
31.7
29.6
43,5
RobaMambaa(Orry
2.7B
224
63.3
79.6
64.2
57.1
86.3
1297.2
60.9
29.4
42.8
62.7
RoboManbu (uurs)
2.7B
336
62.7
77.7
63.3
58.1
87.0
1335.5
60.7
31.4
41.8
61.9
achieves competitive results across all benchmarks. Specifically, our model achieves satisfactory results on the POPE benchmark, helping to reduce failed robot actions caused by hallucinations. Although some performances of RoboMamba are still below those of LLa and SPHINX, we prioritize using a smaller and faster Mamba to balance the efficiency of the robotic model. In the future, we plan to develop RoboMamba- 7B for scenarios where resources are not limited.
Robotic-related reasoning. To comprehensively compare RoboMamba's robotic-related rea- soning abilities, we benchmark it against LLaMA-AdapterV2 [45) and TinyLLa VA
[83 on the RoboVQA [27] validation set. We choose LLaMA-Adapterr2 as a baseline because it serves as the base model for the current state-of-the-art (SOTA) robot MLLM, ManipLLM [15] Meanwhile, TinyLLaVA is chosen as a representative tiny MLLM, enabling a comparison of robotic-related reasoning abilities. For a fair comparison, we load the pre-trained parameters of both LLaMA- AdapterV2 and TinyLLaVA and fine-tuned the baseline models on the Robo VQA training set for two epochs, using their official instruction-tuning method. As shown in Table [ RoboMamba achieves superior performance across BLEU-1 to BLEU-4. The results indicate that our model possesses advanced robotic-related reasoning capabilities and confirms the effectiveness of our training strategy. In addition to higher accuracy, our model achieves inference speeds 7 times faster than LLaMA- AdapterV2 and ManipLLM, which can be attributed to the content-aware reasoning ability and efficiency of the Mamba language model (25). Finally, we visualize the qualitative results in Figure
4.3 Manipulation quantitative results
Baselines. To evaluate RoboMamba's manipulation abilities, we compare our model with four baselines: UMPNet [63, Flowbot3D (57). RoboFlamingo (14) and ManipLLM [4] Before comparison, we reproduce all baselines and train them on our collected dataset. For UMPNet, we execute manipulation on the predicted contact point, with the orientation perpendicular to the object's surface. Flowbot3D predicts motion direction on the point cloud, selecting the largest flow magnitude as the interaction point and using the direction of the flow represent the end-effector's orientation. RoboFlamingo and ManipLLM separately load the pre-trained parameters of OpenFlamingo 44 and L.LMA-Adaptter 45 and follow their respective Ane-tuning and model updating strategies.
Results. As shown in Table 2 our RoboMamba achieves a 7.0% improvement on seen tasks and 2.0% improvement on unseen tasks compared to the previous SOTA ManipL.MM. Moreover, our method showcases SOTA performance across 14 of 20 seen tasks, highlighting its effectiveness and stability in predicting action poses. For unseen tasks, the recent three MLLM-based meth- Mls-RoboFlamingo ManipLLM, and our method achieved promising performance. The results demonstrate that leveraging the strong generalization abilities of MLLMs can effectively improve the policy's generalizzinnnn ability while enhancing accuracy on unseen objects. Regarding efficiency, RoboFlamingo updates 35.5% (1.8B) of the model parameters, ManipLLM updates an adapter (41.3M) comprising 0.5% of the model parameters, whereas our fine-tuned simple policy head (3.7M) only constitutes 0.1 % of the model parameters. Robo effectively updates times fewer parameters than previous MILL- methods while achieving seven times faster inference speeds. The results reveal that our Mobba not only possesses strong reasoning abilities but also can acquires manipulation capabilities in cost-effective manner.
8

---

Table 2:
Comparison of the success rates between RoboMamba and baselines across various training (seen) and test (unseen) tasks. The representation for each task icon is shown Table 3)
Seen Categories
Method
UMPNet [63]
0.28 0.41 0.25
0.20
0.49
0.20
0.35
0.57
0.51
0.25
0.66
0.17
0.17
0.26
0.27
0.40
FlowBot3D [57
0.50 0.53 0.26 0.36
0.34
0.36
0.54
0.26
0.12
0.34
0,41
0.23
0.36
0.30
0.17
0.37
RoboFlamingo 14
0.48 0.51 0.50
0.35
0.11
0.47
0.54
0.35
0.19
0.46
0.18
0.64
0.26
0.42
0.15
0.87
Ma
15
0.68 0.62 0.45 0.74 0.42 0.25 0.61 0.66
0,56 0.52 0.50 0.42 0.64 0.76
0.63
0.60
RoboMamba (Ours)
0.81 0.73 0.33 0.85 0.86 0.60 0.81
0.42 0.56 0.54 0.68 0.81 0.26
0.86
0.39
0.91
Seen Categories
Unseen Categories
Method
AVG
AVG
UMPNet 663
0.27
0.37
0.19
0.60
0.34
0.32
0.36
0.18
0.37
0.21
0.12
0.04
0.53
0.28
0.13
0.26
FlowBot3D [57
0.21 0.57 0.29
0.45
0.35
0.36
0.36
0.18
0.30
0.21
0.50
0.13
0.53
0.28
0.09
0.30
RoboFlamingo
[1
0.20 0.42
0.58
0.60
0.41
0.36
0.62
0.64
0.33
0.14
0.34
0.44
0.66
0.41
0.31
0.43
ManipLLM
15
0.41
0.78
0.41
0.59
0.56
0.21
0.25
0.79
0.76
0.52
0.76
0.43
0.85
0.26
0.52
0.51
RoboMambaa Ours))
0.40 0.55
0.37
0.80
0.63
0.19
0.23
0.67
0.66
0.57
0.45
0.65
0.68
0.30
0.93
0.53
.Maamb--.48
RWKV-3B
Mamba-2.78
Secn
Unsseen
63.3
642
0.5
0.46
0.41
0.3
9.26
9.1
GUA
POPE
Openfalninge
ELaMAsAdeeteeV
Ours-1.0
Ours-1.7B/wlo C)
0ure27B
The of reasoning abilities
by The of reavening ability manipulation aecuracy
Figure 3: Ablation study a) The impact of LLM on reasoning abilities. Ablation study b) The
impact of reasoning ability on manipulation accuracy.
4.4 Ablation study
The impact of LLM reasoning abilities. As shown in Figure 3 a). explore the impact of different LLMs on general and robotic-related reasoning abilities. Given that efficiency is crucial in robotic tasks and directly affects the practicality policy models, we compare Mamba-2.77 7B with other linear complexity LLMs. For all experiments, we utilize the same training data and strategy. Compared with RWKV-3B (24), Mamba-2.7B achieves significant improvements on both common sense and robotic-related reasoning benchmarks. The results demonstrate that the Mamba-2.7B model not only possesses linear complexity but also efficiently acquires strong reasoning abilities through our proposed training strategy. Meanwhile, our proposed RoboMamba VLA framework and training strategy can also be adapted other, more advanced cimpll-coplyxxtty LLM models.
The impact of reasoning abilities on manipulation accuracy. We explore whether utilizing MILMS with different reasoning abilities affects manipulation skill learning. For fair comparison, the same manipulation fine-tuning strategy, injecting and fine-tuning a simple MLP policy head after the MLLM (while freezing other parameters). We compare our RoboMamba-2.77 (Ours-2.7B) with OpenFlamingo LLaMA-Adatter
and our RoboMamba- 1.4B. As shown in Figure 3 b), Ours-2.7B achieves promising results compared with other methods, which proportional its reasoning ability, Meanwhile, Ours-2.7B (w/o C) indicates that we did not use the instruction co-training method, omitting the robotic-related RoboVQA dataset during -unint. We find that this also impacts the accuracy of manipulation, especially reducing the model's 's generalization ability when facing unseen objects. The results confirm our finding: fine-tuning an MLLM learn robot skills does not require extensive resources; it only requires that the MILMM possesses strong robotic-related reasoning abilities. Additionally, we present more ablation studies in Appendix C. including explorations of different vision encoders, training datasets, and policy head design.

---

Long Horizon Planing
Generative Affordance
Planning
place the Intre for me pleass, leal 20 stspe I- Lorigt stace tup the detle
molos baxt
Qureelion-cmm emageniing sngoiingumettt gosl p pless remone the
chip foom boud and piace the diew but t-rpen
Dusetiorc oraggga
preaib now? Janaweer
coup
put brown bug the dianer put the orange bag in the drower 6 pur black bag in the drower inmediali janswert
the table
scoop
holder
Take out some sooop of Pistachlo fiom the ?
Put the Pisachio indile the sup Takes out some
RobooMnnbujOurst
NoboManmbarGural To dose drawer harve
with the
Discriminative Affordance
Planning With Context
cup de Erter the cacer de Plaoe the cup the lable b- dane
LLaMA Addapto/ done
Gusetion-dmaggee
packel
Quaetion-dinng aetlliindiigggllauuuauum
Gusestion: dimagenint poal empty stops Janswerr
the hote prise the right now?
goal le Plence memove
Janewe:: Rcbolombaa(uunn yeu
the bostl inmeddade
R RelboMambaa(uur) the packet on the b put the packet the tatée 3 the pasket LL MA-ddaatteri i- doon
Future Prediction
Janewwer:
Raboffarbaa@Ounn place the
aople
twble
Question weslloncinageifooorccc
Pose Prediction
Janawer: HabboManba(Durai : taname
the drawer pick crips from top diower and iut vired bessel and creae the top draweer last 20 stapss ppen diowvrr nest stepa//armmenr
Question: dinogee/a
Prediction
chip ena benket drop grean
Question Seseiiionniangganaefeeee just
packet chip into the basket 5- dlose the top drower d done Mdd---ptr 1- close the crineet 2-dome
RobolMonnbaCuuns the bock on the Indil
Figure 4: The visualization of RoboMamba's s abilities across various robotic downstream tasks in real-world scenarios, including task planning, long-horizon planning, discriminative and generative affordance, past and future prediction, and low-level pose prediction.
4.5 weal-world experiments
As shown in Figure we visualize RoboMamba's reasoning results across various robotic down- stream tasks. For task planning, compared to LLaMA-AdapterV2. RoboMamba demonstrates more accurate and long-horizon planning abilities, thanks to its strong reasoning capabilities. For a fair comparison, we also fine-tuned the baseline LLaMA-AdapterV2 ally. RoboMamba accurately performs fundamental robotic tasks such as affordance generation and discrimination, proving that can understand robotic scenes. Notably, our model also possesses past and future prediction capabilities. further highlighting its robust reasoning capabilities. Prediction of past and future actions crucial in robotic manipulation, as not only enables effective rethinking of past failure actions but also enhances the robustness of future manipulation pose generation. For low-level action, we use a Franka Emika robotic arm to interact with various household objects. Due to the direct visualization of the gripper causing occlusion, we project Robo.Mamba's 's predicted 6 DoF pose onto a 2D image, using a red dot to indicate the contact point and the end-effector to show the rotation, shown in the bottom right corner of the figure.More real-world demonstrations are provided in Appendix D and the supplementary video file. Meanwhile, as shown Figure [] we also visualize the failure cases of RoboMamba's 's predictions in both reasoning and manipulation tasks.
the RoboVQA dataset. Addition-
5 Conclusion and future plan
In this paper, we introduce RoboMamba, an efficient VLA model that combines a vision encoder with the linear-complexity Mamba LLM,, equipped with visual common sense reasoning and robotic reasoning abilities. Based on our RoboMamba, we can impart new manipulation skills to the model by fine-tuning a simple policy head (0.1% % of the model) in few dozen minutes. This finding reveals how to efficiently equip an VLA model with manipulation abilities without compromising its inherent reasoning capabilities. Finally, RoboMamba excels in reasoning on both general and robotic-related evaluation benchmarks and showcases impressive pose prediction results. Regarding limitations, while our proposed RoboMamba achieves efficient inference speed, its reliance on 2.7B LLM leads to limitations on certain complex reasoning tasks when compared to MLLMs built on 7B/13B LLMs. Looking ahead, our future work will focus on two main directions. 1) We plan adapt the RoboMamba VI.A framework and training strategy to more advanced linear-complexity LLM models to further enhance its reasoning and manipulation capabilities. 2) Constructing a 4D Robot VI.A model 84 48 85 as 3D point cloud and temporal data contain more robotics-specife information
that aids in predicting robust low-level actions.
10

---

Acknowledgements. This work was supported by the National Natural Science Foundation of China (62476011).
References
[1] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397,, 2023.
[2] OpenAl. GPT-4 technical report, 2023.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901 2020.
[4] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08444 08485, 2023.
[5] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision- laiiuula understanding and generation. In International Conference on Machine Learning, pages 12888-12900. PMLR, 2022.
[6] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Lue, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23736 2022.
[7] Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, et al. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models. ICML, 2024, 2024.
[8] Renrui Zhang, Dongzhi Jiang, Yichi Zhang. Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wer Chang, Peng Gao, et al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403.1 14624, 2024.
[9] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0169) 2022.
[10] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303 03378, 2023.
[11] Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973 2023.
[12] Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint arXiv:2309.000 00611.009000,..... 2023.
[13] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:22
2022.
[14] Xinghang Li. Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, al. Iision-language foundation models as effective robot imitators. arXiv preprint arXiv:2311.01378 2023.
[15] Xiaoqi Li, Mingxu Zhang, Yiran Geng, Haoran Geng, Yuxing Long, Yan Shen, Renrui Zhang, Jiaming Liu, and Hao Dong. Maniplim: Embodied multimodal large language model for object-centric robotic manipulation. arXiv preprint arXiv:2312.16217. 2023.
11

---

[16] Brianna Zitkovich. Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision caaion-lnngag--atttnnnn models transfer web knowledge to robotic control. In 7th Annual Conference on Robot Learning, 2023.
[17] Hao-Shu Fang, Chenxi Wang, Hongjie Fang, Minghao Gou, Jirong Liu, Hengxu Yan, Wenhai Liu, Yichen Xie, and Cewu Lu. Anygrasp: Robust and efficient grasp perception in spatial and temporal domains. IEFE Transactions on Robotics, 2023.
[18] Kaichun Mo. Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su. PartNet: A large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[19] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303. 2023.
[20] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[21] Xiao Wang, Shiao Wang, Yuhe Ding, Yuehang Li, Wentao Wu, Yao Rong, Weizhe Kong, Ju Huang, Shihao Li, Haoxiang Yang, et al. State space model for new-generation network alternative to transformers: A survey. arXiv preprint arXiv:2404.09516. 2024.
[22] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021.
[23] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023.
[24] Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao. Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint 2301::23055/3445 2023.
[25] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00755: 2023.
[26] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabricl Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8788888 8763. PMLR, 2021.
[27] Pierre Sermanet, Tianli Ding, Jeffrey Zhao Fei Xia, Debidatta Dwibedi, Keerthana Gopalakrish- nan, Christine Chan, Gabriel Dulac-Amold, Sharath Maddineni, Nikhil Joshi, et al. Robovqa: Multimodal long-horizon reasoning for robotics. arXiv preprint arXiv:2311 1.00899, 2023.
(28) Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X. Chang, Leonidas J. Guibas, and Hao Su. SAPIEN: A simulated bast- interactive environment. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[29] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint UrXiv:24006002240 2024.
[30] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances neural information processing systems, 34:572-585, 2021.
[31] Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling, arXiv preprint arXiv:2208.049: 2022.
12

---

[32] Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212. 14052, 2022.
[33] Ethan Baron, Itamar Zimerman, and Lior Wolf. 2-d ssm: A general spatial layer for visual transformers. arXiv preprint arXiv:2306.066. 06635, 2023.
[34] Jiacheng Ruan and Suncheng Xiang. Vm-unet: Vision mamba unet for medical image segmen- tation. arXiv preprint arXiv:2402.02491, 2024.
[35] Ziyang Wang and Chao Ma. Semi-mamba-unet: Pixel-level contrastive cross-supervised visual mamba-base based unet for semi-supervised medical image segmentation. arXiv preprint arXiv:2402.07245 2024.
[36] Ziyang Wang,
Jian-Qing Zheng. Yichi Zhang, Ge Cui, and Lei Li. Mamba-uner: Unet-like pure visual mamba for medical image segmentation. arXiv preprint arXiv:2402.05507) 2024.
[37] Eric Nguyen, Karan Goel, Albert Gu. Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher Ré. S4nd: Modeling images and videos as multidimensional signals with state spaces. Advances in neural information processing systems, 35:2846-2861, 2022.
[38] Chenhongyi Yang. Zehui Chen, Miguel Espinosa, Linus Ericsson, Zhenyu Wang, Jiaming Liu, and Elliot J Crowley. Plainmamba: Improving non- non-hierarchical mamba in visual recognition. arXiv preprint arXiv:2403. 122400..7699 17695, 2024.
[39] Hang Guo, Jinmin Li, Tao Dai, Zhihao Ouyang, Xudong Ren, and Shu-Tao Xia. Mambair: simple baseline for image restoration with state-space model. arXiv preprint 2564482400,,5488 2024.
[40] Xuanhua He, Ke Cao, Keyu Yan, Rui Li, Chengjun Xie, Jie Zhang, and Man Zhou. Pan-mamba: Effective pan-sharpening with state space model. arXiv preprint arX1::240../2/92, 2024.
[41] Zhengcong Fei, Mingyuan Fan, Changqian Yu, and Junshi Huang. Scalable diffusion models with state space backbone. arXiv preprint arXiv:2402.05608, 2024.
[42] Jiageng Mao, Yuxi Qian, Hang Zhao, and Yue Wang. Gpt-driver: Learning to drive with gpt. arXiv preprint arXiv:23/0.014: 2023.
[43] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: pre-training with frozen image encoders and large language models, 2023.
Bootstrapping language-image
[44] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390 2023.
[45] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Klaa--adapte v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304. 15010, 2023.
[46] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36, 2024.
[47] Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, and Zhou Zhao. Chat-3d: Data- efficiently tuning large language model for universal dialogue of 3d scenes. arXiv preprint arXiv:2308.0877 08769, 2023.
[48] Senqiao Yang, Jiaming Liu, Ray Zhang, Mingjie Pan, Zoey Guo, Xiaogi Li, Zehui Chen, Peng Gao, Yandong Guo, and Shanghang Zhang. Lidar-llm: Exploring the potential of large language models for 3d lidar understanding. arXiv preprint (xXiv:2312.1407 2023.
[49] Yichen Zhu, Minjie Zhu, Ning Liu, Zhicai Ou, Xiaofeng Mou, and Jian Tang. Llava-phi: Efficient multi-modal assistant with small language model. arXiv preprint arXiv:2401.022: 2024.
13

---

[50] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open small language model. arXiv preprint urXiv:2401.02385, 2024.
[51] Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, al. Mobilevim v2: Faster and stronger baseline for vision language model. arXiv preprini arXiv:2402.03766, 2024.
[52] Han Zhao, Min Zhang, Wei Zhao, Pengxiang Ding, Siteng Huang, and Donglin Wang. Cobra: Extending mamba to multi-modee large language model for efficient inference. arXiv preprint arXiv:2403. /4520, 2024.
[53] OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz. Bob Mc- Grew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
[54) Yiran Geng, Boshi An, Haoran Geng, Yuanpei Chen, Yaodong Yang, and Hao Dong. End-to-end affordance learning for robotic manipulation. In International Conference on Robotics and Automation (ICRA), 2023.
[55] Shirin Joshi, Sulabh Kumra, and Ferat Sahin. Robotic grasping using deep reinforcement learning. In 2020 IEEE 16th International Conference on Automation Science and Engineering (CASE), pages 1461-1466. IEEE, 2020.
[56] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint ar0 00645, 2021.
[57] Ben Eisner, Harry Zhang, and David Held. Flowbot3d: Learning 3d articulation flow to manipulate articulated objects. arXiv preprint 1xXiv:2205.04382 2022.
[58] Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 909-918, 2019.
[59] Weikang Wan, Haoran Geng, Yun Liu, Zikang Shan, Yaodong Yang. Li Yi, and He Wang. Unidexgrasp++: Improving dexterous grasping policy learning via geometry-aware curriculum and iterative generalist-specialist learning. arXiv preprint arXiv:2304. 00464, 2023.
[60] Qianxu Wang, Haotong Zhang, Congyue Deng, Yang You, Hao Dong, Yixin Zhu, and Leonidas Guibas. Sparsedff: Sparse-vieu view feature distillation for one-shot dexterous manipulation. arXiv preprint arXiv:2310. 16838, 2023.
[61] Kaichun Mo, Leonidas J Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani. Where2act: From pixels to actions for articulated 3d objects. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 681 6813-6823, 2021.
[62] Siyuan Huang, Iaroslav Ponomarenko, Zhengkai Jiang, Xiaoqi Li, Xiaobin Hu, Peng Gao, Hongsheng Li, and Hao Dong. Manipvqa: Injecting robotic affordance and physically grounded information into multi-modal large language models. arXiv preprint arXiv:2403. 1129, 2024.
[63] Zhenjia Xu, Zhanpeng He, and Shuran Song. Universal manipulation policy network for articulated objects. IEEE Robotics and Automation Letters, 7(2):2447-2454, 2022.
[64] Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint 1.XXv:331100777: 2023.
[65] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint ar arXiv:2304.07193, 2023.
14

---

[66] Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16133-16142, 2023.
[67] Haotian Liu, Chumyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction tuning. arXiv preprint 3rXiv:2310.03744: 2023.
[68] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.1279, 2023.
[69] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024.
[70] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975-11986, 2023.
[71] Yash Goyal, Tejas Khot, Douglas Summers- Stay,, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding Visual Question Answering. In Conference Computer Vision and Pattern Recognition (CVPR), 2017.
[72] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/evf conference on computer vision and pattern recognition, pages 3195-3204, 2019.
[73] Drew A Hudson and Christopher Manning. Gqa: new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[74] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference computer vision and pattern recognition, pages (608-3617 2018.
[75] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.1033 2023.
[76] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.1 13394, 2023.
[77] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281 2023.
[78] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490 2023.
[79] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purper vision-language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
[80] Jun Chen, Deyao Zhul Xiaoqian Shenl Xiang Li, Zechun Liu2 Pengehuan Zhang, Raghuraman Krishnamoorthi2 Vikas Chandra2 Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: Large language model as a unified interface for vision-language multi-task learning. arXiv preprint 1700v:2310.09478, 2023.
15

---

[81] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. ArXiv, 0etp2238.119666 2023.
[82] X Chu, L Qiao, X Lin, S Xu, Y Yang, Y Hu, F Wei, X Zhang, B Zhang, X Wei, et al. Mobilevlm: A fast, strong and open vision language assistant for mobile devices. arXiv preprint arXiv:2312.16880 2023.
[83] Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: A framework of small-scale large multimodal models. arXiv preprint arXiv:2402. |4289, 2024.
[84] Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-lIm: Injecting the 3d world into large language models. arXiv preprint arXiv:2307.12995 2023.
[85] Yiwen Tang, Jiaming Liu, Dong Wang. Zhigang Wang, Shanghang Zhang, Bin Zhao, and Xuclong Li. Any2point: Empowering any-modality large models for efficient 3d understanding. arXiv preprint S4V::2204.07989, 2024.
[86]
https//shareept.comm/ Sharegpt. 2023.
[87] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision, pages 146-162. Springer, 2022.
[88] Oleksii Sidorov, Ronghang Hu, Mareus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow; UK, August 23-28, 2020, Proceedings, Purt II 16, pages 742-758. Springer, 2020.
[89] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference empirical methods in natural language processing (EMNLP), pages 787-798, 2014.
[90] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy, Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 11 11-20, 2016.
[91] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73. 2017.
[92] Alaacldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Cross- covariance image transformers. Advances in neural information processing systems, 34:20014- 20027, 2021.
16

---

A Appendix
Due to space limitations, we provide additional details of the proposed method in this supplementary material. In Appendix [B] we offer a more detailed description of our training dataset, including alignment pre-training, instruction co-training, and robot manipulation fine-tuning datasets. Addi- tional ablation studies are presented in Appendix [C] exploring the impact of different image encoders on reasoning ability, the impact of different training datasets reasoning ability, and the effect of various head designs on manipulation fine-tuning. In Appendix [D] we show additional qualitative results across multiple robot-related downstream tasks. Finally, provide the metric selection rationale and the usage of prompts during testing in Appendix
B Dataset description
Stage 1.1: Alignment pre-training dataset
1) LLLVA-LCSS 558K: This LLaVA Visual Instruct Pretrain LCS-558K dataset is curated subset of the LAIONICC/SBU dataset, specifically filtered to achieve a more balanced distribution of concept coverage. Additionally, it includes captions paired with BLIP synthetic captions for reference purposes.
Stage 1.2: Instruction co-training dataset.
LLaVA-v1.5 655K: This dataset is mixture of ten distinct datasets, including LLaVA
4
ShareGPT 86
VQAv2 71.. GQA (73, OKVQA (72), OCRVQA (72), A-OKVQA 87]
TextCaps
88 RefCOCO 88 90], and Visual Genome (VG) 19.. This mix dataset is also of
the most renowned datasets used for instruction tuning in several works 4 67].
2) ShareGPT4V-SFT dataset: The ShareGPT4V-SFT dataset is similar to LLaVA-v1.5 655K [67]. except that the 23K detailed description entries in LL5VA-I,5-SFT are replaced with detailed captions randomly sampled from the 100K ShareGPT4V data [86).
3) VA-Net dataset: Compared LLaVA-v 655K 67. LLaVA-Nex 669 enhances the instruction data mixture by prioritizing high-quality user instruction data and expanding multimodal document/chart data sources. For high-quality user instruction data, L.L.VA-Nex ensure task diversity and response quality by using existing GPT-V data (LAION-GPT-V and ShareGPT-4V) and a carefully curated 15K visual instruction dataset from LLaVA demos. Additionally, LLaVA-Next replaces TexICaps with Doc and SynDog-EN EN to improve OCR capability.
RoboVQA 800K: co-training, use this dataset to enhance our model's 's robot-related reasoning abilities. RoboVQA 27 comprises realistic data collected by performing various user requests and using multiple embodiments, such as robots, humans, and humans with grasping tools. This dataset includes 5,246 long-ho long-horizon episodes and 22.948 medium-hooe horizon episodes of robotic tasks. each paired with image and text prompt inputs. our experiments, we randomly select 300K image-text paired instruction samples from RoboVQA to construct the co-training dataset.
Safc
Door
Display Refrigerator Laptop
Lighter Microwave
Mouse
Box
Trashcan
Kitchen pot Suitcase
Pliers
Storage
Remote Bottle Folding chair Toaster Lamp Dispenser
Toilet
Scissors
Table
Stapler
Kettle
USB
Switch
Washing Faucet
Phone
Table Representation of each category icon.
Stage 2: Robot manipulation fine-tuning dataset.
Representation for Each Category Icon In Table 3 we provide an overview of the meaning of each category icon presented in Table of the main paper. Following Partnet (58), different tasks designed for each category. For instance, opening the door control panel of refrigerator,
17

---

opening the cap of a bottle, and rotating the lid of a box. The detailed task design can be found at: https::/saapien.use edd/boowse
Simulator Data Collection In the simulator, we use a Franka Panda Robot with a suction gripper as the robotic actuator. During data collection, randomly select a contact point the movable part of the object and orient the end-effector's 's z-axis opposite to the object's 's normal vector, with random y-axis direction interact with the object. Successful operations are categorized successful samples and integrated into the training dataset. For the training set, we collect 10K images across 20 categories, including Safe, Door, Display, Refrigerator, Laptop, Lighter. Microwave, Mouse, Box, Trash Can, Kitchen Pot, Suitcase, Pliers, Storage Furniture, Remote, Bottle, Folding Chair, Toaster, Lamp, and Dispenser. For testing. we use set of 1.1K images that include both seen categories from training and unseen categories, such as Toilet, Scissors, Table, Stapler. Kettle, USB, Switch, Washing Machine, Faucet, and Phone. Regarding the variation between training and testing data, we followed the data collection settings of where2act [6]] and ManipI.IM [15]. The specific variations can be divided into two aspects: 1) Asset Variation and 2) State Variation.
1) Asset Variation: We use 20 categories from PartNet (58) for seen objects and reserve the remaining 10 categories for unseen objects analyze if RoboMamba can generalize to novel categories. Specifically, we further divide the seen objects into 1037 training shapes and 489 testing shapes, using only the training shapes to construct the training data. Thus, the shapes of the seen objects encountered during training and testing are different. For unseen categories, there are a total of 274 shapes, which are used exclusively in the testing data.
2) State Variation: We observe the object in the scene from an RGB-D camera with known intrinsics, mounted 4.5-5.5 units away from the object, facing its center. The camera is located at the upper hemisphere of the object with a random azimuth between [-45, 45] and a random altitude between [30, 60]. Since the tasks involve 'pulling.' we also initialize the starting pose for each articulated part randomly between its rest joint state (fully closed) and any position up to half of its joint state (half-opened). These state settings are utilized for both training and testing data, aiming boost the model's 's generalization ability.
C Additional ablation study
Table 4: Ablation study of different image encoders reasoning abilities.
Encoder
Image Resolution
OKVQA
GQA
POPE
RoboVQA(BLEU-4
CLIP
224 224
46.7
50.7
79.7
35.2
XCIT
224 x 224
63.3
64.2
86.3
42,8
CLIP
336 336
62.7
63.3
87.0
41.8
SigLIP
384 x 384
62.4
64.4
86.0
40.6
The impact of different image encoders on reasoning abilities this section, we replace the CLIP encoder used in our initial submission with other linear-complexity encoders, such as XCIT [92] Additionally, we supplement our experiments by using SigLIP 70 as an image encoder. As shown in Table 4 we analyze the impact of different image encoders and input resolutions on reasoning abilities. The training dataset and strategy remain consistent with those in our main experiment. The results indicate that the choice of image encoder and input resolution does not significantly impact reasoning ability within our RoboMamba VLA framework. However, using an image encoder without cross-modaling alignment (i.e., XCTT) presents challenges in converting image tokens to LLM language embeddings. Although our training process includes an alignment pre-training stage, this primarily trains the projection layer. Therefore, in future work, we aim to develop robotics-specific image encoder capable of projecting image tokens into language embeddings while maintaining linear computational complexity to further improve inference speed.
The impact of training datasets on reasoning abilities As shown in Table we examine the impact of different training datasets on common sense and robotic-specific reasoning abilities. Specifically, we conduct these experiments using 224 x 224 input images and the CLIP vision encoder. First, we observe that Ex2 outperforms Ex1 across two benchmarks, confirming that incorporating robotic instruction data can effectively enhance specific reasoning abilities. Similarly, comparing Ex3 and Ex4 shows comparable results, though performance on the GQA benchmark declines. However, in
18

---

Table 5: Ablation study of training strategies on MLLM reasoning benchmarks.
LLaVA 1.5 ShareGP14V-SFT LLaVA-Next Robo-300k
GQA
POPE
RoboVQA
Ex1
65.3
85.6
26.5
Ex2
64.2
86.3
42.8
Ex3
64.3
85.2
26.7
Ex4
V
62.1
85.5
42.5
Ex5
62.9
86.6
25.1
Ex6
60.8
85.4
43.0
our generated descriptions within robotic scenes, we find that the inclusion of robotic instruction data enhances understanding of geometric relationships. Consequently, specific geometric reasoning benchmark to more accurately assess the spatial reasoning capabilities of VLA models. Finally, comparing Ex1, Ex3, and Ex5, we find that using more advanced general instruction datasets does not yield significant performance improvements, which may be due to the model capacity of the 2.7B LLM.
plan propose robotic-
The impact of policy head designs on manipulation accuracy As shown in Table we explore the impact of different policy head designs manipulation skill learning. In this table, MLPx means using only one MLP heads to predict the position and direction of the end-effector pose. MLPx2 means using one shared head predict direction and another head predict position separately, (SSM block+MLPP) M is similar to MLPx2 2 but adds a State Space Model (SSM) block before the MI.P increase the parameter count of the policy head. The experimental results show that the manipulation accuracy across the three configurations is quite similar, indicating that the parameter count of the fine-tuning policy head has small impact the results. Combined with Figure 3b) this further supports our finding that once RoboMamba achieves sufficient robotic reasoning capabilities, it can acquire pose prediction skills at low cost, regardless of the policy head design.
Table 6: Ablation study of policy head design on manipulation dataset.
result
MLPx2
MLPx1 (SSM
block+MLP)
Acc (Seen)
63.7%
62.1%
63.2%
Parameters
3.7M
1,8M
45.2M
Percentage
0.11%
0.05%
1.3%
D
Additional real-world experiments
We conduct real-world experiments involving interactions with various household objects using Franka Emika robotic arm. We modify the finger gripper by attaching double-sided tape to convert it into a suction gripper, providing the gripper head with adhesive properties. The video demonstrations are included in the supplementary video file. As shown in Figure we visualize our model's 's reasoning results on a series of robotic downstream tasks, including long-horin hong-horizon planning, discriminative affordance, generative affordance, past description, and future prediction. Additionally, failure cases reasoning are illustrated Figure Compared to the ground truth, RoboMamba demonstrates limitations in reasoning ability on some complex tasks, occasionally misinterpreting the current task objective or the target manipulated object.
E
Reasoning evaluation bencharks
VQAv2 and OKVQA: These benchmarks utilized to assess the model's proficiency in basic vision question answering, which a foundational skill in embodied AI. This ability ensures that the model can understand and respond visual content effectively.
POPE and VizWiz: These benchmarks are chosen to evaluate the model's capability to answer questions without falling prey to visual illusions or ambiguities. This aspect crucial for avoiding significant errors in robotic applications.
19

---

Long Horizon Planing
Generative Affordance
Planning
Seteeeioooimaaaannaannn ln the canbe the cante ner rext sxepe "anaawer:
Queedtinn emageniingunn sonniinaumentttt gost strck the cupe tu brocks aony. 20 stepe: atace caga lop
Queetiorce cimageoi prssit now? janaweer
lebollamtaa(uur put the apple the
our container
block move Sox side Janswon
RabooMamba(Ours) put block in the box
Durba place stawwerr bowl
apple
conta put the apple the container
LLaMte dudippi 1ig place the replee
put
Discriminative Affordance
Planning With Context
Guestion winggelian goa weise wh willetoorn: lest 20 stepec In pick up momber ciioc macker on bable
Gusetion--maage (wggae/aaaaoaoooooooeoaooo biin candy on the bet prootiil right nins?
Gusetion-dmagee- (ourmn goal le glease nemrown
eiepe Daanever: while braed write th boand So letter
Janewar:
Rcbolambas(uung
slep")anesenn Reboofamna/duuni Place the popcoon passed tatle
bowed 2o with beter
leter on the aftiie board wrthe aelett on the boand
Future Prediction
Pose Prediction
the stops, Stodk the lid on top of ceisting stack
Question: simageeentoumennt
coffoe
Janawer:
RoboManbaaDrraa open the
nost slope 7Janswor
yslow Croneet
tap stack & Stook the top of cxising slock d Stack The id on top of enaling slack bo Stack the weaking slack
RobooMornaaauur
Prediction
Question- dergie//niinnngga//////eo just
RobolMannaaCuunj pul the hoitier
Stack 4-done
Figure 5: Additional visualization of RoboMamba'ss is abilities across various robotic downstream tasks in real-world scenarios, including task planning, long-horizon planning, discriminative and generative affordance, past and future prediction, and low-level pose prediction.
Long Horizon Planning
Generative Affordance
current pleasse insert the hole-punched paper the S-ring binder, leet staps: Move towards the white paper 2- Drag the white papar to the bottom 3- Greep the paper 4- Drag the white paper to right 5- Move your away from the paper on the table 6- Plck the paper on the 9- place the paper the file 10- Re lease the paper 11- Grasp the steel slep 7Janswer:
Question: <image>
goal
Please remove the from top draw then place them in the bowl. immediate next slep?] answer
RoboMamba (Durs):
orange
Move
lofts:
towards steel
bowl
GroudTruth:
Place the apple inside the
boul
object
RoboMamba(Ours))
paper
Planning With Context
Ground Move away the ring binder
Question: c cimage>
sreettionn<mmge>lnn the
gool first
possible right now
move the dask to the largar box on the floor. lest 20 staps: 1- move lowards the bottom cardboerd box 2- open the bottom cardboard box move away from bottom cardboard 4- pick the sorub packet from the bottom cardboard box next sisps?)ande ?Janswer:
RoboMambal (uurs): popen
sleave of the hoodla
GroudTruth: Fold the hoodie upwards
Pose Prediction
pick tape dispanser box from the table 3- drop the tape disporsar bax in the carthoard box 4- piók the brown box from the table drop the brown box in the cardhoard
AaboMambaj(uusia drop scrub packot the cardboard
drop brown belt bottom cardhoard hox 3- drop the orange object into the carthoard box 4- Mova the arm In clock wise direction S Graap the brown objoct
GroudTruth:
cardboard
Figure 6: The visualization of reasoning failure cases. In the bottom right corner of the image, we re-select the qualitative results from our real-world demonstration. Additionally, we replace the red dot and virtual end-effector with physical Franka Panda Robot.
GQA: These benchmarks are employed to test the model's 's ability to identify and comprehend the types and positions of important objects within image. Such spatial identification skills are vital for tasks related to robotic manipulation and interaction with the environment.
This benchmark used assess the model's 's ability to plan and understand actions based on both textual and visual imputs. This skill is indispensable in the realm of robotics, where understanding and executing complex actions necessary.
RobotVQA:
MM-Vet, MME and MMB: These benchmarks are utilized evaluate multimodal large language models's 's ability to integrate on complex multi-modal tasks including Recognition, Spatial awareness, OCR, and Math. All of them contain wealth of evaluation indicators, such as perception and cognition, which can fully demonstrate the performance of the model under different tasks, and this performance is the best embodiment of the comprehensive application performance of multimodal large language models(MM.L.M)..
20

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We did.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We did.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Justification:
We do not have theoretical proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We did.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer:
[No]
Justification:
We will open source the code as soon as it is ready.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.. cc/
pub i quides Codeeubmees ss isionn cisionnoliin icy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips. cc/ wwwpiicggiiicpoooooo / //odiddee/Cdeeeeeeee CooeeeoeeeSeeeeeee SodeSuumiieee Submiss www.sssssiiigoooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We did.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We did.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
23

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We did.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:::////uriss..//e/ httpa//wpp....cpooaacccccccaaaea
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We did not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We did.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssw hopersswiccoe com/ dataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
25

---

Answer:
[NA]
Justification:
We did not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We did not involve with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We didn't.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "PEnnHlPPCn", "reviewer": "Reviewer_QK1g", "summary": "The paper proposes a method for robot reasoning and manipulation by developing a multimodal large language model (MLLM) based on the Mamba state space model (SSM). The joint model is able to both attack robot reasoning tasks assessed via VQA, and solve robot manipulation tasks by fine-tuning a dedicated head that contains only a small fraction of the model parameters.", "questions": "- Can you please provide results regarding comparison with SOTA methods in terms of time efficiency?\n- Can the training strategy, including the fine-tuning of only the manipulation policy head, be applied to other MLLMs?\n- Is the performance in the manipulation tasks affected by the number of parameters in the corresponding head and how?", "limitations": "Limitations are discussed in the text.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces an MLLM based on the Mamba model built specifically for robotic tasks. The exposition of the method is clear and well motivated. The proposed method takes advantage of the linear complexity of Mamba and, in combination to the reduced number of parameters used for the manipulation policy head, achieves higher efficiency and better performance with respect to state-of-the-art robot MLLM methods. Another strong point is the combination of natural language reasoning abilities and task specific manipulation policies based on the MLLM output. \n\nThe experimental evaluation is comprehensive, considering both simulated and real-world experiments, where the proposed method achieves competitive results with respect to SOTA methods across multiple benchmarks.", "weaknesses": "Although time efficiency is one of the major claimed contributions of the work, the aspect of time efficiency is not well covered in the experimental evaluation. The fact that the proposed method is 7 times faster than SOTA MLLM methods is mentioned a couple of times, however detailed results regarding inference times are not provided. \n\nAnother aspect not covered in detail, regards the effect the number of parameters in the manipulation head has on the performance and efficiency of the manipulation task.\n\nAnother question that comes to mind, is whether it is possible to apply the idea of fine-tuning only a small-sized manipulation head to other MLLMs. Does Mamba work better with this constrained policy head and, if yes, why?\n\nFinally, another issue regards reproducibility, as details regarding the structure of the projection layers and the MLPs used in the manipulation policy head are not provided. \n\n### Minor comments\n- L.13: \"action pose prediction abilities\" is not very clear\n- L.100 \"ManipVQA, enhancing robotic\" verb missing\n- L.192: what is the angle representation considered?\n- L.235: \"on when\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **_(Weakness 1 & Question 1). Detailed inference speed comparison_**\n\nThank you for the constructive comments. Inference efficiency is a crucial evaluation metric in robotic manipulation and poses a major challenge for existing Multimodal Large Language Model (MLLM) based policy models. We compare the control frequency (Hz) of our proposed RoboMamba with previous robotic MLLM. All inferences are conducted on the NVIDIA A100 GPU without any quantization or inference speed-up techniques.\n\n**Control frequency (Hz).** This measures the number of times per second the model can complete inference and generate a manipulation pose. As shown in Table 1 below, RoboMamba-2.7B achieves a control frequency of 9.0 Hz even with the highest input image resolution (336 x 336). These results demonstrate that RoboMamba not only possesses robust reasoning and manipulation capabilities but also maintains efficient inference speed. \n\nTable 1: Comparison of control frequency with previous robotic MLLMs\n|  | Input Resolution | Parameters | Control Frequence（Hz） |\n| --- | --- | --- | --- |\n| RT-1 | 300 x 300 | 35M | 3 Hz |\n| Open-VLA [a] | 224 x 224  | 7B | 3.4 Hz |\n| ManipLLM | 336 x 336 | 7B | 0.7 Hz |\n| RoboMamba | 336 x 336 | 2.7B | 9.0 Hz |\n\nMeanwhile, we compared the token output speed (tokens/s) with ManipLLM. This indicates the efficiency of Robotic MLLM in performing language output tasks (e.g., task planning or visual question answering). Specifically, we compare the number of tokens output per second by each model for the same question. ManipLLM can generate 133.1 language tokens per second after receiving 336 x 336 input images and a question. RoboMamba-2.7B is more efficient, generating 898.4 language tokens per second. These results demonstrate that our model maintains efficient inference speed when answering robotic high-level reasoning tasks using language responses.\n\n- **_(Weakness 2 & Question 3). The effect of manipulation head parameters on performance and efficiency_**\n\nWe have investigated the impact of manipulation head parameters on performance by conducting experiments detailed in Lines 625-634 and Table 5 of Appendix C. The findings reveal that the manipulation success rate across the three configurations is similar. This suggests that RoboMamba, due to its sufficient robotic reasoning capabilities, can achieve effective pose prediction skills at a low cost, as the impact of policy head parameters is not sensitive. Meanwhile, as shown in Table 2 below, the effect of manipulation head parameters on efficiency is negligible, since the number of head parameters is very small compared to the overall RoboMamba parameters.\n\nTable 2. The effect of the number of parameters in the manipulation head\n|  | MLPx2 | MLPx1 | (SSM block+MLP)x2 |\n| - | - | - | - |\n| Accuracy(Seen) | 63.7 | 62.1 | 63.2\n| Parameters | 3.7M | 1.8M | 45.2M\n| Percentage | 0.11% | 0.05% | 1.3%\n| Frequency（Hz） | 9.0 Hz | 9.0 Hz |8.6 Hz\n\n- **_(Weakness 3 & Question 2). Fine-tuning only a small-sized manipulation head on other MLLMs?_**\n\nIn Lines 296-308 and Figure 3 b) of the submission, we have conducted experiments applying the idea of fine-tuning only a small-sized manipulation head to other MLLMs. For convenience, we present the results in Table 3 below, including additional experiments with TinyLLaVA [72]. With the same fine-tuned policy head and training dataset, our RoboMamba-2.7B achieves promising results compared to other MLLMs. These results demonstrate our finding: if MLLMs possess strong robot-related reasoning abilities, they can be efficiently fine-tuned to learn robot manipulation skills. Meanwhile, the Mamba architecture exhibits strong sequence modeling ability [22]; it produces hidden-space output tokens in an autoregressive manner, effectively capturing complex semantic and spatial information in visual scenes. Therefore, RoboMamba can only fine-tune a simple and efficient MLP policy head to convert Mamba’s output tokens into action poses, rather than using a complex policy head for decoding. This significantly enhances RoboMamba's learning efficiency when facing new manipulation categories or tasks.\n\nTable 3. Fine-tuning only a small-sized manipulation head to other MLLMs\n| | Openflamingo | LLaMA-AdapterV2 | TinyLLaVA | Ours-1.4B | Ours-2.7B \n -|-| -|-|-|-\nAcc(Seen) | 0.26 | 0.46 | 0.51 | 0.39 | 0.63 \nAcc(Unseen) | 0.33 | 0.20 | 0.34 | 0.41 | 0.53 \n\n\n- **_(Weakness 4). The structure of projection layers and the MLPs_**\n\nEach MLP used in the projection layer, position head, and direction head consists of two linear layers with a ReLU activation function in between. For the projection layer, the input dimension for the first linear layer is B×N×1024, and the output dimension remains B×N×1024, where B and N represent batch size and tokens, respectively. The input dimension for the second linear layer is B×N×1024, and the output dimension is B×N×2560. For the position head and direction head, both have input dimensions of B×N×128 for the first linear layer, and the output dimensions are B×N×128. The input dimensions for the second linear layer are B×N×128, and the output dimensions are B×N×3, collectively forming the 6-DOF poses of the end-effector. We will publish all the code to ensure RoboMamba's reproducibility.\n\n- **_(Weakness 5). Minor comments_**\n\nThank you for your detailed comments; we will fix all the typos in the final version. For example: 1) We will revise Line 13 to: \"To further enhance RoboMamba with action pose predictions for robot control in SE(3) space, we explore an efficient fine-tuning strategy using a simple policy head.\" 2) We will revise Line 100 to: \"ManipVQA enhances robotic manipulation with physically grounded information processed by MLLM.\" 3) In Line 192, we utilize rotation matrices to represent pose direction. 4) We will revise Line 235 to: \" As detailed in Appendix E, we describe the key aspects each benchmark considers when assessing models in the field of robotics.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FrLo18xccj", "reviewer": "Reviewer_CsAK", "summary": "This paper proposes RoboMamba, which applies the Mamba state space model architecture for robotic manipulation policy learning. Prior MLLM-based robot policy learning finetunes transformer-based models and suffers from two major problems: reasoning capabilities degrade with visual input; and training computational cost is high for end-to-end policy learning. RoboMamaba leverages the efficient Mamba architecture to improve learning efficiency and demonstrates simple vision encoder enables visual understanding. Similarly, a simple policy head is shown to be sufficient for generating performant robot behavior after RoboMamba had converged on robotic reasoning tasks.", "questions": "What limits evaluating RoboMamba to predict one-step action? How does it generalize to multi-step or closed loop policy? Is inference spped, i.e. control frequency, the major bottleneck? \n\nHow does RoboMamba compare with more structured LLM policies such as Code as Policies [1] and VoxPoser [2]?\n\n[1] https://code-as-policies.github.io/\n[2] https://voxposer.github.io/", "limitations": "see weakness", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper’s major contribution is tailoring a state space model for robotic manipulation control. RoboMamba starts with general pretrained vision and language models and fine-tunes different parts of the network for policy learning while maintaining reasoning capabilities of the model. First, it introduces a visual input encoder for Mamba architecture and demonstrates its effectiveness to enable visual understanding within a pretrained Mamba model. Second, to pretrain the model for policy learning, it trains the model on both general VQA and robot VQA datasets. Last, it learns a light-weight policy head for predicting end-effector poses. Experiments show RoboMamba retains performant reasoning capabilities in VQA tasks while being able to generate competent policies.", "weaknesses": "The evaluation of robotic manipulation tasks is limited to one benchmark and contains only single-step open-loop action prediction (instead of learning a closed loop reactive policy). It is unclear how much training data is being used and how much variation there is in testing the policy.  At the same time, this model has many moving parts and design choices that were not carefully ablated. \n\nThe presentation of this work can be improved in general for clarity. The authors could better motivate certain design choices as well as evaluation metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **_(Weakness 1 & Question 1). Additional close-loop experiments_**\n\n1）Thank you for the constructive comments. RoboMamba is not limited to performing single-step open-loop action predictions. In the submitted paper, for fair comparison, we follow the experimental settings of the latest Robotic MLLM (ManipLLM [12]) to implement action pose prediction for articulated objects. RoboMamba can also efficiently generalize to closed-loop policy learning. Specifically, instead of directly predicting the final contact pose of the end-effector, we continuously input the current state image to predict the next end-effector pose. Additionally, we utilize an extra MLP to encode the robot's current state and concatenate it with the image features. For the close-loop benchmark, we select Meta-World [e] to validate RoboMamba's closed-loop action prediction capabilities. Meta-World is a collection of tasks in which agents command a Sawyer robot arm to manipulate objects in a tabletop environment. We consider six tasks from Meta-World: Assembly, Bin-Picking, Box Close, Coffee Pull, Hammer, and Button Press. As shown in Table 1 below, RoboMamba still achieves satisfactory results in the closed-loop experiment. Finally, we will present more closed-loop experiments in the final version.\n\n_**Table 1. The closed-loop experiments of RoboMamba, with the success rate metric.**_\n\n||Assembly|Bin-Picking|Boxclose|CoffeePull|Hammer|ButtonPress\n-|-|-|-|-|-|-\nMVP[f]|0.89|0.79|0.58|0.62|0.98|0.70\nCLIP[23]|0.71|0.68|0.72|0.80|0.90|0.48\nRoboMamba|0.98|0.80|0.80|0.94|1.0|0.98\n\n2）Since the Mamba LLM used in RoboMamba balances context-aware reasoning ability with linear computational complexity, inference speed is not a limiting factor for RoboMamba in learning closed-loop policies. We provide detailed inference speed information in **global rebuttal Q1**.\n\n- **_(Weakness 1). How much training data is being used and how much variation there is in testing the policy._**\n\nThe size of the training dataset (10K) is described in Lines 218-223 of the submitted paper. We provide details of the simulator data collection and categories in Lines 603-612 of Appendix B. Regarding the variation between training and testing data, we followed the data collection settings of where2act [45] and ManipLLM [12]. The specific variations can be divided into two aspects: 1) asset variation and 2) state variation.\n\n**1) Asset Variation**: We use 20 categories from PartNet [42] for seen objects and reserve the remaining 10 categories for unseen objects to analyze if RoboMamba can generalize to noval categories. Specifically, we further divide the seen objects into 1037 training shapes and 489 testing shapes, using only the training shapes to construct the training data. Thus, the shapes of the seen objects encountered during training and testing are different. For unseen categories, there are a total of 274 shapes, which are used exclusively in the testing data. \n\n**2) State Variation**: We observe the object in the scene from an RGB-D camera with known intrinsics, mounted 4.5-5.5 units away from the object, facing its center. The camera is located at the upper hemisphere of the object with a random azimuth between [-45, 45] and a random altitude between [30, 60]. Since the tasks involve 'pulling,' we also initialize the starting pose for each articulated part randomly between its rest joint state (fully closed) and any position up to half of its joint state (half-opened). These state settings are utilized for both training and testing data, aiming to boost the model's generalization ability.\n\n- **_(Weakness 1 & 2). Additional Ablation Study of Moving Parts and Design Choices_**\n\nThe submitted paper includes ablation studies such as the impact of MLLM reasoning ability on manipulation policy learning (Figure 3b), the effect of using different training datasets on reasoning ability (Table 4), and the impact of policy head parameters on manipulation accuracy (Table 5). However, based on your valuable comments, we find that some ablation studies are missing. To address this, we explore the impact of different image encoders and LLMs on the reasoning and manipulation capabilities of our proposed framework, while also examining the effect of using different training datasets on manipulation accuracy. Due to space limitations, please refer to **global rebuttal Q2**, which includes 1) Image Encoder Ablation, 2) Large Language Models (LLMs) Ablation, and 3) The Impact of Training Strategies on Manipulation Abilities.\n\nFor evaluation metrics, we select several MLLM benchmarks to comprehensively assess our model’s reasoning capabilities and generalizability, prioritizing those related to robotics. Detailed descriptions can be found in Lines 647-665 of Appendix E. For manipulation, we follow previous works [41, 51, 12] and selected manipulation successful accuracy as the evaluation metric, as described in Lines 240-246 of Section 4.1. Finally, we will include all the aforementioned ablation studies in the final version and conduct more analysis for each moving part.\n\n- _**(Question 2). Comparisons with structured LLM policy method.**_\n\nDue to time limitations, we reproduce the more recent VoxPoser method in the SAPIEN environment to evaluate its performance on tasks identical to ours, including 'Open Drawer,' 'Open Door,' and 'Open Faucet.' The manipulation success rates achieved by VoxPoser are 0.19, 0.36, and 0.12, respectively, which are lower compared to our results (i.e., 0.86, 0.73, and 0.30). Notably, while VoxPoser employs motion planning based on affordance and obstacle maps generated by an LLM, it still lacks comprehensive visual scene understanding. In contrast, RoboMamba possesses strong visual scene and robotic reasoning capabilities, conditioned on both image and text inputs. Therefore, RoboMamba demonstrates more robust performance in our experiments.\n\nReference:\n[e](Global rebuttal)\n[f] Real world robot learning with masked visual pre-training", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0T69J2UXCu", "reviewer": "Reviewer_iHpb", "summary": "This work introduces RoboMamba to leverage SSM model’s capabilities in non-trivial sequence modeling with linear inference complexity. A simple policy head is employed for finetuning to enable RoboMamba to predict action poses. Evaluation is conducted both in simulation(SAPIEN) and real-world settings and shows consistent improvement compared to previous sota methods such as ManipLLM. Besides, fine-tuning parameters and time are significantly reduced and inference speeds achieve 7 times faster than existing models.", "questions": "1. Can RoboMamba support more complex robot manipulation tasks?\n2. For delicate manipulation objects, RoboMamba seems have inferior performance in Table 2. Could you provide some analysis for this?", "limitations": "1. As mentioned in Question 1, RoboMamba may not support more complex manipulation tasks.\n2. RoboMamba only output 3D poses instead of direct policies.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. On both general and robotic evaluation benchmarks, RoboMamba on average all shows strong performance.\n2. Compared to previous methods, RoboMamber has fewer fine-tuning parameters and time, faster inference speed.", "weaknesses": "1. In Table 1, RoboMamba(2.7B)’s performances on MME, MMB, MM-Vet are still below TinyLLaVA’s results.\n2. In Table 2, RoboMamba does not achieve sota performance on some of the both training and test categories. \n3. Some minor mistakes like: In Figure 3(b), “Openfalmingo”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **_(Weakness 1). Comparison of RoboMamba-2.7B with TinyLLaVA_**\n\nThank you for the constructive comments. The design goal of RoboMamba is to introduce a new paradigm for adapting Mamba to multimodal robotic tasks, resulting in an innovative Robotic MLLM that integrates high-level reasoning, low-level manipulation actions, and efficiency. Our primary focus is on the robotic domain rather than general multimodal scenarios. We select various general scene MLLM benchmarks to showcase our model's generalizability. However, since RoboMamba and TinyLLaVA use different training datasets, this directly affects their scores on various benchmarks. For example, as shown in Table A1 of TinyLLaVA [81], using ShareGPT4V training datasets with different model architectures can improve MM-Vet performance. Additionally, to more comprehensively compare RoboMamba with TinyLLaVA in the robotic domain, we conducted additional experiments using high-level robotic reasoning benchmarks (RoboVQA) and low-level manipulation pose prediction.\n\n1）For a fair comparison on RoboVQA, we load TinyLLaVA's parameters and fine-tune it on the RoboVQA training set. As shown in Table 1 below, RoboMamba achieves superior performance across BLEU-1 and BLEU-4, indicating that our model possesses advanced robot-related reasoning capabilities. Meanwhile, TinyLLaVA shows stronger reasoning abilities than LLaMA-AdapterV2 (7B), highlighting its significant potential for robotic-related tasks.\n\n|**Table 1:**|BLEU-1|BLEU-4|\n-|-|-\nLLaMA-AdapterV2(7B)|27.8|8.1\nTinyLLaVA(2.7B)|43.5|29.6\nRoboMamba(2.7B)|54.9|36.3\n\n2）To compare TinyLLaVA in the manipulation, we use the same fine-tuning strategy by only fine-tuning a simple MLP policy head after the MLLM. As shown in Table 2 below, RoboMamba achieves promising results compared to TinyLLaVA and other methods. These results further support our finding that RoboMamba’s strong reasoning capabilities and framework design enhance the learning of manipulation pose prediction.\n\n|**Table 2:**|LLaMA-AdapterV2|TinyLLaVA|Ours-2.7B\n-|-|-|-\nAcc(Seen)|0.46|0.52|0.63\nAcc(Unseen)|0.20|0.34|0.53\n\nOn the other hand, inference efficiency is a crucial attribute for robotic manipulation models and poses a major challenge for existing MLLM-based methods. As shown in **global rebuttal Q1**, RoboMamba-2.7B achieves a control frequency of 9.0 Hz, while TinyLLaVA, under the same inference settings, only achieves 3.9 Hz. These results demonstrate that our approach not only delivers robust performance in the robotic domain but also offers superior efficiency and practicality. Finally, we will incorporate TinyLLaVA's dataset into our co-training stage and update all the aforementioned experiments in the final version.\n\n- **_(Weakness 2 & Question 2). The analysis of Manipulation results_**\n\nWe provide a comprehensive analysis of our method's manipulation results compared to the previous SOTA method (ManipLLM), especially for delicate objects. We conduct our analysis from two aspects: 1) different test strategies, and 2) more manipulation fine-tuning parameters.\n\n1）During inference, after contacting the object with the predicted 6-DOF pose, following where2act [45], our method simply pulls the objects along the predicted z-axis of the end effector pose. In contrast, ManipLLM adopts an 'Active Impedance Adaptation Policy (AIAP)', which heuristically selects the best pulling direction from all proposed directions (i.e., x-axis, y-axis,and z-axis). This strategy significantly increases the testing time, as it requires trying each predicted direction and then selecting the optimal pulling direction. Therefore, we re-evaluated ManipLLM by testing its accuracy in pulling objects along the z-axis. This adjustment resulted in decreased performance, particularly for delicate categories, as shown in Table 3 below. Consequently, we conclude that the improved performance of ManipLLM in some categories is attributed to the heuristic design of the AIAP interaction rather than to the model's learning capabilities.\n\n|**Table 3:**|Display|Mouse|Pliers|Remote|Foldingchair|Toaster|USB|Washing|\n-|-|-|-|-|-|-|-|-\nManipLLM(z-axis)|0.33|0.35|0.24|0.43|0.36|0.42|0.40|0.56\nRoboMamba(z-axis)|0.33|0.42|0.26|0.39|0.40|0.55|0.45|0.68\n\n\n2）However, for some delicate categories, RoboMamba still fails to achieve SOTA performance, such as Display, Mouse, Remote, Toilet, and Faucet. This leads us to propose that delicate categories are more challenging to manipulate, so we need to update more parameters during the manipulation learning process. As shown in Table 4 below, increasing the number of MLP layers to 5 improves performance to SOTA levels. Your question is very insightful, and we will continue to explore more detailed experiments.\n\n|**Table4:**|PolicyHead|Display|Mouse|Remote|Toilet|Faucet\n-|-|-|-|-|-|-\nRoboMamba|MLP×2(3.7M)|0.33|0.42|0.39|0.19|0.30\nRoboMamba|MLP×5(9.1M)|0.50|0.50|0.65|0.36|0.42\n\n\n- **_(Question 1 and Limitation 1&2). More complex robot manipulation tasks_**\n\nRoboMamba can support more complex robot manipulation tasks, such as multi-step closed-loop tasks. For the closed-loop benchmark, we selected Meta-World [e], a multi-step tabletop environment. We choose six tasks from Meta-World: Assembly, Bin-Picking, Box Close, Coffee Pull, Hammer, and Button Press. Specifically, we continuously input the current state image to predict the next end-effector pose. Additionally, we utilize an extra MLP to encode the robot's current state and concatenate it with the image features. As shown in **Table 1 of Reviewer CsAK**, RoboMamba still achieves satisfactory results in the relatively more complex multi-step manipulation experiment. For Limitation 2, instead of heuristically setting the manipulation direction, RoboMamba can also control the direction through closed-loop trajectory prediction. Finally, we will present more closed-loop experiments in the final version.\n\n- **_(Weakness 3)_** We will correct all the typos in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sX6IIYYSrK", "reviewer": "Reviewer_vV5n", "summary": "This paper proposes a Mamba-based framework called RoboMamba that utilizes multimodal state space model for robotic reasoning and manipulation. It addresses the limitations of existing MLLMs in complex reasoning and high computational costs. RoboMamba integrates a vision encoder with the Mamba model to align visual data with language embedding, enabling robust visual reasoning and action capabilities. The model uses a simple lightweight policy head for action pose prediction, requiring minimal fine-tuning. Experiments show RoboMamba excels in reasoning and pose prediction tasks, achieving faster inference speeds and state-of-the-art performance in both simulated and real-world environments.", "questions": "Please refer to the weaknesses of the paper.", "limitations": "Please refer to the weaknesses of the paper.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Using architecture with linear computational complexity like Mamba for improving efficiency fits quite well with the robotic reasoning and manipulation. \n2. The design of training strategy is simple and high-efficient, verifying the transferability of the proposed framework.\n3. Experimental results on public benchmarks as well as simulated and real-world environments demonstrate the effectiveness and efficiency of the proposed method.", "weaknesses": "1. Besides Mamba-like architecture, there exists many network architectures with linear computational complexity, like XCiT[1], xLSTM[2], hence my first concern is can these architectures work for the proposed framework? \n[1] Ali, Alaaeldin, et al. \"Xcit: Cross-covariance image transformers.\" Advances in neural information processing systems 34 (2021): 20014-20027.\n[2] Beck, Maximilian, et al. \"xLSTM: Extended Long Short-Term Memory.\" arXiv preprint arXiv:2405.04517 (2024).\n\n2. Following the first question, does Mamba has some unique attributes and advantages specially fit the robotic field? This work lack of related comparison and analysis.\n\n3. The Fig.1 seems a bit of rough, for example, the formats and colors of arrows are inconsistent, some borders have jagged edges, and the overall resolution is not high enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **_Weakness 1. Can other architectures work for the proposed framework._**\n\nThank you for the constructive comments. We explore whether other architectures could be applied to the proposed framework from two perspectives: 1) replacing the Image Encoder and 2) replacing the LLM.\n\n1）As you mentioned, we attempt to replace the CLIP encoder used in our submission with a linear computational complexity encoder (XCiT [1]). Meanwhile, we also supplemented the experiment by using SigLIP [c] as an image encoder. As shown in Table 1 below, we explore the impact of different image encoders on reasoning and manipulation abilities. The training dataset and strategy remained consistent across experiments. The results indicate that the choice of image encoder does not significantly affect reasoning ability; rather, the input image resolution is more critical, with higher resolutions improving accuracy. However, using an image encoder without cross-modality alignment (i.e., XCiT) makes it difficult to convert image tokens to LLM language embeddings. Even though our training process includes an alignment pre-training stage, this primarily serves to train the projection layer. Therefore, in future works, we aim to develop a robotic-specific image encoder capable of projecting image tokens into language embeddings while maintaining linear computational complexity. On the other hand, the manipulation results indicate that stronger reasoning abilities in RoboMamba and higher input image resolution aid in policy learning, thereby improving the manipulation accuracy. However, increasing the input resolution also results in additional inference time. Therefore, in our RoboMamba framework, we choose the CLIP (336 x 336) image encoder, which achieves a balance between extracting image semantic knowledge and efficiency.\n\n**Table 1. Ablation Study of the Impact of Different Image Encoders**\n|  | Image Resolution | OKVQA | GQA | POPE | RoboVQA- BLEU-4 | ManipACC(seen) |\n| --- | --- | --- | --- | --- | --- | --- |\n| CLIP | 224 | 63.1 | 62.4 | 85.3 | 36.3 | 52.5 |\n| XCiT [1] | 224 | 46.7 | 50.7 | 79.7 | 35.2 | 41.2 |\n| CLIP | 336 | 62.3 | 63.8 | 86.9 | 40.9 | 63.7 |\n| SigLIP [c] | 384 | 62.4 | 64.4 | 86.0 | 40.6 | 63.4 |\n\n2）We also attempt to replace Mamba with a linear computational complexity LLM in our proposed framework. However, we could not find official published pre-trained parameters for xLSTM [2] in either 2.7B or 1.3B versions. Therefore, we select RWKV-2.7B [d] and Mamba-1.4B, both with linear complexity. As shown in Table 2 below, we explore the impact of different LLMs on reasoning and manipulation abilities. For all experiments, we use the same training data and strategy. The results show that the Mamba-2.7B model possesses superior visual scene reasoning abilities and can more efficiently learn robotic manipulation skills. For other models, due to the sequence modeling capabilities of the LLM itself, the reasoning and manipulation performance achieved is limited. Therefore, we choose Mamba-2.7B as the LLM for our RoboMamba framework to balance high-level reasoning, low-level manipulation actions, and efficiency. Lastly, we will include all the insightful experiments you suggested in our final version. We will also open-source our codebase, continuously experimenting with the latest linear computational complexity architectures to provide a more efficient MLLM framework for the robotics field.\n\n**Table 2. Ablation Study of the Impact of Different LLMs**\n\n||Imageresolution|OKVQA|GQA|POPE|RoboVQA-BLEU-4|ManipACC(seen)\n-|-|-|-|-|-|-\nRWKV-3B[d]|224|32.9|41.0|67.1|9.1|46.5\nMamba-1.4B|224|28.5|40.8|66.8|34.5|50.1\nMamba-2.7B|224|63.1|62.4|85.3|36.3|52.5\nMamba-2.7B|336|62.3|63.8|86.9|40.9|63.7\n\n\n- **_Weakness 2. Does Mamba has some unique attributes and advantages specially fit the robotic field?_**\n\nThank you for the valuable feedback. We would like to explain the robotic-related attributes and advantages of Mamba from the following two perspectives: 1) Balance of Reasoning Ability and Efficiency, and 2) Sequence Modeling Ability.\n\n1）The design goal of the RoboMamba framework is to simultaneously equip the model with high-level reasoning, low-level manipulation actions, and efficiency, all of which are critical attributes in the robotics field. For reasoning abilities, as shown in Table 2 of the Weakness 1 response shows that RoboMamba exhibits stronger reasoning and manipulation capabilities than other linear LLMs (e.g., RWKV, Phi2). For inference efficiency, which is crucial in robotic manipulation and a major challenge for existing MLLM-based policy methods, RoboMamba-2.7B achieves a control frequency of 9.0 Hz. This is significantly faster than other robotic MLLMs, thanks to Mamba's linear computational complexity.\nTherefore, to balance the attributes needed in the robotics field, we choose Mamba as our LLM, as it possesses both context-aware reasoning ability and efficiency.\n\n2）Meanwhile, the Mamba architecture demonstrates strong sequence modeling ability [22] by generating hidden-space output tokens autoregressively, which effectively captures complex semantic information. During the robot manipulation fine-tuning phase, we only need to fine-tune a simple and efficient MLP policy head to convert Mamba’s output tokens into action poses, rather than using a complex policy head for decoding. This significantly enhances RoboMamba's learning efficiency when facing new manipulation categories or tasks.\n\n- **_Weakness 3. Figure 1 improvement._**\n\nAs shown in the **global rebuttal PDF**, we refined Figure 1 by using arrows with consistent formats and colors, aligning the bounding blocks, and replacing them with a higher-resolution image.\n\nReference:\n[c][d] (Global rebuttal);", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s1j4IWg4fr", "reviewer": "Reviewer_16K4", "summary": "This paper introduces RoboMamba, a Multimodal LLM, for robotic reasoning and low-level manipulation using the popular State-Space Model architecture. RoboMamba introduces a multi-stage joint language and vision training pipeline with robot-specific data fine-tuning. RoboMamba demonstrates strong reasoning performance on general and robot evaluation benchmarks while using a small amount of fine-tuned parameters.", "questions": "- Why would the baselines perform worse than RoboMamba if they're trained on the same data, unless the main factor is the architectural choice?\n- Similarly, why would RoboMamba perform better than general VLMs on those benchmarks in Table 1 and how is this comparable if they are not trained on the same data?", "limitations": "The authors mention that using MLLMs for task planning occasionally results in hallucinations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Paper is well-written and easy to follow\n- Well-motivated and comprehensive set of experiments on different general reasoning benchmarks and robot manipulation specific benchmarks\n- Training dataset and evaluation details are clearly stated", "weaknesses": "- There is not much technical novelty in the method. It seems like RoboMamba is combining CLIP with SSMs and fine-tuning on robot-specific data (RoboVQA) which is a generic pipeline that several other MLLMs works have proposed. There are no new insights from the paper.\n- The parameter-efficiency argument is mainly due to the use of SSM blocks instead of Transformer blocks. I think this is already obvious given the architecture design and is not a major contribution of this work.\n- Using the term “real-world experiments” is confusing because it implies there is a physical robot executing a trained policy to perform some task which is not the case in this paper. Please provide quantitative results for the real-world reasoning experiments if possible and also provide some failure cases of your method otherwise it seems like the qualitative examples are cherry-picked.\n- Please provide explanation as to why the RoboMamba training strategy may help learn better representation or capture knowledge that baselines or prior work fail to.\n- Ablation study in Table 4 seem to suggest that the different fine-tuning strategies doesn’t seem to provide much benefit in terms of the general reasoning benchmarks. The improvement in downstream performance is minimal if you take out certain datasets. What about for the robot manipulation benchmarks?\n- It would be nice to see experiments showing the scalability of RoboMamba, e.g. more SSM blocks, more data, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **_(Weakness 1) Technical Novelty._** Thank you for your detailed comments. We would like to reiterate the technical novelties of RoboMamba in two aspects: 1) an all-round and generalist robotic MLLM framework and 2) a robotic-specific training strategy.\n\n**1) An all-round and generalist robotic MLLM framework.** The framework designs goal of RoboMamba is to equip the model with high-level reasoning, low-level manipulation actions, and efficiency, all of which are critical attributes in the robotics field. RoboMamba is the first Robotic MLLM framework to effectively balance these three important attributes. Transferring an SSM-based LLM with a completely different architecture into a new task is not trivial. To achieve this, we innovatively chose the Mamba model, which has strong sequence modeling and linear complexity. Conversely, as shown in **Q2 2) and Table 3 of the global rebuttal response**, simply replacing the LLM in our proposed RoboMamba with another efficient LLMs results in a decline in both high-level reasoning and low-level manipulation capabilities. Meanwhile, as highlighted in **global rebuttal Q1**, RoboMamba achieves a more efficient reasoning speed than previous robotic MLLMs. The results demonstrate that building a Robotic MLLM with the aforementioned three attributes is a highly meaningful exploration and innovation, requiring appropriate architectural design and a specialized training strategy.\n\n**2）Robotic-specific training strategy.** We do not simply use a robot-specific dataset (i.e., RoboVQA and manipulation dataset); instead, we meticulously design a training strategy tailored to the robotic domain and our proposed framework. Our training strategy comprises three steps: alignment pre-training, instruction co-training, and robot manipulation fine-tuning. In the instruction co-training stage, we innovatively combine high-level general VQA data with high-level robotic data for integrated training, demonstrating that the fusion of these two data types can mutually enhance each other. As shown in Table 1 and Figure 3 a) of the submitted paper, the results demonstrate that RoboMamba possesses both visual common sense and robot-related reasoning abilities, which previous robotic MLLMs lacked. The performance improvement is due to the robotic data containing a large amount of complex multi-step reasoning data, while the general VQA data includes rich visual scene understanding data. These complement each other, providing RoboMamba with better representation. However, we did not convert low-level manipulation data to high-level data for mixed training. As shown in Table 1 below, we find that the previous SOTA method [12], which uses low-level manipulation data for fine-tuning LLM, leads to a decline in reasoning abilities compared to its pre-trained MLLM model (LLaMA-AdapterV2 [48]). In contrast, for our robot manipulation fine-tuning, we innovatively find that once RoboMamba possesses sufficient reasoning capabilities, it can acquire pose prediction skills with minimal policy head parameter fine-tuning, without compromising the inherent abilities of the MLLM.\n\n|**Table1:**|GQA|OKVQA|VQAV2\n-|-|-|-\nManipLLM[12]|42.2|30.1|56.1\nLLaMA-AdapterV2[48]|45.1|49.6|70.7\n\n- **_(Weakness 2) Parameter-efficiency._** Sorry for the confusion. As indicated in Lines 16, 54, and 71 of the submitted paper, our use of \"parameter-efficiency\" refers to the minimal parameter fine-tuning (MLP Policy Head) required for learning low-level manipulation skills. It does not suggest that SSM-based LLMs are more parameter-efficient than Transformer-based LLMs.\n\n- **_(Weakness 3) Real-world experiments._** As shown in the bottom right corner of Figure 4 (Pose Prediction), we use a physical Franka Panda robotic arm to manipulate real-world objects. The demonstration video is available in our supplementary material. We apologize for the confusion. We use a red dot to indicate the contact point and a virtual end-effector to show the direction, solely to better demonstrate RoboMamba's predicted 6-DOF pose, as the physical robotic gripper is quite large and occludes the object. Meanwhile, all the reasoning quantitative results in Figures 4 and 5 are based on real-world data from the RoboVQA validation set. Finally, based on your suggestion, we revised the visualization and provided reasoning failure cases in the **global rebuttal PDF**.\n\n- **_(Weakness 4) Better representation._** Please refer to section **2) Robotic-specific training strategy of Weakness 1**. \n\n- **_(Weakness 5) Ablation study in Table 4._** The LLaVA-v1.5 655K mix dataset is already a carefully selected MLLM training dataset [56]. The introduction of additional datasets beyond LLaVA-v1.5 is specifically aimed at enhancing the model's capabilities. We introduce LRV-INSTRUCT 400K to further prevent hallucinations in robotic tasks, which led to improvements in POPE accuracy. Additionally, we included RoboVQA to empower RoboMamba with robot-related reasoning abilities. Finally, as shown in **global rebuttal Q2 3)**, we explored the impact of different training datasets on manipulation abilities.\n\n- **_(Weakness 6) Scalability._**  We further explore the scalability of RoboMamba by increasing model parameters and training datasets. The results are shown in Table 1 of the **global rebuttal PDF**.\n\n- **_(Questions 1&2)._**\nFor manipulation experiments, differences in model architecture and training strategies led to varied performance. For reasoning experiments, different MLLM methods [48,52,56,72] are trained on different datasets, as the design of training strategies are core aspects in the MLLM field. For instance, LLaMA-AdapterV2 [48] trains on COCO Caption and ScienceQA; SPHINX [52] uses additional OCR datasets; TinyLLaVA [72] employs a 1246k pre-training dataset from ShareGPT4V. However, these papers directly compare reasoning capabilities across common benchmarks. We will integrate all your valuable suggestions in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**To all the reviewers:** First, we greatly appreciate all the reviewers' valuable comments and time. Due to character limits in the separate responses, we address some of the reviewers' questions in this global rebuttal. Please review the individual rebuttal response first and then come back to this global rebuttal; we have clearly indicated the points that refer to the global rebuttal. We hope our responses have resolved all your questions. If there are any unclear explanations, please kindly provide further comments and we are committed to promptly addressing them and providing you with a comprehensive response.

_**(To all the reviewers) Q1. Detailed Inference Speed Comparison with Other Robotic MLLMs**_
The design goal of RoboMamba is to introduce a new paradigm for adapting Mamba to multimodal robotic tasks, resulting in an innovative Robotic MLLM that integrates high-level reasoning, low-level manipulation actions, and efficiency. Inference efficiency is a crucial attribute in robotic manipulation and poses a major challenge for existing Multimodal Large Language Model (MLLM) based policy models. Therefore, we compare the control frequency (Hz) of our proposed RoboMamba with previous robotic MLLMs. All inferences are conducted on the NVIDIA A100 GPU without any quantization or inference speed-up techniques. The control frequency measures the number of times per second the model can complete inference and generate a manipulation pose. As shown in Table 1 below, RoboMamba-2.7B achieves a control frequency of 9.0 Hz even with the highest input image resolution. These results demonstrate that RoboMamba not only possesses robust reasoning and manipulation capabilities but also maintains efficient inference speed. The efficient reasoning speed makes RoboMamba more practical and scalable in a wider range of robotic downstream tasks.

_**Table 1: Comparison of control frequency with previous robotic MLLMs**_

|  |Input resolution|Parameters|control frequency(Hz)
-|-|-|-
RT-1[10]|300| 35M|3 Hz
Open VLA[a]|224|7B|3.4 Hz
ManipLLM[12]|336|7B|0.7 Hz
RoboMamba|336|2.7B|9.0 Hz



_**(To reviewer CsAK and 16K4) Q2. More ablation study on moving parts and design choices**_

**1) Image Encoder Ablation**: As shown in Table 2 below, we explore the impact of different image encoders on reasoning and manipulation accuracy. The training dataset and strategy remained consistent across experiments. XCiT [b] is an image encoder with linear computational cost that has not undergone cross-modality alignment similar to CLIP and SigLIP [c]. The results demonstrate that the choice of image encoder does not significantly affect reasoning ability; rather, the input image resolution is more critical, with higher resolutions improving accuracy. However, using an image encoder without cross-modality alignment (i.e., XCiT) makes it difficult for Mamba LLM to comprehend image features, resulting in poorer performance. Meanwhile, the manipulation results indicate that stronger reasoning abilities in RoboMamba and higher input image resolution aid in policy learning, thereby improving the final manipulation accuracy.

_**Table 2. Ablation Study on the Impact of Different Image Encoders**_

| |Image resolution|OKVQA|GQA|POPE|RoboVQA-BLEU-4|Manip ACC(seen)
-|-|-|-|-|-|-
CLIP|224|63.1|62.4|85.3|36.3|52.5
XCiT[b]|224|46.7|50.7|79.7|35.2|41.2
CLIP|336|62.3|63.8|86.9|40.9|63.7
SigLIP [c]|384|62.4|64.4|86.0|40.6|63.4


_**2) Large Language Models Ablation**_: As shown in Table 3 below, we explore the impact of different LLMs on reasoning and manipulation abilities. Given that efficiency is crucial in robotic tasks and directly affects the practicality of policy models, we compare Mamba-2.7B with other linear complexity LLMs. For all experiments, we utilize the same training data and strategy. The results demonstrate that the Mamba-2.7B model not only possesses linear complexity but also efficiently gains visual reasoning abilities through our proposed training strategy. Meanwhile, after robot manipulation fine-tuning, it achieves outstanding manipulation accuracy. Therefore, we chose Mamba-2.7B as the LLM for our RoboMamba framework to simultaneously provide high-level reasoning, low-level manipulation actions, and efficiency.

_**Table 3. The Impact of Different LLMs**_

||Imageresolution|OKVQA|GQA|POPE|RoboVQA-BLEU-4|ManipACC(seen)
-|-|-|-|-|-|-
RWKV-3B[d]|224|32.9|41.0|67.1|9.1|46.5
Mamba-1.4B|224|28.5|40.8|66.8|34.5|50.1
Mamba-2.7B|224|63.1|62.4|85.3|36.3|52.5
Mamba-2.7B|336|62.3|63.8|86.9|40.9|63.7

_**3) The impact of training strategies on manipulation abilities:**_
In Lines 614-624 and Table 4 of the submitted paper, we explored the impact of different training strategies and datasets on RoboMamba’s reasoning abilities. As shown in Table 4 below, we further examine the impact of various training datasets on manipulation abilities. In this table, AP refers to alignment pre-training, and IC refers to instruction co-training with different dataset combinations. The results indicate that each dataset contributed to improved manipulation policy learning, particularly with the application of the RoboVQA dataset. This further supports our finding that when RoboMamba possesses sufficient reasoning capabilities, it can facilitate the learning of low-level manipulation, especially when it has robotic-related reasoning abilities.

_**Table 4. Ablation study of training strategies on robot manipulation benchmark.**_

||AP|IC(LLaVA-655K)|IC(LRV-400k)|IC(Robo-800k)|GQA|POPE|Manipulation ACC(seen)|
-|-|-|-|-|-|-|-
|Ex1||✓|||62.2|85.5|58.2|
|Ex2|✓|✓|||62.7|85.9|58.8|
|Ex3|✓|✓|✓||62.6|86.6|60.3|
|Ex4|✓|✓|✓|✓|63.8|86.9|63.7|

Reference: 
[a] OpenVLA: An Open-Source Vision-Language-Action Model; 
[b] Xcit: Cross-covariance image transformers; 
[c] Sigmoid loss for language image pre-training; 
[d] RWKV: Reinventing RNNs for the Transformer Era; 
[e] Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.

Meta-review: The submission initially received mixed reviews; the authors did a great job during the rebuttal, after which all reviewers became positive about the submission.  The AC agrees with the recommendations.  The authors should incorporate the rebuttal into the camera ready.
Decision: Accept (poster)