AC-DiT: Adaptive Coordination Diffusion Transformer for Mobile Manipulation
Authors: Sixiang Chen
Jiaming Liu
Siyuan Qian
Han Jiang
Zhuoyang Liu
Chenyang Gu
Xiaoqi Li
Chengkai Hou
Pengwei Wang
Zhongyuan Wang
Renrui Zhang
Shanghang Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Mobile Manipulation; Robotic Manipulation
Primary area: applications

Abstract: Recently, mobile manipulation has attracted increasing attention for enabling language-conditioned robotic control in household tasks.
However, existing methods still face challenges in coordinating mobile base and manipulator, primarily due to two limitations.
On the one hand, they fail to explicitly model the influence of the mobile base on manipulator control, which easily leads to error accumulation under high degrees of freedom.
On the other hand, they treat the entire mobile manipulation process with the same visual observation modality (e.g., either all 2D or all 3D), overlooking the distinct multimodal perception requirements at different stages during mobile manipulation.
To address this, we propose the Adaptive Coordination Diffusion Transformer (AC-DiT), which enhances mobile base and manipulator coordination for end-to-end mobile manipulation.
First, since the motion of the mobile base directly influences the manipulator's actions, we introduce a mobility-to-body conditioning mechanism that guides the model to first extract base motion representations, which are then used as context prior for predicting whole-body actions. 
This enables whole-body control that accounts for the potential impact of the mobile base’s motion.
Second, to meet the perception requirements at different stages of mobile manipulation, we design a perception-aware multimodal conditioning strategy that dynamically adjusts the fusion weights between various 2D visual images and 3D point clouds, yielding visual features tailored to the current perceptual needs.
This allows the model to, for example, adaptively rely more on 2D inputs when semantic information is crucial for action prediction, while placing greater emphasis on 3D geometric information when precise spatial understanding is required.
We empirically validate AC-DiT through extensive experiments on both simulated and real-world mobile manipulation tasks, demonstrating superior performance compared to existing methods.

AC-DiT: Adaptive Coordination Diffusion Transformer for Mobile Manipulation
Sixiang Chen 1:: Jiaming Liu Siyuan Qian Han Jiang', Xiaoqi Li', Renrui Zhang', Zhuoyang Liu', Chenyang Gu Chengkai Hou' Pengwei Wang',
Zhongyuan Wang', Shanghang Zhang 1.4
State Key Laboratory Multimedia Information Processing. School of Computer Science, Peking University; NNajjig University; CUHK; 'Beijing Academy of Artificial Intelligence (BAAI) Project web page: ac-dit.github.io
Abstract
Recently, mobile manipulation has attracted increasing attention for enabling language-conditioned robotic control in household tasks. However, existing meth- ods still face challenges in coordinating mobile base and manipulator, primarily due to two limitations. On the hand, they fail to explicitly model the influence of the mobile base on manipulator control, which easily leads to error accumulation under high degrees of freedom. On the other hand, they treat the entire mobile manipulation process with the same visual observation modality (e.g., either all 2D all 3D), overlooking the distinct multimodal perception requirements different stages during mobile manipulation. To address this, we propose the Adaptive Coordination Diffusion Transformer (AC-DiT), which enhances mobile base and manipulator coordination for end-to-end mobile manipulation. First, since the motion of the mobile base directly influences the manipulator's actions, we introduce a mobility-to-bol -o-yyoyy conditioning mechanism that guides the model to first extract base motion representations, which are then used as context prior for predicting whole-body actions. This enables whole-body control that accounts for the potential impact of the mobile base's 's motion. Second, to meet the perception requirements at different stages of mobile manipulation, we design a perception- aware multimodal conditioning strategy that dynamically adjusts the fusion weights between various 2D visual images and 3D point clouds, yiclding visual features tailored to the current perceptual needs. This allows the model to, for example, adaptively rely more 2D inputs when semantic information is crucial for action prediction, while placing greater emphasis on 3D geometric information when precise spatial understanding is required. We empirically validate AC-DiT through extensive experiments on both simulated and real-world mobile manipulation tasks, demonstrating superior performance compared existing methods.
1 Introduction
Mobile manipulation, which integrates mobile platforms with manipulative capabilities to perform complex houschold tasks under natural language conditions, has gained substantial attention. Some two-stage approaches [ 2
3 4 [5] leverage task planners (c.g., VLMs or LLMs) to decompose long-horizon tasks into sub-tasks, which are then executed sequentially by atomic skill modules. In contrast, end-to-end methods 7 g 9 10) typically employ imitation learning or reinforcement learning to jointly optimize the control of both the mobile base and the manipulator. However, these
"Equal contribution, Project lead, "4 "Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

AC-DIT
Performance
Real-Worid Experiments
Image
Print Clood
Lazgroge
Saewc to
callernes"
AC-DIT
Imagyy Proodkrr
Teat Ensdder
Proutze
Cucumber basket
Store breads
Hung
table
Simulation Experiments
Real Studdation
Exterie View
Bateriee View
Wiit View
Lighteggtt
Preceplan-nnaeee Mattimodal Adapeation
Gidac tic appic:
kitches countic andffick bowk.
RoholTonn
Maripoiction Action Head
Mabilk-ta-Bee Budy
DDa 100 #501
Hascorn
Bontes Palk
Vieei
Figure I: Overview of AC-DiT. The proposed end-to-end mobile manipulation framework enhances the coordination between the mobile base and the manipulator by introducing two key mechanisms: mobile-to- to-boyy conditioning and perception- aware multimodal adaptation. The former enables action prediction that conditioning on how upcoming mobile base movements may affect manipulator control, thereby reducing error accumulation. The latter constructs multimodal features tailored the perception requirements at different stages of the mobile manipulation process. Under this paradigm, AC-DiT demonstrates superior performance in both simulation and real-world environments.
approaches still face significant challenges in effectively coordinating the two components. One key limitation is the lack of explicit modeling of the coordination relationship between the mobile base and manipulator control, which is erucial because even minor errors in the mobile base's movement can severely impact the manipulator's performance. In addition, they use the same visual observation modality throughout the entire mobile manipulation process, such consistently relying on either 2D images or point clouds as input, which overlooks the distinct visual perception requirements of each stage. For example, during object localization, semantically rich 2D images enable better scene understanding and target identification, while during interaction, geometrically detailed point clouds are essential for ensuring precise manipulation. Building on these challenges, we ask: "Can we develop an end-to-end mobile manipulation framework that effectively leverages the relationship between the mobile base and the manipulator, as well the distinct characteristics of different task stages, to achieve accurate coordination?"
To solve the above challenge, as shown in Figure propose the Adaptive Coordination Diffusion Transformer (AC-DiT), an end-to-end diffusion transformer (DiT) model for mobile manipulation. First, to better formulate the relationship of mobile base while mitigating potential error accumulation from the base's motion that could affect manipulator control, we introduce a mobility-to-De to-dyy condition mechanism enhance whole-body coordination. Specifically, we attach a lightweight mobility policy head (c.g., DiT (IT) to the encoders and pretrain it using only mobile base actions, enabling it to extract latent mobility features that capture base motion representation. These features are then used as conditional inputs in the subsequent training of the mobile manipulation action head, guiding the prediction of mobile manipulation actions. By providing prior information about the mobile base's 's movement, the model can better plan its actions in response to the expected movement of the base, thereby improving the overall coordination and avoiding error accumulation.
Second, to adapt the perception system to the varying demands of different stages in the mobile manipulation process, we propose a perception-aware multimodal adaptation mechanism that dynamically assigns appropriate weights to different visual modalities. Specifically, given extracted language features, image features from multiple views, and point cloud features, we first project them into a shared feature space using lightweight projector. We then compute the cosine similarity between each visual modality (image and point cloud) and the language features estimate their importance. These similarity-bai -based weights are applied to the original visual tokens and used to produce pereeption-avee aware visual representations, which are then fed into policy head for action prediction. In this way, we empirically discover that AC-DiT can adaptively adjust its reliance on different modalities based on stage context-for example, relying more on 2D inputs when semantic information is more critical, and focusing more on 3D inputs when geometric precision is required. Moreover, it can actively downweight the influence of uninformative views, such as images that capture only floors or walls.

---

To comprehensively evaluate AC-DiT, as shown in Figure we conduct extensive experiments across multiple tasks, including both simulated enviromments (e.g.. ManiSkill-HAB [2]) and real-world scenarios. We compare our method with several imitation learning (e.g., DP [3] DP3 [1], and
RDT IT] and mobile manipulation approaches (e.g., ACT 6 and To [SS), ) and our method
demonstrates a significant advantage over others. In summary, our contributions are as follows:
We introduce the Adaptive Coordination Diffusion Transformer (AC-DiT), an end-to-end framework designed to enhance coordination for mobile manipulation by explicitly modeling the relationship between the mobile base and the manipulator, and by adapting the distinct visual perception requirements at different mobile manipulation stages.
To achieve this, we propose mobility-to to-bbiy conditioning mechanism that conditions over- all mobile manipulation actions on latent mobility representation. enhancing coordination and reducing cumulative errors. We also design perception aware multimodal adaptation mechanism that adaptively assigns importance weights to different visual inputs to satisfy the perception requirements of each stage in the mobile manipulation process.
Experiments demonstrate that our method outperforms state-of-the-art baselines in both simulated and real-world mobile manipulation tasks, highlighting its enhanced coordination and robust action generation capabilities.
2 Related works
Mobile Manipulation. Mobile manipulation tasks require robots to coordinate the motions of mobile base and a manipulator arm in response to human instructions. A common approach [I] 2 5 leverages a high-level planner [16] 17), typically a Vision-Language Model (VLM), to
decompose tasks into subtasks, which are then sequentially executed by low-level atomic skill
modules (e.g., imitation learning [18 6 7 19 20), reinforcement learning 21 22 23
24 25]. or
foundation models 266 27) Alternatively, end-to-end frameworks jointly model the degrees of freedom of both the base and the manipulator within unified action space, using reinforcement I0] to enable holistic optimization for scenarios requiring concurrent or rapidly alternating locomotion and manipulation. However, existing methods rely on single-modale granularity required by mobile manipulation. Moreover, they fail to explicitly capture the kinematic and dynamic interdependencies between locomotion and manipulation. Therefore, we propose AC-DiT to address these critical gaps in an end-to-end manner.
learning 28 29 30 or imitation learning 6
7
g 9
perception (either 2D or 3D) and overlook the modality-specific perceptual
Dision-Language-ae
(VLA) models. Visual- Sosual-Language-ACee
(VLA) models empower large-
scale models interpret visual and linguistic inputs while generating corresponding robot actions [31 34.. Representative works such as VLA (35), RT-2 [36]. and ManipL.LM (37) discretize
32 33
the continuous action space into bins and adopt next-token prediction for action generation, which inevitably compromises the natural continuity of robotic motion. To mitigate this, recent methods replace the VLM's tokenizer with continuous-value regression heads [38
39 40
42 (c.g.. MLPs,
LSTMs), or integrate diffusion-based transformer modules 113 43 14
46 48 to
better capture the multimodal distribution of robot trajectories. Despite these advancements, existing VLA frameworks remain primarily limited to tabletop manipulation tasks, leaving the complex challenges of mobile manipulation largely unexplored. To address this gap, we propose the AC-DiT architecture, which adopts diffusion based action generation paradigm and is specifically designed to meet the distinct perceptual demands of different stages in mobile manipulation, while explicitly modeling the interdependencies between the mobile base and the manipulator.
3 Methods
In Section 3.1 and 3.2 we begin with the problem formulation and overall architecture of our proposed AC-DiT framework. In Section 3.3 and Section 3.4 we provide detailed introduction to the mobility-to-body conditioning mechanism and perception-aware multimodal adaptation mechanism, respectively. We then demonstrate training objectives in Section 3.5
3.3
3

---

Leit-wrist
Exterior
Right-wrist
Point cloud
AC-DiT
Language Input
Noised
Mobility Action
c+d 4+
Image Encoder
3D Encoder
Text Encoder
Fine-tune
wi LoRA
Lightweight Mobility Action Head (DiT)
Mobile Base
Actions
MLP
MLP
MLP
Denoising
Cosine
Latent Mobility
Similarity
Feature
Sdas
W
X
Mobility-to- Body
Perception-aware Multimodal Adaptation
Conditioning
Conditions
Robot
state
Self- Attention
Cross Attention
Self Self-Attennl Attention
Cross- Attention
Self-Artention
Cross- Attention
arei
Mobile Manipulation
MLP
auu
aqez
Action Head (DiT)
Control
Clakk
Noised action
Figure 2: AC-DIT framework. We first train the modules in the grey- grey-shaded region under the supervision mobile base actions, allowing the lightweight mobility action head to learn extract latent mobility features. After this, we optimize the entire AC-DiT model, enabling the mobile manipulation action head to predict both mobile base and manipulator actions. With the Mobility-to Body Conditioning mechanism, this action head conditions the extracted latent mobility features, allowing whole-body action prediction to account for the influence of mobile base motion. Meanwhile, the Perception-Awad Multimodal Adaptation mechanism enables this action head to adaptively assign different importance weights to various visual input features, resulting in a perception-awar visual condition tailored to the perception needs of different manipulation stages.
3.1 Problem Formulation
We model mobile manipulation as temporal sequence prediction problem, where the agent processes a history of observations 0 = (o,-r.... .............. 0,} and language condition & to predict a future action sequence A++1,++ (a.-I......8 a++*) The observation o, each timestamp is represented as o, (V V 3D: že, c), where V.2 contains images from exterior, left-wrist, and right-wrist cameras, ViD represents the point cloud captured at timestamp t, 2, the robot state including the linear velocity and the angular velocity of the mobile base and joint positions of the manipulator, and the control frequency of the robot. Meanwhile, we define coordination in mobile manipulation as end-to-end control paradigm where multi-component robot (in our case, mobile base and arm system) simultancously generates synchronized action decisions for all components at each time step 7 [5]. rather than employing sequential phase-based control strategics. Therefore, the predicted action chunk a++1:4+K includes the mobile base control a I:t+ (linear/angular velocities)
buse
and the manipulator control manipulator (joint positions). The goal is learn end-to-end policy *++1++k
(14 124 (), which is trained on dataset D {(e(1) of ( ())
< 10
of N trajectories, where ( is the length of the i-th trajectory.
3.2 Overall Architecture
As shown in Figure 2 AC-DiT comprises an image encoder, 3D encoder, and text encoder to extract features from 2D images of three camera views, 3D point cloud, and language input,

---

respectively. To enable a unified feature representation across modalities, we adopt SigLIP A9 as the backbone for all three encoders. Specifically, the SigLIP image encoder is used to extract exterior view feature F2p. west- view feature F' 22 right-wrist view feature F2n. which are then concatenated as 2D visual features F2D. For the point cloud input, we follow the strategy introduced in Lift3D [50]. which transforms point clouds into token sequences compatible with 2D foundation models (e.g., SigLIP) with LoRA adapter [51]. This is achieved using 3D tokenizer that processes the raw point cloud and aligns each 3D point with multiple 2D positional embeddings derived from several virtual planes projected from the 3D coordinates. The resulting 3D tokens, enriched with positional information, are then fed into the SigLIP image encoder to extract 3D geometric features F3D. For language input, we utilize the SigLIP text encoder to extract language features Fg.
AC-DiT incorporates two action heads: a lightweight mobility action head H, and a mobile manipu- lation action head H. The lightweight mobility action head Hi H N stacked DiT blocks followed by a final MLP with a total of only 170 million parameters- the features from all three modalities-imagee images, point clouds, and language its input. It aims to extract latent mobility feature that captures the representation of mobile base control, which then serves as mobility prior for the mobile manipulation action head H. (refer to Section B.3). Beyond the latent mobility feature, the mobile manipulation action head H. also takes the reweighted multimodal features- adaptively adjusted according to the varying perceptual demands of different mobile manipulation states as input (refer to Section (344), and outputs both mobile base and manipulator actions to control the movement of robot. The detailed model parameter summary is shown in Appendix
3.3 Mobility-to-Body Conditioning
The core premise of this mechanism is that mobile base motion provides essential contextual priors for effective manipulator planning [9. To explicitly capture the substantial influence of base movement on manipulator control and reduce error accumulation in geggh-dere-oo-frreedeee propose the Mobility-to-Bol Bo-Body Conditioning mechanism. Its goal is to incorporate prior knowledge of base motion into full-body action prediction, facilitating robust and coordinated control between the mobile base and the manipulator.
(DoF) systems, we
Specifically, before training the full AC-DiT model n, we first pretrain all modality encoders together with the lightweight mobility action head Hi (i.e., the modules highlighted with gray background in Figure[] for a few epochs. This pretraining phase aims to enable the model to learn to predict mobile base actions effectively. After pretraining, we enable the lightweight action head Hi to extract latent mobility features that effectively capture the base motion representation. These features are then used as informative mobility -related priors for the subsequent full-body action prediction.
Next, we optimize the full AC-DiT model, which employs the mobile manipulation action head H to predict both mobile base and manipulator actions. These predictions are conditioned not only on features extracted from all modalities but also on the latent mobility features generated by the pretrained lightweight mobility action head Hi. Specifically, to extract these latent mobility features, we collect the action tokens output by the final DiT block in the lightweight mobility action head Hi (prior to the final MLP layer) at each of the five denoising time steps. These tokens are then concatenated to form the latent mobility feature Fm. consisting of mobile base action representation. This latent feature is concatenated with the multimodal features, and the combined feature is fed into the cross-attention layers of mobile manipulation action head H. enabling the generation of both mobile base and manipulator actions conditioned on the mobile base prior. By doing so, this conditioning mechanism enables the manipulator to plan its actions with awareness of base movement, thereby enhancing coordination and improving the overall effectiveness of whole-body control.
3.4 Perception-Aware Multimodal Adaptation
To address the diverse perceptual needs across different stages of mobile manipulation, such as relying on semantic, rich 2D information for localization and geometrically precise 3D information for interaction-we design perception-aware multimodal adaptation mechanism. This mechanism dynamically computes the similarity of 2D images and 3D point cloud modalities relative to language instructions, then adaptively assigns different importance weights to generate stage-specific visual representations.
5

---

Specifically, we firstly use three MLPs projectors (ProjzD. PRoj&D and Proje) to project 2D visual feature 22 consisting of three camera views, 3D geometric features Fan- and language feature F, into shared latent space. We then compute the cosine similarity between each visual modality feature- feature-speciffe specifically, the 2D image features from three views (front, left, and right) and the 3D point cloud feature- and the language feature. These similarity scores are subsequently normalized to obtain the importance weights for each modality the current stage, denoted as w/ w/, wt, w.. The formulation is as follows:
{w: w w = cos(Projz (F2D), Proje (Fr))
(1)
w. = cos(Projan) (FDD), Proje (Fr)
(2)
FL. P2
Fy = w/F2D F w F2D: F. Concat( (f F Pt FT FW)
F = w.Faa
(3)
(4)
In this formulation, wt is scalar weight and 22/3n are feature vectors, resulting in scalar-vector multiplication. The reweighted visual features are then concatenated to form the perception-aware visual representation F,. This weighted visual feature F., together with language features Ft and latent mobility features Fm, serves the conditioning imput for the mobile manipulation action head. This aims to measure the relevance of different visual observations to the different mobile manipulation stages. For example, can adaptively assign less weight to left-wrist right-wrist view images when it shows limited information (c.g., capturing floor or wall), or taking more 3D geometry feature into account when it is about manipulate the target object. We then apply importance weights on corresponding features to obtain perception-awan language feature F, and extracted latent mobility feature F m. forming the conditioning input for the mobile manipulation action head to predict both mobile base and manipulator actions. In this way, different importance weights adaptively assigned to each visual input, satisfying the distinct perceptual demands at various stages of the mobile manipulation process and enabling AC-DiT to leverage appropriate perceptual priors for robust action prediction.
visual features F., together with
3.5 Training Objectives
During pretraining lightweight mobility action head to extract latent mobility feature, we only finetune the LoRA adapter on 3D encoder and the mobility action head, while keeping the rest of the model frozen. The predicted mobile base actions are supervised under denoising MSE loss. When training the model to predict both mobile base and manipulator actions, update modules with fire icon in Figure 2] except those from the pretrained SigLIP model. The predicted mobile base and manipulator actions also supervised by denoising MSE loss.
4
Experiments
In Section 4.1
we compare the mobile and bimanual manipulation capabilities of our method against prior approaches in simulation environments. Section presents both quantitative and qualitative results of AC-DiT in real-world scenarios. Finally, Section 4.3 evaluates the effectiveness of each component through detailed ablation studies.
4.1 Simulation Experiments
4.1.1 Mobile Manipulation Experiments
Benchmark. We employ the Maniskill-Hab (MSHab) simulation benchmark, built upon the Sapien simulator [52]. MSHab further benefits from the integration of ReplicaCAD's [53 photorealistic scene assets and YCB's [54] standardized object models, enabling rich, interactive environments for diverse manipulation tasks. Our experiments include 7 tasks within the Set-Table scenario: pick_apple, place_apple, pick_bowl, blace_bowl open_ open_fridge, open_kitchen coouttrr and flose_kitcher wwwthheenccunneeeee counter. Data collection. Following MSHab [12), training trajectories are collected with agents trained by reinforcement learning (RL) algorithms. Specifically, PPO [22 used for open and close tasks, while SAC 21 is used for pick and place tasks. Additional

---

Table Simulation results on mobile manipulation Maniskill-Hai Benchmark. Mobile refers the target location the agent should navigate to (e.g.. Fridge or Counter). while Manipulation specifies the task the agent should perform upon arrival (e.g., Pick Apple).
Mobile
Fndge
Table
Fridge
Counter
Table
Counter
Counter
Mompulltion
Pick
Place
Open
Fick
Place
Open
Close
Obj.
Apple
Apple
Door
Bowl
Bowl
Drawer
Drawer
Mean S.R.
Alg.
ACT
2.2
8.7
3.3
2. 22
28.0
13.0 0.8
0.0
85.7 60.712 1.2
23.6
DP
21.3 3.3
28.0 6.0
T.3 4 5.8
20.7 1 3.3
69.3 1: 3.3
0.0 0.0
55.0 i 5.7
28.8
3DP
0.0
31.0
0.0
20.0
92.0 0.8
0.00
68.0
21.6
RDT
12.0 11.3
32.0 5.7
82.7 10.5
10.7 6.8
18.7
44.0
100.0 0.0
42.9
EquiBot
5.3 0.6
1:.3
16.3 1: 3.2
4.3 4.0
3.0 3.0::10 1.0
53.7 3:2
16.0
To
1..
1.7
31.3++2.6
15.7
21.3 2.6
60.0 + 0.8
70.0 2.2
33.5
AC-DiT
33.3=1. 1.9
934
6.0
6.0
17.3
81.3
97.3 = 1.9
55.6
details of simulated mobile manipulation experiments are shown in Appendix C.1 We generate 1000 successful demonstrations for each task and ensuring the same demonstrations are used by all baseline algorithms. Baselines. Due to the limited availability of open-source implementations for many state-of-the-ar baselines 2 g [0] for mobile manipulation, which inherently complicates reproducibility, we focus our comparative analysis on end-to-end imitation learning methods for mobile manipulation that are publicly accessible and technically reproducible. Therefore, we adopt three representative methods as our baselines: Diffusion Policy [3] (DP), a representative 2D imitation learning approach; 3D Diffussin Policy [14] (3DP), representative 3D imitation learning approach; Robotics Diffusion Transformer (RDT) [.]. diffusion-based foundation model for generalizable manipulation; EquiBor [55], a diffusion-basee method for mobile manipulation; and model capable of mobile manipulation. Evaluation metric. For all methods, we evaluate 100 episodes 3 times and then compute the average success rate. We report the mean and standard deviation of success rates across the three runs. A successful test episode is defined as one in which the agent completes the task within maximum of 200 steps. Training Details. To ensure fair comparison across methods, we employ identical observation and action spaces for all algorithms. For the observation space, we adopt historical sliding window of length 2, incorporating the base velocity and absolute positions of upper-body joints. In addition also
To [1). flow-based risionlanguageeactionn
to align with the recommended settings provided by MSHab and ensure fair comparison, incorporate several values into the robot states. These include the goal pose, object pose, end-effector pose relative to the base, and binary indicator of whether the object is grasped. The action space mirrors this structure but uses a horizon of length 2 and replaces absolute joint positions with relative positional changes, while maintaining absolute base velocities. For baselines, we follow their original settings.
Result Analysis. We visualize the executed tasks in the top part of Figure 3 From the results in Table T. AC-DiT consistently achieves the highest success rates in most tasks, with a mean success rate of 7%. outperforming all baselines by significant margin. For example, AC-DIT excels in challenging tasks such as "go to fridge, then open the door" (90.7%) and "go to counter, then close the drawer" (97.3%), demonstrating its strong capability in precise and coordinated of mobile manipulation. In contrast, methods like DP and 3DP exhibit relatively low performance across most tasks. In particular, 3DP struggles with processing large-scale scene-level point clouds during navigation, which makes it difficult to accurately approach the target region and interact with the target object. While the Robotics Diffusion Transformer (RDT) achieves reasonable success on some tasks, it falls short in others due to its lack of explicit modeling of the coordination between the mobile base and the manipulator. As a result, errors in the mobile base's movement tend to accumulate and adversely affect the manipulator's performance during execution. Since EquiBot is a diffusion algorithm that relies on object-centric point clouds, while our experimental environment supports scene-level point clouds, such point clouds may cause some interference to the algorithm, which could be the reason for its lower scores. As for To
[15], while it demonstrates competitive performance exceeding ACT (6) DP 13]. and DP3 [4] it is outperformed by AC-DiT in mobile manipulation tasks. One key reason that piO [SS] lacks explicit modeling of the locomotion- manipulation interplay, despite successfully demonstrating the feasibility of end-to-end control in specific scenarios (such as clothing pickup and folding). This performance gap underscores the critical contributions of AC-DiT's perception-awar multimodal adaption mechanism and mobility-to-box conditioning framework, which enable seamless integration of mobility and manipulation through dynamic coordination of motor loops. Overall, the results indicate that AC-DiT's adaptive
7

---

Table Simulation results on dual-arm RoboTwin Benchmark.
Handover
Contrainer Place
Cup Place
Bottles Easy
Bottles Hard
Pick Apple
Mean S.R.
DP
0.7 0.5
33.7 = 2.9
56.1 16.1:.11
81.0 3.3
49.0
6.7 1.2
37.9
3DP
73.3 5.7
74.7 +.3
83.7 10.7
61.0 4.9
65.3 +.7.
RDT
85.3
18.0
56.0 1 3.3
48.0 1. 6.5
93.3 1 3.8
68.0 9.8
69.3 8.2
70.0
AC-DIT
100.0 - 0.0
81.3 1 8.2
86.7 = 13.6
100.0 0.0
84.7 1 4.7
88.0 3.3
90.1
coordination and perception mechanisms significantly enhance task success across diverse mobile manipulation scenarios.
ManiSkill-HAR
Move Fridge Pick Apple
Move to Table Place Apple
Move to Fridge Open Fridge
Move Table Pick Bowl
Move Table Place Bowl
Move Counter
Move Counter Close Drawer
Open Drawer
Robo Twin
Block Handover
Confainer
Emply Cup
Bottles Easy
Bottles Hard
Apple Messy
Figure 3: Robot execution visualization of tasks in mobile simulator ManiSkill-HAB and 6 tasks in bimanual simulator RoboTTwin
4.1.2 Bimanual Manipulation Experiments
The proposed training mechanisms can be readily extended to dual-arm robotic tasks, enhancing coordination between the arms. Specifically, the perception-aware multimodal adaptation mechanism can be seamlessly applied to bimanual scenarios, while the mobility-to-body conditioning mechanism can be adapted into dual-arm-tor aum--o-duallarm conditioning mechanism. This facilitates cross armm prior conditioning, enabling each arm to anticipate the other's actions before action prediction, thereby improving bimanual coordination. Therefore, evaluate the effectiveness of these training mechanisms the bimanual setting and assess their effectiveness on dual-arm coordination.
Benchmark. We use RoboTwin [56] as our simulation environment, which is also built upon the Sapien simulator [52]. We adopt six commonly used dual-arm tasks from RobotWin: Mockkhandover container_plaaee
ddwal_bottlee __w_ttoesecc__________ harrd, empty_cup_ ppaaee _ppespppple_eeye....meee messy. Additional details of simulated bimanual manipulation experiments are shown in Appendix C.2 Data collection. In RoboTTwin demonstrations are automatically generated by motion plamning tools on the basis key poses annotated by GPT-4 [16]. We generate 1000 tra- jectories for each task. Result Analysis. We visualize the executed tasks in the bottom part of Figure 3 We adopt the same basclines and evaluation metrics as described in Section 4.1.I which 3DP is used by RoboTTwin evaluate bimanual performance, while RDT also supports bimanual manipulation. can be observed in Table 2 that the Diffusion Policy (DP) performs poorly across all tasks, with average success rate of only 38.2%. In contrast, though the 3D Diffusion Policy (3DP) and Robotics Diffusion Transformer (RDT) show more stable and consistent performance, our proposed method, AC-DiT, achieves the best results across all tasks, demonstrating its strong bimanual coordination performance. These results demonstrate that our proposed method can be effectively extended to bimanual tasks, enhancing coordination between multiple robotic arms and highlighting the scalability of AC-DiT.
pootles_picc__o_
and pock pick_apple
4.2 Weal-World experiments
Dataset collection. In our real-world robot experiments, we employ the Agilex Cobot Magic plat- form, which consists of four Agilex Piper arms and a Tracer mobile base. In addition to the three Orbbec Dabai cameras located at the wrists and chest positions, we apply an extra RealSense L515 depth camera on the head to capture point clouds. We conduct four mobile manipulation tasks: mo- bile bile_pick_ _icpppplccceeee___eppeppppepa bread, mobile crobileppick_ pick_place- lall__plaaaaaeeeeeaaa cumcuber, mobile_cledd clean_table, and mobile mobile_hong hang_lowwr These tasks involve various objects and manipulation actions, and they all long- mor-horizon tasks,

---

Task Progress
Task Progress
Hong Towel Cucuubber in Basker
"Pick the cucumber und move it to the busker"
"Put the breads into the hasket, lift the basket and tum left."
Clear Table
F
"Pick up the lowel, hand it to right band, move bang up
"Move the trash on the table and collect them to the bowl."
Figure 4: Robot execution progress of AC-DiT in four real-world tasks.
Table 3: Comparison with baselines in the real world. We report the success rate of the overall task
(e.g., cucumber in basket) as well its corresponding sub-tasks (e.g., move, grasp, move, place).
Task
Cucumber in Basket
Store Breads
Tang Towel
Clean Table
Handows
More
Graip
Mov
Placet
Gralp
Place
Lik
Moo
Place
Gaa
Move
Hang
Bowl
Bonte
Speite
Move
Peal
Nopkin
0.0
6.3
ACT
68.8
6.3
0.0
50.0
12.5
12.5
0.0
0.0
37.5
81.3
6.3
0.0
6.3
68.8
25.0
18.8
12.5
18.8
37.5
31.3
25.0
18.8
12.5
75.0
53.0
62.5
68.8
68.8
56.3
81.3
75.0
68.8
81.3
56.3
68.8
43.8
813
56.3
75.0
68.8
62.5
50.0
50.0
56.3
43.8
43.8
AC-DIT
93.8
75.0
68.8
93.8
87.5
100.0
81.3
87.5
81.3
75.0
$1.3
68.8
68.8
100.0
75.0
$1.3
56.3
81.3
75.9
including at least subtasks. For each task, we collect 100 spatially generalized demonstration trajec- tories frequency of 30 fps. Due space limitations, details of the hardware setup, real-world experimental settings, and corresponding results are provided in Appendix A and B respectively. Training and Evaluation Details. We train a single model for all the tasks. For evaluation, evaluate the model for 17 trials with diverse initial object poses. Given that our real-robot tasks are all nong-horizon in nature, we conduct tests each atomic subtask of these long-horizon tasks to enable more granular comparison of each method. Note that if the previous sub-task fails, we manually reset allow evaluation of its performance in the next sub-task. However, for the overall task success rate, all sub-tasks must be completed successfully.
Quantitative Results. As shown in Table 3 compare AC-DiT with ACT 66) method imple- mented for bimanual mobile manipulation, and To [15), a caaiionllnguage-acttinnn model also applied to bimanual mobile manipulation. Same as the results in simulators, AC-DiT performs the best across all tasks, showing its capability of mobile bimanual manipulation in real-world. Qualitative Results. As shown in Figure we visualize the execution of our real-world tasks, all of which long-horizon and involve multiple steps. These tasks are designed to showcase the capability of the proposed AC-DiT in handling complex, multi-stage mobile manipulation. Especially in the bimanual task of hanging towel, the dual robot arms must first cooperate to hand over the towel, while the mobile base simultancously navigates the hanger. During this process. coordination between the mobile base and the manipulators is essential to ensure the towel is stably transferred to the hanger. As the agent approaches the hanger, must then accurately hang the towel in place. Additional qualitative results are shown in Appendix Eland the failure cases are analyzed in Appendix F
further visual- ize how the importance weights of differ- visual inputs change during the real- world clean table task to show the effectiv- ness of perception-aware multimodal adap- tation mechanism, as shown in Figure When the robot approaches the table and prepares to grasp the crumpled tissue, the
As shown in Figure 5
Exterior
Right Whiss
Knaap Toras
Moxe as The Right Table"
importance weights of the right-wrist cam- era and the point cloud increase. This is because these views provide rich seman- tie and spatial information about the target object, which helps the model accurately predict grasping actions. In contrast, the
Visual Importance Weightt
Unchenge
Decresse
Incresse
Figure 5: Effectiveness of Perception-aware Multimodal Adaption mechanism.

---

importance weight of the external view decreases, as the tissue is not clearly visible from that angle. The left-wrist view's importance remains unchanged, since the left arm consistently holds bowl and does not need to take action during this phase. While the robot is moving, the external camera's ss im- portance increases, as it provides a wide field of view that supports navigation and spatial awareness. Meanwhile, the right-wrist camera's s weight decreases because its highly localized perspective offers limited help for movement.
4.3 Ablation Study
We use MSHab as the benchmark for our ablation study. Different from the main experiments, which were con- ducted with three independent runs for each configuration and reported the mean success rate and standard deviation, only report the results from a single experimental run this section due to computational limita- tions. Table A presents the results of ablation study evaluating the im- pact of different components in the proposed AC-DiT framework. In Expl, using only 2D visual inputs results in baseline mean success rate of 37.5%. Introducing both 2D and 3D modalities in Exp2 yields a notable improvement (44.8%), demonstrating the benefit of multimodal fusion. Exp3 further incorporates the Mobility-to-BB Conditioning (MBC) mechanism, boosting the performance to 47.0% and suggesting that explicitly modeling the influence of the mobile base improves coordination. Finally, Exp4 integrates both MBC and the Perception-Awee Aware Multimodal Adaptation (PMA) mechanism, achieving the best performance (49.0%). This confirms that adaptively weighting multimodal features further enhances the system's 's ability to handle complex, long-horizon tasks by taking into account different perception requirements at different mobile manipulation stages. Overall, each component contributes positively, with the full model showing a total gain of 11.5 In Appendix [ we explore the impact of how the conditions are injected into the DiT, and examine the impact of how establish the conditioning relationship.
Table 4: Ablation study. 2D and 3D represent whether take images and points cloud as input respectively. MBC and MA denote the proposed Mobility-to-Bol Conditioning (MBC) and Perception-aware Multimodal Adaption (PMA) mechanisms, respectively.
2D
3D
PMA
MBC
Mean
Gain
Expl Exp2
37.5
V
44.8
+7.3
Exp3
V
47.0
+9.5
Exp4(ours)
/
49.0
+11.5
5 Conclusion and Limitations
We present AC-DiT (Adaptive Coordination Diffusion Transformer), an end-to-end framework that advances mobile manipulation by explicitly modeling the coordination between the mobile base and the manipulator, while dynamically adapting to stage-specific perception needs. To this end, we intro- duce two key mechanisms: mobility-to--out conditioning module, which leverages latent mobility representations to improve coordination and reduce error accumulation; and a perception aware multi- modal adaptation module, which assigns adaptive weights to 2D and 3D visual inputs based stage demands. Extensive experiments in both simulation and real-world settings demonstrate that AC-DIT consistently outperforms state-of-the-arr baselines, showcasing its effectiveness in robust mobile manipulation. As for limitations, since AC-DiT adopts imitation learning, the performance would fluctuate with data quantity and quality, as well inheriting suboptimal or biased demonstrations. Finally, we state the social impact of our work in Appendix H
10

---

6 Acknowledgements
This work was supported by the National Natural Science Foundation of China (62476011).
References
[1] Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, al. Do i can, say: Grounding language in robotic affordances. In 6th Annual Conference on Robot Learning, 2022.
[2] Ri-Zhao Qiu, Yuchen Song, Xuanbin Peng, Sai Aneesh Suryadevara, Ge Yang, Minghuan Liu, Mazeyu Ji, Chengzhe Jia, Ruihan Yang, Xueyan Zou, et al. Wildlma: Long horizon moc-mmaniuulttion in the wild. arXiv preprint arXiv:2411.15131, 2024.
[3] Weixin Mao, Weiheng Zhong, Zhou Jiang, Dong Fang, Zhongyue Zhang, Zihan Lan, Fan Jia, Tiancai Wang, Haoqiang Fan, and Osamu Yoshie. Robomatrix: A skill-centric hierarchical framework for scalable robot task planning and execution in open-world. arXiv preprint arXiv:2412.00171, 2024.
[4] Peiqi Liu, Yaswanth Orr,, Jay Vakil, Chris Paxton, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Ok-robot: What really matters integrating open-knowledge models for robotics. arXiv preprint arXiv:240I. 12202, 2024.
[5] Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin S Wang, Mukul Khanna, Theophile Gervet, Tsung- Yen Yang, Vidhi Jain, Alexander Clegg, John M Turner, et al. Homer- obot: Open vocabuloy mobile manipulation. 7th Annual Conference on Robot Learning, 2023.
[6] Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn, Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv: 13705, 2023.
[7] Zipeng Fu, Tony Zhao, and Chelsea Finn. Mobile aloha: Learning bimanual mobile manipu- lation using low-cost whole-body teleoperation. In 8th Annual Conference Robot Learning, 2024.
[8] Sixu Yan, Zeyu Zhang, Muzhi Han, Zaijin Wang, Qi Xie, Zhitian Li, Zhehan Li, Hangxin Liu, Xinggang Wang, and Song- Chun Zhu. M 2 diffuser: Diffusion-based trajectory optimization for mobile manipulation in 3d scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
Song-Chun
[9] Yunfan Jiang, Ruohan Zhang, Josiah Wong, Chen Wang, Yanjie Ze, Hang Yin, Cem Gokmen, Shuran Song, Jiajun Wu, and Li Fei-Fei. Behavior robot suite: Streamlining real-world whole- body manipulation for everyday household activities. arXiv preprint arXiv:2503. 05652, 2025.
[10] Josiah Wong, Albert Tung, Andrey Kurenkov, Ajay Mandlekar, Li Fei-Fei, Silvio Savarese, and Roberto Martín-Martin. Error-aware imitation learning from teleoperation data for mobile manipulation. In 5th Annual Conference on Robot Learning. 2022.
[11] Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang. Ke Xu, Hang Su, and Jun Zhu. Rdt-lb: lb: a diffusion foundation model for bimanual manipulation. arXiv preprint 211::2410007864, 2024.
[12] Arth Shukla, Stone Tao, and Hao Su. Maniskill-han hab: A benchmark for low-level manipulation in home rearrangement tasks. arXiv preprint arXiv:2412.13211, 2024.
[13] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273668 2023.
[14) Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy. arXiv e-prints, pages arXiv-2403, 2024.

---

[15] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi. Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. тO: A laaooon-lnnuugg-aaCaa action flow model for general robot control, 2024. URL. https://arxiv. (agg/abx/210.244100 2024.
[16] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint (2ti:2333.087744, 2023.
[17] Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, and Katsushi Ikeuchi. Gpt- 4v (ision) for robotics: Multimodal task planning from human demonstration, IEEE Robotics and Automation Letters, 2024.
[18] Priyam Parashar, Vidhi Jain, Xiaohan Zhang, Jay Vakil, Sam Powers, Yonatan Bisk, and Chris Paxton. Spatial-languag attention policies for efficient robot learning. arXiv preprint arXiv:2304. / 1235, 2023.
[19] Xuxin Cheng, Jialong Li, Shiqi Yang, Ge Yang, and Xiaolong Wang. Open-television: Teleop- eration with immersive active visual feedback. arXiv preprint arXiv:2407.015/2 2024.
[20] Chenhao Lu, Xuxin Cheng, Jialong Li, Shiqi Yang, Mazeyu Ji, Chengjing Yuan, Ge Yang. Sha Yi, and Xiaolong Wang. Mobile-television. Predictive motion priors for humanoid whole-body control. arXiv preprint arXiv:2412.07773, 2024.
[21] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. Pmlr, 2018
[22] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1.XXv:1707.06444 2017.
[23] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint (00iv::07000644 2021.
[24] Shirin Joshi, Sulabh Kumra, and Ferat Sahin. Robotic grasping using deep reinforcement learning. In 2020 IEEE 16th International Conference on Automation Science and Engineering (CASE), pages 1461-1466. IEEE, 2020.
[25] OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob Mc- Grew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
[26] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8763. PmLR, 2021.
[27] Hao-Shu Fang, Chenxi Wang, Hongjee Fang, Minghao Gou, Jirong Liu, Hengxu Yan, Wenhai Liu, Yichen Xie, and Cewu Lu. Anygrasp: Robust and efficient grasp perception spatial and temporal domains. IEFF Trunsactions on Robotics, (9(5)39929-39)) -345,, 2023.
[28] Haoyu Xiong, Russell Mendonca, Kenneth Shaw, and Deepak Pathak. Adaptive mobile manipulation for articulated objects in the open world. arXiv preprint arXiv:2401. 240.144331, 2024.
[29] Ruihan Yang, Yejin Kim, Rose Hendrix, Aniruddha Kembhavi, Xiaolong Wang, and Kiana Ehsani. Harmonic mobile manipulation. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3658-3665. IEEF, 2024.
[30] Shagun Uppal, Ananye Agarwal, Haoyu Xiong, Kenneth Shaw, and Deepak Pathak. Spin: Si- multaneous perception, interaction and navigation. In 2024 IEEF/CV Conference on Computer Vision and Pattern Recognition (CVPR), pages 18133-18144 142. IEEE, 2024.
12

---

[31] Jiaming Liu, Hao Chen, Pengju An, Zhuoyang Liu, Renrui Zhang, Chenyang Gu, Xiaoqi Li, Ziyu Guo, Sixiang Chen, Mengzhen Liu, al. Hybridvla: Collaborative diffusion and autoregression in unified vision-language-actionn model. arXiv preprint arXiv:2503. 10631, 2025.
[32] Zhuoyang Liu, Jiaming Liu, Jiadong Xu, Nuowei Han, Chenyang Gu, Hao Chen, Kaichen Zhou, Renrui Zhang, Kai Chin Hsieh, Kun Wu, et al. Mla: A multisensory language-action model for multimodal understanding and forecasting in robotic manipulation. arXiv preprint WrXi:22509.26642 2025.
[33] Xiaoqi Li, Jingyun Xu, Mingxu Zhang, Jiaming Liu, Yan Shen, laroslav Ponomarenko, Jiahui Xu, Liang Heng. Siyuan Huang, Shanghang Zhang, al. Object-centric prompt-driven vision- language-action model for robotic manipulation. In Proceedings of the Computer Vision and Puttern Recognition Conference, pages 27638-27648 2025.
[34] Xiaoqi Li, Liang Heng. Jiaming Liu, Yan Shen, Chenyang Gu. Zhuoyang Liu, Hao Chen, Nuowei Han, Renrui Zhang, Hao Tang, et al. 3ds-vla: A 3d spatial-aware vision language action model for robust multi-task manipulation. In 9th Annual Conference on Robot Learning, 2025.
[35] Moo Jsn Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source lision-language-actinn model. arXiv preprint arXiv:2406.092: 2024.
[36] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language- action models transfer web knowledge robotic control. arXiv preprint arXiv:2307.15818 2023.
[37] Xiaoqi Li, Mingxu Zhang, Yiran Geng, Haoran Geng, Yuxing Long, Yan Shen, Renrui Zhang, Jiaming Liu, and Hao Dong. Manipllm: Embodied multimodal large language model for object-centric robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18061-18070, 2024.
[38] Jiaming Liu, Mengzhen Liu, Zhenyu Wang, Lily Lee, Kaichen Zhou, Pengju An, Senqiao Yang, Renrui Zhang, Yandong Guo, and Shanghang Zhang. Robomamba: Multimodal state space model for efficient robot reasoning and manipulation. arXiv preprint arXiv:2406.04333 04339, 2024.
[39] Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, et al. Vision-language foundation models as effective robot imitators. arXiv preprint 1rXiv2231..03777 2023.
[40] Hongtao WU, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. arXiv preprint 23 2023.
[41] Chi-Lam Cheang. Guangzeng Chen, Ya Jing, Tao Kong, Hang Li. Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, et al. Gr-2: A generative aadoo-languge--attoon model with web-scale knowledge for robot manipulation. arXiv preprint arXiv:2410.06158. 2024.
[42] Siyuan Huang, Iaroslav Ponomarenko, Zhengkai Jiang. Xiaoqi Li, Xiaobin Hu, Peng Gao, Hongsheng Li, and Hao Dong. Manipvqa: Injecting robotic affordance and physically grounded information into multi-modal large language models. In 2024 IEEE/RSJ International Confer- ence on Intelligent Robots and Systems (IROS), pages 7580-7587. IEEE, 2024.
[43] Tsung-Wer Wei Ke, Nikolaos Gkanatsios, and Katerina Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations. arXiv preprint arXiv:2402. 2024.
[44] Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy, arXiv preprint arXiv:2405.122233 2024.
13

---

[45] Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng. Sicheng Xu, Yizhong Zhang, al. Cogact: A foundational laaion-lngguge-actto model for synergizing cognition and action in robotic manipulation. arXiv preprint 2rXiv:2411.196533 2024.
[46] Junjie Wen, Minjie Zhu, Yichen Zhu, Zhibin Tang, Jinming Li. Zhongyi Zhou, Chengmeng Li, Xiaoyu Liu, Yaxin Peng, Chaomin Shen, et al. Diffusion-vil vla: Scaling robot foundation models via unified diffusion and autoregression. arXiv preprint arXiv:2412.03293 2024.
[47] Junjie Wen, Yichen Zhu, Jinming Li, Minjie Zhu, Zhibin Tang. Kun Wu, Zhiyuan Xu, Ning Liu, Ran Cheng, Chaomin Shen, al. Tinyvla: Towards fast, data-efficient languaggggntonttii models for robotic manipulation. IEEE Robotics and Automation Letters, 2025.
[48] Hao Chen, Jiaming Liu, Chenyang Gu, Zhuoyang Liu, Renrui Zhang, Xiaoqi Li, Xiao He, Yandong Guo, Chi- Thi-Wing Fu, Shanghang Zhang, et Fast-in-slow: A dual-system foundation model unifying fast manipulation within slow reasoning, arXiv preprint arXiv:2506.01953 2025.
[49] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference computer vision, pages 11975-11986, 2023.
[50] Yueru Jia, Jiaming Liu, Sixiang Chen, Chenyang Gu, Zhilue Wang, Longzan Luo, Lily Lee, Pengwei Wang. Zhongyuan Wang, Renrui Zhang, et al. Lift3d foundation policy: Lift- ing 2d large-scale pretrained models for robust 3d robotic manipulation. arXiv preprint arXiv:2411.1822 2024.
[51] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
[52] Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al. Sapien: A simulated part-based interactive environ- ment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11097-11107, 2020.
[53] Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, et al. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in neural information processing systems, 34:251-266 2021.
[54] Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. The yeb object and model set: Towards common benchmarks for manipulation research. In 2015 international conference on advanced robotics (ICAR), pages 510-517. IEEE, 2015.
[55] Jingyun Yang, Zi-ang Cao, Congyue Deng, Rika Antonova, Shuran Song, and Jeannette Bohg. Equibot: Sim eau-evivrrr diffusion policy for generalizable and data efficient learning. arXiv preprint arXiv:2407.00 2449..1477, 2024.
[56] Yao Mu, Thanxing Chen, Shijia Peng, Zanxin Chen, Zeyu Gao, Yude Zou, Lunkai Lin, Zhiqiang Xie, and Ping Luo. Robotwin: Dual-arm rohot benchmark with generative digital twins (early version). arXiv preprint arXiv:2409.02920 2024.
[57] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the JEEE/CVF international conference on computer vision, pages 4195 4205, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and/or introduction clearly state the claims made, including the contributions made in the paper and important assumptions and limitations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper has discussed limitations in the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The results reproducible.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer:
[No]
Justification:
We will release after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//guiii wwwticccguie//eeieeeeeeii CoodeSubeiiieeee ssi wswinnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experimental setting is clearly presented in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper reports error bars
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper indicates the type of compute workers.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ptctps://neuriie cc. httpa//pp......poooo.ccccccccaaa
Answer:
[Yes]
Justification:
We follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is social impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
18

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited the original paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, snperswwihcoode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
19

---

Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
20

---

Justification:
The core method development in this research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to LLM policy htttap///eaarriio.cccc/oooorreep for what should or should not be described.
21

---

Appendix A Robot Hardware Details. In this section, we provide a detailed description of the robotic platform used in our experiments, including the Agilex Piper robotic arms, the Tracer mobile base, and the camera configurations. Specific joint ranges, control modes, and sensor parameters are outlined.
Appendix [ World Experiment Details. In this section, we provide comprehensive details on the real-world experiments. covers the robot Configuration and Assets (Appendix B.1:; detailed Task Descriptions (Appendix [.2 for the four complex manipulation tasks; specifics on Computa- tional Resources and Time Cost (Appendix B.3). such as GPU configuration, training stages, iteration counts, model parameters, and memory usage; and the achieved Control Frequency (Appendix B.4).
Appendix C Simulation Experiment Details. In this section, we provide qualitative and quanti- tative results of AC-DiT in simulated tasks. We highlight key challenges in simulation and further demonstrate the effectiveness of our proposed Mobility-t--Bodd Conditioning mechanism and Perception-Aware Multimodal Adaptation mechanism.
Appendix D Additional Ablation Study. In this section, present in-depth ablation study investigating the impact of how the conditions are injected into the DiT and the impact of how to establish the conditioning relationship.
Appendix Additional Qualitative Results. To further validate the effectiveness of the Perception- aware Multimodal Adaptation mechanism, we supplement the visualization experiment from Figure in the main paper with additional results.
Appendix F Failure Case Analysis. In this section, we discuss common failure modes observed in real-world experiments, categorizing them into bimanual coordination errors, locomotion control failures, and issues caused by poor manipulation triggering.
Appendix G Model Parameter Breakdown. In this section, we provide comprehensive breakdown of the model parameters, explaining the rationale behind selecting the Lightweight Mobility Head for base control and the Large Mobile Manipulation Head for whole-body control based task complexity and action space dimension.
Appendix H Broader Impact. In this section, we reflect on the potential broader impact of our work, highlighting its contribution to the development of end-to-end mobile manipulation systems and its societal implications.
A Robot Hardware Details
Table 6: Configurations for cameras
Table 5: Agilex Piper Arm Position Ranges
Parameter
Value
Joint Name
Range
Orbbec Dabai Cameras
J1
- -154 ~ 154
J2
0 ~ 195
FOV (HxW) 80 x 66
J3
-175 ~ 0
Frequency
120 fps
J4
-006 ~ 106
Realsense 1,555 Camera
15
-75e ~ 75
J6
-100 ~ 100P
FOV (Hx W) 70 x 43
Frequency
30 fps
Arms. As illustrated in Figure 6a the Cobot Magic platform incorporates four 6-DoF Agilex Piper robotic arms. The two puppet arms for inference and execution each feature a parallel gripper with a 1 110 mm stroke. The robotic arms operate in joint position control mode, and the motion range of each joint is detailed in T
Mobile base The mobile base, named Tracer, is wheel-based and capable of in-situ steering and forward/backward locomotion. idler wheels. The Tracer is controlled via a 2-DOF velocity controller, governing two independent parameters: forward velocity and rotational velocity. The base achieves maximum speed of 1.8 meters per second.
equipped with two active drive wheels and four passive
22

---

Real L515
Piper Are (24)
AC-DiT
Orbbec Dabai
Mobile
(a) Robot hardware configuration.
(b) Real-Wail World assets.
Figure 6: Robot hardware configuration and real-world assets.
Cameras. As shown in Figure 6a the Cobot Magic comes pre-installed with three Orbbec Dabai cameras. To capture more precise point cloud data, we add an additional Intel RealSense L515 depth camera at the head. Notably, the two central cameras have specific design requirements for their positions and orientations. The central Dabai camera is mounted at chest level with a frontal view to enable the robot observe sufficiently wide field of view during mobility. Meanwhile, the 1.515 camera positioned at the head with an overhead perspective to fully observe the workspace during the manipulation phase. The specific parameters of the two cameras shown in Table
B Real-Words Experiment Details
Task Progress
Cueumber in Busker
Store Breads
Hong Towa!
Cleun Table
Figure 7: Robot execution progress of AC-DiT in real-world tasks.
B.1 Configuration and Assets
The robot hardware configuration and real-world assets are shown in Figure 6a and Figure 6b respectively. Except that AC-DiT uses the RGB information from three Dabai cameras the depth information from an 1,55 camera, all other baseline algorithms use the RGB information from all four cameras. Besides, as shown Figure we provide more detailed visualization of the progress of real-world tasks.
23

---

Lift-wrist
Right-rris
Point
Cucumber in Desket
Store Breads
Hong Towel
Clean Table
Figure 8: Observation examples in real-world tasks.
B.2 Task Descriptions
Detailed descriptions of all real-world robot tasks are listed as follows:
Cucumber basket. The robotic setup includes primary worktable in front of the robot holding cucumber and secondary table on the robot's s left side with a basket for object placement, where robot starts at an initial distance from the primary worktable with its arm in the rest pose. The robot must first navigate to the cucumber then retreat slightly, perform left turn maneuver to reorient toward the secondary table, navigate to the basket, and precisely deposit the cucumber into it before retracting its arm the predefined rest position to complete the task.
the primary table, execute grasp action to acquire it,
2. Store bread. The robotic workspace comprises a primary worktable positioned in front of the robot, equipped with two bread items and basket, and secondary left-side table initially empty. The robot starts at an initial position relative to the primary worktable with its arms in the rest configuration. The task requires the robot to first navigate to the primary worktable and execute bilateral arm manipulation to transfer the bread items into the basket, followed by lifting the basket with coordinated dual-arm motion. Subsequently, the robot performs backward translation, left rotational reorientation, and navigates to the secondary table, where it deposits the basket with precise placement to complete the task.
3. Hang towel. The robotic task environment consists of a workdbench positioned in front of the robot, bearing a towel as the target object, and hanging rack located on the robot's left lateral side. The robot initiates the task at an initial pose in front of the workbench with its arms in the rest configuration. The operational sequence begins with the robot using its left arm execute precision grasp of the towel, followed by a bimanual transfer to the right arm. The robot then performs minor backward translation, left rotational reorientation, and navigates executes towel-hanging action with precise placement on the rack. The task is concluded by the robot returning both arms to their predefined rest positions.
hanging rack, where
4. Clean table. The robotic workspace features a workbench equipped with four waste objects: Sprite bottle, Cestbon bottle, orange peel, and crumpled tissue. large bowl and two randomly positioned waste objects are initially placed on the workbench's 's left side, with the remaining two objects on the right side. task initiation, the robot is positioned on the left side of the workbench with its arms in the default configuration. The task sequence unfolds as follows: first, the robot uses its left arm grasp to lift the large bowl, then utilizes right arm to transfer the two left-side waste objects into the bowl. Subsequently, the robot performs short backward translation, reorients toward the front-right direction, and navigates to the workbench's right side, where it continues using its right arm to deposit the remaining two objects into the bowl, thereby completing the task.
24

---

B.3 Computational Resource and Time Cost
Our proposed approach was trained on a server equipped with eight NVIDIA A (80 GB) GPUs. The training process consisted of two stages: an imitial Mobility pretraining phase (Stage 1) comprising 20,000 iterations, followed by training of the full AC-DiT model (Stage 2), which comprised 30,000 iterations. The lightweight mobility head of our model contains 170 million parameters, while the full AC-DiT model has a total of 1.2 billion parameters. During training with a batch size of 16, the system utilized approximately 60 GB of GPU memory. Inference requires less memory, consuming about 20 GB of GPU memory.
B.4 Control Frequency
The model achieves an inference speed of 5Hz on a single NVIDIA RTX 4090 GPU. This speed, combined with action chunking, allows for robot control frequency of 30Hz. The total processing time for a single control step is 0.2s, which is composed of the model inference time (0. (0.16s). perception information acquisition (0.03s), and point cloud computation (0.01s).
C Simulation Experiment Details
C.1
ManiSkill-Har Simulationi Environment
Task Progress
Move to Fridge Pick Apple
Move to Fridge Place Apple
Move to Fridgs Open Fridge
Move to Counter Pick Rowl
Move to Table Place Bool
Move Couneer Open Drawer
Move Counter
Close Drower
Figure 9: Robot execution progress of AC-DiT in seven MSHab simulation tasks.
Figure illustrates the process of AC-DiT executing simulation tasks in MSHab environments and Figure Toa shows the observation examples of each task in MSHab. Notably, in the MSHab simulator, since trajectories are collected by the RL agent, the mobile base and manipulator frequently exhibit
25

---

Eaterior
Wriss
Point Cloud
Left-Wrist
Exterior
Right-WWist Winsst
Point Cloud
Movero Frage
Plak Apyle
Comtainer Hundorer
Move Thhie
Place Apple
to Frage Moretti Cotanee Open krale Moxs to Couter pure Pioli Polld ConnteCypen Droney Cove Draves
Empty
Bottle Easy
Move to
Foetee Hredd
Apple Mewcy
Move la
(a) Observation examples in
(b) Observation examples in RoboTwin simulation tasks.
MSIlab simulation tasks.
Figure 10: Observation examples in MSHab and RoboTwin simulations.
concurrent motion. This scenario rigorously evaluates the coordination capability between the mobile base and manipulator while further underscoring the efficacy of the mobility-to-body condition mechanism. Specifically, perform concurrent motions to approach the target object, yet the gripper still accurately executes the manipulation at the optimal timing. This demonstrates that AC-DiT's modeling of coordination plays critical role in achieving such seamless integration of locomotion and manipulation.
the task depicted in Figure g the mobile base and the manipulator
C.2 RoboTwin Simulation Environment
Figure [I illustraaes the process of AC-DiT executing simulation tasks in RoboTwin environments and Figure T0b present the observation examples of each task in RoboTwin. Notably, the Handover task necessilates coordinated collaboration between two arms rather than independent actions, under- scoring the critical requirement for bimanual coordination. This further highlights that AC-DiT's dual-arm-to- to-dual-ann aarllarm conditioning mechanism in RobotWin-derre conditioning mechanism plays pivotal role by providing each arm with short-term motion priors, thereby enhancing bimanual coordination.
from the mobility to--bdy
D Additional Ablation Study
Table 7: Ablation Study. We explore the im- pact of how the conditions are injected into the DIT.
Table 8: Ablation Study. We explore the im- pact of how to establish the conditioning rela- tionship.
Condition Injection
Mean
Condition Mechanism
Mean
Cross-Attenion Alternatively
43.7
Up-to-Body 47.3
Cross-Attention
44.8
Mobility-o--Body 49.5
26

---

Tusk Progress
Hanover
Continnr
Empty Cup
Bottles Easy
Bottles Hard
Heatii
Apple Messy
Figure 11: Robot execution progress of AC-DiT seven RoboTwin simulation tasks.
The impact of how the conditions are injected into the DiT. We empirically evaluated two different methodologies for condition injection into DiT. First, following the RDT framework (Α], we use cross-attention for alternating condition injection. Second, we implement cross-attention-baat condition injection with concatenation of all conditional features in each injection step (57). As shown in Ta cross-attention variants (with or without alternatively injection) exhibit comparable performance, with the alteenatively--jjeeaaa to its higher information throughput relative to the alternatively-ineccii one.
scheme achieving marginally superior results due
The impact of how to establish the conditioning relationship. In AC-DiT, first extract latent mobility feature to serve as a condition for mobile manipulation action prediction, as shown in the Hooolitt-tt-Booyy setting in Table 8 In this section, we compare this approach with using latent manipulator feature feature--seccffccall the "Up-to-Body" setting in Table [ We observe that using the latent mobility feature outperforms using the latent manipulator feature. This is because the mobile base actions have a significant impact manipulator control; errors in the mobility prediction can accumulate and propagate the manipulator stage. Therefore, we adopt the strategy of first extracting the latent mobility feature as the condition alleviate potential error accumulation and enhance coordination.
the arm joint position- as the condition, corresponding to
E
Additional Qualitative Results
Adaptive importance weights analysis. To further validate the effectiveness of the Perception-aware Multimodal Adaptation mechanism, we supplement the visualization experiment from Figure 5 in the main paper with additional results, as shown in Figure [2] We visualize the observations at three different time steps. As shown, the importance of the left-wrist camera remains unchanged, since the left arm consistently holds a bowl and does not need to take action during this phase. In contrast, at time steps and 3, the importance weights of the right- -ight-writt camera and the point cloud increase obviously, as they provide critical information for completing the manipulation task. Meanwhile, the exterior camera maintains relatively high importance, as it still captures both the manipulated object and the robot arms from broader perspective. In contrast, during the movement phase at step the importance weight of the point cloud drops significantly, due to the limited sensing range of the depth camera.
27

---

Lift-wrist
Exterior
Right-wrist
Point cloud
t1
t2
t3
X.7
1.0
Figure 12: Visualization of Perception-aware Multimodal Adaption mechanism whole episode.
F
Failure Case Analysis
Despite extensive physical robot experiments, we identify three primary error categories affecting AC-DiT's performance. First category: Bimanual coordination errors. Bilateral arm collaboration required for tasks such as transferring a towel from the left hand the right hand, where the right hand occasionally fails to grasp the correct pose, or the right arm inaccurately deposits waste into a bowl, causing objects to fall outside due to minor positional deviations. Second category: Locomotion errors. Since robot movement is controlled by velocity controller, whose actual motion results from the integration of velocity commands over time, cumulative errors may occur. The model sometimes outputs excessively high velocities, leading the robot into unrecoverable error states, or insufficient velocities that prevent chassis movement. Mitigating this issue relies on dataset quality-ideally, data should be collected by single operator with minimal velocity fluctuations maintained at a suitable constant value during recording. Third category: Suboptimal manipulation triggering due to poor locomotion termination. When the robot stops state outside the dataset's distribution, the subsequent manipulation primitives often fail, as the end pose does not align with the precondition for successful object interaction.
G Model Parameter Summary
We display the comprehensive parameter breakdown in Table specifically detailing all components shown Figure of the main paper. The relative small DIT architecture (Lightweight Mobility Head, 170M DiT) is selected for mobile base control as it has relatively fewer output degrees of freedom
28

---

Table 9: Model Parameter Summary.
Component
Parameters
Architecture Details
SigLIP--assed multi-view vi- sual feature extraction
Image Encoder
428M (frozenn
SigLIP-based language in-
Text Encoder
450M (frozenn
struction encoding
Point cloud processing with
Encoder w/. LoRA
430M (0.09% LoRA)
LoRA adaptation
Perception-aware modal Adaptation
Multi-
1M
Linear Projectors
Main DīT-based head for whole-body action predic- tion
Mobile Manipulation Action 1.2B
Head (DiT)
Lightweight Mobility Action 170M
DiT blocks for base motion prediction
Head (DiT)
Total Trainable
~ 1.37B
Total trainable parameters
Including frozen founda- tion models
Total Model
2.7B
(linear/angular velocities), providing sufficient modeling capacity while reducing computational overhead. The relative large architecture (Large Mobile Manipulation Head, DiT) is selected for the more complex mobile manipulation task, where higher parameter count provides better robustness and generalization capability needed for coordinated whole-body control with higher-dimensional action spaces.
H
Broader Impact
Our work develops an end-to-end mobile manipulation framework based on DiT [17 by explicitly modeling the collaborative relationship between the mobile base and the manipulator, and by adap- tively adjusting weights according to perceptual needs at various stages of mobile manipulation. This research aims to innovate by extending end-to-end models to mobile manipulation tasks, and it does not pose direct societal impacts. We hope our efforts can foster research in the field of end-to-end control for mobile manipulation while promoting the healthy, controlled, and sustainable development of the entire field.
29


Reviews:
[{"review_id": "fWMdnp2ZQx", "reviewer": "Reviewer_KuR9", "summary": "This paper presents a novel strategy to control both the mobile base and the dual-arm of a robot. This is mainly achieved by adding a lightweight mobility action head. Another contribution is a \"conditioning strategy\" of multimodal data (images and point cloud). Overall, this paper reports a practice of multimodal fusion on mobile manipulation tasks. A good paper.", "questions": "Please see my comments above.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "1. I'm not sure whether the word \"coordination\" is a terminology introduced by this paper. (There definitely exist works that handled mobile manipulation). If so, the authors should claim it explicitly and define it formally: what kind of end-2-end mobile manipulation is with coordination, and what is NOT.\n\n2. The author should summarize the parameters of this model. \n\n3. Will the mobile base wiggle during the manipulation? or stay stably? In my own experience, if the output of one dimension (such as the velocity of the robot base) is always zero, then the algorithm does not know when to generate non-zero value. \n\n4.There are ?? in the appendix.", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer KuR9\nWe appreciate the reviewer's constructive feedback and positive assessment of our contributions. We provide detailed responses to the raised questions in the following.  \n\n## **Response to Weakness 1: Definition and Formalization of \"Coordination\"**\n**Prior Usage of \"Coordination\" Term:** The term \"coordination\" has not been formally defined in prior mobile manipulation literature. While previous works achieve joint control of mobile base and manipulator, they lack explicit formalization of what constitutes coordinated behavior.\n\n**Formal Definition of Coordination:** We define **coordination** in mobile manipulation as an end-to-end control paradigm where a multi-component robot (in our case, mobile base and arm system) simultaneously generates synchronized action decisions for all components at each time step, rather than employing sequential or phase-based control strategies.\n\nMathematically, let $\\mathcal{A} = \\mathcal{A}\\_b \\times \\mathcal{A}\\_a$ represent the joint action space, where $\\mathcal{A}\\_b$ and $\\mathcal{A}\\_a$ denote the action spaces for the mobile base and manipulator arms respectively. At each time step $t$, our coordinated policy $\\pi$ generates:\n\n$\\pi(s\\_t) = (a\\_b^t, a\\_a^t) \\in \\mathcal{A}\\_b \\times \\mathcal{A}\\_a$\n\nwhere $s\\_t$ represents the multimodal observation state, and both $a\\_b^t$ and $a\\_a^t$ are computed simultaneously through shared feature representations.\n\n**Which Methods Have/Lack Coordination Capability:** On the one hand, methods like Ok-Robot [4] and HomeRobot [5] do not qualify as coordination because they employ sequential control strategies where navigation and manipulation occur in separate phases. On the other hand, Mobile-ALOHA [7] and π0 [15] possess coordination capability because they output joint actions for both mobile base and manipulator simultaneously. However, unlike existing coordinated methods that simply concatenate actions into a unified space, our approach explicitly models inter-component dependencies through mobility-to-body conditioning, where latent mobility features guide whole-body action generation.\n\n## **Response to Weakness 2: Model Parameter Summary**\nThank you for this important suggestion. We will add the comprehensive parameter breakdown to the revised paper, specifically detailing all components shown in Figure 2 of our main paper:\n\n| Component | Parameters | Architecture Details |\n| --- | --- | --- |\n| Image Encoder | 428M (frozen) | SigLIP-based multi-view visual feature extraction |\n| Text Encoder | 450M (frozen) | SigLIP-based language instruction encoding |\n| 3D Encoder w/. LoRA | 430M (0.09% LoRA) | Point cloud processing with LoRA adaptation |\n| Perception-aware Multimodal Adaptation | 1M | Linear Projectors |\n| Mobile Manipulation Action Head (DiT) | 1.2B | Main DiT-based head for whole-body action prediction |\n| Lightweight Mobility Action Head (DiT) | 170M | DiT blocks for base motion prediction |\n| **Total Trainable** | **~1.37B** | **Total trainable parameters** |\n| **Total Model** | **~2.7B** | **Including frozen foundation models** |\n\n\n**Design Rationale:**\n\n+ **Lightweight Mobility Head (170M DiT)**: The relative small DiT architecture is selected for mobile base control as it has relatively fewer output degrees of freedom (linear/angular velocities), providing sufficient modeling capacity while reducing computational overhead.\n+ **Large Mobile Manipulation Head (1.2B DiT)**: The relative large architecture is selected for the more complex mobile manipulation task, where higher parameter count provides better robustness and generalization capability needed for coordinated whole-body control with higher-dimensional action spaces.\n\n## **Response to Weakness 3: Mobile Base Behavior During Manipulation**\nThis is an excellent question that touches on a core aspect of our coordination strategy.\n\n**Q1: Will the mobile base move during manipulation?**  In some tasks, the mobile base wiggles during manipulation, while in other tasks the base remains stable. Specifically, in simulation environments (MSHab), the base moves continuously throughout the manipulation process according to simulator's default setting. In real-world experiments, the base sometimes remains stationary during certain manipulation cases, while in other cases, it wiggles during manipulation.\n\n**Q2: If the base maintains zero velocity for extended periods, how does the algorithm determine when to generate non-zero actions?** The model leverages multimodal perception and internal coordination modeling to autonomously judge whether the current atomic task has been completed and whether it needs to transition to the next atomic task phase. Through this multimodal sensing and implicit reasoning, the model can automatically transit the base from a stationary state to a moving state when required for task progression. For example, in our real-world \"Store Breads\" task, the base outputs velocity values close to zero while the arms place bread items into the basket, and then automatically transits to movement when lifting the basket and navigating to the secondary table.\n\n## **Response to Weakness 4: Manuscript Corrections**\nWe acknowledge the presence of formatting issues (missing references marked as \"??\") in the current submission. In the revised version, we will:\n\n+ Correct all missing references and citation formatting\n+ Fix typographical errors throughout the manuscript\n+ Add the expanded definitions and explanations mentioned above\n+ Include the parameter summary table in the appendix\n+ Enhance clarity in the coordination mechanism descriptions\n\nWe believe these revisions will significantly improve the manuscript's clarity and completeness.\n\n---\n\nWe sincerely thank the reviewer for the thoughtful and positive comments of our work. We hope our detailed responses are helpful in answering the questions raised, and we look forward to further discussion in this valuable academic exchange.\n\n**Reference**\n\n[4] Ok-robot: What really matters in integrating open-knowledge models for robotics.\n\n[5] HomeRobot: Open-vocabulary mobile manipulation\n\n[7] Mobile-aloha: Learning bimanual mobile manipulation using low-cost whole-body teleoperation.\n\n[15] π0: A vision-language-action flow model for general robot control"}, {"review_id": "S0uHkfAq08", "reviewer": "Reviewer_GULZ", "summary": "The paper introduces AC-DiT, an end-to-end framework designed to improve mobile manipulation by addressing two key challenges: coordinating the mobile base and manipulator, and adapting to varying perceptual needs at different task stages.", "questions": "1. Could you clarify the precise methodology for applying the importance weights to the corresponding features? Specifically, is this implemented via element-wise multiplication of the feature vectors, or through dynamic adjustment of attention weights within the cross-attention conditioning mechanism?\n2. What is the evaluation suite in Table.4? The reported values (e.g., 49%) are not aligned with any results in above tables.\n3. The supplementary results (Figure 7) indicate the computed importance weights exhibit a relatively constrained numerical range (e.g., point cloud weights varying between 0.73-0.76). Given this limited variation, what is the underlying mechanism that enables these weights to significantly influence the model's performance?\n4. Regarding the architectural implementation: does the manipulator's DiT policy operate in a strictly sequential manner, requiring completion of the mobile base's DiT head computation (to generate the latent mobility features) before initiating its own inference? Furthermore, what is the achievable control frequency of the complete system in your real-world deployment?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The mobility-to-body conditioning and perception-aware multimodal adaptation mechanisms are novel contributions, addressing key gaps in mobile manipulation.\n2. The proposed Perception-Aware Multimodal Adaptation (PMA) mechanism is quite simple and intuitive. Empirical results shows it also works well in practice.\n3. Real-world demos are given in the supplementary material.\n\n\n **Weaknesses**\n\n1. The weight analysis presented in Figure 7 may introduce potential misinterpretation due to the use of separate y-axes for 2D and 3D inputs, despite their equal treatment in the model architecture. For instance, during time t₁, while the point cloud appears dominant in the visualization, its actual computed weight is lower than the camera's weight, creating a discrepancy between the graphical representation and the underlying data.\n2. Real-world experiments are limited to a few tasks, and generalization to more diverse environments (e.g., cluttered or dynamic settings) is not fully explored.\n\n\n[minor] The reported \"Gain\" metric in the ablation study requires clarification, as the baseline performance (37.5%) appears inconsistent with the subsequent improvements. \n\n[minor] Several references to tables and figures in the appendix (Lines 31, 37, and 43) are either mislabeled or incorrectly cross-referenced", "was_revised": true, "final_justification": "All my concerns are addressed in the rebuttal. I've raised my score accordingly.", "rebuttal": "# Response to Reviewer GULZ\n\nWe sincerely thank the reviewer for the thoughtful feedback and constructive suggestions. We provide detailed responses to the raised questions in the following.\n\n## **Response to Weakness 1: Weight Analysis Visualization Concerns**\n\nWe want to clarify that Figure 7 of Appendix was not intended to mislead readers, but rather to provide an intuitive visualization of how each modality's importance changes throughout the long-horizon task. The rationale and explanation are as follows:\n\n**Our visualization objective:** Figure 7 is specifically designed with good intentions to illustrate the **temporal trends of importance weights for individual sensor modalities** during the mobile manipulation process, rather than to enable direct magnitude comparisons between different sensor types (2D vs. 3D). As explicitly stated in lines 135-137 of Appendix, we focus on describing directional trends (increase/decrease) for individual modalities without making cross-modal magnitude claims.\n\n**Technical explanation for weight differences:** The observed numerical differences between 2D and 3D weights reflect fundamental architectural characteristics rather than relative importance. Specifically, 3D features contain 256 tokens while 2D features contain 4,374 tokens (from 6 input frames including 3 views from 2 timestamps), naturally leading to different weight scales. Additionally, 3D tokens inherently encode less dense information compared to 2D visual tokens, and we employ a single 3D camera positioned at the robot's head (distant from objects) versus multiple 2D cameras including close-proximity wrist cameras. Therefore, it is expected and reasonable that 3D weights exhibit lower magnitudes compared to 2D weights.\n\n**Improvements for clarity:** Based on this feedback, we will revise Figure 7 by: (1) separating it into distinct subfigures for 2D and 3D modalities to eliminate cross-modal visual comparison, and (2) applying appropriate normalization to ensure clear and unambiguous representation of temporal trends within each modality.\n\n## **Response to Weakness 2: Limited Real-World Experiments**\n\nBased on your suggestions, we conduct additional real-world experiments to validate our method's generalization capability across diverse environments, including cluttered scenes and dynamic scenarios:\n\n**Generalization to cluttered environment:** We have conduct additional experiments for cluttered scene adaptation. Using the **Cucumber_in_Basket** task, we modify the desktop environment to resemble the setup shown in the first image of the third row in Appendix Figure 2, but with the towel removed. Specifically, we add potted plants, fruit bowls, tissue boxes, and upright books to the table while keeping the natural language instruction unchanged. Testing across 17 trials, we find that our method demonstrates strong adaptability to cluttered scenes. The detailed success rates for each sub-task and overall performance are as follows:\n\n||Original|Cluttered|\n|-|-|-|\n|Success Rate|50.0|47.1|\n\n**Generalization to dynamic scene:** We conduct experiments to evaluate generalization to dynamic environments. Using the same **Cucumber_in_Basket** task, we dynamically change the cucumber's position while the robotic arm was in the process of grasping it. Specifically, when the gripper was about to make contact with the object, we randomly moved the cucumber approximately 10cm in a random direction. Testing across 17 trials without additional training, we find that our model can re-attempt grasping after the cucumber's position is moved, demonstrating the model's generalization capability to dynamic scenarios. The detailed success rates for each sub-task and overall performance are as follows:\n\n||Original|Dynamic|\n|-|-|-|\n|Success Rate|50.0|35.3|\n\n## **Response to Minor Issues**\n\nWe will correct all typos, missing citations, and errors in the ablation experiment tables in the revised version.\n\n## **Response to Question 1: Importance Weight Methodology**\n\n**Brief Answer:** The importance weights are applied through **scalar multiplication** between the computed weight scalars and their corresponding feature vectors, not through the cross-attention mechanism.\n\n**Detailed Process:** The application of importance weights follows a straightforward scalar multiplication approach. After computing the importance weights $w=(w_{i}^{f},w_{i}^{l},w_{i}^{r},w_{i}^{p})$ for each visual modality through cosine similarity with language features, where each weight is a scalar value, we directly apply these scalar weights to the corresponding visual feature vectors:\n\n**Mathematical formulation:**\n\n$F\\_{weighted}^{{f}}=w\\_{i}^{f}*F\\_{2D}^{{f}}$\n\n$F\\_{weighted}^{l}=w\\_{i}^{l}*F\\_{2D}^{l}$\n\n$F\\_{weighted}^{r}=w\\_{i}^{r}*F\\_{2D}^{r}$\n\n${F\\_{weighted}^{p}=w\\_{i}^{p}*F\\_{3D}^{p}}$\n\nwhere each ${w\\_i^\\*}$ is a scalar weight and ${F\\_{2D/3D}^\\*}$ are feature vectors, resulting in scalar-vector multiplication.\n\nThe reweighted visual features are then concatenated to form the perception-aware visual representation:\n\n$F_{v}=\\mathrm{Concat}(F_{weighted}^{f},F_{weighted}^{l},F_{weighted}^{v},F_{weighted}^{p})$\n\nThis weighted visual feature $F_v$, together with language features $F_l$ and latent mobility features $F_m$, serves as the conditioning input for the mobile manipulation action head.\n\n## **Response to Question 2: Evaluation Suite in Table 4**\n\nThank you for this important clarification question. Table 4 presents ablation study results conducted on the **ManiSkill-HAB simulation experiments**, which is the same experimental setup as reported in Table 1 of the main paper.\n\nTo ensure statistical rigor, we conducted three independent experimental runs for each configuration and reported mean ± standard deviation in our main results tables. However, for the ablation study in Table 4, we report results from a single experimental run.\n\nWe apologize for any confusion this may have caused and will clarify this methodology in the revised version.\n\n## **Response to Question 3: Impact of Limited Weight Variation**\n\nWe appreciate this insightful question about the mechanism behind PMA's effectiveness despite seemingly small weight variations shown in supplementary Figure 7.\n\n**Reasons for constrained numerical range:**\n1. **Normalization compression:** During importance weight computation, we apply normalization operations that compress the raw similarity scores into a more constrained range\n2. **Task-specific characteristics:** The \"Clean table\" task predominantly involves tabletop manipulation scenarios where the robot faces relatively consistent perceptual scenes (cluttered desktop environments), naturally leading to smaller weight variations\n\n**Why small changes yield significant impact:** The weight variations may appear numerically small, but they represent meaningful relative differences in a normalized space, which is sufficient to alter the balance between modalities in the model's decision-making process. More importantly, these weights directly scale the input features before they enter the transformer architecture, where such scaling differences can propagate through multiple attention layers and ultimately influence action predictions. The effectiveness is demonstrated empirically through our ablation studies, which show that removing the PMA mechanism leads to performance degradation.\n\n**Validation with diverse scenarios:** To validate PMA's broader adaptability, we provide supplementary analysis showing importance weights computed for a ManiSkill-HAB simulation task involving navigation and manipulation. The results demonstrate substantially larger weight variations (variation spans: 0.69 for hand camera, 0.12 for head camera and 0.18 for pointcloud), confirming that PMA adapts more dramatically in some other scenarios. This supplementary evidence further supports our main findings and demonstrates that the constrained range in our real-world task is task-specific rather than a fundamental limitation of our approach.\n\n## **Response to Question 4: Architectural Implementation Details**\n\nThank you for these detailed technical questions about our system implementation.\n\n**Question 4.1 - Sequential Processing**\nYes, our system operates in a **strictly sequential manner**. The computational flow follows this sequence:\n1. All modality encoders (image, pointcloud, text) process their respective inputs\n2. The lightweight mobility action head (170M parameters) processes the encoded features to extract latent mobility features\n3. These latent mobility features, together with the reweighted multimodal features from PMA, are fed into the mobile manipulation action head for final action prediction\n\nThis sequential design ensures proper mobility-to-body conditioning while maintaining computational efficiency since the lightweight nature of the mobility head minimizes additional computational overhead.\n\n**Computational overhead details:**\n+ Baseline model (without mobility conditioning): 129ms inference time\n+ AC-DiT with mobility-to-body conditioning: 163ms inference time\n+ **Additional overhead: 34ms (+26%)**\n\n**Question 4.2 - Control Frequency**\nOur model operates at an inference speed of 5Hz, which enables a robot control frequency of 30Hz through **action chunking**. The total processing time of 0.2s comprises model inference time (0.16s) along with perception information acquisition (0.03s) and point cloud computation (0.01s).\n\n---\n\nWe sincerely thank the reviewer for the thoughtful and positive evaluation of our work. We hope our detailed responses have been helpful in answering the questions raised, and we look forward to further discussion in this valuable academic exchange."}, {"review_id": "4gqA8lSieK", "reviewer": "Reviewer_NDbd", "summary": "This paper proposes a diffusion-transformer (DiT)-based multi-modal imitation learning method for mobile manipulation. To model the influence of the mobile base on manipulation control, it proposes a mobility-to-body conditioning mechanism to predict manipulation action based on the extracted base motion representations. To better utilize multi-model observations, a perception-aware multimodal conditioning strateg that dynamicall adjusts the fusion weights between 2D images of three different views and 3D point clouds. The proposed method is evaluated and compared with other baseline methods on both simulated and real-world mobile manipulation tasks, and superior performance is achieved.", "questions": "1) Please add implementation details for the training procedure, computing resource and time cost.  \n2) Please state clearly in the paper how the training and testing data are composed, and how they differ from each in scenes, tasks (action+object) et al.  \n3) Do you use the same set of language input for training and testing? I am curious how the model is robust and generalizable to different language instructions.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**:  \n1) The paper writing is overall good, with clear description of the motivation and the proposed idea.  \n2) The proposed techniques seems interesting and reasonable for the mobility-to-body conditioning by modeling dependency relationship between mobile base and manipulator, and for the perception-aware multimodal adaptation by adjusting dynamic weights of different modalities during instruction execution.  \n3) The experiments on both simulated and real-world mobile manipulation tasks demonstrate superior performance of the proposed method.  \n\n**Weaknesses**:  \n1) Although the idea of the paper is interesting and reasonable, the proposed mobility-to-body conditioning and perception-aware multimodal adaptation are a bit of too straightforward, which makes the technical contribution of paper be of limited significance.  \n2) The paper lacks clarity about the implementation details. For example, the paper only simply describes the single denoising MSE loss for training without explaining the training procedure and computing cost. The model details of the two action heads (DiT) are not described either. In addition, it is difficult to understand how the mobility-to-body conditioning mechanism can be adapted into a dual-arm-to-dual-arm conditioning mechanism for bimanual manipulation.  \n3) As for the experiments, several details are also missing. it is not clear how the testing data is composed and how the testing data differs from the training data. It seems strange to me why the proposed method is compared with two different sets of baselines for simulated and real-world tasks separately. Finally, it is not explained on which benchmark the ablation study is conducted. \nOverall, the latter two weaknesses make the paper lack of reproducibility.", "was_revised": true, "final_justification": "Most of my concerns are addressed by the rebuttal. I also have read feedbacks from other reviewers. Overall, I would increase my rating to borderline accept.", "rebuttal": "# Response to Reviewer NDbd\nWe sincerely thank the reviewer for the thoughtful feedback and constructive suggestions. We provide detailed responses to the raised questions in the following.\n\n## **Response to Weakness 1: Significant Technical Contribution**\nWe would like to clarify that our technical contributions are not \"too straightforward\" or of \"limited significance.\" Our work addresses two fundamental challenges in mobile manipulation that have not been adequately solved by existing methods:\n\n**1. Mobility-to-Body Conditioning Mechanism**: Unlike prior works that treat mobile base and manipulator control independently or trivially concatenate them together, our method explicitly models their interdependency through a novel conditioning framework. This is not merely a straightforward combination. Instead, we carefully design a conditioning mechanism that provides mobile base motion priors to the full-body controller.\n\n**2. Perception-Aware Multimodal Adaptation**: Our dynamic multimodal fusion goes beyond simple concatenation or fixed weighting. We design a similarity-based adaptive weighting mechanism that:\n+ Dynamically adjusts fusion weights based on language-visual modality relevance\n+ Addresses stage-specific perception requirements (2D for semantics, 3D for geometry)\n+ Actively downweights uninformative views during different manipulation phases\n\nThe empirical results demonstrate substantial improvements over strong baselines, validating the significance of our technical contributions.\n\n## **Response to Weakness 2 and Question 1: Implementation Details and Training Procedure**\nWe provide the detailed training procedure and computational specifications below:\n\n### **Training Procedure**\n**Stage 1: Mobility Action Head Pretraining**\n1. **Input Processing**: Extract features from all modalities (2D images $F\\_{2D}$ 3D point clouds $F\\_{3D}$, language $F\\_{\\ell}$) using SigLIP-based encoders, respectively SigLIP-Image-Encoder, SigLIP-based LIFT3D Encoder and SigLIP-Text-Encoder\n2. **Mobility Feature Extraction**: Feed multimodal features into lightweight mobility action head $\\mathcal{H}\\_l$\n3. **Loss Function**: Apply denoising MSE loss for mobile base actions only:\n\n$\\mathcal{L}\\_{\\text{mobility}}=\\mathbb{E}\\_{t,\\epsilon}\\left[\\left\\|\\epsilon-\\epsilon\\_\\theta\\left(\\sqrt{\\bar{\\alpha}\\_t}a^0\\_{\\text{base}}+\\sqrt{1-\\bar{\\alpha}\\_t}\\epsilon,t,F\\_{2D},F\\_{3D},F\\_\\ell\\right)\\right\\|^2\\right]$\n\nIn this formulation, $\\mathcal{L}\\_{\\text{mobility}}$ represents the loss function for mobility action head training, $\\mathbb{E}\\_{t,\\epsilon}$ denotes the expectation over diffusion timesteps $t$ and Gaussian noise $\\epsilon$, $\\epsilon\\_{\\theta}$ is the predicted noise by the lightweight mobility action head with parameters $\\theta$, $a\\_t^{\\text{base}}$ represents the noisy mobile base action at timestep $t$, $F\\_{2D}$ contains 2D visual features from three camera views, $F\\_{3D}$ includes 3D point cloud features, $F\\_{\\ell}$ represents language features, and $|\\cdot|^2$ denotes the L2 norm for mean squared error computation.\n\n4. **Training Details**: Only fine-tune LoRA adapter on 3D encoder and mobility action head, freeze other components\n\n**Stage 2: Full Mobile Manipulation Training**\n1. **Latent Mobility Feature Extraction**: Use pretrained $\\mathcal{H}\\_l$ to extract latent mobility features $F\\_{m}$ from action tokens at each denoising step\n2. **Multimodal Conditioning**: Apply perception-aware adaptation to compute adaptive weights and reweight visual features\n3. **Full-body Action Prediction**: Mobile manipulation action head H predicts both base and manipulator actions conditioned on $F\\_{m}$ and reweighted multimodal features\n4. **Loss Function**: Apply denoising MSE loss for full action sequence:\n\n$\\mathcal{L}\\_{\\text{full}}=\\mathbb{E}\\_{t,\\epsilon}\\left[\\left\\|\\epsilon-\\epsilon\\_\\theta\\left(\\sqrt{\\bar{\\alpha}\\_t}a^0\\_{\\text{full}}+\\sqrt{1-\\bar{\\alpha}\\_t}\\epsilon,t,F\\_v,F\\_\\ell,F\\_m\\right)\\right\\|^2\\right]$\n\nHere, $\\mathcal{L}\\_{\\text{full}}$ denotes the loss function for full mobile manipulation training, $\\epsilon\\_{\\phi}$ represents the predicted noise by the mobile manipulation action head with parameters $\\phi$, $a\\_t^{\\text{full}}$ indicates the noised full-body action sequence encompassing both base and manipulator actions at timestep $t$, $F\\_v$ contains perception-aware reweighted visual features that combine $F\\_{2D}$ and $F\\_{3D}$ with adaptive weights, and $F\\_m$ represents the latent mobility features extracted from the pretrained lightweight mobility action head.\n\n### **Computational Resource and Time Cost**\n+ **Hardware**: 8 × NVIDIA A100 80GB GPUs\n+ **Training Time**:\n    - Stage 1 (Mobility pretraining): 16 hours for 20,000 iterations\n    - Stage 2 (Full model): 30 hours for 30,000 iterations\n+ **Model Parameters**:\n    - Lightweight mobility head: 170M parameters\n    - Full AC-DiT model: 1.2B parameters\n+ **Memory Usage**:\n    - ~60G GPU memory during training (batchsize 16)\n    - ~20G GPU memory during inference\n+ **Inference Speed**: 5Hz on single RTX4090 GPU\n\n### **Dual-Arm Adaptation Details**\nFor bimanual manipulation, we adapt the mobility-to-body conditioning into a dual-arm-to-dual-arm conditioning mechanism by following a similar two-stage training procedure:\n\n**Stage 1: Dual-Arm Action Head Pretraining**\nWe replace the lightweight mobility action head with dual-arm lightweight head that predict a coarse proposal dual-arm action. During pretraining, the training targets change from mobile base actions to dual-arm actions.\n\n**Stage 2: Full Bimanual Manipulation Training**\nIn the full training stage, we use the pretrained head to extract latent dual-arm features from respective action tokens at each denoising step. The bimanual manipulation action head then conditions on the latent features along with reweighted multimodal features to predict coordinated dual-arm actions. This cross-arm conditioning mechanism enables each arm to anticipate the other's actions, thereby improving bimanual coordination and avoiding conflicts between arms during execution.\n\n## **Response to Weakness 3 and Question 2: Experimental Details**\nSince robot testing relies on the testing environment rather than data, no offline datasets were used during testing. Therefore, we provide detailed explanations of our experimental details from the following three aspects: (1) training data composition, (2) differences between training data collection scenes and testing scenes, and (3) rationale for baseline selection. We hope this addresses the reviewer's concerns.\n\n### **Data Composition**\nEach task's training data consists of:\n+ Natural language instructions for the task\n+ 1000 successful trajectories (simulation) / 100 trajectories (real-world)\n+ Each trajectory contains timestamped observations: 2D images, 3D point clouds, robot states, and action decisions\n\n### **Training vs Testing Scene Differences**\n**MSHab Benchmark**: we follow the default training and testing settings in the original MSHab benchmark.\n+ **Training scenes**: Configured layouts with specific object positions (fridges, tables, apples, bowls), target locations, and robot initial positions in ReplicaCAD indoor environments\n+ **Testing scenes**: Same physical environment but different configuration files specifying varied object positions, target locations, and robot starting poses\n\n**RoboTwin Benchmark**: we follow the default training and testing settings in the original RoboTwin benchmark.\n+ **Training scenes:** Configured layouts with specific object positions (bottles, cups, apples)\n+ **Testing scenes**: Same physical environment but different scene IDs to control the layout variations\n\n**Real-world Experiments**:\n+ **Training**: Collect demonstrations with controlled randomization of target object positions within predefined ranges\n+ **Testing**: Same randomization strategy\n\n### **Baseline Selection Rationale**\nWe clarify that the baselines used in simulation and real-world experiments are not entirely different sets. Rather, the baselines used in real-world experiments represent a subset of those employed in simulation. Specifically, baseline π0's performance in simulation is reported in the supplementary material. The reason we use a subset of baselines in real-world experiments is due to the substantial cost of conducting real robot experiments, which necessitated selecting the most representative baselines for comparison.\n\n### **Ablation Study Clarification**\nThe ablation study (Table 4) was conducted on the **MSHab benchmark**. We apologize for any confusion and will clarify this in the revision.\n\n## **Response to Question 3: Language Instruction Generalization**\n**Training-Testing Language Consistency**: Yes, we use the same fixed natural language instruction set during both training and testing, with random sampling at each phase. This demonstrates our algorithm's robustness to language variations within the instruction set.\n\n**Extended Generalization Validation**: To further verify language generalization capabilities, we conduct an additional experiment on the **pick_apple** task in MSHab benchmark:\n+ **Training**: Used original instruction set unchanged\n+ **Testing**: Applied AI-generated instruction paraphrasing to create semantically equivalent but linguistically different instructions\n+ **Results**:\n    - Original instructions: 33.3% success rate\n    - Paraphrased instructions: 31.8% success rate\n    - Performance degradation: ~1.5%, demonstrating strong language generalization\n\nThis validates our method's robustness to natural language variations while maintaining semantic consistency.\n\n---\n\nWe sincerely thank the reviewer for the thoughtful and positive evaluation of our work. We hope our detailed responses have been helpful in answering the questions raised, and we look forward to further discussion in this valuable academic exchange."}, {"review_id": "3w811zWt9b", "reviewer": "Reviewer_3fe8", "summary": "This paper proposes AC-DiT, a novel diffusion transformer framework designed for mobile manipulation. The method introduces two key modules: (1) **Mobility-to-Body Conditioning**, which uses latent mobility features to guide whole-body action generation, and (2) **Perception-Aware Multimodal Adaptation**, which dynamically adjusts the importance of 2D and 3D visual modalities based on task phase. The approach is validated through extensive simulation and real-world experiments, showing improved performance over imitation learning baselines like DP, RDT, and π₀. AC-DiT aims to address coordination challenges in mobile manipulation under vision-language guidance.", "questions": "* **1) How does AC-DiT compare with mobile manipulation-specific baselines like Mobile-ALOHA, TidyBot++, or EquiBot?**\nThe current comparisons seem limited to general-purpose or tabletop-oriented methods.\n\n* **2) Are the two proposed modules (Mobility-to-Body Conditioning and Perception-Aware Adaptation) significantly better than trivial attention mechanisms?**\nCould similar performance be achieved using standard cross-modal or self-attention?\n\n* **3) Can the same AC-DiT policy generalize across diverse mobile manipulation tasks without task-specific fine-tuning?**\nIf not, how scalable is the approach for real-world multi-task deployment?", "limitations": "**Limited Methodological Novelty:**\nThe core mechanisms are customized attention strategies without strong theoretical justification or architectural innovation.\n\n**Incomplete and Possibly Unfair Comparisons:**\nThe evaluation lacks direct comparison to mobile manipulation-focused baselines, weakening claims of state-of-the-art performance.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n* **1) Problem Motivation is Well-Stated:** The paper clearly identifies critical coordination and perceptual gaps in prior mobile manipulation methods and proposes concrete solutions.\n* **2) Simple Yet Effective Architecture:** The proposed modules are computationally lightweight and integrate seamlessly into diffusion-based transformers, contributing to scalability.\n* **3) Strong Empirical Results:** AC-DiT outperforms multiple baselines across both simulated and real-world tasks, demonstrating its practical value.\n* **4) Detailed Ablation Studies:** The experiments carefully evaluate the impact of each component (e.g., 2D/3D fusion, conditioning mechanisms), reinforcing the modular design’s benefits.\n\n**Weaknesses**\n* **1) Method Novelty is Limited:** Both proposed modules—Mobility-to-Body Conditioning and Perception-Aware Adaptation—are essentially task-specific adaptations of attention-based conditioning mechanisms. The core architectural contribution is somewhat incremental.\n* **2) Unfair Baseline Comparison:** AC-DiT is compared primarily against methods like ACT, DP, and RDT, which were designed for tabletop or bimanual tasks, not mobile manipulation. Missing comparisons to Mobile-ALOHA [1], TidyBot++ [2], or EquiBot [3] weakens the empirical claims.\n* **3) Writing and Typographical Errors:** Several typos and formatting issues are present, such as \"RoboTwen\" (should be RoboTwin, line 270), \"issue\" spelled as \"tissue\" (line 322), and wrong reference to “Figure 3” where it should be “Table 3” (line 296), indicating the need for a more careful proofreading.\n* **4) Limited Theoretical Rigor:** The paper lacks a formal analysis or theoretical justification for why the proposed conditioning mechanisms are superior to simpler alternatives, such as vanilla self-attention across modalities or joints.\n\n```\n[1] Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation, CoRL 2024\n[2] TidyBot++: An Open-Source Holonomic Mobile Manipulator for Robot Learning, CoRL 2024\n[3] EquiBot: SIM(3)-Equivariant Diffusion Policy for Generalizable and Data Efficient Learning, CoRL 2024\n```\n\nIn a word, while the paper demonstrates solid engineering and compelling results, its conceptual contributions may be viewed as modest, and comparisons with stronger, more relevant baselines are missing. Strengthening the theoretical motivation and improving experimental fairness could help to raise its final score.", "was_revised": true, "final_justification": "Given the solid empirical results, rigorous ablations, and the additional clarifications provided, I maintain my original positive assessment of the paper and believe the work makes a valuable contribution to the field of mobile manipulation.", "rebuttal": "# Response to Reviewer 3fe8\n\nWe appreciate the reviewer's constructive feedback and positive assessment of our contributions. We provide detailed responses to the raised questions in the following.\n\n## **Response to Weakness 1 & Limitation 1: Clarification on Innovation**\n\n**Clarification on Innovation Focus:** Our core contribution is not task-specific adaptations of attention-based architectures, but rather the overall **framework design** tailored for generalistic mobile manipulation. we propose a mobile manipulation paradigm that can be applied across various architectures. We introduce a new formulation for mobile manipulation that addresses critical coordination and perceptual gaps. Our two proposed innovations (MBC and PMA) present a first attempt to explicitly model the unique challenges of mobile manipulation. **While our current work applies these to a diffusion-based transformer, the underlying principles have the potential to benefit various mobile manipulation architectures that utilize multimodal perception.** The key insight is that mobile manipulation requires fundamentally different coordination mechanisms compared to tabletop tasks.\n\n**Novelty of Our Contributions:**\n\n1. **Mobility-to-Body Conditioning (MBC):** We make the **first attempt** to formulate mobility-to-body control that explicitly models the causal dependency between mobile base motion and manipulator dynamics in end-to-end learning frameworks. Unlike existing approaches that treat base and manipulator actions independently or simply concatenate them, our **novel conditioning paradigm** extracts latent mobility features first and uses them as explicit priors for whole-body action prediction. This represents a **fundamental departure** from conventional joint optimization strategies by establishing a hierarchical conditioning relationship that mirrors the physical causality inherent in mobile manipulation systems.\n2. **Perception-Aware Multimodal Adaptation (PMA):** Different from other works that treat all modalities equally throughout the manipulation process, we make the **first attempt** to dynamically adapt multimodal fusion based on task-stage semantics. Our **novel insight** is that mobile manipulation exhibits distinct perceptual requirements across different phases—semantic understanding during navigation versus geometric precision during interaction. Rather than using static fusion weights or vanilla attention mechanisms, our approach introduces **language-conditioned similarity weighting** that creates instruction-aware modality adaptation. This **pioneering formulation** enables the model to automatically emphasize 2D semantic information when locating objects and prioritize 3D geometric features when executing precise manipulations.\n\n## **Response to Weakness 2 & Question 1 & Limitation 2: Baseline Experiments**\n\n**Clarification on Baseline Selection:** We did not exclusively use tabletop algorithms. Our baseline selection reflects the current state of mobile manipulation research and acknowledges the excellent contributions of recent works:\n\n+ **Mobile-ALOHA** designed the hardware platform and explored ACT models for mobile manipulation tasks\n+ **TidyBot++** designed a comprehensive mobile manipulation robot system\n+ Both works validated their mobile manipulation hardware contributions using existing algorithms (ACT and DP), which is precisely why we selected **ACT and DP** as parts of baselines.\n\n**Additional Baseline Results:** We have implemented and evaluated EquiBot in our simulation environment. Here is the result. We keep all default settings of EquiBot, except modifying its action chunk to 2 (to align with other methods). Since EquiBot is a diffusion algorithm that relies on object-centric point clouds, while our experimental environment supports scene-level point clouds, such point clouds may cause some interference to the algorithm, which could be the reason for its lower scores.\n\n|Method|Pick Apple|Place Apple|Open Fridge|Pick Bowl|Place Bowl|Open Drawer|Close Drawer|Average|\n|---|---|---|---|---|---|---|---|---|\n|EquiBot|5|10|18|14|5|3|50|15|\n\n**Commitment for Revision:** The revised version will include comprehensive comparisons with EquiBot with detailed analysis of the performance differences.\n\n## **Response to Weakness 3: Writing and Technical Issues**\n\nWe acknowledge the typographical errors and will correct all instances in the revised version.\n\n## **Response to Weakness 4: Formal Analysis and Theoretically Motivation**\n\nTo demonstrate and analyze why our method outperforms vanilla attention mechanisms, we provide evidence through two complementary approaches: (1) We prove that representations learned by our method better model the complex behavioral paradigms of mobile manipulation compared to those learned by vanilla attention, through conducting an additional PCA-based representation distribution analysis; (2) We clearly articulate the theoretically motivation of our method to demonstrate its advantages over vanilla attention mechanisms.\n\n### **Formal Representation Analysis**\n\nTo theoretically demonstrate that our method learns representations that better model the complex behavioral paradigms of mobile manipulation compared to vanilla attention mechanisms, we conducted an additional PCA-based representation distribution analysis. We need to verify that our model sufficiently distinguishes different behavioral patterns in the representation space.\n\nSpecifically, we categorized actions from 10 trajectories of the \"pick apple\" task in the MSHab simulator into two distinct behavioral modes: **approaching objects** (mainly mobile base motion) and **grasping objects** (mainly arm motion). We fed these data into both our AC-DiT model and the vanilla attention baseline, then extracted the hidden vectors before the final MLP layer during the last denoising step. We expect the hidden vectors corresponding to these two behavioral categories to be as distant as possible from each other.\n\nTo robustly evaluate representation quality, we employ PCA (Principal Component Analysis) dimensionality reduction to eliminate interference from noisy dimensions and compute the intra-class/inter-class distance ratio in the principal component space. Our method achieved a **significantly smaller ratio**, indicating that **AC-DiT can better distinguish the complex composite action paradigms in mobile manipulation and provides superior modeling of mobile manipulation problems** compared to vanilla attention mechanisms.\n\n|Method|AC-DiT|Baseline (vanilla attention)|\n|---|---|---|\n|intra-class/inter-class ratio|0.197|0.227|\n\n### **Theoretical Motivation**\nMobile manipulation is fundamentally challenging because it requires not only coordinating the mobile base and manipulator to avoid compound errors, but also modeling distinct perceptual requirements across different phases of long-horizon tasks.\nTo address these fundamental challenges, we propose **injecting mobility representations as conditioning priors** and **modulating 2D/3D perceptual features using language-aligned similarity weights**. This approach effectively applies **context-dependent prior biases** to the attention mechanism.\nOur method transforms conventional mobile manipulation modeling into a more robust, structured attention system with explicit priors. As validated by our formal representation analysis, our model achieves clearer representational separation between different behavioral modes, confirming the effectiveness of our framework in enabling the model to better understand distinct action representations in complex and long-horizon mobile manipulation tasks.\n\n## **Response to Question 2: Comparison with Standard Attention**\n\nOur ablation study (Table 4) provides direct comparison with standard attention mechanisms:\n\n+ **Exp1 (vanilla attention):** Uses 2D inputs with vanilla cross-attention for condition injection into DiT model. The DiT internal architecture uses standard alternating self-attention and cross-attention modules. This achieves 37.5% average success rate across 7 MSHab simulation tasks.\n+ **Exp4 (+3D+MBC+PMA):** Further incorporating our Perception-Aware Multimodal Adaptation achieves 49.0% (+11.5% total gain).\n\nThis improvement demonstrates the advantages of our designed mechanisms over standard attention mechanisms. Standard cross-modal attention cannot capture the temporal dynamics and causal relationships that our MBC provides, nor can it adaptively weight modalities based on task stage requirements like our PMA.\n\n## **Response to Question 3: Multi-Task Generalization**\n\nOur model demonstrates strong multi-task generalization capabilities across diverse mobile manipulation scenarios.\n\n**Simulation Evidence:** Our simulation experiments demonstrate multi-task capability across 7 different mobile manipulation tasks without task-specific fine-tuning.\n\n**Real-World Multi-Task Results:** We trained a unified model on all 4 real-world tasks and here is the result:\n\n|Task|Cucumber in the basket|Store breads|Hang Towel|Clean table|\n|---|---|---|---|---|\n|Success Rate|43.8|31.3|18.8|37.5|\n\nThese results demonstrate that our unified model successfully generalizes across diverse mobile manipulation tasks without task-specific fine-tuning. While performance shows modest trade-offs compared to task-specific models (expected in multi-task learning), our approach maintains reasonable performance across all tasks compared to baselines, showcasing the versatility of our framework.\n\n---\n\nWe sincerely thank the reviewer for the thoughtful and positive evaluation of our work. We hope our detailed responses have been helpful in answering the questions raised, and we look forward to further discussion in this valuable academic exchange."}]

Author rebuttal:

Meta-review: The paper introduces AC-DiT, an end-to-end framework designed to improve mobile manipulation. The framework effectively addresses two critical challenges in the field: (1) the explicit modeling of mobility-to-body dependencies via a lightweight conditioning mechanism that prioritizes base motion features to guide manipulator actions, reducing error accumulation in high-DoF control; and (2) the perception-aware multimodal adaptation strategy that dynamically weights 2D (semantic) and 3D (geometric) inputs across task stages, optimizing visual feature fusion for long-horizon tasks. The extensive validation—spanning simulation (MSHab, RoboTwin) and real-world cluttered/dynamic environments—demonstrates state-of-the-art performance, with a 49% success rate on simulated tasks and robust generalization to unseen instructions and scenes.

Reviewers initially raised concerns about novelty (e.g., similarity to prior attention mechanisms) and evaluation scope (e.g., lack of mobile-specific baselines), but the authors thoroughly addressed these in rebuttal: comparative experiments with EquiBot (CoRL 2024) and Mobile-ALOHA validated superiority; PCA-based representation analysis proved the distinctiveness of AC-DiT’s learned features; and supplementary real-world trials (e.g., 35.3% success in dynamic object displacement) underscored adaptability. While the inference time (5Hz) reflects computational overhead from sequential processing, the trade-off is justified by the 30Hz control frequency and performance gains. For camera-ready, authors must: (1) clarify baseline comparisons in the main text, (2) formalize the "coordination" definition introduced in rebuttal, and (3) expand limitations to explicitly discuss compute constraints. This work delivers a reproducible, impactful advance in end-to-end mobile manipulation.
Decision: Accept (poster)