Align-Then-stEer: Adapting the Vision-Language Action Models through Unified Latent Guidance
Authors: Yang Zhang
Chenwei Wang
ouyang lu
Yuan Zhao
Yunfei Ge
Zhenglong Sun
Xiu Li
Chi Zhang
Chenjia Bai
Xuelong Li
Venue: ICLR 2026 Poster (2026)
Keywords: VLA
Generalist Robot Policies
Efficient Fine-tuning
Classifier Guidance
Primary area: applications to robotics, autonomy, planning
TL;DR: We introduce Align-Then-stEer, a method that uses a unified latent space and a guidance mechanism to efficiently adapt pre-trained VLA models to new robots and tasks.

Abstract: Vision-Language-Action (VLA) models pre-trained on large, diverse datasets show remarkable potential for general-purpose robotic manipulation. However, a primary bottleneck remains in adapting these models to downstream tasks, especially when the robot's embodiment or the task itself differs from the pre-training data. This discrepancy leads to a significant mismatch in action distributions, demanding extensive data and compute for effective fine-tuning. To address this challenge, we introduce Align-Then-stEer (ATE), a novel, data-efficient, and plug-and-play adaptation framework. ATE first aligns disparate action spaces by constructing a unified latent space, where a variational autoencoder constrained by reverse KL divergence embeds adaptation actions into modes of the pre-training action latent distribution. Subsequently, it steers the diffusion- or flow-based VLA's generation process during fine-tuning via a guidance mechanism that pushes the model's output distribution towards the target domain. We conduct extensive experiments on cross-embodiment and cross-task manipulation in both simulation and real world. Compared to direct fine-tuning of representative VLAs, our method improves the average multi-task success rate by up to 9.8% in simulation and achieves a striking 32% success rate gain in a real-world cross-embodiment setting. Our work presents a general and lightweight solution that greatly enhances the practicality of deploying  VLA models to new robotic platforms and tasks. Our code is released at \url{https://github.com/TeleHuman/Align-Then-Steer}.

Published as a conference paper at ICLR 2026
ALIGN-THEN-STERR ADAPTING THE VISION- LANGUAGE-ACTION MODELS THROUGH UNIFIED LATENT GUIDANCE
Yang Zhangl.3r, Chenwei Wangt.3,, Ouyang Lu Yuan Zhao', Yunfei Ge', Zhenglong Sun Xiu Li", Chi Zhang' Chenjia Bai Xuelong Li IIstitute of Artificial Intelligence, China Telecom, TTinng University, The Chinese University of Hong Kong, Shenzhen, "Northwestern Polytechnical University baicj@chinatelecom en, xuelong 1etlliiiiee Li@ieee..... org
ABSTRACT
Vision-Langugge-Actio models (VLAs) pre-trained large, diverse datasets show remarkable potential for general-purpose purpose robotic manipulation. However, primary bottleneck remains in adapting these models to downstream tasks, es- pecially when the robot's embodiment or the task itself differs from the pre- training data. This discrepancy leads to significant mismatch in action dis- tributions, demanding extensive data and compute for effective fine-tuning, To address this challenge, we introduce Align-Then-stter (ATE), a novel, data- efficient, and -nug-aad adaptation framework. ATE first aligns disparate ac- tion spaces by constructing unified latent space, where variational autoencoder constrained by reverse KI. divergence embeds adaptation actions into modes of the pre-training action latent distribution. Subsequently, it steers the diffusion- or flow-based VLA's generation process during fine-tuning via guidance mecha- nism that pushes the model's output distribution towards the target domain. We conduct extensive experiments on cooosmembodiment and cross-task manipulation in both simulation and real world. Compared to direct fine-tuning of represen- tative VLAs, our method improves the average multi-task success rate by up 9.8% in simulation and achieves a striking 32 % success rate gain in a real-world cross-embodiment setting. Our work presents general and lightweight solution that greatly enhances the practicality of deploying VLA models to new robotic platforms and tasks.
INTRODUCTION
Aision-Language-Actinnn models (VLAs) (Brohan et al. 2022 Zitkovich al., 2023 Kim al.
2024 Black et al.) 2024) have emerged as powerful solution to empower general-purpor robotic manipulation with broad generalization tasks and environments. Inspired by the success in scal- ing data and parameters of Large Language Models (LL.Ms) and Visual-Language Models (VL.Ms), the VLA models extend the pre-trained VLM models (Li al.) 2023 Driess et al., 2023] Beyer al. 2024 Wang et al. 2024) by integrating action expert module to generate low-level actions, thereby inheriting the common knowledge, semantic reasoning, and instruction-following capabil- ities of VLMs to facilitate policy learning. Specifically, representative training recipe for VLAs typically consists of two phases: a (i) pre-training phase: pretrain the entire VLA model that con- tains strong VLM and action expert a large corpus containing cross-ents emboddmem datasets (Jang et al.] 2021 [oolaboration et al.] 2023 Khazatsky et al. (202)), which spans diverse robots and tasks to learn general visuomotor prior scale, and (ii) adaptation phase: fine-tune the VLA model on a narrowly curated dataset tailored to particular robot platform and associated tasks, aiming enhance the manipulation abilities in specific downstream domains.
However, the two-phase paradigm for pre-trained VLAs faces a significant challenge when adapt- ing to downstream domains with different embodiments. For example, pre- datasets often
Equal contribution
Corresponding authors

---

Published as a conference paper at ICLR 2026
use single-arm robots, while target embodiments might be dual-arm humanoid. Since VLAs tied to imitation learning and its action labels, the resulting action space and distribution mismatch between pre- trainini and adaptation phases create substantial bottleneck. Even with the same robot, differences in tasks and setup lead to similar distribution discrepancies, often requiring ex- tensive data collection resolve. While recent approaches focus on reducing training costs the parameter level (Kim et al. 2024 Zhang et al. (025b) or with different architectural designs (Kim) al., 2025; Pertsch al., 2025), they often struggle address discrepancies in task distributions and overlook the embodiment mismatches that are significantly more challenging to resolve. This motivates our central question: how can we achieve VLA adaptation for both coossemmodiment and
cross-lask scenarios under limited data regime?
To this end, we propose a data-efficient adaptation method, named Align-Then-stEer (ATE), for cross-embodiment and cross-task VLA adaptation by explicitly mitigating cction-distribution mis- match via two-stage strategy. In the first stage, we construct a unified latent action space to bridge the domain gap between embodiments and tasks involved during pre-training and the adaptation phases. It is achieved by training Variational Autoencoder (VAE) on the pre-training action-labeles dataset obtain latent prior, followed by learning smaller VAE on limited adaptation data with a reverse KL divergence constraint toward the learned pre-trained latent distribution. Thanks to the mode-seeking behavior of reverse KI. divergence, this encourages the encoded latent of the target embodiment's actions to be embedded into a specific mode of the learned prior, yielding a unified and hierarchically structured latent space. In the second stage, inspired by massiffier-uiied diffusion models (Dhariwal & Nichol 2021; [Bansal et al., (023),, we design a guidance function grounded in the shared latent space that can be seamlessly integrated into the training objective of diffusion- or flow-based VLAs. This classifier guidance steers the output distribution of the pre-trained VLA toward that of the target embodiment and task, enabling faster and more precise adaptation under limited data. Notably, our method requires no modification to the original VLA architecture and introduces only negligible overhead, i.e., adaptation highly efficiently achieved by training two separate lightweight VAEs. An overview of our method is given in Fig. Our key contributions are summarized as follows:
We propose novel alignment strategy to bridge the action-distribution gap between pre-training and adaptation. By leveraging the mode-seeking behavior of reverse KL, our method constructs unified, structured latent space that embeds target actions into modes of the pre-training latent distribution.
We introduce a classifier classifier-guidance mechanism that operates within this unified latent space to explicitly steer the VLA's output distribution. This enables precise and rapid adaptation to new embodiments and tasks, without requiring additional data.
The proposed ATE framework is model-agnostic and blug-and-play seamlessly integrating with various diffusion- or flow-based VLAs without modifying their architecture and with negligible computational overhead.
We validate our method across different embodiments and tasks in both simulated and real- world manipulation settings. The result demonstrates consistent improvements over baselines that rely on direct supervised fine-tuning, highlighting our approach's 's superior adaptation efficiency.
2 RELATED WORKS
Policy Adaptation of VLAs. Despite the promise of VLAS for general-purposs robot manipulation, current approaches still suffer from high fine-tuning costs for embodiment and task mismatches. Several methods address this by constructing shared latent action models (Ye et al. [025 Bu et al., 2025 NVIDIA 2025). performing kinematics retargeting (Bauer et al., 2025| Yuan et al. 2025 or employing leghtt fine-tuning techniques like LoRA-based updates (Wen et al. 2025b)
and dynamic layer activation (Zhang et al., 2025a). While these approaches improve training ef- ficiency or establish some form of shared space, they are often limited by architectural constraints (e.g., auto-regressive models) or fail to effectively bridge the significant action distribution gap be- tween domains. In contrast, we propose a novel perspective that we can use the mode-seeking property of VAES to explicitly bridge the distribution gap between pre- pre-trainin and adaptation data. This is complemented by latent guidance mechanism that is compatible with both diffusion- or flow-based VLAs. More discussions on VLAs and constructing unified action spaces is deferred to Appendix A.
2

---

Published as a conference paper at ICLR 2026
RoboTwin
Large-scale embodimantt datasets
Small-szale fine Fne-tuning
datasats
Open Io wodmmooonnn
Alignment
Pre- Trained Vision-Lonr Langunge-gttionnn Model
Maniskill
Real-World
Target Domain
Bimaii
Transfrmme
munching
Source
Domain
Unifled Action
Post- Training
Latent Space
Align
Then stEer
Real World
Simulation
Original design
Visual distractor Human disturbance
Dual-Arm Tasks
Bimanual
dexterous
coordination
Make Sandwich
Distractor Objects
Interfereace
Robn 1.0 Single-Arm Tasks
Minute-level Long-horizon tasks
Pick Bun
Light Change
Interference: Replacing the bun into the pot
Maniskill
RoboTwwn 1.0
Figure 1: We present ATE, plug-and-play adaptation framework for pre-trained VLAs.. By aligning disparate action spaces into a unified latent representation and steering generation via guidance, ATE enables data-efficient cross-task and tross-embodiment adaptation. Evaluations RoboTwin, ManiSkill, and a real-world dual-arm RealMan 7-DoF robot show substantial gains in success rates.
3 PRELIMINARIES
Training objective of VLAs. The training of VLA models typically contains two stages: (i) obtain initial policy via pre-training the model cross-embodiment action-labeled dataset Prettain. where we denote the set embodiments included in it as Mpretzain = {E1, ,,,...,,,,, } : (ii) finetune the policy bodiment Enrget for downstream applications. Formally, the optimization objective of the VLA policy my to maximize the log-likelihood of an action chunk a given observation o. and a language- format task instruction sampled from the dataset, which can be formulated as max; (i n. .!)D [log To (am:4h (o. where o, contain single-view or multi-view RGB ob- servations 17 I?...., , I,, and the proprioceptive state qe usually describes the joint position and angle of the robot.
narrow but high-quality dataset Dalsptation collected from the target
Diffusion Models and Flow Matching. Diffusion models (Sohl-Dickstein et al. 2015 Ho et al.) 2020 [oog et al. 202T) are generative models where data generation is modeled a denois- ing process. Consider a forward diffusion process that gradually adds Gaussian noise to sam- ples Xy (xx,) over T' steps and produces sequence of noisy samples X, that sat- A(xx; Va,x,-1, (1 ax)),, where for E (0, 1)) a predefined noise scheduler. The denoising diffusion model reverses the above process to approximate the data distribution Po (xo) which is parameterized as pe = ((xo:)) dxi:r f "(xx) - pe (x, xx) dxi.r. By minimizing the following denoising loss function introduced by Ho et al. (2020).
isfies q(xir1 Xg) IT_ [(xxxxx1) and q(xx[xx-1)
(()) = Ex..
cø (V + VI - âee, 1)] where āi
ai
we can a noise prediction network cx(x;, t) to perform the reverse diffusion process it begins by sampling XT ~ (O, T), and then iteratively applies the update
1
Xk-1
(xk
vI dued (x+; k)) + a&Ek:
(2)
Vor
where eg NO 7), and 0% is function of k and depends on the noise scheduler.
3

---

Published as a conference paper at ICLR 2026
Steawe the hond Loing the toaster
Large scale cross- embodiment datasets
Adaptation data of the target embodiment
Get steameed dow
Moknng Sindwnche
Action Latent pretrained 9p( 9ex
Action Latent target Qa(a (er-e)
Fixed p(a) N((OF)
AplelAesat
Latent
Fncader
min M(0.).
man 9
Latent Adtion 2
Action
Latent Decader Py (Hers
Latent
Docnder
Pa(Mecad
(a) Stage I: Learning the Unified Action Latent Space
Denoising Difitsion Flow Traectury (A (ist...)
Flame Mariching
Diffaiion Moddl
from Adapcationn
Flow-bassed VLAs, (Ra, et..) Diftusion VLAS. (RDT, esi)
"Tee book broad
juto the nuunc
RCB
Insmactionn
Propeiaapiice U.
guldiance
Steer Adoptition
actutt
Empocios Ur
Clanptter Ganiincr
Nokey Aution Latesi Rylaf * Taggei Action Latant (aluoa)))
Rats duis
Caty, lia mal??
Piejest
FRoda
lapue
8
(b) Stage 2: Steering Efficient Adaptation with Latent Guidance
The overview of ATE framework. (a) In the first stage, we construct a unified action space to bridge the embodiment gap in pretraining and adaptation stages by utilizing the mode- seeking behavior of asymmetric VAEs. (b) In the second stage, we integrate classifier guidance in diffusion and flow-based VLAs steer the pretrained policy towards the target action distribution with specific robot platforms.
Figure 2:
Beyond denoising diffusion, flow matching is another generative modeling framework that imple- ments data generation as a probability path from a known source distribution to the data target dis- tribution (Lipman et al. 2023] Liu et al. 2023b Lipman et al., 2024). Assuming the source sample and target sample are denoted as X and Xo respectively, and flow matching timesteps are denoted as E [0, 1). By instantiating the path as a Gauss--G path given by X, (N(Txx, (1 -)) flow matching operates by first sampling noise from a standard Gaussian and then processes this noise through deterministic process to produce a sample from the target distribution, similar the denoising process. The flow that maintains the probability path is typically trained via optimizing a simple conditional flow matching loss function,
(()) Er,xo,e [| (1 T), (Xo <)][]]
In robotic manipulation, there can be dozens of possible actions a, accomplish the task given the same language instruction / and observation. Motivated by the capability of diffusion models and flow matching on providing high-precision continuous representation and modeling multi-modal
distribution, recent VLAs (Black et al.) 2024 Liu et al. 2024 Intelligence et al. 2025 Bjorck et al., 2025) have incorporated diffusion-bast or baseb action prediction boost performance.
Problem Setting. In this work, we assume access to pre-trained diffusion-basee or flow-based policy, which can either a VI.A model or Vision-Action model like Diffusion Policy (Chi et al.) 2023). Our goal is to adapt the pretrained model in D Prettain to a specific embodiment Eurget given an adaptation dataset Padaptation- where Exareet is not be involved in pre-training, i.e., Erageet & MMpeetrinn In addition, the tasks contain in Dadaptation can also be significantly different from that in pre-training. For clarity, we use the superscript & to denote the diffusion or flow-macching timestep, and the subscript to denote the trajectory timestep.
4

---

Published as a conference paper at ICLR 2026
4 METHODOLOGY
The proposed ATE algorithm mainly consists of two stages: (i) we extract unified action latent space to compactly encompass the actions of different embodiments, thus bridging the domain gap between the embodiments in the pre-training and adaptation phase (details in Section 4.1): (ii) then, we design a guidance function grounded in the unified latent space and leverage the resulting clas- sifier guidance explicitly steer the fine-tuning towards our desired data distribution during adap-
tation (details in Section 4.2)
Fig. 2 provides an overview of our entire procedure.
4.1 ALIGNMENT THRROGGI UNIFIED ACTION LATENT SPACE
A pre-trained VLA acquires a general visuomotor prior by modeling distribution p(ägr+ ---looo 1), where action chunk -1-1 of length H, observation o, and language instruction 1 are sam- pled from the pre-training data D pretrain- Such a prior effectively captures the action distribution PD (a1:+H-1) from the pre-training embodiments MMrectainn However, due to the inherent dif- ferences in degrees of freedom (DoFs) and physical representation (c.g.. joint angles, end-effector poses, and joint torques), the raw action-labeled data from the target embodiment Eurget can vary significantly from the action generated by the pre-trained VLA. This leads to a substantial domain gap that hinders efficient adaptation. To bridge this gap, we employ two distinct Variational Autoen- coders (VAEs) to encode the fundamentally disparate pre-training and adaptation action data into single, unified, and compact action latent space.
Learning Pre-training Action Latent.
We build a pre-training action-VAr Vpretrain == EE }
to compress all possible action chunks in the pre-training data into a low-dimensional latent space 2, which consists of Transformer encoder Es and Transformer decoder D The encoder takes an action chunk at:-- of length H as input and outputs the parameters of a Gaussian distribu- tion (z =+ to represent the action latent. Here, we instantiate pretrain as an Info VAE (Zhao itt al. 2017) rather than a vanilla VAE (Kingma & Welling 2013). Formally, it is optimized by max- umizig the log- ligeeiod term together with KL regularization term towards the fixed latent prior distribution and mutual information maximization term that encourages high mutual information between H- and z. The objective can be simplified as,
C(ó; c D Prectiin) = EE ++ Plogg H-1 [2]]-
(1 Em [Dxr.
(a x
(4)
where Vpectrain parameterized by d. and a and À are hyperparameters that control penalty strength. The latent prior p(z) is set as unit Gaussian (O0, I).
Embedding Adaptation Action Latent into Pre-training Action Latent Space. With the pre- training action VAE Ypretrain- we obtain the learned action latent distribution (=(2) by marginal- izing over all action chunks in Prretrain- To create unified action latent space, we train a separate adaptation action VAE V Vulluptation := {Eg, D } which is parameterized s,, with KL regulariza- tion towards the learned pre-training action latent distribution. Denoting the action chunk of length L in the adaptation data as L- the learning objective of Vadaptation is written as follows,
[v;; D + Datiptation = Ep (114, [log pe (ac++
(1 - a 3D (Pau (1 - (a (a- 1) (l(a))) (5)
The adaptation action VAE has the same architecture and variant as the pre-training VAE. Minimiz- ing the reverse KL divergence DKL (g [ instead of DKL qo) ensures that the learned adaptation action latent distribution is mode-secking (Bishop & Nasrabadi] 2006). There- fore, this effectively embeds the adaptation latent distribution qu (z) into one mode of the pre-trained action latent distribution g0 (2), yielding a unified and compact latent space Z.
Implementation Details. The purpose of using two VAES to bridge the distribution gap of the output action chunks between the pre-trained VLA and the adaptation data. Thus, the pre-training VAE V Vpretrain uses the same action chunk length H as in the pre-training phase, while the adapta- tion action VAE Vuiaptation adopts the action chunk length L. used in the adaptation phase. Follow- ing the guidelines of /Zhao et al. (0177). we replace the last KL term in both Eq, (4) and Eq. (5) with Maximun-Mean Discrepancy (Li et al. 2015 Dziugaite et al.) 2015) ) for more tractable op- timization. The learned pre- pre-trainn training action latent distribution qs in Eq. 5) is approximated as ((g. Σg). where He and Eg are calculated over all sampled pre- pre-training action latents. A de- tailed algorithm for training both VAEs is provided in Alg. More details on the design of the Transformer-basee based VAE are also provided in Appendix C
5

---

Published as a conference paper at ICLR 2026
4.2 CLASSIFIER GUIDANCE FOR STEERING ADAPTATION
To facilitate efficient adaptation of pre-trained diffusion or flow-based VLA, we introduce classifier guidance (Dhariwal & Nichol 2021) explicitly steer the fine-tuning process. The core idea is to design a guidance function that measures the discrepancy between the generated actions and the target action distribution within the unified latent space Z, and then use the gradient of this functioon to guide the policy update. This approach enables highly efficient adaptation without requiring additional data budget. In the following, we first introduce the formulation of classifier guidance in diffusion and flow-based VLAs; then we detail the guidance function according to the learned latent action space.
Classifier Guidance Diffusion- VLAs. Note that the action latent 2 is only determined by the action chunk a i.e., independent of the observation o and language instruction 7. As the conditioning label we use for guidance is based solely on the latent we thus omit these conditions and treat as an unconditional diffusion model (Ty()) == pe() for notational simplicity. A diffusion model typically predicts the noise eø ca k) that corrupts initial action chunk L.++ to noisy version a - To generate a desired action chunk conditioned on label y, we have to sample from the conditional distribution pe ( Formula and Bayes' rule, we can obtain calibrated noise prediction é that mcorporates the guidance
Fortunately, as proven by Luo (2022), by Tweedie's
gradient 9 V Vat... log p(y) (lyla..... +h which corresponds to the score of the joint distribution of the action chunk and the given conditioning label y, as
(( k) co k) V ā. at logr(yla(.ca)) co k) v1 o.g.
(6)
where delay the discussion of the exact expression of g subsequently. By incorporating this modified noise prediction into the original denoising objective, we explicitly steer the fine-tuning stage of the VL... The final objective is reformulated as,
(()) = Ex (o............... (.-e |||c - c (a k, 1,, + vI- as - ..91)
(7)
where À is guidance scale that modulates the influence of the guidance.
Classifier Guidance in Flow-based VLAss Similarly, for those VI.As that model the generation of action chunks with Flow Matching framework, we also provide updated objective that directly incorporates the classifier guidance into the learning of the flow. As proven by Lipman et al. (2024), supposing velocity field Uy is an instance of linear Gaussian flows x" rx' + 7)x" where q(x)) is the target data distribution and p(x²) = -O, I) is the source distribution, the transforma- tion between marginal velocity fields and the score function for unconditional distributions satisfies
Vxr log (oo").
0g
We refer readers to Lipman et al. (2024) for a detailed proof. Similarly, when we condition the velocity field uo (a + 17) on label y. have
00 (a (v) = vo (aT.aa)))) + 1-T
Vai.. W.e. log ((
(9)
The final objective for fine-tuning the pre-trained velocity field uo is thus rewritten as,
( = Ey,,, al D...DDDD [100 (2Tr4AATT,0,,)) + A -
- e)1?
(10)
Designing a Guidance Function for Efficient Adaptation. The goal of fine-tuning pre-trained VLA is to fit the action distribution of the adaptation data. Despite the inherent heterogeneity of actions from different embodiments, by leveraging the unified and compact action latent space 2, established in Section[4.1 To facilitate efficient adaptation, we design a guidance function that measures how closely an intermediate ac- tion chunk during the denoising process, â aligns with the ground truth action chunk, al. from the adaptation data. Inspired by CCrvalho et al. (2023); [iang et al. (2025), we formulate the classifier induced by such guidance function
can effectively measure valid discrepancy between them
energy- based model,
1
exp(
Eg (af Ep
Z

---

Published as a conference paper at ICLR 2026
where Egg is the encoder of the adaptation action VAE and Z is normalizing constant dependent on v.. Then the classifier guidance 9 can be easily calculated via,
g
log Pa (y) a ( - sitth E (a (x
(11)
where we use Eg to encode both noisy actions and clean actions. One may ask whether the encoder Eg can robustly encode noisy action chunks. As identified by ZZao et al. (2017), the vanilla VAE (Kingma & Welling 2013) suffers from issues such as posterior collapse and inaccurate amortized latent inference, while the VAE effectively addresses these limitations, making an appropri- ate alternative for our deliberately designed guidance function. Since the guidance relies on the ground truth action chunk, which is not available during VLA inference, we incorporate this guid- ance explicitly into the training objective, shown in Eqs. ( and (TOp. During training, we use the noisy action chunk f corrupted from its ground truth gt++ as a reliable substitute for the corresponding intermediate action chunk during the denoising process.
This gradient guidance allows us to explicitly calibrate the transitions at any noise level k; pushing the VLA's output consistently towards latent representation that reduces the distance to the target distribution throughout the whole denoising process. As shown in Fig. [2 our method not only steers the pre-trained VLA towards the adaptation data, but also constrains its output remain within the unified action latent space which happens to be a specific mode of the pre-training action latent space. By preventing the VLAs from deviating far from this structured latent manifold, our method also implicitly helps preserve valuable visuomotor prior knowledge acquired during pre-training. Detailed algorithmic descriptions for how we steer the adaptation of diffusioon-bsed/ffove VLAs are provided in Algs. and
5 EXPERIMENTS
To demonstrate the efficiency of ATE in facilitating the adaptation process across embodiments and tasks, we evaluate our approach in both simulated and real-world settings. Through extensive exper- iments, our aim to answer the following key questions: Q1: How efficiently does ATE facilitate the VLA's adaptation under cross-task and cross- eoooddmemod regimes? (Section 5.2) Q2: To what extent does ATE enhance the generalization of the fine-tuned VLA against diverse perturbations, such as variations in object placement, lighting conditions, and visual distractors? (Section (.3) Q3: How critical is our proposed aligned and structured action latent space to achieving efficient cross-embodiment and cross-task adaptation? (Section 5.4)
5.1 EXPERIMENTAL SETUP
Experimental Environment. To demonstrate the efficiency of ATE in facilitating the adaptation process across embodiments and tasks, we evaluate our approach in both simulated and real-world settings. For simulation, we evaluate on two benchmarks: RoboTwin 1.0 (Mu et al. 2025). featur- ing diverse bimanual tasks, and ManiSkill3 (Tao et al. 2024), focusing contact-rich single-arm manipulation. Together, these allow us to test ATE under both bimanual and contact-rich regimes, directly addressing Q1. For real-world experiments, we deploy ATE-trained policies on dual-arm RealMan robot, with four long-horizon dual-arm tasks and one tool-use task (Appendix B). This setup provides a strong testbed for adaptation across task and embodiment shifts (Q1) and robust- ness under real-world perturbations such as object placement, lighting, and visual distractors (Ω2).
Further details are in Appendix [D]
Baselines.
Since ATE is designed to be plug plug-and-pp we benchmark it on representative diffusion- and flow-natching architectures, including Diffusion Policy (DP), RDT-1B (Liu al.) (0024). and To (Black et al. 2024) (details in Appendix[D.1] Note that in all cases, the baseline comparison against direct fine-tuning of these architectures without ATE.
Implementation. Firstly, we adopt a two-step VAE training scheme to learn structured latent representations over action chunks. In step-1, Info- is trained on large-scale VLA datasets to learn transferable latent representations, while step-2 trains on small sets of domain-specific data to align the action distribution of the target domain. Detailed dataset configurations and training settings for both steps provided in Appendix C Secondly, the pretrained Info-VAE is used to provide structured latent information for a lightweight guidance module. which facilitates efficient VIAAs. During policy fine-tuning, the guidance
adaptation of diffusion- or fow-matching-
7

---

Published as a conference paper at ICLR 2026
module leverages the Info- VAE latent space measure the discrepancy between predicted and tar- get actions and applies gradient signals to steer the policy updates toward the target distribution. The Info-VAE parameters are frozen, with guidance enhanced via random latent perturbations and dynamically scaled according to the current generation step. A tunable parameter controls the over- all strength of the guidance, allowing flexible trade-off between adaptation efficiency and stability- This approach enables rapid adaptation without requiring additional data, and integration into ex- isting VLA training pipelines straightforward and computationally lightweight. Full algorithmic details, including the latent-space guidance formulation, are provided in the Appendix
5.2 MAIN RESULTS
Simulation Results. To answer Q1, we first evaluate both RDT and To on 17 manipulation tasks from the RoboTwin 1.0 benchmark, with results summarized in Table [ Our ATE variants signif- icantly outperform the original models, achieving average success rate improvements of +10% for RDT and +9% for mo, with gains exceeding +30% on challenging tasks such as put apple cabinet and empty cup place. In addition to higher final performance, our method demonstrates improved sample efficiency. surpassing the RDT baseline at just 70k steps compared 90k. Detailed per- formance of the DP baseline and ATE variants, showing consistent success rate improvements and accelerated convergence, along with full analysis of in-distribution and out-of-distribution results, is provided in Appendix D.2] The performance gain can be attributed to two key advantages of our algorithm: (i) the unified action latent space, which compactly represents actions of different embodiments and tasks, and bridges the domain gap between pre-training and adaptation; and (ii) the latent space guidance, which explicitly steers fine-tuning towards the target action distribution, which ensures efficient adaptation while preserving valuable visuomotor priors, resulting in both faster convergence and higher final success rates. The effectiveness of the proposed approach is further validated the ManiSkil13 benchmark using RDT as the policy backbone. Table 5 in Ap- pendix [D.2]hows consistent improvements from the ATE adaptation method on two representative tasks, indicating strong generalization in highly dynamic simulated environments. Overall, the re- sults demonstrate that our method consistently enhances performance across tasks and architectures, particularly under limited data conditions.
Real-World Results. To further address Q1 in terms of efficient cross-task and cross-embodiment adaptation, we conduct real-world experiments on a dual-arm RealMan robot, with each arm having 7 DoF and equipped with Agibot OminiPicker single-DoF grippers. To be specific, we design four representative manipulation tasks for our experiments. The detailed experimental setups and data collection procedures provided in Appendix [33 and Appendix The experiments focus on long- horizon manipulation sequences, each lasting over one minute and requiring precise coordination between arms and objects, thus providing strong testbed to validate ATE for efficient cross-task
Table 1: The comparison of task success rates on RoboTwin 1.0 benchmark
Method
Task
RDT-IB
RDT-1B B ATE (Ours)
ATE (Ours)
LLu et al. 2024
Black al.
2024
Block Hammer Beat
52%
71 (1 19)
38%
44%
Block Handover
69%
91 96 22)
80%
92%((1 12)
Blocks Stack (Fasy)
10%
11% (1)
30%
50 % 20)
Blocks Stack (Hard)
1%
7% 6)
8 %%
7%) 1)
Bottle Adjust
53%
37%(( 16)
39%
45%
Container Place
34%
55% 21)
56%
59%
Diverse Bottles Pick
18%
24% 6)
20%
40 % 20)
Dual Bottles Pick (Easy)
76%
87 97% 11)
48%
85 % 37)
Dual Bottles Pick (Hard)
39%
58 38% 19)
52%
55%
Dual Shoes Place
6%
9%
22%
24% 9
Empty Cup Place
22%
61 % 39)
32%
36 %c1
Mug Hanging (Fasy)
6%
6%
11%
27 %1 16)
Mug Hanging (Hard)
1%
1%
4%
4%
Pick Apple Messy
35%
39 (9%611 4)
18% %
11 11%6
Put Apple Cabinet
20%
45% 25)
34%
55 %
Shoe Place
44%
43% I)
52%
58% 6)
Tool Adjust
54%
42%)( 12)
70%
69%(( I)
Average
31.8%
41.6% 9.8)
36.1%
44.8 (t 8.7)
8

---

Published as a conference paper at ICLR 2026
Tu+ATE(OUrs)
cook bun
make sendwich
pick bun
use toaster
100
Success Rabe 1ha
30
33
20
26
13
10
15.0
fast plan. coile.
boou' riglt
Take den Phoer stoener
Gbi bood ciyle Janun lete manee.
Figure Real-world evaluation on physical robot experiments. Top panel reports success rates across four tasks and overall average, comparing ATE with the baseline тo. The success rates are measured at 60k, 90k, and 120k steps to assess both convergence speed and final performance. Bottom panel shows full execution trajectories of four representative tasks, covering long-horizon, single-arm, and dual-arm scenarios.
and cross-embodiment adaptation. The four tasks are Cook Bun, Pick Bun, Make Sandwich, and Use Toaster, with detailed definitions provided in Appendix D.3] Figure [ visualizes success rates across tasks different training steps. Our method significantly accelerates learning, achieving higher success rates with fewer training steps (detailed analyses are provided in Appendix (.3) Besides, in real-world experiments, we also observe significant improvements in motion quality (see Appendix for detailed analyses).
5.3 GENERALIZATION ABILITY ANALYSIS
To answer Q2, evaluate the robustness and generalization of ATE under diverse real-world perturbations. Detailed experimental setups, including lighting conditions, spatial shifts, visual dis- tractors, and human disturbances, 5 trials, with success defined as completing the full manipulation sequence. The experimental setups for all perturbations are illustrated in Fig. 9 and the corresponding results reported in Tables and all in Appendix D.4] ATE consistently outperforms the baseline across all generalization ex- periments, demonstrating stronger robustness and generalization in real-world settings (Q2). This strong generalization arises because our method constrains the policy within the structured latent space learned during pre-training, effectively preserving pre-trained visuomotor priors while allow- ing adaptation to new environmental variations.
provided in Appendix D.4] Each experiment is evaluated over
5.4 ABLATION STUDY
To answer 23, we study the impact of the aligned and structured action latent space on cross- embodiment and cross-task adaptation. Our two-step Info-VAF, which embeds adaptation actions into a mode of the pre-trained latent manifold, consistently outperforms a single-step VAE trained directly task-specific data, using both mo and Diffusion Policy backbones. These results confirm that the aligned latent space critical for efficient adaptation, enabling downstream policies to rapidly leverage prior knowledge and generalize across different tasks and robotic platforms (see detailed results and the corresponding figure Appendix D.5).
6 CONCLUSION
In this work, we present ATE. lightweight and data-cfficient framework for adapting VLAs to new embodiments and tasks. By unifying disparate action spaces and steering the pre- pre-trained model via classifier guidance, our method effectively bridges the action distribution gap between pre-training and downstream adaptation. The experimental results on representative VLA architectures, includ- ing DP. RDT, and mg verify the widely applicability of our adaptation method. The cross-task adapta- tion experiments baselines. By constructing complex bimanual collaboration tasks in a new robot platform, we ob- serve ATE obtains significantly better performance than standard adaptation methods, especially for spatial, visual, and disturbance generalization settings. Our results highlight the practicality and scalability of ATF, offering plug-and-play solution that seamlessly integrates with diffusion-
RoboTwin and ManiSkill benchmarks show that ATE consistently outperforms
9

---

Published as a conference paper at ICLR 2026
and flow-based VLAs without altering their architectures. Looking ahead, envision that this alignment-and- gunddncc paradigm can serve as a foundation for fine-tuning generalist robot policies at scale, enabling efficient adaptation to new robotic platforms and diverse tasks.
ETHICS STATEMENT
All procedures in this paper were conducted in accordance with the ICLR Code of Ethics (ttpps:
iclr,
REPRODUCIBILITY STATEMENT
For the practical part, we give experiment setup in Section 5.1 The hyper-parameteen and imple-
mentation details given in Appendix D and
The code is released https: /tttpp://githbb/
com/TeleHumen Align-nninni--- Then Steer
ACKNOWLEDGMENTS
This work is supported by the National Natural Science Foundation of China (Grant No.62306242), the Young Elite Scientists Sponsorship Program by CAST (Grant No. 2024QNRC001), and the Yangfan Project of the Shanghai (Grant No.23YF11462200,,
REFERENCES
Johannes Anschütz and Arthur-Céssi Le Bras. Prismatic dicudonné theory. In Forum of Mathemat- ics, Pi, volume 11, pp. e2. Cambridge University Press, 2023.
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 843-852, 2023.
Erik Bauer, Elvis Nava, and Robert K Katzschmann. Latent action diffusion for cross-embodiment manipulation. arXiv preprint arXiv:2506. 1.....4666 14608, 2025.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang. Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile vlm for transfer. arXiv preprint arXiv:2407.07720 2024.
Hongzhe Bi, Lingxuan Wu, Tianwei Lin, Hengkai Tan, Zhizhong Su, Hang Su, and Jun Zhu. H- rdt: Human manipulation enhanced bimanual robotic manipulation, 2025. URL https:// embodiedfounde iiniinaatiinn. .wt.iiintnniibb............. io/hrdt
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, vol- ume 4. Springer, 2006.
Johan Bjorek, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang. Dieter Fox, Fengyuan Hu, Spencer Huang, al. Gr0Ot nl: An open foundation model for generalist humanoid robots. arXiv preprint 213V 2025.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. no: A lssion-lagguage-action flow model for general robot control. arXiv preprint arXiv:2410.24164
2024.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arX0v:22/2.068117 2022.
Qingwen Bu, Yanting Yang, Jisong Cai, Shenyuan Gao, Guanghui Ren, Maoqing Yao, Ping Luo, and Hongyang Li. Learning to act anywhere with task-centric latent actions. arXiv preprint arXiv:2502. /4420, 2025.
10

---

Published as a conference paper at ICLR 2026
Joao Carvalho, An T Le, Mark Baierl, Dorothea Koert, and Jan Peters. Motion planning diffusion: Learning and planning of robot motions with diffusion models. In 2023 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1916-1923. IEEE, 2023.
Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Car- los Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al. Pali-x: On scaling up multilingual vision and language model. arXiv preprint ArXiv:2305. 18565, 2023a.
Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18000-18010, 2023b.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
Open X-Embodiment Collaboration, Abby O'Neill, Abdul Rehman, Abhinav Gupta, Abhiram Mad- dukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khaz- atsky, Anant Rai, Anchit Gupta, Andrew Wang. Andrey Kolobov, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang, Chenfeng Xu, Cheng Chi, Chenguang Huang. Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton. Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Felipe Vieira Frujeri, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guangwen Yang, Guanzhi Wang, Hao Su, Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik Christensen, Hiroki Furuta, Homanga Bharadhwaj, Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jachyung Kim. Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jay Vakil, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang, Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang. Jordi Salvador, Joseph J. Lim, Jun- hyek Han, Kaiyuan Wang, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krish- nan Srinivasan, Kuan Fang. Kunal Pratap Singh, Kuo-Hao Zeng. Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen, Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi "Jim" Fan, Lionel Out, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip, Mingtong Zhang, Mingyu Ding, Minho Hco, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Muhammad Zubair Irshad, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Ning Liu, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R Sanketi, Patrick "Tree" Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Mart'in-Mart in, Rohan Baijal, Rosario Scalise, Rose Hendrix. Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonca, Rutav Shah, Ryan Hoque. Ryan Julian, Samuel Bustamante, Sean Kir- mani, Sergey Levine, Shan Lin, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shubham Tulsiani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subrama- nian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park. Suraj Nair, Suvir Mirchan- dani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Dar- rell, Trinity Chung, Vidhi Jain, Vikash Kumar, Vincent Vanhoucke, Vitor Guizilini, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiangyu Chen, Xiaolong Wang, Xinghao Zhu,
11

---

Published as a conference paper at ICLR 2026
Xinyang Geng, Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yansong Pang, Yao Lu, Yecheng Ja- son Ma, Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yongqiang Dou, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yue Cao, Yueh-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen Zhang, Zipeng Fu, and Zipeng Lin. Open X-Embodiment: Robotic learning datasets and RT-X mod- els. https: org abs/ /abb//211..m 08864 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch. Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied mul- timodal language model. In International Conference Machine Learning, pp. 8469-8488. PMLR, 2023.
Max Du and Shuran Song. Dynaguide: Steering diffusion polices with active dynamic guidance. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL
https: /ttp::///peeeeeeeeeeeeee ew net. /te///orumm/ d= f 8gN3
Gintare Karolina Dziugaite, Daniel M Roy, and Zoubin Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. In Proceedings of the Thirty-Fire First Con- ference Uncertainty in Artificial Intelligence, pp. 758-267,22 2 2015.
Kevin Frans, Seohong Park, Pieter Abbeel, and Sergey Levine. Diffusion guidance is controllable policy improvement operator. arXiv preprint arXiv:2505. 2025.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Zhi Hou, Tianyi Zhang, Yuwen Xiong, Haonan Duan, Hengjun Pu, Ronglei Tong, Chengyang Zhao, Xizhou Zhu, Yu Qiao, Jifeng Dai, et al. Dita: Scaling diffusion transformer for generalist vision- language-action policy. arXiv preprint arXiv:2503. 19757, 2025.
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, al. To.5 lision-laguaage-action model with open-world generalization. arXiv preprint arXiv:2504. 16054, 2025.
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. BC-z: Zero-shot task generalization with robotic imitation learning. In 5th Annual Conference on Robot Learning, 2021. URL https: httpp//ppnwn....e/e//e/e. net forum? 1d==kbp23tsGYV
https://openrevvew
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty El- lis, et al. Droid: A large-scale in wh-lhe-wild robot manipulation dataset. arXiv preprint arXiv:2403. 12945, 2024.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Open- VLA: An open-source vision-languagg--ciion model. In 8th Annual Conference Robot Learn- ing, 2024. / hwtp:eewwwwwwww.eeeeeeeeeeeeeeee fortt//oom?? 2M 6QZAE6
Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning vision-language-actinn models: Opti- mizing speed and success. arXiv preprint arXiv:2502.19645. 2025.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.61 14, 2013.
arXiv preprint
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-imas image pre- training with frozen image encoders and large language models. In International conference machine learning, pp. 19732-19742. PMLR, 2023.
12

---

Published as a conference paper at ICLR 2026
Yujia Li, Kevin Swersky, and Rich Zemel. Generative moment matching networks. In International conference on machine learning, Pp. 1718-1721 PMLR, 2015.
Zhixuan Liang Yao Mu, Yixiao Wang, Tianxing Chen, Wenqi Shao, Wei Zhan, Masayoshi Tomizuka, Ping Luo, and Mingyu Ding. Dexhanddiff: Interaction-awart diffusion planning for adaptive dexterous manipulation. In Proceedings of the Computer Vision and Pattern Recognition
Conference, pp. 1745-1755, 2025.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference Learning Repre- sentations, 2023, URL
Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky TQ Chen, David Lopez-Paz, Heli Ben-Hamu, and Itai Gat. Flow matching guide and code. arXiv preprint arXiv:2412.06266 2024.
Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt- lb: a diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.00 078644 2024.
Xingchao Liu, Chengyue Gong, and qiang liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Repre- sentations, 2023a. URL https: /ttpss///pppreviiieeee net http:/iwww.....ommrmddd//111111.
Xingchao Liu, Chengyue Gong, and qiang liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Repre- sentations, 2023b. URL htllbttss///peeeeieeee
net wett//ooommlddiiiii......
Calvin Luo. Understanding diffusion models: arXiv:2208. 11970, 2022.
A unified perspective.
arXiv preprint
Yao Mu, Tianxing Chen, Zanxin Chen, Shijia Peng, Zhiqian Lan, Zeyu Gao, Zhixuan Liang, Qiaojun Yu, Yude Zou, Mingkun Xu, al. Robotwin: Dual-arm robot benchmark with generative digital twins. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 27649- 27660, 2025.
NVIDIA. Gr0Ot nl: An open foundation model for generalist humanoid robots, 2025.
Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision-language-action models. arXiv preprint 2xXx:25501.09777, 2025.
Mathis Petrovich, Michael J Black, and Gül Varol. Action-conditioned 3d human motion synthesis with transformer vae. Proceedings of the IEEF/CVF international conference on computer vision, pp. 10985-10995, 2021.
Mustafa Shukor, Dana Aubakirova, Francesco Capuano, Pepijn Kooijmans, Steven Palma, Adil Zouitine, Michel Aractingi, Caroline Pascal, Martino Russi, Andres Marafioti, et al. Smolvla: A vision- langu--a-agttttiinn model for affordable and efficient robotics. arXiv preprint arXiv:2506.01844 2025.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing. pp. 2256-2265, pmlr, 2015.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representattonss 2021. URL https://openreviie
Corum?id=Px11
2RRHS
Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse-kai Chan, et al. Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai. arXiv preprint arXiv:2410000425 2024.
13

---

Published as a conference paper at ICLR 2026
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXfv:2405.12223, 2024.
Andrew Wagenmaker, Yunchu Zhang, Mitsuhiko Nakamoto, Seohong Park. Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine. Steering your diffusion policy with latent space reinforcement learning. In 9th Annual Conference on Robot Learning, 2025. URL net htt///ormmmiiic -or JU7AbGq3se
https://apenrev///
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint
2024.
Junjie Wen, Yichen Zhu, Jinming Li, Zhibin Tang, Chaomin Shen, and Feifei Feng. Dexvla: Vision-language model with plug-in diffusion expert for general robot control. arXiv preprint 20Xi::2502.008555, 2025a.
Junjie Wen, Yichen Zhu, Jinming Li, Minjie Zhu, Zhibin Tang, Kun Wu, Zhiyuan Xu, Ning Liu, Ran Cheng, Chaomin Shen, et al. Tinyvla: Towards fast, data-efficient saioo-language-actin models for robotic manipulation. IEEE Robotics and Automation Letters, 2025b.
Junjie Wen, Yichen Zhu, Minjie Zhu, Zhibin Tang, Jinming Li, Zhongyi Zhou, Xiaoyu Liu, Chaomin Shen, Yaxin Peng, and Feifei Feng. Diffusionvla: Scaling robot foundation models via unified diffusion and autoregression. In Forty-second International Conference on Machine Learning, 2025c.
Siyu Xu, Yunke Wang, Chenghao Xia, Dihao Zhu, Tao Huang, and Chang Xu. Vla-cache: Towards efficient vision-language-action action model via adaptive token caching in robotic manipulation. arXiv preprint arXiv:2502.002 07. 2025.
Seonghyeon Ye, Joel Jang. Byeongguk Jcon, Se June Joo, Jianwei Yang. Baolin Peng, Ajay Man- dlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, Lars Liden, Kimin Lee, Jianfeng Gao, Luke Zettlemoyer, Dieter Fox, and Minjoon Seo. Latent action pretraining from videos. Interna- tional Conference on Learning Representations, 2025.
Haoqi Yuan, Bohan Zhou, Yuhui Fu, and Zongqing Lu. Cross-embodiment dexterous grasping with reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025. URL httt:tt://ppeeeeeeettte
Rongyu Zhang, Menghang Dong, Yuan Zhang, Liang Heng, Xiaowei Chi, Gaole Dai, Li Du, Yuan Du, and Shanghang Zhang. Mole-vla: Dynamic layer-skipping vision language action model via mixture-of-layers for efficient robot manipulation. arXiv preprint arXiv:2503.20384, 2025a.
Rongyu Zhang, Menghang Dong, Yuan Zhang, Liang Heng, Xiaowei Chi, Gaole Dai, Li Du, Yuan Du, and Shanghang Zhang. Mole-vla: Dynamic layer-skipping vision language action model via mixture-of-layers for efficient robot manipulation. arXiv preprint 2rXiv:2533.20384 2025b.
Shengjia Zhao, Jiaming Song, and Stefano Emnon. Infovae: Information maximizing variational 2017.
autoencoders. arXiv preprint
Jinliang Zheng, Jianxiong Li, Dongxiu Liu, Yinan Zheng, Zhihao Wang, Zhonghong Ou, Yu Liu, Jingjing Liu, Ya-Qin Zhang, and Xianyuan Zhan. Universal actions for enhanced embodied foun- dation models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 22508 2200802222999000 2025.
Yifan Zhong, Xuchuan Huang, Ruochong Li, Ceyao Zhang, Yitao Liang, Yaodong Yang, and Yuan- pei Chen. Dexgraspvla: A visionllnnguage-cttoo framework towards general dexterous grasping. arXiv preprint arXiv:2502.20900 2025.
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pp. 2165-2183. PMLR, 2023.
14

---

Published as a conference paper at ICLR 2026
A ADDITIONAL IN-DEPTH DISCUSSIONS
A.1 ADDITIONAL RELATED WORKS
Visoon-Languaag-Acttion Models. The long-term objective of robotic manipulation is to develop general-purpose models that can adapt to diverse embodiments and tasks. Vision-LanguaeeActtion models (VLAs) is currently a mainstream approach that builds upon the perception and language understanding capabilities of pre-trained VLMs and introduces an additional action prediction mod- ule. Leveraging massive datasets of robotic demonstrations for joint pretraining, these models can process language instructions, visual observations, and proprioceptive input to generate continu- ous action sequences that interact with the physical environment to accomplish complex real-world tasks. Early representative works such as Octo and RT-1 (Brohan et al.] 2022 [Tam et al. 2024) trained Transformer-based policies from scratch on large-scale robotic datasets, while RT-2 and VLe (Zikovich et al. 2023] Kim et al., (024) further utilized pre-trained VLMs (e.g.. PaLI- X, Prismatic) (Chen et | 2023a Anschütz & Le Bras 2023) and fine-tuned them robot datasets (e.g., OXE) (Collaboration et al. 2023) via action discretization, achieving improved capabilities by inheriting the knowledge of VLMs. Subsequent methods such as RDT- IB, H-RDT, Dex VLA, DIVLA, DeX Dita (Liu et al. 2024 Bi et al., 2025; Wen et al. 2025a e Zhong et al.,
2025 Hou et al. 2025) adopt diffusion-
architectures to represent the multimodality robot data. Recently, a new generation of VLA models as To. GROOT, To.5. SmoIVLA, and
TLA (Black et al. 2024; NVIDIA 2025 Intelligence et al. 2025; Shukor et al. 2025 Wen
et al. 2025b) adopt dual system compositional architecture that incorporates generative action expert based on diffusion or flow matching (Liu et al.)
[Lipman et al.) (023) to produce contin- uous action sequences, significantly improving the prediction accuracy and physical executability. However, they mostly focus on architectures and objectives in the pre-training phase, while we aim to improve the efficiency for cross- coos-emmbddiment and cross-task adaptation pretrained VLAs.
Constructing Unified Action Space. Establishing a shared discrete token dictionary to discretize continuous robotic actions is common approach for mapping heterogeneous actions into shared space (Brohan et al.] 2022 Zitkovich et al. 2023 Kim et al. 2024 Pertsch et al.] 2025). Further- more, UniVLA (Bu et al., 2025) leveraged Vector- cector-Quantize VAE to extract task-centric latent actions from massive cross erssssembodmment videos for VLA pre-training and post-training. Similarly, UniACT (Zheng al.) 2025) also learned discrete atomic behavior codebook from large-scale robotic data to form a universal discrete latent action representation. However, these methods pri- marily focus on establishing universal latent representation for actions across different embodi- ments, failing to resolve the significant distribution discrepancy that persists between pre-training and adaptation data within the latent space. In contrast, we approach this issue from the perspective of bridging the distribution gap. Our method constructs a unified action space by first encoding the pre-training actions to form a base latent distribution, and subsequently embedding the adaptation actions into the modes of this pre-training action distribution, which results in structured latent space. While/Base et al. 2025) combined separate VAEs with contrastive learning to map different cecooddmmnn-sppecffie instances of the same action into a shared latent space, their method's 's heavy reliance on a hand motion retargeting system to generate these instances makes it challenging to apply in common VLA pre-training and post-training recipe.
More Discussions on Policy Adaptation of VLAs. Several approaches construct latent action rep- resentation for different data sources YYe et al.] 2025 Bu et al.] 2025 NVIDIA 2025). perform Kinematics retargeting or eigengrasps (Bauer et al. 2025 Yuan et al.] 2025), ) or manually design physically interpretable action space for different robot platforms (Liu al. 2024 Zheng et al.) 2025). While they address the high fine-tuning costs for embodiment and task mismatches, they can- not easily adapt to different robot platforms since the target action distribution can be significantly different from the original one. RoboOS adopts advanced architectures by building skill library and a shared action space for different embodiments, but it requires complex module construction and coupling. Other works improve the adaptation efficiency of VLAs by caching visual tokens and selectively updating task-relevant ones (Xu et al., 2025), compressing high- high-frequency action sequences into low-frequency tokens via discrete cosine transform (Pertsch et al. 2025). adopting LoRA-based updates (Wen et al. 2025b), or performing dynamic layer activation for distillation Zhang et al.) (222aa). These methods are often restricted to ruto-regressive models and overlook diffusion and fow-matching architectures. Meanwhile, solely increasing training efficiency is still limited in cross-embodiment adaptation the domain gap would be hard to bridge via direct tuning.
15

---

Published as a conference paper at ICLR 2026
A.2 RELATIONSHIP TO CFGRL
CFGRL (Frans et al. 2025) establishes a principled link between diffusion- generative model- ing and poliey improvement in RL. Specifically, CFGRL formulates the target policy ı as a product distribution:
a(a s) x #(a s) ((Aa(s))s)
where i(a s) is a reference (e.g.. behavior cloning) policy. As = Q=(s, - (s) is the advantage function under â, and f : - I2 is a non-negative, monotonically increasing function. Under this condition, Theorem in the paper proves that guarantees policy improvement: J(x) 2 J(t). To enable controllable improvement, CFGRL introduces an exponentiated family of policies parameterized by guidance weight w 2 0:
Tw (a s) #(a s) (A)
This mirrors the role of the inverse temperature 1/3 in KL-regularized RL objectives such as Advantage-Weighhe Weighted Regression (AWR), where the optimal policy takes the form <(a {) +( s) exp (Aa a). However, unlike AWR-where 3 must be fixed during training CFGRL de- couples the reference and optimality components, enabling post-hoc control of at inference time without retraining.
Sampling from Te is achieved via classifier-free guidance from diffusion or flow models. Let E {0. 1) denote an optimality label (e.g., 0 = 1 if (s a) > 0). The guidance- score (or velocity) field is:
V. log Tw (a s)
V. log (a {s) V log xaa s. s.o=l).
which corresponds to the additive composition of unconditional and optimality-condiite score functions-a direct consequence of the log-linearity of product distributions. This avoids explicit estimation of the normalizing constant in p(o 8, a), and sidesteps instability from backpropagating through learned advantage predictors.
In ATE, the latent-guidangs strategy share a common perspective imprrovng policy behavior by reshaping the action distribution via score-based guidance. However, there exists important con- ceptual distinctions between ATE and CFGRL. Specifically, CFGRL mainly performs reward-based policy improvement, where the classifier signal is derived from an advantage function A(s, a). and the guided policy provably improves expected returns. In contrast, ATE performs reward-free -free do- main alignment, where the guidance signal is defined solely from target-domain action demonstra- tions in the unified latent space and does not assume access to rewards or value functions.
But interestingly. CFGRL unlocks hidden gains in goal-conditioned behavioral cloning introduces conliconditioned behavioral cloning setting where the classifier term can be interpreted as a proxy for advantage without requiring explicit reward labels. The likelihood p(g s, a) of reaching goal from (s, a) inherently satisfies the monotonicity requirement with respect to advantage. Thus, 1 yields provably improved policy over standard GCBC, while retaining the simplicity of supervised training. We posit that under idealized assumptions, the distance metric over the unified latent space used in ATE could similarly be viewed as an equivalent proxy for advantage function. With this assumption. ATE can also be rigorously framed performing one-step policy improvement towards 'target domain optimality', even with- out explicit reward labels. Instead of conditioning on specific goal g. we condition on the Targer Domain Manifold (encapsulating the target embodiment and task). In CFGRL, the optimality vari- able implies reaching the goal g. while in ATE, the optimality variable o implies aligning with the adaptation data distribution the unified latent space.
applying CFGRL with f(A) = p(g s. and
Let the larget domain optimality p(o]s, a) be proportional to the likelihood that action be- longs to the target domain optimal action distribution. The assumption suggests this likelihood is expressed by the adaptation-speeific VAE encoder Eg. Assuming RBF kernel parameterization for this likelihood, the probability an action a matching a target demonstration a* is:
(
[F - (Pu("")l" D11²)
p(o|s,a) a) x exp
20
then the guidance term derived in CFGRL (Eq. i... f(A(s, a)) exp (-11Ep(a) (a M1?).
in [1]) becomes: Va logp(ols, a) (as)F (a
16

---

Published as a conference paper at ICLR 2026
In order to apply the guidance from such a optimality indicator, we have to it during the gen- eration process of score- based model generation process. In other words, a would be the noisy action at arbitrary diffusion timesteps k, and a' would thus be the action labels in the demon- stration trajectory from the target domain. Since our setup is actually goal-conditioned behavioral cloning setup, we always take the label from the demonstration the optimal one by default. Then, V. log p(ols, a) is exactly the guidance term 9 used in ATE (Eq. (II). Under such assumption, ATE not merely performing domain adaptation; it is mathematically equivalent performing policy improvement.
B ADDITIONAL REAL-WORLD EXPERIMENTS: TOOL USE
B.1 EXPERIMENTAL SETUP
To validate our method on the tool-use VLA challenge, we collected an additional 80 trajectories and designed the Make Yogurt Bowl task. This task requires using both shovel and spoon for precise ingredient manipulation.
Data. 80 high-quality trajectories are collected for this task, covering variations in initial object placement.
Platform. All experiments are conducted on a dual-arm RealMan robot, where each arm has 7 DoF and is equipped with Agibot OminiPicker single-DoF grippers. The left arm handles the spatula, and the right arm handles the spoon.
Training. Models are trained for 120k steps on NVIDIA A GPUs with a batch size of 12 per GPU. Execution runs at 20 Hz on single NVIDIA 4090.
Task Description. Use the right gripper to pick up the shovel, scoop out the nuts and oats, and then pour yogurt into the bowl. During the process, use the left gripper to stir with spoon.
Table 2: Success rate (%).
Task
Baseline
ATE(Ourr)
Make Yogurt Bowl
15%
25%
Figure 4: Experimental setup for the Make Yogurt Bowl task. The dual-arm robot ma- nipulates ingredients using spatula (right hand) and a spoon (left hand), demonstrating precise coordination. Key objects are high- lighted.
B.2 RESULTS.
For the Make Yogurt Bowl task, the robot must perform highly coordinated dual-arm manipulation: the right gripper picks up a spatula to scoop nuts and oats and pour yogurt into the bowl, while the left gripper continuously stirs the mixture with a spoon. This task requires precise control of multiple tools simultancously, careful timing to avoid spillage, and maintaining proper spatial relationships between ingredients and utensils. Such multi-tool coordination represents challenging test for VLAs, as small deviations can lead to task failure.
As shown in Table 2. ATE method achieves success rate of 25%, outperforming the base- line at 15%, with each method tested over 20 trials. This improvement highlights ATE's ability to leverage pre-trained latent representations and structured visuomotor priors, enabling more reliable
17

---

Published as a conference paper at ICLR 2026
#2 2
# #3
#
5
#
#1: Pick up the shovel with the right gripper. Scoop the muts using the shovel. #2: Scoop the outs using the shovel. #3: Pick up the vogurt bottle with the right gripper. #4: Pick up the spoon with the left gripper. NS: Pour the yogurt into the howl while stirring original position.
the spoon gripper. the yogurt bottle
Figure Real-world execution of the Make Yogurt Bowl task.
tool handling and fine-grained manipulation. The results indicate that ATE effectively captures the dependencies between dual-arm actions and tool interactions, allowing it to generalize across subtle variations in object positions and motions, whereas the baseline struggles with the complexity of simultancous tool use.
B.3 VISUAL DISTRACTOR GENERALIZATION
To evaluate visual distractor robustness, unexpected objects (e.g., extra fruit picces, small puzzles) randomly placed on the table. The robot must complete the task while ignoring these unexpected items. Success defined as correctly completing all phases with proper tool use. Each experiment repeated 5 times, and the results summarized in Table 3
Table 3: Success rate (%) under Visual Dis- tractors.
Task
Baseline
ATE(Ours)
Figure 6: Experimental setup of the Make Yo- gurl Bowl task under visual distractors. Ad- ditional irrelevant objects (c.g., fruit pieces) are placed on the table, while the dual-arm robot must focus on manipulating the spoon and spatula to complete the task.
Make Yogurt Bowl
0%
20%
C IMPLEMENTATION DETAILS OF ACTION LATENT ENCODERS
Inspired by prior work motion sequence VAEs PPetrovich et al. 2021 Chen al. 2023b similarly prepend two learnable tokens, Maaenn and Liokens to the input of our Transformer based action encoder. The corresponding outputs from the encoder then used to convert fixed-length
18

---

Published as a conference paper at ICLR 2026
action chunks into single, chunk-level latent representation. The decoder, built upon a Transformer decoder with cross-attentin mechanism, takes zero embeddings as a query and the latent vector as key and value, and generates reconstructed action chunk of length H.
Our two-step training scheme, introduced in Section 5.1 is designed to first learn a general latent structure and then embed actions from specific domains into it. In Step 1: General Latent Learn- ing, we train a pre-training VAE on large-scale VLA dataset to capture a general latent structure diverse robot platforms and tasks. Step 2: Domain-specifie Latent Learning, train newly initialized VAE on small sets of domain-specifie data to embed the action distribution of the target domain within the latent space learned by the pre-training VAE.
The model uses latent dimension of 512 and is trained with a mutual information term to ensure informative latent variables. A key advantage this two-stage approach its remarkable efficiency. Due to the quantity of the pre-training data, Step typically requires around 12 hours of training. However, the training in Step 2 is relatively fast, taking less than 0.5 hours. Table provides the detailed training configurations for our Info-VAE across all policies and stages.
Table 4: Info-VAB training configuration across stages and policy types.
Policy
Dataset Source
Episodes
Epochs
Step 1
DP
RoboTwin 1.0: 5 tasks
50/task
1000
RDT
Open X-Embodiment (subset), DROID, Kuka, ALOHA Open X-Bmbodiment (subset), DROID
3000
1000
Wo
3000
1000
Step 2
DP
RoboTwin 1.0: task RoboTwin 1.0: 17 tasks, ManiSkill3: tasks
50/task
200
RDT
100/task
200
Wo
RoboTwwn 1.0: 17 tasks, Real-World: 4 long-horizon tasks
50/task
200
Wo
Real- World: long-horizon horizon tasks
160/task
200
D EXPERIMENTAL SUPPLEMENT
D.1 BASELINE DETAILS
RDT-1B. RDT-1B is diffusion--aser VLA model tailored for bimanual manip- ulation. It leverages scalable transformer and a physically interpretable unified ac- tion space, enabling transfer across dif- ferent robot embodiments. In our experi- ments, it is trained and evaluated multi- task RoboTwin 1.0 and ManiSkil13 set- tings.
Table 5: Task success rates on ManiSkil13.
Task
RDT-1B
RDT-1B ATE(Ours)
Push Cube
65.2%
78.4 (1 13.2)
Pick Cube
7.6%
14.8% 7.2)
Average 36.4%
46.6 %1 10.2)
Ty- To is flow-matching based generalist policy built pre-trained vision-language models. It supports multi-task and multi-embodiment manipulation, exhibits strong instruction-folloving capability, and allows efficient adaptation with limited data. Our implementation uses the Hugging- face Lerobot Community version (https: http:s://qithbb wwwp//ugginggaccee/eocoooooo wwt.:/ffeeeee///bboeoooooo wwwobet.oboo...o across all experiments.
Diffusion Policy (DP). DP is foundational diffusion braseddio--bsedd architecture commonly used for com- parison in VLA and robotic policy benchmarks. It serves as a standard diffusion baseline to evaluate the benefits of ATE.
D.2 FAST ADAPTATION IN
Each model is trained and evaluated across a range of manipulation tasks using the specified number of demonstrations. All experiments conducted using multi-GPU training setups.
19

---

Published as a conference paper at ICLR 2026
RoboTwin 1.0 Benchmark RoboTwin 1.0 is adopted the core evaluation benchmark. As gen- erative digital twin framework tailored for dual-arm tasks. RoboTwin 1.0 integrates 3D generative models and large language models to synthesize diverse object-centric scenes along with spatially grounded robot code. total of 17 tasks are selected, spanning both single-arm and dual-arm ma- nipulation scenarios (e.g., tool adjustment, dual-bottle picking). Besides, RoboTwin 1.0 provides 17 diverse single- and dual-arm manipulation tasks that are largely absent from mainstream VLA pre- training datasets (Open X- X-Embodiment, Droid, etc.). Together, this benchmark allows to evaluate ATE under bimanual regimes, directly addressing Q1 regarding cross-task efficient adaptation.
ManiSkill3 Benchmark In addition, we use ManiSkill3, a fast, GPU-parallelized simulator de- signed for contact-rich single-arm manipulation. This simulator emphasizes contact-rich manipula- tion and offers diverse suite of single-arm tasks. We evaluate our method on 2 representative tasks such push cube and pick cube, which emphasize physical contact and fine- seee-graied control.
Simulation Training Configuration. We summarize the training settings for each policy in Table
Table 6: Training configurations for different policy models and environments. BS/GPU denotes batch size per GPU.
Policy
Benchmark
BS/GPU
GPUs
Training Duration
Dataset Size
RoboTwin 1.0 (17
16
4 A100
100k steps
100/task
RDT
tasks, multi-task)
ManiSkil13 (2 tasks,
12
2 A100
100k steps
100/task
multi-task)
To (PyTorch)
RoboTwin 1.0 (17
A100
60k steps
50/task
tasks, multi-task)
RoboTwin 1.0 (5=
128
I AA10
2k epöchs
50/aakk
DP
task pretrain)
RoboTwin 1.0 (Fine- tune)
128
1 A110
300 epochs
50/task
Diffusion Policy Evaluation. To assess the effectiveness of proposed ATE training strategy, we perform evaluations under both in-distribution and out-of-distribution settings using Diffusion Policy the backbone model:
Pretraining: base model is trained on a fixed set of RoboTwin 1.0 tasks, including block handover, empty cup place, put apple cabinet, blocks stack easy, and blocks stack hard.
Evaluation:
In-distribution setting: We fine-tune and evaluate the model on subset of the same tasks used in pretraining to assess in-distribution task adaptation.
Out-of-distribution setting: We fine-tune and evaluate the model novel RoboTwin 1.0 tasks that were not part the pretraining set, measuring generalization to unseen tasks.
Discussion: Fig. illustrates the performance of both the DP baseline and the ATE variant in in-distribution and out-of-distribution settings. Overall, our method consistently im- proves success rates and accelerates convergence, with particularly large gains on tasks where the baseline struggles. Examining the training curves 100, 200, and 300 epochs shows that our method reaches higher success rates earlier and maintains strong perfor- mance on the most challenging tasks.
This setup allows us to verify whether our ATE policy fine-tuning improves generalization and adaptation performance across both familiar and novel tasks.
20

---

Published as a conference paper at ICLR 2026
dual bottles
coblict
Siocess site (S))
25%
22%
225
126
246
Training spects
Traiiing spociss
Training spochs
Teaining spacks
(a) Out-of-distribution Tasks
(b) In-distribution Tasks
Figure 7: Evaluation of DP baseline with and without ATE across (a) Out-of-distribution tasks and (b) In- in-distribution tasks. Incorporating ATE consistently improves success rates and accelerates con- vergence, with the most significant gains observed on challenging tasks where baseline performance is low.
D.3 FAST ADAPTATION TO DUAL REALMAN ARMS IN REAL WORLD
Experimental Goals. These evaluations aim to measure adaptation efficiency across task and em- bodiment shifts (Q1) and to assess robustness to real-world perturbations such as object placement, lighting, and visual distractors (Q2).
Platform and Robot. All experiments are conducted on a dual-arm RealMan robot, where each arm has 7 DoF and is equipped with Agibot OminiPicker single- DoFF grippers. Note that mainstream VIA pretraining datasets (e.g., RDT, To) predominantly use 6-DoF single-arm robots, highlighting the increased embodiment complexity in RealMan.
Vision Setup. The visual system consists of a main overhead camera (Intel RealSense L515, RGB only, 960x540, 30 Hz, automatic exposure) and wrist-mounted cameras (Intel RealSense D405, RGB only, 960x540, 30 Hz, automatic exposure) to capture the workspace from multiple perspec- tives.
Data Collection. We collect 160 high-quality trajectories for each of the four representative long- horizon manipulation tasks (Cook Bun, Pick Bun, Make Sandwich, Use Toaster), each lasting over one minute and requiring precise bimanual coordination between arms and objects.
Discussion. Figure 3 in Section 5.2 visualizes success rates across tasks at different training steps. Our method significantly accelerates learning, achieving higher success rates with fewer training steps. For example, in Cook Bun, the success rate increases 100% in 90k steps, while the base- line only reaches 15%. In the single-arm Pick Bun task, ATE reaches 50% success at 90k steps and 70% 120k. For dual-arm tasks (Make Sandwich and Use Toaster), ATE consistently im- proves performance across all checkpoints. On average across all tasks, the baseline achieves 16.7 66.7% overall success, while ATE reaches 58.1% % at 120k steps, demonstrating substantial improvement in both final performance and convergence speed. The largest gains observed in long-horizon tasks like Pick Bun, where proper sequencing and coordination are crucial. Besides, in real-world evaluation, also observe manifest improvements in motion quality. Here, we use the Cook Bun task as a representative example, with all related 6-dimensional force data and analyses provided in Appendix H
Training. Models are trained for 120k steps using 8 NVIDIA A100 GPUs, with a batch size of 6 per GPU.
Task Setup and Success Criteria. These tasks require precise bimanual coordination, tool interac- tion, and long-horizon reasoning, posing significant challenges for existing VLA models. We design four long- horizon manipulation tasks and one tool-use task:
The right gripper grasps bun from the right plate and places it on the left plate, while the left gripper lifts the lid and places it aside. The left gripper then picks a bun from the left plate and places it into the steamer, and finally returns the lid to the steamer.
Cook Bun:
Pick Bun: The left gripper lifts the lid temporarily, grasps the bun farthest from the camera inside the steamer, and places it on the left plate. The lid is then returned to the steamer.
21

---

Published as a conference paper at ICLR 2026
Make Sandwich: The left gripper retrieves one toasted bread slice from the toaster and transfers it to the right gripper, which places it on the left plate. The right gripper then adds the patty, lettuce, and top bread slice handed from the left gripper.
Use Toaster (Tool-Use Task): The left gripper picks up the farther bread slice and passes the right gripper for toaster insertion. The remaining slice is inserted similarly, followed by pressing the toaster button. This task specifically evaluates tool-use capabilities.
(a) Cook Bun
(b) Pick Bun
(c) Make Sandwich
(d) Use Toaster
Figure 8: Key objects involved in the four representative long- fagghorizoon manipulation tasks.
Evaluation.
The evaluation is over 20 trials. Success is defined as completing the long-horizon sequence, including coordinated object manipulation and placement. Initial object positions are randomly sampled within radius of 3.5 em. For the Use Toaster task, we adopt a graded success metric: fully inserting a bread slice counts as 1.0, tilted insertion 0.25, and partial insertion as 0.5.
single NVIDIA 4090 20 Hz, and each task is evaluated
D.4 GENERALIZATION ABILITY EVALUATION ON DUAL REALMAN ARMS
This section provides the real-world generalization experiment setups follows, covering illumina- tion variations, spatial shifts, visual distractors, and human disturbances, as illustrated in Figures 9 and with the corresponding success rates summarized in Tables 7 and Overall, across all gen- eralization experiments seerimeets-iinludinng varying illumination, visual distractors, spatial shifts, human disturbances, and object morphology/clutter variations- -AT consistently outperforms the base- line. These results directly address Q2, showing that ATE enhances the robustness of the VLA against diverse real-world perturbations. The strong generalization arises because method con- strains the policy within the structured latent space learned during pre-training, which effectively preserves pre-trained visuomotor priors while allowing adaptation new environmental variations. By maintaining behavior within this latent manifold, ATE enables the policy to recover from un- expected disturbances, focus on rask-relevant features, and adapt to unseen object configurations, demonstrating reliable performance under conditions not encountered during training.
Table 7: Success rates (%) under different illumination settings (15 trials per task).
Illumination Method
Cook Bun
Pick Bun
Make Sandwich
Low
To
0%
0%
40%
wo + ATE
60%
40%
40%
0%
0%
40%
High
wo ATE
60%
40% %
60%
0%
0%
20%
Flickering
ATE
20%
0%
20%
Table 8: Success rates (%) under different generalization challenges.
Cook Bun
Pick Bun
Make Sandwich
Challenge
To ATE
To
To ATE
To ATE
πo
Visual Distractor
45 %
30%
75%
30%
50%
40%
Spatial Generalization
40 %
0%
60 %
40%
40 %
20%
Human Disturbance
5 %
0%
45
10%
50 %
40%
Varying Illumination. We alter the lighting conditions while keeping the color temperature fixed at 5800 K. The illumination levels shown in Table ATE consistently outperform the baseline, particularly under extreme or flickering illumination, where the baseline often failed. This indicates that ATE is robust to lighting intensity
low, high, and flicker around the optimal value. As
22

---

Published as a conference paper at ICLR 2026
variations, maintaining task performance even under conditions not during training. We used set of controllable studio lamps to vary lighting conditions. All lamps have a fixed color temperature of 5800 K. All illumination settings are illustrated in Figure T0] Each illumination condition (low, high, flickering) was evaluated with 5 trials per task, resulting in 15 trials per task in total.
Lamp: COB studio fill light
Low Illumination; 25 lux
High Illumination: 65 lux
Flickering Illumination: alternating between 0 and 45 lux, flicker frequency 2 Hz
Spatial Generalization. To further examine position generalization, we extend the randomized initial placement experiments beyond the original 20 trials. The objects are displaced within larger radius of approximately cm, em, with five trials conducted at each boundary position (up, down, left, right) as well at the initial location. For the Pick Bun and Cook Bun tasks, the buns were moved accordingly, while for the Make Sandwich task, the meat slices and lettuce were repositioned. shows that ATE achieved success rates of 40%, 60%, and 40% Cook Bun, Pick Bun, and Make Sandwich, respectively. These results demonstrate ATE' s ability to adapt to larger spatial shifts, highlighting efficient position generalization across diverse manipulation tasks. As illustrated in Figure 9 each of the five locations was evaluated with five trials. We first compute success rate per location, and the reported result is the mean success rate across the five locations.
Visual Distractor. We introduce irrelevant objects that are not present in the dataset, such as scattered fruits, puzzle pieces, and trays, to place on the table as visual distractions. Table shows that ATE outperformed the baseline in all tasks, with success rates of 40% for Cook Bun, 80% for Pick Bun, and 40% for Make Sandwich. This suggests that ATE effectively focuses on task-relevant objects while ignoring irrelevant clutter, demonstrating strong visual generalization. Each task was evaluated over trials.
Human Disturbance. During the task, when the robotic arm successfully grasped bun other objects, a human quickly interfered example, removing bun from the gripper or repositioning ingredients- forecingg the policy to reattempt. Table shows that ATE retained 0% success on Cook Bun, 40% on Pick Bun, and 60% on Make Sandwich, whereas the bascline largely failed. These results highlight ATE's robustness to sudden external interventions, suggesting that the policy can recover from unexpected disturbances and adapt its behavior accordingly. Each task was evaluated over 20 trials.
D.5 ABLATION STUDY
To further analyze the impact of the aligned and structured action latent space, we conduct a detailed ablation study comparing three adaptation strategies:
1. Direct fine-tuning: The original policy (No or Diffusion Policy) is fine-tuned directly on task-specific adaptation data without using ATE.
2. ATE (Single-step Info-VAE): Adaptation is performed using ATE with single-step Info- VAE trained directly on task-specific data, removing the two-step procedure.
3. ATE (Two-step Info- VAE): Full adaptation using our proposed ATE to fine-tune the policy.
Results
Figure
presents the comparison across these three strategies. The results show that:
Direct fine-tuning achieves lowest success rates.
ATE with single-step Info-VAB performs similarly to direct fine-tuning, with minor im- provements, indicating that merely using ATE without the structured latent space provides limited benefits.
ATE with two-step Info-VAE consistently outperforms both alternatives, achieving higher final success rates. The two-stage latent space effectively reduces the domain gap while preserving pre- pre-trained visuomotor priors, enabling more efficient and robust cross-task and cross-embodiment adaptation.
23

---

Published as a conference paper at ICLR 2026
Cook Bun
Make Sandwich
Pick Bun
Spatial Generalization (Wih radias opgrazimately
Visual Distractor (Unergected Objects
Human Disturbance (CRject repfassment
Figure 9: Real-world generalization settings. For the Spatial Generalization experiment, vary the initial positions of manipulated objects. For the Visual Distractors experiment, we randomly place unseen items (c.g., fruits) the scene. For the human disturbance experiment, we intervene key execution points by resetting already completed sub-tasks.
25
45
65
Figure 10: Illustration of the illumination setup under three conditions: low (25 lux), high (65 lux), and flickering (alternating between O and lux at 2 Hz).
D.6 OVERALL DISCUSSION
Our simulation and real-world experiments consistently demonstrate that ATE facilitates efficient cross-task and crossemmbod adaptation. Across both the RoboTwin 1.0 and ManiSkill3 bench- marks, our framework improves final success rates, accelerates convergence, and enhances sample efficiency compared to haseline policies. Real-world evaluations the dual-arm RealMan robot
24

---

Published as a conference paper at ICLR 2026
Policy
Figure 11: Ablation study on the effect of two-step Info- VAE training. We compare the proposed two-step Info- VAE with a single-step VAE trained only on task-specific datasets, using both mo and Diffusion Policy backbones. For the Diffusion Policy results, we evaluate both in-distribution (f) and anttof-distribuutinn generalization settings ($).
further confirm these advantages, with ATE achieving higher success rates, smoother trajectories, better long-horizon coordination, and stronger generalization.
These results indicate that our approach effectively addresses Q1 by leveraging unified action latent space and latent-space guidance. This combination allows the policy to rapidly adapt to new tasks and embodiments while preserving valuable pre-trained visuomotor knowledge, significantly enhancing the practicality and deployment of pre-trained VLA models.
E TRAINING HYPERPARAMETERS
We summarize the key hyperparameter settings used for both stages of our method, including (i) Learning the Unified Action Latent Space, and (ii) Steering Efficient Adaptation with Latent Guid- ance.
Table 9: Hyperparameters for Info-VAE pretraining,
Hyperparameter
Value
Latent dimension
512 (RDT), 50 (mo), 14 (DP)
Temporal input length Input dimensionality Optimizer
128 (RDT), 32 (mo). 14 (DP)
Adam
Learning rate
1 x 10-4
Weight decay
1 x 10
Batch size
Table 10: Task-specific fine-tuning (RoboTwin).
Hyperparramter
RDT
TT
DP
Chunk size
64
50
8
Batch size
64
24
128
Learning rate
1 1.0e-4
2.5e-5
1.0c-4
Training duration
100k steps
60k steps
300 epochs
Other platforms. fine-tuning settings:
For ManiSkill and Real Robot experiments, the following task-specific
ManiSkill (simulator): Chunk size 64 (RDT), 24 batch size, learning rate 1.0c-4, training duration 100k steps.
Real Robot (physical): Chunk size 20 (mo), batch size 48, learning rate 2.5e-5, training duration 120k steps.
25

---

Published as a conference paper at ICLR 2026
F ALGORITHMIC DESCRIPTION OF ATE
Algorithm Stage I: Learning the Unified Action Latent Space
Input:
Pre-training dataset-Dpretni pretriin- Adaptation dataset D Dddatationn Pre-training action VAE 6, Adaptation action VAE vi, Number of epochs N
// Learn the pre-training action VAE
for epoch = to do
Sample batch of actions (()
(R from Ppretrain
Update pre-training action VAE á via maximizing ((Ø; D Prettain) in Eq. 44)
end for
// Calculate the pre-training action latent distribution Initialize the running mean μ4 and variance Eg of the pre-trained action latent distribution 4o (=)
for all possible sample a4;-#H1 in do
Encode the latent the action chunk a via
Update the running mean p4B and variance Es end for
Learn the adaptation action VAE
for epoch = 1 to N do
(i) Sample batch of actions (At++L-IJe from addapatioon
Update pre-training action VAE 3 via maximizing L(y;; D Padaptation の) in Eq. end for
Algorithm Stage 2: Steering Adaptation Flow-based VLAs with Latent Guidance
Input: Flow-based VLA vg, Adaptation dataset addaptation Adaptation action encoder Eg,, Guid- ance scale A, Learning rate r)
repeat
Sample Ennervvaioon-lanuagg--teee Sample NN(0,) I), 44[0,, 1])
pair (1:tta11)+)11) PDiajenn (-)
Corrupt clean action chunk a = Tat++L-1 + (1 T)E
|/ Compute the guidance
Get the latents = E.g (1))+ Ep of noisy action chunk and clean action
chunk
Obtain the latent guidance g = -V -f 1|z,,,11=~0000 -
// Steer the tunn-ttniing
Get the ground truth velocity v, -
Update the VLA via 0 = 0 - until Converged
+ - 1
Algorithm 3 Stage 2: Steering Adaptation of Diffusion-based VLAs with Latent Guidance
Input: Diffusion-based VLA Adaptation dataset D adaptations Adaptation action encoder E Guidance scale X, Learning rate
repeat
Sample an observation
observationn-anguu---ccco language-action pair (o,, 1, a1:++-11) ~ PDasuestios (.)
Sample (O, I), k ~ ((,,,,, T})
Corrupt clean action chunk a = Vāganitt-- + VI age
Compute the guidance
Get the latents é E, (a (af.4t-1), Ep(an.tt-1) of noisy action chunk and clean action
chunk
Obtain the latent guidance g - - nt 2112
Steer the fine-tuning
Get the ground truth velocity v; -
Update the VLA via 0 - 1) / until Converged
/1- -aadg] - ell2
26

---

Published as a conference paper at ICLR 2026
Parameter
Value (%)
DoF
8 x 2 = 16
Arm weight
7.9 15.8 kg
Arm Payload
5000g
Arm reach
638.5mm
Arm Repeatability
0.05mm
Joint motion range
J1 178 + 130%,J3 178", J4 135%, + 178", J6 = 128", J7 360
Gripper range
120mm
Gripper weight
0.43kg
Gripper Payload
1.5kg
Gripper max force
30N
Table 11: Technical specifications. In this table, the arm refers to the Realman RM75-66 model, while gripper refers to the Ominipicker developed by AGIBOT.
G
HARDWARE DETAILS
We present the dual-arm platform constructed and utilized in this work. Two identical Realman RM75-6s robotic arms are mounted on a white lifting desk using four G-type fixtures. At their end-effectors, we fixed the OminiPicker grippers developed by AGIBOT together with RealSense- D405 short-range depth cameras. Both grippers communicate with the workstation through USB-to- serial protocols. In addition to the two cameras mounted the robotic arm end-effectors, also strategically positioned one RealSense-LL51 high-precision depth camera on tripod, positioned appropriately behind the arms to capture complementary global view. All three cameras di- rectly connected to our workstation via USB 3.2 interfaces. The two robotic arms are connected the same local area network as the workstation and communicate using the Ethernet-based SDK provided by Realman. An overview of the hardware setup is illustrated in Figure [2] while the corresponding hardware specifications are listed Table II for reference.
Exterior Camora
Right Robotic
Left
Robatic
Arm
Right D405
Left
Camern I405
Right Kripper
Left
Gripper
Figure 12: Hardware feature. The wrist-mounted camera the RealSense D405 model, with optimal operating range of 7cm 50cm, while the exterior camera is the RealSense L515 model, with optimal operating range of 0.25mmmmm55m 9m. The robotic of exactly the same model rather than mirror versions.
27

---

Published as a conference paper at ICLR 2026
H THE HIDDEN ADVANTAGE OF ATE: SAFER AND SMOOTHER MANIPULATION
Rakan Ga
eserer
(a) 6-axis force data during single Cook Bun task using the direct fine-tuned Tu model.
(b) 6-axis force data during single Cook Bun task using the model fine-tunce with ATE.
Figure 13: Six-axis force comparison during the Cook Bun task. (a) Using the direct fine-tuned wo model, the force signals exhibit instability and less structured patterns. (b) Using the mo model fine- tuned with ATE, the force signals are smoother and more consistent, indicating improved robustness and better adaptation the target embodiment.
While our primary objective was to facilitate data-efficient adaptation by bridging the action dis- tribution gap, our experiments revealed valuable and unanticipated benefit: superior motion smoothness and more stable force control. We validate this finding through detailed analysis of six-axis force measurements. The term six-axis force refers to forces and torques expressed in the coordinate frame rigidly attached to the target object: the three force components along the pos- itive directions of the coordinate axes, while the remaining three correspond to torques around these axes.
When performing the cook bun task with the direct fine-tumed policy To. we observed that the robot arm frequently exerts excessive force while placing the steam lid back on the steamer, which often led to deformation of the steamer. By contrast, when the same action was executed by the fine-tuned policy mo with ATE, the deformation of the steamer significantly mitigated.
To quantitatively substantiate this observation, we used the built-in six-axis force sensor integrated into the robot's end-effector to collect data during task execution. As shown in Figure 13a and Figure [3b [3b we selected the force component along the x-axis as the primary evaluation criterion. This is because we confirmed that, when placing the lid, the force transmitted from the lid to the gripper is aligned with the x-axis direction. The data presented in the figure are relative measures, obtained by subtracting the sensor readings at the initial pose from the raw sensor values.
From the figures, it can be observed that the force curve of ATE exhibits fewer sharp fluctuations compared that of the baseline and remains at a smaller overall magnitude. More specifically, across four representative peaks and valleys, the baseline policy demonstrates tendency to apply excessive and unnecessary downward force during the bun-grasping and lid-placing actions.
This finding suggests that by constraining the fine-tuning process within the pre-training action latent space, our method implicitly guides the policy toward safer and more robust physical interactions.
I REAL-WORLD DATA COLLECTION DETAILS
In this section, we provide a detailed description of how dataset is collected. As illustrated in Figure 14b] our real-world tasks involve bimanual simultancous actions. we were to rely on traditional single-operator teleoperation, accomplishing such tasks would impose excessive demands on the operator. To address this challenge, we adopt a dual-operator collaborative teleoperation approach for data collection.
28

---

Published as a conference paper at ICLR 2026
LAN
Robotic
L Robotic
Arm
Arm
Lef
Hand
Right Hand
(a) Dual-arm collaborative teleoperation.
(b) Bimanual task involves simultaneous actions.
Figure 14: The setup of real-robot data collection. (a) The overview how we implement dual-arm data Collection. During teleoperation, the operator holds Joycon with the hand which not cared about by our program. (b) example that requires both hands perform logically disparate tasks, simultaneously occurs in the use toaster task.
In the dual-operator teleoperation setup, each operator is first equipped with a separate Vision Pro headset, both of which are connected to the same local area network. We then employ the open- source project TrackingStreammr to extract sand-keypoint estimations from the Vision Pro devices and stream them to workstation. Note that in the wrist pose estimated by Vision Pro, the attached coordinate frame on the wrist differs in orientation from the coordinate frame defined on the Real- robotic arm. To ensure consistency, we apply right-multiplication with predefined rotation matrix T Tretation to the wrist frame, so that when both the wrist and the robotic arm are in their initial poses, the directions of their coordinate frames are aligned.
After both operators place their corresponding hands (which control the robotic arms) in the initial positions and press the initialization button, the program immediately records the wrist estimation from Vision Pro at that moment, and treats it as the initialization matrix for the current data collection session. Next, we assume the existence of a transformation matrix, such that left-multiplying the initialization matrix with mimsfoom maps it to the robotic arm coordinate system, perfectly aligning with the initial pose of the end-effector By right- gightmuutiplyiig both sides of this equation with the inverse of the initialization matrix, we can solve for T transform
Trrnnfoorm Tei (T Viiisoon.iit init rotation
where The init denotes the initial pose of the end effector in the robot coordinate system, and T init denotes the wrist pose estimated by Vision Pro at the time of initialization. For every subsequent step, Vision Pro's 'ss wrist estimation first multiplied by the right Totttion and then by the left T Trnnsfomm- This way, the motion and orientation of the hand can be directly mapped into the robotic arm ordinate system, representing the desired end effector pose. By employing this mapping approach for teleoperation, it is only necessary to know the initial poses of the robotic pose the interface to be used as the teleoperation reference. Moreover, under the requirements of our experiments, this method enables a clear separation of the two different robotic arms, thereby facilitating the realization of dual operator teleoperation. Next, we utilized the Pinocchio library to solve the inverse kinematics to get the joint angles, which are then transmitted directly the robot for execution. Beyond using Vision Pro as our primary sensing hardware, we integrate Nintendo Joy-Con devices: operators can use the Joy-Cons to control the gripper's 's open and close actions, thereby completely eliminating potential misrecognition issues since we use vision-based solutions. Additionally, Joy-Cons serve as auxiliary controllers to notify the operator of code errors, quickly set the robot arms, and confirm dataset recording, among other functions.
and the initial
The advantages of employing dual operator teleoperation can outlined as follows. First, tasks re- quiring both arms to perform different actions simultaneously can be executed effortlessly, enabling the policy which tne-tumed on our dataset to demonstrate superhuman multitasking capabilities. Second, when one arm is idle, its wrist-mounted camera be aimed at the other working arm, thereby providing three distinct visual perspectives for the active robotic arm. Third, operators are granted greater freedom of movement: they walk around to achieve larger control ranges for
29

---

Published as a conference paper at ICLR 2026
their respective arms without worrying about the state of the other arm. Furthermore, since the two operators' hands are spatially separated, the visual occlusion problem in HandOver-type tasks is naturally resolved. Finally, each of the operator only needs to master half of the skill set, which accelerates skill transfer when switching to new tasks and reduces operator workload.
ADDITIONAL EXPERIMENTS
J.1 ADDITIONAL EXPERIMENTS: SAMPLE EFFICIENCY UNDER VARYING TUNINGG DATASET SCALES
To explicitly evaluate the sample efficiency of our method, compared the success rates of the backbone with and without ATE on the RoboTTii 1.0 benchmark across varying fine-tuning dataset sizes: 10, 25, and 50 episodes per task. The results summarized in Table 12
Comparison of sample efficiency on RoboTwin 1.0 tasks. ATE demonstrates significant improvements in low-data regimes. Notably, with 25 episodes, ATE achieves a 3.15x improvement in average success rate (0.290 vs. 0.092).
Table 12:
episodes / task
25 episodes task
50 episodes task
Task
750 ATE
wo
ATE
To
ATE
Put Apple Cabinet
0%
0%(t
2%
27 %6ct 25)
34%
55 %
Blocks Stack (Easy)
0%
1%c(
5%
6 %
30%
50 %e(t
Diverse Boutles Pick
7%
7%(1
4%
% 18)
20%
40 %
Dual Bottles Pick (Fasy)
5%
14% 9)
21%
9 38)
48%
85 45%% 37)
Block Handover
1%
14 %1 13)
31%
31 % 0)
80%
92 (2
Average
2.6%
4.6 96ct 2>
9.2%
29.0 29.0% 19.89
42.4%
64.4% 22)
direct fine-tuning of mo struggles significantly in data-scarce regimes. No- tably, with only 25 episodes, the baseline fails acquire meaningful behaviors for complex tasks (e.g., sg. 2% on Put Apple Cabinet), whereas ATE achieves 27% success rate. Furthermore, ATE with 25 episodes often matches exceeds the performance of the baseline trained with 50 episodes (e.g., 0.59 vs. 0.48 on Dual Bottles Pick Easy), suggesting an approximate 2 improvement in sample efficiency.
As shown Table 12
J.2 ADDITIONAL EXPERIMENTS: EFFECTS OF VAE GUIDANCE SCALE
The guidance scale À in Eq. and Eq. I0) controls the strength of the steering signal during the fine-tuning process. We conducted a sensitivity analysis on To backbone to identify the optimal operating range.
Table 13: Ablation study on the guidance scale parameter RoboTwin tasks.
Evaluation performed on selected
Task
(Baseline)
A=1 1
À = 2 (Ours)
== 3
Mug Hanging (Easy)
11 %
21%ct
27 %
16%(t
Tool Adjust
70
58%CL 12)
69%(L D
65%(L 5)
Shoe Place
52%
56%6
58% (1
56%(t
Dual Bottles Pick (Easy)
48%
82%6t 34)
85 45 37)
84%ett 36)
Table 13 indicates that generally yields the best performance. While stronger guidance (A =
3) can enforee the target distribution more aggressively, it may occasionally disrupt the smoothness of the action chunk generation, leading to slightly degraded performance compared to the optimal setting.
30

---

Published as a conference paper at ICLR 2026
J.3 ADDITIONAL EXPERIMENTS: EFFECTS OF REVERSE KL WEIGHT
The parameter a governs the weight of the reverse KI. divergence term in the adaptation VAE ohjec- tive in Eq. ( which determines the strength of the mode-seeking behavior during latent alignment.
Table 14: Ablation study on the reverse KL penalty weight a. Evaluation performed on selected Tobo tasks.
Task
(Baseline)
ar = 0.1
a 1 (Ours)
= 10
Blocks Stack Fasy (Fasy)
30%
35%
50 96 20)
26%((144
Diverse Bottles Pick
20%
20%ct
40 10 20)
10%(, 10)
Block Handover
80%
97% 17)
92%ct 12)
37%(L
Dual Bottles Pick (Easy)
48%
60%6( 12)
85 (5 37)
43%6(1.5)
As observed in Table 14 an appropriately tuned a (a 1) is crucial. When a is too large (a 10), the adaptation VAE becomes over-constrained to the pre-training prior, losing the expressivity required to represent the target embodiments 's specific actions. Conversely, small a (a = 0.1) results in weak alignment, making the guidance signal less effective.
J.4 ADDITIONAL EXPERIMENTS: ABLATION STUDY ON VAE CHOICES
Our steering module is not constrained to InfoVAE; designed as plug-and-play guidance mechanism that can operate any latent space with meaningful structure.
To verify this generality, we replaced InfoVAE with a vanilla VAE while keeping our alignment- learning stage and steering module unchanged. The results summarized as Table and Table I6
Table 15: Comparison of different latent structures for guiding Diffusion Policy. Info VAE consis-
tently outperforms Vanilla VAE due to better structured latent spaces.
Method
Task
DP (Baseline)
Vanilla VAE
InfoVAE (Ours)
Dual Bottles Pick (Rasy)
32%
39% (1
43 % II)
Shoe Place
14%
18% (1
22%
Put Apple Cabinet
29%
42% (1 13)
46 % 17)
Empty Cup Place
22%
34% (1 12)
37 % 15)
Table 16: Comparison of different latent structures for guiding Tn- InfoVAE consistently outper- forms Vanilla VAE due to better structured latent spaces.
Method
Task
(Baseline)
+ Vanilla VAE
+ Info VAE (Ours)
Blocks Stack (Easy)
30%
31%
50%ct
Diverse Bottles Pick
20%
36%ct 16)
40 10%t 20)
Dual Bottles Pick (Easy)
48%
81 9b
85 (t
Block Handover
80%
94% 14)
92 %1 12)
These findings confirm three important properties of our design:
The steering module consistently improves the success rate even when paired with a standard VAE, demonstrating that directly applies other butent-based methods without architectural changes.
General compatibility:
Why InfoVAE performs better: Standard VAEs commonly suffer from inaccurate amortized latent inference, leading to poorly structured latent representations. This effect prevents reliable cross-domain latent alignment, which is essential for correct steering in the target domain. In- constraints, forming a
fo VAE alleviates this problem by enforcing stronger more structural latent structure. Thus, steering becomes stable and effective.
31

---

Published as a conference paper at ICLR 2026
Modularity design choice: The key contribution of approach lies in factorizing latent space alignment and downstream steering. Better latent models naturally bring larger gains, mak- ing our approach adoptable with stronger latent learning methods. Lattenn Space Vissstned
Latern Space vissetinaiion
Driee
TIAE MMCoomennn
(a) t-SNE of latent representations with VAE.
(b) t-SNE of latent representations with Vanilla VAE.
Latent Space Visualization. We project the latent embeddings of both pre-training (green) and adaptation (blue) datasets using t-SNE. Compared to Vanilla VAE (right), VAE (left) produces a significantly more compact and structured latent space, where the adaptation data distribution tightly aligned with the high-density modes of the pre-training distribution. This alignment is critical for effective cross-domain knowledge transfer.
Figure 15:
To further qualitatively validate these claims, we visualized the latent distributions of both pre- training and adaptation datasets using t-SNE, shown in Figure [5] The visualization clearly demonstrates that InfoVAR produces a compact, well-structured latent manifold where the adapta- tion data (blue points) are embedded closely within the modes of the pre-training data (green points). In contrast, the Vanilla VAE results in a loosely aligned latent space with noticeable gaps between the two distributions. This visual evidence supports our quantitative results, confirming that the superior performance of InfoVAE VAE stems from its ability to enforce better distributional alignment, which is a prerequisite for effective steering during fine-tuning.
J.5 ADDITIONAL EXPERIMENTS: COMPARISON AGAINST OTHER BASELINES
We compared ATE with two related guidance- suisance-based methods: DSRL WWagemmaerr et al. 2025 (an RL-based approach) and DynaGuide (Du Song 2025) test-time steering approach).
We evaluated the methods on the LIBERO-10 task suite, To adapt DSRL for this setting, we implemented it as an offline RL. method using the adaptation dataset as static buffer with sparse rewards for successful trajectories.
Comparison with Offline DSRL.
Table 17: Comparison with Offline DSRL on the LIBERO-10 benchmark. DSRL degrades perfor- mance in this setting, whereas ATE improves
Benchmark
Directly Fine-tuned ma
Offline DSRL
ATE (Ours)
LIBERO
0.78
0.43
0.88
As shown in Table 17 DSRL significantly degrades performance (0.78 - 0.43). This confirms that RL-based objectives be unstable when learning from limited expert data without online interaction, suffering from the duitof-distriuution issue during value estimation. In contrast, ATE's augmented objective preserves the nature of supervised learning and thus robustly improves perfor- mance (0.88). Comparison with DunaG. We compared Diffusion Policy (DP) baselines against DynaGuide and ATE on three tasks. For a fair comparison, we modified DynaGuide to support multi-view inputs.
32

---

Published as a conference paper at ICLR 2026
Table 18: Comparison with DynaCuide specific tasks. and is more stable.
ATE provides consistent improvements
Task (DP Backbone)
Base Policy
DynaGuide
ATE (Ours)
Put Apple Cabinet
0.29
0,46
0.46
Empty Cup Place
0.22
0.30
0.37
Dual Bottles Pick Easy
0.32
0.18
0.43
Table 18 shows that while DynaGuide improves performance on some tasks, exhibits instabil- ity on others (e.g., degrading performance on Dual Bottles Pick). Furthermore, we observed that DynaGuide severely impacts inference speed due iterative gradient calculation during generation, making it less suitable for real-time control compared to ATE, which incurs zero inference overhead.
K LLM USAGE STATEMENT
In accordance with ICLR 2026 policy, we report that large language model (LLM) was used to aid and polish the writing of this manuscript. The authors reviewed all LLM-ggnerated suggestions and take full responsibility for the final content of the paper.
33


Reviews:
[{"review_id": "QQjVQkajpP", "reviewer": "Reviewer_Udtp", "summary": "The paper proposes ATE, a plug-and-play adaptation framework for vision-language-action (VLA) models. Stage 1 “Align” learns a unified action latent space by training an InfoVAE on pre-training actions and a second InfoVAE on target-domain actions with a reverse-KL regularizer toward the pre-training latent distribution. Stage 2 “Steer” adds a training-time guidance term that pushes generated action chunks toward target-domain actions by penalizing the squared distance between their latent encodings. The method claims minimal architectural changes and low overhead.\nThe method claims minimal architectural changes and low overhead. It is evaluated on RoboTwin 1.0 (17 tasks), ManiSkill3 (2 tasks), and a real dual-arm RealMan platform (4 tasks + 1 tool-use). \nIt achieves 9.8% and 32% success rate improvement on Simulation and real-world setting, respectively. There is also a qualitative claim of smoother, safer motions (force plots).", "questions": "Misc:\nMissing Appendix ref in L410", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Tackles an important and practical problem: efficient adaptation of pretrained VLAs across embodiments and tasks.\n- Simple, model-agnostic recipe that integrates with both diffusion and flow matching, with no architectural changes.\n- Clear implementation description and algorithm boxes; the method is plausibly easy to reproduce and deploy.\n- Writing is good and the paper is easy to follow.\n- The improvements look good, and it seems like a good RL alternative.", "weaknesses": "- I feel like the proposed method is more like a self-distillation method which shares similar objective with RL. I wonder whether it can compete with existing offline RL methods. Now the model did not show any other baseline methods besides the model it based on so it is hard to tell.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dfasg4DDq5", "reviewer": "Reviewer_tPwB", "summary": "This paper proposes two-step framework, namely ATE, on efficient robot learning on robot manipulation tasks. ATE consists of two sub-modules: (i) a dual-VAE module for robot action chunk pre-training and downstream action chunk adaptation, where the embedding space of the latter one is trained to align with the pre-trained VAE. (ii) a steering module for diffusion- and flow-based VLA models. ATE is a plug-and-play framework and could be directly applied to different VLA models. Simulation and real-world experiments demonstrate consistent performance gain w.r.t diffusion- and flow-based models.", "questions": "Questions:\n1.\tIt seems that there lacks necessary (and straightforward) illustration of how diffusion- and flow-based methods like pi_0 work, and how ATE is trained/integrated with these models. \n2.\tWhat’s the necessity of training two VAEs? Once the first VAE is pre-trained using large-scale datasets like OpenXE, it could theoretically transfer and fit-in to a smaller downstream dataset. The motivation could be more clearly illustrated (even though it is mentioned at line 254).\n3.\tIs it possible to directly apply the steering module to other latent-based methods (like Q.2)? Or, is it strongly tied to work with the proposed InfoVAE?\nSuggestions:\n1.\tFont size of Figure 1 and Figure 2 is too small to clearly recognize the text.\n2.\tLine 410: missing reference\n3.\tLine 457: ATE in inconsistent font.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method, including the dual VAE and steer module is theoretically sound.\n2. The size of ATE is relatively small w.r.t base models like pi_0, yet it brings consistent performance gain over the base models.\n3. Extensive experiments make ATE technically solid.", "weaknesses": "I found this paper difficult to follow. Please refer to the questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6XLsEBV5xc", "reviewer": "Reviewer_XTcT", "summary": "Align then Steer (ATE) provides a sample-efficient manner to adapt into a new embodiment for pretrained Vision-Language-Actions Models. ATE first trains a VAE that learns a latent space of all actions in the pretraining dataset, $z$, and uses it as a reference for the adaptation dataset. Afterwards, the method then trains another VAE to learn a separate latent for the adaptation dataset. The novelty came from aligning the two distributions together using a reverse KL penalty. Afterwards, for effective policy adaptation, one can use classifier guidance as an additional objective in the adaptation loss.\n\nVia both real-world and simulated benchmarks, ATE achieved considerable improvements in performance across dexterous tasks across various forms of disturbances and difficulties as well as faster convergence.", "questions": "1. How does ATE compare to other guidance methods such as DSRL or DynaGuide (although their formulations are different)?\n2. Can authors also discuss how the sample efficiency of this method stack up compared to the baseline methods across various finetuning set sizes? I believe this can strengthen the claim that ATE allows effective finetuning without large scale data.\n\nMinor questions and remarks:\n\n1. On page 8 (line 410), there is a missing link to the appendix. \n2. The paper is pretty dense in formulae and notations (especially in the preliminary and method section). It would be better to move some of them to the appendix for presentation.\n3. In eq. 5, $\\mathcal{D}$ is not defined (as compared to $\\mathcal{D}_\\text{pretrain}$. What does it stand for?\n4. On line 266, “we introduce the classifier guidance” should have “the” removed.\n5. In figure 3, I think there’s a mismatch between the figure’s numbering system (6w, which I presumed to be 60k) and the description. It would be better to check these typos and correct them as well in the review period.\n\nReferences: \n\n[1] Liu, B. et al., 2023. “LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning”. NeurIPS\n\n[2] Wagenmaker, A. et al., 2025. “Steering Your Diffusion Policy with Latent Space Reinforcement Learning”. CoRL.\n\n[3] Du, M. et al., 2025. “DynaGuide: Steering Diffusion Polices with Active Dynamic Guidance”. ArXiv.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors conducted an extensive suite of real-world and simulated experiments, which does bolster the claim that the paper is making. I like that the author made a lot of validation on the robustness of the experiments by checking ATE against various forms of environment perturbations.\n2. I believe that this type of alignment procedure in the finetuning stage is also doing something similar to that of RL. I believe that you can follow the same line of thinking proposed in CFGRL [1] and show that this form of classifier guidance in finetuning is a form of policy improvement. \n\nReferences:\n\n[1] Frans, K. et al., 2025. “Diffusion Guidance Is a Controllable Policy Improvement Operator”. ArXiv.", "weaknesses": "1. While the paper shows considerable improvement in aggregate, each task is only evaluated with 5 trials (correct me if I’m wrong). If possible, it is better to use a larger number of trials per task (10-20) so that there is less variance in your experiment procedure. \n2. The method section is pretty dense in notation. I believe that the paper can be reformatted for better readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1liwoDbcrB", "reviewer": "Reviewer_RFTf", "summary": "This paper presents Align-Then-Steer (ATE), a framework for adapting pre-trained Vision-Language-Action (VLA) models to new embodiments and tasks.\nATE addresses the action-space mismatch problem when transferring VLAs across robots or manipulation domains. It consists of two stages:\n(1) Align – construct a unified latent action space by aligning source and target action distributions using a dual-VAE with reverse KL regularization;\n(2) Steer – apply latent guidance to direct diffusion or flow-based VLA generation toward the aligned latent manifold during fine-tuning.\nThis approach enhances adaptation efficiency, preserves pre-trained capabilities, and achieves strong results in both simulation and real-world settings.", "questions": "1. I recently came across latent action learning in a survey paper [1]. How does the latent action learning proposed in your work differ from that?\n\n2. This question is more discussion-oriented — I also believe latent learning is a promising approach for cross-embodiment transfer. What other potential solutions do you foresee in the future? And compared to [2], what advantages does your method offer?\n\n3. If there is a large gap between pre-training and fine-tuning data, will this alignment still be effective? How robust is the latent alignment when the target robot has a significantly different kinematic structure or action semantics?\n\n4. How does the reverse KL constraint affect convergence? Did you observe mode collapse or over-constrained behavior during training?\n\n5. What is the model scale, and can it run in real time for control tasks?\n\n6. Are there visualizations (e.g., t-SNE plots) showing how the latent distributions align before and after training?\n\n7. Possibly due to the page limit, some ablation studies are placed in the appendix. However, this is not ideal — key experimental results should be presented in the main text.\n\n[1] Towards a Unified Understanding of Robot Manipulation: A Comprehensive Survey. arXiv 2025.\n\n[2] villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models. arXiv 2025.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is clear and well grounded.\n\n2. The method shows significant improvements in adaptation success rates.\n\n3. The innovation is moderate, but the approach is scalable and extensible.\n\n4. The paper is well organized and easy to follow, with clear figures and mathematical formulations.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)