Hybrid Training for Vision-Language-Action Models
Authors: Pietro Mazzaglia
Cansu Sancaktar
Markus Peschl
Daniel Dijkman
Venue: ICLR 2026 Poster (2026)
Keywords: vision-language-action models; chain-of-thought; robotic manipulation
Primary area: applications to robotics, autonomy, planning
TL;DR: Hybrid Training enables Vision-Language-Action models to learn from chain-of-thought reasoning traces, achieving high performance, while maintaining fast inference, by allowing direct action prediction at test time.

Abstract: Using Large Language Models to produce intermediate thoughts, a.k.a. Chain-of-thought (CoT), before providing an answer has been a successful recipe for solving complex language tasks. In robotics, similar embodied CoT strategies, generating thoughts before actions, have also been shown to lead to improved performance when using Vision-Language-Action models (VLAs). 
As these techniques increase the length of the model's generated outputs to include the thoughts, the inference time is negatively affected. Delaying an agent's actions in real-world executions, as in robotic manipulation settings, strongly affects the usability of a method, as tasks require long sequences of actions. 
However, is the generation of long chains-of-thought a strong prerequisite for achieving performance improvements? In this work, we explore the idea of Hybrid Training (HyT), a framework that enables VLAs to learn from thoughts and benefit from the associated performance gains, while  enabling the possibility to leave out CoT generation during inference. Furthermore, by learning to conditionally predict a diverse set of outputs, HyT supports flexibility at inference time, enabling the model to either predict actions directly, generate thoughts or follow instructions. We evaluate the proposed method in a series of simulated benchmarks and real-world experiments.

HYBRID TRAINING FOR VISIO-LLANUUGGE-CCTION MODELS
Pictro Mazzaglia
Cansu Sancaktar"
Qualcomm Al Research"
University of Tübingen, Max Planck Institute for Intelligent Systems cansu. Lancaktar@tus mpg de
pmazzagl& aateaaglllggiiio. www.guaaaccooo......o....... comm. com
Markus Peschl
Daniel Dijkman
Qualcomm AI Research
Qualcomm AI Research
mpeschleqti. qua..omm..ommmmmmmmmm com
ddi jkman qua comm. com
ABSTRACT
Using Large Language Models to produce intermediate thoughts, a.k.a. Chain-of- thought (CoT), before providing an answer has been successful recipe for solv- ing complex language tasks. In robotics, similar embodied CoT strategies, gener- ating thoughts before actions, have also been shown lead improved perfor- mance when using Vision- LangunLe-agagaaace Action models (VLAs). As these techniques increase the length of the model's generated outputs include the thoughts, the inference time is negatively affected. Delaying an agent's ' actions in real-world executions, as in robotic manipulation settings, strongly affects the usability of a method, as tasks require long sequences of actions. However, is the generation of long chains-of- of-thughtt a strong prerequisite for achieving performance improve- ments? In this work, explore the idea of Hybrid Training (HyT), framework that enables VLAs to learn from thoughts and benefit from the associated perfor- mance gains, while enabling the possibility to leave out COT generation during in- ference. Furthermore, by learning to conditionally predict diverse set of outputs, HyT supports flexibility at inference time, enabling the model to either predict ac- tions directly, generate thoughts or follow instructions. We evaluate the proposed method in series of simulated benchmarks and real-world experiments.
INTRODUCTION
55%
Despite recent advances in robotics, truly generalist robot poli- cies have long been elusive. Thanks to the joint efforts of collecting large-scale robot data (O'Neill et al.) 2224) and making large Vision Language Models (VLM) open-source (Steiner et al. 2024, Tong et al., 2024). we have entered new era in robotics foundation models. By fine-tuning VLMs on robotic datasets containing actions, we obtain so-called models (VLAs) (Kim et al. [0244 Brohan et al. 2023b a): large policy models that trained end-to-end to take language instructions and raw camera im- ages inputs, and output low-level robotic actions.
Success rate
HyT
ECoT
50%
VLA
45%
Hz 2 Hz 3 Hz
Vision-Language-Actin
Inference frequency
Figure 1: Hybrid Training (HyT) of VLAs increases the agent's per- formance similarly to ECoT, but also maintains the same fast infer- ence as standard VLAs. Perfor- mance refers to the ClevrSkills ex- periments (9 tasks, 3000 demos) in the Experiments section.
VLAS possess several advantages over previous work, such multimodal prompting of the agent and the availability of knowledge from the base pre-trained VLM. However, gener- alization to out-of-distribution (OOD) settings, e.g., task figurations not available in the robotics fine-tuning dataset, re- mains challenging. Indeed, the knowledge of the agent is vast
Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc Work done while doing an internship Qualcomm AT Research.

---

about general concepts, but remains limited in the robotics set- tings, where the data distribution is often narrow.
In order to further unleash the capabilities of VLAs with little data, recent works have trained mod- els to predict intermediate outputs, representing the agent's intentions, before predicting actions 2024 Zhao al., )025). One notable example is Embodied CoT (ECoT) (Zawal-
(Zawalski et al.
ski et ala]|]] 2002). where the VLA learns to output useful information about the given task in language form (Wei al., 2023), before generating the actions execute. This not only has shown to improve performance, but it also allows humans to more easily interpret the agent's intentions and potentially intervene on them, i.c. correcting the agent's intentions, before action generation. However, due to the intermediate reasoning outputs generated before actions, the action inference frequency of these models can be significantly lower.
The human cognition process from observation to action is hypothesized to leverage the interaction of two systems (Kahneman, 201T). The fast and intuitive System I handles most daily tasks, taking control in contexts that our brain judges as unchallenging. The slow and deliberate System II is activated when decisions require additional computation, such as comparing options or processing complex information. The tendency of the brain is to delegate System I, to save energy and time. However, in order to do so, humans need to improve their capa- bilities to deal with complex decisions. This is done by developing skilled intuition (Kahneman & Klein 2009 Simon 1992) that allows leveraging previously learned cues to solve familiar tasks, effortlessly.
many decisions as possible to
In this work, we explore the hypothesis that VLA models can similarly develop skilled intuition, when trained with the right objective. Learning from CoT reasoning traces, a model can further internalize knowledge about environments and tasks. Then, at test-time, the model more eagerly recognizes patterns and can leverage such knowledge to generate actions, even in the absence of intermediate thoughts generation. With this hypothesis in mind, we develop Hybrid Training (HyT) framework, which allows the agent to learn from a combination of CoT and actions data.
Hybrid training presents a more flexible learning objective, which encompasses both ECoT and standard VLAs. During training, we implement the HyT objective using Monte Carlo estimate, consisting of sampling variety of conditional inputs and outputs with different probabilities. The model learns to predict a set of ouputs, modelling a multitude of conditional action probabilities, which mainly depends on newly introduced modality variable. During test-time, it's the modality variable that allows us to influence the model's generation. By default, the modality variable con- ditions the model directly predict actions. This allows VLAs trained with HyT to maintain the same inference time as standard VLAs, while benefitting from training on reasoning traces.
Furthermore, the modality variable can be used for manipulating the VLA into operating in different inference modes. The "act" mode, as mentioned, resembles standard VLA's inference and allows to generate actions directly. In addition, termediate thoughts as in ECoT, and a "follow" mode, where the VLA follows a set of provided intentions, e.g. by human an oracle, similarly the lower level policies in hierarchical systems (hh et al. 2025 Hafner et al. 2022).
show "think" mode, where the VLAs generates in-
In addition investigating and validating our proposed HyT framework, on a set of simulated benchmarks (ClevrSkills (Haresh et al. [0244) LIBERO (Luu et al.) 2023)) and real-world tasks (on a UFactory xArm 6), we aim to address a fundamental question regarding VLA models:
What is the contribution of Co'I' techniques to VLAS performance?
Our results and analysis support the idea that one of the main utilities of CoT-based training is im- proving the representation learned by the model (Chen et al. 2025), enabling improved performance, even when no CoT is generated at inference time.
2 RELATED WORK
Vision- Laaion-LLnnnggg--ttttt Actin models. Open-source efforts in the robotics field, such as the Open X- Embodiment dataset (O'Ne) Oe'Neil et al. (024)). , have fueled progress in the development of large VLAs
(Kim et al.] 2024 Black et al. 2024 Wen et al. 2024 Ghosh et al., 2024 Jiang et al., 2023). Recent
works have also explored hierarchical VLA architectures (Shi al. 2025 NVIDIA et al. 2025
2

---

showing they can be beneficial for solving open-ended and long-horizon tasks. Our work aims to improve VLA's performance, by improving the way available reasoning annotations and action data is used, independently of the architecture used.
Chain-of- Thouught (CoT) and reasoning. Generating a chain of thought has shown improved performance in LLMs solving complex reasoning tasks (Wei et al. 2022). Additionally, reasoning has shown notable success using RL with verifiable rewards, coupled with Supervised Finetuning (SFT) on example reasoning traces (DeepSeek-Al et al. 2025 Havrilla al. 2024). CoT techniques
specifically for VLMs (Shao et al. 2022) and VLAs have also been researched (Zawalski et al.)
2024 Zhao al., 2025 Lin al.] 2025 Intelligence et al., 2025). In particular, ECoT (Zawalski et al. 2024) shows that embodied thoughts can greatly improve the agent's 's predictions in robotics, despite the higher inference costs. Our work grounds on their findings and proposes method that accomplishes both strong performance and fast inference.
Hybrid reasoning. Recent works have attempted to distill slow thinking capabilittess into faster models (Deng a (2024) Yu et al.] 2024). Closely related to our method is DualFormer (Su al.) 2025), which proposes to train a language model by systematically dropping out reasoning traces. In robotics domain, RFST (Zhu et al., 2024) proposes a hierarchical setup that uses a discrimina- tor to decide whether to switch to the fast or slow system, with the respective model of the chosen mode being then used the policy. Concurrent work (Chen et al. 2025) also shows that reason- ing pre-training, co-training and/or dropout can improve performance of VLAs,, by improving the representation learned by the model. Our work, instead, provides method to train single hybrid system that learns to conditionally generate a variety of outputs.
3 PRELIMINARIES
Vision-Language Action (VLA) models are multimodal policies generally trained with imita- tion learning. A VLA processes language inputs through a Transformer-baser LLM architecture VVaswani et al. 2023 Brown et al. 2020 Team et al. 2024). Language is first "tokenized" into language tokens that then processed by the LLM. Similarly, VLAs process visual inputs through a vision encoder, e.g., a vision transformer (Dosovitskiy al. 2021), that transforms image patches into visual tokens, which are then processed by the L.LMM.
Given a language description of task 1, the goal of the VLA policy is to solve the task in a given en- vironment. The policy observes the environment through images æ, generally captured by camera in the environment. The policy interacts with the environment using actions a. Through imitation learning, the policy's 's objective is learn, at each discrete timestep, the distribution p(ae (xz, 1) that solves the given task, which is empirically observed from a dataset of demonstrations.
In addition to predicting actions, thinking VLAs, like ECoT (Zawalski et al. (024). also generate intermediate language outputs. These reasonings are expressed as thoughts in language form, predicted by the model. Generally, thoughts include information about the overall plan of action, the current subtask to execute, the location of objects in the image, or the direction of the agent's s ongoing motion (Zawalski et al. 2024). Thinking VLA models are trained to predicts the joint probability distribution over actions and thoughts: p(a,, Te(x,, 1) [or T1) (T)
Thinking VLAs learn single set of parameters 0 to predict both actions and thoughts. Hierarchical VLAs (Shi et al.) 2025 NVIDIA et al. 2025) use a two-level hicrarchy of models, where model provides an actionable language instruction for solving the task, while the second model exccutes the plan. By treating high-level plans and thoughts interchangeably, action prediction in hierarchi- cal VLAS can be modelled as: p(a,, Te Te( 1) = Po, (a, [2,, Tτ) po, (Te Jr,, 1), where 9 denotes the parameters of the "high-level" model and Og of the "low-level" model.
METHOD
VLAs that learn to predict thoughts and actions, like thinking and hierarchical VLAs, have demon-
strated improved performance in several works (Shi et al. 2025 NVIDIA et al. 2025 Zawalski
et al., 2024| Zhao et al. 2025). Compared to standard VLAs, these models learn to: (1) predict
thoughts in language form, and (ii) condition the actions' probability on the generated thoughts.
Thoughts in language form generally consist of significantly more tokens than their action counter-
3

---

GeilonLanguage-Actie (VLA) model
Hybrid Training
Image:
"What the put the red triangle on the mug?"
<thinks
Think
red
mode
right
AM, pripper]
Large Language Model (LLM)
<act> [ax, grlaper)
mode
Figure Hybrid Training (HyT) framework. Given a set of inputs, on the left, including modality variable, the VLA model learns to conditionally generate a variety of outputs. Examples for the 'think' and 'act" conditional distributions are presented on the right.
part. Thus, generating thoughts at test-time comes high inference cost. This can significantly slow down the agent's is action execution in the environment.
We hypothesize that the primary benefits of these models arise not from the generated thoughts themselves, but from the knowledge learned by the model through thought prediction and thought- conditioned actions prediction. This suggests that the model refines its capabilities by internalizing the patterns present in the thoughts, akin to the development of intuitive expertise (Kahneman & Klein 2009). Under this hypothesis, after a learning process that involves thoughts conditioning and thoughts prediction, VLA should be able to predict actions with higher accuracy, independently of the presence of thoughts as an intermediate output.
To address the need for agents capable of producing multiple probability distributions within single model, we introduce a new training strategy, Hybrid Training (HyT), designed to integrate structured reasoning with flexible policy learning.
Definition 4.1 (Hybrid Training) Given a task description and the current environment observa- tion Tr. the conditional distribution over actions at be expressed
p(a((=;;)) 1) = >>Eeo po
ΣΣ po (at, T ( L, m² )p(m²),
(1)
by marginalizing out thoughts and the set of values assumed by the "modality" variable
The hybrid training formulation allows us to describe a singla VLA model, with parameters 0, that learns different conditional action distributions, mainly depending a modality variable.
4.1 HYBRID TRAINING IMPLEMENTATION
The HyT formulation presented in Eq. generalizes the definition of previous VLAs and enables the possibility to combine multiple objectives into a single model. In particular, leveraging the insights from other VLA models, we can use the hybrid training framework to conditionally learn three distributions:
1) = po ( (x,, I, (mo(")) + (o
we. (ar (o m. (po((m)) + Po po(aal Ts, m (m
think
fellow
This way the agent will learns to generate different outputs depending the modality variable value m E (m*, (m²,m" ,m'm.. The meaning of each distribution is defined as follows:
m²,
"Act" distribution: similarly to standard VLAs, it instructs the model to directly predicts actions. follows from p*(r f 0/m²)= =
"Think" distribution: similarly to BCOT, it instructs the model to first predict intermediate
thoughts and then generate actions.
similarly to low-level policy in hierarchical system, it instructs the model to closely follow provided thoughts/instructions for action prediction. It follows from
"Follow" distribution:
pa (r,, Tr, m/) po (a ( xe, I, Te, m²) (conditional independence) and pT 0lmm) 1.

---

Referring to our hypothesis, combining the three distributions should enable the agent to learn to output actions directly (act distribution) while also learning to predict thoughts and to follow in- structions. The learning objective for the model can be described as:
min Syy (0) w/ Leer (0) + w, Lihink (0) + y Lrollow (f)
(3)
where all the terms Ln are negative log- log-likel losses with respect to the corresponding outputs (actions or thoughts and actions
While one could compute the loss directly as the weighted sum of the three terms, for each datapoint, this would reduce variability in the batches, the model would contain the same thoughts and actions multiple times in the same batch. In order to avoid this, we propose to use a Monte Carlo estimate of the objective in Equation 3 Rather than being used as loss coefficients, the weights Wa. wr, wy define the probabilities of sampling the corresponding inputs and outputs for the model during training. This way, at each batch sampled during training, the model receives modality tokens, actions and thoughts, with probabilities that are defined by the coefficients.
To summarize, in order to train VLA with HyT, one needs to specify: (i) the probability distri- butions to be learned, (ii) set of values to use for the modality variable, and (iii) a set of coeffi- cients/probabilities for sampling data during training. In this work, we adopt the distributions de- fined in Eq. 2 with modality variables defined as simple texts, such as "< act >" or "< think >" and with the set of coefficients {wo : 0.25, Wt- : 0.5, wy : 0.25]. An ablation study on the co- efficients and how each distribution impacts performance is presented in Appendix. A simplified diagram illustrating HyT (with only two kinds of outputs) is shown in Figure 2
4.2 INFERENCE TIME
At test-time, the VLA is tasked to predict actions that solve a task. Thanks to HyT, it is possible to predict actions directly, leveraging the model's "act" distribution. In order to do so, we provide the model with the task description, the environments inputs, c.g. a camera image, and the modality variable m =< act > which instructs the model directly output actions. This enables the model to leverage the more extensive knowledge coming from learned thoughts and actions, without incurring any additional inference costs compared to standard VLÁs.
While the main goal of this work is to show that HyT enables higher performance at high control frequency, the flexibibility of the HyT framework also enables the possibility to instruct the agent to operate in different "modes". ", i.e. conditionally predict different forms of outputs. Other than using the VLA in 'act' mode, it is indeed possible to use the model in 'think' or 'foloww' modes.
Q: What's the value in using other inference modalities?
While we expect no major performance difference, e.g. when operating the model 'think' mode rather than in 'act' mode, additional inference modes support greater flexibility in the use of the VLA. For example, the 'think' mode can be used to read the VLA's intentions, for greater inter- pretability. Instead, the 'follow' mode enables to provide set of more fine-grained embodied in- structions for the agent to follow, such as 'move to the left' or 'pick up the object below the gripper'. Note that, also shown in the ECoT work (Zawalski al., 2024), the 'thought' mode can also able
by overriding the agent's 's intentions. We verify whether these modalities can be useful with HyT in the Experiments section.
Q: How to operate the VLA in different modalities?
Generally, set the modality variable to m =< act 2, enforcing the model directly predict actions. If we want to use a different mode, e.g. to predict intermediate thoughts, we can use the mn =< think variable. In practice, we observed that models trained with HyT always diligently attend to the modality token and generate their outputs accordingly.
In this work, we do not explore the possibility of switching the modality token dynamically during executions, as we observed that, after HyT training, different operating modes have different gener- ative behaviors, but similar performance. The modality variable is thus set the beginning of a task
For action prediction, the loss is negative log-likelihood in case the actions discretized and predicted by the L.M.M Alternativel, the loss can be [.1 1.2 loss, for continuous action prediction.
5

---

Overall (9 tasks)
0.4
Success rate
0.3
0.0
300
750
1500
3000
Dataset size of demos)
ECoT
HiRobot
HyT
VLA
Figure 3: ClevrSkills benchmark Aggregated performance on 9 ClevrSkills environments (exam- plcs shown on the left). Shaded areas indicate standard errors.
execution and not updated during the episode. Further understanding and studying the tradeoffs and potential advantages that could derive from switching mechanism between different modalities as in (Lin et al. 2025) is left for future work.
EXPERIMENTS
We validate the proposed HyT method in a series of simulated benchmarks, which we use to exten- sively assess the framework's capabilities, and on a of real-world tasks, which demonstrates the practical utility of the approach.
General setup. Across all experiments, the task description is provided in the same format, ie. "What should the robot do to {task]?". Image inputs are 224 x 224 RGB images from a camera pointing at the robot's workspace. The action space is 7-dimensional and defined as: [Ax, AΦ, gripper). The end-effector position x and orientation & are controlled in delta space, while the gripper pose is in absolute value.
5.1 CLEVRSKILLS BENCHMARK
The ClevrSkills benchmark (Haresh et al. 2024) is based on the ManiSkil12 manipulation environ- ments (Gu et al. 2023). The environment includes an oracle solver to collect demonstrations, and reasoning traces from the oracle, which we can use to discriminate subtasks and extract thoughts. ClevrSkills also adopts a vacuum gripper and mostly uses simple shapes and objects, isolating the challenges of manipulating complex objects, and focussing on planning and generalization instead.
Baselines. In this Section, we focus on comparing the Hy training methodology with other training paradigms, namely the standard VLA training (Kim al. 2024), ECoT-like thinking VLA training Zawalski et al. 2024), and HiRobot-like hicrarchical VLA training (Shi et al. 2025).
Dataset. Using the oracle, collect diverse dataset that spans three main task groups: Place At, Place OnTop, and Stack Tower, with several variations in terms of object types and numbers. The overall dataset is made of 3000 trajectories. For all the thoughts, we adopt the same format, which includes the current subtask and the high-level motion, i.e. coarse direction instructions. This simple definition has proven effective in early stages of our analysis and related work (Shi et al. 2025). A more detailed dataset description is given in the Appendix.
Training. For all approaches, we start training the VLA from the PaliGemma-2 VLM model with 3B parameters (Steiner et al. 2024), which is based on the Gemma-2 LLM (Team al. 2024) and on the SigLIP vision encoder ZZai et al. 2023). For action prediction, actions are tokenized using a set of 256 discrete bins, and predicted by the LLM (Kim al. 2024). We perform hall-finetuning of the model with a batch size of 32 and a learning rate 2e 5, using the Adam optimizer (Kingma & Ba 2017). For the HiRobot approach, we train two distinct PaliGemma-2 models (Shi al. 2025)

---

Place At (4 obj)
Stack Tower (3 obj)
Stack Tower (4 obj)
Overall (3 tasks)
0.8
0.6
0.7
0.7
0.35
0.5
0.6
0.5
0.4
Success
0.4
0.3
0.35
0.2
0.2
0.30
0.1
0.05
0.1
0.00
MyT (think mode)
MyT
HiRobut
ECoT
MyT (think mode) Oracle
-Oaale
HiRobot
Dracle
ECoT Dracle
Figure 4: Instruction-follooing and other inference modalities. Comparing performance when using own generated thoughts (default) or following instructions generated by an oracle.
Evaluation. During evaluation, we run the agent in the environment for 100 evaluation episodes. At each episode, objects and positions are resampled randomly (but consistently among evaluation runs), making no evaluation environment exactly the same as any of the training examples. The agent is, thus, required to generalize actions to new settings in order to succeed. For each model, we evaluate 3 checkpoints taken at epochs 5, and 10 during training. In general, we found no strong correlation between validation losses and performance on the task. Though, with additional training after 10 epochs, models tend to start overfitting.
Q: How does HyT performance scales with data, compared with other training paradigms?
With this experiment, we aim to verify the hypothesis that VLAs using HyT can develop intuitive 'thinking'. If the hypothesis is correct, we expect increased performance compared to standard VLAs, thanks to the knowledge internalized by training on reasoning traces.
As imitation learning performance can greatly vary with the number of demonstrations, we evaluate models trained with different amounts of demonstrations: 300, 750, 1500, 3000. Overall perfor- mance is summarized in Figure 3 The most evident result is that the hypothesis behind this work is confirmed: the models trained with HyT not only outperform standard VLAs, but they also gen- erally perform better than models trained with the ECoT and HiRobot recipes, at all data scales. In Appendix, we additionally provide detailed performance per task. These results highlight that the HyT approach is particularly beneficial, compared to standard VLA training, for more complex and longer-horizen tasks.
Among the baselines ECoT performs (second) best at all data scales. Hierarchical VLAs have higher performance than standard VLAs with smaller amount of demos, but their performance increase is slower than for other methods, eventually being outperformed by standard VLAs from 1500 demos on.
In terms of inference time, we found that standard VLAs and HyT output actions at around 3Hz on A100 GPUs (4 models acting in parallel). ECoT models are 3x slower. HiRobot hierarchical generation is, overall, 4x slower.
Q: How can we use the other inference modalities enabled by HyT and how do they perform?
As described in the Method section, HyT enables the VLA to be used in multiple inference modali- ties, other than the 'acting' mode, directly generating actions. This can be useful in situations where interpretability or the ability to follow fine-grained instructions are useful. In order to verify em- pirically that these modalities are actually usable, we perform study on the instruction following capabilities of the models on a restricted set of tasks.
In order to test instruction following from external sources, provide the agent with set of "Oracle thoughts", which we extract using the code from the the ClevrSkills' oracle. Oracle thoughts replace the agent's thoughts only during moving subtasks, i.e. "move to location X". This is because the oracle has precise conditions for picking and placing, compared to the VLAs. A learned policy not always satisfies them, while still being successful, causing the agent to get stuck.
7

---

Figure LIBERO benchmark Aggregated performance on 4 LIBERO task suites (examples shown on the left). Mean performance evaluated on 100 evaluation episodes.
LIBERO
Spatial
Object
Goal
Long
Avg.
Finetuned from VLA
Trace VLA Zhang
2025
84,6
85.2
75.1
54.1
74.8
Octo Ghosh
78.9
85.7
84.6
51.1
75.1
Open LA
Kim et al
2024
84.7
88.4
79.2
53.7
76.5
CoT-VLA
Zhao
2025
87.5
91.6
87.6
69.0
81.1
ThinkAct
Huangel
2025
88.3
91.4
87.1
70.9
84.4
To-FAST
Pertsch
96,4
96.8
88.6
60.2
85.5
MolmoAct
87.0
95.4
87.6
77.2
86.6
Finetumed from
VLA-OFT Kim et 2025
94.2
97.8
91.4
84.8
92.1
HyT (ours)
94.0
97.2
96.2
89.4
93.7
We use the models trained on the multitask dataset with the largest size (3000 demos) and present results in Figure for three of the most complex tasks in the ClevrSkills benchmark adopted. HyT performance is shown all inference modalities: 'act' mode (default, in blue), 'think' mode (in green), and 'follow' and 'think' modes with oracle thoughts (barred). First, we can observe that for all the methods tested, following instructions from an oracle, rather than self self-predicted thoughts, can improve the performance of the agent. For HyT. this means that the additional inference modalities ('think' and 'follow") can actually be useful in such contexts.
Furthermore, observe that, without oracle thoughts, models trained with HyT perform similarly in "act' and 'think' modes, corroborating the idea that, with Hy'. in the evaluated settings, interme- diate thought generation test-time may be unnecessary. Nonetheless, for future work, would be worth verifying whether this finding holds for tasks requiring more complex embodied reasoning.
5.2 LIBERO BENCHMARK
The LIBERO benchmark one the most adopted in the VLA community (Zheng et al. 2025
Ghosh et al. 2024 Kim et al. 2024 Zhao et al.) [025 Huang al. 2025 Pertsch et al. 2025 Lee
jt al. 2025
Kim et al. 2025).
The agent is evaluated on suites tasks: Spatial, Object, Goal and
Long (10).
In LIBERO, oracle thoughts are not available. Thus, to extract CoT, we first use the simulator obtain labelled object bounding boxes and high-level motion primitives, including the gripper changes. Then, we feed this information step-wise, along with the task description, to an LLM that generates plan made of subtasks and associate subtasks with temporal steps. More information about the thoughts extraction and structure can be found in the Appendix.
Q: Can HyT be employed in combination with different VLA designs and how does perfor- mance compare to other VLAs in the literature?
Previous work on LIBERO has shown that there are some important choices in the VLA design that are crucial to obtain high performance, such as the adoption of action chunking (Lee et al. 2025 Zhao et al., 2025 Pertsch al. [025 Kim al. (022). For this reason, we chose to implement Hy?l in combination with the OFT fine-tuning recipe (Kim et al.] 2025), which employs action chunking
and continuous actions prediction (LI head).
For both the VLA-OFT model and HyT, we fine-tune the model starting from the Prismatic VLM KKaammhhti al. 2024). This allows us to better leverage the language pre-training of the VLM, as also shown in (Zawalski et al.) 2024). Hyperparameters and training settings are the same in (Km et al. 2025). ). Results are presented Figure
Compared to other fine-tuning recipes, the OFT strategy leverages data efficiently, outperforming all other baselines in all suites of tasks but LIBERO Spatial. Training the model with HyT allows the model to increase performance even further, especially in the most complex suites of tasks: Goal and Long. This shows that HyT can successfully combined with well-established VLA designs, potentially improving tfte-of-thheaa performance.
8

---

Table 1: Real-world experiments. Success rates with standard error on the real-world tasks (on the
left). Additional details about experimental settings are provided in the Appendix.
OpenVLA
HyT
In-distribution tasks
52% +10
72% +9
banana in green bowl
70% 115
70% 215
red cube in brown bag
50% 420
100% =0
tomato left of lettuce
60% #22
60% =22
zucchini in front of green cube
0%
50%
citto--distribution tasks
29%
54% +10
rubber duck in green bowl
40% #15
20% =13
mushroom in brown bag
0% a0
100%
purple die left of lettuce
0%
50%
zucchini front of red hexagon
75% a22
75% A22
Overall
41 %
63%
We also tested fine-tuning VLA-OFT and HyT starting from the Open VLA pre-trained VLA model. However, in this settings, we found no difference in performance (overall success: 95.3%+t.1 ) as both methods nearly saturate the benchmark's performance. One plausible conclusion that HyT and other Chain-of-Thouse can compensate for the lack of robotics pre- pre-training or the scarcity of fine-tuning data, as similarly shown in the other experimental results in this work.
(CoTT) strategies, by leveraging additional thought data during training,
5.3 REAL-WORLD EXPERIMENTS
One of the major benefits of the HyT approach is the capability to achieve higher performance while retaining lower inference time. This is particularly useful in weal-world use-cases, where faster execution is important to carry out tasks quickly and to increase the perceived quality of the agent.
For our real-world experiments, collected dataset comprising 320 trajectories using robotic setup featuring an UFactory xArm 6 with a flexible two-fingered gripper, operating on a white tabletop. The agent observes the environment through RGB images captured by RealSense D435 camera, positioned at corner of the table. For the models, we start from the pre-trained Open VLA model with 7B parameters (Kim et al., 2024) and compare directly it. We perform LoRA fine- tuning with rank 32 (Hu et al., 2021). a batch size of 32 and learning rate of 5e 4, using the Adam optimizer (Kingma & Ba 2017).
Our evaluation spans two categories of tasks: in-distribution and out-of- distribution set includes tasks for which the dataset contains least 10 demonstrations. For the -ut-of-distiibution set, we modify the in-distribution tasks with alterations, such as different objects or placements, ensuring the agent encounters novel scenarios not present in the training data. See the Appendix for additional details.
The in-
The results, shown in Table | show that HyT overall outperforms OpenVLA, especially in out-of- distribution tasks. From qualitative perspective, we notice that VLe and HyT have similar flaws, c.g., they tend to pick objects with the wrong orientation. However, HyT tends to be more precise when reaching picking and placing positions, c.g. it never reached for the wrong object while VLe did, eventually leading to a noticeable performance gap.
DISCUSSION
Thinking strategies for VLAs ZZawalski al.) 2024 Shi et al. 2025) have shown important bene- fits in terms of performance and interpretability over standard VLAs, with the drawback of slower inference. In this work, we support the idea that the thinking process can be "internalized" by the model, developing some form of expert intuition (Kahneman & Klein 2009). We proposed the Hybrid Training framework, which enables the possibility of learning from thoughts, for higher performance, while also being able to predict actions directly for faster inference, empirically validated through simulated and real-world experiments. To conclude, we attempt to answer the question: What is the contribution of CoT techniques VLAs performance?

---

From analyzing the HyT framework, understanding is that learning generate CoT and learning to predict actions from CoT improves the agent's 's understanding of the environment. This is closely related to how auxiliary losses have been used for imitation and reinforcement learning (Yrrats) et al. 2020 Srinivas et al. 2020), improving the learning dynamics of the model and generalization performance. Concurrent work (Chen et al. 2025) also supports the idea that learning from thoughts improves representation learning in VLAs.
Our work does not exclude that thoughts generation might be useful at test-time in more complex settings. Instead, it demonstrates that primitive tasks such as moving objects, opening/closing draw- ers, building stacks of objects, can be solved robustly and with low inference costs. by improving the model internal representations through HyT. More complex tasks requiring memory and/or ad- vanced reasoning would still benefit from reasoning at test time. We leave the investigation of these tasks, which are currently sparse and rarely adopted in the robotics literature, for future work.
One limitation of CoT, hierarchical and HyT methods is that they require additional human labelling, providing thoughts for the agent to learn from. In this respect, one advantage of HyT approach is that it does not require the CoT to present for the whole dataset. Instead, given that HyT relies on sampling for approximating its objective, thoughts can be sampled just for the trajectories that contain them.
Automated reasoning approaches to extract useful thoughts in sepe-super manner would be a promising direction to explore in future work (DeepSeck-Al et al.) (025).. Furthermore, we are convinced that the cost of labelling data will become negligible state-of-the-art vision and vision- language models become more and more performant on visual and spatial reasoning tasks. this context, HyT offers scalable framework distill human and foundation models' knowledge into VLA models, while strongly reducing inference costs compared to reasoning-only approaches like ECoT.
10

---

ETHICS STATEMENT
Robotic agents that are interpretable could have positive societal impact, as they allow users to read and/or verify the agent's s intention. Intervenability can both have positive and negative social impact, as it allows users to correct the agent's s intentions, but it could also allow malicious users to steer the agent's behavior towards unethical behaviors.
REPRODUCIBILITY STATEMENT
In order to ensure reproducibility, made use of open-source simulation benchmarks (ClevrSkills, LIBERO) and open-source models (Paligemma, Prismatic, Open In the main text and/or the Appendix, state all the important hyperparameters for fine-tuning the models using approach and replicate our results. We also provide details about all the tasks, chains-of-thoughts, and models' prompts employed.
REFERENCES
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubezak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. no: A vision- language-action flow model for general robot control, 2024. URL https: http::::/aaxxvvorr
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wes Wei Edward Lee, Sergey Levine, Yao Lu, Hen- ryk Michalewski. Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt- 2: Vision-language-action models transfer web knowledge robotic control, 2023a. URL 2tt.////..///3....// 15818
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalash- nikov, Yuheng Kuang. Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deek- sha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vin- cent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-1: Robotics transformer for real-world control at scale, 2023b. URL aattps://arkee htt/a/ivvv.org/b////2/333 06817
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voos Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https::/aaxii: htt.
org. abs 2005 14165 14165
William Chen, Suneel Belkhale, Suvir Mirchandani, Oier Mees, Danny Driess, Karl Pertsch, and Sergey Levine. Training strategies for efficient embodied reasoning, 2025. URL https://
erxiv. www./rg.../.....oooo
httpg//ab/2550.... /2005 08243
11

---

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangho Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Honghui Ding, Huajian Xin, Huazuo Gao. Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang. Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen. Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng. Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang. Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang. Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li. Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepscek-rl: Incentivizing reasoning capability in llms via reinforce- ment learning. 2025. URL /tappp:://rrxeee //tp://a.vvvv//.. www./rgg/////oooo////o//eee
abs 2501 12948
Yuntian Deng, Yejin Choi, and Stuart Shieber. From explicit cot to implicit cot: Learning to inter- nalize cot step by step, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition scale, 2021. htttt::t/://// http:s///aaxvvoorr/./.
22100 11929
Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong. Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy, 2024. URL
wttpd//.rrg/a////eeeeeeeee 12213
Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie. Zhiao Huang, Rui Chen, and Hao Su. Maniskil12: A unified benchmark for generalizable manipulation skills, 2023, URL https:
//arxiv.or//alt
20000322222 04659
Danijar Hafner, Kuang- Lee, Ian Fischer, and Pieter Abbeel. Deep hierarchical planning from pixels, 2022.
2206. 04114
Sanjay Haresh, Daniel Dijkman, Apratim Bhattacharyya, and Roland Memisevic. Clevrskills: Com- positional language and visual reasoning in robotics, 2024, UtLhtps:: org/abss 241 09052
Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. arXiv preprint arXiv:2403.0446 2024.
12

---

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https:
09685
Chi-Pin Huang, Yueh-Hua Wu, Min-Hung Chen, Yu-Chiang Frank Wang, and Fu-En Yang. Thinkact: Vision-language-action reasoning via reinforced visual latent planning, 2025. URL
https::/aaxxt http://axxvvv.ogaaaaaae httg//bb//530 16815
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen 7. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang. Lili Yu, and Ury Zhilinsky. To.5 vision-language-aattoo action model with open- world generalization, 2025. URL httpp://a/aar: azq/abs/2504.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang. Yonggiang Dou, Yanjun Chen, Li Fei- Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts, 2023.
org/ abs/ 2/tp//2221.0099
Daniel Kahneman. Thinking, fast and slow. macmillan, 2011.
Daniel Kahneman and Gary Klein, Conditions for intuitive expertise: failure to disagree. Am Psychol, 64(6))515-526. September 2009.
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Investigating the design space of visually-oonditione language models, 2024. holLltttss:/aaxi..orrt
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-langunge-actinn model. arXiv preprint 1rXiv:24000092466 2024.
Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning lssion-lnnguage-ee models: Opti- mizing speed and success. arXiv preprint arXiv:2502.19645 2025.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL
/tttps://arxiv org/ www.//////////oo//oooooooaaeeee/ 20.1//141..690
Jason Lee, Jiafei Duan, Haoquan Fang, Yuquan Deng, Shuo Liu, Boyang Li, Bohan Fang. Jieyu Zhang, Yi Ru Wang, Sangho Lee, Winson Han, Wilbert Pumacay, Angelica Wu, Rose Hendrix, Karen Farley, Eli VanderBilt, Ali Farhadi, Dieter Fox, and Ranjay Krishna. Molmoact: Action reasoning models that can reason in space, 2025. URL
07917
Fanqi Lin, Ruigian Nai, Yingdong Hu, Jiacheng You, Junming Zhao, and Yang Gao. Onetwovla: A unified lision-language-ee model with adaptive reasoning, 2025. URL https://axxiv.. org/abs 2505. 11917
Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning, 2023. URL https:///axiv:
htg/aba/2300 03310
NVIDIA, :, Johan Bjorek, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi "Jim" Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, Joel Jang, Zhenyu Jiang, Jan Kautz, Kaushil Kundalia, Lawrence Lao, Zhiqi Li, Zongyu Lin, Kevin Lin, Guilin Liu, Edith Llontop, Loic Magne, Ajay Mandlekar, Avnish Narayan, Soroush Nasiriany, Scott Reed, You Liang Tan, Guanzhi Wang, Zu Wang. Jing Wang, Qi Wang, Jiannan Xiang, Yuqi Xie, Yinzhen Xu, Zhenjia Xu, Seonghyeon Ye, Zhiding Yu, Ao Zhang, Hao Zhang, Yizhou Zhao, Ruijie Zheng, and Yuke Zhu. Gr00t nl: An open foundation model for generalist humanoid robots, 2025. URL https://arrs wwwp:iv.oorgggaa..
14734
13

---

Abby O'Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acom Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew Wang, Andrey Kolobov, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Bal- akrishna, Ayzaan Wahid, Ben Burgess-Limerie
Beomjoon Kim, Bernhard Schölkopf. Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang. Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Felipe Vieira Frujeri, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schi- avi, Glen Berseth, Gregory Kahn, Guangwen Yang. Guanzhi Wang, Hao Su, Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik I Christensen, Hiroki Furuta. Homanga Bharadhwaj. Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jachyung Kim, Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jay Vakil, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang, Jitendra Malik. João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang. Jordi Salvador. Joseph J. Lim, Junhyek Han, Kaiyuan Wang. Kan- ishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrn, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Ku- nal Pratap Singh, Kuo-Hao Zeng. Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen, Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi "Jim" Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip, Mingtong Zhang, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Ning Liu, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R Sanketi, Patrick "Tree" Miller, Patrick Yin. Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Mart' iun'inn-arttin in, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Ju- lian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Lin, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shubham Tulsiani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Ste- fan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suncel Belkhale, Sungjac Park, Suraj Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vikash Kumar, Vincent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiangyu Chen, Xiaolong Wang, Xinghao Zhu, Xinyang Geng, Xiyuan Liu, Xu Liangwei. Xuanlin Li, Yansong Pang, Yao Lu, Yecheng Ja- son Ma, Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yongqiang Dou, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yue Cao, Yuch- Wu, Yujin Tang. Yuke Zhu, Yunchu Zhang. Yunfan Jiang. Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen Zhang. Zipeng Fu, and Zipeng Lin. Open x-mboeimmen: Robotic learning datasets and rt-x models, 2024. URL. https: /tbps:///axvvv wrg. /2310. 08864
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Karl Pertsch, Kyle Stachowiez, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision-languagg-action models, 2025. htt.httts:://aat
org. aalaab///2500 2501 09747
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song. Zhuofan Zong, Letian Wang. Yu Liu, and Hong- sheng Li. Visual cot: Advancing multi-modal language models with comprehensive dataset and
14

---

benchmark for chain- chain-of-thought reasoning, 2024.
16999
Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, Adrian Li-Bell, Danny Driess, Lachy Groom, Sergey Levine, and Chelsea Finn. Hi robot: Open-ended instruction following with hierarchical vision- Assion-angguage-ccto models, 2025,
19417
Herbert A Simon. What is an "explanation" of behavior? Psychological science, 3(3):150-161, 1992.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning, 2020. URL
04136
Andreas Steiner, André Susano Pinto, Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey Gritsenko, Matthias Minderer, Anthony Sherbondy, Shangbang Long, Siyang Qin, Reeve Ingle, Emanuele Bugliarello, Sahar Kazemzadeh, Thomas Mesnard, Ibrahim Alab- dulmohsin, Lucas Beyer, and Xiaohua Zhai. Paligemma 2: A family of versatile vlms for transfer, 2024. /tt.ttpsss//arx..voorrr/o//
03555
DiJia Su, Sainbayar Sukhbaatar, Michael Rabbat, Yuandong Tian, and Qinqing Zheng. Dualformer: Controllable fast and slow thinking by learning with randomized reasoning traces, 2025. URL
https://arxtt org/ abs 241 09918
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju. Léonard Hussenot, Thomas Mesnard, Bobak Shahriari. Alexandre Ramé, Johan Fer- ret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Char- line Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanezyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchi- son, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal. Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Chooe Danila Sinopalnikov, David Wein- berger. Dimple Vijaykumar, Dominika Rogozińska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Slimczak-Plucifsse Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kar- tikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Lauuent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, I Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moyni- han, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Osear Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culli- ton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej. Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Me Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Amold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ron- strom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky. Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang. Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dra- gan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Fara- bet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving open language models at a practical size, 2024. URL https: / wtw.///aiiii wtt.oorg///bsaoo 00118
Harleen
15

---

Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Ziteng Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-l: A fully open, vision-centric exploration of multimodal Ilms, 2024. URL wtl.httss///axiv..ooo
/406. 1....6666 6860)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. [tttps::///arxx
htte//bb/177..... 03762
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24888 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le. and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. /ttpp::/arxxi.....t/////// org/ abs abs 11903
Junjie Wen, Yichen Zhu, Jinming Li, Minjie Zhu, Kun Wu, Zhiyuan Xu, Ning Liu, Ran Cheng, Chaomin Shen, Yaxin Peng, Feifei Feng, and Jian Tang. Tinyvla: Towards fast, data-efficient vision-language-action models for robotic manipulation, 2024. URL httptttss//axii.....//
abs/2403 12514
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Im- proving sample efficiency in model-free reinforcement learning from images, 2020. URL
https://arxiv. wttp 01741
Ping Yu, Jing Xu, Jason Weston, and Ili Kulikov. Distilling system 2 into system 1, 2024. URL
06023
Michal Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic control via embodied chain-of-thought reasoning. arXiv preprint arXiv:2407.08603 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training, 2023. URL httttttss///aiiii/
abs///b///// /ww.///....... 2303.15343 15343
Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma. Song Han, Chelsea Finn, Ankur Handa, Ming- Yu Liu, Donglai Xiang, Gordon Wet- zstein, and Tsung- Yi Lin. Cot-vla: Visual chain-ol of thought reasoning for lisionllanguage-action models, 2025. URI.
wwtp://.orrr//////eaa//o 2503. 22020
Ruijie Zheng, Yongyuan Liang, Shuaiyi Huang, Jianfeng Gao. Hal Daumé III, Andrey Kolobov, Furong Huang, and Jianwei Yang. Tracevla: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies, 2025. URL
10345
Minjie Zhu, Yichen Zhu, Jinming Li, Junjie Wen, Zhiyuan Xu, Zhengping Che, Chaomin Shen, Yaxin Peng, Dong Liu, Feifei Feng, and Jian Tang. congunge- robotic manipulation with fast and slow thinking, 2024. URL httpss//acavv..oo///bb////////// 04181
16

---

A APPENDIX
A.1 HYT COEFFICIENTS
In this Section, we investigate the impact varying the HyT cofficients during training, using the
ClevrSkills benchmark.
In Figure 6 we observe that the 'think' distribution is particularly important when the dataset avail- able is small (300 demos). The ability to generate thoughts about the environment and the tasks may be providing a stronger training signal compared to just following detailed instructions for action prediction. As the dataset size increases, we observe that this is difference is no more present, and that models that include the 'follow' distribution actually tend to perform better.
The usefulness of the "follow" distribution is further demonstrated on the instruction- instruction-folloiinng ab- lation, presented in Figure / where we observe that despite the model trained without the 'follow' distribution has the same performance when the model is used alone, training with the 'follow' distribution improves performance when following instructions from an oracle.
OnTep obj)
OnTop abj)
OnTop abj)
Overall (9 tasks)
0.75
0.75
0.6
0.50
025
0.25
0.25
750
3000
750
3000
750
3000
Place At (2 ob))
Place At ob))
Place At (4 ob))
Success rate
1.0
Success
0.5
0.75
0.50
0.25
3000
750
3000
750
3000
Stack Tower obj)
Stack Tower obj)
Stack Tower obj)
0.2
1.0
Succes
0.5
0.5
0.1
750
3000
750
3000
750
3000
300
750
1500
3000
Dataset size (# of demos)
ECoT (t=1.0)
HyT (t=0.2, a=0.8, f=0.2) HyT (t=0.5, a=0.25, f=0.25)
HyT (t=0.5, a=0.5)
HyT (a=0.5, f=0.5)
Figure 6: Ablating HyT coefficients used during training on the ClevrSkills benchmark.
Place At obj))
Stack Tower (3 obj)
Stack Tower (4 obj]
Overall (3 tasks)
0.7
0.5
0.20
0.6
0.5
0.5
Success
0.4
Saccess
0.3
0.10
0.1
0.1
0.0
0.0
0.00
0.0
HyT (t-0.5, am0.5)
HyT [tw0.5, am0.25, f=0.25)
HyT (t-0.5, a-0.5) Orade
HyT (t=0.5, 2-0.25, f=0.25) Oracle
Figure 7: Ablation on the inclusion of the "follow" distribution during HyT training the instruction-folloving benchmark.
17

---

A.2 CLEVRSKILLS DETAILED RESULTS
In order to provide further insights on the ClevrSkills benchmark, we provide additional results, where we grouped tasks in sets:
1. Easy lasks: requiring only 2 actions (pick, place) and containing no distractors;
2. Medium tasks: requiring only 2 actions (pick, place) but containing 1+ distractors;
3. Hard tasks: requiring more than 2 actions or containing 2+ distractors.
For each task set, we analyze the results with 300 demos (Ix data) and 3000 demos (10x data). Results are showin in Figures 8
9 and T0
Overall, we observe that HyT always shows large improvements over standard VLA training in the low-data regimes. When more data is provided, the performance gap is reduced for easy and medium tasks (~ 4% difference), while it remains larger for the hard tasks (~~ 9% difference). This corroborates the idea that more complex and longer-horizon tasks benefit from training reasoning traces more than simpler and short-horizon tasks.
Clevikkilts Hard tasks
Place OnTopp ohj)
Stack Tower
Stack Tower
Overall 13 tasks)
0.8-
0.22
0.5
9.30
0.4
Saccnsrrat
9.08
Seccess att
3.28
0.2
3.22
Figure 8: Detailed results on the ClevrSkills benchmark hardest tasks.
ClevrSkills diuum tasks
Place On
Place obj)
Place (4
Overal (3 tasks) 0.8
0.7
9.6
0.6
0.6-
Saccassraee
0.5-
Success
0.4
0.4
Figure Detailed results the ClevrSkills benchmark medium tasks.
ClevrSkills Easy tasks
Place At (2 obj)
Place Alt obj)
Stack Tower ob))
Overall (3 tasks)
1.5
1.0
bucosss
2.8
0.5
0.8
0.2-
52-
Figure 10: Detailed results on the Clevrškills benchmark easiest tasks.
18

---

A.3 CLEVRSKILLS DETAILS
Tasks definition The tasks are defined as follows:
Place Al tasks evaluate the agent's spatial understanding. The tasks require the agent to bring an object to a location specified in relation to another object location, i.e. left/right/hhinddinn front of the object. The objects used are mainly simple-shaped objects of different colors as in (Jiang et al. 2023).
Place OnTop tasks evaluate the agent's understanding of object interactions, they require plac-
ing an object on top of another in a stable position. The set of objects used contains simple shapes, but also more complex objects from YCB, such mugs, wooden blocks, bowls, etc.
Stack Tower evaluates the long-horizon capabilities of the agent, as it requires multiple "place on top" with simple object shapes, where the order of the objects the stack defined in the prompt.
For each task, we train and test the agent with three variants, containing varying number of objects. In Place OnTop and Place At the number additional objects only distracting or increasing the amount of clutter in the scene, e.g. making placing objects the table harder. For the Stack Tower tasks more objects also require more actions, as the agent should stack all the objects present the scene.
Training details. The training for all experiments and approaches is done using PyTorch DDP (Paszke et al. 2017) on 4 A100 GPUs. Inference requires less than 20GB VRAM and is performed on a multi-instance A100 for simulated environments.
Evaluation details At each episode, objects and positions resampled randomly (but consistently among evaluation runs), making no evaluation environment exactly the same as any of the training examples. The agent thus, required generalize actions to new settings in order to succeed. For each model, we evaluate 3 checkpoints taken at epochs 5,7 and 10 during training. In general, we found no strong correlation between validation losses and performance on the task. Though, with additional training after 10 epochs, models tend to start overfitting.
Dataset definition. For each task group, we collected a set of 1000 demo trajectories composed as follows: 250 demos in the 2 objects task, 500 demos in the 3 objects task, 250 demos in the 4 objects task. Then, to train agents with different sizes of the dataset, we subsample fixed subsets of trajectories from each dataset. For the multitask dataset, the task group datasets are subsampled and then aggregated.
Extracting thoughts. In ClevrSkills' demonstrations (Haresh et al. 2024) a variety of solvers is ap- plied to the task, following a pre-specificied order oracle's 's plan. Each solver takes executes one subtask from the overall plan and the solver parameters can be recovered in the demonstrations. In order to create thoughts, we can use the ClevrSkills library to transform each subtask from the solver into natural language instruction, e.g. "Move X" "Pick up Y". Then, for moving instructions e.g. "Move to..." and "Carry Z to..." we extract the motion direction of the agent. Similarly to (Shi et al. 2025), this is obtained each timestep by computing the distance between the current end-effector position and the position at the end of the motion subtask. This is then transformed into language, in the form of "left/right", "forward/hackwaal
and "up/down" instructions, as in (Zawal- ski et al.] 2024). Finally, for distances that are less than I cm the agent receives "close" moving instruction. Two examples of thoughts and actions, in the HyT format, are provided in Figure
<think>
<think>
carry the red triangle to mug move: forward right down <act>
pick up the red triangle
<act>
[Ax, 4d, gripper]
[Ax, A&, gripper)
(b)
Examples of thoughts and actions for a moving (a) and a non-moving (b) subtasks, expressed in language format following the format used for HyT.
Figure 1I:
19

---

Gripper information. The ClevrSkills benchmark adopts vacuum-gripe mecuum-gripper robot which creates vacuum in the gripper's s suction cups to maintain contact with objects. Compared to a fingers- based gripper, suction gripper's state, i.c open closed, cannot be easily assessed through images. Thus, in order for the agent to be able to perform grasping actions reliably, we concatenate the state of the gripper in language form the task description (or the thoughts for the hierarchical low- level VLA). The gripper state information we pass to the agent is: (i) whether the gripper suction cups are making contact with something, (ii) whether the gripper was on, grasping something, not. There's s need for the gripper information in the real-world experiments, as we use fingers-base basee gripper, where the gripper state visually observable.
A.4 LIBERO DETAILS
CoT extraction. In order generate plans and subtasks the Gemma-2 9B model.
For plan generation we use the following prompt:
want previ robotic with short 1ist of high-leve] steps to complete the task: (instzuction]'.
The robotic arm is operating in a scene that contains the following objects:
(object_l), (object_2), .... (object_n)
Break down the instruction into a Python list of high-level steps. These steps describe the coarse primitive actions that the robot needs to take to complete the task.
It important not to invent subtasks unless they are explicitly required by the instruction.
For example, if the task does net mention opening er closing drawer, do not include those actions.
After listing the steps, please previde brief explanation for each one of the subtasks.
Listing 1: Prompt for LLM to generate high-level robotic steps
20

---

For inferring step-wite wise subtasks use the following prompt:
want to analyze each step my rebot took to accomplish the task: fi instructionl'.
To complete this task, it followed these subtasks in order: Isubtask 1
(subtask 2}
(subtask
Below is a mapping from each timestep to the motion the robot executed. The final item in each entry represents the object closest to the gripper. This information is used solely for reasoning purposes and not part (...)
motion itself:
Please previde de Python dictionary that maps each of the aubtasks (as strings) the integer timestep at which that subtask begins. The mapping should include only the subtasks listed above. Since the subtasks are sequential, later subtasks must correspond to later timesteps. Each subtask must be assigned a single timestep, and the first subtask should always begin at step 0.
Listing 2: Prompt for LLM to associate subtasks with steps in an episode
The final CoT' look like this:
locate the cabinet, posit in above the mi middle drawer, open the drawer VISIBLE OBJECTS: akita akita_blast bllck__aaoaae 1: 1138, 139, 140, 1501, cream cheese_ plate [161, 21,, wine_rack_l; 1: [0, 94, 81,212), $lat_stove_1:
PLAN:
wine sate__ootte__l [104, 96, 1177,44,,77,,66,
[99,171, 141, 196], wooden cabinet_
wooden_cabinet__1
25, 92,1
SUBTASK REASONING: the robot needs to move into the correct spatial locatien relative to the cabinet
SUBTASK: position above the middle drawer
MOVE: move left
GRIPPER POSITION: [106, 149]
Listing 3: CoT for LIBERO experiments
A.5 REAL-WORLD EXPERIMENTS DETAILS
(a)
(b)
(c)
(d)
Figure 12: The setup for some of the real-world tasks: (a) banana in green bowl (b) red cube in brown bag (c) zucchini in front of green cube (d) tomato left of lettuce.
Dataset preparation. The dataset for real-world experiments is collected via human tele-operation, using VR set and controller map the human motions and intentions, e.g., closing the gripper, to the robot. After collecting the trajectories, we apply some basic pre- pre-processing operations. First, we take the sequences collected 60 Hz, and subsample them at 10 Hz. Then, extract actions in the format [Ax, Ad, gripper] from the end-effector and gripper positions of the demonstrations. Finally, filter no-motion operations, i.e. having Ar and A0 equal to zero.
21

---

Extracting thoughts. For extracting the subtasks to use in the thoughts, based approach. This is based on the assumption that all the tasks in the dataset are in the format "place the {obj1} } {position) the {obj2)'. e.g. 'place the {zucchini} {in front of} the {green cube)'. We identify the position when the robot closes the gripper as the grasping position, and the posi- tion when the robot opens the gripper as the releasing position. Then, we identify the following keyframe moments: (i) the moment when the agent reaches within 3 cms distance from the grasping position, (ii) the grasping moment, (iii) the moment when the agent reaches within 3 cms distance from the releasing position. Finally, the subtasks are defined in the trajectory as follows:
simple heuristic-
Move to the {objl}: between the start of the trajectory and keyframe (i):
Pick up the {objl}: between keyframe (i) and keyframe (ii);
Move (pasition) the {ohj2): between keyframe (ii) and keyframe (iii);
. Place object {position] the {obj2}: between keyframe (iii) and the end of the trajectory.
Also, to have the same thought structure as in the ClevrSkills' experiments, we concatenate a 'move: {direction] to the moving subtasks, where the direction is computed using the end-effector position (see Section A.3).
Evaluation.
During evaluation, we limit the execution time to 150 agent steps, which is ~ 2 minutes per trajectory, considering actions/second from the model and control loop running at 2Hz for stable motions. In case of failed grasps, we allow up to 3 attempts to the agent, before assessing the outcome of the episode. In the following, you find a detailed description of how each in-distribution task is executed and randomized. For ouitr--distribution tasks, follow the same procedures, but we swap one of the main actors in the scene (e.g. the object to pick or the placing target). See Figure 12 for reference. Note: fruits are stuffed toys, while vegetables are hard foam models.
Place the banana in the green bowl (10 trials). The scene includes a green bowl and three objects: a banana, tomato, and blue hexagon, each with predefined positions. We conduct five trials with the banana starting in one location, and five more from a different location. In each location, the banana appears three orientations: vertical (2/5 trials), horizontal (2/5 trials), and diagonal (1/5 trial).
Place the red cube the brown bag. (6 trials). The setup features a brown bag and three objects: a red cube, banana, and lettuce leaf, all placed at specific locations. We run two trials for each location of the red cube. Its orientation varies between being aligned with the robot's base and being tilted.
Place the tomato left of the lettuce (5 trials). The scene contains three objects: a lettuce Icaf, carrot, and a tomato. Their positions are randomized within defined regions, though the overall layout remains consistent. We perform five trials with these randomized placements. A trial is only considered successful if the tomato ends up on the table, near the lettuce, and to its left.
Place the zucchini in front of the green cube (4 trials). The environment includes five ob- jects: a shape sorting box, a zucchini, purple die, a rubber duck, and green cube. Object positions are randomized within certain bounds, while maintaining the general layout. We carry out four trials with these randomized setups. Success is defined as the zucchini being placed on the table, close and in front of the green cube.
22


Reviews:
[{"review_id": "YuCxFSG8JY", "reviewer": "Reviewer_cYRE", "summary": "This paper proposes that the benefits of chain-of-thought (CoT) reasoning in VLAs can be reaped without the very slow inference times that generating reasoning chains at test-time requires. Namely, they propose a method called HyT (hybrid training) in which they train the VLA to exhibit multiple modes of behaviors, either predicting actions directly, predicting a reasoning chain and then actions, or predicting actions from an oracle human-provided reasoning chain, with these modes selectable by configuring a modality variable. This hybrid training scheme allows the model to effectively learn from the chain-of-thought data at train time, not having to actually generate these reasoning chains at test-time to still get equally good performance. The authors test their approach against a standard VLA, a reasoning VLA, and a hierarchal VLA on the ClevrSkills simulated benchmark and find that their approach performs the best across multiple dataset sizes. On the Libero simulated benchmark, there approach, adapted off of the OFT-VLA model, performs the best when compared to many prior VLAs on Libero. And finally on eight tasks in the real-world, their model exhibits better performance than a standard VLA while requiring the same inference time (~3 Hz).", "questions": "(1) For the three different modes, the authors chose weights 0.25 for act, 0.5 for think, and 0.25 for follow. Did they ablate different values of these weights? Generally how important is a good selection of these weights to model performance? Which of the three modes matter the most/least?\n\n(2) HiRobot + Oracle seems to be doing the best, as shown in Figure 4. Am I correct in assuming that the main difference between HiRobot and the proposed approach is that two separate PaliGemma models are used? What leads to the improved performance? More capacity? Less feature sharing?\n\n(3) Why do you think ECoT performed worse than the proposed approach? They are very similar, and it would seem that ECoT should perform slightly better since it generates reasoning at test-time?\n\n(4) Why aren’t any of the ClevrSkills baselines compared against in Libero? E.g., ECoT or HiRobot?\n\n(5) For the real-world tasks, what do the reasoning traces look like and how were they generated? Were the types of reasoning traces generated such that reasoning might be particularly useful in the OOD setting?\n\n(6) There appears to be substantial overlap with a contemporaneous, peer-reviewed publication ([1], accepted Aug 1, 2025). Per the ICLR 2026 reviewer FAQ, I will not penalize the paper for missing a citation or comparison to contemporaneous work. However, as written, the submission does not clearly articulate contributions beyond that work. It would be great if the authors could clarify what is genuinely new here (theory, method, or empirical evidence) beyond this prior work.\n\n[1] Chen, W., Belkhale, S., Mirchandani, S., Mees, O., Driess, D., Pertsch, K., & Levine, S. (2025). Training Strategies for Efficient Embodied Reasoning. arXiv preprint arXiv:2505.08243.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) The authors tackle a very relevant problem in robotics, and arrive at a nice result in that the benefits of reasoning trace generation can be obtained without having to generate expensive reasoning chains at test time. The proposed hybrid training strategy is general and permits different manners of either conditioning on or not conditioning on reasoning at test time.\n\n(2) The experimental results are reasonably through, and include real-world results, which is nice. In particular, the authors’ method scores the highest on the Libero simulated benchmark when compared to other fine-tuned VLAs.\n\n(3) The paper is clear and easy to read.", "weaknesses": "(1) The analysis of the results could be improved. In particular, it remains unclear what the specific reason is that reasoning trace generation can be avoided at test-time, but yet the benefits of reasoning can still be obtained. In their discussion section the authors wrote that “from analyzing the HyT framework, our understanding is that learning to generate CoT and learning to predict actions from CoT improves the agent’s understanding of the environment” —> could you expand on what it mean to improve the understanding of the environment? I.e., is it that the model can obtain better internal representations if trained with reasoning, or can follow language better, or something else?\n\n(2) Reasoning is perhaps the most appealing when the task involves some complex decision making/logical inference before picking what atomic action to take. It seems that the authors missed trying their approach out on these types of tasks. Indeed from the discussion: “we do not exclude that thoughts generation might be useful at test-time in more complex settings, but this would require evaluating on tasks that require advanced reasoning capabilities, which are currently sparse and rarely adopted in the robotics literature”. It would have been interesting to see whether the hybrid training scheme can bring benefits to these types of tasks as well, or if reasoning at test-time becomes strictly necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T5TeCTOEf4", "reviewer": "Reviewer_PRFm", "summary": "This paper introduces Hybrid Training (HyT), a framework for training Vision-Language-Action models that enables learning from chain-of-thought (CoT) reasoning traces while maintaining fast inference speeds. The key insight is that VLAs can internalize knowledge from thought supervision during training without needing to generate thoughts at test time. HyT trains a single model to conditionally predict three distributions based on a modality variable: direct actions (\"act\"), thoughts then actions (\"think\"), and following provided instructions (\"follow\"). The method uses Monte Carlo sampling during training for the three modes. Experiments on ClevrSkills, LIBERO, and real-world tasks demonstrate that HyT achieves performance gains similar to ECoT while maintaining standard VLA inference speeds (3Hz vs 1Hz for ECoT).", "questions": "- Why does HiRobot+Oracle outperform HyT+Oracle in Figure 4?\n- How were the loss coefficients (0.25, 0.5, 0.25) determined? What is the sensitivity to these values?\n- What is the ablation comparing Monte Carlo sampling versus direct weighted loss computation? Does the sampling variance help or is it just for computational efficiency?\n- Can you provide ablations showing individual contributions of Lthink and Lfollow? What happens with just Lact + Lthink?\n- What are the specific inference speeds compared to π0-FAST, and can HyT be combined with their efficient tokenization approach?\n- In what types of tasks (complexity, horizon length, reasoning requirements) does HyT show the largest improvements over standard VLA training?\n- Do all compared baselines (Table in LIBERO results) use the same PaliGemma-2 backbone for fair comparison?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The hypothesis that models can internalize CoT reasoning without explicit generation is compelling and well-articulated through the System 1/System 2 cognition analogy.\n- HyT outperforms competitive baselines including MolmoAct (86.6% → 93.7%) and π0-FAST (85.5% → 93.7%) on LIBERO, while maintaining 3× faster inference than ECoT.\n- The paper includes extensive experiments across simulated (ClevrSkills, LIBERO) and real-world environments with ablations on data scaling.\n- The ability to maintain ECoT-level performance at standard VLA inference speeds addresses a critical deployment constraint for robotic systems.", "weaknesses": "- The loss coefficients (wa:0.25, wτ:0.5, wf:0.25) appear arbitrary without ablation\n- The Monte Carlo sampling approach is not compared to direct weighted loss computation\n- Individual contributions of Lfollow and Lthink to overall performance are not analyzed\n- HiRobot with oracle thoughts often outperforms HyT with oracle, suggesting the \"follow\" mode may not be optimally implemented. \n- The paper lacks systematic investigation of which task characteristics benefit most from internalized CoT training versus standard VLA training.\n- Unclear if all baselines use the same VLM backbone (PaliGemma-2)\n- No inference speed comparison with π0-FAST (or other baselines)\n- Compatibility with other VLA architectures not explored", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B1tA565ZH3", "reviewer": "Reviewer_zK8v", "summary": "This paper proposes Hybrid Training (HyT), a method that unifies thinking and acting behaviors in a single vision-language-action model via a learnable modality variable controlling “think/act/follow” modes. HyT trains one model to generate both reasoning traces and executable actions, using a Monte-Carlo sampling scheme to mix modalities during training. The method is evaluated across simulated (ClevrSkills, LIBERO) and real-robot (xArm6) environments, demonstrating competitive task success and faster inference compared to full chain-of-thought reasoning.", "questions": "See Weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. HyT uses a modality variable to avoid maintaining separate reasoning and acting modules, simplifying deployment and enabling run-time mode control (think/act/follow) without model-switching.\n2. HyT shows good performance on the ClevrSkills benchmark, suggesting HyT can learn complex, compositional behaviors required there. And robotic experiments on an xArm6 support practical applicability beyond simulation.", "weaknesses": "1. In Figure 4, reported task success and efficiency gains relative to ECoT are small; it remains unclear whether HyT achieves comparable performance at lower latency or merely approximates ECoT under similar compute.\n2. How does HyT compare to the π₀.₅ model (https://github.com/Physical-Intelligence/openpi) and OneTwoVLA (https://arxiv.org/pdf/2505.11917)? Can the BOA/BOR control mechanism (as in OneTwoVLA) achieve similar adaptive switching without retraining?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HJ1k9EYTmB", "reviewer": "Reviewer_LZtX", "summary": "This paper proposes Hybrid Training (HyT), a framework that lets VLAs learn from CoT reasoning for better performance while allowing inference without generating thoughts. HyT flexibly enables direct action prediction, thought generation, or instruction following, and shows strong results in both simulation and real-world experiments.", "questions": "All of my qeustions are listed in the weakness section. If my concerns are well addressed, I will raise my rating.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Performance-speed breakthrough: HyT resolves the key “reasoning vs. reflex” trade-off in VLAs, matching the success rates of CoT methods while maintaining high inference speed (e.g., 3 Hz vs. 1 Hz for ECoT)—a major step toward practical deployment.\n\n2. Robust empirical results: Extensive experiments show HyT’s clear advantages: strong data-scaling on ClevrSkills, state-of-the-art results on LIBERO (especially “Goal” and “Long” tasks), and real-world robot transfer. Its 54% OOD success vs. 29% baseline highlights superior generalization.\n\n3. Versatile and interpretable model: A single HyT-trained model supports fast “act,” interpretable “think,” and guided “follow” modes—allowing deployment, analysis, and human-robot collaboration without retraining.", "weaknesses": "1.  HyT requires CoT or reasoning traces for training—using oracle thoughts on ClevrSkills and Gemma-2 9B-generated plans on LIBERO. This reliance on costly, curated reasoning data limits scalability and makes adaptation to new domains harder than learning directly from demonstrations.\n\n2. While “act” mode matches “think” mode performance, the paper doesn’t test harder, long-horizon, or abstract tasks where explicit reasoning might still be needed. Without such analysis, the boundaries of HyT’s internalized reasoning remain uncertain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)