ViPRA: Video Prediction for Robot Actions
Authors: Sandeep Routray
Hengkai Pan
Unnat Jain
Shikhar Bahl
Deepak Pathak
Venue: ICLR 2026 Poster (2026)
Keywords: vision-language-action models
robotics
video prediction
imitation learning
Primary area: applications to robotics, autonomy, planning
TL;DR: We present ViPRA, a framework that turns video prediction models into robot policies by learning latent action priors from unlabeled videos and refining them with flow-matching to enable high-frequency control with minimal labeled data.

Abstract: Can we turn a video prediction model into a robot policy? Videos, including those of humans or teleoperated robots, capture rich physical interactions. However, most of them lack labeled actions, which limits their use in robot learning. We present *Video Prediction for Robot Actions* (**ViPRA**), a simple pretraining-finetuning framework that learns continuous robot control from these actionless videos. Instead of directly predicting actions, we train a video-language model to predict *both future visual observations and motion-centric latent actions*, which serve as intermediate representations of scene dynamics. We train these latent actions using perceptual losses and optical flow consistency to ensure they reflect physically grounded behavior. For downstream control, we introduce a chunked *flow-matching decoder* that maps latent actions to robot-specific continuous action sequences, using only 100 to 200 teleoperated demonstrations. This approach avoids expensive action annotation, supports generalization across embodiments, and enables smooth, high-frequency continuous control upto 22 Hz via chunked action decoding. Unlike prior latent action works that treat pretraining as autoregressive policy learning, ViPRA explicitly models both what changes and how. Our method outperforms strong baselines, with a 16% gain on the SIMPLER benchmark and a 13% improvement across real world manipulation tasks. We have released models and code [here](https://vipra-project.github.io/).

Published as a conference paper at ICLR 2026
VIPRA: VIDEO PREDICTION FOR ROBOT ACTIONS
Sandeep Routray 1.2
Hengkai Pan"
Unnat Jain 2.31
Shikhar Bahl
Deepak Pathak 1.21
Carnegie Mellon University Skild AI
University of California, Irvine
Denotes joint mentoring
ABSTRACT
Can we turn video prediction model into robot policy? Videos, including those of humans or teleoperated robots, capture rich physical interactions. How- ever, most of them lack labeled actions, which limits their use in robot learning. We present Video Prediction for Robot Actions (ViPRA), a simple pretraining- finetuning framework that learns continuous robot control from these actionless videos. Instead of directly predicting actions, we train a Video-language model to predict both future visual observations and motion-centric latent actions, which serve as intermediate representations of scene dynamics. We train these latent ac- tions using perceptual losses and optical flow consistency to ensure they reflect physically grounded behavior. For downstream control, we introduce chunked flow matching decoder that maps latent actions robot-specific continuous ac- tion sequences, using only 100 to 200 teleoperated demonstrations. This approach avoids expensive action annotation, supports generalization across embodiments, and enables smooth, high-frequency continuous control upto 22 Hz via chunked action decoding. Unlike prior latent action works that treat pretraining as autore- gressive policy learning, ViPRA explicitly models both what changes and how. Our method outperforms strong baselines, with a 16% gain on the SIMPLER benchmark and a 13% improvement across real world manipulation tasks. We have released models and code at https: //vipra http//vppaappojjeccc github.
INTRODUCTION
Robots learn by doing, but collecting robot demonstrations, particularly at scale, expensive, time-consuming, and limited by embodiment. In contrast, videos are abundant. From YouTube clips (Abu-El-Haija et al., 2016) of people performing tasks (Grauman et al., 2022: 2024; Goyal et al., 2017; Damen et al., 2018) to logs of teleoperated robots (Collaboration et al., 2023), they capture rich physical interactions, diverse objects, and long- morizon behaviors that are difficult to script reproduce. The challenge is that most of these videos may not include action labels.
At the same time, recent advances in video prediction models (Liu et al., 2024; Blattmann et al., 2023; Singer et al., 2022; Zhou et al., 2022; NVIDIA et al., 2025a) open up new opportunity: learn- ing directly from large corpora of actionless videos. Beyond preserving high-level task semantics, these generative models exhibit strong grasp of object dynamics and fine-grained physical inter- actions. This naturally leads to a central question: Can a video prediction model be transformed into control policy for physical robots? this work, we explore this question through a simple and scalable pretraining-iiettnigg framework that adapts a powerful video-language model (Liu al., 2024) into a robot policy capable of learning from passive videos.
During pretraining, we co-train on two intuitive objectives: (i) predicting what happens next, in the form of future visual observations, and (ii) predicting how the scene evolves, using a compact intermediate representation known as latent actions'. By training with both objectives, the model learns to capture both semantic intent and physical dynamics. contrast, prior latent action pre- training methods (Ye et al., 2024; Bu et al., 2025; Chen et al., 2024; NVIDIA et al., 2025b) treat
Latent actions can be viewed as action-like like tokens that summarize the transition hetween states without requiring access to ground-truth control commands

---

Published as a conference paper at ICLR 2026
Hunan videos (no action labels)
0443
O242
Video and Latent Action Prediction
Hex1 Oct2
Sinulation benchnarks
Or43
ViPRAD
Unified Latent Actions X
Real
Robot videos (no action labels)
Action Chunk with Flow-Matching
View Agnosticc
Enoss-Enbodied
High Frequencyy Control
We present ViPRA. framework learn generalist robot policies from large-scale actioneless vidcos. It extracts motion-centric latent action sequences, pretrains ridco-language model to jointly predict future visual observations and latent action chunks, and finetunes a flow matching decoder to map these latents to smooth, continuous action chunks with minimal labeled data for smooth, high-frequency control.
Figure 1:
pretraining purely as policy learning in latent space, without leveraging video prediction or mod- eling state transitions, and often use temporally coarse task-centric latent actions. Our framework instead predicts state transitions through video prediction and outputs sequence of fine-grained motion-centric latent actions (3 to 6 Hz) over short horizons, capturing high-frequency dynamics critical for control. We further incorporate optical flow consistency signal, promoting physically plausible and motion-aware latent representations.
additional supervision
Importantly, our pretraining leverages both unlabeled human and robot videos, which enables gen- eralization across embodiments (see Figure 1, left). This broad exposure to passive visual data sets the foundation for effective finetuning with only small number of teleoperated robot demonstra- tions. For finetuning on these demonstrations, we employ a flow matching decoder (Lipman et al., 2022) that maps latent actions smooth, continuous robot action chunks (see Figure 1, right). Un- like prior rision-language-actinn 2024a; Black et al., 2024; Qu al., 2025), which required thousands of hours of labeled action trajectories?, our decoder aligns latent transitions with embodiment-spe -preenfc motor behaviors. This design amortizes inference latency via action chunking, enabling smooth, high-frequency control by producing multiple low-level actions in a single forward pass. Our policy can support control rates upto Hz, to our knowledge matched only by one other 7BB-parameter model (Kim al., 2025).
models (VLAs) (Zitkovich et al., 2023; Kim et al., 2024; Li et al.,
In summary, our contributions are as follows.
(i) A scalable method to extract fine mne-grained motion-centric latent actions from unlabeled hu- man and robot videos using perceptual and optical flow consistency losses.
(ii) A novel pretraining framework for robot control that jointly predicts future visual states and motion-centric latent actions within unified video-language model.
(iii) A data-efficient pretraining-fiettninng framework that integrates flow matching and action chunking to enable smooth, frequeryy continuous control, operating at up to 22 Hz. (iv) Demonstrate empirical gains of 16% on the SIMPLER benchmark (L.i et al., 2024b) and 13% real world tasks over the strongest prior continuous control baselines.
RELATED WORK
Lision-Languagg-Acttio Models VLAs (Zitkovich et al., 2023; Kim et al., 2024; Li et al., 2024a; Black et al., 2024; Qu et al., 2025) extend vasion-language models (VLMs) (Touvron et al., 2023; Chen et al., 2023; Driess et al., 2023; Karamcheti et al., 2024: Beyer et al., 2024) by imitation learning action-labeled robot demonstrations (Collaboration et al., 2023). Recent works explore auxiliary objectives including visual trace prediction (Niu et al., 2024), chain-of-thought reason-
10000 hours of pretraining data (Collaboration al., 2023) and 5-100 hours of finetuning demonstrations
2

---

Published as a conference paper at ICLR 2026
ing (Wei al., 2022), and conversational instruction tuning (Li et al., 2025b). However, all existing VLAs require extensive labeled action data, creating a fundamental scalability bottleneck due to the prohibitive cost of data collection. Furthermore, these models focus primarily grounding language in visual semantics while lacking explicit mechanisms for modeling physical dynamics or temporal structure in action generation. In contrast, ViPRA eliminates the labeled data requirement by leveraging unlabeled vidcos during pretraining and incorporates temporal dynamics through joint prediction of future visual states and multi-step latent actions, which provides robust priors for high- frequency control.
Robot Learning from Videos Videos offer a scalable source of information about object dynamics, task structure, and human behavior. Visual planning methods (Du et al., 2023a; Wu et al., 2024a; Ko et al., 2024; Du et al., 2023b; Baker et al., 2022; Liang et al., 2025b; Luo et al., 2025a) use generative video models to plan in video or video-language space and rely on an inverse dynamics model to convert predicted frames into actions. While effective for long-horizon reasoning. these methods often incur high inference costs, limiting their suitability for high-frequency dexterous control. Different from the above, policy supervision approaches (Luo et al., 2025a;b) use video models as supervision or reward sources train policies, though evaluations have so far focused on simulation or relatively constrained real world tasks.
Recent work explores joint training for video generation and action prediction (Li et al., 2025a; Guo et al., 2024), with Li et al. (2025a) also introducing decoupled action decoding to mitigate inference overhead, but evaluations mostly smaller-scale datasets, simulation, and do not demonstrate scaling to internet-scale passive videos.
Other efforts leverage human videos to pretrain visual representations for downstream visuomotor control (Nair et al., 2023; Dasari et al., 2023; Xiao et al., 2022; Karamcheti et al., 2023), or extract intermediate cues such as affordances (Bahl et al., 2023; Kannan et al., 2023; Bharadhwaj et al., 2023; Liu et al., 2022: Goyal et al.. 2022), interactions (Zeng et al., 2024), or visual traces (Wen et al., 2024: Bharadhwajj et al., 2024; Mandikal & Grauman, 2022: Bahl al., 2022) from unlabeled videos to guide policy learning. These approaches depend on structured priors or explicit cue extraction, which can constrain scalability. In contrast, we learn motion-centrie latent actions that capture temporal dynamics and align them with video-language representations, enabling scalable learning from large action-free video corpora.
Latent Action Spaces Latent action representations improve data efficiency by enabling learning from action-free videos via supe-supe learning (Dwibedi et al., 2018; Liang et al.. 2025a; Sco et al., 2022; Schmidt & Jiang. 2024; Cui et al., 2024). Recent methods impose discrete information bottlenecks with vector quantization encoders (van den Oord et al., 2017) and predict these tokens during policy learning (Ye et al., 2024; Lee al., 2024; Yang al., 2024a; Bu et al., 2025; Chen et al., 2024; NVIDIA et al., 2025b), achieving strong real world results through imitation. Some train inverse dynamics models on limited labeled demonstrations before applying them to unlabeled video (Baker et al., 2022: Xu et al., 2023), while others treat latent actions as abstract embodiments and jointly train policies with inverse dynamics model predictions across embodiments (Jang al., 2025). Another line of work uses these abstractions to build world simulators (Gao et al., 2025; Bruce et al., 2024) or plan in latent spaces (Ha Schmidhuber, 2018; Racanière et al., 2017: Hafner et al., 2019a;b; Lee et al., 2020; Wu et al., 2023; Sekar et al., 2020). While these methods capture physical dynamics effectively, they struggle to generalize to novel settings due to limited seman- tic grounding. Video Video-language models can provide such multimodal grounding (Du et al., 2023a; Ko et al., 2024; Liang et al., 2025b; Guo et al., 2024; Du et al., 2023b), but existing approaches typically computationally heavy and slow inference. In contrast existing methods, ViPRA learns fine- -na-grddddd, motion-centric latent actions that capture temporal dynamics while leveraging a video-language model (Liu et al., 2024) for semantic grounding. We train unified latent space from large-scale, action-free human and robot videos, enabling cross-embodiment transfer. By predict- ing action chunks during both latent pretraining and real-action finetuning, we amortize inference latency and achieve smooth, high-frequency control.
BACKGROUND
We defer discussion on VAE for discrete latent actions, optical flow estimation, behavior cloning, and flow matching for continuous control to Appendix A.
3

---

Published as a conference paper at ICLR 2026
Latent Action Learning
Nultimodal Pretreining
Continucus Finetuninn
Aignent
Parstrin Sing
Fux
Latent Action Prediction
Flow-natching
Predicttio
Forward Decoder
Tokenizer
Latent Action Decoder
Flow Decoder
Latent
aratialy
Large-Scar Video
Language Model
Spatio- Teeporal Transformer
finz
Vestet
DENOu2 Spattal Encoder
Latent Action Enbedding Ep
Action Encoder
Toke
Latent: Actionn Takenizer is
Token
Nitsy Acttons
C: tewer basl with
Hunan videos
Robot videos
franes
Future franes
Robot Image history
Figure ViPRA framework comprises of: (1) Latent Action Learning (left): neural quantization bottleneck extracts discrete latent action sequences Zr++L-1 provided image sequences 0ts++L from action- less human and rohot videns, trained via latent loss Liaeet to capture motion-centric dynamics. (2) Multi- modal Pretraining (center): A video-language model jointly predicts future observations o and latent action sequences R- from past frames (0) and task description c, using loss Creamnn (3) Contin- uous Finetuning (right): A flow matching decoder maps latent actions to continuous robot actions H1 using noisy action conditioning and loss LiM, cnabling smooth, high-frequency control.
4 VIDEO PREDICTION FOR ROBOT ACTIONS
generalist robotic agent must combine precise low-level control with euvironment-agnostee high- level intelligence. Video generation models are well-suited to this goal, as future-state prediction captures both physical interaction detail and task-related semantic context. Achieving this requires effectively utilizing large-scale data, architectures that expose motion-centric signals, and stable training pipelines. To this end, we present ViPRA: (i) learning motion-centric discrete latent actions from large-scale human and robot videos without action supervision, guided by perceptual and op- tical flow consistency; (ii) pretraining a multimodal video-language model to jointly predict future visual observations and latent action sequences, enabling semantic grounding of temporal dynamics; and (ii) finetuning a flow matching decoder that maps latent actions to smooth, continuous control using only few hundred demonstrations. This hierarchy leverages the rich physical priors of video models while ensuring the precision needed for real world robot control.
4.1 LATENT ACTION LEARNING FROM ACTIONLESS VIDEOS
We first train latent action model represent behavior in both human and robot videos without requiring action supervision. As illustrated in Fig. 2 (left), this phase extracts motion-cent latent action sequences that capture how the environment changes (inverse dynamics) and reciprocally also models the visual observations in response to these implicit actions (forward prediction).
Given a length-(L- 1) observation sequence Oct+L sampled from human or robot videos, our ob- jective is to learn latent action sequences --:+--11 where each 2 = [=] ... Numu uses Niment discrete tokens encode the motion dynamics at timestep k. Here, each latent action component =f is quantized from shared codebook C of size [C] 8.
We train an inverse dynamics encoder L=( | 0t.++)) that predicts latent action token 2, by conditioning the full observation sequence Ot.t+L. This non-causal design allows each latent action 2ạ to incorporate both past and future context, making it sensitive to local motion intent-for instance, distinguishing pickup from putdown based on surrounding frames. By providing the encoder with the entire clip, we reduce reconstruction ambiguity and force 2g to encode the minimal but sufficient information explain the local transition.
4

---

Published as a conference paper at ICLR 2026
We jointly train forward decoder (ō Ot.c-k; 0 that predicts the future frame 0s+k++1 given the history of observations OLi+* and latent actions 2t:++k- This reconstruction task ensures that the learned latent actions = contain sufficient information explain scene dynamics. The model is optimized using three complementary loss components: pixel-level L1 reconstruction loss free for accurate frame prediction, perceptual loss LLPPS (Zhang al., 2018) to capture high-level structure beyond pixel accuracy, and optical flow consistency loss Laow to encourage physically plausible motion patterns:
4+L-1 Σ
Laow Σ km1+1
[OF(Ok: ōk-1) OF(Ok, 0k- 0k11)l1
1 L [OF( ōk+1) OF(O,, 0k+1)
)l1
(1)
where OF(a, b) denotes optical flow between frames a and b computed via RAFT (Teed & Deng, 2020). This loss encourages predicted frames to exhibit motion patterns consistent with ground truth, supporting temporally coherent dynamics. The total latent action learning loss combines all components:
Diatent Lise ALPIP [(tte Ortoww Aow
where the flow loss is activated only after Oftow warm-up steps to avoid instability from poor early reconstructions. Thus, latent actions serve as representation of scene dynamics effectively bridging between visual observations and any embodiment-speriin control commands. We provide further architecture details in Appendix B.
4.2 PRETRAINING MULTIMODAL VIDEO MODEL WITH LATENT ACTIONS
After obtaining discrete latent actions, design a pretraining scheme leveraging a powerful multi- modal video prediction model. Such models are trained on large-scale datasets to jointly reconstruct video tokens and predict aligned language captions, thereby encoding rich semantic cues and dy- namic priors about how the world changes in their latent space. By aligning our discrete latent actions with the outputs of the generative model, we can effectively pretrain high-level controller that can learn from video clips.
To this end, we jointly predict future visual tokens and latent actions, unifying dynamic scene under- standing and abstract control representation in temporally coherent latent space. Given the most recent observations (O:-1, 0z) and a task description e, the model predicts a future frame Or+-1 that H steps ahead, along with latent action sequence H [Ze, #t-1: Rt+H-11 representing the transitions leading to O++-H- This multi-step horizon encourages meaningful and distinct scene changes, providing robust conditioning for downstream action inference.
As shown in Fig. 2 (center), we build on the instruction-tunet LWM-Chat-IM (Liu et al., 2024) base policy Gg. We use LWM's VQ-VAE encoder Evo tokenize each frame og into discrete visual tokens I& = Evo(ox). We also extend with two modules for latent action mod- eling: (i) a Latent Action Embedding head Es that maps each discrete latent token z E to de dimeennsion vector z (=t) in the model's token space, and (ii) Latent Action Token Decoder H., multi-layer perceptron (MLP) that autoregressively predicts the next latent token gi+1
from the transformer hidden state till position i. This allows the model to generate latent action sequences in the same autoregressive manner as language or video tokens, leveraging the multimodal token space learned during pretraining.
" H Go (c, T4+ 3et.
During training, we apply teacher forcing to both visual and latent action predictions. Given to- kenized observations xz-1 and It, the model autoregressively predicts the future visual tokens ft-H = Go(c, I2 2.). The ground truth tokens I4+H = Evo(o++H) serve as supervision targets for visual prediction. The pretraining objective Epectrin combines both components
Nasies
t- Numn
Cpeetrain
CB(
CB( =
(3)
Cyn
where CE(a, b) denotes the standard cross-entropy loss between logits for and label b.
5

---

Published as a conference paper at ICLR 2026
4.3 CONTINUOUS ADAPTATION
While the latent action pretrained video model provides robust semantic grounded physical priors, lacks the physical precision needed for smooth, low-level robot control. To address this gap, we augment the pretrained model to output continuous actions. utilizing a flow matching decoder trained on real robot trajectories. This adaptation enables temporally smooth, physically consistent motor commands conditioned on visual and linguistic contexts.
As shown Fig. (right), we augment the video model Go with two action-specific components: (i) an Action Encoder E,,, and (ii) a Flow Decoder Hy. The encoder E, embeds continuous noisy actions u, IR' "xx into the token space, while the decoder H,, predicts a flow field over the action chunk. Following the flow matching framework from Eq. 7, we sample target action sequence Ox RHDD draw a noise sample Mg (O, 7), and interpolate:
uo (1 8) Hn
(eta(a,,)) b).
We use a = 1.5 and = 1 for sampling from Beta distribution. This noisy input is encoded via fa = E, (u,, s). and passed into the transformer along with VQ-encoded image tokens (xt=1, and language prompt The model predicts flow field و H, (Ga (c, 1e, Sa)), which is supervised using the flow matching objective from Eq. 9:
FFM ||aaa Mo (1 8) 011
iteratively solve for the continuous action chunk 0t:+ using forward Euler integration (Eq. 10) of the predicted flow field from 0 to 1, over 10 uniform steps with As 0.1. This continuous control refinement layer injects dynamics consistency and smoothness unavailable to the discrete latent tokens alone.
At inference time, given the visual history (O--1, 0z) and task instruction
5 EXPERIMENTS
To evaluate ViPRA, we conduct extensive experiments in both simulation and the real world to address the following research questions: (i) Can generalist policy be trained to leverage both the physical dynamics and semantic understanding of video models? (ii) Does ViPRA efficiently exploit large-scale, actionless video data? (iii) Can multimodal pretraining yield strong high-level priors for downstream policy? (iv) How well does ViPRA adapt high-frequene, continuous control settings? (v) Does ViPRA outperform methods that do not exploit video foundation models?
5.1 ENVIRONMENTS & TRAINING
Training Dataset For learning latent actions and pretraining the video-language model, we use 198k human videos from Something-Somethiing v2 (Goyal al., 2017) and subset of actionless robot videos from the OpenX (Collaboration et al., 2023) dataset: 87k Fractal (Brohan et al., 2022) videos, 25.4k Bridge (Ebert et al., 2021), and 85.6k Kuka (Kalashnikov et al., 2018) videos. We describe training details and hyperparameters for latent action learning in Appendix B, pretraining in Appendix C, and flow matching finetuning in Appendix D.
Simulation Benchmarks Following prior latent action works (Ye et al., 2024; Bu et al., 2025), we benchmark ViPRA in SIMPLER (Li al., 2024b), an open-source suite for evaluating generalist manipulation policies. We evaluate on four Bridge task with 7-DoF WidowX arm, benchmark designed to test generalization across diverse manipulation goals. Since SIMPLER lacks finetuning data, we collect 100 diverse multi-task trajectories using a pretrained VLA model (Ye et al., 2024). We provide details about our SIMPLER tasks and LIBERO Long benchmarks in Appendix F.
Real World Manipulation While SIMPLER already provides a strong correlation between sim- ulated and real world policy performance, we further strengthen our findings with rigorous eval- uations on physical robots. We evaluate ViPRA on bimanual setup with two 7-DoF Franka Panda robots. For single-arm experiments, we finetune on three multiiinstruction tasks: (1) pick up cloth and cover (object), (2) pick up (object1) and place on (objectz), and (3) pick up (colorı) cup and stack on (colorz) cup. We use GELLO (Wu et al., 2024b) tele- operation to collect 180 trajectories, per task spanning 5 cup colors and 10 object types. For both

---

Published as a conference paper at ICLR 2026
Partial
FULL
Binanual Setup
19.3
24.0
20.1
35.7
23.0
cover cover.ob) obj
pack, place
stack. cup
AVERAGE
cover 001
pack place
stack CUB
AVERAGE
Scratch- PM
Ours- PH
Figure 3: Real World Evaluations (Left) We report full and partial success rales three manipulation tasks. ViPRA-FM significantly outperforms baselines. (Right) We show our physical robot setup and lask objects.
Discrete Actions
Continuous Actions
Scratch-AR
OpenVL.A
LAPA
ViPRA-AR
Scratch-FM
UniPT
me
UniVLA
Success Rates
StackG2Y
54.2
45.8
25.0
33.3
66.7
16.7
0.0
CarrolPPlaee
58.3
37.5
20.8
41.7
62.5
33.3
2.7
20.8
50.0
Spoon2Cleth
37.5
70.8
50.0
66.7
66.7
50.0
4.17
66.7
Fogplan/2Rack
58.3
50.0
58.3
70.8
83.3
66.7
0.0
83.3
79.2
AVG
52.1
51.0
38.6
53.1
69.8
41.7
27.1
42.7
Grasp Rutes
StackG2Y
62.5
62.5
70,8
66.7
66.7
45.8
20.8
12.5
62.5
ClatotPPltec
54.2
37.5
62.5
62.5
45.8
33.2
25.0
54.2
Spoon2Cloth
75.0
79.2
75.0
87.5
75.0
62.5
22.2
16.7
79.2
Bggplani2Bask
66.7
70.8
91.7
79.2
100
87.5
16.0
91.7
91.7
AVG
65.6
66.7
68.8
73.9
76.1
60.4
23.1
36.5
50.0
71.9
Table 1: We report success rates and grasp rates on four bridge tasks in SIMPLER benchmark suite.
simulation and real world settings, we report full success and partial success; partial success is de- fined as grasping the correct object, and full success requires completing the task (e.g.. placing, stacking, covering). We evaluate with both seen and unseen objects, textures, colors, and shapes to test generalization. For real world evaluation, policies run using only a front-facing camera. We pre- dict action chunks of length =/114 and replan after executing the first 7 steps. For this evaluation, we cap our policies at an effective closed-loop control rate of 3.5 Hz, though they can also operate at higher frequencies upto 22 Hz.
5.2 BASELINES
We evaluate ViPRA against strong baselines across discrete and continuous action formulations.
Scratch. As a reference, Scratch finetunes the LWM (Liu et al., 2024) video-language backbone directly downstream tasks with image history and action chunking. without any pretraining. establishes baseline performance when no latent action or video-based pretraining is used.
VLA baselines. We include OpenVLA (Kim et al., 2024) and no (Black et al., 2024). Open discretizes actions and adds a one-step autoregressive (AR) action predictor on top of Prismatic- 7B (Karamcheti et al., 2024), while To augments PaliGemma-3B (Beyer al., 2024) with chun- ked flow matching (FM) decoder. Both use al., 2023) containing 970k trajectories, while To also uses proprictary robot data.
robot demos from OpenX (Collaboration
Latent action baselines. We include LAPA (Ye et al., 2024) and UniVLA (Bu et al., 2025), both of which learn one-step temporally coarse latent actions without video prediction during pretrain- ing. UniVLA improves upon LAPA by learning language-cone coniiti--od task-centric actions in DI- NOv2 (Oquab et al., 2023) space. UniVLA uses a Prismatic-7B backbone with a LI action decoder, whereas LAPA uses an LWM backbone with one-step AR prediction. Both rely OpenX demos, with UniVLA additionally leveraging Ego4D (Grauman et al., 2022) and GNM (Yang et al., 2024b).
Video learning baselines. We include UniPI (Du et al., 2023b) and VPT (Baker et al., 2022), both of which leverage videos for pretraining. UniPI trains a video diffusion model and trains an inverse dynamics model (IDM) to recovers actions, while VPT trains IDM labeled data extract pseudo-actions that are then used to pretrain an LWM backbone. Reported results are from (Ye et al., 2024), which evaluated them SIMPLER in comparable setting.
We include both ViPRA-AR, aligned with discrete autoregressive baselines, and ViPRA-FM, aligned with continuous flow matching methods.

---

Published as a conference paper at ICLR 2026
Vertical Hortontal
Left-Rightt Right-Laft
Up Down
Tolaoo Posttonn
0001
Tokeen Poetion
001
12 10 6
6
0000
0.0
0.
12 10
-2.05
Codebook
Codaboook
Figure 4: Positional codebook usage differences across action categories. Each heatmap shows the difference per-position token usage between two groups: (left) vertical horizontal, (middle) left right vs. right - left, and (right) up vs. down. ViPRA learns positionally sensitive codes, with certain entries (e.g.. 0, 2, S) showing systematic variation, indicating that both token index and positions encode action dynamics.
5.3 SIMULATION RESULTS
Table shows that ViPRA achieves the highest average success rate across both discrete and con- tinuous settings. In the discrete setting. ViPRA-AR surpasses LAPA and OpenVI.A by a large margin (69.8%, vs. 53.1 and 38.6%), with particularly strong performance on precision-heavy tasks such as StackG2Y. These results indicate that temporally fine-grained latent actions improve contact-sensitive manipulation. In the continuous setting, ViPRA-FM outperforms Scratch-FM by 20.8%, To by 35,4%, and UniVLA by 19.8% This suggests that motion-centria latent pretraining combined with joint future-state modeling provides stronger control priors than both training from scratch and temporally coarse latent approaches. Although ViPRA-AR slightly exceeds ViPRA-FM in the low-noise simulation regime, likely due to faster convergence, ViPRA-FM remains compet- itive and surpasses all other continuous and discrete baselines. ViPRA also surpasses other video learning, i.e., UniPI and VPT. UniPI frequently generates action sequences that diverge from the given instruction in longer-horizon settings, while VPT provides only limited gains, indicating that IDM-derived pseudo-labels are sensitive to environment shifts. In contrast, ViPRA's joint use of la- tent action prediction and future state modeling yields stronger cross-enviromment transfer and more reliable task execution.
5.4 REAL WORLD RESULTS
Figure reports performance on three real world manipulation tasks. ViPRA-FM achieves the high- est average success rate at 54. 1 %, outperforming =n (40) and Scratch-FM (23.8%). Notably, this performance is obtained using substantially fewer labeled demonstrations, indicating that dynamics priors learned from actionless videos translate effectively to high-frequency continuous control. We also observe robust retry behavior: after failed grasps, the policy re-attempts interaction, leading to high partial success rates, particularly in Cover-Obj, where the cloth is reliably grasped even when placement remains challenging. We exclude discrete policies from real-world evaluation, as bin-based action predictions exhibited unstable spikes under physical noise and occasionally trig- gered safety stops on the Franka arm. In Appendix we provide a detailed comparison of action trajectories across discrete and continuous policies, analyzing the effects of quantization and loss formulations on smoothness and deployment behavior. We further report robustness and generaliza- tion analyses in Appendix G, along with evaluations on challenging bimanual tasks in Appendix H. Although the pretraining corpus contains only single-arm robot clips and human videos, ViPRA transfers effectively to coordinated dual-arm settings, suggesting that motion-centric latent pretrain- ing provides embodiment-agnostic priors that adapt with minimal finetuning.
5.5 ABLATIONS ANALYSIS
Isolating effect future state and latent prediction. Table disentangles the contributions of future state prediction and latent action chunking. The LAPA baseline (latent-only) reaches 53.1 %, while adding state prediction ViPRA-AC boosts performance to 59.2%, showing that anticipating future observations improves control even with 1-step latents. Removing state prediction from setup ViPRA-SP2 causes a drop from 69.8% to 59.4% (AR) and from 62.5% to 53.2% (FM), state-only variant ViPRA-LA achieves 60.7%,
underscoring its importance for policy transfer. comparable to ViPRA-AC but still below the full model. indicating that state prediction alone is not sufficient. Finally, adding state prediction at finetuning ViPRA+SP3 degrades performance (53.1
8

---

Published as a conference paper at ICLR 2026
(a) Vertical motion transfer (down - up).
(b) Horizontal motion transfer (right left).
Figure 5: Latent action transfer rollouts illustrating how latent actions extracted from one video can be transferred to different video. Each column organized as: Top shows the source clip og used extract latent actions zd's = (o(0))) using the inverse model. Middle shows the target clip 06s4 with different action which provides a new scene Bottom shows the transferred rollout, generated by the forward model on B (a 20:3), where the latent actions simulated from the first frame of the target clip. This produces novel behaviors in which the target scene undergoes the source motion.
Joint
36lee
Exp.
Pretrain
Finetune
Succ.
LAPA
1-step L
1-step A
53.1
ViPRA-AR
FS H-step L
H-step
69.8
AC
FS 1-step
1-step A
59.2
SPP
H-step
H-step
59.4
VIPRO Laurs)
-LA
FS only
H-step A
60.7
Jodnt
Joleet
+SP3
H-step
FS + H-step A
53.1
ViPRA-FM FS + H-step L H-step
62.5
-AC
FS + 1-step L
1-step A
44.8
-SP2
H-step L
H-step A
53.2
+SP3
H-step
FS+ HI-ttp A
31.3
Timc Stops
Figure 6: Action smoothness: ViPRA-FM (blue) produces smooth, continuous trajectories, while LAPA (green) exhibits local discontinuities and random spikes, often around contact events, despite tracking the overall trend. During real world deployment, such dis- continuitics triggered the emergency brake mechanism of the robot due to abrupt motor torque jumps.
Table 2: Ablation study: Average success rates in SIMPLER on four Bridge tasks, evaluating the con- tributions of future state prediction (FS), latent ac- tion prediction (L), and action chunking (chunk size #=14) to justify ViPRA's 's design choices.
Exp.
Suce. Action MSE
Co-train (Human Robot)
0.79
0.84
Exp.
Perplexity Entropy Action MSE
Robot-only
0.72
0.91
Default (with flow) No flow loss
5.01
1.59
0.84
Human-only
0.69
0.99
5,63
1.74
0.92
Table Data composition ablation comparing Table 4: Ablation on optical flow consistency Lao loss human-only, robot-only, and co-train on LIBERO-10. during latent action learning. Lower values are better.
AR, 31.3% FM), since the autoregressive structure couples action prediction with video prediction, causing compounding errors that drift irrecoverably on -uitof-distribution states. ViPRA mitigates this by jointly predicting future visual tokens and latent action chunks during pretraining only.
Effect of action chunking. We apply chunking (Zhao al., 2023) in both latent and real action spaces: during pretraining the model predicts latent action sequences, and during finetuning it out- puts continuous chunks via flow matching. Removing chunking ViPRA -AC reduces performance to 59.2% (AR) and 44.8% (FM), single-step actions fail to capture fine temporal dynamics. By combining chunking with future state prediction, the full model achieves the best results of 69.8% (AR) and 62.5% (FM), showing that the two objectives complement each other. Finally, action chunking is not only critical in pretraining but also enables robust, high-frequent time. With KV caching, ViPRA's flow matching decoder runs at 1.95 Hz per chunk, supporting ef- fective control rates up to 22 Hz on hardware (chunk size 14). We provide additional discussion on the connection between action chunking and nigh-frequency execution in Appendix G.5.
control at test
ViPRA enables smooth continuous control. To evaluate action smoothness, we compare ViPRA- FM (blue) against LAPA (Ye al,, 2024) (green), discrete latent policy, during closed-loop rollouts

---

Published as a conference paper at ICLR 2026
in Figure 6. Both models are deployed within the same inference pipeline and and replayed over trajecto- ries from the finetuning dataset, simulating execution under real visual observations. Although both methods capture the overall motion trend. LAPA exhibits sharp local spikes, particularly at contact events or under partial occlusions, where small perceptual changes induce abrupt bin transitions. In contrast, ViPRA-FM's flow matching decoder produces smooth, demonstration-alignee by modeling actions in continuous space. Because such discontinuities can be unsafe on real hard- ware, we restrict physical robot comparisons continuous baselines. Additional analysis of discrete and continuous policies is provided in Appendix F, where we examine the effects of quantization, loss design, and action parameterization on control smoothness and deployment stability.
commands
Latent action analysis. In Figure 5, a cross-video rollout test illustrates the correlation between latent actions and real action dynamics: injecting latents encoding upward motion from one video (top) into the opening frame of downward moving video (middle) causes the reconstructions to move upward (bottom), demonstrating transferable, dynamics-aware semantics. Finally, Figure analyzes codebook usage across categories by computing token-position x code-index histograms. The results indicate that both the choice of codebook entry and its position within the latent action sequence encode structured information about motion direction and dynamies. To further test how motion semantics are shaped during training, we ablate the optical flow consistency loss. As shown in Table 4, removing this supervision increases perplexity and entropy and lowers the accuracy of action prohe with frozen backbone to BridgeV2 actions, confirming that flow alignment encourages the latent space to encode motion rather than background appearance.
Human and robot data composition ablation. We repeat latent action learning and pretraining using only robot only human videos and compare them our co-training setup. As shown in Table 3, human-only training still yields positive transfer on LIBERO-10, indicating that short- horizon motion cues from human videos remain useful despite the embodiment gap. Robot-only training performs better due to closer kinematic alignment. Co-training gives the best results, with the highest LIBERO-10 success rate and lowest action-probe MSE, suggesting that human videos add motion diversity while robot videos ground the latents in robot dynamics.
LIMITATIONS AND FUTURE WORK
ViPRA shows that motion-centric latent actions learned from large-scale actionless videos can form strong foundation for generalist robot policies. Combined with video-language pretraining and flow matching adaptation, this enables smooth, high- high-frequency control with limited labeled data. While effective across simulation and real robots, several limitations remain and point to important directions for future research.
Embodiment and dexterity. Our approach generalizes across embodiments such as WidowX and Franka, including bimanual setups, but extending to more dexterous domains like dual-arm hu- manoids or multi-fingered hands introduces new challenges. These platforms demand finer latent actions capable of representing precise contact dynamics and inter-arm coordination. Incorporat- ing tactile force feedback, learning embodiment-specific adapters, may allow latent actions to better capture such motion primitives.
Data scope and generalization. Although ViPRA leverages large-scale actionless videos for pre- training, the diversity of available data remains limited. Expanding to richer ego-centric datasets with human-hand interactions, tool use, or short navigation episodes could improve embodiment invariance and temporal grounding. Integrating additional sensing modalities such as wrist cam- eras, proprioception, tactile feedback, or depth can further strengthen perception-control alignment, especially in dynamic or unstructured environments.
Scaling behavior and data efficiency. An open question how passive video pretraining scales and where diminishing returns emerge. Characterizing scaling behavior and balancing unlabeled video with smaller quantities of demonstrations would clarify efficient paths for large-scale robot learning.
Predictive modeling. The latent action decoder can also be interpreted as a predictive world model. By sampling latent actions, the model can generate visual rollouts, which naturally sup- port -eaarrennnn--nniinngg.... basedd alignment Bai et al. (2022); Rafailov et al. (2023) and test time planning with VL---based reward models (Ma al., 2024).
10

---

Published as a conference paper at ICLR 2026
7 REPRODUCIBILITY
All models and datasets used in our work are taken from open-sourced components. We take the publicly released LWM-Chat-IM (Liu et al., 2024) on top of that. For latent action learning and pretraining video-language model, we use pub- licly available datasets: Something-Something (Goyal al., 2017), Fractal (Brohan et al., 2022), BridgeV2 (Ebert et al., 2021), and Kuka (Kalashnikov et al., 2018). For SIMPLER (Li et al., collected finetuning trajectories by deploying a pretrained VLA model (Ye
base video-language model and build
2024b) benchmarks, et al., 2024). We will release this dataset for the community to reproduce our benchmarks. More- over, we describe training details, hyperparameters. and model architecture for latent action learn- ing in Appendix B, pretraining in Appendix C, and flow matching finetuning in Appendix D. We have released code, models, latent action labeled pretraining data and benchmark scripts at hatps://vvpraappojeeeee github.io.
ACKNOWLEDGMENTS
We thank Jason Liu and Tony Tao for their assistance in conducting the robot experiments, and Jim Yang. Mohan Kumar Srirama, and Tal Daniel for helpful discussions and feedback. This work was supported in part by the Air Force Office of Scientific Research (AFOSR) under Grant No. FA9550- 23-1-0747 and by the Office of Naval Research (ONR) MURI under Grant No. N00014-24-1-2748.
REFERENCES
Sami Ha-jaa, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijaanarasimman.. YouTube-SM: A Large-Scale Video Classifica- tion Benchmark. arXiv preprint MxXiv:160.088677 2016.
Shikhar Bahl, Abhinav Gupta, and Deepak Pathak. Human-to-Robot Imitation the Wild. In Robotics: Science and Systems, 2022.
Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak. Affordances from Human Videos as Versatile Representation for Robotics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 13778-13790, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-oodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. 2022.
arXiv preprint arXiv:2204.0525
Bowen Baker, 1lge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (VPT): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639-24663 2022.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang. Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alcxey Grit- senko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos. Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Bal- azevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai. PaliGemma: A versatile 3B VLM for transfer. arXiv preprint arXiv:2407.07726. 2024.
Homanga Bharadhwaj, Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. Zero-Shot Robot Manipulation from Passive Human Videos. arXiv preprint arXiv:2302.0200 2023.
Homanga Bharadhwaj. Roozbeh Mottaghi, Abhinav Gupta, and Shubham Tulsiani. Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulation. arXiv preprint arXiv:2405.01522 2024.
11

---

Published as a conference paper at ICLR 2026
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. To: A Vision- Mision-Languagg-Acttin Flow Model for General Robot Control. arXiv preprint arXiv:441022414 2024.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable Video Diffusion: Scaling Latent Video Diffusion Models Large Datasets. arXiv preprint urXiv:2311.15122 2023.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalash- nikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deek- sha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong. Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-1: Robotics Transformer for Real-World Control at Scale. arXiv preprint (rXiv:2212.068/1 2022.
Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai. Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Bechtle, Fer- yal Behbahani, Stephanic Chan, Nicolas Heess, Lucy Gonzalez, Simon Osindero, Sherjil Ozair, Scott Reed, Jingwei Zhang, Konrad Zolna, Jeff Clune, Nando de Freitas, Satinder Singh, and Tim Rocktäschel. Genie: Generative Interactive Environments. arXiv preprint arXiv:2402.1 15391, 2024.
Qingwen Bu, Yanting Yang, Jisong Cai, Shenyuan Gao, Guanghui Ren, Maoqing Yao, Ping Luo, and Hongyang Li. UniVLA: Learning to Act Anywhere with Task-centric Latent Actions. In Robotics: Science and Systems, 2025.
Xi Chen. Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Car- los Riquelme Ruiz, Sehastian Goodman, Xiao Wang, Yi Tay, et al. Pal.IX: On Scaling Up Multilingual Vision and Language Model. arXiv preprint arXiv:2305. 18565, 2023.
Yi Chen, Yuying Ge, Yizhuo Li, Yixiao Ge, Mingyu Ding, Ying Shan, and Xihui Liu. Moto: Latent Motion Token as the Bridging Language for Robot Manipulation. arXiv preprint arXiv:2412.0444. 04422, 2024.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. In Roboties: Science and Systems, 2023.
Collaboration, Abby O'Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishck Padalkar, Abraham Lee, Acom Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog. Alex Irpan, Alexander Khazatsky, Anant Rai, An- chit Gupta, Andrew Wang, Andrey Kolobov, Anikait Singh, Animesh Garg, Aniruddha Kem- bhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan. Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Ed- ward Johns, Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Felipe Vieira Fru- jeri, Freek Stulp. Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guangwen Yang, Guanzhi Wang, Hao Su, Hao- Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik I Christensen, Hiroki Furuta, Homanga Bharadhwaj, Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jachyung Kim, Jaimyn Drake, Jan Peters, Jan Schnei- der, Jasmine Hsu, Jay Vakil, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu. Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu,
12

---

Published as a conference paper at ICLR 2026
Jingpei Lu, Jingyun Yang, Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang. Jordi Salvador, Joseph J. Lim, Junhyek Han, Kaiyuan Wang, Kan- ishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrn, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Ku- nal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen, Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi "Jim" Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Cas- tro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip, Mingtong Zhang, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Muhammad Zubair Irshad, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil Joshi, Niko Suenderhauf, Ning Liu, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R Sanketi, Patrick "Tree" Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fa- gan, Peter Mitrano, Pierre Sermanet, Picter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, and Roberto Martin-Martin. Open X- X-Embodiment: Robotic Learning Datasets and RT-X Models, arXiv preprint ar arXiv:2310.08866 2023.
Zichen Cui, Hengkai Pan, Aadhithya Iyer, Siddhant Haldar, and Lerrel Pinto. DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Control. Advances in Neural Information Processing Sys- tems, 37:33933-33661, 2024.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella. Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling Egocentric Vision: The SPIC-KITCHENS Dataset. In Proceedings the European Conference on Computer Vision, pp. 720-736, 2018.
Sudeep Dasari, Mohan Kumar Srirama, Unnat Jain, and Abhinav Gupta. An Unbiased Look at Datasets for Visuo-Motor Pre-training. In Conference on Robot Leurning, pp. 1183-1198. PMLR, 2023.
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. PaLM-E: An Embodied Multimodal Language Model. arXiv preprint 230.:23333.3337... 2023.
Yilun Du, Mengjiao Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B Tenenbaum, et al. Video Language Planning. arXiv preprint arXiv:2310.10625, 2023a.
Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning Universal Policies via Text-Guided Video Generation. Advances in Neural Information Processing Systems, 36:9156- 16-55672 2023b.
Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding Pretraining in Reinforcement Learning with Large Language Models. arXiv preprint arXiv:2302.06669 06692, 2023c.
Debidatta Dwibedi, Jonathan Tompson, Corey Lynch, and Pierre Sermanet. Learning Actionable Representations from Visual Observations. arXiv preprint 1xXiv:/808.0092: 2018.
Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniilidis, Chelsea Finn, and Sergey Levine. Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets. arXiv preprint arXiv:2109.13333 2021.
Shenyuan Gao, Siyuan Zhou, Yilun Du, Jun Zhang, and Chuang Gan. AdaWorld: Learning Adapt- able World Models with Latent Actions. In International Conference on Machine Learning. 2025.
Mohit Goyal, Sahil Modi, Rishabh Goyal, and Saurabh Gupta. Human Hands Probes for Inter- active Object Understanding. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 3293-3303, 2022.
13

---

Published as a conference paper at ICLR 2026
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag. et al. The "Something Something" Video Database for Learning and Evaluating Visual Common Sense. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5842-5850, 2017.
Kristen Grauman, Andrew Westbury. Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger. Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4D: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18995-19012, 2022.
Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos
Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, al. Ego-Exo4D: Understanding skilled human activity from first- and third-person perspectives. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 19383-19400, 2024.
Yanjiang Guo, Yucheng Hu, Jianke Zhang, Yen-Jen Wang. Xiaoyu Chen, Chaochao Lu, and Jianyu Chen. Prediction with Action: Visual Policy Learning via Joint Denoising Process. Advances in Neural Information Processing Systems, 37:1 123811286--21 1108866-11410,, 2024.
David Ha and Jürgen Schmidhuber. World models. arXiv preprini arXiv: 1803. 10122, 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to Control: Learn- ing Behaviors by Latent Imagination. arXiv preprint arXiv: 19/2.0/00. 2019a.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning Latent Dynamics for Planning from Pixels. In International Conference on Machine Learning, pp. 2555 2565. PMLR, 2019b.
Haoran He, Yang Zhang. Liang Lin, Zhongwen Xu, and Ling Pan. Pre-Trained Video Generative Models World Simulators. arXiv preprint arXiv:2502.07825, 2025.
Joel Jang, Seonghyeon Ye, Zongyu Lin, Jiannan Xiang, Johan Bjorck, Yu Fang. Fengyuan Hu, Spencer Huang, Kaushil Kundalia, Yen-Chen Lin, Loic Magne, Ajay Mandlekar, Avnish Narayan, You Liang Tan, Guanzhi Wang, Jing Wang, Qi Wang, Yinzhen Xu, Xiaohui Zeng, Kaiyuan Zheng, Ruijie Zheng, Ming- Yu Liu, Luke Zettlemoyer, Dieter Fox, Jan Kautz, Scott Reed, Yuke Zhu, and Linxi Fan. DreamGen: Unlocking Generalization in Robot Learning through Video World Models. arXiv preprint arXiv:2505. 12705, 2025.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Iharz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable Deep Reinforce- ment Learning for -aseed Robotic Manipulation. In Conference Robot Learning, pp. 651-673. PMLR, 2018.
Aditya Kannan, Kenneth Shaw, Shikhar Bahl, Pragna Mannam, and Deepak Pathak. DEFT: Dex- terous Fine-Tuning for Hand Policies. In Conference on Robot Learning, pp. 928-942. PMLR, 2023.
Siddharth Karamcheti, Suraj Nair, Annie S Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang. Language-Drrveen Representation Learning for Robotics. arXiv preprint 126.. 2023.
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic VLMs: Investigating the Design Space of Visually-Condttioned Language Models. In International Conference on Machine Learning, 2024.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong. Thomas Kollar, Ben- jamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Open VLA: An Open-Source Vision- Lision-Language-Actinn Model. arXiv preprint 24060220000 2024.
Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-Tuning Vision Lagiio-Lnnguage-Acttee Models: Opti- mizing Speed and Success, arXiv preprint 2966:5502./16455. 2025.
14

---

Published as a conference paper at ICLR 2026
Po- To-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B Tenenbaum. Learning Act from Actionless Videos through Dense Correspondences. In International Conference on Learn- ing Representations, 2024.
Wilhelm Kutta. Beitrag zur nüthrunggsweise Integration totaler Differentialgleichunnen. Teubner, 1901.
Alex X Lee. Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic Latent Actor Critic::: Deep Reinforcement Learning with a Latent Variable Model. Advances in Neural Information Processing Systems, 33:741-752, 2020.
Seungjae Lee, Yibin Wang, Haritheja Etukuru, H Jin Kim, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Behavior Generation with Latent Actions. In International Conference on Machine Learning, pp. 26991-27008. PMIR, 2024.
Qixiu Li. Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, et al. CogACT: A Foundational Asiio-Languaag-Acttne Model for Synergizing Cognition and Action in Robotic Manipulation. arXiv preprint arXi::2411.19650 2024a.
Shuang Li, Yihuai Gao, Dorsa Sadigh, and Shuran Song. Unified Video Action Model. arXiv
preprint arXiv:2503.00200, 2025a.
Xiang Li, Cristina Mata, Jongwoo Park, Kumara Kahatapitiya, Yoo Sung Jang, Jinghuan Shang, Kanchana Ranasinghe, Ryan Burgert, Mu Cai, Yong Jae Lee, and Michael S. Ryoo. LLaRA: Supercharging Robot Learning Data for Vision-Language Policy. In International Conference on Learning Representations, 2025b.
Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong. and Ted Xiao. Evaluating WorldWorld Robot Manipulation Policies in Simulation. arXiv preprint 2rXiv:2405.0594 2024b.
Anthony Liang, Pavel Czempin, Matthew Hong, Yutai Zhou, Erdem Biyik, and Stephen Tu. CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrations. arXiv preprint arXiv:2505. 250.0999000 2025a.
Junbang Liang, Ruoshi Liu, Ege Ozguroglu, Sruthi Sudhakar, Achal Dave, Pavel Tokmakov, Shu- ran Song, and Carl Vondrick. Dreamitate: Real-World Visuomotor Policy Learning via Video Generation. In Conference on Robot Learning, pp. 3943-3960. PMLR, 2025b.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow Matching for Generative Modeling. arXiv preprint arXiv:2210.02747. 2022.
Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning, arXiv preprint arXiv:2306.033 03310, 2023.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World Model on Million-Length Video and Language with Blockwise RingAttention. arXiv preprint arXhv:2402.08268 2024.
Shaowei Liu, Subarna Tripathi, Somdeb Majumdar, and Xiaolong Wang. Joint hand motion and interaction hotspots prediction from egocentric videos. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 3282-3292 2022.
Calvin Luo, Zilai Zeng, Yilun Du, and Chen Sun. Solving New Tasks by Adapting Internet Video Knowledge. In International Conference on Learning Representations, 2025a.
Calvin Luo, Zilai Zeng, Mingxi Jia, Yilun Du, and Chen Sun. Self-Adapting Improvement Loops for Robotic Learning. arXiv preprint arXiv:2506.06658 2025b.
Yecheng Jason Ma, Joey Hejna, Ayzaan Wahid, Chuyuan Fu, Dhruv Shah, Jacky Liang, Zhuo Xu, Sean Kirmani, Peng Xu, Danny Driess, Ted Xiao, Jonathan Tompson, Osbert Bastani, Dinesh Jayaraman, Wenhao Yu, Tingnan Zhang, Dorsa Sadigh, and Fei Xia. Vision-Language Models Are In-Context Value Learners. In International Conference Learning Representations, 2024.
15

---

Published as a conference paper at ICLR 2026
Priyanka Mandikal and Kristen Grauman. DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video. In Conference on Robot Learning, Pp. 651-661. PMLR, 2022.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3M: A Uni- versal Visual Representation for Robot Manipulation. In Conference Robot Learning, pp- 892-909. PMLR, 2023.
Dantong Niu, Yuvan Sharma, Giscard Biamby, Jerome Quenum, Yutong Bai, Baifeng Shi, Trevor Darrell, and Roei Herzig. LLaRVA: Vision-Action Instruction Tuning Enhances Robot Learning. arXiv preprint arXiv:2406. 11.4406.11822 1115, 2024.
NVIDIA, Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvi- jit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, Daniel Dworakowski, Jiaojiao Fan, Michele Fenzi, Francesco Ferroni, Sanja Fidler, Dieter Fox, Songwei Ge. Yunhao Ge, Jinwei Gu, Siddharth Gururani, Ethan He, Jiahui Huang, Jacob Huffman, Pooya Jannaty, Jingyi Jin, Seung Wook Kim, Gergely Klár, Grace Lam, Shiyi Lan, Laura Laixe Anqi Li, Zhaoshuo Li, Chen-Hsuan Lin, Tsung- Yi Lin, Huan Ling, Ming- Yu Liu, Xian Liu, Alice Luo, Qianli Ma, Hanzi Mao, Kaichun Mo, Arsalan Mousavian, Seungjun Nah, Sriharsha Niverty, David Page, De- spoina Paschalidou, Zeeshan Patel, Lindsey Pavao, Morteza Ramezanali, Fitsum Reda, Xiaowei Ren, Vasanth Rao Naik Sabavat, Ed Schmerling, Stella Shi, Bartosz Stefaniak, Shitao Tang, Lyne Tchapmi, Przemek Tredak, Wei-Cheng Tseng, Jibin Varghese, Hao Wang. Haoxiang Wang. Heng Wang, Ting-Chun Wang, Fangyin Wei, Xinyue Wei, Jay Zhangjie Wu, Jiashu Xu, Wei Yang. Lin Yen-Chen, Xiaohui Zeng, Yu Zeng, Jing Zhang, Qinsheng Zhang, Yuxuan Zhang, Qingqing Zhao, and Artur Zolkowski. Cosmos World Foundation Model Platform for Physical AL. arXiv preprint arXiv:2501.03575 2025a.
NVIDIA, Johan Bjorck, Fernando Castañeda, Nikita Cherniade,, Xingye Da, Runyu Ding, Linxi "Jim" Fan, Yu Fang. Dieter Fox, Fengyuan Hu, Spencer Huang, Joel Jang. Zhenyu Jiang, Jan Kautz, Kaushil Kundalia, Lawrence Lao, Zhiqi Li, Zongyu Lin, Kevin Lin, Guilin Liu, Edith Llontop, Loic Magne, Ajay Mandlekar, Avnish Narayan, Soroush Nasiriany, Scott Reed, You Liang Tan, Guanzhi Wang, Zu Wang, Jing Wang, Qi Wang. Jiannan Xiang, Yuqi Xie, Yinzhen Xu, Zhenjia Xu, Seonghyeon Ye, Zhiding Yu, Ao Zhang, Hao Zhang, Yizhou Zhao, Ruijie Zheng. and Yuke Zhu. GROOT N1: An Open Foundation Model for Generalist Humanoid Robots. arXiv preprint XIXv:25533.14344 2025b.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nico- las Ballas, Wojciech Galuba, Russell Howes, Po- Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve. Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning Robust Visual Features without Su- pervision. arXiv preprint arXiv:2304.07193, 2023.
Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang, Jia Yuan Gu, Bin Zhao, Dong Wang, and Xuelong Li. SpatialVLA: Exploring Spatial Representations for Visual-Language-ACri
Model. arXiv preprint arXiv:2501.15831 2025.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin. Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 Technical Report. arXiv preprint arXiv:2412.151155 2024.
Sébastien Racanière, Théophane Weber, David Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adrià Puigdomènech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, Raz- van Pascanu, Peter Battaglia, Demis Hassabis, David Silver, and Daan Wierstra. Imagination- Augmented Agents for Deep Reinforcement Learning. Advances in Neural Information Process- ing Systems, 30, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya
16

---

Published as a conference paper at ICLR 2026
Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Inter- national Conference on Machine Learning, pp. 8748-8763, PMLR, 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly Reward Model. Ad- vances Neural Information Processing Systems, 36:53728-537411 2023.
Carl Runge. Über die numerische Auflösung von Differrentalgleichungen Mathematische Annalen, 46(2):167-178, 1895.
Dominik Schmidt and Minqi Jiang. Learning to Act without Actions. In International Conference on Learning Representations. 2024.
Ramanan Sekar, Olch Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to Explore via Self-Superviedd World Models. In International Conference on Machine Learning. pp. 8583 8592. PMLR, 2020.
Younggyo Seo, Kimin Lee, Stephen I. James, and Pieter Abbeel. Reinforcement Learning with Action-free Pre-training from Videos. In International Conference on Machine Learning, pp. 19561-19579, PMLR, 2022.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. Make-A- A-iieo: Text-to-Videe Generation without Text- Video Data. arXiv preprint arXiv:2209. 14792, 2022.
Zachary Teed and Jia Deng. RAFT: Recurrent All-Pairs Field Transforms for Optical Flow. In In Proceedings of the European Conference on Computer Vision, pp. 402-419. Springer, 2020.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura. Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie. Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv preprint
2023.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features. arXiv preprint arXiv:2502. 14786, 2025.
Mohammad Hassan Vali and Tom Bäckström. NSVQ: Noise Substitution in Vector Quantization for Machine Learning. IEEE Access, 10:13598- 13610, 2022.
Aaron van den Oord, Oriol Vinyals, et al. Neural Discrete Representation Learning. Advances in Neural Information Processing Systems, 30:6309-633 6318, 2017.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Prompinnn Elicits Reasoning Large Language Models.
Le. and Denny Zhou. Advances in Neural Information Processing Systems,
2022.
Chuan Wen, Xingyu Lin, John So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-Point Trajectory Modeling for Policy Learning. In Robotics: Science and Systems, 2024.
Hongtao WU, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li. Minghuan Liu, Hang Li, and Tao Kong. Unleashing Large-Scale Video Generative Pre- Training for Visual Robot Manipulation. In International Conference Learning Representations, 2024a.
17

---

Published as a conference paper at ICLR 2026
Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg, DayDreamer: World Models for Physical Robot Learning. In Conference on Robot Learning, pp. 2240,, 2023.
Philipp Wu, Yide Shentu, Zhongke Yi, Xingyu Lin, and Pieter Abbeel. GELLO: General, Low- Cost, and Intuitive Teleoperation Framework for Robot Manipulators. In 2024 JEEE/RSJ Inter- national Conference on Intelligent Robots and Systems, pp. 12156-12163. IEEE, 2024b.
Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked Visual Pre-training for Motor Control. arXiv preprint arXiv:2203.06175 2022.
Liang Xu, Ziyang Song, Dongliang Wang. Jing Su, Zhicheng Fang. Chenjing Ding, Weihao Gan, Yichao Yan, Xin Jin, Xiaokang Yang, et al. ActFormer: A GAN-based Transformer towards General Action- Conditiondi 3D Human Motion Generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2228-2238, 2023.
Chenyu Yang, Davide Liconti, and Robert K Katzschmann. VQ-ACE: Efficient Policy Search for Dexterous Robotic Manipulation via Action Chunking Embedding. arXiv preprint arXiv:2411.03556, 2024a.
Jonathan Yang. Catherine Glossop, Arjun Bhorkar, Dhruv Shah, Quan Vuong, Chelsea Finn, Dorsa Sadigh, and Sergey Levine. Pushing the Limits of Coos--Embodiment Learning for Manipulation and Navigation. arXiv preprint arXiv:2402. 240..4492244 /9432, 2024b.
Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Sejune Joo, Jianwei Yang, Baolin Peng, Ajay Man- dlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, et al. Latent Action Pretraining from Videos. 2024.
arXiv preprint arXiv:2410.
Jia Zeng, Qingwen Bu, Bangjun Wang, Wenke Xia, Li Chen, Hao Dong, Haoming Song, Dong Wang, Di Hu, Ping Luo, et al. Learning Manipulation by Predicting Interaction. In Robotics: Science and Systems, 2024.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The Unreason- able Effectiveness of Deep Features Perceptual Metric. In In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 586-595, 2018.
Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning Fine-Graine Grained Bimanual Manipulation with Low-Cost Hardware. In Robotics: Science and Systems, 2023.
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magic Video: Efficient Video Generation with Latent Diffusion Models. arXiv preprint arXiv:2211.110111 2022.
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, Huong Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R. Sanketi, Grecia Salazar, Michael S. Ryoo, Krista Reymann, Kanishka Rao, Karl Pertsch, Igor Mordatch, Henryk Michalewski, Yao Lu, Sergey Levine, Lisa Lee, Tsang- Wei Edward Lee, Isabel Leal, Yuheng Kuang, Dmitry Kalashnikov, Ryan Julian, Nikhil J. Joshi, Alex Irpan, Brian Ichter, Jasmine Hsu, Alexander Herzog, Karol Hausman, Keerthana Gopalakrishnan, Chuyuan Fu, Pete Florence, Chelsea Finn, Kumar Avinava Dubey, Danny Driess, Tianli Ding, Krzysztof Marein Choromanski, Xi Chen, Yevgen Chebotar, Justice Carbajal, Noah Brown, Anthony Brohan, Montserrat Gonzalez Arenas, and Kehang Han. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. In Conference on Robot Learning, pp. 2165-2162111111 -2833. PMLR, 2023.
18

---

Published as a conference paper at ICLR 2026
As part of the supplementary material, include additional details about the following.
A Background: Covers key paradigms relevant for ViPRA: VQ- "Q-VAE for learning discrete latent actions, optical flow estimation, behavior cloning for direct action prediction and flow matching for smooth continuous control.
Latent Action Learning: Architecture design, loss formulations, and training protocols for dis- crete latent action codebook learning, including hyperparameter configurations and optimization strategies.
Multimodal Video Pretraining with Latent Actions: Implementation details for extending
LWM with latent actions, including embedding architecture, decoder design, and joint training methodology.
Flow Matching Decoder for Continuous Control: Specification of noise scheduling, action encoder/decoder architectures, and finetuning procedure.
E Simulation Benchmarks: Detailed description of SIMPLER tasks and additional LIBERO Long benchmark.
Action Output Analysis: Comparative visualization and discussion of predicted action trajecto- ries across discrete and continuous policies, highlighting the impact of quantization, loss formu- lations, and control space choices on smoothness and deployment behavior.
G Real World Experiments: Detailed description of hardware setup, task design, policy general- ization, retrying behavior, and the impact of action chunking on control frequency and real-time performance in physical deployments.
H Bimanual Manipulation Tasks: Evaluation of ViPRA-FM
real world dual-arm tasks requiring spatial coordination and tool use, including task setup, challenges, quantitative results, and rollout visualizations from real robot executions.
We have released code, checkpoints, and latent action labeled data and rollout videos at: https: //vipra-project project. iereettgtttee github.ie.
19

---

Published as a conference paper at ICLR 2026
A BACKGROUND
We review key paradigms relevant for ViPRA: VQ-VAE for imposing information bottleneck to learn discrete latent actions, optical flow estimation, behavior cloning for direct action prediction and flow matching for smooth continuous control.
Vector-Quantize VAEs (VQ-VAE). An encoder eç maps an observation Or to a continuous la- tent vector htg [ ]. which is quantized to a sequence of nearest codewords 2; = [2] ..... ..... 21"1 C using a shared discrete codebook. A decoder dę reconstructs the observation 6 = du (2t + Err). where Err is a gradient estimator used to allow backpropagation through the non-differentiable quantization operation.
In traditional VQ-VAE (van den Oord et al., 2017), this estimator takes the form
EITSTE sg(h; - =2),
(4)
where sg(-) denotes the stop-gradien operator. The decoder input 2 + ErTSTE preserves the forward pass while enabling gradients bypass the non-differentiable argmin. However, this approach typically requires auxiliary losses, such as the codebook and commitment losses, to stabilize training and encourage codebook usage. We adopt the NSVQ formulation (Vali Bäckström, 2022), which replaces the deterministic STE with a stochastic noise- injjec-njected surrogate
Errnsvo 1| hell
where ē = c/|lel| and (0, I). The decoder thus receives 6; = d2s + ErrNsvo). Crucially, NSVQ enables gradients to flow to both the encoder and codebook using only the reconstruction loss, without requiring additional codebook loss terms. The noise-injected gradient estimator, com- bined with the unused codebook replacement technique applied during early training, significantly improves training stability and mitigates codebook collapse, common issue in VQ-VAE training.
RAFT based Optical Flow Given two images, Oa and og, RAFT (Teed & Deng. 2020) obtains the that maps each pixel in frame o, to its location in os- first extracts feature maps Φ(0,), Φ(os) E 1g xd (where is the feature extractor) and builds (ér)(oa), ozu(o))) The flow prediction is then refined iteratively: ((++1) (()) + AP( (R). When applied to temporally close frames in a video, this flow field f can give good estimate of motion consistency.
dense displacement field fa-sb E # X2
the all-pair correlation R, with Rij,Al
Behavior Cloning (BC) is supervised learning paradigm in robotics that learns policies directly from expert demonstrations. Given a dataset = {(o,, at) )T observation-action pairs from expert trajectories, BC trains parameterized policy TBC 0) to minimize distance metric between predicted and ground-truth actions:
min ((o,,.)) ( (), a;)] 8
where -(.,-) is typically the L1 or L2 distance for continuous actions or cross-entropy for discrete actions. This framework has been extended with high-capacity architectures: diffusion models (Chi et al., 2023; Zhao et al., 2023) parameterize (ac(a, oo; 0) as denoising process that learns p(a, (o:) through iterative refinement, while VLAs leverage pretrained language models (Touvron et al., 2023; Qwen et al., 2024) and visual encoders (Radford et al., 2021: Oquab et al., 2023; Tschannen et al., 2025) as the backbone architecture for 0, enabling multimodal grounding of actions in visual and linguistic contexts.
Flow Matching (Lipman et al., 2022) provides an alternative to diffusion models for learning con- tinuous normalizing flows. While diffusion models learn the full denoising process, flow matching directly learns the vector field that transports samples from a source distribution to a target distri- bution. This approach offers computational advantages for robotics applications where real-time inference is critical.
Flow matching trains neural network ge predict the velocity field along a straight-line interpo- lation path. Given a source sample xo (typically Gaussian noise) and target sample Φ1 (c.g., robot actions), the interpolation creates a path:
U2 = 8 FT + (1 8) æ1; where
[0, 1] parameterizes the interpolation
20

---

Published as a conference paper at ICLR 2026
The model learns to predict the velocity field that guides samples along this path:
i
ga (u, sly), where represents conditioning inputs
(8)
0s
In robotics applications, y typically includes visual observations and language commands that spec- ify the desired behavior.
The training objective teaches the model to predict the correct velocity by minimizing the difference between predicted and true flow direction:
LFM = (...
Ji0 (1 s)
go gosus: v)
(9)
disection
predicted velacity
At inference time, samples are generated by integrating the predicted velocity field from noise (s 0) to the target (s = 1):
Us+ u, + As (M,, sly), where As is the integration step size
(10)
This produces the final sample efficiency (Black et al., 2024), though more sophisticated solvers like Heun's method or Runge- Kutta can improve stability for high-dimensional control tasks (Kutta, 1901; Runge, 1895). Flow matching has demonstrated superior smoothness and precision compared to direct action prediction, particularly for temporally extended manipulation tasks (Black al., 2024; NVIDIA al., 2025b).
Forward Fuler integration is commonly used due to its
21

---

Published as a conference paper at ICLR 2026
B LATENT ACTION LEARNING
We detail our latent action learning framework in Algorithm 1, which extracts latent action se- quences &t:t+- L from video sequences 0t:t4- using combination of reconstruction, perceptual, and optical flow consistency losses. A detailed diagram of this procedure is shown in Figure 7, and the complete training intiguurrttionnnllddiii model architecture and optimization hyperparameters provided in Table 5.
The inverse dynamics encoder maps each frame оş into a sequence of spatial features using DI- NOv2 (Oquab et al., <02ta)-nitialized backbone. These features are enriched with clip-level context through factorized spatio-temporal attention layers, where the temporal branch employs bidirec- tional attention to aggregate information across the full sequence. The contextualized features are then discretized via coiss-Substitution Vector Quantization (NSVQ) (Vali Bäckström, 2022), pro- ducing Niment discrete codes selected from the shared codebook C, which serve the latent action tokens for Zg.
The forward decoder mirrors this architecture with factorized spatio-temporal transformer, but applies causal temporal attention so that predictions depend only on the past. It jointly attends to the latent action sequence Ze:f++k and the observation history Oct++& reconstruct the next frame +++k+1- In addition, we integrate the action-conditioning modules proposed by (He et al., 2025) before each spatio-temporal block in the decoder better align latent action tokens with visual dynamics.
Algorithm Latent Action Learning (Training Step)
Require: Video clip L++1 observations O E R(L+1)xHxWxx5
Require: Hyperparameters: LPIPS weight ALPIPS Flow weight Anow. Flow start step Offlow Require: Codebook C E IR K with K codes dimension
1: // Inverse model 13 comprises of Ig and NSVQ module
2: Compute contextual embeddings: hea4 &- Ig(ors+4.)
3: for k 0 L do
4: Quantize embedding to latent: 24+k + NSVQ(h C)
5: Decode next frame: ôc+k+1 FalD Z64+k)
6: end for
7:
8: free EL_ 10:44 - octoll 1/1 pixel reconstruction loss
9: LIPIPS +- Et= (P // Perceptual similarity loss
10:
11: // Optical consistency loss
12: if step Oiflow then
1
13: Liow -
(100++ OF(or+k, -F00
L
(p(00+R,0044++1 OF(or+&, oct++1)ll (0f04++)0044+1))l
14: else
15:
Lnow f-
16:
end if
17:
18: Listent -ree ALPIPSCPIIS + Lnow
19:
Update parameters via AdamW optimizer
22

---

Published as a conference paper at ICLR 2026
Latent Action Learning
Hyperparameter
Value
Llatent
Training Configuration
oc+1
Öc+z
Oc+i
Optimizer
AdamW
Base Learning Rate
lc-4
Forward Decoder Fa
DINO Enc. Learning Rate
le.5
Optimizer Momentum
81.
0.9,
Or11
Orez--1
Batch Size
128
Er+t-1
Grad. Norm Clip
4.0
Latent Action Tokenizer lg
Total Steps
240000
Image Augmentation
RandomResizeCrop
Quantization (NSVQ)
Flow Start Step Ottow
60000
Losses
Lix + PIPs CiPPP + Ano Caow
LPIPS Weight ALPIPS
0.5
argmin, ||
syll
Codebook
Flow Weight Anow GPU
0.1 (after Cktow)
hp41
Prat-1
hew
8 Nvidia H100 (168 hours)
SpapoolTemporal Transformer
Inverse Dynamics Encoder I
f+
fe-1-1
fe+l
Backbone Init.
DINOv2 (Oquab et al., 2023)
DINOv2 Init. Spatial Encoder
Embedding Dim
768
Spatio-temporal Layers
6
0,
Op11
O
Oc+l
Attention Heads
16
Attention Head Dim
Latent Action Quantization
Codebook Size [C]
8
Quantized Token Dim
32
Human videos
Robot videos
Quantization Method
NSVQ (Vali & Blickström, 2022)
Latent action learning framework. Given sequence of frames OtsLL an inverse dynamics encoder J=(Fe=+Z-1 0ut+ L) maps observations into discrete latent action sequence via vector quantization. A forward decoder Fo (6 O Z(:++) then reconstructs the future frame conditioned the observation and latent action history. v Training combines LI pixel reconstruction, perceptual (LPIPS), and optical flow consistency losses to ensure that the latent to- kens capture physically grounded and temporally localized action information.
Figure 7:
Codebook Refresh Interval
Every 10 till 100, every 100 till 1000, every 1000 10000
Codebook Refresh Strategy
Re-init Unused, Re-shuffle Used
Forward Decoder F.,
Embedding Dim
768
Spatio-temporal Layers
8
Attention Heads
Attention Head Dim
Table 5: Hyperparameters for latent action learning.
23

---

Published as a conference paper at ICLR 2026
C MULTIMODAL VIDEO PRETRAINING WITH LATENT ACTIONS
We augment LWW-Chat--IN (Liu et al., 2024), a pretrained multimodal video model Go with a latent action embedding layer Esg and a latent action decoder Hg. The model jointly predicts end- of-horizon visual tokens F++H and latent action sequences H- leading to it, conditioned on history visual tokens (z)) and task context
The latent action embedding head Eg maps each code of C into the token space of Ge. and the decoder head H& predicts next-token logits over the latent vocabulary. Training uses teacher forcing for both visual tokens and latent tokens with cross-entropy loss. The complete training procedure and hyperparameters are detailed in Algorithm 2 and 6.
Algorithm 2 Pretraining Multimodal Video Model via Video and Latent Action Prediction
Require: History frames: (+)
Require: Target frame: ot+H
Require: Task description: e Tokenizee text string
Require: Target latent action chunk: H- E CCHNNus Numen
Require: Pretrained models: VQ-VAE encoder Evo. video model Go
Require: Trainable components: Ga. random initialized embedding layer Es, decoder head H
1: Tokenize input frames: It-1 Tt f- Evo(ot-1), Ev)
2: Tokenize target frame: Z4+H Evg(o1+H)
3: Embed latent actions: 24:4H-11 + Es (2)4+-11)
4:
5: for i = Niokens do
6: $ +- Go I,, F4HH (4t)
// Autoregressive prediction with teacher forcing
7: end for
8:
9:
Ling + Ex Nutten CE
2 (1
|/ Image token loss
11: for k = lto l. + H do
for
to Matent
13:
of +- He (Go (Ge(c,,t (c, FHHH Zek, 2k =))))
14: end for
15: end for
16:
17: Lus
En Σ Di=1
Narn CE (2 |/ Latent action token loss
18:
Cpretrain &- Limg + Lue Total loss
20: Update Gg. Esg, and Hi using Adam W optimizer
24

---

Published as a conference paper at ICLR 2026
Hyperparameter
Value
Model Setup
Video Model
LWM-Chat- 1M (Liu et al., 2024)
(initialized)
Tokenization Backbone
VQ-VAE (frozen)
Prompt Tokenizer
BPE tokenizer
Latent Action Vocabulary Size [C]
8
Latent Embedding Dim (Fi))
4096
Latent Decoder Type (H
MLP
Latent Decoder Layers
1
Latent Decoder Hidden Dim
2048
Training Configuration
Optimizer
AdamW
Learning Rate
4e-5
Weight Decay
0.0
Optimizer Betas
(0.9, 0.95)
Batch Size
Total Steps
50,000
Dropout
0.1
Gradient Clipping
1.0
Mixed Precision
bfloat16
GPU
8 Nvidia H100 (144 hours)
Prediction Targets
Prediction Horizon H
14
Image Token Loss
Cross Entropy
Latent Action Loss
Cross Entropy
Table 6: Hyperparameters for multimodal video pretraining to jointly predict future visual state and latent action sequence. The video model is initialized from LWM-Chat- IM and trained jointly with lightweight latent action modules.
25

---

Published as a conference paper at ICLR 2026
D FLOW MATCHING DECODER FOR CONTINUOUS CONTROL
To enable continuous control, we augment the pretrained video model Go with a flow-based action decoder. Specifically, we introduce (i) noisy action encoder E,, that embeds time-intrrpolate noisy action chunk W, into the embedding space of the model, and (ii) a flow decoder H, that predicts the velocity field used transport the noisy actions toward the ground-truth action chunk 41:+H-1- The full training procedure is described in Algorithm 3, and architectural details and hyperparameters are provided in Table 7.
Algorithm 3 Flow Matching for Continuous Control
Require: Image history frames (0t-1,
Require: Task description: e Tokenized text string
Require: Action chunk at:++ H-1 HHx // D-diieensional actions H-steps into the future
Require: Pretrained models: VQ-VAE encoder Evo, video model Gg
Require: Trainable components: Go, noisy action encoder Eny, flow decoder H
1: Sample timestep ~ Beta 1.5,
2: Sample noise ug ~ ~N(0,
3: Compute interpolation:
uo + (1-s) Oct+H-1
4: Tokenize input frames: It- 1, It f- Evo Evolor)
5: Encode noisy actions:
(I,,8)
6: Predict flow field: 9 & H, (Ga 11,
7: LFM <-
[[aaH -11 Uo (1-8) gll2 Flow matching loss
8: Update Gg. E,, H, using AdamW optimizer
Hyperparameter
Value
Noisy Action Encoder Head (E,)
Architecture
2-layer MLP
Hidden Dim
4096
Embedding Dim (da)
4096
Activation
GELU
Dropout
0.1
Flow Decoder Head (G.,)
Input Dim
7 (End-Effector Deltas) or (Absolute Joint States)
Architecture
Single linear projection
Flow Matching Setup
Interpolation Timestep
Beta(1.5, (15, 1.0)
Noise Distribution X
Standard normal (O, I)
Prediction Horizon
14
Integration Method (Inference)
Forward Euler, N 10 steps
Training Configuration (follows Table 6)
Total Steps
12000 (SIMPLER)
Table Hyperparameters used for flow matching action decoder to enable continuous control.
26

---

Published as a conference paper at ICLR 2026
E SIMULATION BENCHMARKS
E. SIMPLER BENCHMARK
Following prior latent action works (Ye et al., 2024; Bu et al., 2025), we benchmark ViPRA SIM- PLER (Li et al., 2024b), an open-source suite for evaluating generalist manipulation policies. We evaluate on four Bridge tasks with 7-DoF WidowX arm, a benchmark designed to test general- ization across diverse manipulation goals. Since SIMPLER does not provide finetuning data, we collect 100 diverse multi-task trajectories using a pretrained VLA model (Ye et al., 2024) to adapt policies before evaluation. The tasks are follows:
Spoon2Cloth: The instruction is put the spoon the towel. The spoon is placed on vertex of a 15 cm square on the tabletop, and the towel on another vertex. The spoon's 's orientation alternates between horizontal and vertical, requiring the robot to re-orient its gripper. This task evaluates both grasp selection and orientation adjustment.
The instruction put carrot on plate. Same setup as Spoon2Cloth, but with a carrot and a plate. While similar in layout, this introduces different geometry and surface, requiring adaptation in grasping and placement.
Carrot2Plate:
StackG2Y: The instruction is stack the green block the yellow block. A green block is placed on a vertex of tabletop square (10 and 20cm cm edges) and yellow block on another. Success requires precise alignment and careful release, making it fine-grained ma- nipulation task that stresses stability and accuracy.
Eggplant2Bask: The instruction is put eggplant into yellow basket. An eggplant is dropped into the right basin of sink and yellow basket in the left basin. The eggplant is randomized in pose but ensured to be graspable. This task evaluates robustness to shape variabil- ity and placement under uncertainty, the object must be reliably picked and transferred across workspace regions.
We evaluate performance using two metrics: success rate and partial success rate (grasp rate). Success rate measures whether the full task goal is completed (e.g., spoon placed towel, block stacked without falling, eggplant deposited into basket). Grasp rate captures whether the robot at least able to establish successful grasp on the object, even if the subsequent placement or stacking is not achieved. This distinction is important: grasping reflects a fundamental capability for initiating manipulation, while successful completion requires the integration of grasping with precise transport and placement. Together, these metrics provide more comprehensive view of policy competence, distinguishing between failures due to perception/grasping versus those arising from downstream control and placement.
E.2 SIMPLER RESULTS
We report both end-to-end success rate and grasp rate in Table 1. Across discrete actions set- ting, ViPRA-AR attains the best average success (69.8%), exceeding LAPA (53.1 %), VPT (51.0%) and VLA (38.6%). It leads on precision-heap heavy StackG2Y (66.7% vs. 54.2% Scratch-AR, 45.8% VPT) and Carrot2Plate (62.5%), and remains competitive on Spoon2C1oth (6.77, near VPT's 70.8%). On Eggplant2Bask, ViPRA-AR (83.3%) significantly outperforms other methods, demonstrating strong transport and placement.
In the continuous setting, ViPRA-FM achieves the highest average success (62.5%), outperform- ing Scratch-FM (41.7%), Mo (27.1%), and UniVLA (42.7%). is the strongest continuous model on StackG2Y (54.2%), Carrot2Plate (50.0%) and Spoon2Cloth (667%) while remaining competi- tive (79.2%) with mo (83.3%) on Eggplant22ass UniPI frequently generates action sequences that diverge from the given instruction longer-horizon settings, while VPT provides only limited gains, indicating that IDM-derived pscudo-labels are sensitive to environment shifts. In contrast, ViPRA's joint use of latent action prediction and future state modeling yields stronger cross-environment transfer and more reliable task execution.
ViPRA converts grasps into task completion more reliably. On StackG2Y, OpenVLA achieves 70.8% grasp but only 25.0% success (a 45.8 pt gap), indicating post-grasp placement failures. ViPRA-AR maintains 66.7% grasp and 66.7% success (0 pt gap), and ViPRA-FM 62.5% grasp
27

---

Published as a conference paper at ICLR 2026
Method
Success Rate
UniPI (Du al., 2023c)
0.00
OpenVLA Kim (2024)
0.54
To To-FAST (Black et al., 2024)
0.60
Wo (Black et al., 2024)
0.85
UniVLA (Bu et al., 2025)
0.92
UVA ViPRA-FM
0.90
0.79
Table 8: Success rates LIBERO-10 benchmark.
vs. 54.2% success (8.3 pt gap), evidencing stable transport and release. On EggplanttBask,, Open- VLA's 91.7% grasp falls to 58.3% success (33.4 pt drop), whereas ViPRA-AR (100% 83.3%) and ViPRA-FM (91.7% + 79.2%) show markedly smaller drops, consistent with smoother post-grasp control and accurate instruction following.
E.3 LIBERO LONG BENCHMARK
We also evaluate on LIBERO Long (a.k.a LIBERO 10) (Liu et al., 2023), the most challenging subset of the LIBERO simulation benchmark. Unlike the Spatial, Object, or Goal subsets, LIBERO Long focuses on long-horizon manipulation tasks that require sequencing multiple sub-goals with heterogeneous objects, layouts, and task dependencies. This setting stresses robustness and temporal compositionality, since errors can accumulate across long horizons.
The evaluation consists of a suite of 10 long-horizon tasks, each paired with natural lan- guage goal description. For example, one of the task instructions includes "put the white mug on the plate and put the chocolate pudding to the right of the plate". requir- ing reasoning over both symbolic relations (object identities, spatial references) and low-level con- trol. For each task, the environment initialized with objects placed in varied locations, increasing the difficulty of generalization.
Each task evaluated across 10 runs with 5 different random seeds, and results are reported as the average reward over all 10 tasks (500 episodes in total). This protocol provides stringent test of both semantic grounding and long-horizon policy execution, making LIBERO Long a valuable complement SIMPLER's shorter-horizon manipulation tasks.
E.4 LIBERO LONG RESULTS
On LIBERO-10, which emphasizes long-horizon, multi-stage manipulation, ViPRA achieves a 79% success rate. This is substantially higher than Open VLA (54%) and -FST (60%), and close to UVA (90%), which is specifically optimized for LIBERO. These results demonstrate that ViPRA's motion-centric latent pretraining transfers effectively to simulated long- long-horizon tasks, outperforming methods trained primarily with labeled actions direct policy supervision.
We observe that ViPRA performs reliably on coarse manipulations (e.g., cups, bowls, books), which easy to grasp, but struggles with precision grasps such cylindrical cans that require diameter- aligned control. We attribute this to Mella-EEF drift: since LIBERO's action space is delta end- effector, small prediction biases accumulate over time, leading imprecise grasps in the absence of absolute cues to re-anchor the trajectory. For instance, To mitigates this issue by conditioning on proprioceptive state history and wrist-camera inputs. Despite lacking such additional signals, ViPRA surpasses Open VLA under the same sensing setup (image-only, delta-EEF), underscoring the benefits of cotionn-enttre latent pretraining for long horizon manipulation.
28

---

Published as a conference paper at ICLR 2026
F ACTION OUTPUT ANALYSIS
We provide more in-depth analysis of the action outputs of various policies introduced in Sec- tion 5.5, highlighting their differences in smoothness, consistency, and suitability for real world deployment. Policies differ their action representations and control spaces:
Absolute Joint Space. ViPRA-FM and LAPA (Ye et al., 2024) output full 7D joint posi- tions (Franka), directly supervised in joint space.
Delta End-Effector Space. OpenVILA (Kim et al., 2024), To (Black et al., 2024), and operate 7D Cartesian delta commands (position, Fuler rotations, gripper), decoded from visual inputs.
Continuous vs Discrete. ViPRA-FM and To (Black et al., 2024) p continuous actions via flow matching decoder, whereas LAPA (Ye et al., 2024) and VLe (Kim et al., 2024) use quantized logits over discretized action bins.
To better understand the behavioral differences between discrete and continuous policies, we analyze the predicted action trajectories across different models during closed-loop visual rollout on real robot observations. We evaluate policies by loading their finetuned checkpoints into inference pipeline and simulating replay on the training trajectories from the finetuning dataset. This allows us to visualize their motor command trends without introducing new generalization factors. In particular, we compare ViPRA-FM (ours), LAPA (Ye et al., 2024), Open (Kim et al., 2024), and To (Black al., 2024).
LAPA (Ye et al., 2024) and OpenVLA (Kim et al., 2024) rely on a discretization scheme in which each dimension of the robot's action space is uniformly quantized into 255 bins using equal-sized quantiles over the training distribution. That is, for each joint or end-effector dimension, bin bound- aries are chosen so that each bin contains roughly the same number of training points. This quantile- based discretization ensures equal data coverage across bins but introduces two key limitations in how actions are represented and learned:
1. Contact-Sensitive Flipping: At test time, small perturbations in the input (c.g., due to occlusions or slight viewpoint drift) may cause the model to flip from one bin another near the quantile coundar--especiale at contact points. Since adjacent bins can correspond different action magnitudes, these minor visual shifts can lead abrupt discontinuities in motor output.
2. Loss Granularity: The cross-entropy loss used for training treats each action bin as distinct class label. As result, all incorrect predictions are penalized equally, regardless of how close they are to the ground-truth bin. For example, predicting bin 127 instead of 128 incurs the same loss as predicting bin O. This is fundamentally at odds with the structure of continuous action spaces, where the cost of an error should scale with its magnitude.
We hypothesize that the combination of bin boundary flipping and non-metric loss leads to the spiky or erratic behavior seen in discrete action models, particularly around moments of contact or high-frequency motion. These effects are amplified in high-dimensional control settings, where discretization artifacts can arise independently in each action dimension compruundioooo into visibly unstable or jerky behaviors across the full joint trajectory.
By contrast, continuous policies such as ViPRA-FM and wo (Black al., 2024) operate directly in RD using flow matching. These losses naturally reflect the structure of the action space-penalee penaalzzng predictions in proportion to how far they deviate from the ground truth. As a result, the output trajectories tend to be smoother, better aligned with demonstrations, and more robust to perceptual jitter.
We note that it may be possible to mitigate some of the above issues by increasing the number of bins or by using non-uniform binning schemes (c.g., higher resolution in frequently visited regions). However, these approaches increase model complexity and still inherit the fundamental limitation of using classification loss in a regression setting. Continuous decoders trained with distance-awand objectives offer more natural and principled solution for low-level control.
To gain deeper insight into how different action representations influence control behavior, we ex- amine the temporal structure of predicted actions across several rollout trajectories. We organize
29

---

Published as a conference paper at ICLR 2026
Pelet
Jalnt
Seint
20lnt
...03
Juint trippur
120
222
1.0
4.11
2.3
(a) Absolute joint space. Predicted 7D joint positions time for ViPRA-FM (blue) and LAPA (Ye et al., 2024) (green). ViPRA-FM produces smooth, continuous trajectories. while LAPA (Ye et al., 2024) exhibits local discontinuities and random spikes-often around contact events-despite tracking the overall trend. In real world deployment, such discontinuities triggered Franka's 's emergency brake mechanism due to abrupt torque jumps.
0.00
Grippor
Sytu
Grlpper
Time Stops
(b) Delta end-effector space. Predicted 7D delta actions (position, rotation, gripper) for no (Black et al., 2024) (magenta) and OpenVLA (Kim et al., 2024) (orange). Although delta control provides structured low- level modulation, OpenVLA exhibits sharp fluctuations due to discretized output. Notably, the gripper signal shows large, momentary switches during contact events resulting failed grasps premature object drops. contrast, Mo maintains stable gripper behavior during fine manipulation.
Figure 8: Visualization of predicted actions across different control spaces. Discrete policies often pro- duce sharp discontinuities due binning artifacts and classification loss, whereas continuous policies exhibit smoother, dynamiex-consistent behavior.
analysis by control space-absolute joint angles vs. delta end-effector motions-and visualize per-dimension action trends across time in Figure 8.
These visualizations support our hypothesis: discrete policies, trained with cross-entropy over fixed bins, lend to produce abrupt transitions around perceptually sensitive regions-especiall near bin boundaries or occlusions. This manifests as random spikes, high-frequency jitter, or contact-time instability, all of which can destabilize robot behavior in deployment.
In contrast, continuous policies like ViPRA-FM and ny trained with flow matching losses, yield consistently smooth, physically plausible actions that better reflect real world constraints. The ability interpolate naturally between states-not potees-notjust just classify mnem-proves critical for robust closed- loop performance in contact-rich manipulation.
30

---

Published as a conference paper at ICLR 2026
G REAL WORLD EXPERIMENTS: SETUP, CHALLENGES, AND OBSERVATIONS
To complement real world results in Section 5.4, we provide additional details hardware setup, task design, and policy behavior under realistic sensing and control constraints. We also ana- lyze generalization to unseen objects, retry behavior, and how chunked continuous actions support efficient closed-lopp control.
G.1 HARDWARE AND DATA COLLECTION SETUP
All experiments conducted real world robotic platform with two 7-DOF Franka Emika Panda arms. The workspace is observed by single front-mounted ZED stereo camera. There are no wrist-mounted side-view cameras, so all perception is monocular and from fixed third- person viewpoint. We use the GELLO teleoperation system (Wu et al., 2024b) to collect human demonstrations 15Hz. Demonstrations are collected directly in task-relevant environments, with each policy trained using only single camera view.
Our decision to use image history as part of the observation is motivated by the inherently temporal nature of the video model architecture, well as the absence of auxiliary views. Stacking obser- vations over time allows the model internally infer dynamics and compensate for occlusions or ambiguous single-frame cues.
G.2 TASK DESCRIPTIONS AND CHALLENGES
We evaluate policies on three real world single-arm tasks, each with unique control and perception challenges (Figure 9):
1. Cover-Object:
The robot must pick up piece of cloth and drape it over a specified ob- ject. This task is challenging due the deformable nature of cloth, which requires reliable grasping from the table surface. Slight changes in cloth configuration or object geome- try can affect dynamics drastically. Generalization requires reasoning over unseen cloth textures and novel target objects.
2. Pick-Place: The robot must pick up a named object (e-g- sponge, bowl, duck) and place it a destination surface (plate board). Object shapes vary significantly, leading to different grasp affordances. Grasping wide bowl vs. narrow-handled cup requires distinct motor strategies. The task is highly multimodal-ther perform the task, depending on object shape, pose, and placement surface.
multiple correct ways to
3. Stack-Cups: The robot must follow language instructions to stack cup of colorl onto cup of color2. Success requires grounding object properties and executing precise stacking. Evaluation setups include unseen cup types, color shades, and geometries test language understanding and spatial generalization.
G.3 GENERALIZATION TO NOVEL OBJECTS
core goal of our real world evaluation is to assess how well the policy generalizes to unseen object instances and configurations not encountered during training. We design test-time setups that introduce meaningful variation across tasks:
Test scenarios include cloths of varying texture, size, and stiffness, well as new target objects such as jars, boxes, and toys. These variations require the policy to generalize grasp strategies and adapt deformable material dynamics.
Cover-Object:
Pick-Place: We evaluate on previously unseen objects with diverse geometries and affor- dances (e.g., bowls, mugs, fruits), and destination surfaces of varying size and texture. The task requires flexible grasping and reliable placement across a range of object shapes and destination surfaces.
Stack- Evaluation includes cup types with unseen shapes, rim sizes, and fine- grained color variations. The policy must generalize language grounding to new color references and execute precise stacking across novel physical configurations.
31

---

Published as a conference paper at ICLR 2026
Cover- -0vert
Pick-Place
Stack- Cups
Traín Setup
Test Setup (Unseen Objects)
pick cloth and cover <obj>
pick <obj1> and place on <obj2>
pick <coli> cup stack on <col2> cup
Task Prompt
Figure 9: Task Setup Overview. (Top row) Training environments for each of the three single-arm manipula- tion tasks: (Overt. Pick-Place, and Stack-Cupa Cups. (Bottom row) Evaluation environments featuring novel objects, textures, or placements not seen during training. Note the variety in cloth shape, object geometry, plate type, and cup color/size combinations.
Despite these shifts, method consistently exhibity robust generalization across all tasks. We attribute this to the combination of latent dynamics pretraining, language-conditionee perception, and unified architecture that integrates semantic, spatial, and temporal cues. Pretraining on diverse unlabeled videos teaches the model general priors about object motion and interaction. Condition- ing on task instructions guides object selection and interpretation contexts. Finally, the architectural design ensures that learned representations capture not just ap- pearance, but how objects behave across time, enabling transfer new instances that were not explicitly seen during supervised finetuning.
in ambiguous unfamiliar
G.4 RETRYING BEHAVIIR ENABLED BY TEMPORAL PRETRAINING
Our method consistently exhibits robust retry behavior: when initial grasp attempt fails, due to occlusion, misalignment, or object shift, the policy often reattempts until successful. This is especially evident in Cover-Object, where the robot frequently retries grasping if the cloth slips, and in Pick-Place, where wide or irregularly shaped objects like bowls may require multiple grasp attempts from different angles.
We attribute this robustness to our temporal pretraining objective. By learning to predict future video frames and latent actions over multiple steps, the model develops a sense of longer-horizon dynamics and recoverability. Rather than depending on single-step feedback, it implicitly plans through extended temporal mentext-enabling course, correct and persist through partial failures.
G.5 ACTION CHUNKING AND INFERENCE EFFICIENCY
ViPRA produces continuous actions using a chunked flow matching decoder, generating sequences of 14 actions per inference step. At test time, cap control frequency by evaluating rollout strategies: 7/14 rollout, where the first actions of each chunk are executed before re- re-planning, and 14/14 rollout, where all 14 actions to effective closed-loop update rate of ~3.55 Hz, while the latter doubles this to 7 Hz. Because predicted action trajectories are smooth and temporally coherent, ViPRA remains stable even under open-loop execution within each chunk. This property is particularly beneficial for contact-rich phases that demand reactive yet fitter-free behavior.
executed before the next inference. The former corresponds
KV caching for fast inference We further optimize inference with key-value (KV) caching. Lan- guage and image attention states cached once and reused across flow matching Euler steps, so only action tokens are recomputed during integration. This reduces redundant computation, enabling the entire 14-step chunk be produced in 510 (~ 1.95 Hz), which corresponds robot-side
32

---

Published as a conference paper at ICLR 2026
control frequency of up to 22 Hz. Our setup stably support control rates approaching 20 Hz, to our knowledge matched only by one other 7B- 18-parameter model (Kim et al., 2025).
Comparison with baselines. Table summarizes model sizes, action rollout lengths, and infer- ence times. Unlike prior approaches that also use a 7B model (e.g., LAPA and Open and operate at 200 ms per step but predict only single actions, ViPRA amortizes inference across long, smooth action chunks, enabling high frequency reactive control.
Method
Model Size
Action Steps
Inference Time (ms)
LAPA (Ye et al., 2024)
7B
220
OpenVL. (Kim al., 2024)
7B
190
To (Black al., 2024)
3.3B
16
90
UniPI (Du al., 2023b)
24000
UVA (Li et al., 2025a)
0.5B
230
ViPRA (ViPRA-FM)
7B
14
510
Table 9: Inferenee speed comparison across models, ViPRA achieves high effective control frequencies by amortizing computation over action chunks.
33

---

Published as a conference paper at ICLR 2026
H VIPRA-FM ON CHALLENGING BIMANUAL TASKS
Bimanual manipulation introduces significant complexity beyond single-arm control. The combined action space spans 14 degrees of freedom, and inter-arm arm coordination requires precise spatial align- ment, collision avoidance, and timing consistency. The solution space also highly multimodal- there are many valid ways to execute task depending on object geometry, initial configurations, and movement variability. These challenges make bimanual tasks a strong test policy's ability to generalize and coordinate under real world constraints.
H.1 BIMANUAL SETUP
We test our framework using both arms of the Franka Panda robot. While only the right arm per- forms active grasping, both arms are controlled jointly using single policy conditioned on shared language instructions. The system receives monocular observations from a front-mounted ZED camera and generates chunked continuous actions for both arms in synchronized control loop.
Place-in- in-Bowl
Mix-wt wi wowtth-whise
Initial Setup
ViPRA-FM
Mid-task
Task Prompt
Pick up <obj> and place in bowl pick up whisk mix the bowl
Figure 10: Bimanual task execution by ViPRA-FM. (Top row) Initial setup for the two tasks: placing tomato into a bowl and mixing with a whisk. (Bottom row) Mid-execution rollout of ViPRA-FM: the right transports tomato toward the bowl held by the arm (left), and mixes the contents using the whisk while the left arm maintains bowl stability (right). These examples highlight coordinated two-arm control and fluent execution of tool- and object- handlinn behaviors.
We evaluate two bimanual tasks of increasing complexity:
(1) Place-in-Bowl: The right arm must grasp target object (e.g., fruit or kitchen item) and place it into a bowl held by the left arm. Success requires fine-grained spatial alignment above the bowl, smooth object transfer, and collision-fre approach and retreat trajectories in close proximity the support arm.
(2) Mix-with-Whise of the bowl, and returns the whisk to its original location. This task involves tool use, curved and sustained motion, and doose-proxmmity coordination with the left arm, which dynamically maintains the bowl pose throughout the sequence.
The right arm retrieves a whisk from a nearby basket, mixes the contents
These tasks pose significant challenges for real world bimanual coordination. Both arms must op- erate in close proximity, requiring precise spatial alignment avoid collisions-especiall during approach and retreat phases. With only a single fixed camera and no moustedounted sensors, the policy must infer depth and object interactions purely from visual input. Timing mismatches cal- ibration drift between the arms can further compound errors, making successful execution sensitive to both perception and control stability.
34

---

Published as a conference paper at ICLR 2026
Figure 11: ViPRA-FM rollouts in world bimanual tasks. Top: Place-in-Bowl. the robot picks up tomato and places it into bowl held by the left arm. Bottom: Kix-with-Whin the robot retrieves a whisk, stirs the bowl contents, and returns the tool. Each sequence shows 10 evenly spaced frames sampled from real world executions.
H.2 BIMANUAL RESULTS
ViPRA-FM is deployed using 14-step action chunks, executed at 7Hz control frequency. This high- frequency chunked control allows the policy to maintain smooth, temporally coherent trajectories while remaining responsive to changing visual inputs. The model also receives short history win- dows input, which helps stabilize motion during contact-heavy transitions and multi-step interac- tions.
In Place-in-Bor the robot completes 10 of 18 trials. Failures were primarily due unsuc- cessful grasps caused by the limited span and compliance of our custom 3D-printed gripper, not the bimanual coordination itself. In all successful grasps. the object was consistently placed into the bowl without collision or instability. This suggests that the policy reliably handles the spatial rea- soning and coordination demands of the task, with grasp robustness being the primary bottleneck, limitation that could be mitigated with more capable gripper design.
In Mix-wi th th-Whisk, the robot completes motion in close proximity to the left arm, requiring continuous spatial alignment between the whisk and bowl. The policy leverages temporal history to stay anchored to the mixing target and uses its chunked control output to produce smooth stirring behavior. The whisk's small, symmetric handle makes easier to grasp. allowing the policy to focus on trajectory accuracy and contact stability throughout the sequence.
of 12 trials. The task involves sustained, curved
Together, these results demonstrate that ViPRA-FM capable of executing complex bimanual tasks using a single vision-conditioned policy and continuous action generation. Additional results, com- parisons, and rollout videos will be shared on our project website. https: ceciipaappjjjjcc. github.io.
35


Reviews:
[{"review_id": "IRoA6pAwJR", "reviewer": "Reviewer_QDdF", "summary": "Authors address the question of how to convert a video prediction model into a robot policy.  They show that instead of directly predicting actions, the video model can predict future visual observations and motion-centric latent actions.  A flow matching decoder to map  these latent actions to robot-specific action sequences.  The entire system runs at 22Hz for low-latency control.  Authors show that the method outperforms baselines.", "questions": "- Fig. 2: The connections between the left / right figures is unclear.  Where does the left figure fit in on the right?  Where does the output of the left figure (the latent actions) go as input on the figure on the right? Is the \"Latent Action Embedding E_\\phi\" block in the right figure represented by the entire left figure?\n\n- If I understand correctly, the paper mentions both discrete latent actions as well as continuous latent actions.  If this is right, then it's unclear what is the difference between these and why the need for both discrete and continuous latent actions.  (The continuous latent actions are then decoded to continuous action chunks.)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method is view agnostic, gets data from humans or robots without the need for action labels, can be applied across robot embodiments, and enables low-latency control.\n\n- The appendix provides a lot of detail on the approach and results in the paper.  This is great for reproducibility.", "weaknesses": "- While the paper claims that the method can generalize across embodiments, I didn't seen strong evidence to substantiate this.  Experiments seem to be on one manipulator arm.\n\n- It will be good to include the limitations of the proposed work in the paper - this is currently missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c7wcvaaBrL", "reviewer": "Reviewer_jonM", "summary": "This paper presents ViPRA, a framework that converts video prediction models into robot policies by learning motion-centric latent actions from unlabeled human and robot videos. Its core contributions are a method to extract these physically-grounded latent actions using perceptual and optical flow losses, a pretraining strategy that jointly predicts future video frames and action sequences, and a flow-matching decoder that enables smooth control.", "questions": "1. How does the model reconcile the fundamental kinematic and dynamic differences between human and robot arms when transferring latent actions? Is there a negative transfer from the \"noise\" of human motion?\n\n2. The entire framework is dependent on the visual perspective of the training videos. Latent actions are also inherently grounded in pixel space. How would performance degrade if the test-time camera viewpoint is different from the training data？", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Creatively combines video prediction, latent actions, and flow matching in a novel \"what\" (future state) + \"how\" (latent action) pretraining paradigm.\n\n2. The paper is well-structured and logically presented, with a clear narrative.\n\n3. The qualitative analysis on latent action representations is interesting.", "weaknesses": "1. The authors did not perform a systematic ablation study on the loss components for the latent action model.\n\n2. The real-world tasks, while commendable, are primarily table-top pick-and-place variants. The paper does not demonstrate generalization to tasks requiring significant non-prehensile manipulation (e.g., pushing, sliding, re-orienting), dynamic environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eBuGBj23k3", "reviewer": "Reviewer_mq1Q", "summary": "This paper proposes ViPRA, a framework for learning robot policy from action-free videos by incorporating video prediction and latent policy learning. The key idea is to pretrain a video–language model to jointly predict both (i) future visual frames and (ii) motion-centric latent actions that summarize local dynamics, guided by perceptual and optical flow consistency losses. These latent representations are mapped to continuous action space through a flow-matching decoder trained on a small number of teleoperated demonstrations.\nThe paper claims that this “video prediction + latent action” pretraining allows robots to leverage large-scale unlabeled human and robot videos, achieving improvements on both simulation benchmark and real-world tasks.", "questions": "Please refer to the weakness part. I would like to raise my score once the concerns are resolved.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper tackles a major challenge in robot learning—leveraging large-scale actionless videos for control. Using video prediction to inject physical dynamics into latent actions is conceptually coherent and builds upon trends in world-model-based control. Evaluation spans both simulation (SIMPLER benchmark) and real-world manipulation (Franka bimanual setup). ViPRA gets better performance comparing against plausible baselines including LAPA, UniVLA, π0, and diffusion-policy variants.", "weaknesses": "1. **Contribution**\n- Unclear novelty: It is not fully clear whether ViPRA introduces a fundamentally new paradigm, or whether it can be viewed as a hybrid of LAPA (latent action tokenization) and UVA (unified video and action prediction).\n\n2. **Codebook Design**\n- Codebook size (|C| = 8) appears extremely small compared to typical VQ-based latent action works (e.g., 128–8192).\n- No ablation or justification is provided for this choice, nor evidence that such a small capacity suffices to capture motion diversity.\n- The paper should include:\n(i) Ablations over codebook size (8 / 32 / 128 / 512 / 8192).\n(ii) Quantitative codebook utilization metrics (entropy, perplexity, diversity).\n\n\n3. Data Scale, Composition, and Scalability\n- Scaling behavior: Although the model claims to leverage “large-scale actionless videos,” the dataset (~400K clips) remains moderate.\nThere is no scaling analysis showing how performance varies with the number of pretraining videos.\n- A performance–vs–data-size curve would clarify whether ViPRA is still data-limited.\n- Human–robot ratio: The impact of mixing human and robot videos is not studied. Different ratios may have large effects on transfer and generalization; sensitivity curves or ablations are needed.\n- Generalization limitation: Without scaling or compositional studies, it is unclear if the model can extend to larger internet-scale video corpora.\n\n4. Latent–Action Semantics and Alignment\n- The alignment between learned latent actions and ground-truth actions (on datasets where GT is available) is not quantified, which would verify whether latent tokens actually capture actionable dynamics rather than visual motion.\n- Would the latent action learned from multiple sources enable a unified action space between multiple embodiments? \n\n5. Optical-Flow Supervision and Robustness\n- The $L_{flow}$  loss is claimed in the contribution but lacks ablation and robustness analysis.\n- How much performance degrades without L_flow?\n- How stable is RAFT-based flow under high ego-motion, blur, or occlusion?\n- Would learned or multi-frame flow estimators perform better?\n\n6. Missing or Incomplete Baselines\n- UVA is cited but not directly compared, despite strong overlap in video-conditioned policy learning. Including UVA as an explicit baseline would help contextualize improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fcc2MwQsfo", "reviewer": "Reviewer_cNx8", "summary": "This work proposes a pretraining-finetuning framework, ViPRA, that learns discrete latent actions from videos without action labels by minimising perceptual and optical flow consistency objectives. Furthermore, this work proposes to use a flow matching decoder to convert the discrete actions to smoother continuous actions, which greatly prevent local discontinuities of previous works. The authors experiments on both simulation and real-world manipulation tasks and shown that the proposed method can achieve stronger performance.", "questions": "I will repost some questions that were mentioned in the weakness section here for clarity.\n\n\n$$\\textbf{Suggestions}$$\nS1: Clarify about how the evaluation metrics is calculated in Table 1 and Figure 3.     \nS2: It would be great if the authors can either amend the claim of 22Hz a bit, or perform some real world experiments at 22Hz.     \nS3: Could provide some clarity about ViPRA-AR since it's not really explained anywhere.    \nS4: Minor: I assume the optical flow model RAFT is pretrained and frozen although it is not explained anywhere. It would be great if the authors could clarify.     \nS5: Minor: Could improve page 22’s readability a bit by aligning them properly (e.g. at the top).    \n\n$$\\textbf{Questions}$$\nQ1: Some related works suggest that pixel-level reconstruction is somewhat not as efficient [1][2], what do the authors think about this? Since this work proposes to learn latent actions, can we argue that it would be more effective to learn latent representation features rather than the full pixel reconstruction?     \nQ2: If I understand correctly, in Table 1, compared to LAPA[3] and UniVLA[4], the proposed ViPRA performs quite better in full success rate, showing that leveraging video prediction is quite helpful to learn robust representations for SIMPLER tasks. However, in LIBERO’s experiments, UniVLA performs best. What do the authors think about this?    \nQ3: Following up on Q2, UniVLA is the best performing framework in LIBERO-10, the authors attributed this to UniVLA being optimised for LIBERO. Can the authors further clarify what this means? For example, what kind of performance can we expect on SIMPLER if we somehow optimise UniVLA for it?    \nQ4. Can we also use KV caching for other related works (LAPA, UniVLA) to get improved speed as well (I believe UniVLA already uses action chunking)? If so, it’s unclear to me how much speed is gained compared to previous works.     \n\n---\n\n[1] G. Zhou et al., DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning, arXiv 2024  \n[2] R. Sun et al., Learning Latent Dynamic Robust Representations for World Models, ICML 2024   \n[3] S. Ye et al., Latent Action Pretraining from Videos, ICLR 2025   \n[4] Q.Bu et al., UniVLA: Learning to Act Anywhere with Task-centric Latent Actions, RSS 2025", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- Very comprehensive related works, greatly aiding the readers in positioning the paper's contributions. \n- The writing was easy to follow and clear, and while the background section is in the appendix, the context provided is enough to have a intuitive understanding of the proposed method. \n- The proposed method regarding representation learning that leverages joint-prediction of pixels and latent actions seems intuitive and shows good performance.\n- The proposed method regarding converting discrete actions to continuous control via flow matching is to my knowledge novel. \n- Conducted comprehensive experiments and analysis, and the experiment results seem impressive (if we can ignore a major weakness for the moment).", "weaknesses": "- Major weakness: For Table 1 and Figure 3, which are the main results in the main manuscript, it is unclear how many seeds/runs/rollouts are experimented and how these values are calculated, so it is not possible to draw any conclusion about the (statistical) significance of the results presented at the moment. \n- \"Up to 22Hz\" in the abstract and introduction is a bit misleading in my opinion, because the real-world experiments were done in 3.5Hz. (i.e. experiments with 22Hz were not done to show the efficacy). \n- Since this work listed on real-world speed as a contribution, several comparison works (LAPA[1]) are not experimented on real world evaluations (i.e. Figure 3) but only in simulation, so it’s unclear how much speed is gained compared to previous works.\n- Minor issues:\n  - Figure 5 is introduced in the manuscript before Figure 4.\n  - Table 6 is not referred nor explained (I assume it’s for appendix E.3, E.4). Furthermore, in Table 6 ViPRA is labelled but should be ViPRA-FM I think.\n  - Apart from KV caching, which is more of an engineering technique for me, it is not obvious why inference speed is gained, since at inference the proposed method performs an extra flow-matching step compared to previous works. \n  - The robot for the real world experiments in Figure 6/8 is not clearly written (I assume it’s Franka).\n  - Not sure how to interpret the results where in LIBERO tasks, $\\pi_0$ and UniVLA both outperform the proposed method ViPRA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KsLzYx9XGm", "reviewer": "Reviewer_1YWh", "summary": "This paper introduces ViPRA, a hierarchical video-based robot control framework that learns discrete latent action representations from large-scale passive human and robot videos and then maps these to continuous robot actions via a flow-matching decoder. The approach jointly predicts future visual observations and latent action sequences during pretraining, and requires only a small amount of teleoperated data for downstream adaptation. The method is evaluated on SIMPLER simulation tasks and several real-world tabletop manipulation tasks, showing improved performance over recent latent-action and vision-language-action baselines.", "questions": "- One of the most exciting implications of this work is that dynamics priors learned from human interactions may transfer across embodiments. Do the authors have any preliminary evidence or insights into what kinds of human video content (e.g., fine manipulation vs. gross motion) most help downstream control? Any failure cases that suggest limits of human-to-robot transfer?\n\n- Have the authors attempted to manipulate latent action tokens intentionally to achieve specific motion patterns (e.g., alter motion direction, smoothness, or speed)? Observing consistency or interpretability here could reveal whether the learned latents are physically meaningful beyond reconstruction.\n\n- Since the method leans heavily on scaling with passive data, at what point does additional video stop helping? Are there any observed regressions or negative-transfer effects when including noisier human videos from less structured environments?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The work provides evidence that hierarchical visuomotor control can benefit from large-scale passive human and robot videos, reducing dependence on action-labeled demonstrations.\n\n- The system attains competitive performance on physical manipulation tasks with only 100–200 teleoperated demonstrations, suggesting good data efficiency in downstream adaptation.\n\n- The integration of flow-matching for action decoding leads to stable, high-frequency command generation suitable for real hardware, addressing common control smoothness limitations in discrete latent policies.\n\n- The paper includes both simulation (SIMPLER) and real-world evaluations, with comparisons against several strong recent methods in the latent-action and VLA literature.", "weaknesses": "- **Limited technical novelty.**\nAside from data sacle, this work appears to be primarily an engineering integration of existing components rather than a new conceptual contribution (i.e. latent action learning, chunked action decoding, and flow-matching control) Similar hierarchical designs have been explored in recent systems such as UniVLA, LAPA, and UniPI.\n\n- **Scaling claim not sufficiently validated.**\nThe paper attributes improvements to pretraining with large-scale passive human videos, but no controlled comparison is provided against training with robot-only video or reduced data subsets. Such experiments would help confirm that data scale, rather than architecture or training strategy, drives the gains.\n\n- **Generalization scope is limited.**\nThe approach is framed as cross-embodiment and generalist, yet evaluations focus on similar 7-DoF manipulators and relatively simple tabletop tasks (pick and place). Broader embodiment diversity or clearer claim boundaries would improve alignment between motivation and evidence.\n\n- **Ablation coverage.**\nThe system includes multiple engineered components (optical flow consistency, VQ bottleneck, multi-stage training). More detailed ablations isolating the effect of each would aid in understanding which design elements are critical to performance.\n\n- **Compute and practicality not discussed.**\nAlthough labeled data requirements are low, the approach requires substantial large-scale video pretraining. Reporting approximate compute/time requirements would help practitioners assess feasibility and compare with alternative methods.\n\nOverall, this work offers a valuable scaling demonstration with solid empirical results, but the methodological novelty is limited, placing it at a borderline accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)