Viveka-500M (preview)

A 523M-parameter Vision-Language-Action model for real-robot manipulation (Piper 7-DoF single arm; ALOHA 14-DoF bimanual), from the paper draft "Viveka: Self-Predictive Embodied Slots for Surprise-Gated Action Chunking".

Status: research preview, trained for one night on one GPU. Not a general-purpose policy. It was trained on 1,057 Piper teleoperation episodes from essentially one scene (3 instructions) plus 150 ALOHA episodes; expect it to work only in that setting. See Limitations.

Architecture

• Vision: DINOv2 ViT-B/14 (pretrained, fine-tuned at 0.1x LR), 224 px, 2 views + 0.2 s history frame, instruction FiLM on the last 4 blocks
• Language: 6-layer byte-level transformer trained from scratch (no pretrained tokenizer or LM)
• Embodied State Former: 32 learned slots + proprio + embodiment tokens (8 layers, d=1024); slot-temporal (4 layers)
• Action expert: 12-layer flow-matching DiT, 48-step chunks (1.6 s at 30 Hz), 1-10 Euler steps
• Self-predictive slots: action-conditioned JEPA head forecasting the slots at +16/+32/+48 steps (EMA target); its forecast error ("latent surprise") can trigger re-planning at run time

Component	Parameters
Visual encoder (ConvViT + FiLM)	88,880,640
Language encoder (byte transformer)	18,867,456
Multimodal fusion (projections)	1,314,816
Embodied representation (ESF slots)	135,343,104
Temporal modelling (slot history)	50,605,568
Latent dynamics predictor (JEPA)	44,263,168
Action policy (flow DiT)	183,010,816
Other (instruction-matching head)	1,050,625
Total trainable	523,336,193
EMA target (not trained, buffer-like)	244,406,016

Measured results (held-out Piper episodes)

Metric	Viveka-500M	Hold-position baseline
Open-loop arm error, 48-step chunk (deg)	3.62	4.64
Open-loop arm error, first 8 steps (deg)	2.62	1.32
Gripper open/close agreement	93.5%	-
Error increase with cameras blanked / swapped	+170% / +288%	-
Error under dark / colour-cast / occlusion / blur	0.0605 / 0.0607 / 0.0611 / 0.0714 rad (clean 0.0605)	-
Stalled-scene detection ROC-AUC (+48 steps), own forecast vs no-dynamics	0.968 vs 0.008	-
Latency per chunk, batch 1, 1 / 5 / 10 flow steps	25 / 35 / 48 ms	-
Peak inference VRAM	2.0 GB	-

The history frame is currently unused (removing it changes error by +0%). 420 queries from 12 held-out episodes, single training run. Note: over the first 8 steps of a chunk the policy is less accurate than holding still; keep the node's rate limiter on.

Files

• model.safetensors: fp32 weights (EMA training target and optimizer removed), 1.95 GB
• config.json: full training config; norms.json: per-embodiment action/state normalisers; eval.json: checkpoint validation metrics
• Code: https://github.com/Aditharavind/viveka-500m (private)

Use (Piper)

from inference.piper_agent import VivekaPiperAgent      # from the GitHub repo
agent = VivekaPiperAgent("path/to/this/folder", instruction="pick the yellow block put into the box", ode_steps=10)
chunk_rad = agent.predict(joint_state_rad_7, {"cam_global": rgb_1280x720, "cam_gripper": rgb_640x480})  # (48, 7)

ROS 2 node: robotics/piper_node_viveka500m.py (see its docstring for the recommended flags). Inference uses < 2 GB of VRAM.

Limitations

• One scene, 3 instructions: language following and transfer to new environments are barely tested. Strong appearance augmentation was used, but real robustness needs more diverse demonstrations.
• Offline (open-loop) metrics only so far; closed-loop real-robot success rates are pending.
• The latent-surprise trigger has been evaluated only against synthetic deviations so far.
• ALOHA (bimanual) heads were trained on 150 episodes and are not evaluated here.