A Vision-Language-Action-Critic Model for Robotic Real-World
  Reinforcement Learning
Authors: name: Shaopeng Zhai; name: Qi Zhang; name: Tianyi Zhang; name: Fuxian Huang; name: Haoran Zhang; name: Ming Zhou; name: Shengzhe Zhang; name: Litao Liu; name: Sixu Lin; name: Jiangmiao Pang
Published: 2025-09-19 12:44:29

Abstract: Robotic real-world reinforcement learning (RL) with vision-language-action
(VLA) models is bottlenecked by sparse, handcrafted rewards and inefficient
exploration. We introduce VLAC, a general process reward model built upon
InternVL and trained on large scale heterogeneous datasets. Given pairwise
observations and a language goal, it outputs dense progress delta and done
signal, eliminating task-specific reward engineering, and supports one-shot
in-context transfer to unseen tasks and environments. VLAC is trained on
vision-language datasets to strengthen perception, dialogic and reasoning
capabilities, together with robot and human trajectories data that ground
action generation and progress estimation, and additionally strengthened to
reject irrelevant prompts as well as detect regression or stagnation by
constructing large numbers of negative and semantically mismatched samples.
With prompt control, a single VLAC model alternately generating reward and
action tokens, unifying critic and policy. Deployed inside an asynchronous
real-world RL loop, we layer a graded human-in-the-loop protocol (offline
demonstration replay, return and explore, human guided explore) that
accelerates exploration and stabilizes early learning. Across four distinct
real-world manipulation tasks, VLAC lifts success rates from about 30\% to
about 90\% within 200 real-world interaction episodes; incorporating
human-in-the-loop interventions yields a further 50% improvement in sample
efficiency and achieves up to 100% final success.