HyCodePolicy: Hybrid Language Controllers for Multimodal Monitoring and
  Decision in Embodied Agents
Authors: name: Yibin Liu; name: Zhixuan Liang; name: Zanxin Chen; name: Tianxing Chen; name: Mengkang Hu; name: Wanxi Dong; name: Congsheng Xu; name: Zhaoming Han; name: Yusen Qin; name: Yao Mu
Published: 2025-08-04 17:18:14

Abstract: Recent advances in multimodal large language models (MLLMs) have enabled
richer perceptual grounding for code policy generation in embodied agents.
However, most existing systems lack effective mechanisms to adaptively monitor
policy execution and repair codes during task completion. In this work, we
introduce HyCodePolicy, a hybrid language-based control framework that
systematically integrates code synthesis, geometric grounding, perceptual
monitoring, and iterative repair into a closed-loop programming cycle for
embodied agents. Technically, given a natural language instruction, our system
first decomposes it into subgoals and generates an initial executable program
grounded in object-centric geometric primitives. The program is then executed
in simulation, while a vision-language model (VLM) observes selected
checkpoints to detect and localize execution failures and infer failure
reasons. By fusing structured execution traces capturing program-level events
with VLM-based perceptual feedback, HyCodePolicy infers failure causes and
repairs programs. This hybrid dual feedback mechanism enables self-correcting
program synthesis with minimal human supervision. Our results demonstrate that
HyCodePolicy significantly improves the robustness and sample efficiency of
robot manipulation policies, offering a scalable strategy for integrating
multimodal reasoning into autonomous decision-making pipelines.