Title: Compositional Multimodal Reasoning for Long-Horizon Robotic Manipulation in Scientific Experiments
Primary area: applications to robotics, autonomy, planning
Keywords: long-horizon planning, manipulation planning, vision-language-action, multimodal reasoning

Abstract: Long-horizon robotic manipulation in laboratory environments presents great challenges requiring strict procedural dependencies, fine-grained object manipulation, and domain expertise. While recent work integrates large language models (LLMs) with multimodal perception for task planning, they often require task-specific fine-tuning and struggle to generalize to scientific workflows that demand compositional reasoning beyond spatial grounding. To address these challenges, we propose Compositional Multimodal Planner (CoMP), a decoupled hierarchical reasoning framework for long-horizon robotic planning in scientific experiments. (1) a task-level planner using chain-of-thought prompting to capture task logic, (2) a mid-level multimodal planner that integrates future scene prediction to enable visually grounded reasoning, and (3) a low-level skill controller that executes actions via reinforcement learning. This compositional and decoupled design allows independent optimization of each component, enabling controllable and extensible planning without requiring fine-tuning of large models. To support evaluation, we introduce a benchmark dataset for scientific experiment tasks. Experiments on both our benchmark and RLBench demonstrate that CoMP consistently outperforms strong baselines, demonstrating its effectiveness and generalizability in long-horizon robotic planning.