Title: Are Vision Language Models Blind Thinkers on Zero-Shot Multimodal Planning Tasks?
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords: computer vision, machine learning, large language model

Abstract: The advancement of vision language models (VLMs) has empowered embodied agents to accomplish simple multimodal planning tasks, but not long-horizon ones requiring long sequences of actions. In text-only simulations, long-horizon planning has seen significant improvement brought by the LLM-as-formalizer methodology. Instead of directly generating action sequences, LLMs translate the planning domain and problem into a formal planning language like the Planning Domain Definition Language (PDDL), which can evoke a formal solver to derive the plan in a verifiable manner. In multimodal environments, research of VLM-as-Formalizer remains scarce, usually involving gross simplifications such as predefined object vocabulary or overly similar few-shot examples. In this work, we present a suite of five \vaf pipelines that tackle one-shot, open-vocabulary, and multimodal PDDL formalization. We evaluate those on two existing datasets while presenting another two on the Blocksworld and ALFRED domains that for the first time account for authentic visual scenes, multi-view, occlusion, and motion blur. We conclude that VLM-as-Formalizer outperforms end-to-end plan generation across multiple domains, while revealing VLMs as blind thinkers bottlenecked by vision rather than language especially in capturing an exhaustive set of necessary object relations. We find that inference-time techniques such as scene graph generation compensate for visual performance, though leaving headroom for future research directions on visual PDDL generation.