KUDA: Keypoints to Unify Dynamics Learning and Visual Prompting for
  Open-Vocabulary Robotic Manipulation
Authors: name: Zixian Liu; name: Mingtong Zhang; name: Yunzhu Li
Published: 2025-03-13 16:59:17

Abstract: With the rapid advancement of large language models (LLMs) and
vision-language models (VLMs), significant progress has been made in developing
open-vocabulary robotic manipulation systems. However, many existing approaches
overlook the importance of object dynamics, limiting their applicability to
more complex, dynamic tasks. In this work, we introduce KUDA, an
open-vocabulary manipulation system that integrates dynamics learning and
visual prompting through keypoints, leveraging both VLMs and learning-based
neural dynamics models. Our key insight is that a keypoint-based target
specification is simultaneously interpretable by VLMs and can be efficiently
translated into cost functions for model-based planning. Given language
instructions and visual observations, KUDA first assigns keypoints to the RGB
image and queries the VLM to generate target specifications. These abstract
keypoint-based representations are then converted into cost functions, which
are optimized using a learned dynamics model to produce robotic trajectories.
We evaluate KUDA on a range of manipulation tasks, including free-form language
instructions across diverse object categories, multi-object interactions, and
deformable or granular objects, demonstrating the effectiveness of our
framework. The project page is available at http://kuda-dynamics.github.io.