UniAff: A Unified Representation of Affordances for Tool Usage and
  Articulation with Vision-Language Models
Authors: name: Qiaojun Yu; name: Siyuan Huang; name: Xibin Yuan; name: Zhengkai Jiang; name: Ce Hao; name: Xin Li; name: Haonan Chang; name: Junbo Wang; name: Liu Liu; name: Hongsheng Li; name: Peng Gao; name: Cewu Lu
Published: 2024-09-30 17:52:05

Abstract: Previous studies on robotic manipulation are based on a limited understanding
of the underlying 3D motion constraints and affordances. To address these
challenges, we propose a comprehensive paradigm, termed UniAff, that integrates
3D object-centric manipulation and task understanding in a unified formulation.
Specifically, we constructed a dataset labeled with manipulation-related key
attributes, comprising 900 articulated objects from 19 categories and 600 tools
from 12 categories. Furthermore, we leverage MLLMs to infer object-centric
representations for manipulation tasks, including affordance recognition and
reasoning about 3D motion constraints. Comprehensive experiments in both
simulation and real-world settings indicate that UniAff significantly improves
the generalization of robotic manipulation for tools and articulated objects.
We hope that UniAff will serve as a general baseline for unified robotic
manipulation tasks in the future. Images, videos, dataset, and code are
published on the project website at:https://sites.google.com/view/uni-aff/home