InstructPart: Task-Oriented Part Segmentation with Instruction Reasoning
Authors: name: Zifu Wan; name: Yaqi Xie; name: Ce Zhang; name: Zhiqiu Lin; name: Zihan Wang; name: Simon Stepputtis; name: Deva Ramanan; name: Katia Sycara
Published: 2025-05-23 18:36:13

Abstract: Large multimodal foundation models, particularly in the domains of language
and vision, have significantly advanced various tasks, including robotics,
autonomous driving, information retrieval, and grounding. However, many of
these models perceive objects as indivisible, overlooking the components that
constitute them. Understanding these components and their associated
affordances provides valuable insights into an object's functionality, which is
fundamental for performing a wide range of tasks. In this work, we introduce a
novel real-world benchmark, InstructPart, comprising hand-labeled part
segmentation annotations and task-oriented instructions to evaluate the
performance of current models in understanding and executing part-level tasks
within everyday contexts. Through our experiments, we demonstrate that
task-oriented part segmentation remains a challenging problem, even for
state-of-the-art Vision-Language Models (VLMs). In addition to our benchmark,
we introduce a simple baseline that achieves a twofold performance improvement
through fine-tuning with our dataset. With our dataset and benchmark, we aim to
facilitate research on task-oriented part segmentation and enhance the
applicability of VLMs across various domains, including robotics, virtual
reality, information retrieval, and other related fields. Project website:
https://zifuwan.github.io/InstructPart/.