SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by
  Imitating Human Annotator Trajectories
Authors: name: Muzhi Zhu; name: Yuzhuo Tian; name: Hao Chen; name: Chunluan Zhou; name: Qingpei Guo; name: Yang Liu; name: Ming Yang; name: Chunhua Shen
Published: 2025-03-11 17:08:54

Abstract: While MLLMs have demonstrated adequate image understanding capabilities, they
still struggle with pixel-level comprehension, limiting their practical
applications. Current evaluation tasks like VQA and visual grounding remain too
coarse to assess fine-grained pixel comprehension accurately. Though
segmentation is foundational for pixel-level understanding, existing methods
often require MLLMs to generate implicit tokens, decoded through external pixel
decoders. This approach disrupts the MLLM's text output space, potentially
compromising language capabilities and reducing flexibility and extensibility,
while failing to reflect the model's intrinsic pixel-level understanding.
  Thus, we introduce the Human-Like Mask Annotation Task (HLMAT), a new
paradigm where MLLMs mimic human annotators using interactive segmentation
tools. Modeling segmentation as a multi-step Markov Decision Process, HLMAT
enables MLLMs to iteratively generate text-based click points, achieving
high-quality masks without architectural changes or implicit tokens. Through
this setup, we develop SegAgent, a model fine-tuned on human-like annotation
trajectories, which achieves performance comparable to state-of-the-art (SOTA)
methods and supports additional tasks like mask refinement and annotation
filtering.
  HLMAT provides a protocol for assessing fine-grained pixel understanding in
MLLMs and introduces a vision-centric, multi-step decision-making task that
facilitates exploration of MLLMs' visual reasoning abilities. Our adaptations
of policy improvement method StaR and PRM-guided tree search further enhance
model robustness in complex segmentation tasks, laying a foundation for future
advancements in fine-grained visual perception and multi-step decision-making
for MLLMs.