PixelRefer: A Unified Framework for Spatio-Temporal Object Referring
  with Arbitrary Granularity
Authors: name: Yuqian Yuan, name: Wenqiao Zhang, name: Xin Li, name: Shihao Wang, name: Kehan Li, name: Wentong Li, name: Jun Xiao, name: Lei Zhang, name: Beng Chin Ooi

Multimodal large language models (MLLMs) have demonstrated strong
general-purpose capabilities in open-world visual comprehension. However, most
existing MLLMs primarily focus on holistic, scene-level understanding, often
overlooking the need for fine-grained, object-centric reasoning. In this paper,
we present PixelRefer, a unified region-level MLLM framework that enables
advanced fine-grained understanding over user-specified regions across both
images and videos. Motivated by the observation that LLM attention
predominantly focuses on object-level tokens, we propose a Scale-Adaptive
Object Tokenizer (SAOT) to generate compact and semantically rich object
representations from free-form regions. Our analysis reveals that global visual
tokens contribute mainly in early LLM layers, inspiring the design of
PixelRefer-Lite, an efficient variant that employs an Object-Centric Infusion
module to pre-fuse global context into object tokens. This yields a lightweight
Object-Only Framework that substantially reduces computational cost while
maintaining high semantic fidelity. To facilitate fine-grained instruction
tuning, we curate PixelRefer-2.2M, a high-quality object-centric instruction
dataset. Extensive experiments across a range of benchmarks validate that
PixelRefer achieves leading performance with fewer training samples, while
PixelRefer-Lite offers competitive accuracy with notable gains in efficiency.