Detect Anything via Next Point Prediction
Authors: name: Qing Jiang; name: Junan Huo; name: Xingyu Chen; name: Yuda Xiong; name: Zhaoyang Zeng; name: Yihao Chen; name: Tianhe Ren; name: Junzhi Yu; name: Lei Zhang
Published: 2025-10-14 17:59:54

Abstract: Object detection has long been dominated by traditional coordinate
regression-based models, such as YOLO, DETR, and Grounding DINO. Although
recent efforts have attempted to leverage MLLMs to tackle this task, they face
challenges like low recall rate, duplicate predictions, coordinate
misalignment, etc. In this work, we bridge this gap and propose Rex-Omni, a
3B-scale MLLM that achieves state-of-the-art object perception performance. On
benchmarks like COCO and LVIS, Rex-Omni attains performance comparable to or
exceeding regression-based models (e.g., DINO, Grounding DINO) in a zero-shot
setting. This is enabled by three key designs: 1) Task Formulation: we use
special tokens to represent quantized coordinates from 0 to 999, reducing the
model's learning difficulty and improving token efficiency for coordinate
prediction; 2) Data Engines: we construct multiple data engines to generate
high-quality grounding, referring, and pointing data, providing semantically
rich supervision for training; \3) Training Pipelines: we employ a two-stage
training process, combining supervised fine-tuning on 22 million data with
GRPO-based reinforcement post-training. This RL post-training leverages
geometry-aware rewards to effectively bridge the discrete-to-continuous
coordinate prediction gap, improve box accuracy, and mitigate undesirable
behaviors like duplicate predictions that stem from the teacher-guided nature
of the initial SFT stage. Beyond conventional detection, Rex-Omni's inherent
language understanding enables versatile capabilities such as object referring,
pointing, visual prompting, GUI grounding, spatial referring, OCR and
key-pointing, all systematically evaluated on dedicated benchmarks. We believe
that Rex-Omni paves the way for more versatile and language-aware visual
perception systems.