ChatRex: Taming Multimodal LLM for Joint Perception and Understanding
Authors: name: Qing Jiang; name: Gen luo; name: Yuqin Yang; name: Yuda Xiong; name: Yihao Chen; name: Zhaoyang Zeng; name: Tianhe Ren; name: Lei Zhang
Published: 2024-11-27 14:11:10

Abstract: Perception and understanding are two pillars of computer vision. While
multimodal large language models (MLLM) have demonstrated remarkable visual
understanding capabilities, they arguably lack accurate perception abilities,
e.g. the stage-of-the-art model Qwen2-VL only achieves a 43.9 recall rate on
the COCO dataset, limiting many tasks requiring the combination of perception
and understanding. In this work, we aim to bridge this perception gap from both
model designing and data development perspectives. We first introduce ChatRex,
an MLLM with a decoupled perception design. Instead of having the LLM directly
predict box coordinates, we feed the output boxes from a universal proposal
network into the LLM, allowing it to output the corresponding box indices to
represent its detection results, turning the regression task into a
retrieval-based task that LLM handles more proficiently. From the data
perspective, we build a fully automated data engine and construct the
Rexverse-2M dataset which possesses multiple granularities to support the joint
training of perception and understanding. After standard two-stage training,
ChatRex demonstrates strong perception capabilities while preserving multimodal
understanding performance. The combination of these two capabilities
simultaneously unlocks many attractive applications, demonstrating the
complementary roles of both perception and understanding in MLLM. Code is
available at https://github.com/IDEA-Research/ChatRex.