GRIT: Teaching MLLMs to Think with Images
Authors: name: Yue Fan, name: Xuehai He, name: Diji Yang, name: Kaizhi Zheng, name: Ching-Chen Kuo, name: Yuting Zheng, name: Sravana Jyothi Narayanaraju, name: Xinze Guan, name: Xin Eric Wang

Recent studies have demonstrated the efficacy of using Reinforcement Learning
(RL) in building reasoning models that articulate chains of thoughts prior to
producing final answers. However, despite ongoing advances that aim at enabling
reasoning for vision-language tasks, existing open-source visual reasoning
models typically generate reasoning content with pure natural language, lacking
explicit integration of visual information. This limits their ability to
produce clearly articulated and visually grounded reasoning chains. To this
end, we propose Grounded Reasoning with Images and Texts (GRIT), a novel method
for training MLLMs to think with images. GRIT introduces a grounded reasoning
paradigm, in which models generate reasoning chains that interleave natural
language and explicit bounding box coordinates. These coordinates point to
regions of the input image that the model consults during its reasoning
process. Additionally, GRIT is equipped with a reinforcement learning approach,
GRPO-GR, built upon the GRPO algorithm. GRPO-GR employs robust rewards focused
on the final answer accuracy and format of the grounded reasoning output, which
eliminates the need for data with reasoning chain annotations or explicit
bounding box labels. As a result, GRIT achieves exceptional data efficiency,
requiring as few as 20 image-question-answer triplets from existing datasets.
Comprehensive evaluations demonstrate that GRIT effectively trains MLLMs to
produce coherent and visually grounded reasoning chains, showing a successful
unification of reasoning and grounding abilities.