Title: GRIT: Teaching MLLMs to Think with Images
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Reasoning model

Reinforcement learning
Abstract: Recent studies have demonstrated the efficacy of using Reinforcement Learning (RL) in building reasoning models that articulate chains of thoughts prior to producing final answers. However, despite ongoing advances that aim at enabling reasoning for vision-language tasks, existing open-source visual reasoning models typically generate reasoning content with pure natural language, lacking explicit integration of visual information. This limits their ability to produce clearly articulated and visually grounded reasoning chains. To this end, we propose Grounded Reasoning with Images and Texts (GRIT), a novel method for training MLLMs to think with images. GRIT introduces a grounded reasoning paradigm, in which models generate reasoning chains that interleave natural language and explicit bounding box coordinates. These coordinates point to regions of the input image that the model consults during its reasoning process. Additionally, GRIT is equipped with a reinforcement learning approach, GRPO-GR, built upon the GRPO algorithm. GRPO-GR employs robust rewards focused on the final answer accuracy and format of the grounded reasoning output, which eliminates the need for data with reasoning chain annotations or explicit bounding box labels. As a result, GRIT achieves exceptional data efficiency, requiring as few as 20 image-question-answer triplets from existing datasets. Comprehensive evaluations demonstrate that GRIT effectively trains MLLMs to produce coherent and visually grounded reasoning chains, showing a successful unification of reasoning and grounding abilities. All code, data, and checkpoints will be released.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GRIT, a method to train Multimodal Large Language Models (MLLMs) to generate reasoning chains that interleave natural language with explicit bounding box coordinates from input images. By combining reinforcement learning (via GRPO-GR) with task-specific rewards focused on answer accuracy, format correctness, and grounding validity, GRIT enables MLLMs to produce interpretable, visually grounded reasoning without requiring dense annotations. The method achieves strong performance on diverse vision-language tasks (e.g., counting, spatial reasoning, mathematical reasoning) using only 20 training samples. Experiments on models like Qwen2.5-VL and InternVL3 demonstrate improved QA and grounding accuracy.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. GRIT introduces a paradigm shift by explicitly integrating visual grounding into reasoning chains, enabling MLLMs to \"think with images\" rather than just \"about images.\" This bridges the gap between textual reasoning and visual evidence.\n2. The method requires only 20 training samples, showcasing exceptional efficiency compared to existing approaches that often rely on large annotated datasets.\n3. The authors validate their approach across seven datasets and provide both qualitative examples and quantitative analyses. Results show consistent improvements over baselines in answer accuracy and grounding precision.\n4. The generated reasoning chains, with interleaved bounding boxes and text, offer transparency into the model's decision-making process, which is critical for trust and debugging.\n\n### Weaknesses\n1. While GRIT demonstrates success on smaller MLLMs (Qwen2.5-VL-3B, InternVL3-2B), the paper does not evaluate its scalability to larger models or more complex architectures, leaving questions about its applicability to state-of-the-art systems.\n2. Experiments show diminishing returns with larger training datasets (e.g., 7,000 samples), suggesting that further improvements may require more diverse data. This highlights a potential bottleneck in real-world deployment.\n\nQUESTIONS:\n1. The training was conducted using only 20 samples, but it involved 200 training steps with a total batch size of 128, resulting in a computational cost of approximately 96 A100 hours. If we scale this setup linearly, would training with 7,000 samples consume as much as 33,600 A100 hours?\n2. Furthermore, what would be the impact on model performance if the batch size or the number of training steps were reduced?\n3. The data scaling experiments were conducted using `Qwen2.5-VL 2B` (Line 326), which does not appear to exist. Is this a typo?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GRIT (Grounded Reasoning with Images and Texts), a novel framework designed to train multimodal large language models (MLLMs) to generate reasoning chains that are both coherent and visually grounded. Unlike prior work that relies solely on natural language to represent reasoning in vision-language tasks, GRIT enables models to interleave textual explanations with explicit bounding box references, thereby making the reasoning process more interpretable and grounded in visual input.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- The integration of textual explanations with explicit bounding box references enhances the interpretability and expandability of MLLMs, providing more transparent and grounded reasoning.\n- The evaluation is conducted on test sets sampled from seven different datasets, showcasing the model’s generality across diverse vision-language tasks.\n- The paper investigates the effect of scaling the training data and demonstrates that the proposed approach achieves strong performance with very limited supervision, highlighting its data efficiency.\n\n**Weaknesses:**\n\n- The proposed GRPO-GR method is more of an incremental modification—adding new reward terms to the existing GRPO algorithm—rather than a fundamentally novel algorithmic contribution.\n- Even with 7,000 training samples, the model's in-domain accuracy remains below 64%, which may be insufficient for practical deployment in real-world applications.\n\nQUESTIONS:\n1. **Clarification of Acronyms and Terminology**\n\n   * In line 8, the acronym *MLLM* (Multimodal Large Language Model) is used without first introducing its full form. Similarly, *GRPO-GR* in line 13 is not expanded on first use. For clarity and accessibility, please ensure that all acronyms are defined when first introduced.\n   * Additionally, it would be helpful to provide brief definitions for key technical terms such as *\"grounding\"*, particularly for readers from broader machine learning communities who may not be deeply familiar with vision-language grounding terminology.\n\n2. **Details on Dataset Sampling and Test Splits**\n\n   * The paper mentions that evaluation is conducted on data sampled from seven datasets. Could the authors clarify how these samples were selected (e.g., random, stratified) and how many examples from each dataset were used for testing? This would help readers better assess the diversity and robustness of the evaluation protocol.\n\n3. **Missing Appendix or Supplementary Material**\n\n   * The main text (e.g., lines 186, 210, and 215) references an Appendix, but no Appendix or supplementary material was found in the submission. Please clarify whether this content is missing or was inadvertently excluded.\n\nLIMITATIONS:\n* The proposed training method relies on the evaluation of another MLLM (e.g., a GPT-based model) to compute the answer-accuracy reward. This dependency may introduce bias or circular evaluation, and raises questions about the method’s independence and generalizability.\n\n* While GRIT demonstrates effectiveness on tasks such as object counting and spatial reasoning, it remains unclear how well the approach scales to more complex reasoning tasks (e.g., multi-step inference, causal reasoning, or abstract visual questions). Exploring these directions could be a valuable avenue for future work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-06"}

comments: SUMMARY:
This paper proposes GRIT, a framework using GROP to train VLMs to enable models with the grounded reasoning capabilities, where the model outputs grounding boxes during reasoning. Extensive experiments show that the proposed methods can improve the performance of VLMs on various benchmarks effectively.

STRENGTHS AND WEAKNESSES:
Strengths
- An empirically reliable method to improve the performance of VLMs, and in the meantime to incorporate reasoning evidence during output.
- Extensive experiments have conducted to investigate the results.

Weakness
- The proposed method leverages grounded evidence to improve the reasoning accuracy of VLMs. However, no experiments are conducted to show if this method can also facilitate the improvements on Visual Grounding task.

Missing References:
- Wang W, Lv Q, Yu W, et al. Cogvlm: Visual expert for pretrained language models[J]. Advances in Neural Information Processing Systems, 2024, 37: 121475-121499.
- Qi J, Ding M, Wang W, et al. Cogcom: A visual language model with chain-of-manipulations reasoning[C]//ICLR. 2025.

QUESTIONS:
Will the Visual Grounding task (e.g., asking more complex compositional questions) benefit from this grounded reasoning method?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-06