Title: MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Large Laguage Models

Chain-of-Thought

Mathematics
Abstract: Chain-of-Thought (CoT) has widely enhanced mathematical reasoning in Large Language Models (LLMs), but it still remains challenging for extending it to multimodal domains. Existing works either adopt a similar textual reasoning for image input, or seek to interleave visual signals into mathematical CoT. However, they face three key limitations for math problem-solving: *reliance on coarse-grained box-shaped image regions, limited perception of vision encoders on math content, and dependence on external capabilities for visual modification*.  In this paper, we propose **MINT-CoT**, introducing **M**athematical **IN**terleaved **T**okens for **C**hain-**o**f-**T**hought visual reasoning. MINT-CoT adaptively interleaves relevant visual tokens into textual reasoning steps via an Interleave Token, which dynamically selects visual regions of any shapes within math figures. To empower this capability, we construct the MINT-CoT dataset, containing 54K mathematical problems aligning each reasoning step with visual regions at the token level, accompanied by a rigorous data generation pipeline. We further present a three-stage MINT-CoT training strategy, progressively combining text-only CoT SFT, interleaved CoT SFT, and interleaved CoT RL, which derives our MINT-CoT-7B model. Extensive experiments demonstrate the effectiveness of our method for effective visual interleaved reasoning in mathematical domains, where MINT-CoT-7B outperforms the baseline model by +34.08% on MathVista and +28.78% on GeoQA, respectively.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a chain-of-thought method, MINT-CoT, for multimodal mathematical reasoning. Its core contribution lies in introducing the Interleave Token mechanism, which dynamically selects visual tokens related to the current mathematical semantics at each reasoning step and interleaves them into the textual reasoning process to achieve collaborative visual-textual reasoning. To support this method, the authors construct a 54K-scale MINT-CoT dataset and design a three-stage training process, gradually transitioning from purely textual supervision to reinforcement learning-based visual interleaving training.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\t1.The proposed Interleave Token mechanism enables dynamic insertion of visual tokens at each step of textual reasoning.\n\t2.The performance gains are significant, with improvements of 26.92% and 32.59% on GeoQA and MathVista-Math subtasks, respectively.\n\nWeaknesses:\n\t1.The method essentially relies on traditional similarity matching principles; the Interleave Token is effectively a formalized expression of a query-like vector in an attention mechanism, lacking structural novelty.\n\t2.The method heavily depends on high-quality alignment between visual regions and reasoning steps in the dataset. Although the token index annotation process is automated, inaccuracies in the mapping can significantly degrade performance, limiting generalizability in real-world applications.\n\t3.Figure 3 is not clearly presented, with small fonts and repetitive data processing descriptions, making the methodological pipeline appear redundant and less intuitive. The coordination between textual explanation and illustration is suboptimal.\n\t4.Although the reinforcement learning stage adopts the GRPO strategy, there is no description of implementation details, making it difficult to assess the effectiveness of this phase.\n\nQUESTIONS:\n1.The paper does not specify whether there is any limit or sparsity constraint on the number of visual tokens inserted. Have there been instances where excessive tokens caused reasoning failures?\n2.Does token selection behavior differ statistically between algebraic and geometric problems? Are there task-specific biases in token filtering?\n3.You use GPT-4o to extract keywords in dataset annotation. Have you evaluated the impact of alternative extraction methods on final performance? There appears to be full reliance on GPT-4o, raising concerns about potential error propagation.\n4.If OCR or keyword extraction fails, would visual token selection break down entirely? Have you incorporated such simulated error scenarios during training to improve robustness?\n\nLIMITATIONS:\nNo. The checklist claims that the limitations of the work are included in the appendix, but in fact, no such discussion could be found.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes MINT-CoT, a novel approach to enhancing mathematical reasoning in multi-modal large language models (MLLMs) through interleaved token-level interleaving Chain-of-Thought (CoT). Central to this approach is the MINT-CoT dataset of 54K problems, where each reasoning step is explicitly grounded to relevant visual regions. The authors design a three-stage training pipeline involving text-only CoT supervised fine-tuning (SFT), interleaved CoT SFT, and interleaved CoT reinforcement learning (RL), culminating in the MINT-CoT-7B model. The model achieves strong gains on visual math benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n1. The paper addresses a fundamental challenge in multi-modal mathematical reasoning.\n2. The introduction of an Interleave Token that dynamically selects arbitrary-shaped visual regions per reasoning step is novel.\n3. The construction of a large-scale dataset (54K samples) with token-level alignment between text and vision is a significant contribution to the community. \n4.  MINT-CoT-7B achieves substantial performance improvements over baselines on two challenging benchmarks (MathVista and GeoQA).\n\n## Weaknesses:\n1. Unclear Visual Perception Capability: While the paper emphasizes fine-grained interleaving of visual tokens, it relies on a math-domained variant of CLIP (MAVIS) for visual grounding. However, CLIP and its derivatives are known to capture coarse-grained semantics, which raises concerns about their ability to accurately perceive fine-grained visual elements such as small geometric cues, labels, or spatial relationships crucial in mathematical diagrams. The paper does not provide direct evidence that MINT-CoT meaningfully overcomes these perceptual limitations.  Although the model shows improved final-answer accuracy on benchmarks like MathVista and GeoQA, it remains unclear whether the interleaved visual steps are always semantically or visually correct. There is no diagnostic analysis or qualitative study examining failures in intermediate reasoning, which could be masked by correct final answers but reflect flaws in the model’s step-wise visual understanding.\n2. Questionable Necessity of Multimodal CoT for Geometric Figures:  The paper assumes that visual interleaving is beneficial for geometric figures. However, many such diagrams consist of simple symbolic elements—lines, points, circles, triangles—which lack rich semantics and can be easily and precisely described in natural language. Given that LLMs are already capable of understanding these concepts from large-scale text-only training, it's unclear whether introducing visual tokens truly adds value. In fact, there is no direct evidence provided to show that the model understands these visual tokens better than their natural language descriptions. For example, does the model truly understand the meaning of the selected visual tokens? what happens if a portion of these selected tokens is randomly dropped or replaced with irrelevant ones?\n3. Unclear Advantage of Token-Level Visual Grounding over Box-Level: While the paper criticizes box-level grounding for including irrelevant visual content, visual tokens are also subject to noise and overlap, especially in complex diagrams. For example, a token covering a \"circle\" may still partially include other elements (e.g., lines or text). Furthermore, the empirical improvement in Table 4 over the box-based OCT baseline is minimal. \n4. Missing Ablation on the Effectiveness of Interleaved CoT: The paper lacks a key ablation study to isolate the impact of the proposed interleaved multimodal CoT reasoning. Specifically, it remains unclear how much of the performance gain comes from the MINT-CoT dataset itself versus the interleaving strategy. A natural baseline would be to train the model on the MINT-CoT dataset without interleaved visual tokens, using only text-based CoT supervision, and compare performance. Without this, it is difficult to determine whether the interleaved design is truly responsible for the observed improvements or whether the gains stem largely from exposure to large-scale curated training data.\n\nQUESTIONS:\n1. Resolution Dependency of Token Selection: Since the number of visual tokens depends on input image resolution, how do you handle ground-truth labels for active visual tokens when resolution changes? Do you apply any post-processing to align token indices across different resolutions?\n2. Efficiency Concerns: How many visual tokens are input during reasoning on average? Could you provide details about the training time and inference time overhead introduced by interleaving visual tokens compared to the baseline?\n3. Number of Interleave Tokens: How many Interleave Tokens are leant during training? \n4. Design Choice in Token Selection Loss (Eq. 7): Why is the token selection supervised with a binary cross-entropy loss (Eq. 7) rather than integrated directly into the next-token prediction objective (Eq. 6)? For instance, could token selection be formulated as a number secretion over L visual tokens, jointly optimized during sequence modeling? You can number the visual tokens from 1 to L, and predict which token number is selected.\n5. Domain Specificity of Visual Tokens: What justifies the use of selected visual tokens specifically for geometric math reasoning? I think this token election can be generalize to natural images or more general domains.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a method for visual math reasoning where an interleave token at each reasoning step is used to select additional relevant visual tokens for reasoning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: the research is well-motivated. The proposed method is conceptually simple, and its effectiveness is validated on a standard visual math benchmark.\n\nWeaknesses: \n- the method is task or dataset specific. I am not convinced that the method can be easily applied to other visual reasoning tasks.\n- the evaluation is performed only on MathVista. More, different benchmarks need to be used to validate how general the method can be applied.\n- Need to clarify the use of \"interleave tokens\".  e.g., Note that an interleave token is the output of a reasoning step. (without visual tokens), Do we assume that the interleave token might not be optimal since it is generated without visual info, or do we simply assume that the ground-truth reasoning steps are given -- therefore the proposed method is a data curation method that enhance the reasoning steps with additional visual tokens?\n\nQUESTIONS:\nAs listed in the \"weaknesses\" section above, copied below\n\n- Need to clarify the use of \"interleave tokens\".  e.g., Note that an interleave token is the output of a reasoning step. (without visual tokens), Do we assume that the interleave token might not be optimal since it is generated without visual info, or do we simply assume that the ground-truth reasoning steps are given -- therefore the proposed method is a data curation method that enhance the reasoning steps with additional visual tokens?\n- the method is task or dataset specific. I am not convinced that the method can be easily applied to other visual reasoning tasks.\n- the evaluation is performed only on MathVista. More, different benchmarks need to be used to validate how general the method can be applied.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims at improving the mathematical reasoning capabilities of the VL models using the visual signals in the chain-of-thoughts. Specifically, it argues that the prior work does not focus on the fine-grained visual details (e.g., rely on box-level cues), and require external tools which come with additional inference costs and large-scale training. To resolve this, the paper proposes MINT-CoT framework. In particular, the authors (a) curate 50K problems with fine-grained visual cue annotations, (b) train a vision-language model to focus on the cues via an interleave tokens, and (c) performs progressive training (e.g., text-only, SFT, and RL) to further refine the model performance. Ultimately, the authors show that the method achieves large improvements in the performance on MathVista and GeoQA dataset.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The experimental design is quite interesting. The idea behind creating a nice dataset with fine-grained visual cues will be a very usual artifact for the community. \n2. The method to train an interleave token that decides the regions to focus on automatically is novel and seems to work well on MathVista and GeoQA.\n3. The three-stage training pipeline is also sensible and the ablation studies showcase the usefulness of diverse design choices (Table 3-4).\n\nWeaknesses\n\n1. The choice of the base model as Qwen-2-VL-7B seems strange to me given Qwen-2.5-VL-7B exists in Table 1. For instance, Qwen-2.5-VL achieves 66.6% on MathVista instead of 40% Qwen-2-VL-7B. It is unclear if the method can be utilized to improve the frontier VLMs or not. I imagine that the gains will not look as large as ones claimed in the paper (30% on MathVista) once you train with Qwen-2.5-VL model.\n2. The evaluation is quite narrow – just MathVista and GeoQA. Infact, GeoQA is a part of MathVista already. There are many more evaluations that are quite popular – MathVerse, MathVision, We-Math, MMMU-Pro that should be tested. There should be some general-purpose evals too like VQAv2, GQA, Blink, ChartQA etc to understand the impact of such training on overall visual capabilities of the model. In addition, it would be better to include the numbers for the entire mathvista data instead of a particular subset.\n3. As suggested by Table 3 and 4, it seems that the biggest gain over the base model is from text-only CoT which is the default method nowadays. In this light, I believe that the paper overclaims its numbers. Instead of saying +34% on MathVista over the base model, I think the right baseline should be text-only CoT (default) method for reasoning training. \n4. It is not clear if text-only SFT followed by text-only RL is worse than MINT-CoT framework or not. It is a valid baseline which seems missing in the current setup.\n\nQUESTIONS:\nMentioned in strengths and weaknesses\n\nLIMITATIONS:\nMentioned in strengths and weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-07-02"}