Title: Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Vision-Language Models

Multimodal Chain-of-Thought
Abstract: Large Vision-Language Models (LVLMs) have achieved significant success in multimodal tasks, with multimodal chain-of-thought (MCoT) further enhancing performance and interpretability. Recent MCoT methods fall into two categories: (i) Textual-MCoT (T-MCoT), which takes multimodal input and produces textual output; and (ii) Interleaved-MCoT (I-MCoT), which generates interleaved image-text outputs. Despite advances in both approaches, the mechanisms driving these improvements are not fully understood.  To fill this gap, we first reveal that MCoT boosts LVLMs by incorporating $\textit{visual thoughts}$, which convey image information to the reasoning process regardless of the MCoT format, depending only on clarity and conciseness of expression. Furthermore, to explore visual thoughts systematically, we define four distinct forms of visual thought expressions and analyze them comprehensively. Our findings demonstrate that these forms differ in clarity and conciseness, yielding varying levels of MCoT improvement. Additionally, we explore the internal nature of visual thoughts, finding that visual thoughts serve as intermediaries between the input image and reasoning to deeper transformer layers,  enabling more advanced visual information transmission. We hope that the visual thoughts can inspire further breakthroughs for future MCoT research.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDespite advancements in T-MCoT and I-MCoT, few studies analyze the underlying unified mechanisms behind these improvements in MCoT. The paper attributes these performance enhancements to visual thoughts, the reasoning steps that convey visual information from the input images.  The authors further propose four strategies that represent different modalities and approaches to integrate visual thoughts, providing a comprehensive analysis of each strategy.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The paper conducts a systematic evaluation of four visual thought expressions (N-LANG, S-LANG, E-IMG, G-IMG) across multiple tasks (MMVP, V*Bench, M3CoT, CoMT). This multi-task, multi-model approach strengthens the robustness of the findings by demonstrating the effectiveness of visual thoughts across diverse scenarios.\n2. The internal analysis of visual thoughts (Section 5) using attention distribution (Figure 7) and information flow (Figures 8–9) within LVLMs provides a novel perspective on how visual information is processed. This adds depth to the study, as it connects empirical results to model mechanics.\n\n**Weakness:**\n1. The experiments primarily focus on a small set of LVLMs (e.g., GPT-4o, LLaVA), which limits the generalizability of the findings. Without testing on a broader range of architectures, the claim of a “unified perspective” is weakened.\n2. The correlation analysis of clarity and conciseness (Figure 6) relies on manual scoring (0–3 scale). This introduces subjectivity and reduces the reproducibility of the results.\n3. While Table 1 reports performance across tasks, the paper lacks a detailed discussion of why certain visual thought expressions perform better in specific tasks. For example, the superior performance of E-IMG in V* Bench-attributes is noted but not thoroughly analyzed, which limits the interpretability of the results.\n\nQUESTIONS:\n1. The experiments primarily use LLaVA-1.5-7B for attention and information flow analyses, and it is unclear how generalizable the findings are across different LVLM architectures or scales. Could the authors extend their analysis to at least one additional LVLM with a different architecture or scale to validate the robustness of visual thoughts?\n2. The authors claim that removing visual thoughts and forcing reasoning solely from the original image can impair performance more than reasoning directly from the query, as demonstrated by the Math-Breakpoint results in Figure 4 (b). This finding appears counterintuitive and lacks adequate discussion. Furthermore, is it appropriate to draw such a broad conclusion based on a single experiment?\n3. Image-based visual thoughts (E-IMG, G-IMG) rely on external tools such as Grounding-DINO and DALL-E 3, which may introduce inaccurate segmentations or irrelevant generated images. How do such errors impact reasoning performance, and how are they mitigated? Could the authors evaluate the sensitivity of MCoT performance to errors in visual thought generation (e.g., by injecting controlled noise or using lower-quality tools)?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces and analyzes the concept of 'visual thoughts', intermediate reasoning steps in Multimodal Chain-of-Thought (MCoT) reasoning with Vision-Language Models (VLMs). These visual thoughts, expressed as either text or images, serve as distilled representations of visual input that differentially enhance VLM performance across tasks. Acting as a form of internal visual cache, visual thoughts reduce the need for repeated image processing and help propagate visual information into deeper layers of the model during reasoning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- The paper offers practical insights into the suitability of different types of CoT strategies for various tasks. These observations are well-motivated and could guide future work in selecting or designing appropriate MCoT strategies depending on task characteristics.\n\n- The mechanistic analysis of internal model behavior, particularly attention shifts and information flow, is interesting and contributes meaningfully to our understanding of how visual reasoning unfolds in large vision-language models.\n\n**Weaknesses:**\n\n- **Effectiveness verification of visual thought.** I have some concerns regarding the methodology for the 'w/o visual thought' and 'text-form visual thought' baselines. If these are generated by simply replacing the image in the image-form visual thought outputs with a placeholder or caption, the surrounding text may still implicitly reference the image, which could introduce inconsistencies in the context. This may negatively impact performance not solely due to the absence of the visual thought, but because the resulting context is no longer coherent. Additionally, the authors state that removing visual thoughts leads to performance worse than using the query alone, but Figure 4b shows that the query-only setting is consistently the lowest-performing or close to it, calling this claim into question.\n\n- **Baselines.** The 'w/o VT' baseline of Section 4 is unclear: does it correspond to the 'w/o visual thought' from Figure 4? If so, the paper should clarify this explicitly. Additionally, it would be useful to include a vanilla baseline without any CoT reasoning at all, to better isolate the value added by CoT versus visual thoughts.\n\n- **Visual Attention Analysis.** The conclusions drawn from the attention plots are not entirely convincing. While the paper claims that visual thought tokens receive higher attention in deeper layers, the observed differences appear subtle, and the attention to visual thoughts does not clearly dominate over original attention to the image. Stronger quantitative evidence would strengthen this analysis.\n\n**Typos:**\n\nThe manuscript contains some minor typos, including:\n\n- line 154 'Does these different Visual Thoughts all works?' -> 'Do these different Visual Thoughts all work?'\n- line 188 'Which scenarios does different Visual Thoughts work better?' -> 'In which scenarios do different Visual Thoughts work better?'\n- line 176 it is unclear what 'in rationale' means in this context\n- line 264 it is unclear what 'boarder' means in this context\n\nQUESTIONS:\n- What metric is plotted on the y-axis of Figure 5? Why do all columns add up to 1? Please clarify in the caption or main text.\n- In Section 5.2, how exactly is the intervention performed to disrupt the information flow from visual thoughts or the image? Is this based on manipulating attention scores?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a taxonomy and analysis of multimodal chain-of-thought (MCoT) reasoning, categorizing thoughts into two main types: textual and visual, with four subtypes—natural language, structured language, image editing, and image generation thoughts. The study emphasizes the distinct roles of textual vs visual thoughts, supported by quantitative experiments on four datasets. The analysis includes computing correlation, flow, and attention scores to understand the impact of each thought type. Key findings include visual thoughts are crucial for enhancing MCoT reasoning, with their effectiveness depending on conciseness and clarity. Also, input image information is propagated to deeper transformer layers primarily through visual thoughts.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n* Interesting topic with a clear categorization of multimodal CoT.\n* Insightful analysis with good visualizations.\n* Multiple perspectives and interesting findings.\n\nWeaknesses:\n* Some analyses could go deeper, would benefit from more extended results.\n\nQUESTIONS:\n1. The finding in Section 5.1 is interesting. I’m curious whether this phenomenon only appears in deep models. Furthermore, is visual thought more beneficial for deeper models? Could you test this with a quantitative experiment?\n\n2. The differences between visual and textual thoughts in Flow and Disturbation might be related to the LLAVA training pattern. If you have access to models with different training patterns, it would be interesting to see whether these phenomena still hold.\n\n3. In Figure 6, manually annotating relevance may limit the data scale. While human annotation is valuable and should be kept, you might also consider using auto methods (e.g., CLIP) to generate scores.\n\n4. Beyond image similarity, I’d like to better understand the role of visual elements through representation analysis—e.g., by examining similarity between visual thought key/value pairs in attention.\n\nLIMITATIONS:\nno potential negative societal impact", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the concept of visual thoughts as a unifying abstraction for intermediate reasoning steps in multimodal chain-of-thought (MCoT) systems. It categorizes four modalities of visual thought expressions (natural language, structured language, edited image, generative image) and empirically compares their effectiveness across a variety of benchmarks and large vision-language models (LVLMs). The authors further analyze attention patterns and information flow to argue that visual thoughts function as a “cache” that facilitates deeper reasoning.\n\nThis paper focuses on categorizing the idea of visual thoughts, and analysis of it, without clear technical contributions of new methods or datasets.  To improve the paper, show that explicitly designing models or prompts informed by the “visual thoughts” concept yields improvements over existing approaches.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n* Provides a clear, structured categorization of intermediate visual reasoning mechanisms.\n* Attention and information-flow studies provide some internal evidence supporting the “cache” hypothesis.\n* Presents thorough experiments across several tasks and models, supported by ablation and diagnostic analyses.\n\n# Weaknesses\n* The core contribution — the concept of visual thoughts — is more an interpretive narrative than a substantive technical advance.\n* The work primarily re-labels existing techniques (scene graphs, image editing, image generation, captions) rather than proposing new methods or benchmarks.\n\n# Suggestions\n* Benchmark directly against the most related prior works (e.g., Visual Sketchpad [15], Image-of-Thought [55], Scene Graph CoT) under comparable conditions.\n* Add an experiment showing that explicitly designing prompts or models informed by the “visual thoughts” perspective achieves measurable improvements over existing baselines.\n* Report stronger statistical analysis where possible: include confidence intervals, p-values, or more runs to strengthen claims about significance.\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}