Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought
Authors: Zihui Cheng
Qiguang Chen
Xiao Xu
Jiaqi WANG
Weiyun Wang
Hao Fei
Yidong Wang
Alex Jinpeng Wang
Zhi Chen
Wanxiang Che
Libo Qin
Venue: NeurIPS 2025 poster ()
Keywords: Large Vision-Language Models
Multimodal Chain-of-Thought
Primary area: deep_learning

Abstract: Large Vision-Language Models (LVLMs) have achieved significant success in multimodal tasks, with multimodal chain-of-thought (MCoT) further enhancing performance and interpretability. Recent MCoT methods fall into two categories: (i) Textual-MCoT (T-MCoT), which takes multimodal input and produces textual output; and (ii) Interleaved-MCoT (I-MCoT), which generates interleaved image-text outputs. Despite advances in both approaches, the mechanisms driving these improvements are not fully understood.  To fill this gap, we first reveal that MCoT boosts LVLMs by incorporating $\textit{visual thoughts}$, which convey image information to the reasoning process regardless of the MCoT format, depending only on clarity and conciseness of expression. Furthermore, to explore visual thoughts systematically, we define four distinct forms of visual thought expressions and analyze them comprehensively. Our findings demonstrate that these forms differ in clarity and conciseness, yielding varying levels of MCoT improvement. Additionally, we explore the internal nature of visual thoughts, finding that visual thoughts serve as intermediaries between the input image and reasoning to deeper transformer layers,  enabling more advanced visual information transmission. We hope that the visual thoughts can inspire further breakthroughs for future MCoT research.

Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought
Zihui Cheng 1.4+ Qiguang Chen?* Xiao Xu²
Jiaqi Wang" Weiyun Wang
Hao Fei? 13,41
Yidong Wang
Alex Jinpeng Wang' Zhi Chen Wanxiang Che² Libo Qin
School of Computer Science and Engineering, Central South University
Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology
3 Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen
4 Text Computing and Cognitive Intelligence Ministry of Education Engineering Research Center,
Guizhou University 5 Chinese University Hong Kong
SSannhaii Al Laboratory ByteeDacce Seed (China) t. edu... cn
National University of Singapore Peking University fxh_upêccee edu. cn, qgchen i hí edu. cn,
Abstract
Large Vision-Language Models (LVLMs) have achieved significant success in multimodal tasks, with multimodal chain-of-thought (MCoT) further enhancing performance and interpretability. Recent MCoT methods fall into two categories: (i) Textual-MCoT (T-MCoT), which takes multimodal input and produces textual output; and (ii) MorrleaveedMooof (I-MCoT), which generates interleaved image- text outputs. Despite advances both approaches, the mechanisms driving these improvements are not fully understood. To fill this gap, we first reveal that MCoT boosts LVLMS by incorporating visual thoughts, which convey image information to the reasoning process regardless of the MCoT format, depending only on clarity and conciseness expression. Furthermore, to explore visual thoughts system- atically, we define four distinct forms of visual thought expressions and analyze them comprehensively. Our findings demonstrate that these forms differ clarity and conciseness, yielding varying levels of MCoT improvement. Additionally, we explore the internal nature of visual thoughts, finding that visual thoughts serve as intermediaries between the input image and reasoning to deeper transformer layers, enabling more advanced visual information transmission. We hope that the visual thoughts can inspire further breakthroughs for future MCoT research.
Introduction
Recently, Large Vision-Language Models (LVLMs) have made remarkable advancements in tackling various multimodal tasks [57, 22, 52]. Inspired by the success of Chain-of- Seain-of-Thouggeee Thought (CoT) reason- ing [45, 17, 31, LVLMs have integrated Multi-modal CoT (MCoT) reasoning, enabling step-by-step generation of reasoning paths in multimodal contexts. This advancement has enhanced their reasoning abilities, facilitating more sophisticated interactions with multimodal inputs [54, 4]. Specifically, current MCoT techniques are generally divided into two paradigms: (1) Tox (T-MCoT): This approach adheres to the traditional CoT framework, generating text-based rationales from multi- modal inputs (see Figure (a)). For example, some methods require LVLMSs to interpret and describe visual elements before producing an answer [55, 51J, while others enhance reasoning by integrating json-format scene graphs derived from images [30, 31]. (2) Interleaved-MCot (1-MCoT): A more recent approach, such as 03-mini [33] and Visual Sketchpad [15], generates interleaved image-text
"Equal Contribution
'Correponndigg Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Questioa: many breskpoitss does function
Question: breukpoints following function have?
fits
fuin
LVLMs
WS-MMat 18.4
LVLMS
Image Processing
MCOT Input
Inget
Textual-Only Rationale
Interleaved Rationale
Visual Thought: When 14.29-cx 898, ((0 - --23.26x 30885, Whey 8.98 < 67...
According to the function expression, we can draw flanction
image as follows:
Visual Thought: Let's
Visual Thought: Let's
Visual Thought: Let's
Based on
expression, we can the following: When -14.29 XX -898,
generate image for
genernte image for Eq.
generate image for Eq
decressing linear
function. As incresses, f(t continoously decreases. When -9.98 x 6.?. this an increasing linear function, As increases, (it) also increases...
Conclusion: Therefore, function has two breakpolnts.
Conclusion: Therefore, the function has total of two breakpoints
(a) Meo (TMCoTT)
Interlavvd-MMoo (I-MCoT)
Figure 1: Comparison between (a) Textual MCoT (T-MCoT) with purely textual rationale, and (b) Interleaved MCoT (I-MCoT) with the image-text interleaved rationale. VT: visual thoughts.
rationales (see Figure 1 (b)). This method employs external tools, like code interpreters or specialized visual models, modify images for reasoning [28, 15, 56], while others use image generation models to create new images, enhancing the reasoning process [27, 19].
However, the debate between these two paradigms remains unresolved. Some researchers contend that the interleaved rationale in I-MCoT better reflects human cogmitive processing of multimodal inputs, potentially offering advantages over T-MCoT [15, 19]. In contrast, other studies suggest that in mathematical contexts, text-only rationales may yield superior performance [56, 7]. This disagreement highlights fundamental gap in understanding the mechanisms behind different MCoT approaches. In addition, in the literature, there is a lack of a unified framework to explain MCoT's s effectiveness, identify optimal MCoT paradigms, or derive generalizable insights across tasks. Motivated by this, in this work, we aim to investigate the following research question: Is there a unified explanation for that different MCoT paradigms enhance LVLMs in distinct ways?
To this end, we reveal that MCoT unificdly improves LVLMs by integrating visual thoughts [18] into the rationale generation. Visual thoughts intermediate, logic-driven cross-modal representations that facilitate and accelerate multimodal reasoning within a unified perspective. By caching distilled visual information, they bridge raw pixels and linguistic rationales, enabling fast, context-aware access without reprocessing the image. Conceptually, as shown in Figure 2, image features and visual thoughts function like external memory and cache: raw images require slow, resource- intenssvve retrieval, while visual thoughts retain key content for rapid access, improving reasoning efficiency and reducing computational cost. Moreover, the effectiveness and efficiency of visual thought transmission can further explain performance differences across MCoT paradigms.
To verify this, during experiments, we first verify the effectiveness of visual thoughts in improving MCoT performance. Then, to extensively explore how visual thoughts work in different expressions, categorize four major strategies: Natural Language, Structured Language, Edited Image, and Generative Image visual thoughts. Moreover, we also investigate the role of internal attention mechanisms and information flow within LVLMs to analyze the rationale behind visual thoughts. Our findings reveal the following: (1) Removing visual thoughts and forcing reasoning solely from the original image can impair performance, even more than reasoning directly from the query. (2) Different expressions of visual thoughts are more effective in certain scenarios, depending on their expression clarity and efficiency. (3) Visual thoughts not only carry visual information but also serve as primary intermediaries, connecting the imput image to deeper transformer layers and enabling more advanced cognitive processing in LVLMs.
Our main contributions can be summarized as follows:
We present the first comprehensive exploration of the underlying mechanisms driving visual thoughts in MCoT reasoning. This unified perspective provides novel insights into how visual cognition unfolds during drcision-making processes.
We introduce four distinct strategies for systematically exploring visual thoughts, demonstrating the advantages and disadvantages of different strategies.
Our analysis further delves deeply into the nature of visual thoughts, revealing critical insights, such the way in which visual information is integrated into the reasoning path, which enables

---

(External
How manty apples and hov?
How many apples are hend?
Raw Image (External Storage)
Visual Thought (Cache)
Derivative Steps
@
Derivative Steps
There thrre apples
Editing Tool
Counting Applex
There are apples the
Visual Thought (Cache)
2 There are three apples on the right.
Derivative Steps
Derixative Steps
NO Visual Information CACHE HERE!!
Counting Apples the Right
There are thrre apples on the rigitt
Answer: Six apples
Answer: Five apples
Visual Information Flow
Visual Information
Multimodal Reasoning without visual thoughts
(h) Multimodal Reasoning with visual thoughts
Comparison of multimodal reasoning from compuler-system perspective: (a) visual thoughts as an internal visual cache versus (b) direct access to raw images as external storage.
Figure 2:
conveying more visual information into the deeper transformer layers of LVLMs. We hope these findings will inspire more breakthroughs for future research in this area.
2 Visual Thoughts
2.1 Definition Visual Thoughts
To explain how MCoT improves performance, contend that its primary benefit the incorporation of visual thoughts, intermediate reasoning steps that explicitly convey visual information and enable LVLMs to perform deeper visual reasoning. As shown in Figure 2, LVLMs treat the image as external memory, forcing the model to iteratively reprocess the entire visual input at each step so as to cap the depth of reasoning. In contrast, visual thoughts extract only the ietrruction-elevaann regions (e.g., the left and right apples) and store them as a cache. Subsequent reasoning steps query this cache rather than the full image, reducing computational overhead and enabling deeper, multi-step MCoT.
Formally, a visual thought (Rev) a reasoning step that conveys information from the visual input Vi and all previous steps R (R1, R2...., 1. These steps driven by the task question Qr and by explicit instructions Ig requesting the MCoT expression Eyr. The model then generates the next reasoning step R; as follows:
arggax # (Rei, V1 - REW Qr, Dr,I Re)) if n
R; arggaax (Reg, REv RDV, Qr Iem Rei), if < n,
Rew
RD
where RD represents the derivative reasoning steps [58] following R, and grasp visual information from R The function (-) denotes the probability of generating visual thoughts, and iT-) rep- resents the probability of generating derivative reasoning steps. The symbol - y indicates the reasoning step y with the flow of reasoning information from x to y.
2.2 Categories of Visual Thoughts
Visual thoughts can be expressed in different modalities depending on the MCoT variant: (1) In T-MCoT, which generates text-based rationales, they appear as textual expressions; (2) In I-MCoT, which produces cross-moda rationales. they manifest as visual expressions.
2.2.1 Textual Multimodal Chain-of Thought (T-MCoT)
We first define visual thoughts within T-MCoT, where the model generates visual thoughts as textual tokens. As shown in Figure (a), the traditional T-MCoT generates text-only output from multimodal inputs, representing visual thoughts as Evr = Exea- Formally, the visual thought can be expressed as:
R2 argmax Tren (Rei, V, - &un VV, Q, DEme Rei), Ree
where Tee (-) denotes the probability of generating the rationale Rew from the textual tokens. Expression 1: Natural Language (N-LANG) facilitates effective visual information transfer by natural language expression, such as describing images based on question, enhancing vision- lision-language

---

Instrction (lemn)
Visual Thoughts (Ri
Instenctin
Visual Thoughts (
Please
naplier ampph Ein...
Queetion (8x)
Whs'' the cake hai!
LVUMC (Frse)
M-LANG
sara Ilt santin in
liaboay
Was''s la: saae has?
UUUN: (naw--)
E ING
Tosk
bus
Inputt (92)
Visual Input (9))
SLANN
G-ING
Model
Prestous Ressoning
Prestoua Beasoning
Let's ccnadi the bea.
Information
Information
Vistal Thoughs with Vertual Expression
(b) Visual Thought with Visual Expresioon
Figure 3: Visual Thoughts in textual expression (a) and visual expression (b). Specifically, the textual expression includes N-LANG and S-LANG, while the visual expression includes E-IMG and G-IMG.
aligmment in LVLMs through richer visual descriptions [2. 12. 26], as shown in Figure (a). Its reasoning process can be formally defined as:
R R.
argmaa Trear (Rei V1 - R N-LANG V1, O, I.IN-LANG, Reg).
(3)
To implement N-LANG, we prompt LVLMs to generate image captions as a precursor to reasoning. Expression 2: Structured Language (S-LANG) [5] has demonstrated superior performance over traditional MCoT reasoning on math, by effectively incorporating structures language into reasoning pipelines [38, 13, 16], as shown in Figure 3 (a), which can be formally expressed as:
R₂- = argmax Trear (Rei, 11 - RS-LANG [V, Is-LANG R.i).
R.S-LAG
To investigate S-LANG visual thoughts as a medium for expressing visual thoughts, we implement by prompting LVLMs to generate scene graph from input queries, which is then used for reasoning.
2.2.2 Interleaved Multimodal Chain-of Thought (I-MCoT)
We then introduce the MCoT through visual expressions, suggesting that image tokens are integral to visual thoughts. As depicted in Figure (b), the I-MCoT framework expands on the traditional T-MCoT by integrating image editing and generation into the reasoning process, thus enabling visual thoughts be conveyed through images. This can be mathematically represented as:
R2- argmax IT Timage (Rea, V/ - Rime [V,, Q I. R2d)),
(5)
where (.mage()) (-) denotes the probability of the model incorporating an Image-basede rationale step.
Expression 3: Edited Image (E-IMG) processes the original image and performs various visual operations, such grounding [23], depth estimation [48], and segmentation [35]. By conveying image tokens, E-IMG enhances the ability of LVLMs to interpret visual data, thereby improving reasoning capabilities, shown in Figure 3 (b), defined as:
R. = argmax Timage (Rei, V, R E-IMG 1E-,MG,, Q, Red),
(6)
R R-E-M
To explore the E-IMG visual thought, we provide LVLMs with edited images using vision tools. enabling them to incorporate the edited results into subsequent reasoning.
Expression 4: Generative Image (G-IMG) are required to prompt generative models to generate logical-related image based on the advancement of LVLMs (1, 9, 43, 8), shown in Figure (b), which is defined as:
R2 argmax Timage (Rei, V,
G-IG M,, I. Re))
IMG
To explore G-IMG, we use DALL-E 3 [1] as a tool for visual thought to generate novel images based on input queries, which are then employed supplementary inputs to assist reasoning.
3 Effectiveness Verification of Visual Thought
Integrating visual thoughts is essential for MCoT's effectiveness in both image and text expression. To validate this, as depicted in Figure (a), we compare system performance under three conditions:
(I) "Image-form visual thoughts", the original I-MCoT with interleaved image-and-ter text rationales;

---

Image frems Visnal Thoughtt
Query Onaly
Quanyytelly
What's the color ofe bux?
will desect the bus with touh.
Siral Visual Thought foom Visual Thought
Acconding to tse sraonited image. Bitk...
The hos tmage to white and blue
Roylar the Deags Link de Parviallr
Whst's
the
Whae''s color oft hoe?
Thought
deset bus with too..
Caption Only
CoNTT Selection
CoNT Update
Mach-limakpsintt
Accooding the sencersd imags, thrk...
(b) Effectiveness verification on GPT-4o.
w/l Visaal Though Imagg-Sorm Viuual Thonghi
Captins-Oull
The ben image while.
Caption: The
Text form Visual Thought
Keplace de Placoliiie
Lash iouge Captior Thntggt
bus with shack desig....
shows pesserges
What'ethe oolerof hudl
will dktect with the tooks Caption: The image bosing lang
What's the color of ds: Ius?
Sei Conteen ao
which
You moniional that the bas hsca
bas imsgg while blac.
Hard to Descrite
Mediums to
Baayst--Doarrtee Image
Image
Descritte Image
(a) Workflow of effectiveness verification.
(c) Visual Thoughts v.s. Caption-Only.
Figure 4: Effectiveness Verification for Visual Thoughts. More details are in Appendix B.
(2) "w/o visual thoughts", where the visual thought cache is cleared, forcing the model to reanalyze the input image; (3) "Text-form visual thoughts", where the cache is restored with descriptions of the images from I-MCOT. As illustrated in Figure 4 (b), the results reveal a consistent trend: omitting visual thoughts leads to decrease in accuracy (even worse than reasoning only from the query), while including them consistently improves reasoning performance. These findings emphasize the essential role of visual thoughts in conveying visual information and enhancing model accuracy.
Image-form visual thoughts consistently surpass text-form visual thoughts across different com- plexities. Owing to the inherent strength of the image modality in conveying visual information, image-form visual thought can facilitate cache-like visual logic propagation more effectively than text-form visual thought, thereby triggering the subsequent reasoning process with higher efficiency. As shown in Figure 4 (b), the results reveal that image-form visual thought consistently outperforms text-form visual thought across scenarios of varying complexity, with especially higher improvement pronounced in hard scenarios (47.83% on CoMT-Selection). This demonstrates that the image-form visual thought offers superior channel for conveying detailed visual information.
Visual thoughts represent a reasoning process that conveys visual information beyond simple image captions. Unlike vanilla captions, visual thoughts provide a more cache-like function that integrates detailed visual cues dynamically during reasoning. They evolve through sequential reasoning steps, encompassing visual information and contextual logic to support downstream tasks. As shown in Figure 4 (c), simple scenes, models with visual thoughts perform comparably caption-only baselines. However, as scene complexity increases, caption-only models lose accuracy, falling to the "w/o visual thoughts"' Further, when brief captions omit essential details, visual thoughts improve performance by over 7%, showing their strength in visual information conveyance for MCoT.
4 Exploration for Different Categories of Visual Thoughts
Building on the effectiveness of visual thoughts observed during preliminary verification, we will further provide a comprehensive evaluation of the four classic expressions of visual thoughts?.
4.1 Does these different Visual Thoughts all works?
MMVP V'Rench
MCCT O CoMT
Explicitly incorporating visual thoughts with different expressions can all enhance the performance of almost all LVLMs. As shown in the results in Table 1, compared to the w/o VT expression without extra instructions, four strategies incorporating visual thoughts in the rationale achieve performance improvements across different tasks across almost all LVLMs, demonstrating the effectiveness of visual thoughts in enhancing the performance of MCoT,
Shareoo Performence Improveemnnt
80%
60%
40%
20%
0%
N-LANG
S-LANG
E-IG
G-IMG
Figure The proportion of performance improvement rate across tasks.
Moore details can be seen in Appendix C.
5

---

Model
MMVP
V-Bench
McCoT
CoNT
plysical
temporal
deletion
update
LLaVA-I.5:79 [21]]
w/o
29.57
59.
26.83
16.00
23.50
N-LARG
$2.33
52.63
34.78
45.57
60.33
32.52
21.30
17.50
29.00
38.58
5 LAMS
51.33
$2.63
35.65
51.11
61.57
31.21
22.00
20.50
20.00
39.50
B-ING
49.33
50,00
36,52
48.59
64.05
34,15
25.
23,00
29.50
10.10
G-ING
49.07
48.08
34.78
55.56
63.22
39,02
29.50
25.00
35.00
42.27
Qwem2: VL-70 [42]
w/c VT
70,00
55.26
65,70
80.00
75.21
74,80
26.00
18.00
57.00
56.11
83:33
28.00
20.00
71.00
G-IMG
65.00
39.21
51.30
84.44
80.17
82.93
29.50
25.50
44.30
$8.06
GPT GPT-te (o-mm [32)
w/o VT
72.67
44.74
56.52
78.80
70.12
80.49
10.00
19.50
24.00
48.55
N-LAME
75.33
52.17
61.84
84.44
79.34
81.30
27.30
20.00
27.00
56.55
S-LAMG
74.33
52.63
54.55
84.44
74.65
81.30
26.00
20.10
33.00
55.06
R-1M5
73.58
52.63
70.18
84.44
76.85
83, 74
29.00
21.50
33.00
58,33
G-ING
72.67
30,00
53.04
86.67
78,93
87.80
30.00
20,00
40.50
57.73
[32]
Wio VI
79:67
LANI
S-LAMG
84.33
63.16
64.35
90.00
78.51
82.93
29. 19.50
18.00
42.00
61.42
2-ING
83.00
99.21
65:22
90.00
78.10
86.18
34.00
28.50
50.00
63,80
G- TM4
78.00
59.21
50.13
92.22
78.03
86.18
53.50
28.50
46.50
62.46
Table 1: Main results various LVLMs. The bold content indicates the best performance within each LVLM. w/o VT refers to prompting LVLMs without additional visual thoughts.
Tasks requiring more complex visual operations can benefit more from visual thoughts. Further analysis, as shown Figure 5, reveals that visual thoughts yield the most substantial performance improvements in CoMT. This benchmark primarily focuses complex multimodal operations such as visual deletion, selection, and update during reasoning, rather than on other perception tasks. These findings suggest that visual thoughts can significantly enhance the reasoning capabilities of LVLMs in complex scenarios.
T-MCoT demonstrates superior reasoning performance on coarse-grained perception tasks, whereas I-MCoT excels in scenarios that require fine-grained visual operations. Furthermore, we examine the efficiency of visual thought transmission in T-MCoT and I-MCoT across different sce- narios. As shown in Table 1, T-MCoT outperforms in coarse- guanseerained perception tasks (e.g.. MMVP, V*Bench- V=Bench-position)) while I-MCoT enables more efficient transmission in fine- mne-grained tasks (c.g.. V*Bench-attribee attributes) and in tasks demanding visual operations (e.g., M CoMT). Consequently, should adapt different categories of MCoT for different features of tasks.
Visual thoughts with visual expression cost more in ratio- nale than textual expression. To compare the reasoning costs of four visual thought expressions, we calculate the average number of text tokens in responses, as well average number of generated image tokens for each expression. As shown in Table 2, N-LANG and S-LANG have slightly higher average text token counts, particu- larly S-LANG, which requires extensive structured snip- pets. However, for E-IMG and G- G-IMG, the higher average number of image tokens, due to the inclusion of more images, substantially increases the reasoning burden on LVLMs, both in terms of time and expense, indicating higher reasoning costs compared to text expressions, which highly limits the reasoning efficiency.
# Text Token
Image Token
N LANG
139.02
8 --NNNGG
364.37
B-IMG
91.65
1,112.51
IMG 89.18
393.00
Table 2: Reasoning costs of different Visual Thoughts across all LVLMs and datasets. #X: the average number of X.
4.2 Which scenarios does different Visual Thoughts work better?
strong performance on coarse-grained, perception-oriented reasoning tasks. natural language turn visual input into high-level semantic cues and efficiently extract macro features. As shown in Table 1, the MMVP benchmark. where identifying salient object matters, N- LANG first spots elements like "a butterfly", guiding subsequent analysis and achieving top results in coarse-grained perception.
When rapid overview of an entire scene is needed, N-LANG
S-LANG excels in reasoning about object relationships. S-LANG converts input images into detailed scene graphs, enabling precise modeling of both spatial and semantic relationships. As shown in Table for the V=Bench-position benchmark, S L-LANG not only identifies entities such as

---

N-LANG LANG A E-ING I
N-LANG S A Amm Be
N.LAAN ACLANC A FIMG GGIMG
Image Relevnnce
3.50
Expression Clarity
3,50
Coneire Expression
3.50
LDD
2,50
2.50
2.50
AD
1.50
7,50
1.50
0.50
0.50
0.2
0.4
0.6
0.8
0.2
0.4
0.5
0.8
0.2
Accurncy
Ascuracy
Accurncy
(P.0.4441 P>0.4; twil.113 P50.5
[Pm0.8273 Pcll:0;; Tm0.8525 Pic001
p=0,7229 P-00:01; r=0,8500 P=0.01
(6) The porrelation between
Input Image
(b) The correlation between the clarity of visual
(c) The correlation between the conciseness of visual thought expression and sucuracy.
relevannce of visuall thought and sccurac..
thought expression and its accuracy.
Figure 6: Analysis of the correlation between accuracy and visual thought quality. p: Spearman's correlation coefficient; r: Pearson's 's correlation coefficient; P: p-value of related assumptions,
"table" and "chair" but also accurately infers their relative positions, which underlies its state-of-the-art performance in relational reasoning tasks.
E-IMG achieves strong results in detailed image analysis. Emulating human editing workflows, E- IMG refines visual content to aid fine grainer feature detection. As shown in Table in the B#Bench-attribtees it magnifies and annotates areas of interest, improving the accuracy of attribute predictions. This detailed focus yields the highest average performance across models.
G-IMGis well-suited for multi-step reasoning through iterative image generation. G-IMG generates images dynamically to refine and test reasoning hypotheses, enabling adaptive visual thinking. As shown in Table I, in the McCoT, requiring multiple interaction rounds, integrates logical steps with new visuals to deepen understanding of complex concepts. This flexible generative pipeline excels in long-term, multi-turn reasoning scenarios.
4.3 What are the core factors that influence different Visual Thoughts effectiveness?
Visual thoughts function as a distilled visual information cache, not a faithful replica of the original image. A key question is whether visual thoughts accurately preserve all the content of the original image, acting as cache instead of external storage. To explore this, we manually assess the fidelity of visual thoughts shown in Figure 6 (a), both Spearman's 's (p) and Pearson's (r) correlation coefficients are below 0.15 (P > 0.4), indicating no significant relationship. This suggests that visual thoughts function more as a condensed cache of visual information rather than a direct substitute for the original image.
0-3 scale and examine their correlation with model accuracy. As
Visual thoughts provide a clearer, more interpretable expression of visual logic, enabling more effective retrieval of reasoning-relevan information. Unlike direct image representations, visual thoughts do not aim to reconstruct the original image, but instead distill and communicate cross-modal visual logic like a computer cache. To assess whether the effectiveness of visual thoughts depends on how well this visual logic is expressed, we manually score the clarity of logical representation on 0-3 scale. We then examine the correlation between these scores and model accuracy. As shown in Figure 6 (b), both Spearman and Pearson correlation coefficients exceeded 0.8 (P < 0.01), indicating a strong positive association. These findings suggest that the clearer the visual logical representation, the more effectively the model can reason using information stored in the visual thought cache.
Concise visual logic expression further enhances the effectiveness of visual reasoning. Beyond clar- ity. we further explore what other factors affect the effectiveness of visual thoughts. We hypothesize that more compact visual expressions, those stripped of redundant or extraneous elements- enable faster and more accurate retrieval from the internal visual cache during inference. To test this, manually assign a conciseness score to each visual expression, reflect- ing the efffccincy with which it conveys the visual information and logic required for multimodal reasoning. We then examine how these scores relate to model performance. As shown in Figure 6 (c), reasoning accuracy correlated strongly with concise- ness, which indicates that the effectiveness of visual thoughts is not solely determined by clarity, but is also enhanced by the compactness of the underlying visual logic.
Aucutay
Ancerac
021/115
8351
Dau Tarnbald Tee Throluld
Galdance Stale
60
G-GG
Figure 7: Factors Correleation of I-MCoT.
Visual thoughts affected by external noise can negatively impact the reasoning performance of MCoT. In I-MCoT, since the visual thoughts (i.e., edited images and generative images) are typically
7

---

Oiginal Imsgo Attantion No-VS
Percentages (%g
in lattrr 50% leyers acuss all brgers
60 75
Original Image
0.6
N-LANG
S-LANG
G-IMG
Antenton
0.4
Poroemtage (%0
0.2
0.0
13
22
Visual Thought Attention
0.5
N-LANG
S-AANG
Dilleese
Atsetion
F-MG
G G-M
0.4
0.2
LLaVA-1.5- 1.5-
LLaVN-15-
Qwen2.5.
Qven25 VL-7B
138
0.0
More results the percentage of the layers which the atiention to visual thoughts exceeds that to the inpui original image, and the difference attention score to visual thoughts relative to the original image across all layers.
28
Layers
The attention distribution with the model.
Figure 8: Attention distribution of Visual Thought and Image in MCoT.
produced by external tools, it is worth investigating whether the noise introduced by these tools affects MCoT's reasoning process. Specifically, we employ Grounding DINO and Stable Diffusion v1.5 to generate the edited and generative images, respectively. external tools, we adjust their parameters to manipulate the quality of visual thoughts and evaluate the noise levels using MLLM evaluation and CLIPScore metrics. As shown in Figure 7, both E-IMG and G-IMG exhibit clear negative correlation between accuracy and noise level. When the injected noise becomes excessive, the model's reasoning performance even fall below that of the native reasoning (Direct). This suggests that managing external noise within visual thoughts is critical factor for maintaining effective reasoning.
control the noise introduced by
5
Internal Rationale Behind the Visual Thought
To comprehend the rationale visual thoughts, we examine how visual information is conveyed through attention mechanisms and information flow analysis in LVLM's internal structures."
5.1 Visual Attention Analysis
During the reasoning process, the attention of the original image will be seattered into visual thought. To investigate the internal mechanism of visual thoughts within LVLMs, we first analyze the model's 's attention distribution shifts of visual thoughts using LLaVA. As illustrated in Figure 8 (a), in the reasoning process without visual thought (No-VIS), the model exhibits high attention to the image, whereas in the reasoning process with visual thought, the attention to the raw input image significantly decreases across all visual thought expressions. Furthermore, we can observe that the model shifts its attention from the raw input image to various visual thoughts. This redistribution enables the flow visual information, which supports the entire logical framework.
Visual thoughts can convey visual information more deeply than the original image itself. More interestingly, as shown in Figure 8 (a), we observe striking phenomenon in deeper layers of the model: attention to the original image without the incorporation of visual thoughts diminishes sharply after the 12th layer, nearly reaching zero. contrast, when visual thoughts are included, attention to all expressions exhibits marked increase, surpassing the 12th layer and even maintaining levels comparable to those seen in earlier layers. This finding suggests that visual thoughts play a crucial role in transferring visual information into the deeper layers of the model, thereby promoting enhanced cross-modal interaction and supporting more sophisticated logical reasoning.
Model architecture has a greater impact on visual thoughts transmission than the parameter scale. Based on the interesting observation in Figure 8 (a), we further investigate the internal attention distribution of visual thoughts. To this end, we select the 7B(32 layers) and 13B(40 layers) model
More details can be seen in Appendix D.
8

---

No-VIS
lenn 20
Tranddormar Layer
Transforme
Layer?
Transformet Layer ...
Transformet Layer 18
Transformes Layer 19
Trannformas Layer
Trandorn
Layers 32
Answer:
H
Query: Deternine
Thought
functionns breakpoint number.
Answer:
Cholces: (A) (B) (CC)
Remove Visual Thought Information Flow in Layer
Answer:
Remove Visual Thought
Answer:
Information Flow in Layer 19"
Figure The Disturbation Analysis of Information Flow within LLaVA model.
from LLaVA-1.5 and select 3B(36 layers) and 72828 layers) model from Qwen2.5-VL series to extract internal attention to the original input image and visual thoughts seperately. As shown in Figure 8 (b), for both the LLaVA and Qwen models, in the majority of layers, the attention toward visual thoughts exceeds that toward the original image, especially in the latter 50% of layers, suggesting that visual thoughts can convey visual information more effectively and deeply than the image itself. Furthermore, the results of the difference in attention scores reveal that both LLaVA (7B - 13B) and Qwen (7B - 3B) exhibit increasing attention gains with deeper model layers. It reveals that the influence of visual thoughts depends more on the model architecture (e.g., number of layers) than on the parameter scale. Additionally,
notice that in the n-lang and s-lang settings for LLaVA, the model shows relatively low attention to visual thoughts due to the self geff-generteed weaker language capabilities, resulting in captions and scene graphs that are overly simplistic and fail to accurately convey visual information.
5.2 Visual Information Flow Analysis
Input questions query visual information from the visual thought cache. Subsequently, we manually disturb the model's internal information flow to understand and quantify the cache-like impact of visual thoughts on MCoT. As shown in Figure 9, without visual thoughts (No- VIS), the model erroneously selects choice A, while incorporating visual thoughts allows it to correctly choose C. Additionally, intercepting the flow of information from the query to the visual thought cache before layer 19 will significantly prevent the model from selecting the correct answer, whereas disrupting the flow from the image has no effect on the inference. These findings suggest that querying visual information from the visual thought cache is the key mechanism enhancing the model's 'ss predictions.
The visual information retrieved from visual thought cache will be further passed to advanced reasoning processes across the boarder and deeper model layers. Furthermore, we examine the directional information flow distribution within the model, focusing on the flows between the the image and reasoning and the flow between visual thoughts and reasoning process at various layers. As shown in Figure 10, our findings indicate that the information flow from visual thoughts to reasoning stages is considerably stronger than the direct flow from the image to reasoning. This emphasizes the critical role visual thoughts play in mediating and organizing visual data, optimizing it for reasoning. By doing visual thoughts enable the model to leverage visual information more effectively, resulting in more accurate and coherent textual outputs.
The information of the original image primarily flows to the visual thoughts. As illustrated in Figure 10, rather than directly entering reasoning processes, most of the image's ' input information is initially passed to visual thoughts during the derivation steps, which then feed into the reasoning stage. This demonstrates that nearly all image-derived signals pass through visual thoughts before reaching deeper reasoning. This two-stage process (raw pixels visual thoughts - deeper reasoning) emphasizes the role of visual thoughts as crucial bridge, allowing LVLMs to generate better MCoT.
Related Work
The development of chain-of-thoughh (CoT) [45, 17, 34] has led to research exploring its extension to multimodal reasoning [25, 44, 14, 41, 31, 53). In this context, Zhang et al. [54] formally introduce the concept of multimodal chain-of-thougeee (MCoT), employing a two-stage framework that separates rationale generation and answer formulation. These studies fall under the category of Textual-MMoT (T-MCoT), where multimodal inputs are used, but the output is text-only. Zheng et al. [55], Yao et al. [49] improve multimodal interaction through step decoupling; and Wei et al. [46], Chen et al. [4] introduce multi-hop reasoning to capture more complex relationships. Furthermore, Chen et al. [6] extend T-MCoT for evaluation in commonsense reasoning tasks.

---

Query:
Select the best answer. Which property do these three abjects
have common?
Choices:
Image
A. Blue
B. Hard
C. sticky
N-Lang:
S-Lang:
E-ing:
G-Img:
Reasuning
blue pair shorts with while stripe down the side.
F'objects" i [[name"
"Toothpaste", 'artributes": ('color:: "blue" Jook
Proportion Value
Qwen D LLaVA
20
30
1e-2
bA
20 Layer Index
30
20
30
10
20
30
0
10
30
Layer
Layer Index
Layer
Figure 10: Information flow distribution of Visual Thought and Image in MCoT within LLaVA-1.5- 7B and Qwen2-VL-2BB
Due the limitations T-MCoT's purely text-based output, more literature has explored MCoT using the image-text interleaved output paradigm (I-MCoT) [27, 28]. Hu et al. [15], Zhou et al. [56]. Su al. [37] explore using vision expert models annotate input images for I-MCoT. Additionally, Cheng et al. [7] propose benchmark of four task types requiring multimodal output to assess LVLMs' I-MOTT reasoning capabilities. While most works rely on external executors vision expert models, Gao et al. [1]] extract image patches directly from raw inputs for interleaved images, and Li et al. [19] achieve I-MCoT reasoning by fine-tuning LVLMs with multimodal generation capabilities.
Despite the advancement in T-MCoT and I-MCoT, few studies analyze the underlying unified mechanisms behind these improvements by MCoT. We attribute these performance enhancement to visual thoughts, the reasoning steps that convey visual information from the input images. We further propose four strategies that represent different modalities and approaches to integrate visual thoughts and provide a comprehensive analysis on the four strategies respectively, hoping that the findings can provide practical and systematic guidance for future research on MCoT.
Discussion
Limitations To facilitate variable control and streamline analysis, we exclude multiple rounds of visual thought interactions, which should be explored in future complex scenarios. Additionally, due to the difficulty in generating I-MCoT during the Any-to-Any LVLM inference process, which often leads poor logic quality, instead use DALLE call for G-IMG.
Broader Impacts This work represents the first comprehensive investigation into the unified mechanisms underlying MCoT. We hope our work can serve as a valuable reference and guide future research into the mechanisms of MCoT. For social impact, this work may contribute to advancing the understanding of interpretability in multimodal AGI systems.
Conclusion
In this study, we first propose the concept of visual thoughts, which facilitate the transfer visual information from input images to the reasoning process and deeper transformer layers. Visual thoughts are crucial for the effectiveness of MCoT approaches. We also propose and evaluate four strategies for expressing visual thoughts, demonstrating their role in enhancing MCoT performance. Furthermore, we analyze the underlying rationale of visual thoughts to better understand their functioning. We aim for this work advance the understanding of the unified mechanism behind MCoT and inspire further innovations in future research.
10

---

Acknowledgements
This work was supported by the National Natural Science Foundation of China (NSFC) via grant 92570120 and 62306342. This work was supported by the Scientific Research Fund of Hunan Provincial Education Department (24B0001). This work was sponsored by the Excellent Young Scientists Fund in Hunan Province (2024JJ4070), the Science and Technology Innovation Program of Hunan Province under Grant 2024RC3024 and CCF-Zhipu Large Model Innovation Fund (NO. - Zhipu202406). This study was also funded by the Open Project of the Text Computing and Cognitive Intelligence Ministry of Education Engineering Research Center (No. TCC1250101).
References
[1] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https//ccdn.opendie
2(3):8, 2023.
[2] Yandong Bi, Huajie Jiang, Yongli Hu, Yanfeng Sun, and Baocai Yin. See and learn more: Dense caption-aware representation for visual question answering. IEEE Transactions on Circuits and Systems for Video Technology, 2023.
[3] Qiguang Chen, Libo Qin, Jiaqi Wang, Jinxuan Zhou, and Wanxiang Che. Unlocking the capabilities of thought: A reasoning boundary framework to quantify and optimize chain-of- thought. arXiv preprint 1rXiv:2410005695, 2024.
[4] Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M A novel benchmark for
multi-ster multi-modal chain-of- th-ouugh... In Lun- Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages $199-8221) Bangkok, Thailand, August 2024. Association for Computational Linguisties. doi: 10.18653/ v1/2024.acl-long.44. URL https : ac anthology, wwwgor///00 2024 2.000..cll-1
446.
[5] Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv: 125888 2022.
[6] Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. Measur- ing and improving chain-of-thought reasoning in vision-language models. arXiv preprint arXiv:2309.. 04061, 2023.
[7] Zihui Cheng, Qiguang Chen, Jin Zhang, Hao Fei, Xiaocheng Feng, Wanxiang Che, Min Li, and Libo Qin. Comt: A novel benchmark for chain of multi-modal thought on large vision-language models. arXiv preprint arXiv:2412.129%
2024.
[8] Ethan Chern, Jiadi Su, Yan Ma, and Pengfei Liu. Anole: An open, autoregressive, native large multimodal models for interleaved image-text generation. arXiv preprint 2rXiv:2407.06135 2024.
[9] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- ers for high-resolution image synthesis. In Forty-first International Conference Machine Learning, 2024.
[10] Deqing Fu, Ruohao Guo, Ghazal Khalighinejad, Ollie Liu, Bhuwan Dhingra, Dani Yogatama, Robin Jia, and Willie Neiswanger. IsoBench: Benchmarking multimodal foundation models on isomorphic representations. In First Conference on Language Modeling (COLM), 2024. First four authors contributed equally.
[11] Jun Gao, Yongqi Li, Ziqiang Cao, and Wenjie Li. Interleaved-modal chain-of-thought. arXiv preprint arXiv:241 1. 19488, 2024.
[12] Deepanway Ghosal, Navonil Majumder, Roy Ka-Wei Lee, Rada Mihalcea, and Soujanya Poria. Language guided visual question answering: Elevate your multimodal language model using knowledge-enricher prompts. arXiv preprint arXiv:2310.20119, 2023.
[13] Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 14953-14963 2023.
11

---

[14] Liqi He, Zuchao Li, Xiantao Cai, and Ping Wang. Multi-modal latent space learning for chain- of-thought reasoning in language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 181 80-18187, 2024.
[15] Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. Visual sketchpad: Sketching as a visual chain of thought for multimodal language models. arXiv preprint arXiv:2406. 09403, 2024.
[16] Yushi Hu, Otilia Stretcu, Chun-TTa Ta Lu, Krishnamurthy Viswanathan, Kenji Hata, Enming Luo, Ranjay Krishna, and Ariel Fuxman. Visual program distillation: Distilling tools and program- matic reasoning into vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9590-9601, 2024.
[17] Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199-22211 2022.
[18] Robert G Kunzendorf, Kara Young, Tamara Beecy and Karen Beals. Is visual thinking "imageless thought"? Perceptual and Motor Skills, 91(3):981-955 2000.
[19] Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vulić, and Furu Wei. Imagine while reasoning in space: Multimodal visualization cotiirattioooo---ooee thouhht arXiv preprint arXiv:2501.07542, 2025.
[20] Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianwei Yang, Lei Zhang, and Jianfeng Gao. Segment and recognize anything at any granularity. In European Conference on Computer Vision, pages 467-484. Springer, 2024.
[21] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
[22]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in newral information processing systems, 36, 2024.
[23] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Qing Jiang. Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, 2024. URL https: /ttp:///aavvvoooooeeoo 05499.
[24] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang. Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, pages 38-55. Springer, 2024.
[25] Pan Lu. Swaroop Mishra, Tony Xia. Liang Qiu, Kai-Wei Chang, Song- Chug Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Process- ing Systems (NeurIPS), 2022.
[26] Ziyu Ma. Shutao Li, Bin Sun, Jianfei Cai, Zuxiang Long, and Fuyan Ma. Gerea: Question- aware prompt captions for knowledge-based visual question answering. arXiv preprint arXiv:2402.02503, 2024.
[27] Fanxu Meng, Haotong Yang, Yiding Wang, and Muhan Zhang. Chain of images for intuitively reasoning. arXiv preprint arXiv:2311.088 1.09241, 2023.
[28] Sachit Menon, Richard Zemel, and Carl Vondrick. Whiteboard-dl of tf-thought: Thinking step-by- step across modalities. arXiv preprint arXiv:2406. 14562, 2024.
[29] Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019.
[30] Chancharik Mitra, Brandon Huang. Trevor Darrell, and Roei Herzig. Compositional chain-of- thought prompting for large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14420-14431. 2024.
[31] Debjyoti Mondal, Suraj Modi, Subhadarshi Panda, Rituraj Singh, and Godawari Sudhakar Rao. Kam-cot: Knowledge augmented multimodal chain-of-thie thhoogghss reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18798-18806, 2024.
[32] Openai. Openai got- 2024. URL https: / oppeaai com/ www.i//wweeeeeeee gpt- 3pt-40/.
12

---

[33] Openai. Openai o3-mini system card, 2025. URL https::/openai.com///
mini- miin--ssstem-card/
[34]
Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 2695-2709, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v 1/w023.222222222 http222....lllpmmmmeem ciliip-mmin...66 163. URL https: http//aaaaanttttooooooo org/ 2023. emnlp main.163/.
[35]
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam 2: Segment anything in images and videos, 2024, URL https:
/ //arxiv.ors wwooor/aasaao. abs 2408 02448.00077 00714.
[36] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint (rXivv13/2.6034. 2013.
[37] Zhaochen Su, Linjie Li, Mingyang Song, Yunzhuo Hao, Zhengyuan Yang, Jun Zhang, Guanjie Chen, Jiawei Gu, Juntao Li, Xiaoye Qu, et al. Openthinking: Learning to think with images via visual tool reinforcement learning. arXiv preprint arXiv:2505. 08617, 2025.
[38] Didac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings the the IEEE/CVF International Conference on Computer Vision, pages 11888- 118--11888... 2023.
[39] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9568-9578, 2024.
[40] Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng. Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. arXiv preprint 2rX5v:2305.144 14160, 2023.
[41] Lei Wang. Yi Hu, Jiabang He, Xing Xu, Ning Liu, Hui Liu, and Heng Tao Shen. T-sciq: Teaching multimodal of--nnofft reasoning via large language model signals for science question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19162-19170, 2024.
[42] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- language model's 's perception of the world at any resolution. arXiv preprint 12111 2024.
[43] Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprini arXiv:2409. 11199/8880 18869, 2024.
[44] Zefeng Wang, Zhen Han, Shuo Chen, Volker Tresp, and Jindong Gu. Towards the adversarial robustness of lasion-l model with chain-of ofiinoof-thougt reasoning. 2023.
[45] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of of--hhu prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24877, 2022.
[46] Jingxuan Wei, Cheng Tan, Zhangyang Gao, Linzhuang Sun, Siyuan Li, Bihui Yu, Ruifeng Guo, and Stan Z Li. Enhancing human-like multimodal reasoning: a new challenging dataset and comprehensive framework. Neural Computing and Applications, 36(33): 2024.
[47] Penghao Wu and Saining Xie. V?: Guided visual search llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13084-13099 13094, 2024.
mechanism in multimodal
[48] Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10371-1038) 2024.
13

---

[49] Fanglong Yao, Changyuan Tian, Jintao Liu, Zequn Zhang, Qing Liu, Li Jin, Shuchao Li, Xiaoyu Li, and Xian Sun. Thinking like an expert: Multimodal appergrph--of-thoee (hot) reasoning to boost foundation modals. arXiv preprint arXiv:2308. 06207, 2023.
[50] Hao Yin, Guangzong Si, and Zilei Wang. Lifting the veil on visual information flow in milms: Unlocking pathways to faster inference. arXiv preprini arXiv:2503. 13 13008, 2025.
[51] Daoan Zhang, Junming Yang, Hanjia Lyu. Zijian Jin, Yuan Yao, Mingkai Chen, and Jiebo Luo. Cocot: Contrastive chain-or chain-of-thous of-thoughttttt prompting for large multimodal models with multiple image inputs. arXiv preprint arXiv:2401 02582, 2024.
[52] Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[53] Yongheng Zhang, Xu Liu, Ruihan Tao, Qiguang Chen, Hao Fei, Wanxiang Che, and Libo Qin. Vitcot: Video-text interlcaved chain-of-thoupe for boosting video understanding in large language models. arXiv preprint wxXi:2507.09877 2025.
[54] Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Mul- timodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923, 2023.
[55] Ge Zheng, Bin Yang, Jiajin Tang, Hong- Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models. Advances in Neural Information Processing Systems, 36:5 66:5168-5191, 2023.
[56] Qiji Zhou, Ruochen Zhou, Zike Hu, Panzhong Lu, Siyang Gao, and Yue Zhang, Image-of- thought prompting for visual reasoning refinement multimodal large language models. arXiv preprint arXiv:2405. 13872, 2024.
[57] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.. 10592, 2023.
[58] Ziyu Zhuang, Qiguang Chen, Longxuan Ma, Mingda Li, Yi Han, Yushan Qian, Haopeng Bai, Zixian Feng, Weinan Zhang, and Ting Liu. Through the lens of core competency: Survey evaluation of large language models. arXiv preprint arXiv:2308.07902, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: As shown in lines 7-16 of the Abstract and 62-71 1 of the Introduction, present our main claims and outline the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Our work is not strictly purely theoretical work, we provide more of an empirical analysis. We provide theoretical statement the concept in Section 2, including the corresponding assumption and formula.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: As shown in Appendix B to Appendix D, we have provided detailed descrip- tions and analyses of the experimental setups for all our investigations.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
16

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We will release our code in the official version of the subsequent paper to provide reproduction and provide more help to the future community.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https: /tt///isss.... cc/
public/ guides: / SuodeSuomii
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips- cc/ /tppppplcc//oc/cpp ceooliccguiiee feiides//e Code SudmeSSuuiiss ss ciiionolli oooicyycy cy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: As shown Appendix A, we have provided detailed descriptions and analyses of the experimental setups for all our investigations.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide error bars in Figure 6, and explain the error variables in Section 4.3. However, error bars are reported for all tasks because it would be expensive for human annotation and computational resource consumption.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: As shown in Appendix A, outline the specific model compute resources provided.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't itdn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp//euurp..///ee//ee wwtpp//www../.tttttoooo.ee
Answer: [Yes]
Justification:
We convinced that comply with NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed the broader impacts of our work in Section 7. In addition, since our work is more like providing an empirical analysis and has no additional social harmfulness, we do not discuss this part.
Guidelines:
18

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papersw sww thoode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: As shown Appendix F, We describe and analyze the details and ethical considerations of our crowdsourcing in detail.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
supplemental material is fine, but if the main contribu-
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [No]
Justification: Since our region and institution are not required to provide IRB approval, we do not describe this section. We are convinced that our work complies with the NeurIPS Code of Ethics and the guidelines.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Our core method this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: //nuurirsspp./ cc/Confereecces /20cl// 2025/ LLM) for what should or should not be described.
21

---

Appendix
A Experiment Setup
Model Settings We conduct all our experiments using four models, including LLaVA-I.55 [21],
Qwen2-VL [42], GPT-4o-mini [32], and GPT-4o [32]. For the GPT series models, we adjust the temperature parameter within [0.2]; for the open-source models, adjust the temperature parameter within the range [0.2]. In addition, all open source models complete inference on 2 A6000 48G.
Benchmark Settings We select benchmarks from both math and commonsense categories. For the math tasks, we choose IsoBench [10] involving tasks such as chess, math, graph, etc. For the com- monsense tasks, select datasets including MMVP [39], V*Bench [47], MSCoTTCCommoneenss [4], and CoM' [7], which assess the LVLMs' capabilities such as visual grounding and object detection, fine-grained identification, and CoT reasoning.
B Effectiveness Verification of Visual Thought
In this experimental section, we utilize the CoMT [7] and IsoBench [10] datasets.
B.1 Visual Thought Setting
For the generation of Image-form Visual Thought, for CoMT, we utilize the interleaved -oodal rationales provided within the benchmark itself to construct reasoning process that incorporates image-form visual thought, which then provided to the model as context for answer prediction; for IsoBench, we leverage the function graphs included in the dataset the image-form visual thought, embedding them into a template-basee reasoning process that serves as context.
Subsequently, we construct two variants by modifying the generated image: in the w/o Visual Thought setting, the image replaced with an <image> placeholder; the Text-form Visual Thought setting, the image is substituted with its corresponding caption according to the query, generated by GPT-4o [32].
For the Caption Only setting, we employ GPT-4o [32] to generate textual descriptions directly from the original input image without consider the questions. These descriptions are then provided the model as context for further answer prediction.
B.2 Image Classification
In this section we investigate how the complexity of an image's textual description influences the effectiveness of visual reasoning. We first prompt the model to produce captions that are as precise as possible, and we quantify description difficulty by the number of tokens in each cap- tion. Consistent with commonsense expectations and our manual annotations, images depicting spot-the-difffrence scenarios are the most challenging to describe and are therefore labeled Hard-to-Descrrbee Tanngam-relatee im- ages present moderate level of difficulty and are labeled Medium-to- Micdium-to-Dessrrbbe. By contrast, images showing mathe- matical functions detected through optical character recog- nition (OCR) require the fewest tokens and are labeled Pasy-to-Descriln
SMMil-Selection
CoMT-Uppdaee
Math-breakpcint
100
$4.0
Direct
X w/o
80
Performanee
60
42.0
31.0-
20
13.012.0
Figure 11: Results of Direct and w/o
Visual Thought prompting on GPT-4o.
B.3
Additional Discussion on w/o Visual Thoughts
To better substantiate that the performance degradation observed in the w/o Visual Thought scheme proposed in Figure 4 (a) does not merely stem from contextual perturbations, we compare direct prompting with GPT-40 to generate responses (Direct) against explicitly instructing to avoid producing any visual content (w/o VT prompting). As shown in Figure 11, the results obtained by w/o VT prompting demonstrate a noticeable performance drop when visual information entirely omitted by the model, even though the surrounding context remains complete.
22

---

Query Only
Imsago-orm Thought O Text-form Visual Thought
Visual Thought
+ Image-form
Caption-Only
Thought O
O w/o "Toought
toens Visual Thought
100
90
Accameyy
5 8:
Accurac
O0
Graph
Hard-to-
Describe
Medium Describe Imsge
Parity
Convexity
Breakpaint
Image
Emage
(a) Effectiveness verification on Qwen2-VL.
(b) Visual Thoughts v.s. Caption-Only.
Figure 12: Effectiveness Verification for Visual Thoughts pure text problem.
B.4
More Results
Beyond the exploration in Section 3, we further investigate the effectiveness of incorporating visual thought into pure-text problems that require spatial imagination. In this setting, visual thought serves to convey visual information from the imagined spatial domain that is necessary for solving the textual problems. Specifically, we select tasks including chess, function, and graph from IsoBench and Qwen2-VL VL the experimental model. We employ VisualSketchpad[ to generate reasoning processes that include image-form visual thought, keeping all other settings the same.
Integrating visual thoughts is essential for MCoT's effectiveness in both image and text expression. As shown Figure 12 (a), consistent with the conclusions Section 3, the absence of visual thoughts results in a noticeable drop in accuracy (even worse than relying solely on the query-only). This highlights the critical function of visual thoughts in transmitting visual information and boosting model performance.
Visual thoughts can encode visual information more efficiently than captions in more complex scenarios. As shown in Figure 12 (b), in simple scenes, where descriptions are straightforward, the gain is modest (7.24%). In scenes of moderate complexity, the gain rises 19.54%. In highly complex images, where concise captions often struggle, visual thoughts deliver over 50% efficiency improvement. These findings demonstrate that visual thoughts scale with image complexity and offer superior channel for transmitting detailed visual information.
C The Categories of Visual Thoughts
In this part of the experiment, we evaluate the performance of the four types of visual thoughts using four models, including LLaVA-1.5 [21], Qwen2-VL [42], GPT-4o-mini [32], and GPT-4o [32].
C.1 Prompt Design
In the w/o VT setting. the model directly generates both the reasoning path and the final answer based solely on the given query. In contrast, for the four visual thought paradigms (N-LANG, S-LANG, E-IMG, and G-IM)), a two-stage framework is employed. In the first stage, the model prompted generate corresponding visual thought. In the second stage, this visual thought, combined with the original query, serves as additional context to guide the model in deriving the reasoning path and the final answer.
C.1.1 Prompt Design for w/o VT
In the w/o VT setting, the model is tasked with generating both the reasoning path and the final answer directly from the given query, without relying on intermediate visual representations. To ensure minimal influence from visual reasoning, we explicitly instruct the model to avoid referencing or incorporating any visual descriptions during the reasoning process. The specific prompts used in experiments are presented below:
23

---

### Question:
<Q>
### Choices: <C>
Let's think step by step, but try to avoid any visual descriptions during the process! End your thinking process with the most appropriate answer in the format "ANSWER: (x)" followed by the choice.
Your Response:
C.1.2 Prompt Design for N-LANG
N N-LANG facilitates effective visual information transfer by natural language expression, such as describing images based on question, enhancing vision-language alignment in LVLMs through richer visual descriptions.
Stage 1: Visual Thought Generation At this stage, the model is tasked with extracting meaningful information and transforming into natural language, forming foundation for visual thought and enabling downstream reasoning. The specific prompts used in our experiments are as follows:
### Question: <Q>
### Choices: <C>
Please generate comprehensive caption for the given image based the provided query, ensuring accurately reflects the content and context the image and the query.
### Caption:
Stage 2: Visual Thought Reasoning Then, the model advances to cross-modal reasoning, build- ing its previously generated visual thoughts. Below the prompts used in this stage of experiment:
Based on the question and the caption that is related to the question and generated by yourself, let's think step by step, but try to avoid adding visual descriptions during the process! End your thinking process with the most appropriate answer in the format "ANSWER: (x)" followed the choice.
### Question: <Q>
### Choices: <C>
### Caption: <N-LANG>
Your Response:
C.1.3 Prompt Design for S-LANG
L-LANG facilitates effective visual information transfer by effectively incorporating structures language into reasoning pipelines.
Stage 1:
Visual Thought Generation In this stage, the provided image constitutes the primary source of visual information. The model required carefully analyze the visual content and interpret it in a structured manner that facilitates subsequent reasoning processes. The specific prompts employed in our experiments are follows:
### Question: <Q>
#### Choices: <C>
For the provided image and its associated question, generate a scene graph in JSON format
that includes the following:
1. Objects that are relevant answering the question.
2. Object attributes that are relevant to answering the question.
3. Object relationships that are relevant to answering the question.

---

Stage 2: Visual Thought Reasoning Next. the model performs advanced cross modal reasoning based on the visual thoughts generated in the previous stage. The specific prompts used in our experimental setup are listed below:
Based on the question and the scene graph that is related to the question and generated by yourself, let's think step by step, but try to avoid adding visual descriptions during the process! End your thinking process with the most appropriate answer in the format "ANSWER: (x)" followed by the choice.
### Question: <Q>
### Choices: <C>
### Scene Graph: <S-LANG>
Your Response:
C.1.4 Prompt Design for E-IMG
E-IMG transforms the original image through tasks such as grounding, depth estimation, and seg- mentation. By encoding image tokens, it strengthens LVLMs' capacity to interpret visual information, thereby enhancing reasoning performance.
Visual Thought Generation At this stage, E-IMG employs vision tools (e.g., Grounding- DINO [24], Semantic-SAM [20], DeythAnything [48]) to edit images based on action series. These processed images serve as extended inputs, facilitating the formation of visual thoughts in the subsequent reasoning stage. The prompts used our experiments are as follows:
Stage 1:
### Question: <Q>
#### Choices: <C>
Given the image, questions, and options, please think step-by-step about the image to answer the question, design series of image processing steps to extract pointing visual features based the available actions.
The available actions are:
1. segment and mark():
2. detection(objects)::
You encouraged few steps as possible achieve the goal.
# Action Series:
Stage 2:
Visual Thought Reasoning At this stage, the model advances to cross-man modal reasoning tasks, guided by visual thought representations formed earlier. The prompts used in our experimental procedure are detailed below:
Based on the question and the additional annotated image (all images except the first one) that related to the question and created according to Image Processing Series generated by yourself, let's think step by step, but try to avoid adding visual descriptions during the process! End your thinking process with the most appropriate answer in the format "ANSWER: (x)" followed by the choice.
### Question: <Q>
### Choices: <C>
<Extra Image Input>
Your Response:
C.1.5 Prompt Design for G-IMG
G- G-IMG is designed prompt generative models to produce images that support logical reasoning, leveraging the progress of LVLMs.
25

---

Model
VeRench
MrCaT
CoNFT
position
auribute
physical
temperal
selection
upslne
$3.95
54.77
8989
19.50
LANG
S-IM2
83.00
59.21
65.22
90.00
86.18
34,00
28.50
63,80
G-1162
75.00
59.21
39:13
92.22
73.93
86.15
33.50
28.50
46.50
02.46
w/a GoT
77.67
37.49
63.48
87.78
76.03
58.54
32,50
21,50
41.50
57.45
Table Results of w/o CoT on GPT-4o.
Stage 1: Visual Thought Generation At this stage, IMG employs the vision model DALL-E [l] to generate images based on tailored image generation prompis. These images embody the model's visual thoughts and are used additional imputs in the subsequent reasoning phase. The prompts used in our experiments are as follows:
You are an expert in writing prompts for text-to-image generation. Now, based on the following image and the corresponding textual query, please write precise and detailed prompt to generate an image that is highly relevant to the query. This image will be provided you later as auxiliary tool to help answer the query. Therefore, the generated image should be as clear, detailed, and closely aligned with the query possible, helping you extract necessary information from the image to answer or resolve the query accurately.
When writing the prompt, please consider factors such as the composition, color, style, and details of the image to ensure its practicality and effectiveness.
#### Question: <Q>
### Choices: <C>
### Prompt Generated:
Stage 2:
Visual Thought Reasoning In this phase, the model performs cross-modal reasoning tasks, drawing on visual thought representations established in the prior stage. The corresponding experimental prompts are presented below.
Based on the question and the additional synthesized image (the second one) that is related to the question, let's think step by step, but try to avoid adding visual descriptions during the process! End your thinking process with the most appropriate answer in the format "ANSWER: followed by the choice.
##### Question: <Q>
### Choices: <C>
<Extra Image Input>
Your Response:
C.2
More Results on w/o CoT
To gain a deeper understanding of the role of visual thoughts, we supplement the results in Table with additional vanilla baseline which is without any CoT reasoning, implemented using GPT-4o. As shown in the Table 3, we observe that the trends of removing Visual Thoughts (VT) and CoT are not consistent. For benchmarks that require extensive multi-step reasoning, the performance of the w/o CoTT variant significantly degrades.
reasoning steps, such as M&CoT, which involves
C.3 Core Factors Evaluation Mechanism
In Section 4.3, we conduct human evaluation to assess the conciseness and efficiency of visual thoughts. Specifically, we randomly sample 100 instances evenly distributed across MMVP [39]. V*Bench [47], M3CoT [4], and CoMT [7].
26

---

To evaluate the effectiveness of visual thoughts generated in response to user queries, we introduce three evaluation metrics. Each metric is designed to capture specific aspect of visual reasoning quality, as defined below:
This metric assesses the degree which the visual thoughts aligns with the semantic content and context of the imput image. A highly relevant visual thoughts output should accurately reflect the objects, relationships. or scenes depicted in the image, ensuring that the generated content remains faithful to the source material.
Image Relevance:
Expression Clarity: This criterion measures how clearly the visual thoughts conveys the in- tended reasoning or logic in response to the input query. A high score indicates that the visual elements, such as spatial arrangements, symbols, or visual cues, are intuitively understandable and unambiguous, allowing viewers to easily grasp the underlying rationale.
Concise Expression: This metric evaluates the efficiency and succinetness of the visual thoughts in communicating information. An effective visual thoughts should avoid unnecessary visual complexity while preserving essential content, thereby enhancing interpretability and reducing cognitive load on the viewer.
Each of the three metrics scored on a 3-point ordinal scale: (1) Low: The visual thoughts fails to meet the criterion or introduces confusion. (2) Medium: The visual thoughts partially meets the criterion, with minor ambiguities or limitations. (3) High: The visual thoughts fully satisfies the criterion in a clear, accurate, and efficient manner.
Detailed annotation guidelines provided as below:
Each metric is rated on a 3-point scale: low (1), medium (2), and high (3). The evaluation criteria for each score level are detailed below:
Image Relevance: measures the consistency between the visual thought and the content of the input image.
Low: The visual thought misrepresents incorrectly describes the image content.
Medium: The visual thought accurately captures the main content of the image.
High: The visual thought not only accurately represents the image but also provides
comprehensive description.
Expression Clarity: Assesses how clearly the visual thought represents the visual logic implied by the input query.
Low: The visual thought fails convey any visual logic relevant to the query.
Medium: The visual thought partially captures the visual logic relevant the query.
The visual thought fully and clearly expresses the visual logic associated with the query.
High:
Concise Expression: Evaluates the comprehensibility and succinctness of the visual thought in conveying visual information.
Low: The visual thought is verbose, redundant, difficult understand.
Medium:
The visual thought conveys its visual content and logic in generally clear
manner.
High: The visual thought presents the visual content and logic clearly, concisely, and in an easily understandable way.
D Internal Rationale Behind the Visual Thought
In Section 5, we analyze the internal mechanisms by which MCoT transmits visual information, focusing on two perspectives: the attention mechanism and information flow.
27

---

Model
MMVP
V"Beach
MrCoT
CoMT
AVG.
posttion
atrrbttes
sacial
deletion
selection
updlie
w/o
74,33
53.15
54.76
58.59
76.86
79.67
26.50
19.50
57,00
56,83
N- L (Sa 184)
$5.50
57.89
663.45
58.89
73.53
K3.74
33.50
25.50
57.50
02.60
S-LaNG (843
$3.67
02,15
66.35
90:000
73.51
K2.93
29.50
18.00
42.00
02.35
E-1MG (Bta 524)
$2.33
59.21
65.22
92.22
78.10
85.15
33.50
25.00
49.00
63.75
3-236 IBta 524)
79,00
59.21
63.45
92.22
78.93
86.15
33.50
28.55
66.50
02.95
Diverse-97
$5.00
63.16
65.22
92.23
78.93
96.18
34,00
28,50
$0.00
64,80
Table 4: Results of Diverse- VT GPT-4o.
D.1 Visual Attention Analysis
In this section, we employ LLaVA-1.5-7B [21] as the experiment model to analyze the attention distribution during its reasoning process and randomly select one sample from the ScienceQA [25] as case study. Specifically, we compare two input settings: (1) a multimodal query alone, and (2) multimodal query with visual thought. For each setting, we extract the attention weights from the last token to the input image and the visual thought in each transformer layer.
We define A (L.b) as the attention weight of token attending to token j of the h-th head at the 1-th layer, and the attention weight extraction process can be represented as:
H
(L.b)
(8)
last
H
Inst,k
h=1
where 2 ius represented the target extracted attention weight, last represents the index of the last token, I represents the index of the target token, and H represents the number of the head.
As illustrated in Figure 8, we observe that the inclusion of visual thoughts leads to a shift in attention from the original image toward the visual thought. Moreover, the visual thought continues to carry and transmit visual information into deeper layers of the transformer.
D.2 Visual Information Flow Analysis
In this section, we also utilize LLaVA-I.5-72 [21] as the experimental model and sample data from the ScienceQA [25] investigate the role of internal information flow within MCoT. Specifically, conduct two analyses: Attention Blocking Analysis and Saliency-Basee Information Flow Analysis
D.2.1 Attention Blocking Analysis
We manually block the information flow between specific tokens by setting the attention mask between them selected transformer layers to -inf [40, 50J. As shown Figure 9, blocking the information flow between the query and the visual thought leads the model to choose an incorrect option that it would otherwise have predicted correctly. This result highlights the critical influence of information flow between the query and the visual thought on final answer prediction.
D.2.2 Saliency-Based Information Flow Analysis
We compute saliency scores [36] to evaluate the relative importance of different information flows for answer prediction [40, 50]. The computing process utilize Taylor expansion [29] for each element of the attention matrix:
OC(x) AA
1
Σ h
A O
(9)
Here, f(x) represents the loss function, and denotes the input; A₁, is the attention matrix value for the h-th attention head in the (-th layer. The saliency matrix 4 for the (-th layer is obtained by averaging across all heads. The significance information flow from the j-th token the i-th token can be represented by (t..)).
As illustrated in Figure 10, the information flow between the visual thought and the reasoning process plays pivotal role in guiding the model toward the correct answer, and its contribution is significantly greater than that of the image-to-reasonn reasonnnnn pathway.
28

---

E
Future Work
Based on the empirical analysis of visual thoughts, exploring ways to further enhance the performance of MCoT is promising direction for future research. Here, inspired by self-consistency, we design novel risual-thoughh-guiiee strategy that ensembles diverse visual thoughts reasoning paths, which are refered as diverse-VT. We implement diverse- VT on GPT-4o. As shown in Table 4, we observe that Diverse- VT achieves the best performance, demonstrating the feasibility of integrating multiple visual thoughts to enhance visual information transmission in MCoT, thereby improving reasoning performance.
Ethical Considerations
We engage multiple human annotators to evaluate the quality of visual thoughts in Section 4.3.
Quality Check We initiated the project with introductory interview task, in which participants answered 10 example questions for each visual thought expression. To ensure participant engagement and familiarize them with the task, each participant was compensated $20.
Dataset Annotation For the subsequent stages of data annotation, we employed two PhD students and two graduate students who possessed proficiency in both Chinese and English (CET-6 level) and demonstrated strong mathematical capabilities. These students were compensated $15 per hour, which is higher than the local average salary.
29


Reviews:
[{"review_id": "h348n7EBf8", "reviewer": "Reviewer_7efu", "summary": "This paper introduces the concept of visual thoughts as a unifying abstraction for intermediate reasoning steps in multimodal chain-of-thought (MCoT) systems. It categorizes four modalities of visual thought expressions (natural language, structured language, edited image, generative image) and empirically compares their effectiveness across a variety of benchmarks and large vision-language models (LVLMs). The authors further analyze attention patterns and information flow to argue that visual thoughts function as a “cache” that facilitates deeper reasoning.\n\nThis paper focuses on categorizing the idea of visual thoughts, and analysis of it, without clear technical contributions of new methods or datasets.  To improve the paper, show that explicitly designing models or prompts informed by the “visual thoughts” concept yields improvements over existing approaches.", "questions": "see above", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 2, "strengths_and_weaknesses": "# Strengths\n* Provides a clear, structured categorization of intermediate visual reasoning mechanisms.\n* Attention and information-flow studies provide some internal evidence supporting the “cache” hypothesis.\n* Presents thorough experiments across several tasks and models, supported by ablation and diagnostic analyses.\n\n# Weaknesses\n* The core contribution — the concept of visual thoughts — is more an interpretive narrative than a substantive technical advance.\n* The work primarily re-labels existing techniques (scene graphs, image editing, image generation, captions) rather than proposing new methods or benchmarks.\n\n# Suggestions\n* Benchmark directly against the most related prior works (e.g., Visual Sketchpad [15], Image-of-Thought [55], Scene Graph CoT) under comparable conditions.\n* Add an experiment showing that explicitly designing prompts or models informed by the “visual thoughts” perspective achieves measurable improvements over existing baselines.\n* Report stronger statistical analysis where possible: include confidence intervals, p-values, or more runs to strengthen claims about significance.", "was_revised": true, "final_justification": "after reading the rebuttal, other reviews and discussion, I've increased my score from BR to BA.  The rebuttal highlighted the contribution of the paper, and its transferrable insights.", "rebuttal": "Thanks for your acknowledgment and interest in our work! We sincerely appreciate your thorough and insightful comments on our work, and we will address each of your main concerns below:\n\n**Q1**: The concept of visual thoughts is more an interpretive narrative than a substantive technical advance, like new methods and benchmarks.\n\n**R1**: Thanks for your kind feedback. We sincerely believe there may be a misunderstanding. We sincerely think that the main contribution of our work is not to introduce new methods or benchmarks, but to present visual thoughts as a unifying perspective to better understand the mechanisms underlying MCoT (one of the most influential paradigms in the field). This contribution has also been acknowledged by other reviewers (Reviewer #NrVc, Reviewer #Z1x4, Reviewer #79f4).  From this viewpoint, our study yields several notable and thought-provoking findings:\n- First, we identify visual thoughts as the central functional component of MCoT. Regardless of their specific form, visual thoughts act as cache-like carriers of visual information during the reasoning process.\n- Second, we conduct a comparative analysis of various MCoT implementations and reveal that the key determinants of performance are the clarity and efficiency of these visual thoughts.\n- Third, we examine attention patterns and information flow to uncover the internal mechanisms by which visual thoughts serve as core intermediaries. Our findings suggest that they play a crucial role in sending more visual information to deeper model layers to produce the better reasoning trajectory of MCoT.\n\nWe hope this study can serve as a  foundation for understanding how visual reasoning unfolds in large vision language models. We will **definitely** add more discussion in the next version to better address your technical concerns.\n\n**Q2**: Benchmark directly against the most related prior works (e.g., Visual Sketchpad [15], Image-of-Thought [55], Scene Graph CoT) under comparable conditions.\n\n**R2**: Thank you for your constructive feedback. Actually, the experiments shown in Figure 10 of our paper’s appendix were conducted directly on the Visual Sketchpad. In Table 1, our N-LANG setting is aligned with MM-CoT, the S-LANG setting corresponds to Scene Graph CoT, and the E-IMG setting is consistent with the Visual Sketchpad (note that the Image-of-Thought setting actually involves multi-turn visual reasoning).\n\nIn addition, to ensure what you referred to as “under comparable conditions,” we constrained all methods to generate responses within two interaction rounds, thereby ensuring comparability. Moreover, our comparison is not only direct but also accompanied by a detailed analysis of their differences and the underlying reasons.\n\nTherefore, we have in fact conducted a comparison with the most relevant prior works under comparable conditions. We will revise the next version of the manuscript to clarify this point more explicitly.\n\n**Q3**: Add an experiment showing that explicitly designing prompts or models informed by the “visual thoughts” perspective achieves measurable improvements over existing baselines.\n\n**R3**:  Thanks for your insightful feedback. Inspired by self-consistency, we follow your suggestion to design a novel visual-thought-guided strategy that ensembles diverse visual thoughts reasoning paths, which are refered as diverse-VT. As shown in Table 1, we observe that Diverse-VT achieves the best performance.\n>Table 1: Result of Divserse-VT.\n|| MMVP  | V\\*Bench-Position | V\\*Bench-attributes | M3CoT-physical | M3CoT-social |  M3CoT-temporal  | CoMT-deletion  | CoMT-selection  | CoMT-update | AVG |\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| w/o VT | 74.33 | 53.95 | 54.78   | 88.89   | 76.86  | 79.67 | 26.50 | 19.50 | 37.00 | 56.83 |\n| N-LANG (Maj@4)    | 85.00 | 57.89 | 63.48 | 88.89 | 78.93 | 83.74 | 33.50 | 25.50 | 37.50 | 61.60 |\n| S-LANG (Maj@4)    | 83.67 | 63.16  | 64.35  | 90.00 | 78.51 | 82.93 | 29.50 | 18.00 | 42.00 | 61.35 |\n| E-IMG (Maj@4)     | 82.33 | 59.21| 65.22   | 92.22  | 78.10  | 86.18 | 33.50 | 28.00 | 49.00 | 63.75|\n| G-IMG (Maj@4)     | 78.00 | 59.21 | 63.48    | 92.22    | 78.93   | 86.18 | 33.50 | 28.50 | 46.50 | 62.95|\n| Diverse-VT        | 85.00 | 63.16   | 65.22    | 92.22 | 78.93  | 86.18 | 34.00 | 28.50 | 50.00 | 64.80 |\n\nWe will add more discussion in the next version.\n\n**Hope our clarifications can address your concerns, and we sincerely hope that you can reconsider our work in light of these clariìcations.**"}, {"review_id": "rIr5L8OOoV", "reviewer": "Reviewer_NrVc", "summary": "The authors present a taxonomy and analysis of multimodal chain-of-thought (MCoT) reasoning, categorizing thoughts into two main types: textual and visual, with four subtypes—natural language, structured language, image editing, and image generation thoughts. The study emphasizes the distinct roles of textual vs visual thoughts, supported by quantitative experiments on four datasets. The analysis includes computing correlation, flow, and attention scores to understand the impact of each thought type. Key findings include visual thoughts are crucial for enhancing MCoT reasoning, with their effectiveness depending on conciseness and clarity. Also, input image information is propagated to deeper transformer layers primarily through visual thoughts.", "questions": "1. The finding in Section 5.1 is interesting. I’m curious whether this phenomenon only appears in deep models. Furthermore, is visual thought more beneficial for deeper models? Could you test this with a quantitative experiment?\n\n2. The differences between visual and textual thoughts in Flow and Disturbation might be related to the LLAVA training pattern. If you have access to models with different training patterns, it would be interesting to see whether these phenomena still hold.\n\n3. In Figure 6, manually annotating relevance may limit the data scale. While human annotation is valuable and should be kept, you might also consider using auto methods (e.g., CLIP) to generate scores.\n\n4. Beyond image similarity, I’d like to better understand the role of visual elements through representation analysis—e.g., by examining similarity between visual thought key/value pairs in attention.", "limitations": "no potential negative societal impact", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n* Interesting topic with a clear categorization of multimodal CoT.\n* Insightful analysis with good visualizations.\n* Multiple perspectives and interesting findings.\n\nWeaknesses:\n* Some analyses could go deeper, would benefit from more extended results.", "was_revised": true, "final_justification": "This paper provides a detailed analysis of the mechanisms of visual thought and will help the community gain a better understanding of VLM inference. Although it does not propose a significant new method, I still find it valuable and above the bar.", "rebuttal": "Thanks for your acknowledgment and interest in our work! We sincerely appreciate your thorough and insightful comments on our work, and we will address each of your main concerns below:\n\n**Q1**: Is the finding in Section 5.1 only appearing in deep models? Is visual thought more beneficial for deeper models?\n\n**R1**: Thank you for your constructive feedback. To investigate how visual thoughts influence internal attention across models of different depths. We select four models with varying numbers of layers: LLaVA-1.5-7B (32 layers), LLaVA-1.5-13B (40 layers), Qwen2.5-VL-3B (36 layers), and Qwen2.5-VL-7B (28 layers).  \n\nSpecifically, we extract the model’s internal attention to the original input image during inference using the method without incorporating visual thoughts (no-vis), and extract the internal attention to visual thoughts using methods that incorporate visual thoughts (n-lang, s-lang, e-img, g-img), following the approach illustrated in Figure 7(b) of the paper. This allows us to compare the shifting trend of attention after integrating visual thoughts. \n\nWe report the percentage of layers in which the attention score to visual thoughts exceeds that to the input image, as well as the percentage within the latter 50% of layers (shown in parentheses). The results are as follows:\n> Table 1: The percentage of layers in which the attention to visual thoughts exceeds that to the input original image. The number in parentheses indicates the result only discussed in the latter 50% layers of the model.\n|| Layer |N-LANG(%)| S-LANG(%) | E-IMG(%)| G-IMG(%)|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n| LLaVA-1.5-7B    | 32    | 43.75 (12.50)  | 40.63 (6.25)   | 96.88 (100.00) | 93.75 (100.00) |\n| LLaVA-1.5-13B   | 40    | 20.00 (0.00)   | 35.00 (5.00)   | 95.00 (100.00) | 92.50 (100.00) |\n| Qwen2.5-VL-3B   | 36    | 97.22 (100.00) | 86.11 (100.00) | 72.22 (88.89)  | 58.33 (100.00) |\n| Qwen2.5-VL-7B   | 28    | 100.00 (100.00)| 92.86 (100.00) | 60.71 (100.00) | 60.71 (100.00) |\n\nAs shown in Table 1, **for both the LLaVA and Qwen models with various number of layers**, in majority of layers, the attention of visual thoughts exceeds that of the original image, **demonstrating that the majority attention are distributed to visual thoughts during the reasoning process**.\n\nFurthermore, the results in \"()\" indicate that in the latter 50% layers of the model, attention to visual thoughts almost entirely surpasses that to the original image, even 100% visual thought attention is higher than original image, suggesting that **visual thoughts can convey visual information more deeply than the original image itself**. These findings are consistent with our observations in the paper.\n\n\\* In the n-lang and s-lang settings for LLaVA, the model shows relatively low attention to visual thoughts due to the self-generated weaker language capabilities, resulting in captions and scene graphs that are overly simplistic and fail to accurately convey visual information.\n\nWe further compute the difference in attention score to visual thoughts relative to the original image, and average the results across all layers. The results are as follows:\n> Table 2: The difference in attention score to visual thoughts relative to the original image across all layers.\n|| Layer | N-LANG | S-LANG| E-IMG | G-IMG|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n| LLaVA-1.5-7B| 32 | -0.0529 | -0.0472  | 0.0156 | 0.0256|\n| LLaVA-1.5-13B | 40| -0.0481| -0.0275  | 0.0552 | 0.0637 |\n| Qwen2.5-VL-3B| 36| 0.1280| 0.0438| 0.0229 | 0.0388 |\n| Qwen2.5-VL-7B| 28 | 0.1069| 0.0553  | 0.0189 | 0.0152|\n\nAs shown in Table 2, both LLaVA (7B->13B) and Qwen(7B->3B) exhibit increased attention gains with deeper model layers. In the case of Qwen2.5-VL, the 3B variant, **which has more layers, shows a more significant increase in attention** compared to the 7B variant. This suggests that the impact of visual thoughts depends more on the specific model architecture than on the parameter scale.\n\n**Q2**: The findings about information flow can be related to the LLAVA training pattern and it would be interesting to explore the different models.\n\n**R2**: Thank you for your valuable comment. We follow your suggestion to further investigate the distribution of information flow under the integration of the four types of visual thoughts using Qwen2-VL-2B.\n> Table 3: Information flow distribution of Visual Thought and Image using Qwen2-VL-2B.\n|| N-LANG| S-LANG| E-IMG| G-IMG|\n|-|:-:|:-:|:-:|:-:|\n| original image to reasoning | 0.0028 | 0.0007 | 0.0022 | 0.0022 |\n| visual thought to reasoning | **0.0056** | **0.0010** | **0.0038** | **0.0038** |\n\nAs shown in Table 3, we observe that visual thoughts can achieve larger information flow than the original image across all visual thought variants. Therefore, we can obtain the same conclusion as with the LLaVA models: *“The information flow from visual thoughts to the reasoning stages is considerably stronger than the direct flow from the image to reasoning.”*\n \nFurthermore, we observe that, in contrast to the behavior of LLaVA, Qwen2-VL achieves higher information flow distribution in the n-lang setting, owing to its stronger language generation capabilities.\n\nWe will add more discussion in the next version.\n\n**Q3**: Auto methods (e.g., CLIP) can be beneficial for conclusions within Figure 6. \n\n**R3**: Thanks for your insightful feedback. To address the concerns regarding human evaluation bias and data scale limitations, we adopt your suggestion and use CLIP-based clarity scores to evaluate all visual thoughts.\n\nSpecifically, we compute the classification probability between two categories: “This is a clear (annotated logical) image” and “This is not a clear (annotated logical) image” (for G-IMG, content in parentheses is removed). These probabilities serve as quantitative indicators for clarity score of visual thought.\n> Table 4: Correlation analysis between clip-based clarity scores and model performance on GPT-4o-mini.\n|| Pearson Correlation| P |\n|-|:-:|:-|\n| S-LANG | 0.692| 0.06 |\n| N-LANG | 0.693| 0.04 |\n| G-IMG  | 0.731 | 0.04|\n| E-IMG  | 0.646| 0.06|\n\nAs shown in Table 4, there is a clear correlation between clarity scores and model performance, indicating that high-quality visual thoughts are critical for achieving strong results. This aligns with our observations in paper.\n\n**Q4**: The role of visual thoughts through representation analysis.\n\n**R4**: Thank you for your constructive suggestions. Combining the results from Table 1 and Table 2, we observe that the internal attention mechanism reflects the effectiveness in conveying visual information of visual thoughts. For instance, in the case of LLaVA, which has relatively weaker language capabilities, the generated captions (N-LANG) or scene graphs (S-LANG) tend to be of lower quality and fail to effectively convey visual information, resulting in lower attention scores. This highlights the correlation between visual thoughts and the model’s internal representations."}, {"review_id": "9xj0UgsWEo", "reviewer": "Reviewer_Z1x4", "summary": "The paper introduces and analyzes the concept of 'visual thoughts', intermediate reasoning steps in Multimodal Chain-of-Thought (MCoT) reasoning with Vision-Language Models (VLMs). These visual thoughts, expressed as either text or images, serve as distilled representations of visual input that differentially enhance VLM performance across tasks. Acting as a form of internal visual cache, visual thoughts reduce the need for repeated image processing and help propagate visual information into deeper layers of the model during reasoning.", "questions": "- What metric is plotted on the y-axis of Figure 5? Why do all columns add up to 1? Please clarify in the caption or main text.\n- In Section 5.2, how exactly is the intervention performed to disrupt the information flow from visual thoughts or the image? Is this based on manipulating attention scores?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- The paper offers practical insights into the suitability of different types of CoT strategies for various tasks. These observations are well-motivated and could guide future work in selecting or designing appropriate MCoT strategies depending on task characteristics.\n\n- The mechanistic analysis of internal model behavior, particularly attention shifts and information flow, is interesting and contributes meaningfully to our understanding of how visual reasoning unfolds in large vision-language models.\n\n**Weaknesses:**\n\n- **Effectiveness verification of visual thought.** I have some concerns regarding the methodology for the 'w/o visual thought' and 'text-form visual thought' baselines. If these are generated by simply replacing the image in the image-form visual thought outputs with a placeholder or caption, the surrounding text may still implicitly reference the image, which could introduce inconsistencies in the context. This may negatively impact performance not solely due to the absence of the visual thought, but because the resulting context is no longer coherent. Additionally, the authors state that removing visual thoughts leads to performance worse than using the query alone, but Figure 4b shows that the query-only setting is consistently the lowest-performing or close to it, calling this claim into question.\n\n- **Baselines.** The 'w/o VT' baseline of Section 4 is unclear: does it correspond to the 'w/o visual thought' from Figure 4? If so, the paper should clarify this explicitly. Additionally, it would be useful to include a vanilla baseline without any CoT reasoning at all, to better isolate the value added by CoT versus visual thoughts.\n\n- **Visual Attention Analysis.** The conclusions drawn from the attention plots are not entirely convincing. While the paper claims that visual thought tokens receive higher attention in deeper layers, the observed differences appear subtle, and the attention to visual thoughts does not clearly dominate over original attention to the image. Stronger quantitative evidence would strengthen this analysis.\n\n**Typos:**\n\nThe manuscript contains some minor typos, including:\n\n- line 154 'Does these different Visual Thoughts all works?' -> 'Do these different Visual Thoughts all work?'\n- line 188 'Which scenarios does different Visual Thoughts work better?' -> 'In which scenarios do different Visual Thoughts work better?'\n- line 176 it is unclear what 'in rationale' means in this context\n- line 264 it is unclear what 'boarder' means in this context", "was_revised": true, "final_justification": "I acknowledge the efforts made by the authors in the rebuttal, and particularly their explanations related to the Effectiveness Verification of Visual Thoughts and the Visual Attention Analysis. While I share the view of other reviewers that the paper introduces limited technical innovation, I believe that the unified analysis of multimodal CoT introduced in the paper is valuable, and could inform future research in the field. Therefore, I am voting for borderline acceptance.", "rebuttal": "Thanks for your acknowledgment and interest in our work! We sincerely appreciate your thorough and insightful comments on our work, and we will address each of your main concerns below:\n\n**Q1**: Conclusions in effectiveness verification of visual thought can be incoherence of the resulting context.\n\n**R1**:  Thank you for your constructive feedback. To address your concern, we compare direct prompting with GPT-4o to generate results (Direct) against explicitly instructing it to avoid generating any visual content (w/o VT prompting). As shown in Table 1, the results obtained by w/o VT prompting indicate a performance drop when visual information is will never be mentioned by model, even when the context is complete.\n\nIt also aligns our major conclusion: Visual thoughts are the main visual information carrier.\n\nWe will include more detailed discussions in the next version.\n> Table 1: Results on Direct prompting vs. w/o VT prompting.\n|| CoMT-Selection | CoMT-Update | Math-Breakpoint |\n|:-:|:-:|:-:|:-:|\n| Direct | 13.00| 42.00| 84.00|\n| w/o VT prompting | 12.00| 32.00| 72.00 |\n\n**Q2**: The authors state that \"removing visual thoughts leads to performance worse than using the query alone\" maybe not correct.\n\n**R2**: Thank you for your meticulous and insightful comments. We sincerely believe there may have been a misunderstanding regarding our explanation. Actually, what we intended to convey is that in Figure 4(b), removing the visual thought module can, in some cases, lead to performance even lower than the Query-Only baseline. However, this does not imply that all instances of the “w/o Visual Thought” condition perform worse than the Query-Only baseline (as clarified in the first paragraph of Section 3).\n\nMoreover, this phenomenon is observed only in the Math subset, primarily because many Math questions contain sufficiently detailed textual descriptions that allow the model to perform reasoning without relying heavily on visual features. Removing visual thoughts in such cases disrupts the effective transmission of visual cues, which unexpectedly degrades performance. In contrast, for other datasets, the queries themselves carry minimal information (e.g., questions such as “How many people are in the image?”), and thus the Query-Only condition naturally performs worse than “w/o Visual Thought”.\n\nTo further address your concerns, we have included additional experiments, as illustrated in Table 2. Similar trends are observed not only in GPT-4o but also in other models such as GPT-4o-mini and Gemini.\n>Table 2: Results on w/o visual thought v.s. Query-Only.\n|| chess  | graph  |math|\n|-|:-:|:-:|:-:|\n|GPT-4o-mini w/o visual thought|6.00%|38.00% |9.00%|\n|GPT-4o-mini Query-Only|0.00%|3.00%|26.00%|\n|Gemini-2.0-flash w/o visual thought |4.00%| 24.00%|17.00%|\n|Gemini-2.0-flash Query-Only|1.00%|0.00%|29.00%|\n\nWe will incorporate more detailed discussion on this issue in future versions of the paper to ensure clarity and alleviate any potential confusion.\n\n**Q3**: The problem with the baseline and suggestions of including the vanilla baseline without any CoT reasoning at all.\n\n**R3**: Thanks for your insightful feedback. The w/o VT of Section 4 and 'w/o visual thought' from Figure 4 represent the same meaning but not the same implementation as not including the visual thoughts in the rationale. Specifically, we implement methods in Section 4 by explicitly prompting the model and we implement methods in Figure 4 by removing the VT manually.\n\nIn addition, we follow your suggestion to add the vanilla baseline without any CoT reasoning and the results are shown as:\n> Table 3: Results on w/o CoT.\n|| MMVP | V\\*Bench-Position | V\\*Bench-attributes | M3CoT-physical | M3CoT-social | M3CoT-temporal | CoMT-deletion | CoMT-selection |CoMT-update|AVG|\n|-|-|-|-|-|-|-|-|-|-|-|\n| w/o VT | 74.33 | 53.95| 54.78| 88.89 | 76.86 | 79.67 | 26.50|19.50|37.00 |56.83 |\n| N-LANG |85.33| 57.89 | 63.48 | 88.89 | 78.93  | 83.74| 33.50| 25.50  |37.50|61.64|\n| S-LANG|84.33|63.16| 64.35| 90.00  | 78.51 | 82.93 | 29.50  | 18.00 |42.00|61.42|\n| E-IMG | 83.00 | 59.21 | 65.22  | 90.00 | 78.10 | 86.18 | 34.00| 28.50 |50.00 |63.80|\n| G-IMG| 78.00 | 59.21 | 59.13  | 92.22 | 78.93| 86.18 | 33.50 | 28.50  |46.50 |62.46|\n| w/o CoT | 77.67 | 57.89| 63.48| 87.78| 76.03 | 58.54  | 32.50 | 21.50 |41.50|57.43|\n\nWe observe that the trends of removing Visual Thoughts (VT) and CoT are not consistent. As shown in Table 3, for benchmarks that require more reasoning steps, such as M3CoT, which involves extensive multi-step reasoning, the performance of the w/o CoT variant significantly degrades.\n\nMoreover, in most vision-reasoning scenarios, removing VT leads to even worse performance compared to removing CoT. This further substantiates our claim that visual thoughts are the core of MCoT. Rather than merely providing an additional reasoning process, VT plays a crucial role in conveying visual information throughout the reasoning pipeline.\n\nWe will add more discussion in the next version.\n\n**Q4**: The conclusions drawn from the attention plots are not entirely convincing.\n\n**R4**: Thank you for your constructive feedback.\n\n**Attention Distribution Analysis**: To further support the conclusions drawn in Figure 7, we perform additional ***quantitative analyses*** to four models with varying layer counts: LLaVA-1.5-7B (32 layers), LLaVA-1.5-13B (40 layers), Qwen2.5-VL-3B (36 layers), and Qwen2.5-VL-7B (28 layers).\n1. For each model, we extract internal attention to the original image using the NO-VIS method and compare it to attention directed at visual thoughts using the N-LANG, S-LANG, E-IMG, and G-IMG methods, as shown in Figure 7(b).\n2. We then compute the percentage of layers where attention to visual thoughts exceed that to the input image.\n3. This percentage is also calculated specifically for the latter 50% of layers (values shown in \"()\").\n> Table 4: The proportion of model layers in which the attention associated with visual thoughts surpasses that of the original input image. Values in \"()\" correspond to results calculated from the latter 50% of the model’s layers.\n||Layer |N-LANG(%)|S-LANG(%) |E-IMG(%)|G-IMG(%)|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|LLaVA-1.5-7B|32|43.75 (12.50)|40.63 (6.25)| 96.88 (100.00) | 93.75 (100.00) |\n|LLaVA-1.5-13B|40|20.00 (0.00)|35.00 (5.00) | 95.00 (100.00) | 92.50 (100.00) |\n|Qwen2.5-VL-3B| 36 |97.22 (100.00) | 86.11 (100.00) | 72.22 (88.89) | 58.33 (100.00) |\n|Qwen2.5-VL-7B| 28|100.00 (100.00)| 92.86 (100.00) | 60.71 (100.00) | 60.71 (100.00) |\n\n**Results:**\n- Table 4 shows that in most layers across both LLaVA and Qwen models, attention favors visual thoughts over the original image, suggesting that reasoning relies more on visual thoughts. \n- Notably, in the latter half of the layers, attention to visual thoughts often dominates entirely, reaching 100% in some cases, indicating their superior capacity to convey visual information. These findings align with our overall conclusions.\n\n\\* Under the N-LANG and S-LANG settings for LLaVA, attention to visual thoughts is relatively low, likely due to the model’s limited self-generated language capacity. The resulting captions and scene graphs are overly simplistic, failing to capture visual content adequately.\n> Table 5: The difference in attention score to visual thoughts relative to the original image across all layers.\n|| Layer | N-LANG | S-LANG| E-IMG | G-IMG|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|LLaVA-1.5-7B| 32 | -0.0529 |-0.0472| 0.0156 | 0.0256|\n|LLaVA-1.5-13B |40| -0.0481| -0.0275| 0.0552 | 0.0637 |\n|Qwen2.5-VL-3B|36|0.1280| 0.0438| 0.0229 | 0.0388 |\n|Qwen2.5-VL-7B|28|0.1069| 0.0553  | 0.0189 | 0.0152|\n\n**Average Attention Score Anlaysis:** We compute the attention score differences induced by visual thoughts relative to the original image, averaged across all layers. The findings are:\n- As shown in Table 5, both LLaVA (7B->13B) and Qwen(7B->3B) exhibit increased attention gains with deeper model layers.\n- For Qwen2.5-VL, the 3B variant, despite its smaller parameter size, exhibits a greater attention increase than the 7B variant, likely due to its deeper architecture. This indicates that the influence of visual thoughts is more dependent on model design than parameter count.\n\nWe will emphasize this section further and expand the discussion in the next version.\n\n**Q5**:  What metric is plotted on the y-axis of Figure 5? Why do all columns add up to 1? Please clarify in the caption or main text. \n\n**R5**: Thank you for your kind reminder. The y-axis in Figure 5 represents the normalized relative accuracy improvement of each dataset incorporating visual thoughts. The y-axis metric of $i$-th dataset can be mathematically represented as:\n\n$Metric_i^y = \\frac{\\delta acc_i}{\\sum_{j=1}^{n} \\delta acc_j},$ \n\nwhere $\\delta acc$ represents the relative accuracy improvement, $n$ represents the total number of datasets.\nWe then calculate this metric for each type of visual thought, which produces the results in Figure 5 in our paper.\n\nWe will add the details in the next version.\n\n**Q6**: Questions about the experiments on disruption of information flow.\n\n**R6**: Thank you for your valuable suggestion. In the information flow disruption experiment presented in Section 5.2, we adopt the methodology outlined in [1,2], in which the information exchange between designated token pairs is obstructed by assigning a value of $-\\infty$ in selected transformer layers.\n\nWe will provide more explanation and clarification in the next version.\n\n[1] Label words are anchors: An information flow perspective for understanding in-context learning.\n\n[2] Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference.\n\n**Q7**: Typo Problems\n\n**R7**: Thanks for your valuable feedback. We will follow your suggestions to fix all typo problems in the next version.\n\n**Hope our clarification can address your concerns, and we sincerely hope that you can reconsider our work in light of these clarification.**"}, {"review_id": "TiBY8CKRZQ", "reviewer": "Reviewer_79f4", "summary": "Despite advancements in T-MCoT and I-MCoT, few studies analyze the underlying unified mechanisms behind these improvements in MCoT. The paper attributes these performance enhancements to visual thoughts, the reasoning steps that convey visual information from the input images.  The authors further propose four strategies that represent different modalities and approaches to integrate visual thoughts, providing a comprehensive analysis of each strategy.", "questions": "1. The experiments primarily use LLaVA-1.5-7B for attention and information flow analyses, and it is unclear how generalizable the findings are across different LVLM architectures or scales. Could the authors extend their analysis to at least one additional LVLM with a different architecture or scale to validate the robustness of visual thoughts?\n2. The authors claim that removing visual thoughts and forcing reasoning solely from the original image can impair performance more than reasoning directly from the query, as demonstrated by the Math-Breakpoint results in Figure 4 (b). This finding appears counterintuitive and lacks adequate discussion. Furthermore, is it appropriate to draw such a broad conclusion based on a single experiment?\n3. Image-based visual thoughts (E-IMG, G-IMG) rely on external tools such as Grounding-DINO and DALL-E 3, which may introduce inaccurate segmentations or irrelevant generated images. How do such errors impact reasoning performance, and how are they mitigated? Could the authors evaluate the sensitivity of MCoT performance to errors in visual thought generation (e.g., by injecting controlled noise or using lower-quality tools)?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. The paper conducts a systematic evaluation of four visual thought expressions (N-LANG, S-LANG, E-IMG, G-IMG) across multiple tasks (MMVP, V*Bench, M3CoT, CoMT). This multi-task, multi-model approach strengthens the robustness of the findings by demonstrating the effectiveness of visual thoughts across diverse scenarios.\n2. The internal analysis of visual thoughts (Section 5) using attention distribution (Figure 7) and information flow (Figures 8–9) within LVLMs provides a novel perspective on how visual information is processed. This adds depth to the study, as it connects empirical results to model mechanics.\n\n**Weakness:**\n1. The experiments primarily focus on a small set of LVLMs (e.g., GPT-4o, LLaVA), which limits the generalizability of the findings. Without testing on a broader range of architectures, the claim of a “unified perspective” is weakened.\n2. The correlation analysis of clarity and conciseness (Figure 6) relies on manual scoring (0–3 scale). This introduces subjectivity and reduces the reproducibility of the results.\n3. While Table 1 reports performance across tasks, the paper lacks a detailed discussion of why certain visual thought expressions perform better in specific tasks. For example, the superior performance of E-IMG in V* Bench-attributes is noted but not thoroughly analyzed, which limits the interpretability of the results.", "was_revised": true, "final_justification": "The paper offers a thorough analysis of Visual Thoughts. Its findings have the potential to inform future research; however, the insights are somewhat limited. Therefore, I recommend a rating of Borderline Accept.", "rebuttal": "Thanks for your acknowledgment and interest in our work! We sincerely appreciate your thorough and insightful comments on our work, and we will address each of your main concerns below:\n\n**Q1**: The experiments primarily focus on a small set of LVLMs.\n\n**R1**: Thanks for your insightful comment. In our paper, we select a range of LVLMs with diverse architectures and parameter scales, spanning from proprietary models (GPT-4o, GPT-4o-mini) to open-source models (Qwen2-VL, LLaVA-1.5), in order to provide the broadest possible interpretation of the role of visual thoughts.\n\n**Q2**: The correlation analysis of clarity and conciseness (Figure 6) relies on manual scoring.\n\n**R2**: Thanks for your constructive comment. We also employ the automatic evaluation method CLIPScore to assess all the visual thoughts presented in Table 1. Specifically, we compute the classification probability between two categories: “This is a clear (annotated logical) image” and “This is not a clear (annotated logical) image” (for G-IMG, content in parentheses is removed). These probabilities serve as quantitative indicators for clarity score of visual thought.\n> Table 1: Correlation analysis between clip-based clarity scores and model performance on GPT-4o-mini.\n|| Pearson Correlation| P |\n|-|:-:|:-|\n| S-LANG | 0.692| 0.06 |\n| N-LANG | 0.693| 0.04 |\n| G-IMG  | 0.731 | 0.04|\n| E-IMG  | 0.646| 0.06|\n\nFrom the results, we can observe that the trends from human evaluation and automatic evaluation are consistent.\n\nWe will add more discussion in the next version.\n\n**Q3**: Certain sections lack a detailed discussion of why certain visual thought expressions perform better in specific tasks.\n\n**R3**: Thanks for your kind feedback. To investigate the performance differences of visual thoughts across various tasks, we analyze them in conjunction with the characteristics of specific task sets. For instance, CoMT—which places greater emphasis on visual information interaction—tends to benefit more from image-form visual thoughts that effectively convey information. In Q6, we further examine the impact of noise within visual thoughts on task-specific performance, thereby further revealing the factors that contribute to the performance gains brought by visual thoughts. We will add more discussion in the next version.\n\n**Q4**: The experiments for attention and information flow analyses is unclear how generalizable the findings are across different LVLM architectures or scales.\n\n**R4**: Thanks for your insightful feedback. We follow your suggestion to add more models for evaluation. \n1. **For attention analysis**:\n\nWe conduct attention analysis across four models (LLaVA-7B/13B and Qwen2.5-VL-3B/7B) by comparing attention to the input image (w/o visual thoughts) and to visual thoughts (with VT). According to the result, we observe that across most layers—and especially in the latter half—attention to visual thoughts consistently exceeds that to the original image, indicating that visual thoughts can more effectively convey visual information during reasoning, **consistent with findings in the paper**.\n\nMoreover, we find that models with more layers exhibit stronger attention to visual thoughts, suggesting that architecture depth is a key factor in leveraging visual thoughts.\n\nFor methodological details and specific experimental results, please refer to our detailed response to **Reviewer #NrVc (R1)**.\n\n2. **For information flow analysis**:\n\nWe incorporate *Qwen2-VL-2B* to examine information flow under different types of visual thoughts. According to the experimental results, all variants of visual thoughts yield stronger information flow to reasoning stages than the original image, **consistent with our findings in LLaVA models**.\n\nNotably, unlike LLaVA, Qwen2-VL exhibits the highest information flow in the n-lang setting, likely due to its stronger language generation capabilities.\n\nFor a detailed explanation and discussion, please refer to our response to **Reviewer #NrVc (R2)**.\n\n**Due to space constraints, we summarize only the key insights here. We sincerely appreciate your understanding!**\n\nWe will add more discussion in the next version.\n\n**Q5**: The finding of 'forcing reasoning solely from the original image can impair performance more than reasoning directly from the query' appears counterintuitive and lacks adequate discussion.\n\n**R5**: Thanks for your valuable comment. We sincerely believe there may have been a misunderstanding regarding our explanation. Actually, what we intended to convey is that in Figure 4(b), removing the visual thought module can, in some cases, lead to performance even lower than the Query-Only baseline. However, this does not imply that all instances of the “w/o Visual Thought” condition perform worse than the Query-Only baseline (as clarified in the first paragraph of Section 3).\n\nMoreover, this phenomenon is observed only in the Math subset, primarily because many Math questions contain sufficiently detailed textual descriptions that allow the model to perform reasoning without relying heavily on visual features. Removing visual thoughts in such cases disrupts the effective transmission of visual cues, which unexpectedly degrades performance. In contrast, for other datasets, the queries themselves carry minimal information (e.g., questions such as “How many people are in the image?”), and thus the Query-Only condition naturally performs worse than “w/o Visual Thought”.\n\nTo further address your concerns, we have included additional experiments, as illustrated in Table 2. Similar trends are observed not only in GPT-4o but also in other models such as GPT-4o-mini and Gemini.\n> Table 2: Results on w/o visual thought v.s. Query-Only.\n||chess|graph|math|\n|-|:-:|:-:|:-:|\n|GPT-4o-mini w/o visual thought|6.00%|38.00% |9.00%|\n|GPT-4o-mini Query-Only|0.00%|3.00%|26.00%|\n|Gemini-2.0-flash w/o visual thought |4.00%| 24.00%|17.00%|\n|Gemini-2.0-flash Query-Only|1.00%|0.00%|29.00%|\n\nWe will incorporate more detailed discussion on this issue in future versions of the paper to ensure clarity and alleviate any potential confusion.\n\n**Q6**: How do errors injected from inaccurate segmentations or irrelevant generated images impact reasoning performance, and how are they mitigated?\n\n**R6**: Thanks for your constructive feedback. To evaluate the potential performance variations in MCoT caused by errors introduced through image-form visual thoughts, we conduct the following experiment:\n\n**For E-Img**: To evaluate the potential errors introduced by E-Img, we controll the editing effects by adjusting the parameters of the editing tool and observe corresponding changes in MCoT performance. Specifically,\n1. We select GroundingDino as the editing tool and manipulate the *box threshold* and *text threshold* parameters to vary the accuracy of grounding. Images with poor grounding mistakenly annotate irrelevant objects as grounded, resulting in inaccurate edited images.\n2. These edited images are then ranked by GPT-4o based on grounding quality, and the resulting ranking is used to quantify the introduced errors by assigning specific scores to specific ranks.\n3. We sample 100 instances from V*Bench-attributes which are sensitive to E-Img and remove the instance failing to perform grounding, using GPT-4o-mini as the model. The results are shown in Table 3.\n4. We include the method without incorporating visual thoughts (no-vis) as the baseline.\n\n**For G-Img**: To facilitate the control of the quality of the G-IMG, we controll the generation effects by adjusting the parameters of the generation models and observe corresponding changes in MCoT performance. Specifically,\n1. We adopt the open-sourced *Stable Diffusion v1.5*. We manipulate the model’s *guidance scale* parameter to control the degree to which the generated image adheres to the input prompt. Images generated with a low guidance scale often fail to reflect visual information relevant to the original query, thereby resulting in irrelevant generated images.\n2. We then used CLIPScore to evaluate the similarity between the generated images and the input prompts, serving as a metric to quantify the introduced error.\n3. We sample 100 instances from M3CoT which is sensitive to G-Img as the experiment data, using GPT-4o-mini as the model. The results are presented in Table 4.\n4. We include the method without incorporating visual thoughts (no-vis) as the baseline.\n> Table 3: Evaluation of E-Img error sensitivity.\n|box threshold & text threshold(↑)|noise score(↓)|acc(↑)|\n|:-:|:-:|:-:|\n|0.07 & 0.05| 0.9278|29.17|\n|0.14 & 0.10| 0.7806|33.33|\n|0.21 & 0.15| 0.6333| 43.06|\n|0.28 & 0.20|0.4167|54.17|\n|0.35 & 0.25|0.2417| 55.56 |\n|no-vis|-|34.72|\n> Table 4: Evaluation of G-Img error sensitivity\n| guidance scale(↑) | clipscore(↑) | noise score(↓) | acc(↑) |\n|:-:|:-:|:-:|:-:|\n|0|0.6720|0.3280|68.57|\n|2.5|0.7815|0.2185|73.33 |\n|5|0.8168|0.1832|74.29|\n|10|0.8401|0.1599|77.14|\n|15|0.8519|0.1481|77.14|\n|no-vis|-|-|72.38|\n\nBased on the results in Tables 3 and 4, we observe a positive correlation between the quality of edited and generative image and the MCoT performance.\n\nWe further observe that poor-quality edited images (box threshold & text threshold=0.07 & 0.05) or generated images (guidance scale=0) can even impair the reasoning capabilities of MCoT (lower than *no-vis*). Through manual inspection, we find that those images contain substantial misleading information (e.g., grounding numerous objects that do not meet the specified conditions or generating images that convey incorrect visual content). This indicates that inaccurate visual thoughts may introduce noise and, in turn, lead to degraded performance.\n\nTo mitigate the introduced errors, we can adopt more powerful editing and generative models, and leverage general-purpose LVLMs to evaluate the resulting images for timely correction.\n\nWe will **DEFINITELY** add more the discussion for mitigate your concerns in the next version."}]

Author rebuttal:

Meta-review: This paper introduces the concept of Visual Thoughts as a unifying framework to interpret and analyze intermediate reasoning steps in Multimodal Chain-of-Thought (MCoT). All reviewers converged on a borderline accept rating. While initial concerns centered around limited methodological novelty and clarity, the detailed rebuttal and new results convinced reviewers of the paper’s utility. 

This is not a “breakthrough” paper in the traditional NeurIPS sense, but it offers an important perspective that clarifies and systematizes an emerging area (MCoT). The empirical results and mechanistic insights could directly inform both academic and applied work in multimodal reasoning. Given this, the paper is recommended for poster acceptance.
Decision: Accept (poster)