Enhancing LLM Reasoning via Vision-Augmented Prompting
Authors: Ziyang Xiao
Dongxiang Zhang
Xiongwei Han
Xiaojin Fu
Wing Yin YU
Tao Zhong
Sai Wu
Yuan Jessica Wang
Jianwei Yin
Gang Chen
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Multimodal Large Language Models
Dual-Modality Reasoning
Primary area: other
TL;DR: A dual-modality reasoning framework that enhances reasoning capabilities by incorporating self-synthesized visual and spatial information for problem-solving.

Abstract: Verbal and visual-spatial information processing are two critical subsystems that activate different brain regions and often collaborate together for cognitive reasoning. Despite the rapid advancement of LLM-based reasoning, the mainstream frameworks, such as Chain-of-Thought (CoT) and its variants, primarily focus on the verbal dimension, resulting in limitations in tackling reasoning problems with visual and spatial clues. To bridge the gap, we propose a novel dual-modality reasoning framework called Vision-Augmented Prompting (VAP). Upon receiving a textual problem description, VAP automatically synthesizes an image from the visual and spatial clues by utilizing external drawing tools. Subsequently, VAP formulates a chain of thought in both modalities and iteratively refines the synthesized image. Finally, a conclusive reasoning scheme based on self-alignment is proposed for final result generation. Extensive experiments are conducted across four versatile tasks, including solving geometry problems, Sudoku, time series prediction, and travelling salesman problem. The results validated the superiority of VAP over existing LLMs-based reasoning frameworks.

Enhancing LLM Reasoning via Vision-Augmented Prompting
Ziyang Xiao', Dongxiang Zhang" Xiongwei Han', Xiaojin Fu', Wing Yin Yu² Tao Zhong,, Sai Wu'4, Yuan Wang', Jianwei Yin', Gang Chen
Zhejiang University Huawei Noah's 's Ark Lab School Business, Singapore University of Social Sciences Hangzhou High-Tech Tech Zone(Binjiang) Institute of Blockchain and Data Security {xi ixiaoziyang, zhangdongxiang, wusai, cg) edu. cn (hanxi (hanxiongwei, rocket. YuWingYin, Chongtao5)(huaweel. com fuxiao in32@ho tmail. com, Jessi cassica3aa336)m com zjuyjwocs zju. edu.
Abstract
Verbal and visual-spatial information processing are two critical subsystems that activate different brain regions and often collaborate together for cognitive rea- soning. Despite the rapid advancement of L.M-bbssed reasoning. the mainstream frameworks, such as Chain-of- Thouuhh (Co'T) and its variants, primarily focus on the verbal dimension, resulting in limitations in tackling reasoning problems with visual and spatial clues. To bridge the gap, propose novel dual-modality reasoning framework called Asioo-AAummnted Prompting (VAP). Upon receiving textual problem description, VAP automatically synthesizes image from the visual and spatial clues by utilizing external drawing tools. Subsequently, VAP formulates chain of thought in both modalities and iteratively refines the synthe- sized image. Finally, a conclusive reasoning scheme based on self-alignment is proposed for final result generation. Extensive experiments conducted across four versatile tasks, including solving geometry problems, Sudoku, time series prediction, and travelling salesman problem. The results validate the superiority of VAP over existing LLMs-based reasoning frameworks.
Introduction
The human cognitive system is characterized by the presence of two specialized subsystems within the working memory: the phonological loop, which processes verbal information, and the -ssual-spatial sketchpad, which processes visual and spatial information [1]. Both of them play crucial role in problem-solving by offering qualitatively distinct strategies for comprehending and manipulating information [2]. Recently, with the rapid advancement of L.Mss, there have emerged various reasoning frameworks, such as Chain of Thought (CoT) [3], Self-consistent CoT (CoT-SC) [4], and Tree of Thoughts (ToT) [5]. Although these frameworks have shown impressive performance across wide range of NLP tasks, they primarily focus on the verbal dimension with text-only representations, resulting limitations in tasks that require visual and spatial interpretation (e.g., geometry problems or grid puzzles).
In this paper, propose novel dual-modality reasoning approach called Vision-Augment Prompting (VAP) that analogizes human cognition subsystems with the assistance of multimodal large language models (MLLMs). VAP takes textual problems input and uses self-synthesized
"Correspondigg author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Input:
Input:
Input
Input
Inpot
There circle centered (-2.0, -2.0) with radius 2.0.
There is another circle with its center (0.0. 0.0) that through (-2.0, -2.0).
There is a line segment with a slope of -1 that also passes
through point (-2.0, -2.0).
T
10
2
How many intersection points there in this figure?
How many intersection points are there this figure?
Ourput
Output
Output
Output:
Output:
Standard
Chain
Visual-
There is at least intersection point, which is (-2.-2)
According to the figure, there are 5 intersection points.
Prompting
Thought
Augmented
Prompting
Prompting (VAP)
(a) With text-only input
(b) Enhanced with visual input
(CoT)
Figure An example of solving the same Geometry Intersection Counting problem using different types of input. Outputs are derived from GPT-4; for brevity, only the conclusions are presented.
Figure 2: Compare with Standard Prompt- ing and Chain of Thoughts Prompt- ing(CoT).
images as an additional information channel to enhance reasoning. As shown in Figure 1, we employ the state-of-the-art GPT-4V(ision) [6] to solve a geometry problem. In this case, the model accurately deduces the correct answer when given both image and text inputs. In contrast, it generates an ambiguous answer with purely textual information. This example is analogous the process of human cognition where it is a common practice to use images to enhance comprehension when handling geometry problems [7].
Our proposed vision- augmente prompting comprises three steps. Initially, LLMs generates a high- level plan for the following steps, including selecting appropriate drawing toolkit and creating an initial image. To automate the procedure, we leverage the API documentation external tools context of LLM facilitate drawing tool selection and figure synthesis. In the second step. VAP iteratively performs reasoning on the image, updates it, and generates an accompanying textual thought in each iteration. This process results chain of thoughts in both image and text modalities, as illustrated Figure 2. Lastly, the final image and the chain textual thoughts are jointly fed the MLLM derive conclusive answer. To enhance robustness, we introduce technique called self-alignment, where the MLLM describes the image content first, and the image channel is discarded if the self-description fails align with the initial high-level plan.
We conduct extensive experiments across four versatile tasks, among which VAP establishes new state- of-the-art art performance compared to other training- free LLMs-based methods. These four reasoning tasks include: (1) Geometry Intersection Counting in the domain of geometry word problems (+5.0% absolute accuracy gains); (2) Sudoku Puzzle as a logical reasoning task (+12.9%); (3) Time Series Prediction as numerical analysis task (+99.9% in reducing mean absolute error); and (4) Travelling Salesman Problem classical NP-hard problem in the field of operations research (+1 (+1.8% in reducing the optimal gap).
Related Work
2.1 LLMs-based Reasoning
Improving the reasoning capabilities of LLMs such as GPT-4 [8], PaLM2 [9], and LLaMA2 [10] has been a hot topic recent years. To achieve the goal, Chain-of-Thought (CoT) [3] breaks reasoning task into series of intermediate reasoning steps. Self-consistency [4] replaces the naive greedy decoding CoT by sampling a diverse set reasoning paths and selecting the most consistent answer. Tree of Thoughts (ToT) [5] and Graph of Thoughts (GoT) [11] further allow LLMs to explore and combine thoughts in structured manner. Cumulative Reasoning (CR) [12] decomposes reasoning task into smaller components, which are solved in cumulative and iterative manner.

---

Additionally, Chain-of-Experts [13] enhances reasoning capabilities through the collaboration of multiple LLMs.
2.2 Multimodal Large Language Models
The emergence of Multi-modal Large Language Models (MLLMs) such (iso4V( [6] has fostered new research landscape. Although the architecture of GPT-4V is not publicly available, substantial progress has been made in the open-source domain [14, 15, 16, 17]. These works usually fine-tune traditional text-hased LLMs to align with other modalities. Similar approaches are also adopted by multimodal chatbots [18] and multimodal universal task solvers [15, 19] for vision-related
tasks [20, 21, 22].
2.3 MLLMs-based Reasoning
According recent survey [23], reasoning based on MLLMs can be broadly categorized into LLM- aided visual reasoning and multi-modal Shain-of-Thooggee LLM-aided visual reasoning focuses on solving traditional visual reasoning tasks with the assistance of LLMs. This includes tasks such
as visual question answering [24], image segmentation [25], and video question answering [26]. Some works this category, like ViperGPT [27]. VisProg [28], and LLava-Plus [29], also use MLLMs to call external tools for enhanced reasoning, while their task settings differ from ours. the other hand, the multi-modal Chain-of-Thought extends traditional prompting techniques to the multi-modal context. MM-ReAct [30] extends ReAct [31] to support multimodal data. The Visual Chain of Thought (VCoT) [32] uses CoT with vision- saiionlanguage grounding to bridge the gap in multi-step temporal reasoning. Compared with work, VCoT is designed to bridge the logical gaps within sequential data and facilitate temporal reasoning in tasks like visual storytelling and WikiHow
summarization.
Methodology
Mainstream LLMs-based reasoning frameworks, such CoT, ToT, and GoT, only consider the verbal domain L. To tackle more challenging problems with visual and spatial clues, we propose vision-augmented prompting (VAP) and extend the exploration space from single domain L to dual-modality domain L LUG g. where G represents the visual-spatial domain. It relies on the external image synthesis toolkit to automatically draw image matching the text description of the input problem (Section 3.1). The core challenge lies in how to effectively navigate the joint space and maximize the success rate of problem solving, which will be presented Section 3.2.
3.1 External Image Synthesis Toolkit
While visual-spatial information can be beneficial for reasoning demonstrated in Figure 1, LLMs lack the inherent ability to visualize concepts. Therefore, it is necessary to leverage external image synthesis toolkit. Eision-augmented prompting incorporates graphic rendering tools that rely logical programming to render images, including Python Turtle [33] for graphical visualizations, and Matplodib [34] for drawing analytical figures. In addition to these third-party programmable tools, we further integrate image generative models, such as DALL-E 3 [35], to produce images directly from textual prompts. Consequently, we define the set of drawing tools Sy (Turtle, Matplotlib, DALL-E 3). These tools will be utilized by our method through API calls.
3.2 Procedures of Vision-Augmentedd Prompting
We model the problem-solving process of VAP as an iterative reasoning process. With the aid of the image synthesis tools, VAP progressively updates the image according to the instruction provided by the language model. To maintain a coherent reasoning trajectory, a 'thought' is generated on each iteration. Subsequently, to derive the conclusive answer, the final image, the original problem, and trajectory iterative thoughts are sent to the MLLM to obtain the final answer. However, we find it challenging for L.LMs to perform iterative multi-step drawing and reasoning directly due to the lack of global view. To overcome this limitation, we introduce planning step, where L.MMs will create high-level plan for subsequent steps. Furthermore, it is possible that the synthesized image could be

---

Step I: Planning
Step 2: Iterative reasoning
Input:
Input:
There circle contered at (-2, -2) with a radius
You one juring problese (problens aloug with (figure
There a circle contered at (0, O) thas parres through (-2, -2)
Please aoe Pythow eauiptstttl draw nest shupe the feua....
There is line segment with slope I that paszes (-2,-2).
Give your insight about the new Intervcctinns at each map...
How many intersection points are
Iterate 3 times
Output:
High-level Drawing Phaning:
thoughht Outy one shape, with to interratioos."
Tool Selection: This is goometric problem, we uiilize the various shapes.
buJerminate:; Fialer
nurpiotill libcary for abilit
Derutivv step:
Intiulizaiin: we seed to setinng the cocrlinte system, the fox both
(plirindletert 2, 22) 2. FillFaate)
cocediaate should span
Iteration: Proceed with drowing the shapes lerutively, Each sender respective shupes the cocedinate system.
Step 3: Conclusive Reasoning
Input:
There cirefe costrred f-2, -22 with radus of
Ourpat:
Theve cirefe rostered 0j that pazzes thruuhh (-2, -2)
Thace i live segment with stape- of that paisre (+2, -2).
tool: "Pydhu mosploniis" dram_coniest_gf so_ck_u_upppp "Dran shape Riuure
How intersection points there?
Tenjestory shought:
when ierminate:: 'All dlspes are placed or the figwe". thought oach step: "she intesections at each step."
2. There are addittional intersection poists generated.
Only our shope,
intersections.
3. There are additional insersection poists generated.
(Ploner interpoet
lmugg
as -pi subpooos))
siatt-illnt-, glint-6, 2)"
There three shape in the figare: two circles and one line,
self alignment
Output:
From the provided figure, there are interssetionn prints.
Figure 3: Illustration of the workflow in VAP.
disqualified and mislead the reasoning process. To alleviate the negative effect of such images, we introduce a technique named self-alignment to enforce MIL.Ms to literally describe the synthesized image and check whether its text description is aligned with the initial high-level plan. If disparity is detected, we discard the image and restart the reasoning process.
Step I: Planning As demonstratrd in Figure 3, our method takes a textual problem P as input and firstly generates a high-level reasoning plan. In implementation, we first prompt the LLM to acquire a detailed description of the plan in natural language. The output of the plan consists of five key components: (1) an analysis of the problem's 's characteristics for visualization, along with the chosen drawing tool T from the set of available tools Sy; (2) A description of how to initialize the image using the chosen tool, accompanied by the corresponding API call instruction To: (3) the prompt P. that drives the LLM to draw in each iterative step; (4) the prompt P, that drives the LLM to think step by step during the iterative drawing process: (5) the termination condition C' for the iteration. To facilitate subsequent reasoning, we further transform the unstructured textual description of the planning into semi-structured JSON format. This step can be formalized as shown in Equation 1, where F planning denotes the language model involved in this step with prompt engineering. With the instruction Lg, we can create the initial image 90 E G using the selected tool.
(T, lo, Pa, Pr. C} t Fplanning (P,Sr) Sr), 90 < T(Lo)
(I)
Note that both plan generation and ISON format transformation are implemented using LLMs with prompt engineering. Details of the prompt templates can be found in Appendix 1.
Step 2: Iterative reasoning In the iterative reasoning step, the MLLM is specified by two prompts Pa and P. generated in the previous step, along with the image synthesis tool Sy and the assigned termination condition C. We denote the MLLM in this step as P. In each iteration t, the MLLM takes the original problem P. the partially-complee
image Si and trajectory of thoughts Z4 as input. The MILMM will then generate an instruction I, containing API calls used to update the image 9t- Regarding the update of the image, an accompanying 'thought' 2, that provides a textual interpretation of the current state is generated. The process is formally depicted in Equation 2, with details of the prompt template presented in Appendix 1.2. Here, Ze starts as an empty array and the new 'thought' 2; is appended to the trajectory at each iteration.
{z, I+) + F(PP.P..Sr..)) (P, P9, 2i), Zot1 + Z, U (ze) 9441 + T(I))

---

We observe that the MLLM occasionally failed to follow the instructions provided in the prompt (e.g.. repeatedly drawing the same shape), causing disruption the entire iterative process. To address this issue and improve the stability of reasoning process, we enrich the context of MI.LM by appending the input and output from the last step, which serve as illustrative examples for the MILMM to better follow the instructions.
Step 3: Conclusive reasoning When the iterative reasoning terminates (i.e., condition C' is met), we proceed to the final step of conclusive reasoning using the synthesized image of step 2, denoted by 9n- Since we cannot guarantee 9n is a perfect output and a disqualified In may mislead the conclusive reasoning process, we introduce technique named self-alignment to enforce MLLMs to literally describe the synthesized image and check whether its text description is aligned with the initial high-level plan. If disparity is detected, we discard the image and restart the reasoning process. a qualified image cannot be obtained after a certain number of trials, VAP is degraded to simple input-output reasoning, without leveraging the visual channel.
As shown Equation 3, our conclusive reasoning takes In. the trajectory of thoughts Zn, along with the original problem P as input and leverage the prompt template presented in Appendix A.1.3 to derive the final answer A.
A t F Faelf- aligmment (P, In, Zn.
4 Experiments
We evaluate VAP using four diversified and challenging tasks, including Geometry Intersection Count, Sudoku Puzzle, Time Series Prediction, and Travelling Salesman Problem. We also provide typical input-output example for each task in Appendix A.2.
These tasks share three -LM-based reasoning baselines, including standard prompting, chain-of- thought (CoT) prompting, and CoT with self-consistency (CoT-SC) prompting [4].
For CoT prompting, the reasoning process involves multiple intermediate steps. In the Geometry Intersection Counting task, we define each intermediate step as calculating the numher of intersections between a pair of shapes. In the Sudoku Puzzle task, each step involves considering a position that violates the Sudoku rules. For Time Series Prediction and Travelling Salesman Problem, it is challenging to define specific step content. Therefore, we append a 'think step by step" instruction to the standard prompt and include an additional step to extract the solution from the generated thoughts.
Additionally, for CoT-SC prompting, when the task output is discrete (i.e., Geometry Intersection Counting, Sudoku Puzzle, and Travelling Salesman Problem), we sample & answers and use the majority answer. When the task output continuous (i.e., numeric value in the task of Time Series Prediction), we generate predictions and calculate their average as the final forecast value. The default & is set to 10.
For fairness, employ the 'GPT-4-isionn-prewee and our VAP, The default temperature set to O. For methods that require sampling, such as SC, the temperature is set to 0.7.
the underlying MLLM for these baselines
4.1 Task 1: Geometry Intersection Counting
Task Description The task determines the number of intersection points between a couple of geometric shapes described in natural language. For example, given the input "There a line segment from (-2.5, -1) to (-0.5, -1 1) and circle centered at (-1.5. -1) with radius 1," the correct output should be "2", as there are 2 intersection points between the circle and the line segment.
Task Setup We randomly sample 200 problem instances from Geometry Intersection Counting task in the BIG-bench benchmark" [36] The performance metric used in this task is accuracy, which defined as the percentage of problem instances in which the output number matches the ground truth.
httpps://itthu.. httpeittuu..coo/eeeeeeeeeeeeeeee Btmm//oogl//BI-bbeeno http://teeemmmmiiiieeeeeeee intersect_geometty
5

---

-pec-ip Baselines We introduce two additional baselines specifically developed for solving geometry problems. The first is Inter-GPS [37]. a symbolic reasoner [38] for geometry problems. The second is G-LLaVA [39], MILMM specialized trained on geometry problems. Since geometry figures are not available the problem input, G-LLaVA only leverages the textual modality for reasoning.
0.5
Table 1: Intersection Point Result.
Methods
Method
Accuracy
Accuracy
Standard
Standard
8.5%
0.3
CoT
CoT
10.0%
CoT-SC(k=5)
CoT-SC (km=5)
11.0%
CoT-SC(K=10)
CoT-SCCk==10) (k=10)
11.5%
CoT-SC(K=20)
(AT-SC(A:=00)
11.5%
Inter-GPS
6.0%
4
VAP
G-LLaVA
14.0%
Number of Shapes
VAP
16.5%
Figure Accuracy with different number of shapes.
Results As shown in Table 1, Inter-GPS and G-LLaVA exhibit low accuracy, primarily because they are originally trained on dual-modality datasets and thus show limited effectiveness with text- only input. We also examine their performance when provided with an additional image input in Section 4.5, where the purpose is to validate the effectiveness of the synthesized images by our VAP algorithm. Among the general-purpose reasoning methods, standard prompting and CoT prompting yield relatively low accuracy (8.5% and 10.0% respectively). The CoT-SC method, even with a sample size of 20, only marginally improves accuracy by 1.5%. In contrast, VAP significantly outperforms these methods and achieves an accuracy 16.5%. Furthermore, in the break-down analysis with increasing number of shapes, illustrated in Figure 4, VAP demonstrates clear superiority over other LLM-based methods in more complex scenarios involving four or more shapes. We can see that the accuracy of all baselines is almost close to 0 in these complex scenarios.
4.2
Task 2: Sudoku Puzzle
Task Description this task, an initial state Sudoku board is presented in natural language, with digits filled cells and dots denoting empty cells. For example, in 9 x board, row "......... only contains digit 6 in the third column. The reasoning process iteratively provides the information of the next action, in the form of "x y digit", where 'x' and 'y' specify the cell's coordinates on the board, and 'digit' is the number to be placed in that cell.
Task Setup We utilize the Sudoku puzzle generation program from the BIG-bench" to create dataset that includes 150 Sudoku puzzles, For performance evaluation, we employ correct rate and collision rate as two metrics. The correct rate measures the accuracy in solving the puzzles, and collision rate assesses the frequency at which the position in given command is already filled with numbers, which indicates a violation of Sudoku rules.
Task-specific Baselines For a more comprehensive evaluation, we also incorporate Tree of Thought (ToT) [40] as an additional baseline. The implementation of ToT this task is not troublesome because it is straightforward to decompose the thought process into a tree-structured representation for board games.
Results In Table 2, the standard prompting shows low success rate of 18.0%, with a high rate of rule violations. This can be attributed to Sudoku's demands for reasoning and rule comprehension, which pose challenge for LLMs relying solely prompt engincering. The CoT and CoT-SC methods show improved performance, as they can, to certain extent, enhance the coherence LLM reasoning by providing reference of previous steps. The ToT algorithm is the most effective among these baselines, achieving 22.6% success rate, owing to its tree-structured thought process. Notably,
phttps://gitaee http://ooooooooooooo/////////e/a http://bggppcccbbbbcnnccar.. http:///nccaaak..assseeeeeeeee casks/sudoku

---

VAP outperforms all other methods with correct rate of 35.5%. VAP also reduces the collision rate, implying that integrating the image modality is helpful for LLMs understand game rules.
767
800
Table 2: Sudoku Result.
Methods
617
Method
Correct rate
Collision rate.
556
SMA
Standard
Standard
18.0%
68.7%
400
Col
CoT
17.3%
64.7%
CoT-SC
COT--CC(K=5) (A=5)
20.6%
48.7%
LiMeTme
CoT-SC(k==10)
20.6%
47.3%
VAP-LLMTime
ToT
22.6%
44.0%
Figure Time series prediction results, with y-axis indicating MAE.
0
VAP 35.5%
26.0%
4.3 Task 3: Time Series Prediction
Task Description To evaluate the numerical analysis capabilities, we regard the LLM as a time series predictor. In this task, sequence of data points is provided, and the objective for the LLM is to predict the next values in the series.
Task Setup The dataset for this task is sourced from the Darts library [41], which includes curated collection of 8 real univariate time series datasets. We configure the window size to 100 data points and predict the subsequent Absolute Error (MAB) metric.
values in the series. The performance is evaluated using the Mean
Task-specific Baselines We chose the LLMTime algorithm [42] as the most cutting-edge method for comparison, where a data rescaling strategy and tokenization trick is introduced to enhance numerical precision. For instance, the sequence "8.05, 1, 35" will be tokenized as "8 0.5 0 0 0". We also adopt the simple moving average (SMA) traditional statistical baseline for time series forecasting.
Results As shown in Figure 5, SMA is recognized as a traditionally effective method for time series prediction, surpassing most LLM--ased methods with an MAE of 611. Among the L.M--based approaches, compared to standard prompting, CoT-SC approach significantly reduces the MAE from 767 675, justifying the effectiveness of integrating auto-regression and sample strategy. is noteworthy that LLM'Tim, performs effectively. Its performance even marginally exceeds that of SMA. outperforms the majority of LLM-based methods and is slightly inferior to LLMTime. However, considering that VAP is orthogonal to LLMTime, we introduce a new method that combines the same preprocessing technique with VAP. The results show that this combined version, VAP+LLMTime, outperforms all comparison algorithms.
preprocessing technique for L.MSs to handle sequential data,
the result, VAP
4.4 Task 4: Travelling Salesman Problem
Next, we test our method on travelling salesman problem (TSP), NP-hard combinatorial optimiza- tion problem that poses challenges for neural network solvers [43]. The input for TSP instances consists of set of coordinates, and the output is sequence of city indices representing the route for salesman.
Task Setup We use Fuclidean TSP instances with 10 and 20 cities as our testset. For each city size, 100 instances are generated by program, with coordinates uniformly distributed within a [0, 100]2 integer grid. The average path length and the gap from the optimal solution are used as metrics.
Task-specific Baselines We introduce four traditional TSP solvers as our task-specific baselines: Gurobi, an exact TSP solver using mixed integer programming: Nearest neighbour (NN) algorithm, which greedily selects the closest city; Fastest insertion (FT), an efficient insertion method; and Random baseline that chooses paths randomly for performance benchmarking, Here, note that the
7

---

output of LLM-based methods (i.e., Standard, CoT, CoT-SC, and VAP) might violate constraints of TSP generating duplicate cities). In such cases, we randomly select an unvisited city continue the solution.
Table 3: TSP task performance.
Traditional
LLMs-based
Method
Gurobi
Random
NN
FI
Standard
CoT
CoT-SC
VAP
N=20 N=10
Length
294.8
529.6
327.4
306.8
327.5
327.2
325.0
312.4
Gap
0.0%
79.7%
11.1%
4.1%
11.1%
11.0%
10.4%
6.0%
Length
383.7
989.5
448.2
424.2
544.5
547.2
541.9
499.3
Gap
0.0%
170.7%
16.8%
10.6%
41.9%
42.6%
41.2%
30.2%
Results As shown in Table 3, the performance of standard prompting is comparable to the Nearest Neighbour algorithm when applied to TSP with 10 cities. CoT demonstrates very slight improvement standard prompting. This might be attributed the TSP's requirement for heuristic and complex thought process, which is challenging for CoT to replicate. VAP outperforms all LLM--aaedd algorithms, achieving 6.0% gap for optimality, close to the Fastest Insertion's performance. It's worth noting that when the number of cities a problem instance increases to 20, the performance of LLM-based algorithms significantly decreases as the search space of TSP grows exponentially with the problem size. Such huge search space poses a significant challenge for LLMs not specifically tailored solve TSP. Nonetheless, VAP's superiority over other LLM-based methods still evident when the problem size increases.
4.5 Effectiveness of the Synthesized Images
To demonstrate quality of synthesized images, we first present several illustrative examples in Figure and then provide numerical experiments in the following.
70
SIX
60
40
30
20
20
60
(a) Geometry example
(b) Sudoku example
(c) Time series example
(d) TSP example
Figure Examples images synthesized by our VAP among the four tasks.
In the first experiment evaluate the quality of synthesized images, we use the rate of integrity as the metric, which refers to whether the generated image contains all the elements described in the original problem. This a relatively objective metric for human annotators to reach consensus. shown in Table 4, the images synthesized by VAP demonstrate a high integrity rate across various tasks. Notably, for Time Series Prediction and TSP, the module achieves an impressive integrity rate of 100% and 98.0%. respectively. We also evaluate the performance improvement of VAP when provided with the ground truth image. As shown in Table 4, the column 'Wit ground truth image' refers to the performance boost rate. If the drawing errors had been corrected, we observe notable potential for improvement. These results underscore the crucial role of accurate image rendering in enhancing the final output quality of the VAP.
In the next experiment, we provide the synthesized images by our VAP as additional input for the geometry solvers Inter-GPS and G-LLaVA involved in task of Geometry Intersection Counting. As shown in Table 5, these two approaches can greatly benefit from our synthesized images. Their remarkable accuracy boosting validates the effectiveness of the images generated by VAP.

---

Table 4: Performance of Drawing Module.
Table 5: Performance of dedicated ge- ometry solvers with (denoted by 1) and without synthesized images.
Task
Integrity
With ground truth image
Geometry
83.5%
5.8% (17.0% + 18.0%)
Method
Accuracy
Accuracy
Sudoku
89.3%
5.1% (35.5% +3733%
Time Series
100.0%
Inter-GPS 6.0%
17.0%
TSP
98.0%
1.3% (30.2% 30.6%)
G-LLaVA
14.0%
20.5%
5
Ablation Study
In the ablation study, evaluate the effectiveness of three steps, including high-level planning, iterative reasoning and self-alignment conclusive reasoning. Note that removing the planning step refers to prompting the LLM to draw the image step-by-step without providing a plan in advance. Removing iterative reasoning refers to directly prompting the LLM to generate an entire image and then proceeding to the conclusive reasoning step without intermediate results.
As shown in Table 6, we report the core metric for four tasks: accuracy for Geometry Intersection Problems, correct rate for Sudoku puzzles, MSE for Time Series Prediction, and average tour length for the TSP with 10 cities. The results demonstrate that removing any of the key components causes performance degradation, except for self-alignment in Time Series Prediction. This is because there is no error in the drawing module, as discussed in Subsection 4.5. Hence, in this scenario, self-alignment plays no effect. Interestingly, removing planning step causes the most significant performance drop for Geometry Intersection Problems and TSP. The reason is that planning is crucial for visualizing problems requiring precise spatial relationships. Furthermore, iterative reasoning proves to be an essential component, as its removal leads to considerable performance degradation across all tasks. This finding highlights the importance of view that all mooblem-solving is step-by-step process. Self-alignment plays an important role in the task Sudoku. For example, the I.M will occasionally draw the incorrect coordinate range for the Sudoku board, resulting in board with the wrong size. However, with self-alignment, the model first describes the board size in the figure and cross-checks this against the original plan, preventing such issues.
Table 6: Ablation study results. Each column represents task and the cell indicates the performance when removing a module in VAP.
Geometry
Sudoku
Time Series
TSP (N=10)
VAP
16.5%
35.5%
556
312.4
w/o planning
12.0%
24.7%
578
322.0
w/o iterative
14.5%
19.9%
582
321.3
w/o self-alignment
16.0%
25.1%
556
315.9
6 Limitations of VAP
Even though VAP has been shown to outperform other LLM-based approaches in a set of versatile tasks, there still exists a noticeable performance gap between VAP and task-specific approaches. However, these tailored approaches require nontrivial efforts for domain customization and is not able to support other types of reasoning tasks. In contrast, VAP is a lightweight and training- framework. With negligible customization cost, VAP is general enough to handle a spectrum of complex reasoning tasks with visual and spatial clues.
The other limitation is the black-box functionality of VAP, even though the framework is inspired by the cooperation of two subsystems different regions of human brain that cooperate with each other. In Appendix A.3, we provide two case studies as examples to provide certain insights and justify the effectiveness of VAP. would an interesting future research direction to explore the interpretability of VAP and its relationship with cognitive science.

---

7 Conclusion
In this paper, we introduced a novel reasoning framework called Vision-Augmented Prompting (VAP), designed to enhance the reasoning capabiliiies of large language models (LLMs) by emulating the human cognitive system's ' dual modality processing. VAP leverages the external image synthesis tools to generate visual representations that augment the textual input. The procedure of VAP follows a three-step algorithm: first, automatically generates a high-level plan for reasoning; second, engages in iterative drawing and reasoning based on the partiall-completer image; and finally, original problem, all thoughts and generated image are jointly to derive a solution. To enhance robustness, we introduced a self-alignment technique, where the MLLM describes the image content, and the image channel is discarded if the self-description fails to align with the initial high-level plan. We conducted extensive experiments across four diverse reasoning tasks: intersection counting in Geometry Intersection Problems, Sudoku Puzzles, Time Series Prediction, and the Travelling Salesman Problem. The results demonstrated the effectiveness of VAP, establishing new state-of-the- art performance compared to other training-free LLM-based methods.
Acknowledgments and Disclosure of Funding
This work supported by the National Key Research and Development Project of China (2022YF00002000) and the Fundamental Research Funds for the Central Universities (226-2024. 00145 and 226-2024-00216).
References
[1] Alan Baddeley. Working memory. Science, 255(5044):556 1992.
[2] Mary Hegarty Maria Kozhevnikov and Richard E. Mayer. Revising the visualizer-verbalizer dimension: Evidence for two types of visualizers. Cognition and Instruction, 20(1):47-77, 2002.
[3] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of of-tuuugttthht prompting elicits reasoning in large language models. In NeurlPS, 2022.
[4] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistnney improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[5] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. CoRR, abs/2305.10601, 2023.
[6] Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung- Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of Imms: Preliminary explorations with gpt-iv(ision), 2023.
[7] Judith E Fan, Wilma Bainbridge, Rebecca Chamberlain, and Jeffrey D Wammes. Drawing as versatile cognitive tool. Nature Reviews Psychology, 2(9):556-568 2023.
[8] OpenAl. Gpt-4 technical report, 2023.
[9] Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier- Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Srooquette-hhor Aakanksha Chowdhery, Clément Crepy. Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas
10

---

Gonzalez, Guy Gur-Ari Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui. Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lann Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Mísra, Maysam Moussalem, Zachary Nado. John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang. Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report, 2023.
[10]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai La, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang. Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliaan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
[11] Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. Graph of thoughts: Solving elaborate problems with large language models. CoRR, abs/2308.09687. 2023.
[12] Yifan Zhang, Jingqin Yang. Yang Yuan, and Andrew Chi-Chih Yao. Cumulative reasoning with large language models. CoRR, fbss/2308.0337. 2023.
[13) Ziyang Xiao, Dongxiang Zhang, Yangjun Wu, Lilin Xu, Yuan Jessica Wang, Xiongwei Han, Xiaojin Fu, Tao Zhong, Jia Zeng, Mingli Song. and Gang Chen. Chain-of-experts: When Ilms meet complex operations research problems. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-1,, 2024. OpenReview.net, 2024.
[14] Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. CoRR, 044/2304008885 2023.
[15] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung. and Steven C. H. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. CoRR, 055/2305.06500 2023.
[16] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. CoRR, 1es/2303.16199, 2023.
[17] Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: vision and language model for dialogue with humans. CoRR, abs/2305.04490. 2023.
[18] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning, 2023.
11

---

[19] Wenhai Wang, Zhe Chen. Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, and Jifeng Dai. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. CoRR, abs/2305. 11175, 2023.
[20] Wentao Tan, Changxing Ding, Jiayu Jiang, Fei Wang. Yibing Zhan, and Dapeng Tao. Harnessing the power of milms for transferable text-to-image person reid. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 17127-17137. IEEE, 2024.
[21] Zepeng Li, Dongxiang Zhang, Yanyan Shen, and Gang Chen. Kluman-in-the-looee vehicle reid. In Brian Williams, Yiling Chen, and Jennifer Neville, editors, Thirty-Sevents AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023. Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pages 6048-6055 AAAI Press, 2023.
[22] Zepeng Li, Dongxiang Zhang. Sai Wu, Mingli Song, and Gang Chen. Sampling-resilient resilient multi-object tracking. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 3297-3305. AAAT Press, 2024.
[23] Shukang Yin, Chaoyou Fu, Sirui Zhao Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306. 2023.
[24] Haoxuan You. Rui Sun, Zhecan Wang, Long Chen, Gengyu Wang, Hammad A. Ayyubi, Kai-Wei Chang, and Shih-Fu Chang. Idealgpt: Iteratively decomposing vision and language reasoning via large language models, 2023.
[25] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li. Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint arXiv:2308.0m 00692, 2023.
[26] Antoine Yang, Antoine Miech, Josef Sivie, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[27] Didac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: for reasoning. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 11854-11864, IEEE, 2023.
Visual inference via python execution
[28] Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training, 2022.
[29] Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang, Jianfeng Gao, and Chunyuan Li. Llava-plus: Learning to use tools for creating multimodal agents. CORR, abs/2311.004377 2023.
[30] Zhengyuan Yang. Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. MM-REACT: prompting chatgpt for multimodal reasoning and action. CoRR, abs/2303.1 1181, 2023.
[31] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5. 2023. OpenReview.net, 2023.
[32] Daniel Rose, Vaishnavi Himakunthala, Andy Ouyang, Ryan He, Alex Mei, Yujic Lu, Michael Saxon, Chinmay Sonar, Diba Mirza, and William Yang Wang. Visual chain of thought: Bridging logical gaps with multimodal infillings, 2023.
[33] Ram Rachum, Peter Bittner, et al. Pythonturtle: A learning environment for python suited for beginners and children, inspired by logo, may 2009.
12

---

[34] J. D. Hunter. Matplotii: A 2d graphics environment. Computing in Science & Engineering. 9(3):90-95) 95, 2007.
[35] Zhan Shi, Xu Zhou, Xipeng Qiu, and Xiaodan Zhu, Improving image captioning with better use of captions, 2020.
[36] BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023.
[37] Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Anmual Meeting of the Association for Computational Linguisties and the 11th International Joint Conference on Natural Language Processing, ACI/IJCNLP 2021, (Volume I: Long Pupers), Virtual Event, August 1-6, 2021, pages 6774-6786 Association for Computational Linguistics, 2021.
[38] Ziyang Xiao and Dongxiang Zhang. A deep reinforcement learning agent for geometry online tutoring. Knowl. Inf. Syst., 65(4):1611-1625, 2023.
[39] Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye. Wanjun Zhong, Yufei Wang, Lanqing Hong. Jianhua Han, Hang Xu, Zhenguo Li, and Lingpeng Kong. G-llava: Solving geometric problem with multi-modal large language model, 2023.
[40] Jieyi Long. Large language model guided tree-of-thought, 2023.
[41] Julien Herzen, Francesco Lässig, Samuele Giuliano Piazzetta, Thomas Neuer, Léo Tafti, Guil- laume Raille, Tomas Van Pottelbergh, Marek Pasieka, Andrzej Skrodzki, Nicolas Huguenin, Maxime Dumonal, Jan Koscisz, Dennis Bader, Frédérick Gusset, Mounir Benheddi, Camila Williamson, Michal Kosinski, Matej Petrik, and Gaël Grosch. Darts: User-friendly modern machine learning for time series. J. Mach. Learn. Res., 23: 13:12:11124:6. 2022.
[42] Nate Gruver, Mare Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language models are zero-shot time series forecasters. CoRR, abs/2310.07820, 2023.
[43] Dongxiang Zhang, Ziyang Xiao, Yuan Wang, Mingli Song, and Gang Chen. Neural TSP solver with progressive distillation. In Brian Williams, Yiling Chen, and Jennifer Neville, editors, Seirtt-Seventhh AAAI Conference Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pages 12147-12154, AAAI Press, 2023.
[44] Richard C Larson and Amedeo R Odoni. Urban operations research. Number Monograph. 1981.
A
Appendix
A.1
Prompt Engineering of VAP
A.1.1 Planning
The planning step's prompt is divided into two sub-steps. First, high-level plan's textual repre- sentation is generated using the following prompt. Here, the '(examples)' and "(problem)' are placeholders for the few-shot examples (which will be discussed later) and the problem input, respectively.
13

---

Your role is to visualize problem by creating an image that represents the problem description accurately using a specific tool. This task involves drawing an image that encapsulates all the details provided in the problem description.
The drawing will be executed through an iterative process. This means you will develop the image a step-by-step manner, ensuring that each element of the problem is represented accurately.
Before beginning the drawing, you are required to outline a high-level plan for how you will approach the drawing process. This plan should cover three essential aspects:
1. Tool Selection The choice of software or tool you will use for drawing. Here are drawing tools available:
Python Matplotlib: a powerful Python library used for creating a wide variety of static, animated, and interactive visualizations. It is widely utilized in data science and engineering for generating high-quality plots and graphs.
//=== Continued from previous page ===
Turtle: Python library that provides simple way to draw graphics and shapes using virtual "turtle" that moves around the screen. Inspired by the Logo programming language, it an excellent tool for teaching programming concepts through visual feedback.
-DALLE3: an advanced image generation model, capable of creating detailed and imaginative images from textual descriptions. It leverages deep learning to understand and produce highly realistic or fantastical scenes based on user prompts.
Initialization Approach: How you will begin your drawing, focusing on setting up the initial state of the image.
3. Iterative Drawing Approach: A detailed description of how you will iteratively add details the image step by step.
Output Format: Your plan should organized into three distinct paragraphs, each starting with the respective headings: Tool selection:, Initialization;, and Iteration:. [examples]
Problem Description: (problem)
The few-shot examples are optional to improve the LLM's ability to handle various tasks. The planning step plays crucial role in the overall reasoning process, therefor, use one-shot example as our default setting (the same as other baselines for fairness). For the planning step, we use several pairs of problem input and textual high-level plan as examples. For other standard prompting, we use the problem input and answer output as examples. For CoT prompting and CoT-SC prompting, use problem input, trajectory of thoughts, and answer output examples.
Subsequently, semi-structured plan JSON format extracted using the following prompt. Here, (drawing_plass plan) is the placeholder of the high-level plan generated before.
You are tasked with visualizing problems by creating images. Each image should be constructed step by step, based on a detailed plan you't ve previously prepared. You should first review your plan, and start by examining the high-level plan you't 've made for the iterative drawing process.
Your plan as following: (drawing_plan)
Your task is to extract meta-information from your drawing plan and format it JSON string. Follow this strict JSON structure in your output:
"tool": "Name of the tool you're re using",
"draw_conters _aceeehh__hhaaaeeeeeeeeeeeeeee Lar___ahh____eeeee sttpssstep""" "Describe what you'll draw in each step", "when terminate". "Criteria for completing the drawing". "thought at each step": "Your thought process each step". "initialization": "Initial setup before starting the drawing"
PLEASE ENSURE YOUR ISON OUTPUT IS CORRECTLY FORMATTED! Avoid including extraneous words or characters outside the JSON structure.
14

---

A.1.2 Iterative reasoning
The prompt for the iterative reasoning step is as follows. Please note that current cutting- MLLMs cannot embed images arbitrary positions of prompt. Therefore, provide the image in context and refer to it in prompt description accordingly.
You are problem visualizer, tasked with drawing an image based problem description using a specified tool.
You will draw the image accurately, ensuring that all information in the problem description is included the image. This process will be done iteratively. Each iteration step will involve drawing specific content of the figure using the tool.
The problem description is as follow: (problem) Here the partially completed figure given in context. (figure.png). Please refer it during your thought process and image updates.
//=== Continued from previous page
For each step, provide your thoughts (thought_at aa__achh__________aa se__ec_shttp____________
Next, update the image content (draw_cod hewe_wotten__toe of _ff_each____ _e__ch_sttepp using API calls with the tool tool.
Output Format:
Your output should be a JSON string. Strictly follow the JSON format provided below.
"thought"" "Your thought on this iteration". "is terminal": false, // true this the last iteration, otherwise false. The terminition condition is [when_[teee (erminate) "iterative _ereeeedddweeeeee >_ttsssst""""_t "Description of what was drawn modified in this step"
PLEASE ENSURE YOUR JSON OUTPUT IS CORRECTLY FORMATTED! include any extra words or comments.
A.1.3
Conclusive reasoning
First, we apply self-alignment by prompting the MLLM describe the content of the image.
Please describe the details the given image accurately and comprchensively. Include all visible objects and elements without omitting any details, and avoid adding any imaginary non-existent objects. Ensure that the description is as detailed as possible, faithfully capturing the essence of the image provided. Here the given image (image.png: Your description is follow:
Next, the MLLM prompted to function as binary classifier and is tasked with checking whether the description aligns with the initial high-level plan. The prompt is shown as follow.
You are artist who visualizes problems.
You have made a plan for your drawing: [drawing_plan]
The final content you drew is described
(self description)
Now, you should determine whether the description aligns with your initial drawing plan. Pay close attention to the details in both.
Your output should be a single word: 'true' if they align, and "false" if they do not. Do not provide any additional comments explanations.
Here is your answer:
The prompt of conclusive reasoning step is follow.
15

---

Here the problem: (problem)
Its visualization is given in context. (figure.png)
The description the image is as follows: (seif_ _ededddeccpptonn The trajectory of thoughts is as follows: fnhouugts_trajeccun Provide your final answer:
A.1.4 Prompt used ablation study
In the ablation study, we investigate the impact of the planning step by removing it from the process. The removal of the planning step results in the loss of access to meta-information, such the selected tool, which can no longer be incorporated into the iterative reasoning prompt template. To address this issue, employ alternative prompt template specifically designed for iterative reasoning in the absence of the planning step. This alternative template allows for the continuation of the iterative reasoning process without relying the meta-information that would have been provided by the planning step. The specific prompt we used is shown as follow.
You are a problem visualizer, tasked with drawing an image based on a problem description using a specified tool.
You will draw the image accurately, ensuring that all information in the problem description included in the image. This process will be done iteratively. Each iteration step will involve drawing a specific content of the figure using the tool.
The problem description is as follow: (problem)
Here is the partially completed figure given in context. (igure.pnn), Please refer to it during your thought process and image updates.
For each step, provide your thoughts according to this problem.
Next, update the image content according to this problem using Python API calls.
Output Format: Your output should be a JSON string. Strietly follow the ISON format provided below.
"thought": "Your thought on this iteration", "is "ss_temmnnal": false, // true if this is the last iteration, otherwise false "eerative_draw_step""" "Description
what was drawn or modified in this step"
PLEASE ENSURE YOUR JSON OUTPUT IS CORRECTLY FORMATTED! Do not include any extra words or comments.
A.2
Examples of Four Experimental Tasks
This section presents the onput-output examples for the four tasks involved in the experiments.
A.2.1 Geometry Intersection Counting
Input: There is a circle centered at (-1.5, -1.0) with radius 3.0. There is polygon with coordinates [(-2.2, 4.0). (-3.2, -0.4), (2.4, -2.6), (3.8, 4.1)]. There is a line segment from (0.9. 3.3) to (-1.3, 3.4). How many intersection points are there?
Output: 2
A.2.2 Sudoko Puzzle
Input: Sudoku puzzle Fill the dots with digits "[1-4]". Digits cannot repeat in the same row, column or 2x2cell. Each cell is separated by spaces or empty lines. Specify the new digits with command: "ex> <y> <digit>", ", removing the quotes. The top left corner has coordinates "I !. For example the command to add the digit "4" to the bottom left corner is "I 4.".
Pleuse pay attention the information given in the image, especially the positions of the X-axis and Y-axis.
Board:
16

---

2431
.3
.1
.2 1.
Command:
Output:
A.2.3 Time Series Prediction
Input:
You time predictor. The user will provide sequence and you will predict the value(only one value needed). The sequence is represented by decimal strings separated by commus.
Please continue the following sequence without producing any additional text. Do not say anything like 'the next value in the sequence are', just return the numbers. Sequence:
432.0,399.0,4406.,,52266.
Output: 438.0
A.2.4 Travelling Salesman Problem
Input: You are an TSP solver. Solve a TSP instance:
Here 10 point in city, each point is represented pair numbers, indicating its coordinates in the specified space. The coordinations are as following:
O: (0.9371, 0. 1482)
I: (0.6345, 0.2510)
2: (0.2628, 0.7120)
3: (0.1545, 0.9067)
4: (0.2827, 0.808I)
(0.9533, 0.9086)
6: (0.4199, 0.7617)
7: (0.6315, 0.0414)
8: (0.8694, 0.5878)
9: (0.7709, 0.7211)
What is the shortest possible route that visits each city exactly once and returns to the origin city?
Your answer should be the permutation of all city index (separated by comma). The start point Please give your answer directly, don't II use any other tools.
Output: 0.7.1.3.....26.4..3
17

---

A.3 Interpretability of VAP
To better understand how VAP facilitates problem-solvingg we conduct two experimental studies to gain deeper insight.
012.0. 118.0, 132.0, 129.0, 121.0,
Original Problew
Without an analytical
148.0, 148.0, 136.0, 119.0,
tool, it's hard observe
104.01 118.0 115.0. 126.0....
accurate periodic patterns.
300
Has periodicity?
Period ?
40
GPT: The periodic puttern in the data appears to have cycle that repeats approximately every 15
300
Imagge
(a) Solution given by Standard Prompting
(b) Solution given by
100
80
100
VAP
units the
VAP
Figure 7: Detect periodicity with and without VAP.
Figure 8: A case study of TSP task.
Firstly, we focus on Time Series Prediction task. Figure shows the Air Passengers Dataset from Darts. On the left, we have the textual representation of the sequence alongside an image drawn by VAP. We ask GPT-4V to model the periodicity of this sequence, which is an important factor in Time Series Prediction. Despite the clear periodic pattern in this sequence, GPT-4V struggles to recognize this periodicity when provided only with the textual problem. In contrast, when presented with the image, GPT-4V could easily identify the sequence's periodicity. This study demonstrates how VAP can provide a more efficient and clear information format, thereby enhancing the model's reasoning ability on prediction tasks.
The second study focuses on the TSP, where we visualize two solutions for the same TSP instance in Figure 8. One solution generated using standard prompting, while the other is given by VAP. The comparison of the two solutions reveals that the standard prompting results in numerous crossing paths, which is proven to be suboptimal in Euclidean TSP [44]. In contrast, the VAP solution has few crossing paths. This improvement is likely due to the clear visualization of visited partial paths, which provides the model with heuristic cues easily exclude bad choices.
A.4 More Experimental Results
A.4.1 Effiency of VAP
Rregarding the computational efficiency of VAP, we conduct an experiments comparing the time consumption and accuracy of various methods across geometry and Sudoku tasks. Table presents the results of this analysis.
Table 7: Efficiency and Accuracy Comparison
Geometry
Sudoku
Method
Time Usage
Accuracy
Time Usage
Correct Rate
Standard
0,2
8.5%
0.3 s
18.0%
CoT
0.5
10.0%
0.8 s
17.3%
CoT-SC (k=5)
2.3 s
11.0%
4.1 s
20.6%
CoT-SC (k=10)
4.5 s
11.5%
8.8 s
20.6%
ToT (n_ (n_chillren==))
9.0
22.6%
VAP
4.1
16.5%
9.5
35.5 %
Although VAP is computationally intensive, its time usage comparable ToT and CoT-SC (k=10) in the Sudoku task. Interestingly, for the simpler geometry task, VAP demonstrates faster performance than CoT-SC (k=10). This efficiency can attributed VAP's unique structure: despite its large context (including tool instructions, thought trajectory, and image encoding), the output is concise, consisting primarily API calls and immediate thoughts. This leads to rapid inference during each step, we observed that time usage is predominantly influenced by the number of decoded tokens. Given VAP's superior effectiveness on these tasks, we posit that the observed performance gap in computational efficiency is acceptable.
18

---

A.4.2 Different Foundational Models
Our experimental setup employs unified VLLM. However, observed that VAP necessitates MLLM to process visual-text imput, whereas other baselines solely require textual input. Considering that traditional LLMs are expected to outperform MILMs in text-only tasks, introduced GPT-4 and LLaMA 8B as additional LLMs ensure a more comprehensive and equitable comparison. Table 8 presents the results of this expanded analysis on geometry intersection counting task.
Table 8: Performance on Geometry Task over Different Foundational Models
Method
Accuracy (GPT-4v) Accuracy (GPT-4) Accuracy (LLaMA 3)
Standard
8.5%
10.0%
7.0%
CoT
10.0%
11.0%
8,0%
CoT-SC (k= 5)
11.0%
11.5%
8,0%
CoT-SC (k= 10)
11.5%
11.5%
8.0%
CoT-SC (k = 20)
11.5%
11.5%
8.0%
VAP
16.5% %
The findings reveal several noteworthy points. First, GPT-4 demonstrated marginal improvement in baseline performance compared to GPT-4v, albeit still significantly lower than VAP. Conversely, LLaMA 3 8B exhibited reduced accuracy, which can be attributed to its smaller model size potentially limiting its generalization capabilities for this particular reasoning task. While larger variants of LLaMA 3 might yield superior results, current hardware constraints precluded their evaluation. Interestingly, simpler methods such as standard prompting show greater sensitivity to model variations compared to more complex approaches like CoT with CoT-SC. Despite these differences, believe that the fundamental conclusions of our study remain valid.
19

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiiaton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We have made main claims that accurately reflect the paper's contributions and scope in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
20

---

Justification:
The paper discuss the limitations of this work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sellispecifiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include the content of the theoretical analysis results that require assumptions or proof.
Guidelines:
The answer NA means that the paper does not include theoretical Results
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The paper provides all the information needed to reproduce the main experi- mental results.
Guidelines:
21

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the Results
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We have provided the experimental code and test data facilitate reproduction.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
ooseeeeSuuumsssioooooooocoo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the Results See the NeurlPS code and data submission guidelines (https: / ctips.cc//uulii//gu gewse////g////eeeeeeeeeeeeesssee /CodeSubmisst wwtpbbmmsssiioooooooiioo onoolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and Baselines If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
22

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The paper specify the all details of experiment, including the rationale for their selection and underlying model.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: In our experiment, set the temperature of the LLM to 0 for reproducibility, ensuring that their responses were deterministic and free from randomness. And we also utilize fixed test set. Consequently. there the outputs were consistent and invariable across multiple runs. Guidelines:
bars in experimental results, as
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: The paper does not provided resource usage experiment because the method we proposed is LLMss-bsse, which solely involved API calls to these models and is not computationally intensive all. Any computer with network communication capabilities could execute our method.
23

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /
Answer: [Yes]
Justification: The research conducted in the paper conform with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The work performed has no direct societal impact, as it relies solely on calling LLMs and their behavior is fully controlled. The approach does not involve any external factors or unintended consequences that could potentially affect society.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deploymentt of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
24

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators of the data and LLMs employed in this research have appropriately acknowledged and adhered to the respective licenses governing their use for research purposes.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The new assets, including source code and dataset we use, are well documented and the documentation is provided alongside the assets.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemoodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
25

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjeets
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "jgLQuD44gZ", "reviewer": "Reviewer_VnVX", "summary": "This paper proposes visual-augmented prompting (VAP) for large language models (LLMs) in reasoning tasks. Specifically, VAP translates textual questions into a sequence of self-synthesized images using API calls (Python Turtle, Matplotlib, DALL-E3). These images are then fed back to Vision-LLM (GPT-4o) in a step-by-step manner as deduction steps. The detailed VAP process includes (1) planning, (2) iterative reasoning, and (3) conclusive reasoning. Experiments on several math tasks, such as geometry intersection counting, Sudoku puzzles, time series prediction, and the traveling salesperson problem, demonstrate that VAP helps LLMs perform better than chain-of-thought (CoT) and tree-of-thought (ToT) methods.", "questions": "1. What is the chance of visual drawing code failing? How to reduce this probability.\n\n2. Does the model require customized instructional prompts for LLM on different tasks?\n\n3. Can the model generalize to more general Visual QA tasks?\n\n4. Is it possible to compare with VCoT mentioned in Section 2.3 (line 76)?", "limitations": "The authors discuss and address some of the limitations of VAP.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper extends Chain-of-Thought (CoT) with visual prompt information. In addition to the step-by-step textual deduction in CoT, it uses drawing APIs (e.g., Python Turtle, Matplotlib, DALL-E) to synthesize pictures in the intermediate steps, helping to derive the final answer for mathematical problems. This approach is interesting and has not been explored before.\n\n2. Experiments on math-related problems, such as geometry intersection counting, Sudoku puzzles, time series prediction, and the traveling salesperson problem, demonstrate that VAP outperforms CoT and ToT methods.", "weaknesses": "1. When the LLM generates Python API calls, there is a chance that the generated code may not run successfully (bugs in the code). What is the probability of this phenomenon occurring in the experiments?\n\n2. Generalization: CoT and ToT are more generalized to different LLM tasks, while VAP is limited to a few geometry problems. Does it generalize to normal Visual QA tasks?\n\n3. There is a missing reference to relevant works on tool usage ability for LLMs, such as ViperGPT [1], VisProg [2], and LLava-Plus[3].\n\n\n[1]. ViperGPT: Visual Inference via Python Execution for Reasoning\n[2].Visual Programming for Compositional Visual Reasoning\n[3]. LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the constructive comments from the reviewer. We provide detailed responses to each concern in the following.\n\n### W1&Q1: \"When the LLM generates Python API calls, there is a chance that the generated code may not run successfully (bugs in the code). What is the probability of this phenomenon occurring in the experiments?\"\n\nWe understand the reviewer's concern and have added an experiment to assess the code generation error rate. The following table presents the ratio of syntax error and runtime error:\n\n|             | Syntax Error Rate | Runtime Error Rate |\n| ----------- | ----------------- | ------------------ |\n| Geometry    | 0%                | 0%                 |\n| Sudoku      | 0%                | 2.7%               |\n| Time Series | 0%                | 0%                 |\n| TSP         | 0%                | 0%                 |\n\nThe results show that the componnet of LLM-based code generation is sufficiently robust, with no syntax error. This is because the task of code generation for API call is not challenging for LLM, without involving complex logic. For example,  \"a circle center at (1, -1) with radius 2\" is translated to `plt.circle(c=(1, -1), r=2)`. We can observe slight runtime error in the Sudoku task. These errors occurred when translating Sudoku positions to actual coordinates in the image. For example, in a 9x9 board, the grid position \"1 9 1\" should be translated to `plt.text(\"1\", x=0.5, y=8.5)`(placing the text in the center of the grid). However, the LLM occasionally made mistakes like `plt.text(\"1\", x=1, y=9)`, causing out-of-bounds placement. \n\n### W2&Q3: \"Generalization: CoT and ToT are more generalized to different LLM tasks, while VAP is limited to a few geometry problems. Does it generalize to normal Visual QA tasks?\"\n\nYes, we agree with the reviewer that CoT and ToT are more generalized, but their performances are inferior to our VAP in the four diversified reasoning tasks that can benefit from dual-modality reasoning. \n\nIn the current stage, we are unable to support normal visual QA tasks for two main reasons. First, the problem setting is different. The input of visual QA tasks consists of an image and text question. In our setting, the input is a text question. The image is automatically synthesized according to the text input. Second, VAP is designed to improve numeric reasoning, whereas normal visual QA tasks focus on semantic understanding of the input image. Nonetheless, we agree that it would be very impactful if we can extend VAP to be more general to support VQA.\n\n### W3: \"There is a missing reference to relevant works on tool usage ability for LLMs, such as ViperGPT, VisProg, and LLava-Plus.\"\n\nWe thank the reviewer for providing these relevant works. In revision, we incorporated these works in related work and add necessary explanations and discussions.\n\n### Q2: \"Does the model require customized instructional prompts for LLM on different tasks?\"\n\nIn fact, VAP does not require task-specific prompts for different tasks, which we think is a desirable feature. This ows to the planning step, in which the prompt specifies the role, drawing workflow, external image tools, and output format. These elements are identical for different tasks. The prompt also contains the input problem text, which triggers LLM to understand the task and generate different plans accordingly. \n\nThe key components of the prompt template for the planning include:\n\n> Your role is to visualize a problem by creating an image... // Role play\n>\n> The drawing will be executed through an iterative process... // Workflow clarification\n>\n> Here are drawing tools available:... // Introduction of the drawing tools\n>\n> You must strictly follow the JSON format... // Format control\n>\n> Problem Description:... // Problem input\n\nMore details on this prompt can be found in Appendix A.1.1.\n\n### Q4: \"Is it possible to compare with VCoT mentioned in Section 2.3 (line 76)?\"\n\nVCoT was designed for tasks like storytelling and summarization, instead of the numeric reasoning tasks presented in our work. Furthermore, we found that its source code is not available, with an empty page in the provided github link. To address the reviewer's concern, we came up with an idea to apply our VAP to the sotrytelling task as presented in the VCoT paper. The task setting is to write a story based on the input of text prompts.\n\nIn the implementation of VAP, we incorporate DALLE3 as one of the external image tools. Then at each iterative step, VAP generates a photorealistic image using DALLE3 to render a scene. This image is then sent to GPT-4v to enhance the model's creativity in story generation.\n\nAs to performace metric, VCoT adopts human evaluation to judge creativity etc, which we think are subjective. Therefore, for reviewer's reference, we simply provided an example below to demonstrate the results of two approaches.\n\n| Input short prompt                          | Me and my son decided to take a nice walk on the beach.We both were admiring the scenery. Afterwards, we started driving home. |\n| ------------------------------------------- | ------------------------------------------------------------ |\n| Result of VCoT (reported in original paper) | On the way home from school, my son and I enjoyed the beautiful view of the sky and flowers.We had picked up a presentation of flowers from school, which we admired as we drove home.The sky was blue and white over the car, and we enjoyed the scenic view of the flowers and the sky. |\n| Result of VAP                               | The sun dipped below the horizon as my son and I strolled along the sandy shore, admiring the beautiful colors of the sky and the calming sound of the waves.We were both quiet, watching the seagulls fly above us, the interesting shapes in the sand, and the boats far out on the water.As the day ended, we walked back to our car, feeling happy and closer than ever after our special time together on the beach. |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DWODx4BiAg", "reviewer": "Reviewer_L2VS", "summary": "This paper proposes a new prompting technique, vision-augmented prompting (VAP), to improve the reasoning capabilities of large language models (LLMs). Different from the mainstream chain-of-thought (CoT) frameworks that only involve textual reasoning steps, the proposed VAP framework automatically generates images from visual and spatial clues via external tools. In addition, the VAP framework feeds a chain of thought with both textual and visual context into LLMs to solve the original problems. Evaluations on four tasks (i.e., geometry, sudoku, time series prediction, travelling salesman problem) demonstrate that the proposed VAP outperforms prior CoT frameworks.", "questions": "See the Weaknesses section for detailed questions.", "limitations": "The authors mentioned and attempted to address several limitations in the manuscript.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed VAP framework improves the traditional chain-of-thought (CoT) prompting via augmenting visual context. Specifically, the VAP generates high-level drawing planning from the textual question, and then iteratively outputs instructions to draw the stepwise visual context and provide related textual thoughts. The visual contexts with textual context (i.e., thoughts) provide richer information than the text-only context in the CoT framework and helps the reasoning capability of the LLM (MLLM) models.\n\n2. As the iteratively generated visual and textual contexts may contain errors and lead to wrong reasoning, the authors propose a self-alignment mechanism to check if the visual and textual contexts align with the initial high-level drawing planning. If not aligned, the iterative reasoning procedure will be restarted accordingly. This mechanism helps to improve the iterative reasoning and results in a more accurate conclusion.\n\n3. In the evaluation section, the authors conduct a comprehensive comparison between the proposed models and multiple CoT frameworks on 4 versatile benchmarks. In addition, the authors also introduce several task-specific baselines during the comparison and it further validates the effectiveness of the proposed VAP framework. Moreover, the authors conduct human analysis on some generated drawing and observe an impressive integrity rate. These evaluations are helpful to understanding the strengths of the VAP framework.\n\n4. The paper is well written and easy to read.", "weaknesses": "1. In the VAP framework, it chooses one of three tools (matplotlib, turtle, and dalle3), but the manuscript does not provide much details about how these tools are used\n- For matplotlib and turtle, code generation is needed. However, how can the framework guarantee that the code has the right syntax to properly generate the image?\n- For dalle3, how is the prompt generated for the image drawing? \n- What is the distribution of calls on these three different tools?\n- In each iteration of iteratively reasoning, does the selected tool draw a complete image, or overlay the new drawing on image from previous iteration?\n- Can the framework use a mixture of different tools in resolving one problem?\n\n2. In the evaluation, the questions from all benchmarks are provided in text format. I wonder if the authors can include benchmarks with both text and image in question (e.g., VQA benchmarks). This is a fairer setting since the current baselines ignore the visual capability of the GPT4v model.\n\n3. In ablation studies, there is one experiment that removes the planning step. I wonder if the authors can provide more details. Does it mean that a different set of iterative reasoning prompt is used?\n\n4. Typos: Line 294: is plays -> plays / \"Self-alignment plays an important role to the task of Geometry Intersection Problems\" Shouldn't it be \"Sudoku\" since self-alignment achieves the largest improvement (25.1% -> 35.5%) on Sudoku?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the in-depth review! Below, we respond to the weaknesses raised in the review.\n### W1.1:\"For matplotlib and turtle, how can the framework guarantee the syntax correctness of generated code?\"\nWe understand the reviewer's concern. We add an experiment for code generation assessment and find the LLM-based code generation is sufficiently robust to guarantee syntax correctness by itself. The following table presents the ratio of syntax error and runtime error:\n|             | Syntax Error | Runtime Error |\n| ----------- | ------------ | ------------- |\n| Geometry    | 0%           | 0%            |\n| Sudoku      | 0%           | 2.7%          |\n| Time Series | 0%           | 0%            |\n| TSP         | 0%           | 0%            |\n\nHere, LLM produces no syntax errors. Because the code generation for API call is not challenging for LLM, without involving complex logic. For example,  \"a circle center at (1,3) with radius 2\" is translated to `plt.circle(c=(1,3),r=2)`. We can observe slight runtime error in the Sudoku task. These errors occurred when translating Sudoku positions to actual coordinates in the image. For example, in a 9x9 board, the grid position \"1 9 1\" should be translated to `plt.text(\"1\",x=0.5,y=8.5)`(placed in the center of the grid). However, the LLM occasionally made mistakes like `plt.text(\"1\",x=1,y=9)`, causing out-of-bounds placement.\n### W1.2: \"For dalle3, how is the prompt generated for the image drawing?\"\nFor DALLE3, LLM also generates an API call to create the image(OpenAI offers a package for DALLE3).\n\nAs DALLE3 is not selected in the tasks presented in our experiment(detailed in W1.3), we take another task we have previously tried as the example. This task is creative writing task, where LLM is asked to write storied based on some keywords.(We did't put the task in manuscript due to the lack of a convincing performance metric)\n\nExample of this task:\n\nInput: \"Write a story according to given key words: Mage, Warriors, Priest\"\n\nOutput: \"A light breeze swept the ground, ...\"\n\nIn this example, VAP first make a plan like:\n```json\n{\n  \"tool\": \"DALLE3\",\n  \"initialization\": \"dalle3 = OpenAI().images\",\n  ...\n}\n```\nEach iteration, VAP will generate an image using API call. For example, `img = dalle3.generate(prompt=\"The Mage dressed in a dark cloak, holding a staff and surrounded by a magical aura.\", size=\"1024x1024\")`.\n### W1.3: \"What is the distribution of calls on these three different tools?\"\nAs responded in W1.2, DALLE3 is not selected in the four tasks. So, we report the distribution of calls on the tools of Matplotlib and Turtle in the following:\n|            | Geometry | Sudoku | Time Series | TSP  |\n| ---------- | -------- | ------ | ----------- | ---- |\n| Matplotlib | 86.0%    | 91.3%  | 100%        | 78%  |\n| Turtle     | 14.0%    | 8.7%   | 0.0%        | 22%  |\n\nMatplotlib is used more frequently, especially for time series prediction, due to its ability to construct coordinate systems efficiently.\n\nWe've also explored the relation between selected tool and performance, which can be found in response to Reviewer gmDj.\n### W1.4: \"In iterative reasoning, the selected tool draw a complete image, or overlay the new drawings from previous iteration?\"\nIn each iteration, our method uses the drawing tool to overlay the new drawing on the image from the previous iterations. \n\nFor example, consider a geometry problem with two shapes, as follows:\n```Plain\nThere's a circle centered at (3, 2) with radius 1\nThere's a circle centered at (6, 1) with radius 4\nHow many intersection points are there?\n```\nIn the second iteration, the LLM will use the API call `plt.circle(c=(6,1),r=4)` to draw an additional circle on top of the previous image incrementally. \n### W1.5: \"Can the framework use a mixture of different tools in resolving one problem?\"\nUnfortunately, our method does not support using a mixture of different tools during iterative reasoning. The drawing tool is determined in the planning step by LLM and remains fixed to ensure consistency. \n### W2: \"I wonder if the authors can include benchmarks with both text and image in question (e.g., VQA benchmarks). This is a fairer setting since the current baselines ignore the visual capability of the GPT4v model.\"\nWe understand the reviewer's concern regarding the fairness of baseline comparision. It can be challenging to create an absolutely fair setting when baselines only require text ability of VLLM. Our ablation study may provides valuable insights. When the iterative reasoning stage is removed, it downgrade VAP to standard prompting with visual ability. And the results show a clear superiority of VAP over such a baseline.\n\nWe also take the reviewer’s advice to investigate VQA benchmarks. However, we find our task setting differs from VQA. Our work focuses on enhancing text-only problems using the additional image channel, while VQA typically involves answering questions based on provided images. This represents a distinct area of research.\n### W3: \"In ablation studies, there is one experiment that removes the planning step. I wonder if the authors can provide more details. Does it mean that a different set of iterative reasoning prompt is used?\"\nWe apologize for the lack of details. When the planning step is removed, we lose access to meta-information like selected tool, which is used to fill into iterative reasoning prompt template. Therefore, we use an alternative prompt template for iterative reasoning. The key changes to this new prompt will be detailed in the Appendix in revision. The key changes of this prompt include:\n- No specific draw content and thought content given\n> ..., provide your thoughts according to this problem.\n- No specific tool given\n> ... update the image content using Python API calls.\n### W4: \"Typos at Line 294\"\nThank you for pointing these out. We have fixed the typo and it's indeed that Sudoku benefits most from self-alignment. We have revised the statements accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4PsBcKDwas", "reviewer": "Reviewer_vhJU", "summary": "This paper targeting an intresting topi in VL research: Can VLM understand the organized prompts as LLM? The authors proposed a method called VAP(visual augmented prompt) to improve the prompting learning methods for VLM. The authors argue that human have two specialized subsystems that process verbal inforation and visual-spatial information respectively. Thus, to mimic human's decision making capability, the proposed method will synthesize images and organized a chain-of-thought in both modalities. This method comprises three steps: 1. selecting an anppropriate drawing toolkit and creating an initial image; 2. Iteratively perform reasoning on the synthesized images, and generates a paragraph of accompaning text for the generated image; 3. Finally, feed all the intermediate thoughts and images to the model as input to formulate the COT process.\nThis method is tested on 4 different tasks to show its general capability.", "questions": "1. Since the drawing steps and numbers are decided by the planner. What if the plan is quite long, and we are out of tokens? Do we have a control on the granularity on the intermediate drawing?\n2. How frequent the generated images will content incorrect information? And how well the self-alignment module detect some very specific errors?", "limitations": "Not applicable", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Though COT and prompt learning is not a novel topic in LLM. How to organize and build a CoT process for VLM to effectively activate the reasoning skill for VLM. While most recent work focusing on leveraging the language side to generate the rationale process. This work focusing on how to coordinately generate both visual and language rationale. This is a quite good idea.\n2. The proposed method reflect the human decision process and thus sounds quite reasonable and novel.\n3. The proposed method increase the results on different tasks by a large margin, even compare to many advance prompting and CoT styles. These solid increment prove the effectiveness of this method.", "weaknesses": "1. As mentioned by the authors. The image generation process not always controllable and could generate error-pone content to mislead the thinking process. Therefore I wonder, is generating a explicit image the best practice for this process. As the generation of images could result in unwanted features, we could also keep the representation in the latent space as a prompt.\n2. The method relies on a planner to plan what to draw for the whole process. However, a planner itself can be incorrect and we simply have no control on this planner and the generated plan. Is there a more formulated way to generate the plan, or do we have somewhat of tools to detect the possible problem?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the constructive comments from the reviewer! We provide detailed responses to each concern in the following.\n\n### W1: \"I wonder, is generating an explicit image the best practice for this process. As the generation of images could result in unwanted features, we could also keep the representation in the latent space as a prompt.\"\n\nThis is a really inspiring idea! In the current stage, commercial VLMs we used in this paper have not provided the API interface that allows us to use encoded image feature in the latent space as part of the input. We consider it as our future work to examine whether we can implement the idea on other open-sourced VLMs.\n\n### W2: \"Is there a more formulated way to generate the plan, or do we have somewhat of tools to detect the possible problem?\"\n\nWe appreciate your constructive feedback. We totally agree that a more formalized approach to generate the plan would likely improve the performance and robustness of our method.\n\nThe ideas we currently have include:\n\n- Format control: we can utilize the open-source repository lm-format-enforcer (https://github.com/noamgat/lm-format-enforcer) to strictly formulate the output JSON schema. This tool uses a prefix tree decoder during the streaming output of the language model to enforce the output format.\n- Reflection mechanism: recent works, such as Reflexion, have shown that LLMs can self-reflect on previously generated answers and produce more reasonable responses. This mechanism can be applied on the planning stage, which could help detect potential problems.\n- Controllable parameters in planning prompt: we can introduce controllable parameters in the prompt of planning step, which would provide users with more control over the generated plan and would also contribute to performance (also proved in the experiment in Q1.2).\n- Planning decomposition: for complex tasks, decomposition can be an effective strategy. Techniques like ToT or ReAct could be applied to generate a more reliable plan.\n\n### Q1.1: \"Since the drawing steps and numbers are decided by the planner. What if the plan is quite long, and we are out of tokens?\"\n\nToken length is indeed a common issue in LLM-based reasoning tasks. So far, the maximum number of tokens required by the planner of VAP is 2841, which is safely below the 8k token context limit in the default GPT-4 setting.\n\n### Q1.2: \"Do we have a control on the granularity on the intermediate drawing?\"\n\nThank you for your insightful comment. Currently, the granularity on drawing is determined by planner and we don't provide access to modify it. Inspired by your feedback, we explored the possibility of controlling granularity to enhance drawing efficiency.\n\nWe conducted an experiment on Sudoku task by controlling the number of iterative steps (which is equivalent to controlling the granularity of drawing). Specifically, we inject the prompt \"You must finish within `n_iterations` by drawing multiple rows in parallel\" into the prompt of  iterative reasoning.  `n_iterations` is set to 8, 4, 2, and 1 to investigate performance. The results are as follows:\n\n|                        | Time usage | Correct rate |\n| ---------------------- | ---------- | ------------ |\n| VAP (original)         | 19.0 s     | 35.5%        |\n| VAP (n_iterations = 8) | 17.7 s     | 35.3%        |\n| VAP (n_iterations = 4) | 15.9 s     | **37.3%**    |\n| VAP (n_iterations = 2) | 12.3 s     | 26.6%        |\n| VAP (n_iterations = 1) | 4.4 s      | 19.9%        |\n\nIt is interesting to find that set `n_iterations` to 4 improve performance over the original version and also enhancing efficiency. This suggests that more iterations do not necessarily lead to higher accuracy. When `n_iterations` is set to 1, the iterative reasoning process is almost skipped, resulting in poor performance. These findings support your previous comment (W2) that a more formulated generation approach is beneficial and worth further exploration. We appreciate the reviewer's comment for pointing out this.\n\n### Q2: \"How frequent the generated images will content incorrect information? And how well the self-alignment module detect some very specific errors?\"\n\nWe analyzed the frequency of generated images containing incorrect information and report the results in the following table:\n\n|             | Correct rate without self-alignment | Correct rate with self-alignment |\n| ----------- | ----------------------------------- | -------------------------------- |\n| Geometry    | 77.0%                               | 83.5%                            |\n| Sudoku      | 72.0%                               | 89.3%                            |\n| Time Series | 100.0%                              | 100.0%                           |\n| TSP         | 98.0%                               | 98.0%                            |\n\nWe define an image as \"correct\" if it contains the integral information described in the text. From the result, we observe that the geometry and Sudoku benefited significantly from self-alignment. In geometry task, LLM would occasionally initialize coordinates with the wrong range, causing the shapes outside of the view. In Sudoku, LLM may have difficulty translating position descriptions to image coordinates. For example, in a 9x9 board, the correct API call grid position \"1 2 4\" should be translated to `plt.text(\"1\", x=1.5, y=3.5)`(placing the element in the center of the grid). However, the LLM occasionally make mistakes, such as:\n\n- `plt.text(\"1\", x=2, y=4)`, placing at the bottom right corner\n- `plt.text(\"1\", x=1, y=3)`, placing at the top left corner\n- Missing this element\n\nThese mistakes can be detected by self-alignment because the LLM will describe the content of the image and check it against the original input, which is proved to be crucial in our ablation study.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rJ7Jq7EKNS", "reviewer": "Reviewer_gmDj", "summary": "Traditional large language models (LLMs) struggle with tasks requiring visual and spatial interpretation based solely on text. This study introduces a visual-augmented prompting (VAP) strategy, using an external image generation tool to iteratively create intermediate visual representations that aid reasoning. VAP's effectiveness is validated in four tasks: (1) Geometry Intersection Counting, (2) Sudoku Puzzles, (3) Time Series Prediction, and (4) the Traveling Salesman Problem.", "questions": "For weakness (2), the author claimed that using GPT-4V as the foundational model for the baseline methods ensures fairness. However, VAP requires an MLLM to handle visual-text input, whereas all other baselines only require textual input. I believe traditional LLMs should outperform MLLMs for text-only tasks. Therefore, the author is expected to incorporate a wider range of LLM models (such as LLaMA 3, GPT4) for comparison.\n\nI am curious about the potential of VAP in solving real-world reasoning problems, where the model needs to generate more photorealistic images (not just diagrams) to benefit reasoning.", "limitations": "The author has a section that addresses most of the limitations of the VAP method. However, I suggest that if the VAP method is time-consuming in terms of the iterative figure drawing process, the author should discuss the efficiency problem in the limitations section.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "(1) The VAP method is simple yet effective and well-motivated, aligning with human cognitive processes. It promisingly integrates both intermediate textual and visual results to enhance the accuracy and interpretability of the reasoning process. \n\n(2) Results clearly show that VAP significantly outperforms all baselines across four tasks.\n\n(3) The manuscript is well-written and easy to follow.", "weaknesses": "(1) Efficiency of Iterative Reasoning: The author does not mention the time consumption of the iterative reasoning process of VAP. Additionally, I am curious about the trade-off between the number of images that need to be drawn and model performance.\n\n(2) More LLM models (such as LLaMA 3, GPT4) need to be incorporated for more comprehensive comparison.\n\n(3) Effect of Different Figure Drawing Tools: I am interested in understanding how different graphic rendering tools for image drawing affect the final reasoning results.\n\n(4) Method Scalability on Complex Geometry Problems: Figure 4 shows a significant accuracy drop for VAP when the number of shapes exceeds three. This raises concerns about the scalability of VAP in handling complex geometry problems.\n\n(5) Although VAP performs better than the baselines, its overall performance is still poor and has a noticeable gap compared to traditional methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the constructive comments on our work, which have helped us to enhance the paper. The following are our detailed responses to each comments.\n\n### W1.1:Efficiency of iterative reasoning\n\nThis is a good point. We added an efficiency experiment for VAP in our revision. The results for the geometry and Sudoku tasks are shown below:\n\n|               | Geometry |          | Sudoku |              |\n| ------------- | -------- | -------- | ------ | ------------ |\n|               | Time     | Accuracy | Time   | Correct rate |\n| Standard      | 0.2 s    | 8.5%     | 0.3 s  | 18.0%        |\n| CoT           | 0.5 s    | 10.0%    | 0.8 s  | 17.3%        |\n| CoT-SC (k=5)  | 2.3 s    | 11.0%    | 4.1 s  | 20.6%        |\n| CoT-SC (k=10) | 4.5 s    | 11.5%    | 8.8 s  | 20.6%        |\n| ToT           | -        | -        | 9.0 s  | 22.6%        |\n| VAP           | 4.1 s    | 16.5%    | 9.5 s  | 35.5%        |\n\nWhile VAP is indeed time-consuming, it achieves comparable time usage to ToT and CoT-SC (k=10) in the Sudoku task. The reason is that, although VAP's context is quite large (tool instructions, thought trajectory, image encoding), its output is concise (API calls and immediate thoughts), leading to fast inference during each step (time usage is mainly affected by the number of decoded tokens). We believe the time usage is acceptable given VAP's superior effectiveness.\n\n### W1.2:Trade-off between the number of images and model performance\n\nThis is an inspiring comment. Currently, the number of images is determined by the planner, and we don't provide access to modify it. To address the reviewer's concern, we add an experiment of controlling the number of images. The results of this experiment can be found in our response to all reviewers.\n\n### W2/Q1:More LLM models need to be added for comprehensive comparison\n\nWe follow the reviewer's advice and add GPT4 and LLaMA 3 8B as additional LLMs for more fairer comparison in our revision. The results can be found in our response to all reviewers. Based on results, we believe that the superiority of VAP over other baselines remains valid, even when utilizing different LLMs.\n\n### W3:Effect of different figure drawing tools\n\nWe sincerely thank your comment and it actually pushes us to think deeply about the relationship between tool selection and performance.\n\nFirst, we analyze the distribution of drawing tools selection:\n\n|            | Geometry | Sudoku | Time Series | TSP  |\n| ---------- | -------- | ------ | ----------- | ---- |\n| Matplotlib | 86.0%    | 91.3%  | 100%        | 78%  |\n| Turtle     | 14.0%    | 8.7%   | 0.0%        | 22%  |\n| DALLE3     | 0.0%     | 0.0%   | 0.0%        | 0.0% |\n\nMatplotlib was used more frequently, especially for time series prediction, due to its ability to construct coordinate systems efficiently.  DALLE3 is never selected by planner as the images of four tasks are all diagrams (we have explored another use case of DALLE3, detailed in Q2).\n\nSecond, we are also interested in which tool perform better for these tasks. We conduct an experiment where we forced VAP to choose a specific tool. The results are as follows:\n\n|                 | Geometry↑ | Sudoku↑   | Time Series↓ | TSP(N=10)↓ |\n| --------------- | --------- | --------- | ------------ | ---------- |\n| Original        | 16.5%     | 35.5%     | 556          | 312.4      |\n| Matplotlib only | 16.5%     | **36.6%** | **556**      | 312.6      |\n| Turtle only     | 16.5%     | 19.0%     | 611          | **312.1**  |\n\nWe find that the impact of different tools depends on the task. For geometry and TSP, using Matplotlib or Turtle makes no significant difference, while in Sudoku and time series forecasting, Matplotlib is better than Turtle. Because drawing grids and coordinates are too complicated for Turtle (draws by pen movement). The results also reflect the rationale behind the LLM's tool selection, as it automatically avoids choosing tools that are not suitable for a given task.\n\n### W4: Scalability of VAP\n\nSolving complex reasoning problems has been a well-known weakness of LLMs and this domain has attracted significant attention because once the challanges can be overcome, many applications can well benefit from the power of LLMs. So, it is not surprising to find that the accuracy drops when facing challenging reasoning problems. Compared with other LLM-based reasoning frameworks, our VAP still demonstrates the best performance. The performance gap is even widened when the problem becomes more challenging.\n\n### W5: Although VAP performs better than the baselines, its overall performance is still poor and has a noticeable gap compared to traditional methods\n\nWe totally agree with the reviewer that the current performance of LLMs is still not comparable to traditional methods in many tasks. Its strength lies in generality, with one model to support vairious decision and prediction problems. Thus, how to exploit the potential of LLMs and improve its capabilities in reasoning and prediction has become a hot research topic in recent years. Our work falls in this category and we believe with more efforts devoted in this domain, the performance gap may continue to shrink.\n\n### Q2: The potential of VAP in solving real-world reasoning problems, where the model generates more photorealistic images\n\nWe've actually explored VAP's potential in a real-world sotrytelling task. The task setting is to write a story based on the input of text prompts.\n\nIn the implementation of VAP, DALLE3 is always selected as the external image tool. Then at each iteration, VAP generates a photorealistic image to render a scene. This image is then sent to GPT-4v to enhance the model's creativity in story generation.\n\nOur previous findings show that, based on our subjective judgement, VAP can significantly benefit the task. We did not put this task in the manuscript due to the lack of a convincing performance metric.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all detailed feedbacks provided by the reviewers! We address a few points in this response. All other questions are addressed in reviewer specific responses.



### Trade-off between number of iterations and performance

We add an experiment of controlling the number of iterations in Sudoku task. Specifically, we injected the prompt "You must finish within `n_iterations` by drawing multiple rows in parallel" into the prompt of iterative reasoning. `n_iterations` is set to 8,4,2 and 1. The results are as follows:

|                    | Time   | Correct rate |
| ------------------ | ------ | ------------ |
| VAP (original)     | 19.0 s | 35.5%        |
| VAP (iterations=8) | 17.7 s | 35.3%        |
| VAP (iterations=4) | 15.9 s | **37.3%**    |
| VAP (iterations=2) | 12.3 s | 26.6%        |
| VAP (iterations=1) | 4.4 s  | 19.9%        |

It is interesting to find that set `n_iterations` to 4 improve performance over the original version and also enhancing efficiency. This suggests that more iterations do not necessarily lead to higher accuracy. When `n_iterations` is set to 1, the iterative reasoning process is almost skipped, resulting in poor performance.



### Baselines comparasion with different LLMs

Our experimental setup employs a unified VLLM. However, we observed that VAP necessitates a MLLM to process visual-text input, whereas other baselines solely require textual input. Considering that traditional LLMs are expected to outperform MLLMs in text-only tasks, we introduced GPT-4 and LLaMA 3 8B as additional LLMs to ensure a more comprehensive comparison. Accuracy on the geometry task is presented as follows:

|               | GPT-4v    | GPT4  | LLaMA3 |
| ------------- | --------- | ----- | ------ |
| Standard      | 8.5%      | 10.0% | 7.0%   |
| CoT           | 10.0%     | 11.0% | 8.0%   |
| CoT-SC (k=5)  | 11.0%     | 11.5% | 8.0%   |
| CoT-SC (k=10) | 11.5%     | 11.5% | 8.0%   |
| CoT-SC (k=20) | 11.5%     | 11.5% | 8.0%   |
| VAP           | **16.5%** | -     | -      |

Here, we can get these conclusions:

- GPT-4 slightly improved baseline performance compared to GPT-4v, but still significantly lower than VAP.
- LLaMA3 8B shows decreased accuracy, likely due to its small size limiting generalizability. Note that currently we are unable to run larger versions of LLaMA3 due to machine constrain.
- Simpler methods (standard prompting) benefit more from model changes than complex methods (CoT-SC).

 From the results, we maintain that the superiority of VAP over other baselines remains valid.

Meta-review: This paper proposes visual-augmented prompting (VAP) framework that automatically translates textual questions into a sequence of self-synthesized images from visual and spatial clues using external tools with API calls (Python Turtle, Matplotlib, DALL-E3). These images are then fed back to Vision-LLM (GPT-4o) in a step-by-step manner as deduction steps. The detailed VAP process includes (1) planning, (2) iterative reasoning, and (3) conclusive reasoning. Experiments on several math tasks, such as geometry intersection counting, Sudoku puzzles, time series prediction, and the traveling salesperson problem, demonstrate that VAP helps LLMs perform better than chain-of-thought (CoT) and tree-of-thought (ToT) methods.

In general image generation process is not always controllable and could generate error-pone content to mislead the thinking process. During rebuttal, the authors analyzed the frequency of generated images containing incorrect information. From the result, it is observed that the geometry and Sudoku benefited significantly from self-alignment. In geometry task, LLM would occasionally initialize coordinates with the wrong range, causing the shapes outside of the view. In Sudoku, LLM may have difficulty translating position descriptions to image coordinates
Decision: Accept (spotlight)